Download Stats Extreme Makeover - The Value of XSLT 2.0 and XQuery 1.0

On Friday, I had the opportunity to use XSLT 2.0 and XQuery 1.0 in a way that proved the value of the new standards.

For our Beta programs, we get a weekly report of how many downloads have occurred. Over time we have asked for more and more breakdowns of the data (how many were from IBM'ers vs. non-IBM'ers, how many were from non gmail/hotmail/etc email addresses, how many for code vs. how many for documentation, how many unique users have downloaded, etc.). We use these reports to gauge the interest in our betas and the effectiveness of our beta programs.

Currently the process for doing these reports is:

1. Load a webpage somewhere on the intranet that returns in either HTML or space and newline separated text the download records (and I don't have the time nor contacts to change this "service").
2. Import these download records into Excel
3. Write VBScript that processes the rows into summary tables, but not all summaries have VBScript written for them due to the fact that VBScript can get quite complicated.
4. Read the summary tables and hand compute some summaries and transpose them into emails and presentations to beta teams.

This process can take a few hours and is error prone.

I asked for the raw data for the service as I knew there had to be an easier way.

First I started with the HTML as its pretty much XML and I figured I could just write some simple XQuery summaries. This failed as HTML isn't XML (if its not XHTML). Don't get me started on that rant (in this case, we had things like width=-1 with no quotes)! So I restarted with the text file version of the data.

XSLT 2.0 has a new function - unparsed-text() that lets you load from text files. This allows you to load any data into XSLT 2.0, but only as a string. As the XSLT 2.0 specification shows the real power of this function is magnified by XPath 2.0's support of regular expressions. Combining the unparsed-text() with regex tokenize() with newline as the separator allowed me to for-each across every line in the file. Adding another new XSLT 2.0 function into the mix - analyze-string() with a regular expression that did capturing of each of the fields allowed me to transform each line into a well formed XML element with sub-elements for each of the fields (download element with elements for filename, email address, etc). Ah, now I have data in a well formed structured XML format. Life is good.

Once I had this transformation of the input data, I returned to the task of creating the summaries. With XQuery, any query I want is just moments away. To prove the point, I decided to write a simple query that would break down the downloads for IBM'ers and non-IBM'ers separately. I would also group each unique user and list all the downloads that user had done with summaries at all levels of how many code vs. how many documentation downloads had occured. I want to put this in XML format, so others could query my summaries or I could put the summaries into HTML or load back into Excel (if you really really want to).

It took me approximately 30 minutes to create all of the above queries. The code is approximately 20 lines long and is well designed into re-usable functions that could be customized later. The code is easy for most of my peer Java developers to understand as XQuery looks alot like an imperative language with SQL like queries mixed into templating of output.

My next step is to move this to a web application (what I did so far was prototyped it with sample data) that connects directly to the service. The web application could easily offer up web forms that allowed the user to specify search criteria supported by the back end service (date range for example) along with what XQuery/summary view was required.

There are still some items that can't be automated and need human intervention. As an example, the process of deciding what constitutes an IBM'er is complex as some IBM'ers have "ibm.com" email address and some do not. Also, IBM could be "IBM, International Business Machines, or mistyped". It would take a bit of time to create some services that approximate what a human eye could spot manually. Adding a human facing pop-up that allowed visual inspection of the automated data analysis steps would be valuable. Also, I admit that I didn't create the charts and graphs - just the raw data that could be loaded into Excel to create charts and graphs.

But in the end I have converted a manual error prone data processing scenario into an automated approach (for data query and summaries) that creates all the same valuable raw data for reports with the potential to add more reports much more quickly. All of this was made possible using well documented W3C standards that have all the needed features (some new with XPath/XSLT 2.0 and XQuery 1.0) that make this scenario possible.

WebSphere Application Server for Developers - Now downloadable, No charge

Want to try out WebSphere Application Server V7.0 for development? Check!
Want to do it without having to order a CD? Check!
Want to do it without having to work with purchasing? Check!
Want to do it by yourself as a developer? Double Check!

I talk to many developers who want to install WebSphere Application Server for development, but every license is tracked and installed by their operations team on some remote server. Working with the operations team and (gasp) purchasing would take more time than the time to write the application they wanted to write. So they go off and download some other runtime and later port the application back to WebSphere Application Server or abandon the project altogether.

This won't be the story after today. Go here to download a no-charge copy of WebSphere Application Server for your development needs.

From the download page:

Download WebSphere Application Server for Developers, a no-charge offering that satisfies the need for a no-cost WebSphere Application Server development runtime for projects that don't warrant the expense of a priced and supported runtime on the developer desktop. Reduce testing effort and develop with confidence using a runtime environment that is identical to the production runtime environment your applications will eventually run on.

Enjoy!

WAS V7 XML Feature Pack Beta Refreshed (With focus on XSLT 2.0)

Today we have released an updated XML Feature Pack Beta. The beta is available here.

Since the first release two months ago (which had full XPath 2.0 and partial XSLT 2.0/XQuery 1.0), we have added quite a bit to XSLT 2.0 (making us nearly spec complete) and more to XQuery 1.0.

Among the biggest features added in the XSLT 2.0 area is XML Schema awareness. This includes validation which allows you to validate (make sure it contains data that is allowed per the schema) the incoming data to transformations, temporary trees of data, and output data from transformations. Also, as is important when working with vertical industry standard schemas, our schema awareness support includes schema assessment. This means that your XSLT stylesheets know when an attribute or element content is an integer or date and you can treat it as such. Taking this to the next level, if you extend your industry standard schema type of a bill of materials, you can write templates that match that type (schema-element(*, tns:BillOfMaterialsType)). In the past you would have had to made templates that were of the industry standard element names as well as all extending element names (tns:BillOfMaterialsName, mycompany:FullfillmentBillOfMaterialsName, mycompany:PurchasingBillOfMaterialsName, etc.).

Other new XSLT 2.0 features in this refresh are collections support (allows multiple input data sources in a well defined way), "undeclare-prefixes" serialization parameter(simplifies namespace in XML 1.1), next-match (allows you to call overridden templates) element, required parameters and "as" attributes (helps guarantee stylesheet elements are called correctly) and full support of XSLT keys.

Another major feature of this refresh is ability to pre-compile XML artifacts. This refresh contains tools that compile XPath, XSLT, and XQuery to java classes. These classes are basically the same logic that previously was interpreted by the XML runtime. However, with them now being compiled, they run much faster than interpreted. The refresh offers three modes:

• interpreted - best when the XML artifacts are created ad hoc at runtime and not reused


• compiled - best when the XML artifacts are created ad hoc at runtime but reused


• pre-compiled - best when the XML artifacts are known at compile time of your application

We have also expanded our XQuery 1.0 support in this refresh.

Please let me know if any of these new features help you in your applications. I'm excited to hear of all the applications that are being tested against this functionality.

Finally, I have created another three demos to be posted on YouTube. Over the couple of days, I'll post them with links here. Hopefully they will help you understand how to use this functionality in your own applications.

Rational Application Developer

I have not posted to this blog for a while, and I can now explain why. Today, at the WebSphere Technical Conference in Vienna, we announced the upcoming availability of an open beta for the next major version of Rational Application Developer (RAD 7.5). Those of you who've read my previous posts (hi Mom!) will not be surprised by the themes we're focusing on for RAD 7.5; better support for the complete application development lifecycle, including refactoring operations and quickfixes that cover the complete Java EE programing model.

The RAD 7.5 beta also provides support for the WebSphere EJB3 feature pack, the beta for which is available now for WebSphere 6.1. The RAD support for EJB 3.0 allows us to really exploit annotation based programming models. Previous versions of the spec focused on deployment descriptors, and the matching RAD releases had advanced editors to support that. In RAD 7.5, the focus is on Java developers, and using the Java editor as a base - but extending the already powerful Eclipse editor by adding significant new function and capability to support developing an annotation based application.

The intent is simple; most developers creating applications for WebSphere are quite comfortable using Java, and we have a great Java editor in RAD/eclipse - so it's a mistake to force them to switch to a whole new environment (with wizards and graphical editors) when they want to define some aspect of server interaction (such as Web Services or EJBs). This is a natural extension to what the spec defines, but the way we're supporting that pattern within RAD is new, easy to use, and productive.

The other aspect that I'm thrilled about is just the fact that we're running an open beta, less than a year after we shipped the last major version of RAD. We're not done yet, and we have more good ideas that we're still implementing - but the code is great quality, and the performance is good... and this shows a commitment to delivering a rock solid, robust, and well performing RAD. The intent is to encourage developers to explore the product, and not feel concerned that they'll get in trouble if they try something non-standard or off the golden path.

I'm really pleased with the progress the team has made on RAD 7.5; this is going to be the best version of RAD we've ever delivered, and the open beta gives you a chance to see where we're at, comment on what you see, and help us refine the product to make it exactly what you need. You can sign up for the beta here, and there will be newsgroups available to report problems, ask questions, and give us your feedback.

Tim Francis

Simple Ajax

I've been at a couple of conferences recently, and one of the hot topics that everyone is talking about is "Web 2.0". The great thing about these cool buzzwords is that they mean different things to everyone - but "Web 2.0" generally implies web sites that provide a richer, more interactive experience. In turn, one of the technologies that is used to provide this experience is Ajax (Asynchronous Javascript and XML). Ajax is not new, the technologies have actually been around for many years, but the pattern has recently been popularized by modern browsers and high profile applications such as Google Maps, GMail, etc.

AJAX is not a framework or library, it is an application pattern (although many frameworks are emerging to help, such as Dojo). An Ajax enabled page uses Javascript to send an XML request to call back to the server, the server asynchronously completes the request and sends back results, and then more Javascript in the page updates some parts of the page (rather than refreshing the entire page).

The problem is that developing AJAX based applications is hard - it requires writing complex Javascript and DHTML, and the asynchronous nature of the pattern further complicates debugging. There are also a series of other problems that have solutions - but are easier to get wrong, than they are to get right: Making the browser's 'back' button work is tough; Javascript has a tendency to leak memory; you must test your code on many browser variants, etc. As noted, frameworks that will help are starting to emerge, most notably Dojo - but they are still immature, and still require that you write JavaScript code in your page.

WebSphere and RAD have supported JSF (Java Server Faces) based applications for several releases now, and the combination of JSF and the JWL component library we ship offers a powerful combination that simplifies the development of applications. What's new is that RAD v7 has introduced some common AJAX behaviors into our JSF component library; you can now provide capabilities such as scrolling data tables with no page refresh, dynamic menus, and entry fields that support type ahead. The nice part is that none of this function requires you (the developer) to write any JavaScript - the bulk of the function is encapsulated in the JWL component library, and the tiny snippets that are required can be generated for you by RAD. In turn, this means you get to avoid most of the problems listed above - but still provide the enhanced interaction that people are starting to expect from modern web pages.

JSF based pages can co-exist quite happily with "normal" JSP and HTML pages, so there is no requirement for you to make a huge investment and redo your entire site. Instead, you can update just the pages that you wish to modernize, and provide the basic interaction patterns that are the hallmark of "Web 2.0". There are several resources available to help you understand the new capabilities, but a couple of articles on developer works provide a good starting point. The first one is "JSF and Ajax: Web 2.0 application made easy" and provides a general overview, and a second article titled "Improve the usability of Web applications with type-ahead input fields" walks you through adding typeahead to a JSF based page.

As Ajax continues to mature, frameworks such as Dojo will continue to add value to the development of advanced web pages - but the capability that's available in RAD v7 and WAS 6.1 today is more than enough to allow you to provide your customers with a modern, interactive site, without forcing your developers to go through a painful development process.

Tim Francis