Joint WAS XML Feature Pack and DB2 pureXML Article on FpML Lives

I've mentioned this work a few times on the blog, so I wanted to make sure people saw the final article on DeveloperWorks.

Programming XML across the multiple tiers, Part 2: Write efficient Java EE applications that exploit an XML database server

This article uses the example of Financial Products Markup Language (FpML) to show how to program realistic native XML across the Application Server and DB2 pureXML. It shows how you can use one consistent programming model (XQuery) and one consistent data model (XML) across data access, transformation, and filtering across both the mid tier and database tier. Using this one data model which doesn't require mapping to Java objects should increase the agility of your XML centric applications as no mapping code needs to be generated or maintained across both tiers.

Even though the article is based upon FpML (which is really useful to the financial sector), the concept is applicable to all industries that have substantial amounts of XML data.

The article has code attached (both a Rational Application Developer ear project and Eclipse/ANT builds) and database load scripts, so you can play with the code to see how it works. All you need to do is define the JDBC resources to point to your DB2 instance. I also have a virtual image for this based upon the free of charge WebSphere Application Server For Developers and DB2 Express-C in case you're interested.

Humor for Friday - Java-4-ever

I don't want to post funny videos here too much as I think people come here for technical information. However, this is just too good to not share. There is one slightly NSFW (not safe for work) section. Even mentions XML, so its truly great.

Watch the Java 4-Ever Video

Now on YouTube

The Cure for XML in Web 2.0?

Earlier, I blogged about the Pain of XML in Web 2.0. I alluded to not being happy with the answer I ended up with. I'm happy to say that I'll finally be talking about a possible solution. As you see here, I have submitted a paper for the Balisage 2010 Conference entitled Where XForms meets the glass: Bridging between data and interaction design along with Charles Wiecha and Rahul Akolkar.

Here is the abstract:

XForms offers a model-view framework for XML applications. Some developers take a data-centric approach, developing XForms applications by first specifying abstract operations on data and then gradually giving those operations a concrete user interface using XForms widgets. Other developers start from the user interface and develop the MVC model only as far as is needed to support the desired user experience. Tools and design methods suitable for one group may be unhelpful (at best) for the other. We explore a way to bridge this divide by working within the conventions of existing Ajax frameworks such as Dojo.

Interested? Let me know and we can get a review copy of the paper to you. I have talked to many clients that want to integrate their meta data driven XML dominant data to the Web 2.0 work with DOJO and run into the impedance mismatch wall. Hopefully that wall will be coming down soon.

BTW, if you'd like to attend this great conference to hear about this topic and many others on the jam packed agenda, here is a great link to use to convince your management to let you join us in Montreal.

Video Blogs on Impact 2010 Sessions Next Week

Are you heading to IBM Software Impact 2010 next week? I hope so!

I will be presenting on WebSphere XML Strategy. I will be presenting a session on the WebSphere Application Server Feature Pack for XML talking about usage scenarios, how to use the feature pack, and best practices (Sessions 1635A/B on Monday and repeat Thursday). I will also be providing a general education session on XPath 2.0, XSLT 2.0 and XQuery 1.0 (Session 1634 on Tuesday) focused on basic education as well as noting whats new with the standards - with a cool give away! I will be assisting a lab where you can get hands on experience with the XML Feature Pack and the Rational Application Developer tools (Session 1606 on Monday). Stop by any of these session or hit me up on twitter (@aspyker) if you have any questions about XML or data strategy within your enterprise.

Along with the XML Feature Pack and XML strategy talks, I'll be participating in a SOA and BPM Performance update (1321) talking about SPEC SOA and multiple panel discussions around the values of the application server, performance, and feature packs.

You can hear me talk about the sessions here:



Some other videos about sessions next week are available as well.

Erik Kristiansen on RESTful, dynamic scripting, and OSGi programming models:



Lan Vuong on extreme transaction processing and elastic application architectures:

XML Feature Pack 1.0.0.3 Available

I'm happy to announce another major update to the XML Feature Pack and its impact on our development tooling - Rational Application Developer for WebSphere.

We just released the 1.0.0.3 version of the XML Feature Pack that has two major new features (as well as some small bug fixes). The two new features are:

XQuery Schema Awareness

In the initial release we had Schema Awareness for XSLT 2.0. In this release we add similar function to XQuery. Specifically this means we started to support the optional XQuery 1.0 features of schema import and schema validation. With these features you can use your own type information in XQuery programs. A common scenario would be looking for all addresses in an input document, regardless of they were "billingAddress" or "shippingAddress". Programming based on type information is a powerful concept that leads to more flexible implementations. Also, validation allows you to validate incoming documents, xml trees and whole output documents. This allows for greater reliability in your XML processing.

Debugging support for XSLT 2.0 under Rational Application Developer (RAD) for WebSphere

Previously with RAD you could debug XSLT 1.0 stylesheets. With this new release of the XML Feature Pack and with RAD 7.5.5.1 you can debug XSLT 2.0 stylesheets. This isn't just about moving to a newer stylesheet level. With all the changes in the data model and advanced new concepts like grouping, there are many improvements in visualization with debugging over the XSLT 1.0 debugger.

What is also interesting is that this is a converged debugger. While there are other XSLT 2.0 debuggers out there, they only work on the stylesheet itself. With this support in RAD, you can debug not only the stylesheet, but also the Java code in your web application that invokes the XSLT engine along with any Java extension functions you might have. If you are using XSLT 2.0 in the application server, this is the tool you want for debugging end to end.

I hope to do a video demo of this Rational Application Developer functionality. Imagine setting breakpoints in XSLT as well as Java and being to jump between them. Anyone interested in seeing such a video demo?

Have fun with the new functions!

Programming XML Across Multiple Tiers

In the XML Feature Pack, we ship a sample that shows how to use XML centric programming in the middle tier. The sample shows how to unlock data in Web 2.0 ATOM XML encoded feeds using XQuery and present the data in a typical web application using XSLT. As an extension to this sample, we also have a sample that shows how to persist data from these feeds into an XML Database such as DB2 pureXML or Apache Derby. We included this example as we found, frequently, that people working with XML centric programming typically had large XML datastores in XML centric databases.

While the sample is there, with source code, in the XML Feature Pack, we don't explain why we coded the sample the way we did. In this new developerWorks article (Programming XML across the multiple tiers: Use XML in the middle tier for performance, fidelity, and development ease), we go into detail why for simplicity, performance, and flexibility reasons we coded the sample the way we did.

The article is worth a read. It will walk you through the new features in the XML Feature Pack and JDBC 4.0 that allow an end to end native XML programming model across the XML Feature Pack and an XML database. We hope to expand this article over time to cover more advanced concepts when working with XML databases.

Finally, here are two quick videos that show how to get the sample working with DB2 pureXML and Apache Derby.

DB2 pureXML (Part 1/2)
Direct Link (HD Version)

Apache Derby (Part 2/2)
Direct Link (HD Version)

The pain of XML in Web 2.0

I have continued to think about the end to end XML story across database, middle tier, and the browser client. I have talked to many organizations that work with standard industry XML documents (HL7, OAGIS, ACCORD, etc) where the XML view unifies the data of their entire enterprise. To these organizations, they work data out from the message queues to data storage to middle tier. However, what does it look like when they want to expose this data to the web tier? There are products that handle this well like Lotus Forms based on XML centric standards like XForms. But what about Web 2.0 libraries like DOJO or jQuery?

I took the download sample I described here, and tried to visualize the data to Web 2.0 webpages. The data format of the XML of interest is:



First, I looked at the DOJO bar chart code and did something like this in a server side XQuery program that generated HTML with the following under JavaScript:



This works as XQuery can return sequence of primitive types and in this case, I'm just returning a string and inserting it inside of the JavaScript code that expects value/text values. But what if I want to have a REST endpoint serve up XML directly and have the browser consume it?

DOJO DataGrid can read from a DataStore which can be hooked to an XmlStore. This means I can use a browser side control to read from my server side XML. All seems good until you get into the details. Here are snippets of the code to make this "work":



What are the some of the issues with this? First, the XmlStore has to map to a simpler format for the DataGrid to understand the XML data. That is why I had to manually tell the XmlStore to promote all the attribute values to similarly named element names. Nicely, the XmlStore supports allowing the ability to drill down to something other than the root item for the data, but it really just allows you to pick the name of an element (you'll see I specified "month"). The second problem is that for any complex industry specific data, likely that wouldn't be sufficient. What if I had multiple month elements at different parts of the XML tree? I'd end up getting a table that combined months that meant different things. What I'd really want is XPath as the root selector. Third, even though the Store abstraction is nice for handling multiple data formats, if I wanted data to be combined from different parts of the XML tree or multiple trees, what I really would like is XPath from the DataGrid formatter function itself.

Assuming this might be easier in the other very popular library for JavaScript query, I went off an investigated jQuery. I quickly found articles that talked about jQuery and XML. I patterned the next part of the article after this example. So, rewriting, I ended up with:



Now, with jQuery, I'm actually able to do a little more "native" xml query. You'll see that I can access attributes directly. You'll see that I can navigate only to the months or the monthByMonthDownloadStats. However, as someone that knows XQuery, this syntax seems very unnatural (I'm sure it's very clear to JavaScript and/or CSS writers). Unnaturalness aside, this seems more verbose. In XQuery I can write this like:



With this I get all of the same benefits that jQuery has (plus more - I'm almost sure jQuery wouldn't support the rich Functions and Operations of XPath 2.0 or any mixed XML content common in document centric XML approaches). XQuery mixes the construction of the content with the query of input much better in my opinion (I believe if we showed date comparison for example you'd see a worse comparison). Of course the benefit of jQuery over XQuery is XQuery doesn't run in the browser. I had to run the previous XQuery sample on the server. That is a pretty big benefit.

I think the summary of all of this, if you stayed with me this long, is that Web 2.0 technology in the browser isn't really ready to handle the complex XML documents that exist within most enterprises. This means if you want to marry Web 2.0 with the enterprise XML data, you'll need to write data conversions essentially extending the presentation tier across the browser and middle tier that simplify the data or use feature like the Web 2.0 Feature Pack to do this for you. Also, you'll need to learn two languages (arguably three if you consider jQuery a language) and programming styles when dealing the with XML data.

Given I look at WebSphere XML Strategy, I'm not sure I'm happy with this answer. I am currently looking towards other solutions to this issue. Given I'm rather new to Web 2.0, feel free to point out other things I didn't consider in the Web 2.0 space for XML processing (outside of XForms of course).

Some Learning Experiences with XQuery/XSLT2

While working on a demo of XQuery, I ran into issues with the following things and wanted to share in case others new to XQuery could benefit. The demo was the first time I linked XQuery to Web 2.0 (was populating DOJO graphs from XML data) in an application.

First, DOJO is based upon JavaScript. When you write an XQuery that generates a dynamic web pages that mixes XQuery and DOJO, you need to be careful of the "{" character. JavaScript structures love to use the "{" character, as does XQuery. XQuery allows you to escape the "{" character by using "{{" (similarly for "}"). This isn't a huge issue once you realize what is going on as the XML Feature Pack will complain when compiling a XQuery + JavaScript program telling you that some XQuery script subsection isn't valid (its trying to interpret the JavaScript structure as XQuery).

Second, similar to a problem I had before, you have to be careful with namespaces. I had something like:


<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<title>Title</title>
</head>
<body>
{
for $i in /some/path/in/input/document
return $i
}


And the some path in input document wasn't returning any data, even though I knew there was data at that path. The issue here is documented in the spec. The default namespace of xhtml in the direct constructor becomes the default namespace for the path step elements. I found the simplest way to fix this was to move the path logic into a declared function that was outside of the direct constructor where the XHTML default namespace wasn't in scope. I could have also re-declared the default namespace or prefixed all xhtml nodes, but that wouldn't look as clean.

Speaking of declared functions, I also was tripped up for a little bit by the fact that declared functions don't get the same context passed to them automatically as the does the main execution of the same module. This exhibited by the runtime telling that the path I was executing was invalid as the context was unknown. Again the spec tells me that the context is undefined. In order to deal with this, you just need to pass the context of interest to the function and have all relative paths work off of the passed context.

Finally, I did get tripped up on XSLT 2.0 as well. When running a stylesheet that took no direct input, I mistakenly called setXSLTInitialMode (good for defining multiple paths through a XSLT 2.0 stylesheet) instead of setXSLTInitialTemplate (good for loading data from multiple input docs or unparsed text, etc.). Luckily, the errors of IXJXE0793E and ERR XTDE0045 came out in the logs and helped me spot the code completion generated typo.

Hopefully some small help if, like myself, you're working to use XQuery/XSLT 2.0 more and more in your ever day coding. Now, if I could just stop typing ";" at the end of XQuery let statements.

DevX coverage of CEA, XML, and SCA Feature Packs

Here is a good DevX article that covers the values of the three WebSphere Application Server Version 7.0 Feature Packs - Communication Enabled Applications (CEA), XML (XQuery, XSLT 2.0, XPath 2.0) Applications, and Service Component Architecture (SCA). It also covers why our strategy of feature packs helps our customers save money, get functions easier, and have more stable environments.

External Coverage of XML Feature Pack

Last year InfoQ did a nice article on the XML Feature Pack. The article does a good job of talking to application scenarios where the new XPath 2.0, XSLT 2.0, and XQuery 1.0 standards are valuable. It also talks about why native XML programming is better for performance, multi-core, and cloud strategically as compared to object oriented imperative approaches. The article also mentions comparisons to other technologies.

Today, Dustin Amrheim, wrote an article that focused on the declarative vs. imperative comments in the InfoQ article and talked about how this matters in the cloud. He argues that this is an interesting approach as compared to packaging existing imperative programming models.

Both are worth a read.

WebSphere XML Feature Pack V1.0 Released

I'd proud to announce that we released the XML Feature Pack which means it's ready for production deployment in WebSphere Application Server 7.0 environments.

The WebSphere Application Server V7.0 Feature Pack for XML 1.0.0.0 provides an XML programming model that has support of the W3C XML standards of XSLT 2.0, XPath 2.0, and XQuery 1.0. These newer standards provide developers with innovative capabilities for simplified development of XML-based and document-centric applications. The programming model consists of two parts:

- The new XML Transform and Query (XPath 2.0, XSLT 2.0, and XQuery 1.0) runtime which has been optimized for performance, runs under Java 2 security, has an thread-safe model appropriate for server environments, and provides for reliability, availability, and serviceability.

- A new API to invoke all three languages that allows applications to navigate, transform or query XML from a single consistent Java API. This API also allows the XML runtime to incorporate existing Java business logic and data.

Other important parts of this feature pack are:

- The IBM Thin Client for XML with WebSphere Application Server provides all the same functionality in J2SE clients that are used in WebSphere Application Server environments.

- The samples (with easy to browse source code) which show over 40 different aspects of the new XPath 2.0, XSLT 2.0, and XQuery technologies, three end to end web applications that show how to use these technologies to navigate, transform, and query XML atom blog feeds, and an end to end web application that shows how to integrate data from databases that support XML natively with other XML data sources in the most simple and performant way.

- Command line and ANT tools for pre-compiling XML artifacts for optimal performance

- Command line tools for simple execution of XML artifacts

- The Information Center for complete documentation of the XML Feature Pack

I will be posting all links to public information on the XML Feature on this blog post. Already, there is a video that shows how to get the XML Feature Pack installed (including the samples), so you can get started easily.

Update 2009-01-18: Rational Application Developer 7.5.5 tools add support for XPath 2.0 and XSLT 2.0.

More Web Services Performance Improvements in WAS 7.0.0.7

Back in WAS 7.0, I blogged about impressive web services performance improvements.

We have continued that improvement in fixpack 7 (7.0.0.7). Given this is a fix pack, we decided to make the new functional optional (until 8.0). You can learn about how to turn it on in PK94109. Basically you enable it by setting a system property on your application server instance.

This improvement is on the JAXB to XML marshalling (serialization) side. On the server, this would be important when responses are large (the typical case is responses are larger than requests). On the client, this would be important on requests.

As before, this not only helps web services using JAXB, but also anyone using JAXB directly in the application server.

In our tests in the labs we see approximately 20% increase in performance for some representative web services. If you are using JAX-WS JAXB based web services, turn this one with 7.0.0.7 and let me know what you see.

XML Feature Pack, SCA Refresh, and SAML get a date

Yesterday, IBM announced the dates for the XML Feature Pack, a major update to the Service Component Architecture Feature Pack, and SAML support in the WebSphere Application Server.

Borrowing from the announce letter:

New WebSphere® Application Server V7 capabilities promote application innovation and efficient development and management through updated Service Component Architecture (SCA) support, new XML programming model standards, and support for secure, interoperable Web services using SAML1 Token:

• Key updates to SCA programming model support for composite application assembly and service deployment through the WebSphere Application Server V7 Feature Pack for SCA V1.0.1 Refresh


• Support for the XSLT 2.0, XPath 2.0, and XQuery 1.0 programming model standards to benefit XML application development scenarios through the WebSphere Application Server Feature Pack for XML


• Support for OASIS Web Services Security SAML Token Security Profile 1.1 standard delivered in the WebSphere Application Server V7 Fix Pack 7 (7.0.0.7)

XML and SCA will be generally available on November 20th, and SAML will be available on November 13th. Hopefully you can try these out over the holiday season and be ready for deployment next year.

Speaking at RTP WUG Next Week

If you're in the Raleigh/Durham area come to hear about the XML Feature Pack at the RTP WebSphere User's Group (WUG) on Tuesday. I'll be speaking through dinner to talk about what XPath 2.0, XSLT 2.0, and XQuery 1.0 bring to the table for JEE programmers, what features are in the XML Feature Pack Beta and how to get started. After that there will be a hands on session with WebSphere sMash.

Ryan Boyles from the WebSphere sMash team wrote up a great article that tells you how to get registered and maps to the location, etc here. Hope to see all my "local" people there.

XML-In-Practice Day #1 Summary

I'm at IDEAlliance XML-In-Practice 2009 in DC this week talking about IBM WebSphere XML and learning about other XML products and technologies. There are four tracks -- Publishing and Media, Applications, Foundation and Interoperability AKA the technology track, Electronic Medical Records and President Obama's Economic Plan, and e-Government. Based on my totally unscientific head count, the attendance to each track is 50%, 25%, 12% and 13%.

I attended the following sessions today -

Keynote - XML Enabled Medical Records - Dr. Clement McDonald

---

I learned a) how technology is used to create repositories of information within hospitals/caregivers b) how much workload these systems exchange in the form of HL7 messages c) how distributed systems share data within a localized region for decision making, consistency of care d) how Web 2.0 is helping replace very complicated forms based desktop apps that are trusted currently e) how Dr's are happy to have a wealth of electronic information to help, but see putting new data into the system as something they cannot afford to do given already limited time with patient (call for smarter devices/speech to text) f) how different data is across the various medical interactions all the way from very structured to very narrative. The best two parts of the talk was the 0.5 seconds he showed an XML document which stressed the business aspect of this is key - the technology just has to exist behind the scenes to make it possible and seeing a Dr. throw stuffed pings to the audience (joke on how LOINC standards sounds like OINK).

Overview of President Obama's Electronic Medical Records Plan and Health Information Technology Architecture - John Quinn

---

I a) learned how much money is set aside to rewards providers that move to standardized medical health records and what timelines exist to get these rewards b) learned how these timelines are aggressive based on time to implementation of typical systems c) learned how the rewards are based upon a certified system which is challenging to guarantee for valuable use. I really took away a deeper appreciation for not only the complexity inside of a single hospital, but also how challenging a national mandate will be (especially to individual physicians).

XSLT Stylesheets from Version 1.0 to 2.0 - Priscilla Walmsley

---

I didn't take alot of notes in this session as I'm rather knowledgeable about this topic. However, I'd say it was a great presentation given the example (before and after) based approach.

Customer Use Case: How IBM Simplifies Complex Content Developing and Publishing Across the Enterprise. - Daniel Dionne

---

Great presentation that didn't go into just what DITA for content development/publishing is, but showed the entire lifecycle and processes needed to make a wide adoption work. Went into some rather impressive use cases of the technology, along with challenges, within the IBM company.

Technical Overview of Relax-NG - Bob Ducharme

---

Can't say I was a huge supporter of this talk, but that is likely due to the fact that I'm a data-oriented XML guy and I'm working on standards, customer situations that are very dependent on XML Schema. Bob discussed areas where Relax-NG was better than XML schema for mostly document oriented scenarios. I would have liked to see more mention of XML Schema 1.1 and how that changed the story. I did get some good value out of understanding why some document-centric customers are still using DTD's.

HL7's use of XML - Paul Knapp

---

Learned how HL7 V3 XML isn't really yet used in US e-healthcare apps (every hospital is exchanging internal messages in HL7 V2). Abroad, new projects that are less than three years old are very likely to be HL7 V3. We should be seeing more of this in the states with new projects, especially as we start to consider the need to share information outside of a single hospital, etc. Paul did mention binary XML and how that would help many of the HL7 V3 current issues.

MarkLogic Beer and Demo Jam

---

I did a 4 minute demo along with nine others during the reception. You get 5 minutes to do a demo with no preparation and the best demos win free stuff. I demoed the XML Feature Pack and the 40 samples we have along with the end to end blog checker sample written in XPath 2.0, XSLT 2.0, and XQuery 1.0. The samples I showed had a nice CSS and dashboard we've added since Beta 4 and that visual skinning over the XML technologies drew positive comments from the crowd. Didn't win anything in the end. Oh well.

After hours

---

Finally, I was able to do dinner with about 15 folks who regularly attend these conferences. Some great conversation with people from all parts of the industry.

XML Feature Pack Thin Client Demo - Zero to running in 6 minutes

NOTE: This post is our of date. For the same demo on the released product see this link

As we announced, the latest beta release of the XML Feature Pack contains the Thin Client for XML. As well as allowing you to use this in your client applications to WebSphere Application Servers, the thin client allows for quick and easily evaluation of the technology. Here I show a quick demo of using the following simple XML, XPath, XSLT and XQuery files along with Java files to invoke them.

Demo 7 - XML Feature Pack Beta 4 Thin Client for XML

Direct Link (HD Version)


Here are the files for the demo:

thinclientdemo.zip


Which contains (HelloXSLT.java, HelloXQuery.java, HelloXPath.java, simple.xsl, simple.xq, locations.xml)

XML Feature Pack Beta 4 - Now With Thin Client

A month ago, we announced the Beta 3 refresh which was specification complete on XPath 2.0, XSLT 2.0, and XQuery 1.0. On Friday we released a Beta 4 refresh which continues to remove any remaining restrictions as well as adds one new major feature - The Thin Client for XML with WebSphere Application Server.

As noted on the open beta download page,

The beta includes the IBM Thin Client for XML with WebSphere Application Server. The thin client allows access to the same Feature Pack API and runtime functionality (XPath 2.0, XSLT 2.0, XQuery 1.0) available in the WebSphere Application Server Feature Pack for XML. The thin client can be copied to multiple clients running Java SE 1.6 in support of a WebSphere Application Server V7.0 installation.

This means if you have client applications to WebSphere Application Servers you can copy the XML Feature Pack thin client file to these clients and get the same XML programming model support in your clients.

We also believe this thin client support will help "new to WebSphere" folks evaluate this technology. As such, we have added a download link to the jar file on the open beta website. Click on that link and then click on "Local install using Download Director or HTTP" and follow through to download "IBM Thin Client for XML with WebSphere Application Server
com.ibm.xml.thinclient_1.0.0.jar". I hope to show a demo of how fast you can get up and going with the thin client in the next day or so.

Hidden nodes in XPath - fail on namespaces by me

I was working on a sample with the XML Feature Pack last week to show good integration between the XML Feature Pack Beta and databases that support XML columns, such as DB2 pureXML.

I ran into an issue that stumped me for a while and wanted to write about it so maybe others won't be slowed down as long as I. I was writing a XCollectionResolver and XResultsResolver that connected to the database. For some reason, while these resolvers returned data that looked valid, they couldn't be navigated by XPath 2.0. I saw things like this is XQuery:

let $a := trace($domainSpammers/spammers/spammer/email, "email =")

Traced nothing, while

let $a := trace(node-name($domainSpammers/*/*/*), "threestars = ")

Traced email, uri, and name. I even put domainSpammers into the output of the XQuery and could see the spammers/spammer/email tree:

<spammers xmlns="http://www.w3.org/1999/xhtml" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <spammer>
    <name>Joe Smith</name>
    <email>jsmith@email.com</email>
    <uri>http://joe.uri.com</uri>
  </spammer>
</spammers>

I looked at this for a few hours. Luckily one of my team members saw the issue. You can see by the title of this post and the above xml, the issue was the elements are in the XHTML namespace.

It turns out I was writing to the document from XSLT 2.0 using the new feature of multiple result documents. While I wanted my browser returned page to be in the default namespace of XHTML, I didn't want the data written to the database to be in the XHTML namespace. However, since I didn't clarify this, it mistakenly was written to that namespace.

Next time, and maybe this will help you, I'll add namespace-uri() to my debugging arsenal:

let $a := trace(
  (node-name($domainSpammers/*/*/*),
  namespace-uri($domainSpammers/*/*/*)), "threestars = ")

Which would clearly have shown that email was in the XHTML space:

a = email http://www.w3.org/1999/xhtml;

Which would have saved me a few hours of pulling my hair out.

Other XML Feature Pack Beta 3 Highlights

Yesterday I blogged about one of the major focus areas of the Beta 3 of the XML Feature Pack. In today's video demo, I show the other major items of note in Beta 3. The major features of Beta 3 are:

- Spec complete on XPath 2.0, XSLT 2.0, and XQuery
- Changes to the XML Feature Pack API to adjust for new features and results of usability studies
- Focus on development and deployment issues with full command line support to pre-compile XML artifacts for performance, ANT commands to do the same, and support for running with Java 2 Security enabled.
- As always, more samples. In addition to the end to end sample of XQuery I demoed yesterday, we have over 40 samples that show the new features of the new standards.

Here is the video that overviews these features:

Demo 6 - XML Feature Pack Beta 3 Highlights

Direct Link (HD Version)


For more information on the WAS V7.0 XML Feature Pack Beta, please go here to download the code, samples, documentation as well as see other demos.

Download Stats Extreme Makeover - The Value of XSLT 2.0 and XQuery 1.0

On Friday, I had the opportunity to use XSLT 2.0 and XQuery 1.0 in a way that proved the value of the new standards.

For our Beta programs, we get a weekly report of how many downloads have occurred. Over time we have asked for more and more breakdowns of the data (how many were from IBM'ers vs. non-IBM'ers, how many were from non gmail/hotmail/etc email addresses, how many for code vs. how many for documentation, how many unique users have downloaded, etc.). We use these reports to gauge the interest in our betas and the effectiveness of our beta programs.

Currently the process for doing these reports is:

1. Load a webpage somewhere on the intranet that returns in either HTML or space and newline separated text the download records (and I don't have the time nor contacts to change this "service").
2. Import these download records into Excel
3. Write VBScript that processes the rows into summary tables, but not all summaries have VBScript written for them due to the fact that VBScript can get quite complicated.
4. Read the summary tables and hand compute some summaries and transpose them into emails and presentations to beta teams.

This process can take a few hours and is error prone.

I asked for the raw data for the service as I knew there had to be an easier way.

First I started with the HTML as its pretty much XML and I figured I could just write some simple XQuery summaries. This failed as HTML isn't XML (if its not XHTML). Don't get me started on that rant (in this case, we had things like width=-1 with no quotes)! So I restarted with the text file version of the data.

XSLT 2.0 has a new function - unparsed-text() that lets you load from text files. This allows you to load any data into XSLT 2.0, but only as a string. As the XSLT 2.0 specification shows the real power of this function is magnified by XPath 2.0's support of regular expressions. Combining the unparsed-text() with regex tokenize() with newline as the separator allowed me to for-each across every line in the file. Adding another new XSLT 2.0 function into the mix - analyze-string() with a regular expression that did capturing of each of the fields allowed me to transform each line into a well formed XML element with sub-elements for each of the fields (download element with elements for filename, email address, etc). Ah, now I have data in a well formed structured XML format. Life is good.

Once I had this transformation of the input data, I returned to the task of creating the summaries. With XQuery, any query I want is just moments away. To prove the point, I decided to write a simple query that would break down the downloads for IBM'ers and non-IBM'ers separately. I would also group each unique user and list all the downloads that user had done with summaries at all levels of how many code vs. how many documentation downloads had occured. I want to put this in XML format, so others could query my summaries or I could put the summaries into HTML or load back into Excel (if you really really want to).

It took me approximately 30 minutes to create all of the above queries. The code is approximately 20 lines long and is well designed into re-usable functions that could be customized later. The code is easy for most of my peer Java developers to understand as XQuery looks alot like an imperative language with SQL like queries mixed into templating of output.

My next step is to move this to a web application (what I did so far was prototyped it with sample data) that connects directly to the service. The web application could easily offer up web forms that allowed the user to specify search criteria supported by the back end service (date range for example) along with what XQuery/summary view was required.

There are still some items that can't be automated and need human intervention. As an example, the process of deciding what constitutes an IBM'er is complex as some IBM'ers have "ibm.com" email address and some do not. Also, IBM could be "IBM, International Business Machines, or mistyped". It would take a bit of time to create some services that approximate what a human eye could spot manually. Adding a human facing pop-up that allowed visual inspection of the automated data analysis steps would be valuable. Also, I admit that I didn't create the charts and graphs - just the raw data that could be loaded into Excel to create charts and graphs.

But in the end I have converted a manual error prone data processing scenario into an automated approach (for data query and summaries) that creates all the same valuable raw data for reports with the potential to add more reports much more quickly. All of this was made possible using well documented W3C standards that have all the needed features (some new with XPath/XSLT 2.0 and XQuery 1.0) that make this scenario possible.