Perspective in Benchmarks - My thoughts on Microsoft StockTrader

I wanted to take the time to respond to claims coming from Microsoft. On June 4th, Greg Leake, Technical Marketing Manager for .NET of Microsoft started to discuss an effort with which he was involved. The effort was “porting” IBM’s WebSphere Performance Sample – Trade 6.1 to Microsoft .NET 3.0. Since then, Microsoft has been interviewed by trade press, has created video podcasts, and has started discussions on Microsoft forums and other development community sites. Please do not be fooled by these marketing statements.

In my opinion, Microsoft uses their report to focus on three points

- Attempt to prove that Microsoft is interoperable with WebSphere software
- Attempt to prove Microsoft’s SOA relevancy and readiness for the enterprise
- Attempt to prove Microsoft performance is better than WebSphere software performance

I’d like to state up front that I was personally involved with the development of IBM’s Trade benchmark starting with Trade 3 and ending with Trade 6.1. Others on my team have made significant contributions to the code as well to ensure that Trade continues to be an credible and useful tool to express performance concepts and release to release enhancements to IBM customers for J2EE (Java EE) programming.

Some level of interoperability

I applaud any effort by Microsoft or any other software vendor in pursuit of interoperability. However, the interoperability the Microsoft report speaks to is basic web services functionality (SOAP/WSDL/XML) only. It does not focus on interoperability of transactions, security, reliability, durability and does not use industry standard schemas that many of our customers need for cross enterprise (B2B) or intra-enterprise web services interoperability with .NET clients calling WebSphere servers. Independent of the Microsoft report, Microsoft and IBM have already focused on interoperability of these higher value qualities of service in Web Services through industry standard WS-I efforts as shown here and here. I am very happy to see IBM and Microsoft continue to focus on standards based interoperability. I am confident that WS-I will continue to facilitate customer focused web services interoperability in these higher value web service functionalities.

Is Microsoft enterprise ready?

Press coverage has cited the Microsoft report as helping Microsoft prove it is ready for the enterprise. From our quick look at Microsoft code, this doesn’t seem to be the case. Many of the “enterprise ready” features stressed by the Microsoft report are hand coded into the application. Areas such as multiple vendor database support, load balancing, high availability, scalability, and enterprise wide configuration for services contribute to the significantly higher lines of code count in the Microsoft application as compared to the WebSphere implementation. You should equate higher lines of code count to more maintenance and support costs long term and think about the value of this being provided in the application server product versus in the application itself. Some of the increased lines of code are for comments, but the comments themselves point out how many places where Microsoft deviated from using framework classes and instead implemented custom extensions to fill in gaps of functionality lacking in their framework.

As an author of Trade, I must admit an embarrassing fact. Four years ago, my team added web services capability to Trade as an optional component – to demonstrate web services as an alternative to remote stateless session bean methods. Since that time, I personally have worked hand in hand with many enterprise customers adopting web services. We have found that the Trade approach to web services wasn’t the best. Specifically the fine grained “services” in Trade average around 400-800 bytes of passed data. As you’ll see in my recent blog post, industry standard schemas for B2B typically have much larger payloads. While Trade was a fun exercise for my team to learn web services, it in no way mirrors what we know now our customers have told us are good web service practices. Current SOA principles motivate more coarse gained services. The embarrassing fact is that we never have gotten around to removing the poor examples of web services usage from Trade. However, it is interesting to note that Microsoft did not recognize or point out the obvious flaws in these web service patterns during their analysis – they merely parroted what they saw.

Microsoft entitled this paper as “Service Oriented”. Later in the press, Microsoft alludes to this “Service Oriented” benchmark test to help draw credibility to Microsoft’s SOA strategy. Based on what I just said about Trade web services, you’ll see why IBM has never talked about Trade and web services to help customers understand web service performance. Meanwhile, this Microsoft report focuses on these trivial payload web services to prove they can support “Service Oriented” benchmarks. Follow-up posts and press coverage followed the “Service Oriented” title, proclaiming that this in some way helps Microsoft’s SOA strategy. Drawing that conclusion is incorrect. The industry agrees that SOA is more than trivial web services. SOA is about a business centric IT architecture that provides for a more agile and flexible IT support of business needs. Realistic web service usage is a cornerstone of SOA, but web services alone do not define SOA.

Is Microsoft performance better than WebSphere performance?

I believe Microsoft is trying to draw IBM out of our commitment to standard benchmarking organizations to confuse customers about performance. You can draw your own conclusions from specific comments apparently made by Greg Leake in press coverage, asking for IBM to “collaborate”. Some have commented correctly on community discussions that the Microsoft report isn’t specific enough in terms of topology, hardware, and tuning to make any sensible conclusions based on the performance data. It is a compelling story that this Microsoft report weaves – Microsoft beating IBM on its own benchmark. However, IBM didn’t run Trade as a benchmark in the way shown in the paper’s results. As a customer, you should always be careful how much you trust proprietary benchmark results produced by a single vendor. These things can always be coerced to create FUD and confusion.

We have reviewed the paper and results and found inconsistencies with the best practices for how to run WebSphere Application Server. Assuming items Microsoft chose not to document along with improvements in performance allowed by following best practices, we in fact believe that IBM WebSphere Application Server would win across all the scenarios shown in the results.

You may well ask, if WebSphere Application Server would win, why wouldn’t you say so and publish a contrary IBM report. I don’t believe publishing a proprietary view of performance would help our customers – for all the reasons stated above. At best, if IBM was to respond to this paper, you could expect both vendors to degrade to the lowest common coding styles for implementing the benchmark so they would “win”. As shown already by Microsoft’s implementation, Microsoft wouldn’t choose to use their framework classes and features, but instead code optimal patterns in the application code. When those patterns are not tested and supported by the formal product, no customer wins by seeing the resulting performance.

IBM has a history of competing in standard benchmarking organizations such as SPEC. We do so because such organizations are standards based and unbiased and therefore trusted. Standards-based benchmarking processes give all participating vendors equal opportunity to review run rules, implementation code, and the results of other vendors. Given this, if you find Trade and SOA benchmarks useful, maybe it is time for IBM and Microsoft to jointly propose a SOA benchmark to a standard benchmarking organization. SOA benchmarking under standard benchmarking organizations is where our customers and the industry can truly benefit.

Summary

I personally talk to many customers about performance, SOA, and web services. I stand behind all that I say technically independent of marketing. I build strong long lasting relationships with these customers, many of whom know me personally. In good faith, I can stand in front of them with results from a standards benchmarking organization. I can stand in front of them showing our SOA leadership based on both customer references and analyst reports. On the other hand, I cannot in good faith show a one-off competitive benchmark run by a single vendor. I hope you can understand this position and it helps you discuss the coverage of this Microsoft report and any similar efforts by any vendor that follow.

This Microsoft report shows basic levels of interoperability and work within the WS-I shows higher levels of interoperability. The Microsoft report points out that Microsoft has to hand code many enterprise ready features that are just taken care of for you in WebSphere technology. The title of this Microsoft report and its follow-on press coverage attempts to confuse the industry on SOA, which points out how desperate Microsoft is to get press coverage in SOA. This Microsoft report doesn’t do a good job of showing the true performance story. Specifically on performance, I encourage all customers to put their trust in standard benchmarking organizations

My surreal work experience for the day

I noted the title of an eWeek article called, "Microsoft to Reach Out to IBM, Cisco on Interoperability" and decided to take a look. Microsoft's on-again, off-again relationship with interoperability is frustrating for our customers, for example when they choose subsets of a given specification to support. So if Microsoft wants to be serious about interoperability, it can only be a good thing, right?

So here's what Bob Muglia, Microsoft's senior vice president for the Server and Tools division, had to say:

"[Our customer council] basically told me, pretty directly, that while Microsoft's implementation was in great shape, IBM's and others were not, and that Microsoft needed to do a better job helping them do a better implementation," Muglia said. "And I had to think about that, as it is one thing for us to work with customers around interoperability, but quite another to go out and help a competitor build a better product to enable interoperability."

But Microsoft has now decided to go and talk to IBM and BEA Systems and a few others to help improve and define their interoperability. "Ultimately these guys have to make their products good, but there is a lot we can do working with them to make their products interoperate better with us," Muglia said.

IBM could not be immediately reached for comment on Muglia's remarks or its thoughts about improving interoperability with Microsoft.

After I picked myself off the floor, I asked myself, "Why would eWeek allow itself to be used like this?" Clearly, this was simply a marketing move, not a signal of genuine intention to work with IBM. If Microsoft really wanted to work with IBM on interoperability, would it launch this activity by telling eWeek that the goal was for Microsoft to help IBM with our implementation? And without a response from IBM? This is not a news story, it's an advertisement.

Microsoft Host Integration Server?

Over on the Mainframe Blog, there's been an amusing chain of posts here, here, and here, on the topic of "What's the point of Microsoft Host Integration Server?". Just to pique people's interest, here is what Microsoft had to say about it:

I am most amused at the idea of mainframe guys arguing cost. In the interest of transparency, Timothy might also do the math on what the customer would save if they ripped the mainframe out altogether. Not only do they not need to buy HIS, but they can save a lot more on the recurring costs of the mainframe.

...

Last time I checked (and I haven’t been paying attention for a while), the mainframe was three orders of magnitude more expensive per MIP than x86 and was falling further and further behind. You’re crazy to put any new workload on the mainframe. This is why IBM is always peddling the fad of the moment on the mainframe to see if they can hoodwink people to maintaining or even increasing the workloads on their mainframe. Run Linux and Java on the mainframe, they say, never mind the fact that these cross platform approaches should drive people to the lowest cost hardware, not the highest cost.

...

To put this in perspective, we’ve just been through the biggest computing buildout ever in the last decade with the Web, and the mainframe is nowhere to be seen. Even when IBM has tried to pay customers to run portions of their web sites on a mainframe, they have failed.

Obviously it will be news to our z/OS WAS customers that running portions of their websites on a mainframe has "failed".

More evidence that when it comes to the mainframe, Microsoft just doesn't get it.