Monday, September 24, 2007

Perspective in Benchmarks - My thoughts on Microsoft StockTrader

I wanted to take the time to respond to claims coming from Microsoft. On June 4th, Greg Leake, Technical Marketing Manager for .NET of Microsoft started to discuss an effort with which he was involved. The effort was “porting” IBM’s WebSphere Performance Sample – Trade 6.1 to Microsoft .NET 3.0. Since then, Microsoft has been interviewed by trade press, has created video podcasts, and has started discussions on Microsoft forums and other development community sites. Please do not be fooled by these marketing statements.

In my opinion, Microsoft uses their report to focus on three points

- Attempt to prove that Microsoft is interoperable with WebSphere software
- Attempt to prove Microsoft’s SOA relevancy and readiness for the enterprise
- Attempt to prove Microsoft performance is better than WebSphere software performance

I’d like to state up front that I was personally involved with the development of IBM’s Trade benchmark starting with Trade 3 and ending with Trade 6.1. Others on my team have made significant contributions to the code as well to ensure that Trade continues to be an credible and useful tool to express performance concepts and release to release enhancements to IBM customers for J2EE (Java EE) programming.

Some level of interoperability

I applaud any effort by Microsoft or any other software vendor in pursuit of interoperability. However, the interoperability the Microsoft report speaks to is basic web services functionality (SOAP/WSDL/XML) only. It does not focus on interoperability of transactions, security, reliability, durability and does not use industry standard schemas that many of our customers need for cross enterprise (B2B) or intra-enterprise web services interoperability with .NET clients calling WebSphere servers. Independent of the Microsoft report, Microsoft and IBM have already focused on interoperability of these higher value qualities of service in Web Services through industry standard WS-I efforts as shown here and here. I am very happy to see IBM and Microsoft continue to focus on standards based interoperability. I am confident that WS-I will continue to facilitate customer focused web services interoperability in these higher value web service functionalities.

Is Microsoft enterprise ready?

Press coverage has cited the Microsoft report as helping Microsoft prove it is ready for the enterprise. From our quick look at Microsoft code, this doesn’t seem to be the case. Many of the “enterprise ready” features stressed by the Microsoft report are hand coded into the application. Areas such as multiple vendor database support, load balancing, high availability, scalability, and enterprise wide configuration for services contribute to the significantly higher lines of code count in the Microsoft application as compared to the WebSphere implementation. You should equate higher lines of code count to more maintenance and support costs long term and think about the value of this being provided in the application server product versus in the application itself. Some of the increased lines of code are for comments, but the comments themselves point out how many places where Microsoft deviated from using framework classes and instead implemented custom extensions to fill in gaps of functionality lacking in their framework.

As an author of Trade, I must admit an embarrassing fact. Four years ago, my team added web services capability to Trade as an optional component – to demonstrate web services as an alternative to remote stateless session bean methods. Since that time, I personally have worked hand in hand with many enterprise customers adopting web services. We have found that the Trade approach to web services wasn’t the best. Specifically the fine grained “services” in Trade average around 400-800 bytes of passed data. As you’ll see in my recent blog post, industry standard schemas for B2B typically have much larger payloads. While Trade was a fun exercise for my team to learn web services, it in no way mirrors what we know now our customers have told us are good web service practices. Current SOA principles motivate more coarse gained services. The embarrassing fact is that we never have gotten around to removing the poor examples of web services usage from Trade. However, it is interesting to note that Microsoft did not recognize or point out the obvious flaws in these web service patterns during their analysis – they merely parroted what they saw.

Microsoft entitled this paper as “Service Oriented”. Later in the press, Microsoft alludes to this “Service Oriented” benchmark test to help draw credibility to Microsoft’s SOA strategy. Based on what I just said about Trade web services, you’ll see why IBM has never talked about Trade and web services to help customers understand web service performance. Meanwhile, this Microsoft report focuses on these trivial payload web services to prove they can support “Service Oriented” benchmarks. Follow-up posts and press coverage followed the “Service Oriented” title, proclaiming that this in some way helps Microsoft’s SOA strategy. Drawing that conclusion is incorrect. The industry agrees that SOA is more than trivial web services. SOA is about a business centric IT architecture that provides for a more agile and flexible IT support of business needs. Realistic web service usage is a cornerstone of SOA, but web services alone do not define SOA.

Is Microsoft performance better than WebSphere performance?

I believe Microsoft is trying to draw IBM out of our commitment to standard benchmarking organizations to confuse customers about performance. You can draw your own conclusions from specific comments apparently made by Greg Leake in press coverage, asking for IBM to “collaborate”. Some have commented correctly on community discussions that the Microsoft report isn’t specific enough in terms of topology, hardware, and tuning to make any sensible conclusions based on the performance data. It is a compelling story that this Microsoft report weaves – Microsoft beating IBM on its own benchmark. However, IBM didn’t run Trade as a benchmark in the way shown in the paper’s results. As a customer, you should always be careful how much you trust proprietary benchmark results produced by a single vendor. These things can always be coerced to create FUD and confusion.

We have reviewed the paper and results and found inconsistencies with the best practices for how to run WebSphere Application Server. Assuming items Microsoft chose not to document along with improvements in performance allowed by following best practices, we in fact believe that IBM WebSphere Application Server would win across all the scenarios shown in the results.

You may well ask, if WebSphere Application Server would win, why wouldn’t you say so and publish a contrary IBM report. I don’t believe publishing a proprietary view of performance would help our customers – for all the reasons stated above. At best, if IBM was to respond to this paper, you could expect both vendors to degrade to the lowest common coding styles for implementing the benchmark so they would “win”. As shown already by Microsoft’s implementation, Microsoft wouldn’t choose to use their framework classes and features, but instead code optimal patterns in the application code. When those patterns are not tested and supported by the formal product, no customer wins by seeing the resulting performance.

IBM has a history of competing in standard benchmarking organizations such as SPEC. We do so because such organizations are standards based and unbiased and therefore trusted. Standards-based benchmarking processes give all participating vendors equal opportunity to review run rules, implementation code, and the results of other vendors. Given this, if you find Trade and SOA benchmarks useful, maybe it is time for IBM and Microsoft to jointly propose a SOA benchmark to a standard benchmarking organization. SOA benchmarking under standard benchmarking organizations is where our customers and the industry can truly benefit.

Summary

I personally talk to many customers about performance, SOA, and web services. I stand behind all that I say technically independent of marketing. I build strong long lasting relationships with these customers, many of whom know me personally. In good faith, I can stand in front of them with results from a standards benchmarking organization. I can stand in front of them showing our SOA leadership based on both customer references and analyst reports. On the other hand, I cannot in good faith show a one-off competitive benchmark run by a single vendor. I hope you can understand this position and it helps you discuss the coverage of this Microsoft report and any similar efforts by any vendor that follow.

This Microsoft report shows basic levels of interoperability and work within the WS-I shows higher levels of interoperability. The Microsoft report points out that Microsoft has to hand code many enterprise ready features that are just taken care of for you in WebSphere technology. The title of this Microsoft report and its follow-on press coverage attempts to confuse the industry on SOA, which points out how desperate Microsoft is to get press coverage in SOA. This Microsoft report doesn’t do a good job of showing the true performance story. Specifically on performance, I encourage all customers to put their trust in standard benchmarking organizations

1 comments:

Greg Leake said...


Microsoft Response

Andrew, my responses are in bold/italics inline with your original post comments below.

Thanks,

Greg Leake
Microsoft Corporation



Perspective in Benchmarks - My thoughts on Microsoft StockTrader
I wanted to take the time to respond to claims coming from Microsoft. On June 4th, Greg Leake, Technical Marketing Manager for .NET of Microsoft started to discuss an effort with which he was involved. The effort was “porting” IBM’s WebSphere Performance Sample – Trade 6.1 to Microsoft .NET 3.0. Since then, Microsoft has been interviewed by trade press, has created video podcasts, and has started discussions on Microsoft forums and other development community sites. Please do not be fooled by these marketing statements.

In my opinion, Microsoft uses their report to focus on three points

- Attempt to prove that Microsoft is interoperable with WebSphere software
- Attempt to prove Microsoft’s SOA relevancy and readiness for the enterprise
- Attempt to prove Microsoft performance is better than WebSphere software performance

I’d like to state up front that I was personally involved with the development of IBM’s Trade benchmark starting with Trade 3 and ending with Trade 6.1. Others on my team have made significant contributions to the code as well to ensure that Trade continues to be an credible and useful tool to express performance concepts and release to release enhancements to IBM customers for J2EE (Java EE) programming.

Some level of interoperability

I applaud any effort by Microsoft or any other software vendor in pursuit of interoperability. However, the interoperability the Microsoft report speaks to is basic web services functionality (SOAP/WSDL/XML) only. It does not focus on interoperability of transactions, security, reliability, durability and does not use industry standard schemas that many of our customers need for cross enterprise (B2B) or intra-enterprise web services interoperability with .NET clients calling WebSphere servers. Independent of the Microsoft report, Microsoft and IBM have already focused on interoperability of these higher value qualities of service in Web Services through industry standard WS-I efforts as shown here and here. I am very happy to see IBM and Microsoft continue to focus on standards based interoperability. I am confident that WS-I will continue to facilitate customer focused web services interoperability in these higher value web service functionalities.


Microsoft Response:

The interoperability demonstrated uses the correct technology and Web Services standards (basic Web Services) for what it needs to achieve: tight interoperability from a user-interface done on one platform, to a middle tier done on another platform. The Trade 6.1 application, as specified and developed by IBM, does not utilize client-coordinated transactions, rather these are encapsulated, appropriately, on the middle-tier and hence self-contained within a specific application server (either .NET or WebSphere). For security, an application such as Trade 6.1 does not require (in its current incarnation) federated security between a client and multiple Web Service middle tier endpoints (since there is only one endpoint), and hence does not currently require the use of WS-Security. Using https/SSL over basic Web Services would be the right approach for the current Trade 6.1/StockTrader apps, vs. a full WS-Security implementation.

I also believe, as you point out, that there is good opportunity to explore interop in middle-tier to middle-tier interactions as well, and additionally in the future as you suggest incorporate advanced Web Service standards such as WS Atomic Transactions, WS Security, and WS Reliable Messaging into such interactions. This would require extensions to the Trade 6.1 scenario/specification, but we would love to work with IBM and would incorporate such interop into the .NET StockTrader application as well, as further proof of both companies commitment to open industry standards and interop between platforms. As you point out, MS and IBM collaborate deeply within the Web Services standards bodies precisely with this goal in mind, and further sample applications demonstrating interop I believe would be welcome by customers.

At the same time, do not discount the value in the interoperability achieved currently in the Trade 6.1/.NET StockTrader samples using Basic Web services. This is a real scenario, for any customer that wants to integrate a .NET front end with a backend WebSphere middle tier, or a Java/JSP front end with a backend .NET middle tier. The .NET StockTrader, becuase you exposed all middle tier services within Trade 6.1 as Web Services appropriately, is able to do just that, and it's very cool. You can mix and match front ends and middle tiers across platforms, and this is a deep customer requirement in many, many cases. And all it takes in either app is a simple configuration change, not a single code change! This would not be possible without SOAP and Web Services basic standards. How else would you get a WPF or ASP.NET front end to seamlessly plug into a middle tier WebSphere App Server; or a JSP front end seamlessly working with .NET/Windows Server middle tier application server?


Is Microsoft enterprise ready?

Press coverage has cited the Microsoft report as helping Microsoft prove it is ready for the enterprise. From our quick look at Microsoft code, this doesn’t seem to be the case. Many of the “enterprise ready” features stressed by the Microsoft report are hand coded into the application. Areas such as multiple vendor database support, load balancing, high availability, scalability, and enterprise wide configuration for services contribute to the significantly higher lines of code count in the Microsoft application as compared to the WebSphere implementation. You should equate higher lines of code count to more maintenance and support costs long term and think about the value of this being provided in the application server product versus in the application itself. Some of the increased lines of code are for comments, but the comments themselves point out how many places where Microsoft deviated from using framework classes and instead implemented custom extensions to fill in gaps of functionality lacking in their framework.


Microsoft Response

There is no element in .NET StockTrader that is not fully based on .NET and .NET Framework classes. It is entirely a managed code, 100% pure .NET application--even the database loaders!



As an author of Trade, I must admit an embarrassing fact. Four years ago, my team added web services capability to Trade as an optional component – to demonstrate web services as an alternative to remote stateless session bean methods. Since that time, I personally have worked hand in hand with many enterprise customers adopting web services. We have found that the Trade approach to web services wasn’t the best. Specifically the fine grained “services” in Trade average around 400-800 bytes of passed data. As you’ll see in my recent blog post, industry standard schemas for B2B typically have much larger payloads. While Trade was a fun exercise for my team to learn web services, it in no way mirrors what we know now our customers have told us are good web service practices. Current SOA principles motivate more coarse gained services. The embarrassing fact is that we never have gotten around to removing the poor examples of web services usage from Trade. However, it is interesting to note that Microsoft did not recognize or point out the obvious flaws in these web service patterns during their analysis – they merely parroted what they saw.

Microsoft Response

There really is no flaw in you having exposed Trade 6.1 middle tier operations as Web Services in the way you did. To the contrary, had you not done so, no interop with your existing Trade 6.1 code base could have been achieved seamlessly from .NET; and likewise had we not exposed .NET middle tier business services as Web Services, nor could the JSP Trade 6.1 front end seamlessly integrate with the .NET middle tier in .NET StockTrader. There is a place for both fine-grained and course-grained services...it really depends on the scenario. For UI to middle tier operations, I believe it would be impossible to build a truly interoperable UI without having some fine-grained operations exposed on the middle tier as Web Services. After all, UI's typically invoke some fine grained operations (simple database searches/presentation of datasets, etc.). But Trade 6.1 does not just expose exteremely fine-grained interactions: the order operations actually have transaction lifting going on, and multiple operations performed as atomic transactions against the database--they are not, in, fact, fine-grained. If you believe the use of Web Services in Trade 6.1 is inappropriate, please propose an alternative to having an all-.NET UI app work with an all-Java middle tier, or vice versa? Again, do not discount the value or customer need to be able to use .NET front ends (UI apps, web-based or smart clients) work with WebSphere middle tiers, or Java front ends (JSP or Java thick clients) work with .NET middle tiers/business service layers. In fact, I think this is the most basic needs of all within the enterprise.

To be sure I do agree with you, as pointed out above, that service-to-service interactions, that might be much coarser grained, are important as well. Both are important but it would be sad to see IBM back away from an architecture that allows .NET (ASP.NET and .NET smart clients) work so seamlessly with WebSphere middle tier app servers and business service layers (and vice versa). Just plug .NET StockTrader into WebSPhere Trade 6.1, or the Trade 6.1 JSP app into middle tier .NET elements using your existing approach, and it's hard to argue the value! It works and it works quite well!



Microsoft entitled this paper as “Service Oriented”. Later in the press, Microsoft alludes to this “Service Oriented” benchmark test to help draw credibility to Microsoft’s SOA strategy. Based on what I just said about Trade web services, you’ll see why IBM has never talked about Trade and web services to help customers understand web service performance. Meanwhile, this Microsoft report focuses on these trivial payload web services to prove they can support “Service Oriented” benchmarks. Follow-up posts and press coverage followed the “Service Oriented” title, proclaiming that this in some way helps Microsoft’s SOA strategy. Drawing that conclusion is incorrect. The industry agrees that SOA is more than trivial web services. SOA is about a business centric IT architecture that provides for a more agile and flexible IT support of business needs. Realistic web service usage is a cornerstone of SOA, but web services alone do not define SOA.


Mirosoft Response

Glad we agree that Web Service, open industry standards need to be at the heart of a SOA architecture. We also agree that just exposing middle tiers as Web Services may provide interop, but may not constitue a true SOA design. However, if you look deeply at the StockTrader application and the Windows Communication Foundation and our approach to SOA in our base .NET platform, you will see fundamental principles in the design that do constitute, technically, the deeper definition you strive to provide. At a technical level, WCF (and StockTrader) use services everywhere...well defined interfaces between black-boxed components, based on open industry standards, to separate implementation from interface; based on the notion of Data Contracts and Service Contracts baked into WCF, with bindings supporting *all* the Web Service current basic and advanced specifications. At Microsoft, we have already abandoned the idea of two separate programming models for remote services...one model for all-.NET environments and one for Web Services. WCF is now the SINGLE service-oriented remoting architecture for .NET, totally based on Web Service standards with the ability to separately, if desired, plug in binary encoding bindings without any additional programming or difference in programming model. Hence, any .NET service automatically can be an open Web Service, including support for WS-Security, WS-Atomic transactions, etc. WebSphere and Java still make a distinction in programming model, to my knowledge, between RMI/Java-proprietary remote interfaces, and Web Service interfaces. At MS, we have commited to open interop so deeply that our entire communication stack and programming model is consolidated around WCF/.NET 3.0 and Web Services.


Is Microsoft performance better than WebSphere performance?

I believe Microsoft is trying to draw IBM out of our commitment to standard benchmarking organizations to confuse customers about performance. You can draw your own conclusions from specific comments apparently made by Greg Leake in press coverage, asking for IBM to “collaborate”. Some have commented correctly on community discussions that the Microsoft report isn’t specific enough in terms of topology, hardware, and tuning to make any sensible conclusions based on the performance data. It is a compelling story that this Microsoft report weaves – Microsoft beating IBM on its own benchmark. However, IBM didn’t run Trade as a benchmark in the way shown in the paper’s results. As a customer, you should always be careful how much you trust proprietary benchmark results produced by a single vendor. These things can always be coerced to create FUD and confusion.


Microsoft Response

Here I will have to disagree, and assume you have not read the full disclosure document that fully documents, with diagrams and hardware specs, the full topology. Further, the paper provides all tuning applied to each component for each platform, and the full test script flow, think times, load test tool used, etc. If something is actually missing, please let me know! I will provide it so you can replicate the tests--but I am completely confident you have all you need already becuase such care was taken already in the benchmark 90+ page report. Please do replicate the tests! Then you will see this is not just vendor FUD...it is a real test, with lots of time taken to ensure fair and accurate reporting. As noted in the document, we encourage customers to replicate the testing, or perform their own bake-offs, perhaps inviting the vendors participating to the shoot out for tuning advice and the like.


We have reviewed the paper and results and found inconsistencies with the best practices for how to run WebSphere Application Server. Assuming items Microsoft chose not to document along with improvements in performance allowed by following best practices, we in fact believe that IBM WebSphere Application Server would win across all the scenarios shown in the results.

Microsoft Response

Be careful here. I tend to avoid making such statement until I have actually performed detailed testing. I have performed such testing on these two apps, and you have not. Please do perform the tests and tell us what you achieve! With appropriate disclosure, of course, so we too can replicate! Such testing and being forced to look at your performance will benefit customers in the end. That is the nature of competition. If you find a missing tuining parameter, be very specific---I will re-run the test on the standard hardware we used. But you need to be very specific here!



You may well ask, if WebSphere Application Server would win, why wouldn’t you say so and publish a contrary IBM report. I don’t believe publishing a proprietary view of performance would help our customers – for all the reasons stated above. At best, if IBM was to respond to this paper, you could expect both vendors to degrade to the lowest common coding styles for implementing the benchmark so they would “win”. As shown already by Microsoft’s implementation, Microsoft wouldn’t choose to use their framework classes and features, but instead code optimal patterns in the application code. When those patterns are not tested and supported by the formal product, no customer wins by seeing the resulting performance.


Microsoft Response

This is just wrong, Andrew. To get a good performing application, smart coding has to be applied. That is true for Java and .NET. Interestingly, there are areas where Trade 6.1 could benefit from some smarter code (for example, the way quotes are retrieved involves too many remote requests). However, to achieve interop with the existing Trade 6.1 implementation, we had to use the same design/coding practice, or else the interop would not work as the middle tier operations would not support the same interfaces. Did we use .NET and Visual Studio to bake in some other non-performance features? You bet! There are some things with configuration, for example, and using a backing RDBMS vs. in-memory non-persisted configuration settings (only read at app startup), and how we synchronize these settings across clustered node members that WebSphere Trade 6.1 does not have. The configuration service is not part of the benchmark, but goes beyond what you have with clustered Trade 6.1 setups and how you must separately config each running node (each time it is restarted)....To get the same, you would have to hand-code more Java code into Trade 6.1--and if a benefit is there, why not? In fact the entire config service done in .NET StockTrader is fully based on Web Services; so you could create a Java implementation that fully integrated with .NET nodes if you wanted!


IBM has a history of competing in standard benchmarking organizations such as SPEC. We do so because such organizations are standards based and unbiased and therefore trusted. Standards-based benchmarking processes give all participating vendors equal opportunity to review run rules, implementation code, and the results of other vendors. Given this, if you find Trade and SOA benchmarks useful, maybe it is time for IBM and Microsoft to jointly propose a SOA benchmark to a standard benchmarking organization. SOA benchmarking under standard benchmarking organizations is where our customers and the industry can truly benefit.


Microsoft Response

As does Microsoft, where it makes sense. For example, in industry-standard benchmarks where pricing information and price/performance ratios are mandatory (ala TPC) so that customers can understand the cost of the tested platform, and vendors cannot simply throw escalating hardware at eachother to out do eachother. As for the difference with Trade 6.1/StockTrader, unlike many Java-based SPEC benchmarks, its easy for customers to replicate on realistic hardware setups; its free for customers to download the code; is based on an IBM-created scenario coded to demonstrate performance of their platform; and *clearly* shows the software performance on the exact same hardware stack of the two tested platforms.....There is great value here, assuming full disclosure is included (downloadable code, full specs, etc as done in our benchmark paper and MSDN Web site).


Summary

I personally talk to many customers about performance, SOA, and web services. I stand behind all that I say technically independent of marketing. I build strong long lasting relationships with these customers, many of whom know me personally. In good faith, I can stand in front of them with results from a standards benchmarking organization. I can stand in front of them showing our SOA leadership based on both customer references and analyst reports. On the other hand, I cannot in good faith show a one-off competitive benchmark run by a single vendor. I hope you can understand this position and it helps you discuss the coverage of this Microsoft report and any similar efforts by any vendor that follow.

This Microsoft report shows basic levels of interoperability and work within the WS-I shows higher levels of interoperability. The Microsoft report points out that Microsoft has to hand code many enterprise ready features that are just taken care of for you in WebSphere technology. The title of this Microsoft report and its follow-on press coverage attempts to confuse the industry on SOA, which points out how desperate Microsoft is to get press coverage in SOA. This Microsoft report doesn’t do a good job of showing the true performance story. Specifically on performance, I encourage all customers to put their trust in standard benchmarking organizations


Microsoft Response

On the contrary, industry-standard benchmarks are good for seeing who is the same league with respect to performance, and has the resources to dedicate to developing and implementing the benchmarks and hence likely to be able to back up their products in enterprise scenarios. However, we *always* encourage customers to do their own testing as well; invite vendors to customer-specified shootouts; do pilot/proof of concept projects; and always load test early, not late. Compare performance, but also productivity and manageability. Also, read third party publications lab reports and benchmarks--we always enjoy participating and wish publications would do more shootouts, but frankly have found often we are the only major vendor willing to participate.

In the end, any customer that *just* puts their faith in benchmark organization's number is *not* doing due diligence in terms of investigating performance/scalability and cost.