WebSphere first on SPECjEnterprise2010 Java EE Benchmark

In case you missed it, we published the first two results for a new benchmark - SPECjEnterprise 2010.

This new benchmark covers the Java EE 5.0 programming model running on an application server. You may remember SPECjAppServer 2004 and our work to lead in that benchmark. Given how old the 2004 benchmark is, it no longer represents the common practices of coding of new applications. This third version of Java enterprise application server benchmark covers areas such as the simplified Java EE 5.0 programming model for persistence and web programming, web services, and messaging.

By being first to publish, IBM continues to demonstrate its commitment to driving standard third-party trusted benchmarking. Also, we show how the WebSphere Application Server really shines on Java EE support in terms of being consistently first to market with highly performant programming models that matter to you. We published both on a simple single server (1) as well as a highly available, scalable cluster configuration (2) which demonstrates WebSphere Application Server 7.0's ability to scale from simple to complex application environments.

If you want to chat about this benchmark, IBM's results, or see some other standardized performance work we're doing at SPEC and can be in the San Jose area at the end of the month, please stop by the "First Joint WOSP/SIPEW International
Conference on Performance Engineering". I'll be in attendance talking about the SPEC SOA benchmark work.

(1)IBM SPECjEnterprise2010 result of 7903.16 EjOPS using WebSphere Application Server V7 on IBM BladeCenter HS22 (8 nodes, 64 cores, 16 chips) and DB2 9.7 on IBM System x3850 (1 node, 24 cores, 4 chips).
(2)IBM SPECjEnterprise2010 result of 1,013.40 EjOPS using WebSphere Application Server V7 on IBM System x3650 (1 node, 8 cores, 2 chips) and DB2 9.7 on IBM System x3850 (1 node, 12 cores, 2 chips).
Source: http://www.spec.org; Results current as of 01/08/10.

In the news .. SPEC SOA benchmark effort

I wanted to shared some press coverage of the effort I mentioned back in September.

The original SPEC press release press release talked about how the work group was looking for participation, what the initial focus was on, and the unique challenges of producing a SOA benchmark.

In coverage by SearchSOA and TechTarget, the working group has been asked very interesting questions. The SearchSOA article talks about how SPEC is a good organization to take on the challenge of a SOA benchmark given its past track record in JEE middleware benchmarking and talks about the challenges of benchmarking typical implications of a service oriented architectures. The TechWorld article talks adds discussion of how the benchmark is focused on some core aspects of SOA but will likely grow over time and discusses how the benchmark will need to consider both IT and business requirements on typical SOA applications.

Also, the effort has been covered by TheServerSide and InfoQ.

As mentioned in the original press release, if your organization can contribute to the effort, contact info@spec.org

Also, kind of fun to see your quotes translated into various languages - Japanese, German for example. I do think this level of coverage indicates the value this benchmark will deliver to both technical and business communities.

SPEC working on standard SOA Benchmark

I continue to be interested in helping customers understand the performance of Service Oriented Architect (SOA) applications. As you can see here, I'm working (as the chair) in this SPEC working group, along with considerable input from Oracle and VMware on pushing forward on a standard benchmark for SOA based applications and the middleware infrastructure on which they run.

The interesting parts (in my opinion) of this press release are:

The benchmark will be developed by a trusted benchmarking organization with input from all SPEC members. Also, as mentioned in the press release, we're looking for participation by other interested parties. If you're interested in joining SPEC or providing input, let Bob Cramblitt know. I'm truly excited to see a SOA benchmark come from SPEC as they have a proven track record in creating industry trusted benchmarks for middleware performance.

While the initial focus is Web Services, Enterprise Service Buses, and Business Process Management (BPEL), the group realizes these technologies are only part of the entire SOA picture. It's good to see the group start with a sensible core and grow the effort over time.

The group is working to stay flexible on its support of multiple approaches to implementing these technologies. This is key, as SOA is an architectural approach and there are multiple ways to implement such technologies. However, in an industry standard benchmark it's important to audit and standardize common implementations to confirm they would be used in typical customer implementations.

I'll continue to post publically shareable information as the work group makes progress. If you have any quick questions, post them here and I'll ask them at the working group.

IBM again leads Oracle (and everyone else) on SPECjAppServer2004 performance and cost

I recently had someone email me on a performance topic and they referenced WebSphere Application Server's SPECjAppServer leadership. They actually didn't know that IBM regained total leadership in this benchmark late last year. Maybe the fun of the holidays didn't allow folks to notice? I wanted to post this update, so if other folks missed it they would notice.

In these times of tough economic conditions, it's really important to focus on total cost of ownership and ways to improve efficiency. IBM's continuous leadership in performance proves we care about performance as it relates to end to end J2EE enterprise computing scenarios. This increased performance means you can host more workload and applications on the same resources that you used before. Our performance engineers are continuously improving performance of your applications and we use SPECjAppServer to demonstrate those improvements.

Sometimes people ask me how SPECjAppServer matters to them. It's this continuously increasing performance that matters to them. Originally SPECjAppServer helped the J2EE programming model transition from a programming model to a capable enterprise ready programming model (stressing basic performance and scalability). Now, SPECjAppServer leadership helps customers know the middleware they are buying from IBM is helping them improve their efficiency.

This view is a technology view (leadership at 22634.13 JOPS), however it has business impacts as well. That result was achieved at roughly one half of the cost of our closest competitor. That just acquisition cost. If you consider the power savings on top of that you're not only being more green, but saving green.

Again, great work by the WebSphere Application Server performance team!

Disclaimer: SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks to measure the performance of the newest generation of high-performance computers. Its membership comprises leading computer hardware and software vendors, universities, and research organizations worldwide. For complete details on benchmark results and the Standard Performance Evaluation Corporation, please see www.spec.org. Competitive claims reflect results published on www.spec.org as of January 23, 2009 when comparing SPECjAppServer2004 Total JOPS on all published results.

More information on the TPC SOA benchmark

As I mentioned previously, the TPC (Transaction Processing Performance Council) is looking at SOA benchmarking. SearchDataCenter.com has an article that summarizes alot of the new and upcoming TPC work. You will see the TPC chairman, Mike Molloy, talk about the popular existing TPC benchmarks (TPC-C, TPC-E, TPC-H, and TPC-App) as well as new benchmarking efforts around energy (consumption with focus on greening of the data center), virtualization (across all new workloads), ETL (Database Extract, Transform, Load), and SOA.

On SOA, Mike explains, "TPC's SOA benchmark is only in the proposal stage, but the tentative plan is to focus on common industry-accepted portions of SOA infrastructure, mainly Web services, the enterprise service bus, and business process choreography. As advanced SOA practices become more standard in the industry, TPC will expand the benchmark to incorporate additional SOA infrastructural services".

I'm excited to work within the TPC to continue driving this SOA performance benchmark to reality. There are no other standard benchmarks that tackle these common SOA infrastructural components. If you have interest in seeing such a benchmark or comments on how it would help you, please post a comment. I'll relay them back to the TPC.

EJB 3.0 Performance Improvements in WAS V7.0 (up 23%)

EJB 3.0 was actually available for some time on top of WAS V6.1 in the Feature Pack for EJB 3.0, but now it exists in WAS V7.0 as well.  EJB 3.0 in my mind is all about making the developer's life simpler (annotations, test outside of container, etc.) along with giving us a better model for more performant persistence.  You can see all new features added to EJB 3.0 in WAS V7.0 (including performance features) in Kevin's blog post.  Given this is the second major release of EJB 3.0 for WebSphere, performance of EJB 3.0 on WAS is top notch.  You'll also see that we've recently added an implementation using the standard JPA caching functions to allow caching via WebSphere eXtreme Scale (Formally known as WebSphere Extended Deployment Data Grid) which is a distributed in-memory data grid.

In working on persistence performance in WAS V7.0, we used the Apache DayTrader application with some IBM tweaks.  There are two versions of DayTrader of interest (1.2 which uses EJB 2.1 and 2.0 which uses EJB 3.0).  There are other changes as well to move DayTrader 2.0 to Java EE 5, but most don't have deep performance impacts.  Really, the big change from a performance perspective is the implementation of the persistence layer.

The results of all of the new WAS V7.0 features is a very performant EJB 3.0 implementation that outpaces the EJB 2.1 performance by 23% as shown in the below chart.  This is a rather impressive improvement considering the EJB 2.1 implementation has been grooved by years of leading IBM benchmark results in SPECjAppServer.  It is also worth noting that the below chart is EJB 2.1 to EJB 3.0 heads up (actually both are measured on WAS V7.0).  If you are migrating from WAS V6.1 or earlier the percentage improvement would be far greater.

Again, I'm interested in feedback from users who have tried the new engine.

Web Service Performance Improvements in WAS V7.0 (up to 3x)

Two weeks ago, I said I'd be doing a weeklong series of posts on WAS V7.0 performance.  A week late, but here we go.  I guess I can say I delayed until WAS V7.0 was generally available, which it is now (trial download here)!  WAS V7.0 should be available by other supported download methods as well.

There are many impressive areas of performance improvements in WAS V7.0, but I'll start with the one nearest and dearest to my heart - Web Services.  In the past, I talked about web services performance through the years.  I mentioned a possible follow-up blog post on what Feature Pack for Web Services performance.  I never followed up on that (hrm, a theme in my slow blogging), but now that the same code is available in WAS V7.0, I will.  In fact, we've made even more improvements to web services performance in WAS V7.0 that would truly make the "performance through the years" chart from my previous blog post much more impressive.

WAS V6.1 supports the J2EE 1.4 JAX-RPC web service programming model.  Last year, we added Java EE 5 support for the JAX-WS web service programming model (expanded functionality, standards support, easier programming model, and enhanced management support for services) through the Feature Pack for Web Services.  While WAS V7.0 continues support of both the JAX-RPC and JAX-WS programming models so that existing JAX-RPC web services applications deployed on previous versions of WebSphere will run on V7 unmodified, the development and performance resources were almost exclusively dedicated to the JAX-WS work.  We encourage customers to move to this new programming model to take advantage of the new features as well as the significant performance gains.  Many customers I know of have moved to the Feature Pack for Web Services already, but now that it's natively supported in WAS V7.0, I expect even more customers to make a move from JAX-RPC to JAX-WS.

The chart below compares WAS V6.1 JAX-RPC performance to WAS V7.0 JAX-WS performance.  JAX-WS web services significantly outperform JAX-RPC web services for all message payload sizes shown.  For a 3k(input)/3k(output) message size, JAX-WS achieves a 70% higher throughput and for 10k/10k message size JAX-WS achieves over 2x higher performance.  For 10k/3k, 10k/10k, 100k/100k JAX-WS achieves over 3x/2.5x/2.5x higher performance, respectively.

There are many reasons for these performance gains, but the most significant one is the optimized data binding using JAXB in JAX-WS.  JAXB provides JAX-WS the framework to unmarshal XML documents into Java objects or marshal Java objects into XML.  We have done very impressive optimizations in our JAXB implementation to achieve these performance gains, well beyond what most JAXB implementations can do with StAX or SAX.  Getting high performance XML parsing combined with JAXB's full support for XML Schema is an impressive accomplishment.

Some things that matter to you specifically:  This is a primitive benchmark as it only measures web services performance (no business logic).  Therefore, your application using web services will not go 2-3 times faster just by switching to JAX-WS and JAXB, but it should markedly improve.  Specifically the additional latency involved in exposing business logic to web services and XML should go down by 1/2-2/3, which in many scenarios is substantial.  Also, even though you can handle XML yourself in JAX-WS by getting the data as a stream or DOM, it's best if you use JAXB as you'll get the most performance benefit of our optimizations.

I'm interested in feedback from users that have tried the new engine.

The TPC is working on a SOA Benchmark proposal submitted by IBM

Given the comments I made previously on SOA benchmarking, I'm happy to report that the Transaction Processing Performance Council (TPC) has agreed to evaluate an SOA benchmark proposal submitted by IBM. You can see in the recent "Meet the TPC" working session, the TPC discussed the TPC SOA workgroup on page 8. The summary is rather high level - "Tasked to evaluate SOA benchmark proposal from IBM". I believe if this proposal is accepted with modifications by other TPC participants, customers of SOA solutions and the industry can truly benefit from the TPC's standards-based benchmarking experience and levels of customer trust. I personally will be working to make this workgroup a success, with the intent of making SOA performance as understood as hardware, database, and J2EE performance. If there are companies out there with SOA solutions, I would personally encourage them to join the TPC and participate in this or other benchmarks and workgroups. I hope to, in the future, blog about the success of this workgroup and benchmark.

IBM WebSphere Application Server Total Leadership in SPECjAppServer2004

As noted in the IBM press release, we recently published a SPECjAppServer 2004 result that puts IBM on top of BEA, Oracle, and Sun for the absolutely largest SPECjAppServer 2004 throughput ever achieved. This result comes after we dominated the same competitive crowd on per cpu core performance as I blogged about previously. Both of these results matter. The per cpu core leadership shows application server performance in the most competitive light. The total configuration shows application server scalability in the most competitive light. IBM WebSphere dramatically wins both.

Some numbers from this total leadership publish:

• The result of 14004 JOPS represents roughly 75,000 transactions per second against the DB2 database. Simply impressive.
  
• The closest competitor in total throughput was Oracle at 10519. The IBM result tops that Oracle result by 33%. The IBM result tops the highest Sun result (8439) by 66%. The IBM result tops the highest BEA result (8253) by 70%.
  

Congrats to John Stecher and team. You can read more about the total configuration publish (and John's team) on InfoQ.

SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks to measure the performance of the newest generation of high-performance computers. Its membership comprises leading computer hardware and software vendors, universities, and research organizations worldwide. For complete details on benchmark results and the Standard Performance Evaluation Corporation, please see www.spec.org. Competitive claims reflect results published on www.spec.org as of February 18, 2008 when comparing SPECjAppServer2004 JOPS on all published results as well as derived metrics calculated by SPECjAppServer2004 JOPS / Number of Application Server Processor Cores.

WebSphere Beats BEA and Oracle Performance By Over 37%

I wanted to take the time to congratulate some of my performance team members who have worked to prove our performance leadership, displacing both of the current top SPECjAppServer 2004 leaders - BEA and Oracle. You can see this documented in our latest launch press release here.

"A powerful IT infrastructure is also required to support the growing number of transactions across the enterprise, as well as with customers, partners and suppliers. IBM recently beat competitors by 37 percent in an industry benchmark that measures high-volume transaction processing that are typical in today's customer environments. IBM WebSphere Application Server established record-breaking SPECjAppServer2004 benchmark performance and scalability results involved more than 15,500 concurrent clients and produced 1,197.51 SPECjAppServer2004 JOPS@Standard (jAppServer Operations Per Second), which translates into more than 4.3 million business transactions over the course of the benchmark's hour long runtime."

This shows our commitment to standard benchmarking organizations as I described in my last blog post. By leading this benchmark that was designed and approved by all participating SPEC vendors, IBM customers truly benefit from the demonstrated performance leadership.

We chose to lead in a category that matters most to our customers - per core performance. If you look at the result and the number of J2EE server cores and divide it out, we lead Oracle by 37% and smash BEA by 56% with approximately 300 JOPS/core. Comparing per core (CPU) normalized performance is really the best way to compare one application server and hardware stack to another. This demonstrates IBM's leadership of the industry in J2EE performance without question.

This result also shows the combined power of an IBM stack. Specifically, we achieved this by running WebSphere Application Server v6.1 on the IBM 5.0 JVM on AIX 5L V5.3 with an IBM POWER6 p570 server powered by two dual-core IBM® POWER6® 4.7 GHz processors against IBM DB2 Universal Database v9.1 running on a single POWER6-based server.

Congrats again to my team who yet again dominates where it matters - in customer focused standards based benchmarks.

SPEC is a non-profit organization that establishes, maintains and endorses standardized benchmarks to measure the performance of the newest generation of high-performance computers. Its membership comprises leading computer hardware and software vendors, universities, and research organizations worldwide. For complete details on benchmark results and the Standard Performance Evaluation Corporation, please see www.spec.org/. Competitive claims reflect results published on www.spec.org as of October 03, 2007 when comparing SPECjAppServer2004 JOPS@Standard per CPU Core on all published results.

Perspective in Benchmarks - My thoughts on Microsoft StockTrader

I wanted to take the time to respond to claims coming from Microsoft. On June 4th, Greg Leake, Technical Marketing Manager for .NET of Microsoft started to discuss an effort with which he was involved. The effort was “porting” IBM’s WebSphere Performance Sample – Trade 6.1 to Microsoft .NET 3.0. Since then, Microsoft has been interviewed by trade press, has created video podcasts, and has started discussions on Microsoft forums and other development community sites. Please do not be fooled by these marketing statements.

In my opinion, Microsoft uses their report to focus on three points

- Attempt to prove that Microsoft is interoperable with WebSphere software
- Attempt to prove Microsoft’s SOA relevancy and readiness for the enterprise
- Attempt to prove Microsoft performance is better than WebSphere software performance

I’d like to state up front that I was personally involved with the development of IBM’s Trade benchmark starting with Trade 3 and ending with Trade 6.1. Others on my team have made significant contributions to the code as well to ensure that Trade continues to be an credible and useful tool to express performance concepts and release to release enhancements to IBM customers for J2EE (Java EE) programming.

Some level of interoperability

I applaud any effort by Microsoft or any other software vendor in pursuit of interoperability. However, the interoperability the Microsoft report speaks to is basic web services functionality (SOAP/WSDL/XML) only. It does not focus on interoperability of transactions, security, reliability, durability and does not use industry standard schemas that many of our customers need for cross enterprise (B2B) or intra-enterprise web services interoperability with .NET clients calling WebSphere servers. Independent of the Microsoft report, Microsoft and IBM have already focused on interoperability of these higher value qualities of service in Web Services through industry standard WS-I efforts as shown here and here. I am very happy to see IBM and Microsoft continue to focus on standards based interoperability. I am confident that WS-I will continue to facilitate customer focused web services interoperability in these higher value web service functionalities.

Is Microsoft enterprise ready?

Press coverage has cited the Microsoft report as helping Microsoft prove it is ready for the enterprise. From our quick look at Microsoft code, this doesn’t seem to be the case. Many of the “enterprise ready” features stressed by the Microsoft report are hand coded into the application. Areas such as multiple vendor database support, load balancing, high availability, scalability, and enterprise wide configuration for services contribute to the significantly higher lines of code count in the Microsoft application as compared to the WebSphere implementation. You should equate higher lines of code count to more maintenance and support costs long term and think about the value of this being provided in the application server product versus in the application itself. Some of the increased lines of code are for comments, but the comments themselves point out how many places where Microsoft deviated from using framework classes and instead implemented custom extensions to fill in gaps of functionality lacking in their framework.

As an author of Trade, I must admit an embarrassing fact. Four years ago, my team added web services capability to Trade as an optional component – to demonstrate web services as an alternative to remote stateless session bean methods. Since that time, I personally have worked hand in hand with many enterprise customers adopting web services. We have found that the Trade approach to web services wasn’t the best. Specifically the fine grained “services” in Trade average around 400-800 bytes of passed data. As you’ll see in my recent blog post, industry standard schemas for B2B typically have much larger payloads. While Trade was a fun exercise for my team to learn web services, it in no way mirrors what we know now our customers have told us are good web service practices. Current SOA principles motivate more coarse gained services. The embarrassing fact is that we never have gotten around to removing the poor examples of web services usage from Trade. However, it is interesting to note that Microsoft did not recognize or point out the obvious flaws in these web service patterns during their analysis – they merely parroted what they saw.

Microsoft entitled this paper as “Service Oriented”. Later in the press, Microsoft alludes to this “Service Oriented” benchmark test to help draw credibility to Microsoft’s SOA strategy. Based on what I just said about Trade web services, you’ll see why IBM has never talked about Trade and web services to help customers understand web service performance. Meanwhile, this Microsoft report focuses on these trivial payload web services to prove they can support “Service Oriented” benchmarks. Follow-up posts and press coverage followed the “Service Oriented” title, proclaiming that this in some way helps Microsoft’s SOA strategy. Drawing that conclusion is incorrect. The industry agrees that SOA is more than trivial web services. SOA is about a business centric IT architecture that provides for a more agile and flexible IT support of business needs. Realistic web service usage is a cornerstone of SOA, but web services alone do not define SOA.

Is Microsoft performance better than WebSphere performance?

I believe Microsoft is trying to draw IBM out of our commitment to standard benchmarking organizations to confuse customers about performance. You can draw your own conclusions from specific comments apparently made by Greg Leake in press coverage, asking for IBM to “collaborate”. Some have commented correctly on community discussions that the Microsoft report isn’t specific enough in terms of topology, hardware, and tuning to make any sensible conclusions based on the performance data. It is a compelling story that this Microsoft report weaves – Microsoft beating IBM on its own benchmark. However, IBM didn’t run Trade as a benchmark in the way shown in the paper’s results. As a customer, you should always be careful how much you trust proprietary benchmark results produced by a single vendor. These things can always be coerced to create FUD and confusion.

We have reviewed the paper and results and found inconsistencies with the best practices for how to run WebSphere Application Server. Assuming items Microsoft chose not to document along with improvements in performance allowed by following best practices, we in fact believe that IBM WebSphere Application Server would win across all the scenarios shown in the results.

You may well ask, if WebSphere Application Server would win, why wouldn’t you say so and publish a contrary IBM report. I don’t believe publishing a proprietary view of performance would help our customers – for all the reasons stated above. At best, if IBM was to respond to this paper, you could expect both vendors to degrade to the lowest common coding styles for implementing the benchmark so they would “win”. As shown already by Microsoft’s implementation, Microsoft wouldn’t choose to use their framework classes and features, but instead code optimal patterns in the application code. When those patterns are not tested and supported by the formal product, no customer wins by seeing the resulting performance.

IBM has a history of competing in standard benchmarking organizations such as SPEC. We do so because such organizations are standards based and unbiased and therefore trusted. Standards-based benchmarking processes give all participating vendors equal opportunity to review run rules, implementation code, and the results of other vendors. Given this, if you find Trade and SOA benchmarks useful, maybe it is time for IBM and Microsoft to jointly propose a SOA benchmark to a standard benchmarking organization. SOA benchmarking under standard benchmarking organizations is where our customers and the industry can truly benefit.

Summary

I personally talk to many customers about performance, SOA, and web services. I stand behind all that I say technically independent of marketing. I build strong long lasting relationships with these customers, many of whom know me personally. In good faith, I can stand in front of them with results from a standards benchmarking organization. I can stand in front of them showing our SOA leadership based on both customer references and analyst reports. On the other hand, I cannot in good faith show a one-off competitive benchmark run by a single vendor. I hope you can understand this position and it helps you discuss the coverage of this Microsoft report and any similar efforts by any vendor that follow.

This Microsoft report shows basic levels of interoperability and work within the WS-I shows higher levels of interoperability. The Microsoft report points out that Microsoft has to hand code many enterprise ready features that are just taken care of for you in WebSphere technology. The title of this Microsoft report and its follow-on press coverage attempts to confuse the industry on SOA, which points out how desperate Microsoft is to get press coverage in SOA. This Microsoft report doesn’t do a good job of showing the true performance story. Specifically on performance, I encourage all customers to put their trust in standard benchmarking organizations