Wednesday, September 9, 2009

SPEC working on standard SOA Benchmark

I continue to be interested in helping customers understand the performance of Service Oriented Architect (SOA) applications. As you can see here, I'm working (as the chair) in this SPEC working group, along with considerable input from Oracle and VMware on pushing forward on a standard benchmark for SOA based applications and the middleware infrastructure on which they run.

The interesting parts (in my opinion) of this press release are:

The benchmark will be developed by a trusted benchmarking organization with input from all SPEC members. Also, as mentioned in the press release, we're looking for participation by other interested parties. If you're interested in joining SPEC or providing input, let Bob Cramblitt know. I'm truly excited to see a SOA benchmark come from SPEC as they have a proven track record in creating industry trusted benchmarks for middleware performance.

While the initial focus is Web Services, Enterprise Service Buses, and Business Process Management (BPEL), the group realizes these technologies are only part of the entire SOA picture. It's good to see the group start with a sensible core and grow the effort over time.

The group is working to stay flexible on its support of multiple approaches to implementing these technologies. This is key, as SOA is an architectural approach and there are multiple ways to implement such technologies. However, in an industry standard benchmark it's important to audit and standardize common implementations to confirm they would be used in typical customer implementations.

I'll continue to post publically shareable information as the work group makes progress. If you have any quick questions, post them here and I'll ask them at the working group.


Jinho Choi said...

Oh, it's good idea for ESB folks. But there're many type of ESB in the world and each of them is made by multiple way. What is the criteria of ESB? We can define the benchmark for the 'Java Application Server' as the jAppServer2004 with the Java EE specifications. But how can we define the ESB with some specification?

Andrew Spyker said...


I personally do not believe you need to have a technology standard to define a benchmark. You do need to ensure that the allowable implementations are sufficiently specified by the benchmark rules. Some examples of this in the ESB technology space could be along the lines of requiring that clients are not forced to change is services change in certain ways. You could have only a certain few implementations that the benchmarking organization allowed that are implemented upon standards. An example of this might be XPath/XSLT based implementations vs XQuery based implementations.

I say all this in my personal opinion, as the goal of the working group is to define this in a way that all participants support. I encourage folks with opinions and ideas to join the working group via becoming SPEC members. In that way we can be sure we have considered all options.

PS. I believe in the services space, and service choreography space we have more standards to rely on.