Great set of articles on Native Memory

I am currently helping a customer with a native memory issue causing OutOfMemory (OOM) exceptions. Most folks are familiar with Java heap memory issues (eg. too much caching eating up live memory or leaking objects), but native memory is also important especially when your application isn't so "pure" (spawning threads, lots of JNI, etc). How do you know if you have a native out of memory? You'll see that you'll get out of memory conditions when the Java heap still has plenty of room left over. In the case I'm working on now, we actually saw the out of memory well before the memory in the heap grew to the maximum size (-Xmx) of the heap. In trying to help the customer out, I found the following articles excellent. The first is a very good overview of native memory and how to diagnose. The second is our app server tech note on things to try in the case you believe you have a native memory issue.

Thanks for the memory - Understanding how the JVM uses native memory on Windows and Linux (Windows/Linux version, AIX version).

Troubleshooting native memory issues

Humor for Friday - Java-4-ever

I don't want to post funny videos here too much as I think people come here for technical information. However, this is just too good to not share. There is one slightly NSFW (not safe for work) section. Even mentions XML, so its truly great.

Watch the Java 4-Ever Video

Now on YouTube

WAS V7 XML Feature Pack Beta Refreshed (With focus on XSLT 2.0)

Today we have released an updated XML Feature Pack Beta. The beta is available here.

Since the first release two months ago (which had full XPath 2.0 and partial XSLT 2.0/XQuery 1.0), we have added quite a bit to XSLT 2.0 (making us nearly spec complete) and more to XQuery 1.0.

Among the biggest features added in the XSLT 2.0 area is XML Schema awareness. This includes validation which allows you to validate (make sure it contains data that is allowed per the schema) the incoming data to transformations, temporary trees of data, and output data from transformations. Also, as is important when working with vertical industry standard schemas, our schema awareness support includes schema assessment. This means that your XSLT stylesheets know when an attribute or element content is an integer or date and you can treat it as such. Taking this to the next level, if you extend your industry standard schema type of a bill of materials, you can write templates that match that type (schema-element(*, tns:BillOfMaterialsType)). In the past you would have had to made templates that were of the industry standard element names as well as all extending element names (tns:BillOfMaterialsName, mycompany:FullfillmentBillOfMaterialsName, mycompany:PurchasingBillOfMaterialsName, etc.).

Other new XSLT 2.0 features in this refresh are collections support (allows multiple input data sources in a well defined way), "undeclare-prefixes" serialization parameter(simplifies namespace in XML 1.1), next-match (allows you to call overridden templates) element, required parameters and "as" attributes (helps guarantee stylesheet elements are called correctly) and full support of XSLT keys.

Another major feature of this refresh is ability to pre-compile XML artifacts. This refresh contains tools that compile XPath, XSLT, and XQuery to java classes. These classes are basically the same logic that previously was interpreted by the XML runtime. However, with them now being compiled, they run much faster than interpreted. The refresh offers three modes:

• interpreted - best when the XML artifacts are created ad hoc at runtime and not reused


• compiled - best when the XML artifacts are created ad hoc at runtime but reused


• pre-compiled - best when the XML artifacts are known at compile time of your application

We have also expanded our XQuery 1.0 support in this refresh.

Please let me know if any of these new features help you in your applications. I'm excited to hear of all the applications that are being tested against this functionality.

Finally, I have created another three demos to be posted on YouTube. Over the couple of days, I'll post them with links here. Hopefully they will help you understand how to use this functionality in your own applications.

The Poor (Broke) Man's Java Profiler

When I talk to customers about big performance issues they are experiencing in their applications I usually mention "The Poor Man's Profiler". Actually, when I use this phrase, I am talking about Java core dumps. Java comes with a cool feature where you can send a break signal to the Java process and get a core dumped to the working directory of the application. In that core, if you search for "Full thread dump", you will see what is happening in all the threads of the application. If you do that a few times during load, you can very poorly approximate a Java profiler. If you have an application that won't scale or a Java process that is taking minutes to complete an operation, its amazing how often you can find the issue with Java cores.

Here is how you do it.

1. Get the application running under load (or if its a single threaded long running operation, just let the operation get started).


2. Send a BREAK signal to the Java process. In Unix/Linux, you can do this with kill -3 ProccessId. In Windows, you can do this by using a program called SendSignal (Note I don't endorse this tool, but happen to find it useful).
   
   
   You'll see the process start to consume exactly one CPU in the system (on a uniprocessor box, 100% CPU, on a 4-way box, 25% CPU, for example). During this time the JVM is doing its job collecting up all the information for the core. At the end it will dump the core to the working directory of the application. Once it is done (could take some time) the CPU will go back to normal.


3. Let the process continue for a little while, so the application gets back into its normal state.


4. Then repeat the previous steps starting with the BREAK signal three or more times (the more the better).


5. Look at the information in the thread dump section (search for "Full thread dump") of the Java core and make sure it looks similar in all three (or more) core dumps. If so, you should have enough certainty to trust what you see in the core dumps.

In a future post, I'll talk about what to look for in the Full Thread Dump, but to get you started, if you see the same method repeatably on the top of the stack of multiple threads, that is what you want to go after first.

In the last two months, I used this approach successfully to find problems in an application server application (bad third party libraries) and a migration script (bad algorithm for handling large amount of data). Both showed rather clearly what needed to be fixed in the program. Also, since this comes with Java, you need no tools to use the "poor man's profiler".

If you have questions or if you find this tip useful, please post a comment on this topic.