Wednesday, June 25, 2008

The Poor (Broke) Man's Java Profiler

When I talk to customers about big performance issues they are experiencing in their applications I usually mention "The Poor Man's Profiler". Actually, when I use this phrase, I am talking about Java core dumps. Java comes with a cool feature where you can send a break signal to the Java process and get a core dumped to the working directory of the application. In that core, if you search for "Full thread dump", you will see what is happening in all the threads of the application. If you do that a few times during load, you can very poorly approximate a Java profiler. If you have an application that won't scale or a Java process that is taking minutes to complete an operation, its amazing how often you can find the issue with Java cores.

Here is how you do it.

  1. Get the application running under load (or if its a single threaded long running operation, just let the operation get started).

  2. Send a BREAK signal to the Java process. In Unix/Linux, you can do this with kill -3 ProccessId. In Windows, you can do this by using a program called SendSignal (Note I don't endorse this tool, but happen to find it useful).

    You'll see the process start to consume exactly one CPU in the system (on a uniprocessor box, 100% CPU, on a 4-way box, 25% CPU, for example). During this time the JVM is doing its job collecting up all the information for the core. At the end it will dump the core to the working directory of the application. Once it is done (could take some time) the CPU will go back to normal.

  3. Let the process continue for a little while, so the application gets back into its normal state.

  4. Then repeat the previous steps starting with the BREAK signal three or more times (the more the better).

  5. Look at the information in the thread dump section (search for "Full thread dump") of the Java core and make sure it looks similar in all three (or more) core dumps. If so, you should have enough certainty to trust what you see in the core dumps.

In a future post, I'll talk about what to look for in the Full Thread Dump, but to get you started, if you see the same method repeatably on the top of the stack of multiple threads, that is what you want to go after first.

In the last two months, I used this approach successfully to find problems in an application server application (bad third party libraries) and a migration script (bad algorithm for handling large amount of data). Both showed rather clearly what needed to be fixed in the program. Also, since this comes with Java, you need no tools to use the "poor man's profiler".

If you have questions or if you find this tip useful, please post a comment on this topic.


Anonymous said...

Calling this a profile is pretty poor , the thread dump has little to do with profiling in the true sense of the word.

Andrew Spyker said...

Actually if you were to do thousands of thread dumps and average them, the result is the same as sampling stack walking profilers. Doing a few, as I pointed out in the article, is a good way to find gross performance problems. This is not to say a good profiler isn't needed to do focused finer grainer performance analysis and improvement. For that there are many profilers I would recommend that aren't for "Poor" or "Broke" men.

Anonymous said...

Andrew, thanks for you post, however appears to be down...could you possibly post a copy of SendSingal.


Andrew Spyker said...

I can't provide SendSignal myself as I don't endorse the tool. It looks like the website is up again now. It is worth noting that on Windows, if you're running a console application, you can use control-break in the console window to force java core dump instead of using SendSignal, as described in the IBM JVM diagnostics guide I linked to from the post. Also, if the process is WebSphere Application Server, we have wsadmin commands I could provide to force a java core dump.

Anonymous said...

Is the same as:

Heap dump:

set jvm [$AdminControl completeObjectName type=JVM,process=appserver1,*]
$AdminControl invoke $jvm generateHeapDump

Thread dump:

set jvm [$AdminControl completeObjectName type=JVM,process=appserver1,*]
$AdminControl invoke $jvm dumpThreads

Andrew Spyker said...

anonymous who posted the admin commands .. thanks!