JMX and concurrency problems

Abstract

In this entry I try to explain that introducing Java Management Extensions (JMX) requires some additional care for concurrency control. If JMX is added without extra care, the system could be subject to all kinds of concurrency related problems like data races and visibility problems.

Introduction

A few months ago I was working on an existing batch application and sometimes strange problems occurred: a process was throwing strange runtime exceptions (like ArrayIndexOutOfBoundsException and NullPointerException). The confusing thing was that everything was (unit)tested very heavy and the tests didn’t rapport any exceptions.

After a deeper inspection we figured out that the process was running concurrently (so multiple instances of that process were running at the same time) and that process was reading and writing to a data structure. The consequence of the concurrent execution was that the data structure got corrupted and that caused the exceptions to be thrown. The strange thing was that this process should not be able to run concurrently. It was executed by Quartz (and wired up in Spring) and by making the job stateful, concurrent execution should be impossible.

After a while I found the cause of the problem: the process was not only executed by Quartz, but also by a JMX client. The MBean (acting in behalf of the JMX client) bypassed the Quartz mechanism completely and just called the process method on the service. The fix was quite easy: instead of calling the method directly, the Quartz scheduling mechanism should be used to schedule the task for immediate execution.

The bigger problem

The bigger problem is that JMX is added into a system without realizing that ‘components’ are now used in a multithreaded environment. The consequence is you will get into all kinds of concurrency related problems like dataraces and visibility problems. Even the most simplest case: setting a property on a component is not without problems if the property isn’t volatile or used in a synchronized context (final also is a perfect but under appreciated way to prevent these problems).

Conclusion

The bottom line is that adding JMX to a system is not something that should be thought of lightly. The fact that certain environments (like Spring) make it very easy to JMX-ify beans, doesn’t mean you should forget about your responsibilities.

About these ads

One Response to JMX and concurrency problems

  1. Damien says:

    You hit the nail on the head. I think a lot of people overlook the fact that a JMX MBean is basically a singleton object with public access (if a SecurityManager is not used) by anyone who knows the ObjectName. The appropriate care should be taken when running within a multi-threaded environment.

    — Damien

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: