In this entry I try to explain that introducing Java Management Extensions (JMX) requires some additional care for concurrency control. If JMX is added without extra care, the system could be subject to all kinds of concurrency related problems like data races and visibility problems.
A few months ago I was working on an existing batch application and sometimes strange problems occurred: a process was throwing strange runtime exceptions (like ArrayIndexOutOfBoundsException and NullPointerException). The confusing thing was that everything was (unit)tested very heavy and the tests didn’t rapport any exceptions.
After a deeper inspection we figured out that the process was running concurrently (so multiple instances of that process were running at the same time) and that process was reading and writing to a data structure. The consequence of the concurrent execution was that the data structure got corrupted and that caused the exceptions to be thrown. The strange thing was that this process should not be able to run concurrently. It was executed by Quartz (and wired up in Spring) and by making the job stateful, concurrent execution should be impossible.
After a while I found the cause of the problem: the process was not only executed by Quartz, but also by a JMX client. The MBean (acting in behalf of the JMX client) bypassed the Quartz mechanism completely and just called the process method on the service. The fix was quite easy: instead of calling the method directly, the Quartz scheduling mechanism should be used to schedule the task for immediate execution.
The bigger problem
The bigger problem is that JMX is added into a system without realizing that ‘components’ are now used in a multithreaded environment. The consequence is you will get into all kinds of concurrency related problems like dataraces and visibility problems. Even the most simplest case: setting a property on a component is not without problems if the property isn’t volatile or used in a synchronized context (final also is a perfect but under appreciated way to prevent these problems).
The bottom line is that adding JMX to a system is not something that should be thought of lightly. The fact that certain environments (like Spring) make it very easy to JMX-ify beans, doesn’t mean you should forget about your responsibilities.