Speaking at JFall about the JMM

November 3, 2008

At 12 November I’ll be speaking at JFall about the Java Memory Model. This is my first presentation for a large crowd, but this year I have been speaking at least once a month at my employer about mostly concurrency related subjects (low level concurrency, java memory model , java.util.concurrent, databases, architecture and concurrency, fork join framework, mvcc, stm). A few days later I’ll be giving a similar presentation at the IB-groep. Eventually my goal is to give presentations on large international (Java) conferences to become a concurrency authority, and this should help me to get the cool assignments.

So if you want to know more about the JMM, go to my presentation and don’t hesitate to ask questions.

[edit]
If you want to see my presentation (Dutch), you can check it here:
http://www.bachelor-ict.nl/peter-veentjer

Advertisements

Letting the garbage collector do callbacks

July 14, 2008

Garbage collection in Java is great, but in some cases you want to listen when an object is garbage collected. In my case I had the following scenario:

  1. I have a throw away key (a composition of objects) and a value (statistics) in some map.
  2. When one of the components of the key is garbage collected, the key and value should be removed from the map.

The java.util.WeakHashMap was the first thing that came to mind, but there are some problems:

  1. It is not thread safe. And wrapping it in a synchronized block is not acceptable for scalability reasons.
  2. The key (a WeakReference containing the real key) is removed when the real key has been garbage collected. But in my case this isn’t going to work. I don’t want my throw-away key to be removed when it is garbage collected because it would be garbage collected immediately since nobody but the WeakReference is holding a reference to it.

So I needed to go a step deeper. How can I listen to the garbage collection on an object? This can be done using a java.lang.ref.Reference (java.lang.ref.WeakReference for example) and a java.lang.ref.ReferenceQueue. When the object inside the WeakReference is garbage collected, the WeakReference is put on a ReferenceQueue. This makes it possible to do some cleanup by taking items from this ReferenceQueue. This is the first big step to listen to garbage collection.

The other two constraints are now easy to solve:

  1. thread safety: use a ConcurrentHashMap implementation
  2. listen to an arbitrary object instead of the key: wrap the object inside a WeakReference and store the key & map inside. If you create a thread that consumes these references from the ReferenceQueue, this thread is now able to remove the key (and value) from the map

This weekend I created a proof of concept implementation and it can be found bellow.

public class GarbageCollectingConcurrentMap<K, V> {

    private final static ReferenceQueue referenceQueue = new ReferenceQueue();

    static {
        new CleanupThread().start();
    }

    private final ConcurrentMap<K, GarbageReference<K, V>> map = new ConcurrentHashMap<K, GarbageReference<K, V>>();

    public void clear() {
        map.clear();
    }

    public V get(K key) {
        GarbageReference<K, V> ref = map.get(key);
        return ref == null ? null : ref.value;
    }

    public Object getGarbageObject(K key){
        GarbageReference<K,V> ref=map.get(key);
        return ref == null ? null : ref.get();
    }

    public Collection<K> keySet() {
        return map.keySet();
    }

    public void put(K key, V value, Object garbageObject) {
        if (key == null || value == null || garbageObject == null) throw new NullPointerException();
        if (key == garbageObject)
            throw new IllegalArgumentException("key can't be equal to garbageObject for gc to work");
        if (value == garbageObject)
            throw new IllegalArgumentException("value can't be equal to garbageObject for gc to work");

        GarbageReference reference = new GarbageReference(garbageObject, key, value, map);
        map.put(key, reference);
    }

    static class GarbageReference<K, V> extends WeakReference {
        final K key;
        final V value;
        final ConcurrentMap<K, V> map;

        GarbageReference(Object referent, K key, V value, ConcurrentMap<K, V> map) {
            super(referent, referenceQueue);
            this.key = key;
            this.value = value;
            this.map = map;
        }
    }

    static class CleanupThread extends Thread {
        CleanupThread() {
            setPriority(Thread.MAX_PRIORITY);
            setName("GarbageCollectingConcurrentMap-cleanupthread");
            setDaemon(true);
        }

        public void run() {
            while (true) {
                try {
                    GarbageReference ref = (GarbageReference) referenceQueue.remove();
                    while (true) {
                        ref.map.remove(ref.key);
                        ref = (GarbageReference) referenceQueue.remove();
                    }
                } catch (InterruptedException e) {
                    //ignore
                }
            }
        }
    }
}

I am now able to run my concurrency detector on large projects like JBoss instead of running out of memory.


Now officially a Xitaan

June 30, 2008

Last 5 months I haven’t done a lot of software development for my work. I was given the chance to do more consultancy related tasks like performance/stability trouble shooting and audits. I could never have imagined it, but I lost my passion for ‘just’ writing software. I still do write software at home. At the moment I’m working on a tool that is able to detect which objects are accessed by multiple threads. One of the main challenges now is to replace AspectJ by a custom Java agent that gives me more flexibility and doesn’t interfere with other AspectJ configurations. But I’m not satisfied working on projects anymore; I was an unhappy developer and continuing on this road doesn’t provide a bright future.

A few weeks ago I was asked to make the switch officially from the development department (XSD) to the consultancy department (XITA) and I agreed:

  1. XSD (Xebia Software Development) does projects
  2. XITA (Xebia IT Architects): does more consultancy related tasks like audits, trouble shooting, advice etc.
  3. XBR (Xebia Business Requirements): does a lot on the business side of IT like changing the development process or giving business advice.

Within XITA I want to continue specializing in everything related to concurrency and distributed computing. Also I’m thinking about speaking at JFall (a Dutch Java conference) about the Java Memory Model and I’m planning to give an internal? course on this subject as well.


Minimal Agile documentation

November 13, 2007

One of the things that wonder me when looking at source code of Agile projects, is the lack of documentation. The minimal amount of documentation on classes/interfaces is the description of its core responsibility: what is the purpose of this class? It doesn’t need be be very long; a few sentences would be enough. And if the class/interface is part of some design pattern, this should be added to the documentation as well. If the the class is completely self contained, and the pattern is well known (like a Strategy or Decorator), I could live without the documentation but not everyone is a pattern freak, I would rather see it in the documentation.

Another important document is some sort of architectural overview to get a quick understanding of the system without needing to browse through all sources. This document should at least cover the main components of the system, what the responsibilities are of these components and how they interact and why this approach is being used. Without this information it is very hard to say anything about performance, security, scalability, re-usability.

I know that not writing documentation shouldn’t be blamed on the Agile methodology, but on its misinterpretation.

ps:
If you want to remove redundant parts in the documentation, why is the @author tag not removed from the javadoc? If I want to figure out who wrote it (or changed it) I could look at the blame functionality most version control systems provide.


The use of Maven cripples the mind

November 7, 2007

I have used Maven 1.1 and 2 for the last 1.5 years, but it always am boggled by the utter complexity of the tool. There are a lot of good aspects about Maven (dependency management, standard tasks etc) but these advantages don’t weigh up to all the problems. For example: If I want to create a war and an ear, I need to create multiple projects. How stupid can you be?

With ANT (used it for more than 5 years) I would add an extra target and I’m done. I can do whatever I want. That is why I add all functionality to my script so I get a one button release. So even though some stuff in Maven is much easier to set up, in the end I’m spending much more time configuring maven than it would take in a different tool like ANT. Other problems of maven are badly documented tasks, tasks that break for unclear reasons (ANT is as solid as a rock). My bet (and hope) is that Maven in a few years is seen as something bad like EJB 2 and replaced by modern build environments like GANT, Rake, Raven or Build’r. Build languages that are just internal DSL’s; so you can switch to the full blown language without needing to create plugins and it also prevents programming in XML (XML was a bad choice for ANT from day one).


Design patterns and design problems

October 24, 2007

At the moment I’m playing with Ruby (building a Norton Commander thingy based on GTK) and one of the things I’m realizing (again) is that a lot of ‘design’ complexity disappears because Ruby, in a lot of cases, is cleaner and less verbose. For example: in Java you would have to create a Strategy (aka Policy) or Template Method if you need to inject a function, in Ruby you can use a closure. The consequence is that small grained design in Java is difficult, and this is caused by all the syntactic overhead (interface definitions, anonymous innerclasses, template-methods etc).

I don’t say design patterns are bad, but some design patterns (I would rather call them concepts) are harder to implement in some languages, and this inhibits good design: you are damned if you do, and damned if you don’t (if you don’t, your code gets ugly, and if you do, your code gets ugly).

Design pattern = Concept + Implementation
Concepts often are good, implementations often are limited by the host-language.


DSL’s: a higher abstraction to the rescue

June 29, 2007

One of the things that bother me is that it is very easy to loose overview in large wired structures and I think DSL’s can limit the distance between specification and implementation. Example of something set up with a very low abstraction in Spring:

    <bean id="fileWritingProcess"
          class="com.xebia.clustering.processes.FileWritingProcess"/>

    <bean id="resequenceProcess"
          class="org.codehaus.prometheus.processors.ResequenceProcess"/>

    <bean id="fileWritingProcessor"
          class="org.codehaus.prometheus.processors.standardprocessor.StandardProcessor">
        <constructor-arg index="0"
                         type="org.codehaus.prometheus.channels.InputChannel"
                         ref="fileWritingProcessor-input"/>
        <constructor-arg index="1">
            <list>
                <ref bean="resequenceProcess"/>
                <ref bean="fileWritingProcess"/>
            </list>
        </constructor-arg>
    </bean>

    <bean id="fileWritingProcessorRepeater"
          class="org.codehaus.prometheus.repeater.ThreadPoolRepeater"
          init-method="start"
          destroy-method="shutdown">
        <constructor-arg index="0">
            <bean class="org.codehaus.prometheus.processors.ProcessorRepeatable">
                <constructor-arg ref="fileWritingProcessor"/>
            </bean>
        </constructor-arg>

        <property name="exceptionHandler">
            <bean class="org.codehaus.prometheus.exceptionhandler.Log4JExceptionHandler"/>
        </property>

        <property name="shutdownAfterFalse"
                  value="true"/>
    </bean>

And this is only 1/4 of the XML configuration.

Or now written in a DSL in Groovy (I’m still playing with the concepts and Groovy, so it could be that the syntax isn’t correct).

def piped = environment{
    queues{
        queue(name:'queue1'),
        queue(name:'queue2'),
        queue(name:'queue3')
    }

    pipeline{
        [parserprocess,
         queue1,
         parallel(threadcount:10){fibonacciprocess},
         queue2,
         parallel(threadcount:10){piprocess},
         queue3,
         [resequenceprocess,filewritingprocess]]
    }
}

I think most can imagine what the last script is doing without knowing anything of the application. That is the power of a DSL.