Developing software is like working with iron

When I talk to family about my job, it is always hard to explain what I’m doing. Yes.. I write software.. and no.. I can’t help you with your Windows problems. But some parts of software development can be compared to working with iron:

1. When you start, you can create small parts and connect them to create bigger parts, just like with Lego or Meccano
2. When you have accomplished a lot, in some cases it feels good to do fine tuning and put the dots on the i by getting a very fine sandpaper and polishing it: just feeling proud.
3. If there already is existing software and some parts aren’t not behaving as you want it to behave, you get a crowbar and try to bend it in shape. In some cases a crowbar isn’t sufficient and you need to get something like a sledge hammer or blowtorch
4. In some cases the quality is that bad, that a sledge hammer will not help. The best thing to do is to melt everything and start from scratch.

Multiverse and the 2 Phase Commit

Currently I’m working on a 2 phase commit in Multiverse; a Java based Software Transactional Memory implementation. The cool thing about the 2 phase commit in Multiverse is that isn’t slow (something most developers fear when they hear about 2 phase commit). It only says that the transactions should do the commit in 2 parts:

1. prepare: make sure that the transaction has no read/write conflicts and make sure that the sources are ‘locked’ so that a commit is going to succeed
2. commit: does the actual writing of the changes

So if multiple transactions need to commit/abort atomically; the all need to be prepared first. If this succeeds, they all can continue to the commit (that is not going to fail). And if the prepare of one of the transactions fails, they are all aborted. A 2 phase commit in Multiverse can be done completely in memory in a single JVM without any network communication, so it is very very fast.

CommitGroup

I’m still working on a nice ’synchronization’ structure called the CommitGroup. The CommitGroup is the token that needs to be shared between all transactions that all need to commit atomically:

CommitGroup group = new CommitGroup();

When a transactions want to participate in the commit group, and are not able to complete it, you can do something like:

group.waitForCommit(transaction)

And if it is the transaction that is able to complete all the commits of the CommitGroup, you can do something like this:

group.completeCommit(transaction)

Isn’t that cool? The 2 phase commit and the CommitGroup are going to be part of Multiverse 0.4 which is going to be released at the end of this month on the new location at Codehaus, including a complete new website with a big reference manual, tutorials, introductions and of course a lot of new features and performance improvements. The main focus is going to shift from creating a cool toy project, to something really usable like Hibernate.

Multiverse: Non Blocking Transactions

If an application is using traditional locks, it it quite hard to use a different model then a thread per operation because once a thread enters a lock (a lock you don’t have any control over), the thread is ‘lost’. With IO there is a special extension (non blocking IO) that makes it possible to bypass this behaviour so you can have one thread serving multiple sockets. But if you use traditional concurrency control then you are on your own.

With Multiverse (a Java based STM implementation) it is not only possible to block on multiple resources (for more information see the retry/orelse primitives in the STM literature), but in the 0.4 release it also is going to be possible to use a single thread to run multiple blocking transactions. For some time the idea was in the back of my mind and I had the gut feeling that it should be possible with STM. But I never gave it any serious thought because there is so much else to do. Last Friday I had a day off and apparently it was a good day because in 30 minutes I had something up and running.

In Multiverse a latch is used to register transactions as ‘being interested’ in changes of transactional objects. A latch is a blocking structure that can be opened, but once it is opened it can’t be closed, and others threads can wait (block) on the opening of the latch. If a retry happens (so a retryerror is thrown) the transactionmanager catches it, creates a latch and registers it at the read objects. Once a change happens on one of the read objects in the transaction, the latch is opened and the transaction can continue (technically it is restarted). But because the transaction/transactionmanager controls which latch implementation is used, there is a lot of design freedom.

How do non blocking transactions work?
Instead of relying on a traditional latch (so based on an intrinsic lock or a java.util.concurrent.locks.Condition), I replaced it with a different implementation:


public class NonBlockingLatch implements Latch {

private final AtomicBoolean open = new AtomicBoolean();
private final Runnable task;

public NonBlockingLatch(Runnable task) {
this.task = task;
}

@Override
public void open() {
if (open.compareAndSet(false, true)) {
tasks.put(task);
}
}

@Override
public boolean isOpen() {
return open.get();
}

@Override
public void awaitUninterruptible() {
throw new UnsupportedOperationException();
}

.....
}


This latch is added to all the transactional objects in the transaction, and it does a callback once it is opened. In this case it does a callback to some taskqueue of an executor and stores the task in the taskqueue so that the transaction can restart. In the 0.4 release of Multiverse an initial version of this functionality will be added.

Why should you want it?
The cool thing about non blocking transactions is that you could create 100.000 transactions for example while using a few hundred threads. You could build a trading system e.g. where transactions wait on some stock price to go under or over some threshold and buy/sell the stocks using a transaction on one or more transactional resources.

But the mechanism isn’t completely perfect; there is a lot of room for fairness/starvation prevention and at the moment the transaction is restarted on every change. And with this implementation it also is possible that the same transaction is run in parallel if multiple transactional resources are notified used in the same transaction. It seems that I don’t need to be bored the coming weeks with my Christmas leave.

Plans for Multiverse 0.4

The 0.3 release of the software transactional memory implementation Multiverse is almost complete. I already started with the 0.4 release and it will get the following transaction execution models:

1. readonly transaction with tracking reads; useful blocking operations and also to prevent unwanted load failures once a read has been executed
2. readonly transaction without readtracking (already available in the 0.3 release)
3. update transaction without read tracking (reduces stress on the transaction), but it could be subject to writeskews and can’t be used for blocking operations
4. update transaction with read tracking (already available in the 0.3 release). This is useful for blocking operations but also for detecting write-skew problems. In the 0.4 detection for writeskew detection will be configurable

The 0.4 release also is going to get a mechanism that selects the optimal transaction implementation based on the number of reads/writes done on a transaction.

1. tiny: optimized for a single atomic object (completely optimized to reduce object creation as much as possible)
2. bound length: optimized for a maximum number of atomic objects e.g. 10 (based on an array)
3. unbound length: optimized for bigger transaction (based on a map)

I have created some prototypes the show a big performance improvement for tiny transactions (2 a 3 times faster than growing-transactions). And based on the transaction familyname (if annotations are used, family name will be inferred automatically), the system will select the optimal implementation. The systems starts with tiny transactions, and if an implementation wasn’t ‘big’ enough, the transaction is aborted and a ‘larger’ implementation (or different settings) are used for the following transaction. This is completely invisible to the programmer apart from having some transaction failures in the beginning. But since transactions are retried automatically, this shouldn’t be a big problem.

The following features are also planned (or already partially implemented) for the 0.4 release:

1. TransactionalTreeMap
2. TransactionalTreeSet
3. Transactional primitives (IntRef, BooleanRef etc)
4. Support for subclassing atomic objects.
5. Preventing unwanted object creation in transactions (Tranlocal objects are only cloned for local usage when they are written, not when read)
6. Support for 2 phase commit to make distributed transactions possible. This was requested by Jonas Boner of the Akka project that used Multiverse as the STM implementation.

Am I too stupid for @Autowired?

When I started with Spring, I finally had the feeling that I could write enterprise application in the same powerful way as I could write normal systems. And I have written quite complex Java stuff like expertsystems, prolog compilers and various other compilers and interpreters and currently working on Multiverse; a software transactional memory implementation. I also have written quite a lot of traditional enterprisy Spring applications (frontend, backend/batch processing). So I consider myself at least as smart as most developers.

The first applicationcontext files felt a littlebit strange, finding the balance between the objects that need to be created inside the application context and objects created in the Java objects themselves. But it didn’t took me long to find this balance and realise how powerful the application context is:

1. it acts like a executable piece of software documentation where I can see how a system works, just by looking at a few xml files. I don’t need to look in the source to see how stuff is wired.
   I compare it with looking at Lego instructions, by looking at the instructions I can see what is going to be build (not that I ever followed the design). To understand what is build, I don’t need to look at the individual parts.
2. separation of interfaces and implementation, so testing and using a lot of cool oo design stuff is a dream
3. having the freedom to inject different instances of the same interface on different locations. This makes it possible to do cool stuff like adding proxies, logging etc etc. I wasn’t forced to jump through hoops any more because of all kinds of container specific constraints

With the old versions of Spring (1.2 series) I had the feeling of total control and complete understanding.

The application context made me aware that I needed to work with 2 hats:

1. ‘Object’ designer; so actually writing the Java code and documentation
2. ‘Object’ integrator; where I assembly complete systems based on the objects

And I loved it.. I have more than 1.000 post on the Spring forum just because I believe in this approach.

But if I look at a lot of modern Spring applications, filled with @Component and @Autowired annotations, it feels like I have lost it all. It takes me a lot longer to realise how something works, even though I have a great ide (newest beta of IntelliJ) with perfect Spring integration that makes finding dependencies a lot easier. So I keep on jumping from file to file to understand the big picture. A lot of developers I meet/work with think that the auto-wiring functionality is great because it saves them a lot of time and prevents them from programming in XML (XML was a bad choice, but that is another discussion), but somehow my brain just doesn’t get it.

So my big questions are:

1. am I too stupid?
2. if I’m going to try it longer, am I going to see the light?
3. do I expect too much? So because less characters need to be typed, is it inevitable that some redundant/explicit information is lost
4. is this some hype and eventually people start to realise that it was a nice experiment, but doesn’t provide the value it promises (just like checked exceptions or EJB 2). And in 10 years we can all laugh about it.

2 Types of Dao’s.

From time to time I hear the discussion about how easy it is to replace dao implementations. But this isn’t as easy as it sounds because there are 2 types of dao’s:

1. session-less dao’s; e.g. an JDBC, IBatis or Gigaspaces implementation
2. session-based dao; e.g. a Hibernate/JPA implementation

One big difference between these 2 is that with a session-based dao, it doesn’t matter if an object is changed after a save/update has been called because the object is attached to the session. When the transaction commits it will commit the last state of the object because it can access these objects from the session. With a session-less dao’s, only the state of the object when the save/update was called will be persisted. Personally I prefer to name the save/update methods of a session-based dao ‘attach’ because it makes clear what is really going on. And loaded objects are attached by default, so no need to call save/update on them.

Another big difference between these 2 is that with a session-based dao, you will get the same object instance within a transaction for some entity. So you don’t need to worry about accessing it more than once within a transaction. With session-less dao’s, you can get a new instance within a single transaction, every time you do a load. So you need to take care to update the correct instance.

So even though dao’s look easy to replace because they have an interface, there is a lot of ‘undocumented’ behaviour that makes it a lot trickier than expected.

Greg Young at Devnology

Yesterday I attended a Distributed Domain Driven Design presentation given by Greg Young and organized by devnology at Xebia office. In the past I have been playing with DDD concepts but a lot of questions remained unanswered, even after asking them to Eric Evans when he was at the Xebia office a few years ago.

The information provided by Greg made a lot of sense and it was nice to see that there is a better way for writing domain driven applications instead of the current often awkward approach. Creating different models for different needs and storing the event chain instead of storing the new state of an object really made sense. So it is certainly something I want to experiment with in the future. The well known “Patterns of Enterprise Application Architecture” written by Martin Fowler and “Domain-Driven Design: Tackling Complexity in the Heart of Software” written by Eric Evans, made the first steps in the enterprise domain driven design possible, but I think that Greg could lead us to the next step.

Apart from the technical content (very very important to me) it was also nice to see an enthusiastic and professional speaker. I would certainly place him in the same league as Fowler and Evens and I think that we are going to hear more about Greg in the future (a book perhaps?).

Multiverse: STM for Scala? Part I

Multiverse is a Java based STM implementation and last week I have been playing with integrating Multiverse in Scala. And since I have no practical experience with Scala and the Java language is not going anywhere (any more) it sounded like a nice opportunity to get started with Scala.

Multiverse supports 2 models for creating atomic objects (objects managed by the STM) in the Java language. The simplest approach is adding an @AtomicObject annotation to a Pojo:

@AtomicObject
class Person{
   private int age;
   public int getAge(){return age;}
   public void setAge(int newAge){this.age = newAge;}
}

All changes made to person are atomic and isolated and the cool thing is that the get/setAge methods can participate in larger transactions (this solves the composability problem lock based approaches are suffering from). Using an @AtomicObject annotation is simple because you still can write normal Pojo’s.

The second model is using explicit refs (similar like Clojure refs).

class Person{
   private final Ref age = new Ref();
   public int getAge(){return age.get();}
   public void setAge(int newAge){age.set(newAge;}
}

Multiverse doesn’t care which model is being used. The first has a better syntax and probably is going to perform better when the Pojo has more than 1 field (although it could also lead to false write-conflicts). But the second model is very easy to integrate with Scala and doesn’t rely on instrumentation. In the future the first model is going to be added to Scala as well, but I need to have a better understanding of how Scala is compiled to bytecode.

One of the cool things about Scala is that it is very easy to add new language constructs using closures. If we look at the atomic functionality (so wrapping a transaction around a set of operations) in Java, we need to write a lot of verbose code:

int sum = new AtomicTemplate(){
    public Integer run(Transaction t){
         return person1.getAge()+person2.getAge()
   }
}.execute();

But in Scala you can say:

val sum = atomic{person1.age+person2.age}

Using the following closure:

 def atomic[E](body: => E): E = {
    new AtomicTemplate[E] {
      def execute(t: Transaction) = body
    }.execute()
  }

Isn’t that cool?

In the following blogposts I’ll explain the support for the retry and orelse mechanisms.

Spring AOP: Too complex for day to day use.

I have done quite a lot with Spring and AOP in the past (have been a long time Spring supporter), but every time I need it in a project and look at the reference documentation, I find it more and more confusing and getting too complex for day to day use (especially if not every team member is a Spring expert). It is not because I don’t understand AOP:
- I have used it for quite some time
- I do quite a lot of bytecode manipulation on the Multiverse project.

What is wrong with Spring AOP?
- everything can be combined with everything in all kinds of different ways (so too many permutations)
- there are 10.000 ways to configure it and the configuration can be spread all over the place (including in the source).
- Spring documentation (which is very detailed) is not written in a way to get you up and running fast (so a complete working copy/paste hello world solution).
- bytecode weaving of AspectJ gives problems with load time weaving (administrators, influences other applications on the same jvm) but compile time weaving in combination with Maven also gives problems because the bytecode is modified before the unit tests are executed.

It doesn’t mean that Spring AOP is bad from a technology point of view, but it certainly is bad from a productivity point of view. I love to solve complex problems, but I don’t like to solve problems that are needlessly complex. And Spring AOP imho is a good example of Spring becoming the problem it tried to solve: needless complexity.

Writing an Executor using STM

I’m working on the 0.3 release of Multiverse, a Java based STM implementation, creating replica’s of concurrent datastructures to understand better how it should evolve. The Executor from Java is a different type of datastructure than the standard queues/stacks (already available) etc, because it also needs to manage threads.

That is why I decided to create an example STM based implementation of the Executor interface. It is lazy with thread creation and has a dynamic poolsize.

@AtomicObject
public class StmExecutor implements Executor {

    private final DoubleLinkedList workerThreads;
    private final DoubleLinkedQueue workQueue;
    private StmExecutorState state;
    private int poolSize;

    public StmExecutor(int poolSize, int maxCapacity) {
        if (poolSize < 0) {
            throw new IllegalArgumentException();
        }

        this.workQueue = new DoubleLinkedQueue(maxCapacity);
        this.state = StmExecutorState.started;
        this.workerThreads = new DoubleLinkedList();
        this.poolSize = poolSize;
    }

    public void execute(Runnable task) {
        if (task == null) {
            throw new NullPointerException();
        }

        if (state != StmExecutorState.started) {
            throw new RejectedExecutionException();
        }

        workQueue.push(task);

        if (poolSize > workerThreads.size() && workQueue.size() > 1) {
            createWorkers(1);
        }
    }

    public int getPoolSize() {
        return poolSize;
    }

    public int getActualPoolsize() {
        return workerThreads.size();
    }

    public boolean isShutdown() {
        return state == StmExecutorState.shutdown;
    }

    public void setPoolSize(int newPoolSize) {
        if (state != StmExecutorState.started) {
            throw new IllegalStateException();
        }

        createWorkers(newPoolSize - poolSize);
        this.poolSize = newPoolSize;
    }

    private void createWorkers(int workerCount) {
        if (workerCount > 0) {
            DoubleLinkedList newWorkers = new DoubleLinkedList();
            for (int k = 0; k < workerCount; k++) {
                newWorkers.add(new WorkerThread(this));
            }
            workerThreads.addAll(newWorkers);
            executePostCommit(new StartWorkerThreadsTask(newWorkers));
        }
    }

    public void shutdown() {
        if (workerThreads.isEmpty()) {
            state = StmExecutorState.shutdown;
        } else {
            state = StmExecutorState.shutdownInProgress;
        }
    }

    public void awaitShutdown() {
        if (!isShutdown()) {
            retry();
        }
    }

    //needed to overcome an instrumentation issue
    private StmExecutorState getState() {
        return state;
    }

    //needed to overcome an instrumentation issue
    private void setState(StmExecutorState state) {
        this.state = state;
    }

    //needed to overcome an instrumentation issue
    private DoubleLinkedList getWorkerThreads() {
        return workerThreads;
    }

    //needs to be a static inner class for now.
    static class WorkerThread extends Thread {
        private final StmExecutor executor;

        public WorkerThread(StmExecutor executor) {
            setName("Worker");
            this.executor = executor;
        }

        public void run() {
            boolean again;

            do {
                try {
                    again = runTask();
                } catch (Throwable ex) {
                    ex.printStackTrace();
                    again = true;
                }
            } while (again);
        }

        @AtomicMethod
        private boolean runTask() {
            if (isShuttingDown() || isTooMany()) {
                executor.getWorkerThreads().remove(this);

                if (isShuttingDown() && executor.getWorkerThreads().isEmpty()) {
                    executor.setState(StmExecutorState.shutdown);
                }
                return false;
            }

            Runnable task = executor.workQueue.take();

            task.run();
            return true;
        }

        private boolean isTooMany() {
            return executor.getActualPoolsize() > executor.getPoolSize();
        }

        private boolean isShuttingDown() {
            return executor.getState() == StmExecutorState.shutdownInProgress;
        }
    }

    static class StartWorkerThreadsTask implements Runnable {

        final DoubleLinkedList workerList;

        StartWorkerThreadsTask(DoubleLinkedList workerList) {
            this.workerList = workerList;
        }

        @Override
        public void run() {
            for (WorkerThread thread : workerList) {
                thread.start();
            }
        }
    }
}

enum StmExecutorState {
    started, shutdownInProgress, shutdown
}

As you can see, it is quite easy. And it certainly is a lot better than writing one using old school concurrency. I have written a few implementations for the Prometheus project, so I know how complex they are to write. Especially the closing down part. Another cool thing is that this executor can participate on already running transactions. So you don’t need to be worries about the fact that the work was placed on the queue even though the calling thread fails somehow later, so atomicity you also get for free.

There still are some issues although, if you look at the workqueue, there is lot of contention on the head and tail. A queue has just too strict ordering behaviour. So some kind of pessimistic locking mechanism (perhaps a less extreme version than the shared/exclusive.. where you still are allowed to do a write if others already obtained the lock), could help transactions to find non conflicting executions. This could be combined with something like striping.