Liking Gradle

Last week I have spend a lot of time getting Maven 2 the way I wanted. In most cases Maven is doable, but in case Multiverse, I have some very specific requirements. I need to run the same tests again using different configurations (javaagent vs static instrumentation) and I also need to fabricate a very special jar that integrates all dependent libraries (using jarjar) and does some pre-compilation.

The big problem with Maven is that defining complex builds using the plug-in system is not that great. I have spend a lot of time on configuring the plugins because they were not working like expected or not working at all.

A long time ago I took a look at Gant, but ANT (even if you do it in groovy) still is ANT; meaning a lot of manual labor. So I wanted to give Gradle a try; the power of Groovy combined with a plugin system and predefined build system from Maven. And after 1 day of playing with the scripts, I have got most stuff up and running exactly like the current Maven system (even the artifacts being deployed to the local maven repository!). And I guess that during this week I can make the switch final.

One of the things I really need to get started on, is creating an elaborate set of benchmarks. Using Maven it is fighting the framework, but with the complete freedom you have in Gradle, but still all the facilities provided by plugins, it think it is going to a joy to begin with.

We’ll see what Maven 3 is going to bring, but I’m glad that there is a usable alternative available.

Multiverse: Timed blocking transactions

With traditional concurrency control you can do a timed wait on some resource (a lock for example) to come available. If the resources
doesn’t come available within a certain limit, the operation fails and this needs to be handled in the code. There are a few problems with
this approach:

1. no convenient way to pass the total timeout to each blocking call
2. you need to have several versions of blocking methods (with or without timeout, interruptible or not) and often there is no easy abstraction higher up. So you keep getting almost identical methods, that only differ in the type of blocking method they call
3. no easy way to roll back changes. If you decide to abort an operation because of a timeout, it could be hard to restore the system in the original state (so no atomicity)

With software transactional memory these problems can be solved. I have just added support for timed blocking methods, so you can say something like:

@TransactionalObject
class BankAccount{
    private int balance;

    public int balance(){
      return balance;
    }

    public void setBalance(int balance){
       this.balance = balance;
    }
}

  @TransactionalMethod(timeout = 1, timeoutTimeUnit = TimeUnit.SECONDS)
  public static void tryRemove(Account a, int amount){
      a.setBalance(a.getBalance()-amount);//is rolled back in case of a failure

       if(a.getBalance()<0{
              retry();
       }       
    }

When the tryRemove is called (and no transaction is running), and the balance is not sufficient, the call blocks until a timeout occurs or the balance is increased and the money withdrawn. And if you want the call to be interruptible, you just need to add the InterruptedException:

    @TransactionalMethod(timeout = 1)
    public void tryRemoveInterruptibly(int amount)throws InterruptedException{
       balance = balance - amount;//is rolled back in case of a failure

       if(balance<0{
              retry();
       }       
    }

And the timeunit defaults to SECONDS, so no need to configure that.

In Multiverse you only need to specify multiple blocking versions of a method if that method is exposed to non transactional methods. Transactional methods calling transactional methods, will always lift on the transaction of the outermost transactional call (Multiverse provides a flattened transaction model for nested transactions for now). This also means that it is very easy to create a timed blocking method on datastructures you don’t own.

Multiverse STM: Looking for a Groovy Committer

One of the mission statements of Multiverse is to create a STM that can be used inside any JVM based language. We already have Scala integration available (although it needs some care) and since yesterday we also have someone responsible for integrating Multiverse with JRuby (welcome aboard Sai Venkat).

But it would be cool if we can add Groovy integration in the 0.5 release (in a week) so that it seamlessly integrates with Groovy. So we are looking for someone with Groovy experience, wants to experiment with STM technology and work on an Open Source project.

If you are interested, send a mail to alarmnummer at gmail dot com.

Multiverse and static instrumentation

Multiverse 0.4 relies on a javaagent if you want to seamlessly integrate it with the Java language. The problem is that javaagents suck in production environments, so that is why compiletime instrumentation is a logical next step. The biggest step was to redesign the instrumentation so that the same instrumentation can be used for the javaaagent and the static compiler. This was completed a few weeks ago. But the other step was to create a preinstrumented self contained multiverse jar using Maven so that the same jar can be used as agent, as compiler (jarjar rocks) and as library. After 1 day of struggling and some help of an ex colleague (thanks Silvester) I finally solved the last problems.

So for the 0.5 release of Multiverse, expected next week, it is possible to do static instrumentation (which can be integrated into maven) and to combine this with the Multiverse javaagent (used from your IDE). Already instrumented classes can be mixed with the javaagent and instrumented classes are protected against repeated instrumentation.

Another advantage is that projects that don’t want to rely on instrumentation, but take a more programmatic approach (Multiverse 0.5 will contain a new Api for that as well) still can use the preinstrumented transactional datastructures provided by Multiverse:

• 0.4: TransactionalLinkedList (also implements BlockingQueue and BLockingDeque)
• 0.4: TransactionalArrayList
• 0.5: TransactionalReferenceArray
• 0.5: TransactionalTreeMap (also implements the ConcurrentMap interface).
• 0.5: TransactionalTreeSet
• 0.4: TransactionalThreadPoolExecutor (also implements ExecutorService)

In the 0.6 release more transactional datastructures will be added, and more performance optimizations on the existing collections are to be expected as well.

Multiverse and constructor optimizations

One of other important optimizations I have just implemented and is going to be added to the Multiverse 0.5 release, is that transactions that only create/create+read objects are very cheap (reads already are very cheap) For a construction only one volatile read (for the version) and one volatile write (for the content to be stored). This means that creating transactional objects is going to be very fast and won’t cause scalability problems.

I hope I’ll be able to inline the complete transaction for the constructor when it is very basic (just like the getter/setter optimization).

Multiverse and getter/setter optimization

I’m currently working on removing unnecessary method calls (there is no method as fast as a method that is not called) and I have been able to boost performance on one of the transactional collection classes a few hundred percent and there still is a lot more to gain.

But there is more low hanging fruit waiting to be picked; if there is a getter or setter called on a transactional object, the transaction normally being used to execute these methods, can completely be optimized away be an extremely well performing solution. Getters will have the performance of one threadlocal read and one volatile read. The setters are a little bit more complicated because:

1. they still need to acquire/release a lock (so at least one cas and one volatile write)
2. it also needs one volatile write for storing the content
3. and the biggest influence: increasing a shared clock: an AtomicLong.

I expect that writes will be in the 8+ Million transaction/second (125 ns/write) and around 80 million transactions/second (12.5 ns/read) on a single core. Another thing that also needs to be tested more extensively is how well this is going to scale because that shared counter is going to put a lot of pressure on cpu cache. The atomic writes can be made cheaper by relaxing the increment of the clock, if concurrent threads are increasing that clock; if one of them succeeds, the other is all right with that.

PS:
This functionality is also going to be integrated in the transactional reference support for the Akka project (for the programmatic support the threadlocal read for the transaction can even be bypassed in the get and set).

PPS:
The relaxed clock optimization is explained in the TL2 paper of David Dice and was added to Multiverse quite some time ago (is configurable).