Use the real database when testing

November 24, 2007

While unit/integration testing the repositories (dao’s in yesterdays lingo), try to use the same type of database, instead of some lightweight alternative like hsldb. Unless it also used in the production environment of course.

Using a different type of database could lead to not detecting problems like:

  1. differences in sql interpretation, types, precision and formatting.
  2. missing database constraints. Usually this is functionality I prefer to guarantee and like to have placed in handwritten ddl scripts, instead of it being out of control because it is generated by some or-mapping technology. This approach also invites for creating update scripts from day one, instead of it being an afterthought and causing pain when upgrading to the next release.
  3. differences in concurrency control mechanisms. What leads to a pessimistic lock in one database, could lead to an optimistic locking failure in another.

This means that there are differences in behavior between databases and these should not be ignored. Not using the real database could lead to problems that are only observed in the acceptation or production environment. And the longer it takes to encounter a bug, the more time it is going to cost to solve it.

I know that in continuous build environments using the same type of database can be difficult; not only the environment needs to support different databases, it also needs to support multiple versions. One of the ways to solve this problem is to use a virtualized operating system (using e.g. VMWare) per database version. This should prevent the build environment from changing into a maintenance nightmare.

Advertisements

Lightweight Batch Processing I: Intro

November 12, 2007

If you are lucky, your application is a lot more complex than just the standard request/response webapplication. The complexity in these application can typically be found in the business domain or in the presentation logic. Batch processing systems process large volumes of data, and this is always something that makes me happy to be a software developer, because so much interesting stuff is going on; especially concurrency control and transaction management.

This is the first blog about lightweight batch processing and the goal is to share my knowledge, and hopefully gain new insights by your comments. There are batch frameworks (like the newest Spring module: Spring Batch) but frameworks often introduce a lot more functionality (and complexity) than required and they can’t always be used for a wide range of reasons (sometimes technical, sometimes political). This set of blogs is aimed at these scenario’s. The approach I use is to start from some basic example, to point out the problems that can occur (and the conditions), and eventually to refactor the example.

Lets get started: underneath you can see a standard approach to processing a batch of employees.

EmployeeDao employeeDao;

@Transactional
void processAll(){
    List batch = getBatch();
    for(Employee employee: batch)
        process(employee);
}

void process(Employee employee){
    ...logic
}

As you can see, the code is quite simple. There is no need to integrate the scheduling logic in the processing logic. It is much better to hook up a scheduler (like Quartz for example) from the outside (makes code much easier to test, to maintain and to extend). This example works fine for a small number of employees and if the processing of a single employee doesn’t take too much time. But when the number of employees increases, or the time to process a single item increases, this approach won’t scale well and could lead to all kinds of problems. One of the biggest problems (for now) is that the complete batch is executed under a single transaction. Although this transaction provides the ‘all or nothing’ (atomicity) functionality that normally is desired, the length of the transaction can lead to all kinds of problems:

  1. lock contention (and even lock escalation depending on the database) leading to decreased performance and eventually to a complete serialized access to the database. This can be problematic if the batch process is not the only user of the database.
  2. failing transactions caused by running out of undo space, or the database aborting the transaction because it runs too long.
  3. when the transaction fails, all the items need to be reprocessed, even the ones that didn’t gave a problem. If the batch takes a long time to run, this behavior could be highly undesirable.

In the following example the long running transaction has been replaced by multiple smaller transactions: 1 transaction to retrieve the batch and 1 transaction for each employee that needs to be processed:

EmployeeDao employeeDao;

void processAll(){
    List batch = getBatch();
    for(Employee employee: batch)
        process(employee);
}

@Transactional
List getBatch(){
    return employeeDao.findItemsToProcess();
}

@Transactional
void process(Employee employee){
    ...logic
}

As you maybe have noticed, this example is not without problems either. One of the biggest problems is that the complete list of employees needs to be retrieved first. If the number of employees is very large, or when a single employee consumes a lot of resources (memory for example) this can lead to all kinds of problems (apart from running another long running transaction!). One of the possible solutions is to retrieve only the id’s:

EmployeeDao employeeDao;

void processAll(){
    List batch = getBatch();
    for(Long id: batch)
        process(id);
}

@Transactional
List getBatch(){
    return employeeDao.findItemsToProcess();
}

@Transactional
void process(long id){
    Entity e = dao.load(id);
    ...actual processing
}

A big advantage of retrieving a list of id’s instead of a list of Employees, is that the transactional behavior is well defined. Detaching and reattaching objects to sessions introduces a lot more vagueness (especially if the or mapping tool doesn’t detach the objects entirely). There are different approaches possible: you can keep a cursor open and retrieve an employee only when it is needed, but the problem is that you still have a long running transaction. Another approach is that the only employees are retrieved that can be processed in single run, this has to be repeated until no items can be found.

In the next blogpost I’ll deal with multi-threading and locking.


The use of Maven cripples the mind

November 7, 2007

I have used Maven 1.1 and 2 for the last 1.5 years, but it always am boggled by the utter complexity of the tool. There are a lot of good aspects about Maven (dependency management, standard tasks etc) but these advantages don’t weigh up to all the problems. For example: If I want to create a war and an ear, I need to create multiple projects. How stupid can you be?

With ANT (used it for more than 5 years) I would add an extra target and I’m done. I can do whatever I want. That is why I add all functionality to my script so I get a one button release. So even though some stuff in Maven is much easier to set up, in the end I’m spending much more time configuring maven than it would take in a different tool like ANT. Other problems of maven are badly documented tasks, tasks that break for unclear reasons (ANT is as solid as a rock). My bet (and hope) is that Maven in a few years is seen as something bad like EJB 2 and replaced by modern build environments like GANT, Rake, Raven or Build’r. Build languages that are just internal DSL’s; so you can switch to the full blown language without needing to create plugins and it also prevents programming in XML (XML was a bad choice for ANT from day one).


Versioned id’s

October 22, 2007

If you are using multiple system transactions that span a single business transaction, there is chance of isolation problems: a business transaction could see changes made by other transactions and this could lead to erroneous behavior.

Example:
there is a web page that can be accessed by management, that contains a list of all employees. Each row contains the id, the name, the date of the last bonus and a button to give the bonus. When the button is pressed, the id of the employee (eventually) is send to the following service method:

@Transactional
void giveBonus(long id){
	//load throws a special exception if the employee is not found
	//Spring: ObjectRetrievalFailureException if employee isn't found
	Employee e = employeeDao.load(id);
	e.giveBonus();
}

The problem with this approach is that 2 managers could access the page at roughly the same moment and both see that some employee didn’t get a bonus for some time and decide to give the bonus. The consequence is that the employee has received the bonus twice. Although this example is a little bit hypothetical, in some cases this behavior could lead to serious problems.

Optimistic locking is one of the best known techniques to deal with these issues: it doesn’t lock (unlike pessimistic locking) but hopes (is optimistic) about the fact that no other transaction is going to conflict. If a different transaction does make a change, the version of the data is changed, and this is something that can be checked. As soon as a difference between versions is found, an optimistic locking failure exception can be thrown.

Just carrying around the version and id, doesn’t feel that good. It makes method signatures less pretty and you have to carry it around. Especially when you need to deal with multiple entities, the solution gets more ugly. The solution is simple: integrate id and version into a single structure: VersionedId. VersionedId’s contain the id of an entity and also contain the version of that entity; so a VersionedId can uniquely identify an entity in time and could look like this:

class VersionedId{
	long version, id;
}

the previous example could be rewritten by using a VersionedId:

@Transactional
void giveBonus(VersionedId id){
	//load throws a special exception if the employee is not found or version doesn't match
	//Spring: ObjectRetrievalFailureException if employee not found
	//Spring: OptimisticLockingFailureException if version doesn't match
	Employee e = employeeDao.load(id);
	e.giveBonus();
}

As you can see there is not much difference. But you have to make sure that the load method of the EmployeeDao is able to handle versioned id’s and starts throwing some kind of optimistic locking failure if the Employee with the given id and version can’t be found. And instead of just placing the id of the employee on the html-page, the version needs to be placed as well.

Using detached objects (and reattaching them) is also a solution. But there are some issues:

  1. The detached objects need to be stored and the most likely place is the HttpSession. This could lead to replication issues if you are in a clustered environment
  2. Vague transactional behavior
  3. I have seen some weird issues with detaching and reattaching entities with Hibernate (especially not detaching completely)

Service layer woes

July 18, 2007

I have posted an entry on the blog of my employer:

http://blog.xebia.com/2007/07/18/service-layer-woes/


RIA and scalability

June 21, 2007

At the moment I’m writing a blog for the company website about the new process functionality in Prometheus, and something occurred to me.

With the introduction of multi-cores, parallelism is going to be more important. One of the easiest types of applications to scale is one with small and independent tasks, for example a standard web-application. Requests can be executed concurrent, so increasing the number of cores, is going to increase the throughput (not the latency). When latency also needs to be improved things get much more complicated because you need to introduce parallelism on a much lower level. Web pages are getting more and more complex, so building a single screen is going to take more and more time if no extra parallelism is introduced.

The cool thing is that with the introduction of Rich Internet Applications (RIA), the type of request is going to change. Instead of having one big ‘build me the complete page’ request, you will have a lot of smaller and independent requests ‘please get me the newest value for this page component’. This means that RIA applications could scale a lot easier. I have no numbers to back it up, so don’t shoot me if I’m wrong.


Spring and visibility problems

March 1, 2007

I posted a blogentry on Spring and visibility problems on the company website. If you want to read the post, you can find it here:

http://blog.xebia.com/2007/03/01/spring-and-visibility-problems