Synchronized code in distributed systems

I encountered some code recently where a GUI client was listening for Object updates from a multi-threaded server, and it was using the version number on the Object to determine whether the incoming Object was the latest version. The server incremented the version of the Object by inserting a new row into a table in the database and using the primary key as the version number.

Now consider the case multi-threaded case where two threads, T1 and T2, are making updates to the Object.

  1. Both threads start a transaction.
  2. T1 increments the version to v1.
  3. T2 increments the version to v2.
  4. T2 commits and sends the notification to the client.
  5. T1 commits and sends its notification to the client.

The GUI ignores the update from T1 because it already has v2 of the Object from T2. The problem was that the changes from T1 were not included in the update from T2 because T1 had not committed its database changes.

The underlying problem here is that there is no synchronization around incrementing the version. There are several ways of solving this problem:

Option 1: use Java’s synchronized keyword
In this solution the version number would be incremented in a synchronized block, using the Object instance as the lock on which to synchronize.

synchronized(myObject) {
  myObject.setVersion( myObject.getVersion() + 1);
}

This method works in a multi-threaded environment but does not work when the application is clustered across multiple servers.

Option 2: use the database as the synchronization point.
In this solution the version number is moved from its own table to the master record, and the field is updated when we need to increment the version. However, there is a pitfall that needs to be avoided for this to work as expected.

The incorrect way to do this is to increment the value in Java and save it.

  String sql = "update my_obj set version = ? where id = ?";
  ...
  stmt.setInt( idx++, o.getVersion()+1);
  stmt.setInt( idx++, o.getId());
  stmt.execute();

The problem here is that different threads could write the same version number when they should have incremented it. Consider this scenario:

  1. Both threads start a transaction.
  2. T1 reads the Object (v1).
  3. T2 reads the Object (v1).
  4. T1 increments the version to v2.
  5. T2 increments the version to v2.

T2 should have incremented the version to v3 but T1 hadn’t committed its update when T2 read the record from the database.

The correct way to do this is to update the record in place because this forces all threads across all clustered servers to get a write lock on a single record in the database, and the value that they are incrementing is always the latest. Each thread will block until the previous one has committed its change.

  String sql = "update my_obj set version = version + 1 where id = ?";
  ...
  stmt.setInt( idx++, o.getId());
  stmt.execute();

Learn To Type!

If there was one piece of advice I could give to someone considering a career in software development it would be: Learn to f$%@cking type! Knowing how to touch type is the single most valuable skill that a developer can possess.

Typing allows you to concentrate on what you’re coding rather than how to get the characters on the screen. In other words, you’re able to think about the code rather than thinking about where to find the letters on the keyboard. Jeff Atwood and Steve Yegge both make the same point.

I have had the privilege of working with some great developers over the years and they all share a few common traits.

  • They can all type.
  • They all read books, both technical and recreational.

Doing these things will not necessarily make you a great programmer but it will establish a pattern of learning that will help you get a little bit better each year.

I find it astounding that there are people in IT who haven’t bothered to learn how to type. It is just plain lazy. Learning to type only takes about an hour per day for a few weeks, along with a little commitment to practice it every time you’re at a keyboard.

As developers, the keyboard is the tool of our trade. Would you hire a carpenter that couldn’t use a hammer? Or as Jeff Atwood put it, “When was the last time you saw a hunt-and-peck pianist?”.

Once you can type efficiently a whole world of keyboard shortcuts opens up and the mouse is relegated to a cumbersome annoyance that slows you down. Thoughts seemingly turn themselves into code.

You get to feel like Neo when he became one with the Matrix.

Reflection HashCodeBuilder Performance

I recently embarked upon a performance investigation to figure out why a particular operation was slow. My theory was that it was the serialization/deserialization cost with a set of large objects; my colleague’s theory was the iteration over the set. We were both wrong.

The investigation became intriguing when we started measuring the elapsed time at various points in the application and noticed that the bulk of the time was being spent in a seemingly innocuous call to construct a HashSet from an existing Collection.

Knowing how Java works under the hood is invaluable in diagnosing these issues. Java will make heavy use of hashCode() and equals() when constructing a Set because it needs to determine uniqueness. When we checked the hashCode() and equals() implementations in the class of the set we found that it was using the Apache Commons utility classes.

@Override
public int hashCode() {
  return HashCodeBuilder.reflectionHashCode(this, false);
}

@Override
public boolean equals(Object o){
  return EqualsBuilder.reflectionEquals(this, o, false);
}

This is a pretty lazy way to implement these methods. In an Agile context, it probably falls into the “do the simplest thing that could possibly work” category, but the “proper” implementation would have only taken a few extra minutes and would have saved us a lot of investigation time, and would have kept our users happy.

I baselined this implementation against an explicit implementation and found that the reflection version was 5 times slower on a simple object. In our particular case it was several orders of magnitude slower because of the complexity of our Object graph.

But it gets even better. The object that was using the reflection hashCode builder had a unique identifier that we could use, so implementing an explicit hashCode builder took our run time down to almost zero.

@Override
public int hashCode() {
  HashCodeBuilder b = new HashCodeBuilder();
  b.append(getId());
  return b.hashCode();
}

@Override
public boolean equals(Object o){
  if( o == null){
    return false;
  }
  if( ! (o instanceof MyObject)){
    return false;
  }
  MyObject that = (MyObject)o;
  EqualsBuilder b = new EqualsBuilder();
  b.append(getId(), that.getId());
  return b.isEquals();
}

The sample code is available on GitHub

A new assignment

My ex-project and I had been together for ten months. We had our fair share ups and downs, particularly in the early days, but over the last few months we had settled into a nice routine. It wasn’t particularly exciting any more but at least it was predictable.

And then yesterday I was asked to lead a troubled project. It’s only six months old and already has quite a few warts, and it limped across the line into UAT last week. My first instinct was to cower in the corner in the fetal position but I resisted that temptation and quietly accepted the role.

The challenge with assignments like this is to avoid making drastic changes straight away. Most products are only a few tweaks away from being great so the approach I’m going to take is to make a few small adjustments and re-evaluate: build, measure, learn. Repeat. The key will be to get a deep understanding of the codebase and the business problem its trying to solve before making too many changes.

I have heard the “R” word being bandied around but I refuse to even consider a re-write. If there are significant changes required we can refactor towards the new goal. Joel Spolsky puts a re-write at number one in his list of Things you should never do, and I agree one hundred percent.

It’s going to be an interesting couple of months, but hopefully I can get the project back on track and start shipping better versions every two weeks.

How to insulate yourself from static methods

Sometimes your code needs to call a static method in a 3rd party library and unit testing  suddenly becomes difficult, particularly when that static method requires context in order to work.

Enter the insulating class (also known as a Facade).

  1. Create a new interface that declares a method with the same signature as the static method you want to delegate to
  2. Create a new Facade class that implements the interface
  3. Replace the calls to the static method with calls to your new Facade class

Let’s assume that the method you need to call looks like this

public class ThirdPartyValueComputer {
  public static int computeValue(SomeObject param, SomeOtherObject param){
    ....
  }
}

If you put this call into another class with an interface then you will be able to use mock objects to test your code.

public interface IValueComputer {
  int computeValue(SomeObject p1, SomeOtherObject p2);
}

public class ValueComputerFacade implements IValueComputer {
  @Override
  public int computeValue(SomeObject p1, SomeOtherObject p2){
    return ThirdPartyValueComputer.computeValue(SomeObject p1, SomeOtherObject p2);
  }
}

public interface ISomeService {
  int calculate(SomeObject p1, SomeOtherObject p2);
}

public class SomeService implements ISomeService {
  private IValueComputer valueComputer;
  public void setValueComputer(IValueComputer c){
    this.valueComputer = c;
  }

  @Override
  public int calculate(SomeObject p1, SomeOtherObject p2) {
    int computedValue = valueComputer.computeValue(p1, p2);
    // Now perform the logic you need to test
    int result = ...;
    return result;
  }
}

The service is configured in Spring as follows

  
  
  
  

Now that you’ve created and configured your facade, you can write tests for the service class as follows

Mockery context = new Mockery();

@Test
public void testCalculate() throws Exception {
  final SomeObject p1 = new SomeObject();
  final SomeOtherObject p2 = new SomeOtherObject();

  IValueComputer valueComputer = context.mock(IValueComputer.class);
  SomeService service = new SomeService();
  service.setValueComputer(valueComputer);

  context.checking(new Expectations() {{
    oneOf(valueComputer).computeValue(p1, p2))
    will(Expectations.returnValue(1));
  }});

  int result = service.calculate(p1, p2);
  assertEquals(1, result);
}