This is the answer to the riddle posed in my earlier puzzler posting. You should probably look at that question before looking at this answer.
I suppose I should call it something other than a puzzler, to avoid getting hit by Josh and Neal's angry team of vicious, snarling lawyers...
This program certainly looks straightforward. It just looks as if two threads are writing to two variables. In fact, you probably expected me to say something "who-cares" oriented about compiler optimizations at this point. Well, they are volatile variables, so you can worry a lot less about potential reorderings. In fact, this one has absolutely nothing to do with program transformations, and, if you ran the program on my laptop, you found that it hangs!
It turns out that this is the result of one of those vicious little static initialization quirks that provide so many hours of headaches. What happens is something very like the following. The first thread encounters A; this class has not been initialized, so that thread tries to initialize it. When you try to initialize a class, you acquire a lock on that class, so that no one else can initialize it at the same time. Are you starting to see where this could lead to problems?
At the same time as this, the second thread encounters B, and so acquires the lock on the B class. It then runs the static initializer for B, which encounters A. "Wait!" it says -- "A hasn't been initialized! Better acquire that initialization lock..." It tries to acquire the lock, but the first thread already has it, so it waits for the first thread to finish.
Meanwhile, the same process goes on in the first thread. It runs the static initializer for A, which encounters B. "Wait!" it says -- "B hasn't been initialized! Better acquire that initialization lock..." It tries to acquire the lock, but the second thread already has it, so it waits for the second thread to finish.
Result: Both threads wait forever. Deadlock!
This whole process is scheduling / hardware / OS / JVM dependent, of course. If the first thread runs to completion without letting the second thread start, then it will quite happily initialize both A and B without the other thread acquiring any locks. This will avoid deadlock nicely. This seems to happen on Linux, but not OS X.
How do you avoid this? Well, that's a little tricky. In this case, you would probably rewrite the code so that it doesn't perform the initialization in two separate threads. That's not always a general-purpose solution, though. Your best bet, in general, is to avoid having circularities like this in your static initializers. As with constructors, it is important to keep your static initializers as simple as possible.
Keeping it simple might mean not doing anything that might trigger subsequent static initialization. That's a good first option, if you can manage it, but it is not always possible.
The second option is to make sure that you have an order over your classes. For example, if you have three classes, A, B and C, you could structure them so that C can refer to A and B in its static initializer, B can only refer to A, and A can only refer to itself. This will prevent deadlocks by enforcing a strict order over when the implicit locks can be acquired.
The final option -- if you know this will be a problem -- is to make sure that the initialization can only happen in a single thread. This may mean having to force it to occur earlier, by referencing the class earlier than it would otherwise have been referenced.
I feel like there should be a moral. The moral is that static initialization has all sorts of hidden gotchas, and you should look out for it.