Monday, July 21, 2008

Immutability in Java, Part 3: Deserialization and Reflection

This time, I'll talk about deserialization, immutability and final fields. I'll try to be a little shorter.

I spent the last two posts on immutability (here and here) talking about what it takes to make an object's fields immutable. This advice mostly boiled down to "it has to be final and set before the constructor ends".

When you have deserialization, though, the fields can't be set before the constructor ends. An instance object is constructed using the default constructor for the object, and the fields have to be filled in later. We gave deserialization special semantics, so that when an object was constructed this way, it would behave as if it had been constructed by an ordinary constructor. No problems there.

The problems come when and if you need to write custom deserialization (using, for example, the readObject method of ObjectInputStream). You can do this using reflection:
Class cl = // get the class instance
Field f = cl.getDeclaredField("myFinalField");
f.setAccessible(true);
f.set(enclosingObject, newValue);
Note that you need setAccessible to change final fields using reflection.

ETA: Jared Levy points out that you should probably use Field.getDeclaredField if the field isn't public. The original code used Field.getField(). Thanks to him for pointing that out.

In the last post, I mentioned that something very like a "freeze" action happens at the end of the constructor; as long as your final fields (and everything reachable from them) are fully constructed by the end of that, your data are immutable. There is a similar "freeze" action after you modify final fields using reflection; as long as your final fields (and everything reachable from them) are fully constructed as of when you change them via reflection, your data are immutable. This means that if you have data you want your final field to point to, and you want other threads to see, they all have to be constructed by the time you set the final field. See the previous postings for examples of this.

The other restrictions on immutability apply, too. You can't let another thread see the object before the final field is set via reflection, for example. See the previous blog postings for more on this, too.

This is slightly different for static final fields, since they are often completely removed at javac compile time. You can't change static final fields after class initialization. Well, almost. You can change System.out, System.in and System.err using System.setOut(), System.setIn() and System.setErr(). We felt very dirty after writing these rules.

By the way, the general treatment of static final fields was a big mistake in the original spec. The compiler (javac) is forced to inline static final compile time constants everywhere it can. They can even be inlined into other classes. This means that you can compile a class A with a static final int x set to 1, compile another class B that prints A.x, recompile the first class with x set to 2, and then run the program and have class B print out "1" instead of "2". Weird and bogus. However, we are stuck with it.

29 comments:

Danny said...
This comment has been removed by the author.
Danny said...

This was a big mistake in the original spec.

Was this a big mistake? Wouldn't it make more sense to never allow the changing of final fields (even through reflection?) Calling "setAccessible" seams like a workaround.

Perhaps the deserialization in java could have been designed in such a way such that a constructor is still called. In this case, you could still set final fields as you normally would (in the constructor.)

Jeremy Manson said...


Was this a big mistake? Wouldn't it make more sense to never allow the changing of final fields (even through reflection?) Calling "setAccessible" seams like a workaround.

Perhaps the deserialization in java could have been designed in such a way such that a constructor is still called. In this case, you could still set final fields as you normally would (in the constructor.)


I was referring specifically to the fact that static final fields are treated specially. I've updated the entry to make a little more sense of what I was saying.

It is pretty ridiculous to have to use reflection and setAccessible. The Java serialization and construction models are fundamentally broken; I think anyone who has ever written a FooBuilder class can attest to that.

stinkyminky said...

Jeremy

Thank you your blog. Is it possible for you to talk about 'Lock' in concurrent library. For example, 'Lock' uses 'compareAndSet' to achieve the mutual exclusivity. When do you recommend using 'Lock' instead of 'synchronize'. Also how does Lock achieve 'visibility'? How does 'Lock' work its magic regard to memory barrier?

Thanks in advance

Tom said...

Thanks! It will be very helpful in my project.
I was passing this problam, I need to set some final variables on the readObject method, and need to be a different value (the value i read), not the constructor's value.

Tom said...
This comment has been removed by the author.
Tom said...

But when you makes it accessible, it will be accessible only by reflexion, right?

Jeremy Manson said...

But when you makes it accessible, it will be accessible only by reflexion, right?

Correct.

Tom said...

This is also helpful:
http://forums.sun.com/thread.jspa?messageID=10370899#10370899

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

The Java serialization and construction models are fundamentally broken; I think anyone who has ever written a FooBuilder class can attest to that.

Jeremy, could you elaborate more on why the construction model is broken? How would you fix it if you had a chance to design it from scratch?

Thanks!

Jeremy Manson said...


Jeremy, could you elaborate more on why the construction model is broken? How would you fix it if you had a chance to design it from scratch?


Done and done.

Anonymous said...

To get around inlined static final constants that are primitives, I use wrapper classes that are then unboxed.

Your example, A.x=1, B{print(A.x)}, would have A{static final int x}. If you use A{s-f-Integer x} then B will always print the most recent value of A.x even without a recompile (of B; A.x won't be changed without a recompile).

String constant inlining is obviously not solved this way.

Jeremy Manson said...

To get around inlined static final constants that are primitives, I use wrapper classes that are then unboxed.

Sure, but that's completely horrible. I wouldn't do that on general principles.

hlovatt said...

My pet project, a pattern enforcing compiler (PEC):

http://pec.dev.java.net/nonav/compile/index.html

Is an extensible compiler to which can enforce contracts like immutability. Immutability is one of the supplied pattens with the compiler.

It is nice to have the compiler do extra checks.

Jeremy Manson said...

@stinkyminky
Thank you your blog. Is it possible for you to talk about 'Lock' in concurrent library. For example, 'Lock' uses 'compareAndSet' to achieve the mutual exclusivity. When do you recommend using 'Lock' instead of 'synchronize'. Also how does Lock achieve 'visibility'? How does 'Lock' work its magic regard to memory barrier?

For all intents and purposes, using ReentrantLock is the same as using synchronized. It was slightly faster in 1.5, but the difference now is negligible. Any difference nowadays is mostly stylistic.

Robin Bygrave said...

For what its worth...

... a reference to the:

"freeze" that occurs post construction when setting final fields via reflection

can be found here...

http://www.cs.umd.edu/~pugh/java/memoryModel/jsr133.pdf

9.1.1 Post-Construction Modification of Final Fields

...
Freezes of a final field occur both at the end of the constructor in which the final field is set, and immediately after each modification of a final field via reflection other special mechanism.
...

Cheers, Rob.

Unknown said...

I agree that the standard serialization/deserialization mechanism has some issues.

However, I very rarely face this 'final field' issue as only very few classes need to use these custom methods! Most use the default mechanism and their final fields are just fine. Otherwise, what is the benefit of using this mechanism?

Further more, this is more of an implementation issue.

Have you used any 3rd-party serialisation tools? I have used Skaringa in the past but it has similar issues. However, you get XML with no effort.

Now, I use XStream and I'm very pleased!
XStream uses a "Pure Java" mechanism, but it also provides an "Enhanced Mode" one for the "kitchen sink" set! :-) http://xstream.codehaus.org/faq.html

Jeremy Manson said...

@Tasos - everything you say is correct. But people do roll their own serialization from time to time. The default is often inappropriate.

Unknown said...

But people do roll their own serialization from time to time. The default is often inappropriate.My point was that using custom serialization is not an all-or-nothing decision.
In my experience the classes that require custom serialization are a small minority.

Unknown said...

Perhaps the deserialization in java could have been designed in such a way such that a constructor is still called. In this case, you could still set final fields as you normally would (in the constructor.)How do you see coding such dual-purpose constructors? Constructors initialize new instances, while deserialization re-instantiate old ones with existing data.

For me the whole point of the default serialization mechanism is that it is transparent. And the default mechanism deals with final fields just fine...

Unknown said...

The Java serialization and construction models are fundamentally broken; I think anyone who has ever written a FooBuilder class can attest to that.Jeremy, can you please give an example of this?
I don't see why someone would want to serialize a FooBuilder class. My understanding is that such classes are mainly used as construction utilities; not something I would normally serialize...
Do you typically serialize your whole object-graph?
I must be misunderstanding your comment.

Jeremy Manson said...

Not the key part of that sentence is "and construction". A reasonable language wouldn't have the need for a builder class at all.

Jeremy Manson said...

Er... "note", not "not".

Anonymous said...

String constant inlining is obviously not solved this way.

You could use intern in this case.

Jeremy Manson said...

Of course, calls to intern() are kind of expensive by comparison.

Anonymous said...

"All literal strings and string-valued constant expressions are interned" - from JDK
Could you explain why explicit call of intern() are expensive for static constants?

Jeremy Manson said...

Could you explain why explicit call of intern() are expensive for static constants?

Because they aren't optimized away, so they cost just as much as any other call to intern() (this is from recollection).

Anonymous said...

Nice post. I new deserialization could be made thread-safe, but this article helped me understand the specific technique. And thanks to the follow-up posts, I was able to find the JSR (133) pertinent spec section that backs up the technique.