1 2 Previous Next 28 Replies Latest reply: Jul 2, 2009 12:30 PM by Brian Stansberry RSS

ConcurrentModificationException during session serialization

Stanimir Stamenkov Newbie

I've initially posted this in the RichFaces users forum:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=136368

I haven't got reply there so far and I'm unsure where is the exact problem so I've decided to post it here, too.

The problem as it seems to be is the web session gets modified while it is been serialized. Specifically the serialization of an org.ajax4jsf.util.LRUMap instance holding some JSF(?) state is failing with ConcurrentModificationException, so I've first thought it could be a RichFaces library problem.

This is my development configuration (1 node cluster):

JBossAS 4.2.2.GA
Seam 2.0.2.SP1
RichFaces 3.2.0.SR1
Sun's JDK 1.5.0_15
Windows XP Pro SP2

The same problem appers with 2 nodes configuration on FreeBSD 7.0 with Sun's JDK 1.5.0_11.

Tell me what info I could supply more to clarify the issue. Here's the full stack trace of the exception:

15:27:13,187 ERROR [JBossCacheService] externalizeSession(): exception occurred externalizing session SessionBasedClusteredSession[id: F3m44XY0NwX7HVPjsweIDw** lastAccessedTime: 1211977633015 version: 39 lastOutdated: 0]
java.util.ConcurrentModificationException
 at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:365)
 at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:384)
 at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:383)
 at java.util.HashMap.writeObject(HashMap.java:1037)
 at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
 at java.util.HashMap.writeObject(HashMap.java:1039)
 at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
 at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1380)
 at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
 at org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.writeExternal(SessionBasedClusteredSession.java:175)
 at org.jboss.web.tomcat.service.session.JBossCacheService.externalizeSession(JBossCacheService.java:1027)
 at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:316)
 at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121)
 at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1097)
 at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:652)
 at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
 at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
 at java.lang.Thread.run(Thread.java:595)


  • 1. Re: ConcurrentModificationException during session serializa
    Brian Stansberry Master

    Sounds like your analysis on the RichFaces forum was on the right track.

    The session management layer does not prevent concurrent access to the session. The session's internal data structures can deal with concurrency (e.g. the attribute map is a ConcurrentHashMap) but if the objects you'll store in it are going to be accessed concurrently, they'll need to be thread safe or have access to them synchronized.

  • 2. Re: ConcurrentModificationException during session serializa
    Stanimir Stamenkov Newbie

    Thanks for the feedback. I've prepared a minimal test case demonstrating the problem and I'll submit a report in the JBoss-Jira against the RichFaces project.

  • 3. Re: ConcurrentModificationException during session serializa
    Brian Stansberry Master

    It could be argued that we shouldn't be allowing concurrent access to the session. But that's not really a clustering issue; i.e. the problem can occurs in a non-clustered webapp. The Tomcat classes on which the clustered session management is based don't enforce such a requirement, so our clustered impls also don't .

  • 4. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

     

    The same problem appears with 2 nodes configuration on FreeBSD 7.0 with Sun's JDK 1.5.0_11.


    I am trying to help with JBAS-5615. Could you clarify if "sticky http session" handling is enabled or disabled in this two node cluster?


  • 5. Re: ConcurrentModificationException during session serializa
    Nick Belaevski Master

    Hi Scott,

    The issue was reproducing even on single-cluster node (clean JBoss installation launched with "all" configuration). I can see this exception thrown on JBoss 5.0.1.GA (you should replace attached jsf-facelets.jar with the newest version to run test .EAR file: http://repository.jboss.org/maven2/com/sun/facelets/jsf-facelets/1.1.15.B1/jsf-facelets-1.1.15.B1.jar).

  • 6. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    I agree, that running with only one node in the cluster shouldn't require "sticky sessions=true".

    When we do run with more than one cluster node, we need to enable "sticky sessions" to avoid multiple http session attribute changes (to the same http session attribute) coming from different nodes (at the same time).

    Thank you for explaining this part of the problem. Of course, it goes deeper than not configuring the http load-balancer (please always enable "sticky http sessions" :-)

    The problem as it seems to be is the web session gets modified while it is been serialized. Specifically the serialization of an org.ajax4jsf.util.LRUMap instance holding some JSF(?) state is failing with ConcurrentModificationException, so I've first thought it could be a RichFaces library problem.


    From your description of the problem, it sounds like the following events have occurred:

    An HTTP request is received for user Sharon, during the processing of this request (R1), an HTTP session attribute change is made. As part of the HTTP processing after request processing, we attempt to replicate the attribute change. Meanwhile, another HTTP request (R2) for user Sharon was also processed that generated an HTTP session attribute change. As part of R2 processing, the same attribute change was transmitted (from a different HTTP client service thread) and applied to the collection (while the R1 client request handling thread is iterating through the collection).

    The java.util.ConcurrentModificationException occurs when iterating through the collection, while another thread modifies the collection.

    Before we discuss possible solutions, lets see if we agree on the above. If I am not stating things correctly above, please explain your understanding of what is happening.

  • 7. Re: ConcurrentModificationException during session serializa
    Nick Belaevski Master

    Yes, that's right. The problem is that our (RichFaces) code synchronizes on session facade, but replication code synchronizes on internal instance of session - exception happens.

  • 8. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    What could the solution be? Should SessionBasedClusteredSession.writeExternal synchronize on the same session facade as RichFaces?

  • 9. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    What I mean, is should SessionBasedClusteredSession.writeExternal + SessionBasedClusteredSession.readExternal instead synchronize on getSession(), which should be the same StandardSessionFacade as RichFaces is synchronizing on.

    I haven't recreated the bug yet. I could either recreate it or you could try the following SessionBasedClusteredSession change (two line change to synchronize on common session facade) :

    public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException
     {
     synchronized (getSession()) // change 1 of 2
     {
     // Let superclass read in everything but the attribute map
     super.readExternal(in);
    
     attributes = (Map) in.readObject();
     }
     }
    
     public void writeExternal(ObjectOutput out) throws IOException
     {
     synchronized (getSession) // change 2 of 2
     {
     // Let superclass write out everything but the attribute map
     super.writeExternal(out);
    
     // Don't replicate any excluded attributes
     Map excluded = removeExcludedAttributes(attributes);
    
     out.writeObject(attributes);
    
     // Restore any excluded attributes
     if (excluded != null)
     attributes.putAll(excluded);
     }
    
     }
    



  • 10. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    There are other code changes that go with the change but the above should be enough for you to test.

    The super classes also need to change to synchronize on the getSession() instead of "this".

  • 11. Re: ConcurrentModificationException during session serializa
    Nick Belaevski Master

     

    "smarlow@redhat.com" wrote:
    What I mean, is should SessionBasedClusteredSession.writeExternal + SessionBasedClusteredSession.readExternal instead synchronize on getSession(), which should be the same StandardSessionFacade as RichFaces is synchronizing on.

    I haven't recreated the bug yet. I could either recreate it or you could try the following SessionBasedClusteredSession change (two line change to synchronize on common session facade) :

    public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException
     {
     synchronized (getSession()) // change 1 of 2
     {
     // Let superclass read in everything but the attribute map
     super.readExternal(in);
    
     attributes = (Map) in.readObject();
     }
     }
    
     public void writeExternal(ObjectOutput out) throws IOException
     {
     synchronized (getSession) // change 2 of 2
     {
     // Let superclass write out everything but the attribute map
     super.writeExternal(out);
    
     // Don't replicate any excluded attributes
     Map excluded = removeExcludedAttributes(attributes);
    
     out.writeObject(attributes);
    
     // Restore any excluded attributes
     if (excluded != null)
     attributes.putAll(excluded);
     }
    
     }
    


    Yes, if synchronization will occur on StandardSessionFacade visible to the application/framework developers, than it should work ok. I haven't built JBoss AS before, so not sure whether I'll be able to make these changes and test; in any case, I'll try that and report.

  • 12. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    It is easy to build (assuming 4.2.x):

    cd jboss_4.2.x_tree
    cd build
    ./build.sh
    cd output
    cd jboss-4.2.4.GA/bin
    ./run.sh -c all

    I also attached the potential 4.2 code patch (need to get it reviewed still) to JBAS-5615.

  • 13. Re: ConcurrentModificationException during session serializa
    Scott Marlow Master

    If you have difficulties applying/building the patch, I'll try to recreate/test instead. Let me know either way.

  • 14. Re: ConcurrentModificationException during session serializa
    Brian Stansberry Master

    Thanks, nbelaevski for pointing out how code outside the container can't synchronize on the session, only the facade. And thanks Scott for running with this.

    The concern I have is StandardSession.facade is not thread safe, and what's returned from getSession() needs to be if we are going to use it as a mutex. There's no ref to facade outside StandardSession.getSession(), so ClusteredSession can override the getSession() method and replace the field with a volatile field and synchronize the construction if null.

1 2 Previous Next