1 2 Previous Next 28 Replies Latest reply on Jul 2, 2009 12:30 PM by brian.stansberry

    ConcurrentModificationException during session serialization

    stani0

      I've initially posted this in the RichFaces users forum:

      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=136368

      I haven't got reply there so far and I'm unsure where is the exact problem so I've decided to post it here, too.

      The problem as it seems to be is the web session gets modified while it is been serialized. Specifically the serialization of an org.ajax4jsf.util.LRUMap instance holding some JSF(?) state is failing with ConcurrentModificationException, so I've first thought it could be a RichFaces library problem.

      This is my development configuration (1 node cluster):

      JBossAS 4.2.2.GA
      Seam 2.0.2.SP1
      RichFaces 3.2.0.SR1
      Sun's JDK 1.5.0_15
      Windows XP Pro SP2

      The same problem appers with 2 nodes configuration on FreeBSD 7.0 with Sun's JDK 1.5.0_11.

      Tell me what info I could supply more to clarify the issue. Here's the full stack trace of the exception:

      15:27:13,187 ERROR [JBossCacheService] externalizeSession(): exception occurred externalizing session SessionBasedClusteredSession[id: F3m44XY0NwX7HVPjsweIDw** lastAccessedTime: 1211977633015 version: 39 lastOutdated: 0]
      java.util.ConcurrentModificationException
       at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:365)
       at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:384)
       at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:383)
       at java.util.HashMap.writeObject(HashMap.java:1037)
       at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
       at java.util.HashMap.writeObject(HashMap.java:1039)
       at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
       at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1380)
       at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
       at org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.writeExternal(SessionBasedClusteredSession.java:175)
       at org.jboss.web.tomcat.service.session.JBossCacheService.externalizeSession(JBossCacheService.java:1027)
       at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:316)
       at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:121)
       at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1097)
       at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:652)
       at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
       at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:98)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Thread.java:595)


        • 1. Re: ConcurrentModificationException during session serializa
          brian.stansberry

          Sounds like your analysis on the RichFaces forum was on the right track.

          The session management layer does not prevent concurrent access to the session. The session's internal data structures can deal with concurrency (e.g. the attribute map is a ConcurrentHashMap) but if the objects you'll store in it are going to be accessed concurrently, they'll need to be thread safe or have access to them synchronized.

          • 2. Re: ConcurrentModificationException during session serializa
            stani0

            Thanks for the feedback. I've prepared a minimal test case demonstrating the problem and I'll submit a report in the JBoss-Jira against the RichFaces project.

            • 3. Re: ConcurrentModificationException during session serializa
              brian.stansberry

              It could be argued that we shouldn't be allowing concurrent access to the session. But that's not really a clustering issue; i.e. the problem can occurs in a non-clustered webapp. The Tomcat classes on which the clustered session management is based don't enforce such a requirement, so our clustered impls also don't .

              • 4. Re: ConcurrentModificationException during session serializa
                smarlow

                 

                The same problem appears with 2 nodes configuration on FreeBSD 7.0 with Sun's JDK 1.5.0_11.


                I am trying to help with JBAS-5615. Could you clarify if "sticky http session" handling is enabled or disabled in this two node cluster?


                • 5. Re: ConcurrentModificationException during session serializa
                  nbelaevski

                  Hi Scott,

                  The issue was reproducing even on single-cluster node (clean JBoss installation launched with "all" configuration). I can see this exception thrown on JBoss 5.0.1.GA (you should replace attached jsf-facelets.jar with the newest version to run test .EAR file: http://repository.jboss.org/maven2/com/sun/facelets/jsf-facelets/1.1.15.B1/jsf-facelets-1.1.15.B1.jar).

                  • 6. Re: ConcurrentModificationException during session serializa
                    smarlow

                    I agree, that running with only one node in the cluster shouldn't require "sticky sessions=true".

                    When we do run with more than one cluster node, we need to enable "sticky sessions" to avoid multiple http session attribute changes (to the same http session attribute) coming from different nodes (at the same time).

                    Thank you for explaining this part of the problem. Of course, it goes deeper than not configuring the http load-balancer (please always enable "sticky http sessions" :-)

                    The problem as it seems to be is the web session gets modified while it is been serialized. Specifically the serialization of an org.ajax4jsf.util.LRUMap instance holding some JSF(?) state is failing with ConcurrentModificationException, so I've first thought it could be a RichFaces library problem.


                    From your description of the problem, it sounds like the following events have occurred:

                    An HTTP request is received for user Sharon, during the processing of this request (R1), an HTTP session attribute change is made. As part of the HTTP processing after request processing, we attempt to replicate the attribute change. Meanwhile, another HTTP request (R2) for user Sharon was also processed that generated an HTTP session attribute change. As part of R2 processing, the same attribute change was transmitted (from a different HTTP client service thread) and applied to the collection (while the R1 client request handling thread is iterating through the collection).

                    The java.util.ConcurrentModificationException occurs when iterating through the collection, while another thread modifies the collection.

                    Before we discuss possible solutions, lets see if we agree on the above. If I am not stating things correctly above, please explain your understanding of what is happening.

                    • 7. Re: ConcurrentModificationException during session serializa
                      nbelaevski

                      Yes, that's right. The problem is that our (RichFaces) code synchronizes on session facade, but replication code synchronizes on internal instance of session - exception happens.

                      • 8. Re: ConcurrentModificationException during session serializa
                        smarlow

                        What could the solution be? Should SessionBasedClusteredSession.writeExternal synchronize on the same session facade as RichFaces?

                        • 9. Re: ConcurrentModificationException during session serializa
                          smarlow

                          What I mean, is should SessionBasedClusteredSession.writeExternal + SessionBasedClusteredSession.readExternal instead synchronize on getSession(), which should be the same StandardSessionFacade as RichFaces is synchronizing on.

                          I haven't recreated the bug yet. I could either recreate it or you could try the following SessionBasedClusteredSession change (two line change to synchronize on common session facade) :

                          public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException
                           {
                           synchronized (getSession()) // change 1 of 2
                           {
                           // Let superclass read in everything but the attribute map
                           super.readExternal(in);
                          
                           attributes = (Map) in.readObject();
                           }
                           }
                          
                           public void writeExternal(ObjectOutput out) throws IOException
                           {
                           synchronized (getSession) // change 2 of 2
                           {
                           // Let superclass write out everything but the attribute map
                           super.writeExternal(out);
                          
                           // Don't replicate any excluded attributes
                           Map excluded = removeExcludedAttributes(attributes);
                          
                           out.writeObject(attributes);
                          
                           // Restore any excluded attributes
                           if (excluded != null)
                           attributes.putAll(excluded);
                           }
                          
                           }
                          



                          • 10. Re: ConcurrentModificationException during session serializa
                            smarlow

                            There are other code changes that go with the change but the above should be enough for you to test.

                            The super classes also need to change to synchronize on the getSession() instead of "this".

                            • 11. Re: ConcurrentModificationException during session serializa
                              nbelaevski

                               

                              "smarlow@redhat.com" wrote:
                              What I mean, is should SessionBasedClusteredSession.writeExternal + SessionBasedClusteredSession.readExternal instead synchronize on getSession(), which should be the same StandardSessionFacade as RichFaces is synchronizing on.

                              I haven't recreated the bug yet. I could either recreate it or you could try the following SessionBasedClusteredSession change (two line change to synchronize on common session facade) :

                              public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException
                               {
                               synchronized (getSession()) // change 1 of 2
                               {
                               // Let superclass read in everything but the attribute map
                               super.readExternal(in);
                              
                               attributes = (Map) in.readObject();
                               }
                               }
                              
                               public void writeExternal(ObjectOutput out) throws IOException
                               {
                               synchronized (getSession) // change 2 of 2
                               {
                               // Let superclass write out everything but the attribute map
                               super.writeExternal(out);
                              
                               // Don't replicate any excluded attributes
                               Map excluded = removeExcludedAttributes(attributes);
                              
                               out.writeObject(attributes);
                              
                               // Restore any excluded attributes
                               if (excluded != null)
                               attributes.putAll(excluded);
                               }
                              
                               }
                              


                              Yes, if synchronization will occur on StandardSessionFacade visible to the application/framework developers, than it should work ok. I haven't built JBoss AS before, so not sure whether I'll be able to make these changes and test; in any case, I'll try that and report.

                              • 12. Re: ConcurrentModificationException during session serializa
                                smarlow

                                It is easy to build (assuming 4.2.x):

                                cd jboss_4.2.x_tree
                                cd build
                                ./build.sh
                                cd output
                                cd jboss-4.2.4.GA/bin
                                ./run.sh -c all

                                I also attached the potential 4.2 code patch (need to get it reviewed still) to JBAS-5615.

                                • 13. Re: ConcurrentModificationException during session serializa
                                  smarlow

                                  If you have difficulties applying/building the patch, I'll try to recreate/test instead. Let me know either way.

                                  • 14. Re: ConcurrentModificationException during session serializa
                                    brian.stansberry

                                    Thanks, nbelaevski for pointing out how code outside the container can't synchronize on the session, only the facade. And thanks Scott for running with this.

                                    The concern I have is StandardSession.facade is not thread safe, and what's returned from getSession() needs to be if we are going to use it as a mutex. There's no ref to facade outside StandardSession.getSession(), so ClusteredSession can override the getSession() method and replace the field with a volatile field and synchronize the construction if null.

                                    1 2 Previous Next