8 Replies Latest reply on Jul 21, 2011 1:32 PM by galder.zamarreno

    Strange data being passed to readExternal by unmarshaller

    rocketraman

      I have an Externalizable object that is being cached using Infinispan 4.2.1.FINAL, in async replication mode across two instances. The layout is as follows:

       

      instance 1: cluster name AAA

      instance 2: cluster name BBB

      instance 3: cluster name CCC

      instance 4: cluster name AAA, BBB, CCC

       

      i.e. instance 4 serves as a replicated cache for instance 1, 2, and 3 and is a member of all 3 infinispan clusters.

       

      At some point shortly after system startup, on instance 4 my Externalizable Object's readExternal method is called, but with data that is either corrupted or that belongs to a different type of Object. The (invalid) data passed to my Object's readExternal method by Infinispan starts with:

       

      0x 00 00 00 01 44 00 00 00 24 00 00 4E

       

      The data actually causes an OOME to be thrown by readExternal because of a very large array allocation caused by the invalid data from ObjectInput. The full stack at the point the OOME is thrown (captured via OnOutOfMemoryError and jstack) is here:

       

      https://gist.github.com/1010811

       

      This is with Infinispan 4.2.1.FINAL and Jboss Marshalling 1.2.3.GA and Jgroups 2.12.0.FINAL.

       

      I am tracing through the stack trying to determine why my application code is receiving this corrupted data from Infinispan but so far have not had any success. Any tips would be greatly appreciated.

        • 1. Re: Strange data being passed to readExternal by unmarshaller
          dmlloyd

          OOME from too-large arrays are usually harmless to the JVM but it does underscore that it is essential to validate data read by externalizers or custom serialization implementations.

           

          It looks like you're seeing stream corruption.  This would be very unlikely to be caused by other Externalizable classes due to safety measures employed for those object types.  The possible causes for corrupted data are, in order of decreasing likelihood:

          1. A bug in the user (java.io.)Externalizable class itself
          2. A bug in a ClassTable or ObjectTable implementation which is in use by Infinispan, which trade safety for speed (this would point to Infinispan or possibly a user class delegated to by Infinispan)
          3. A bug in JBoss Marshalling involving a problem containing the isolation between user stream data and internal stream data

           

          Would it be possible to see the Externalizable class in question in a pastebin?

          • 2. Re: Strange data being passed to readExternal by unmarshaller
            rocketraman

            I have provided essentialized/simplifed versions of the relevant Externalizable Objects here:

             

            https://gist.github.com/1013834

             

            Though much of the public interface that are simply accessors and are therefore irrelevant for mashalling have been stripped away, the writeExternal and readExternal methods have not been modified from the original. As you can see, the writeExternal/readExternal methods are pretty simple and I would be surprised to see a bug there. However, I'd be happy to be proven wrong .

             

            Note the OOME is actually being thrown by org.jboss.marshalling.UTFUtils.readUTFBytes when MField.readExternal calls in.readUTF() -- UTFUtils.readUTFBytes reads the length of the String as an int value (in the last example I saw, it reads a length of 0x2400004E or 603,979,854 chars) and it therefore tries to allocate a char[603979854], which obviously fails.

            • 3. Re: Strange data being passed to readExternal by unmarshaller
              galder.zamarreno

              Why does your class implement org.jgroups.util.Streamable as well? I don't see any reason for your class to be implementing that. You're already Externalizable and it's Infinispan that will be dealing with the serialization. I dunno what the effects of Streamable could be here but you do not want JGroups to be doing funny things with it trying to serialize your class again, so I'd try to leave it out and the implemented methods and try your test again.

               

              Btw, it'd be really useful if you could provide us with some test/setup that we can run to try to replicate your issue. Would that be possible?

              • 4. Re: Strange data being passed to readExternal by unmarshaller
                rocketraman

                It implements Streamable because it is also used by some other code that sends these Objects around a different non-Infinispan JGroups cluster. I can certainly try removing it. However, if the Streamable interface were causing JGroups to corrupt the stream, wouldn't the problem happen consistently instead of rarely as it does now?

                 

                In any case, if that doesn't fix the problem, I will try to create a reproducible test case. However, as always with these types of "inconsistent" problems, that may not be possible.

                • 5. Re: Strange data being passed to readExternal by unmarshaller
                  rocketraman

                  Update: removing the Streamable interface did not help. Updating to Infinispan 5.0.0 CR4 and the associated marshalling library (1.3.0.CR9) also did not help.

                   

                  I am trying to create a test case but so far have not been successful in reproducing the issue outside of the application.

                  • 6. Re: Strange data being passed to readExternal by unmarshaller
                    rocketraman

                    SOLVED: Problem was due to unsafe threading. Using Infinispan putAsync, but Object that was being serialized was likely modified by another Thread after putAsync was called and during the Infinispan serialization. Adding the appropriate synchronization seems to have fixed it. Thanks a lot for the help.

                    • 7. Re: Strange data being passed to readExternal by unmarshaller
                      manik

                      Could someone create a FAQ around this?  No one else has reported this as yet but it is concievably something that could affect others.