Seems like a serialization issue (which uses Rive based on stackTrace).

Are you sure you have identical objects and infinispan version in both ends?

see this: http://community.jboss.org/wiki/InfinispanTechnicalFAQs#Marshalling__Unmarshalling

and this: http://docs.jboss.org/infinispan/4.0/apidocs/config.html#ce_global_serialization

Wrt to:

Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:708)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)

What Infinispan version are you using? Can you try using latest CR4?

Hmmmm, could you to mimic what you're doing in a unit test? I mean, from what I understand, your unit test should include:

- start first cache with your config, and do the operations you've been doing in the 1st instance.

- start second cache with your config and it should fail with that StateTransferException

Note that if you mark shared in your cache loader, the cache assumes that both caches are directed to the same file path. If you want them to point to different paths, shared should be set to false. See the configuration documentation in http://infinispan.sourceforge.net/4.0/apidocs/config.html#ce_loaders_loader

Hello,

Any update on this issue? I am getting the same exception.

Caused by: java.io.StreamCorruptedException: Unexpected byte found when reading an object: 0
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:708)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
    at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1637)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1553)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1202)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:270)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
    at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1637)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1553)
    at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1517)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1202)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:270)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
    at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:64)
    at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:264)
    at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:251)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
    at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:64)
    at org.infinispan.marshall.jboss.ConstantObjectTable$ExternalizerAdapter.readObject(ConstantObjectTable.java:264)
    at org.infinispan.marshall.jboss.ConstantObjectTable.readObject(ConstantObjectTable.java:251)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:357)
    at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:207)
    at org.jboss.marshalling.AbstractUnmarshaller.readObject(AbstractUnmarshaller.java:85)
    at org.infinispan.marshall.jboss.JBossMarshaller.objectFromObjectStream(JBossMarshaller.java:207)
    at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:109)
    at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromByteBuffer(MarshallerAdapter.java:26)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:148)
    ... 27 more

Caused by: an exception which occurred:
    in field authentication
    in object of type org.jasig.cas.ticket.TicketGrantingTicketImpl
    in field ticketGrantingTicket
    in object of type org.jasig.cas.ticket.ServiceTicketImpl

I am using Ja-sig's CAS version 3.3.5 and trying to add Infinispan 4.0.0.FINAL support to that and following test case is failing:

@Test
    public void testPutTicket() {
        Principal principal = new SimpleWebApplicationServiceImpl("someurl");
        Authentication authentication = new ImmutableAuthentication(principal);
        TicketGrantingTicket tgt = new TicketGrantingTicketImpl("testId", authentication, new TimeoutExpirationPolicy(3600000));
        this.registry1.addTicket(tgt);
        ServiceTicket st = tgt.grantServiceTicket("ST-ID", new SimpleWebApplicationServiceImpl("CASifiedAppUrl"), new MultiTimeUseOrTimeoutExpirationPolicy(1, 300000), Boolean.TRUE);
        this.registry1.addTicket(st);
    }

Surprisingly, this.registry1.addTicket(tgt) operation succeeds whereas this.registry.addTicket(st) fails in deserializing TicketGrantingTicket inside ServiceTicket.

I debugged RiverMarshaller and RiverUnmarshaller object's doWrite, doWriteFields, doInitSerializable & doRead, doReadFields, doWriteSerializableObject methods and realized one difference between marshaller and unmarshaller. I feel that the code in doInitSerializable & doWriteSerializableObject is causing the issue as the marshaller serializes the parent class's members first and then the child class's members second whereas unmarshaller skips parent and tries to deserialize fields of child class. Marshaller serializes fields of AbstractTicket first and then fields of child class TicketGrantingTicketImpl whereas unmarshaller tries to deserialze fields of TicketGrantingTicketImpl without deserializing fields of AbstractTicket.

Something to do with following lines of code:

RiverMarshaller.java:

protected void doWriteSerializableObject(final SerializableClass info, final Object obj, final Class<?> objClass) throws IOException {

        final Class<?> superclass = objClass.getSuperclass();
        if (Serializable.class.isAssignableFrom(superclass)) {
            doWriteSerializableObject(registry.lookup(superclass), obj, superclass);
        }

        :

        :

}

RiverUnmarshaller.java:

private void doInitSerializable(final Object obj, final SerializableClassDescriptor descriptor) throws IOException, ClassNotFoundException {
        final Class<?> type = descriptor.getType();
        final SerializableClass info = registry.lookup(type);
        final ClassDescriptor superDescriptor = descriptor.getSuperClassDescriptor();
        if (superDescriptor instanceof SerializableClassDescriptor) {
            final SerializableClassDescriptor serializableSuperDescriptor = (SerializableClassDescriptor) superDescriptor;
            doInitSerializable(obj, serializableSuperDescriptor);
        }

        :

        :

}

AbstractTicket's ClassDescriptor is not an instance of SerializableClassDescriptor hence, deserialization of AbstractTicket's fields are skipped (?) and just TicketGrantingTicketImpl's fields are being deserialized which results into corrupted input stream.

FWIW, we are currently using JBossCache 3.2.1.GA and we have no such issue with it.

Let me know if you have any questions and I can be of any help to resolve the issue quicker (assuming that what I am saying above is the real issue).

Thanks,

Mihir

Here is the TestCase which fails.

----------------------------------------

InfinispanCacheTests.java

----------------------------------------

import junit.framework.TestCase;

import org.infinispan.Cache;
import org.infinispan.manager.CacheManager;
import org.infinispan.manager.DefaultCacheManager;
import org.junit.Test;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;

public class InfinispanCacheTests extends TestCase {
    CacheManager cacheManager1;
    CacheManager cacheManager2;
    Cache<String, Object> cache1;
    Cache<String, Object> cache2;

    @Override
    protected void setUp() throws Exception {
        Resource resource = new ClassPathResource("infinispanTicketCache.xml");
        cacheManager1 = new DefaultCacheManager(resource.getInputStream());
        cache1 = cacheManager1.getCache();
       
        cacheManager2 = new DefaultCacheManager(resource.getInputStream());
        cache2 = cacheManager2.getCache();
    }
   
    @Test
    public void testPut() {
        Child1 child1Obj = new Child1(1234, "1234");
        Child2 child2Obj = new Child2(2345, "2345", child1Obj);

        this.cache1.put(child1Obj.getId(), child1Obj); //Works fine as the Child object inside Parent is NULL
        this.cache1.put(child2Obj.getId(), child2Obj); //Fails because Parent has object which is Child of its own
    }
   
}

----------------------------------------

Parent.java

----------------------------------------

public class Parent implements Serializable {
    private String id;
    private Child1 child1Obj;
   
    public Parent(String id, Child1 child1Obj) {
        this.id = id;
        this.child1Obj = child1Obj;
    }
   
    public String getId() {
        return id;
    }
    public Child1 getChild1Obj() {
        return child1Obj;
    }
}

----------------------------------------

Child1.java

----------------------------------------

public class Child1 extends Parent {
    private int someInt;
   
    public Child1(int someInt, String parentStr) {
        super(parentStr, null);
        this.someInt = someInt;
    }
   
}

----------------------------------------

Child2.java

----------------------------------------

public class Child2 extends Parent {
    private int someInt;
   
    public Child2(int someInt, String parentStr, Child1 child1Obj) {
        super(parentStr, child1Obj);
        this.someInt = someInt;
    }
}

Note: See attached cache config. sync replTimeout is set high just so that I have more time to debug.

I found that RiverMarshaller.writeKnownClass passes relative difference to get the class from the classCache with ID_REPEAT_CLASS_NEAR instead of serializing it again, on the unmarshaller side, the ID_REPEAT_CLASS_NEAR tries to read the Class information from the classCache but the problem is that the Parent class is still being "understood" and the reference to it is "IncompleteClassDescriptor" which gets used as the superDescriptor for Child1 (which is also part of Parent) which is causing the issue and following if condition return false and corrupts stream.

RiverUnmarshaller.java

private void doInitSerializable(final Object obj, final SerializableClassDescriptor descriptor) throws IOException, ClassNotFoundException {
        final Class<?> type = descriptor.getType();
        final SerializableClass info = registry.lookup(type);
        final ClassDescriptor superDescriptor = descriptor.getSuperClassDescriptor();
        if (superDescriptor instanceof SerializableClassDescriptor) { //<== superDescriptor is IncompleteClassDescriptor as it was read from classCache while it was still being interpreted from input stream
            final SerializableClassDescriptor serializableSuperDescriptor = (SerializableClassDescriptor) superDescriptor;
            doInitSerializable(obj, serializableSuperDescriptor);
        }

        :

        :

}

Anyone can confirm my understanding?

Thanks,

Mihir

Mihir, thanks for attaching the unit test. I'm currently looking into it.

Mihir, this appears to be a bug, see  https://jira.jboss.org/jira/browse/JBMAR-106. Your suspicions look correct as well. We'll address the issue asap.

Thanks Galder. Any idea as to when this issue might get fixed? Asking so that I can plan moving to infinispan accordingly. Not sure how common is to have such class structure!

You should create an ISPN jira for this, and link it to the JBMAR one.  And vote for it.    It will certainly get in for 4.1.0, whether we release a 4.0.1 with this fix is what we need to think about, based on how quick JBMAR gets patched.  What are your timescales?

Hi Manik,

Our current deployment is working fine with JBossCache 3.2.1.GA, but in the later part of our implementation I realized that Infinispan is the better choice to use for future and I was really excited to try out the Distributed Replication in Infinispan as it promises to have scalability as number of nodes in the cluster increases. We are using jasig's CAS with JBossCache but I see the limitation of having same copy of data on all servers, and exactly as you mentioned in one of your article about distributed replication in Infinispan, we would not get more RAM to store more data as we increase nodes, and I would like to get rid of that limitation by trying out Infinispan's distributed replication with may be 2 or 4 num copies.

Once I test out Infinispan, plan is to add Infinispan support to jasig's CAS and we don't have any hard deadlines there as well, but it would be good to have this issue fixed in a month or two.

I created https://jira.jboss.org/jira/browse/ISPN-389 (left the fix version blank) as you suggested but cannot vote as it says "You cannot vote for an issue you have reported".

Thanks,

Mihir

Mihi, I made a note in the JIRA with instructions on how to get the fix. Basically, the JBMAR issue is closed and fixed, so simply build the 1.2.x branch of JBMAR and you'll have the fix.

Great. I built jar from the 1.2 branch and issue seems to be fixed now.

Thanks for all your help.