8 Replies Latest reply on May 21, 2012 11:42 AM by dex80526

    CacheException: Unable to invoke method JGroupsTransport.start()

    dex80526

      I saw some wird exceptions with 5.1.4.Final code, the stack trace is below.

       

      Here is some conext:

       

      I have a 2 node cluster in Replication mode, and have a seprate StatusUpdater thread polls the status of cahces (e.g., it tries to call size() on a named cache).

       

       

      It seems to me that we the StatusUpdater thread calls size() on the cache will cause the following stacktrace if the cache is not completly "ready" yet, and the cache ends up in that state forever.

       

       

      In addition, it piles up a lot dummy-** protocolStack in the trace.

       

       

      I expect the cache/cache manager should handle the request gracefuly if it is ready.

       

      How to avoid this?

       

       

      org.infinispan.CacheException: Unable to invoke method public void

      org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on

      object of type JGroupsTransport

              at

      org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238)

              at

      org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:639)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:226)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:152)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:167)

              at

      org.infinispan.factories.InterceptorChainFactory.register(InterceptorChainFactory.java:73)

              at

      org.infinispan.factories.InterceptorChainFactory.createInterceptor(InterceptorChainFactory.java:62)

              at

      org.infinispan.factories.InterceptorChainFactory.buildInterceptorChain(InterceptorChainFactory.java:95)

              at

      org.infinispan.factories.InterceptorChainFactory.construct(InterceptorChainFactory.java:272)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:286)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.AbstractComponentRegistry.getOrCreateComponent(AbstractComponentRegistry.java:291)

              at

      org.infinispan.factories.AbstractComponentRegistry.invokeInjectionMethod(AbstractComponentRegistry.java:246)

              at

      org.infinispan.factories.AbstractComponentRegistry.access$000(AbstractComponentRegistry.java:83)

              at

      org.infinispan.factories.AbstractComponentRegistry$Component.injectDependencies(AbstractComponentRegistry.java:792)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponentInternal(AbstractComponentRegistry.java:220)

              at

      org.infinispan.factories.ComponentRegistry.registerComponentInternal(ComponentRegistry.java:154)

              at

      org.infinispan.factories.AbstractComponentRegistry.registerComponent(AbstractComponentRegistry.java:175)

              at

      org.infinispan.factories.InternalCacheFactory.bootstrap(InternalCacheFactory.java:102)

              at

      org.infinispan.factories.InternalCacheFactory.createAndWire(InternalCacheFactory.java:78)

              at

      org.infinispan.factories.InternalCacheFactory.createCache(InternalCacheFactory.java:62)

              at

      org.infinispan.manager.DefaultCacheManager.wireCache(DefaultCacheManager.java:692)

              at

      org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)

              at

      org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)

              at

      .....StatusUpdater.poll.....

      ....

      Caused by: org.infinispan.CacheException: Unable to start JGroups Channel

              at

      org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:186)

              at

      org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:175)

              at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source)

              at

      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:601)

              at

      org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236)

              ... 96 more

      Caused by: java.lang.IllegalStateException: cluster 'HA_Customer_9.3' is

      already connected to singleton transport: [dummy-1336085803144,

      dummy-1336085899401, dummy-1336085785840, dummy-1336085828476,

      dummy-1336085808197, HA_Customer_9.3, dummy-1336085874063,

      dummy-1336085868964, dummy-1336085785779, dummy-1336085818352,

      dummy-1336085792988, dummy-1336085879152, dummy-1336085823409,

      dummy-1336085785693, dummy-1336085848698, dummy-1336085785734,

      dummy-1336085813295, dummy-1336085889257, dummy-1336085853769,

      dummy-1336085894308, dummy-1336085843643, dummy-1336085833534,

      dummy-1336085787888, dummy-1336085798048, dummy-1336085785653,

      dummy-1336085884206, dummy-1336085863914, dummy-1336085785603,

      dummy-1336085838588, dummy-1336085858821]

              at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:909)

              at org.jgroups.JChannel.startStack(JChannel.java:841)

              at org.jgroups.JChannel.connect(JChannel.java:277)

              at org.jgroups.JChannel.connect(JChannel.java:261)

              at

      org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:184)it

        • 1. Re: CacheException: Unable to invoke method JGroupsTransport.start()
          dex80526

          This problem can be reproduced by quickly stop/re-start a cache manager.

          Note: I am using jgroup-tcp.xml.

           

          It looks like Jgroup transport hits the following situation when this happens.

           

          1. after we first start cache manager, jgroups transport establishs TCP connections between 2 nodes one a given porrts (e.g., 7900, 7902).

           

          2. when we call cacheManger.stop() and immedately creats another cache manager with the same configuration, some how, the tcp connections from the previous cache manager instance did not get tear down (I observed by watich netstat output), when this happens, the second cache manager will run into the above situation, and the jgroups transport ends up in the error state.

           

          The question is how to ensure the cacheManager.stop() clears/tears down every things.

          • 2. Re: CacheException: Unable to invoke method JGroupsTransport.start()
            dex80526

            I narrowed the cause of this problem. It is caused by cache manager can not terminate the first cache manager instance becaue of SyncFailedException: sync failed as shown at the end.

             

            I do not see there is any way to froce stop the cache manager when sync failed.  How to force terminat the cache manager (not by killing the JVM)?

             

            ====stack trace==============

             

            2012-05-17/10:11:32.539/MDT [Thread-23] DEBUG com.symplified.service.shared.platform.cluster.InfinispanClusterCacheStoreServiceImpl[183] - exception caught in stop existing cache manager

            org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.stop() on object of type CacheLoaderManagerImpl

                    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238)

                    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:885)

                    at org.infinispan.factories.AbstractComponentRegistry.internalStop(AbstractComponentRegistry.java:674)

                    at org.infinispan.factories.AbstractComponentRegistry.stop(AbstractComponentRegistry.java:553)

                    at org.infinispan.factories.ComponentRegistry.stop(ComponentRegistry.java:199)

                    at org.infinispan.CacheImpl.stop(CacheImpl.java:539)

                    at org.infinispan.CacheImpl.stop(CacheImpl.java:531)

                    at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:742)

            .....

            Caused by: org.infinispan.CacheException: java.io.SyncFailedException: sync failed

                    at org.infinispan.loaders.jdbm.JdbmCacheStore.stop(JdbmCacheStore.java:246)

                    at org.infinispan.loaders.decorators.AbstractDelegatingStore.stop(AbstractDelegatingStore.java:155)

                    at org.infinispan.loaders.decorators.AsyncStore.stop(AsyncStore.java:237)

                    at org.infinispan.loaders.CacheLoaderManagerImpl.stop(CacheLoaderManagerImpl.java:221)

                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                    at java.lang.reflect.Method.invoke(Method.java:601)

                    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236)

                    ... 14 more

            Caused by: java.io.SyncFailedException: sync failed

                    at java.io.FileDescriptor.sync(Native Method)

                    at jdbm.recman.TransactionManager.sync(TransactionManager.java:331)

                    at jdbm.recman.TransactionManager.close(TransactionManager.java:347)

                    at jdbm.recman.TransactionManager.synchronizeLogFromMemory(TransactionManager.java:156)

                    at jdbm.recman.TransactionManager.shutdown(TransactionManager.java:339)

                    at jdbm.recman.RecordFile.close(RecordFile.java:346)

                    at jdbm.recman.BaseRecordManager.close(BaseRecordManager.java:168)

            • 3. Re: CacheException: Unable to invoke method JGroupsTransport.start()
              galder.zamarreno

              To your 1st question, maybe you have multiple Infinispan caches with the same cluster nodeName? Seems like there's two: HA_Customer_9.3.

               

              If the issue is coming from quick restart, do no set the node name and let JGroups give you a dynamic number which avoids such issue. This is an old topic called the reincarnation problem, which is solved by avoiding fixed node names: https://issues.jboss.org/browse/JGRP-130

               

              WRT last comment, you're having an issue stopping the cache manager but you seem to ignore it by simply logging a debug message. Did you propagate this exception?

              • 4. Re: CacheException: Unable to invoke method JGroupsTransport.start()
                dex80526

                Hi Galder:  Thanks for the point to the reincarnation issue.  I think the reincarnation is the root cause.  That might also contribute the OutOfMemoryError I had.

                 

                Yes, I specified nodeName and machineId in trasport config like below:

                 

                <global>

                      <transport clusterName="DexTest_9.3" machineId="dexportal2.dex.net" nodeName="dexportal2.dex.net" rackId="rack_DexTest_9.3">

                         <properties>

                            <property name="configurationFile" value="/usr/local/test/jgroups-tcp.xml"/>

                         </properties>

                      </transport>

                      <globalJmxStatistics enabled="false"/>

                      <!--

                            Used to register JVM shutdown hooks.

                            hookBehavior: DEFAULT, REGISTER, DONT_REGISTER

                      -->

                       <shutdown hookBehavior="DONT_REGISTER"/>

                   </global>

                 

                 

                I like the logical address as described in Belas's blog.

                 

                How do I configure it to use logical addresss?

                • 5. Re: CacheException: Unable to invoke method JGroupsTransport.start()
                  galder.zamarreno

                  Remove nodeName from the configuration and it'll use a logical address.

                  • 6. Re: CacheException: Unable to invoke method JGroupsTransport.start()
                    dan.berindei

                    Galder, the JGroups logical address is actually a UUID, so it'll be unique every time.

                    The address string representation is also unique, because Infinispan actually adds a random number to the nodeName setting. 'HA_Customer_9.3' seems to be the cluster name.

                     

                    Dex, the "cluster 'HA_Customer_9.3' is already connected to singleton transport" error suggests you have enabled shared transport in the JGroups configuration, and stopping the cache manager didn't properly remove the cluster from JGroups' internal list of started clusters (as you're not allowed to start 2 JChannels with the same singletonName in the same JVM that connect to the same cluster).

                     

                    Right now any exception during cache manager shutdown seems to stop the shutdown procedure and put it in the TERMINATED state, without a chance to retry the shutdown. It would probably be a good idea to log any failures and continue with the shutdown of the other components, please open a JIRA for it.

                    • 7. Re: CacheException: Unable to invoke method JGroupsTransport.start()
                      dex80526

                      HA_Customer_9.2 is the cluster name; ISPN does add a random number to node names. The problem is if there is any error in  cache maanger stop, there is no way to retry or force it  quit.  I tried to retry calling cachemanager.stop() when I caguht any exception in stop(), the extra stop() does nothing.  I'll fine JIRA issue on this. 

                       

                      Thanks.

                      • 8. Re: CacheException: Unable to invoke method JGroupsTransport.start()
                        dex80526

                        Here is the JIRA issue: ISPN-2056