8 Replies Latest reply on Oct 14, 2015 7:51 AM by galder.zamarreno

    jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1

    wildfly123

      I'm getting below exception while trying to deploy an app to wildfly9.0.1 in domain mode. I'm deploying to a server group that has two servers residing on different hosts. Please help.

       

      2015-10-04 16:28:35,927 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service jboss.infinispan.ej

      b.dist: org.jboss.msc.service.StartException in service jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException: Unable to invoke method p

      ublic void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

              at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start()

      throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)

              at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850)

              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:599)

              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:554)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:424)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:438)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:118)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:109)

              at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:79)

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

              ... 4 more

      Caused by: org.infinispan.commons.CacheException: Bad response received from coordinator: CacheNotFoundResponse

              at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:525)

              at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:117)

              at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:110)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:497)

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

              ... 18 more

        • 1. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
          wdfink

          Could you attach the configuration and show a bit more details about the application?

          • 2. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
            nadirx

            It means that the cache has not been created on all nodes. As Wolf asked, can you please post the configuration and some logs narrowed to org.infinispan at DEBUG level ?

            • 3. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
              wildfly123

              I have the following infinispan subsystem in domain.xml on both servers (each residing on different hosts). Please let me know if you need more information.

               

              <subsystem xmlns="urn:jboss:domain:infinispan:3.0">

                              <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">

                                  <transport lock-timeout="120000"/>

                                  <replicated-cache name="default" mode="SYNC">

                                      <transaction mode="BATCH"/>

                                  </replicated-cache>

                              </cache-container>

                              <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                                  <transport lock-timeout="120000"/>

                                  <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                                      <locking isolation="REPEATABLE_READ"/>

                                      <transaction mode="BATCH"/>

                                      <file-store/>

                                  </distributed-cache>

                              </cache-container>

                              <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                                  <transport lock-timeout="120000"/>

                                  <local-cache name="local-query">

                                      <eviction strategy="LRU" max-entries="10000"/>

                                      <expiration max-idle="100000"/>

                                  </local-cache>

                                  <invalidation-cache name="entity" mode="SYNC">

                                      <transaction mode="NON_XA"/>

                                      <eviction strategy="LRU" max-entries="10000"/>

                                      <expiration max-idle="100000"/>

                                  </invalidation-cache>

                                  <replicated-cache name="timestamps" mode="ASYNC"/>

                              </cache-container>

                          </subsystem>

              • 4. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
                galder.zamarreno

                Where is the "ejb" cache container? It does not appear in the configuration, so removing it will likely be causing the exception that you see.

                • 5. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
                  wildfly123

                  Here is the ejb subsystem.

                   

                  <subsystem xmlns="urn:jboss:domain:ejb3:3.0">

                                  <session-bean>

                                      <stateless>

                                          <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                                      </stateless>

                                      <stateful default-access-timeout="5000" cache-ref="distributable" passivation-disabled-cache-ref="simple"/>

                                      <singleton default-access-timeout="5000"/>

                                  </session-bean>

                                  <mdb>

                                      <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:hornetq-ra.rar}"/>

                                      <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>

                                  </mdb>

                                  <pools>

                                      <bean-instance-pools>

                                          <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeo

                  ut-unit="MINUTES"/>

                                          <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeou

                  t-unit="MINUTES"/>

                                      </bean-instance-pools>

                                  </pools>

                                  <caches>

                                      <cache name="simple"/>

                                      <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

                                  </caches>

                                  <passivation-stores>

                                      <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

                                  </passivation-stores>

                                  <async thread-pool-name="default"/>

                                  <timer-service thread-pool-name="default" default-data-store="default-file-store">

                                      <data-stores>

                                          <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                                      </data-stores>

                                  </timer-service>

                                  <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>

                                  <thread-pools>

                                      <thread-pool name="default">

                                          <max-threads count="10"/>

                                          <keepalive-time time="100" unit="milliseconds"/>

                                      </thread-pool>

                                  </thread-pools>

                                  <iiop enable-by-default="false" use-qualified-name="false"/>

                                  <default-security-domain value="other"/>

                                  <default-missing-method-permissions-deny-access value="true"/>

                                  <log-system-exceptions value="true"/>

                              </subsystem>

                  • 6. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
                    galder.zamarreno

                    Forget about the EJB subsystem, your cache container list has the `ejb` cache container missing

                     

                    I'd suggest you download a clean Wildfly 9 zip and start from scratch.

                    • 7. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
                      wildfly123

                      Sorry, I actually have ejb cache container under infinispan subsystem, but I removed that piece to make the deployment work. But here is the original configuration.

                       

                      <subsystem xmlns="urn:jboss:domain:infinispan:3.0">

                                      <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">

                                          <transport lock-timeout="120000"/>

                                          <replicated-cache name="default" mode="SYNC">

                                              <transaction mode="BATCH"/>

                                          </replicated-cache>

                                      </cache-container>

                                      <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                                          <transport lock-timeout="120000"/>

                                          <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                                              <locking isolation="REPEATABLE_READ"/>

                                              <transaction mode="BATCH"/>

                                              <file-store/>

                                          </distributed-cache>

                                      </cache-container>

                                      <cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">

                                          <transport lock-timeout="120000"/>

                                          <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                                              <locking isolation="REPEATABLE_READ"/>

                                              <transaction mode="BATCH"/>

                                              <file-store/>

                                          </distributed-cache>

                                      </cache-container>

                                      <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                                          <transport lock-timeout="120000"/>

                                          <local-cache name="local-query">

                                              <eviction strategy="LRU" max-entries="10000"/>

                                              <expiration max-idle="100000"/>

                                          </local-cache>

                                          <invalidation-cache name="entity" mode="SYNC">

                                              <transaction mode="NON_XA"/>

                                              <eviction strategy="LRU" max-entries="10000"/>

                                              <expiration max-idle="100000"/>

                                          </invalidation-cache>

                                          <replicated-cache name="timestamps" mode="ASYNC"/>

                                      </cache-container>

                                  </subsystem>

                      • 8. Re: jboss.infinispan.ejb.dist: org.infinispan.commons.CacheException while deploying to wildfly9.0.1
                        galder.zamarreno

                        ...but I removed that piece to make the deployment work

                        My suspicion is that not ejb cache container defined in all nodes in the cluster is what is causing the issue.

                         

                        Once you put back restore the same configuration in all nodes in the cluster, do you still have the same issue?

                         

                        If so, I'd suggest start from a untouched Wildfly distribution and see whether you still have the original issue

                         

                        Btw, what deployment issue were you having that lead you to remove the ejb cache container? :\