3 Replies Latest reply on Oct 2, 2015 3:41 AM by rbattenfeld

    Wildfly-9.0.1: Distributed web cache not started

    rbattenfeld

      Hi

       

      We are migrating from Wildfly-8.2.0.final to 9.0.1.Final. We face an issue related to the distributed web cache. Somehow, this distributed web cache is not started.

       

      I can see that JGroups is starting all three channels:

      2015-07-26 11:23:51,704 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000078: Starting JGroups channel ejb

      2015-07-26 11:23:51,693 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel web

      2015-07-26 11:23:53,842 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000078: Starting JGroups channel lifecycle-cache

       

      But then I see just the two other caches started:

      2015-07-26 11:23:52,593 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0002: Started dist cache from ejb container

      2015-07-26 11:23:53,872 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0002: Started default cache from lifecycle-cache container

       

      I am completely not sure if this issue is related to an own infinispan (name: lifecycle-cache) cache we need for application specific lifecycle manangement. This worked with 8.2.0 and 8.2.1 fine. We would like to upgrade since the clustered-HA-singleton implementation has an important fix resolved in 9.0.1. The web module which we try to deploy is flagged with 'distributed'.

       

      Is there something wrong in the configuration? Below are the infinispan config and the stacktrace. I have also attached the domain.xml.

       

      Regards,

      Ralf

       

      Infinispan configuration in domain.xml (ha profile):

                  <subsystem xmlns="urn:jboss:domain:infinispan:3.0">

                          <cache-container name="lifecycle-cache" default-cache="default" aliases="lifecycle cache">

                          <transport lock-timeout="60000"/>

                          <replicated-cache name="lifecycle-cache-replicated" mode="SYNC">

                              <transaction mode="BATCH"/>

                              <file-store preload="true" purge="false" passivation="false" relative-to="jboss.server.data.dir" path="lifecycle-cache"/>

                          </replicated-cache>

                      </cache-container>

                      <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">

                          <transport lock-timeout="60000"/>

                          <replicated-cache name="default" mode="SYNC">

                              <transaction mode="BATCH"/>

                          </replicated-cache>

                      </cache-container>

                      <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                          <transport lock-timeout="60000"/>

                          <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                              <locking isolation="REPEATABLE_READ"/>

                              <transaction mode="BATCH"/>

                              <file-store/>

                          </distributed-cache>

                      </cache-container>

                      <cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">

                          <transport lock-timeout="60000"/>

                          <distributed-cache name="dist" mode="ASYNC" owners="2" l1-lifespan="0">

                              <locking isolation="REPEATABLE_READ"/>

                              <transaction mode="BATCH"/>

                              <file-store/>

                          </distributed-cache>

                      </cache-container>

                      <cache-container name="hibernate" default-cache="local-query" module="org.hibernate.infinispan">

                          <transport lock-timeout="60000"/>

                          <local-cache name="local-query">

                              <eviction strategy="LRU" max-entries="10000"/>

                              <expiration max-idle="100000"/>

                          </local-cache>

                          <invalidation-cache name="entity" mode="SYNC">

                              <transaction mode="NON_XA"/>

                              <eviction strategy="LRU" max-entries="10000"/>

                              <expiration max-idle="100000"/>

                          </invalidation-cache>

                          <replicated-cache name="timestamps" mode="ASYNC"/>

                      </cache-container>

                  </subsystem>

       

      Stacktrace:

      2015-07-26 11:23:50,331 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0006: Starting Services for CDI deployment: java-ee-common-lifecycle-war-admin.war

      2015-07-26 11:23:50,380 INFO  [org.jboss.weld.Version] (MSC service thread 1-2) WELD-000900: 2.2.14 (Final)

      2015-07-26 11:23:50,427 INFO  [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0009: Starting weld service for deployment java-ee-common-lifecycle-war-admin.war

      2015-07-26 11:23:50,642 INFO  [org.jboss.weld.deployer] (MSC service thread 1-8) WFLYWELD0006: Starting Services for CDI deployment: request-portal-dispatcher-war-dispatcher-prod.war

      2015-07-26 11:23:50,683 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0009: Starting weld service for deployment request-portal-dispatcher-war-dispatcher-prod.war

      2015-07-26 11:23:51,053 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      2015-07-26 11:23:51,055 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      2015-07-26 11:23:51,056 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      2015-07-26 11:23:51,056 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-4) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      2015-07-26 11:23:51,063 INFO  [stdout] (MSC service thread 1-4)

      2015-07-26 11:23:51,063 INFO  [stdout] (MSC service thread 1-4) -------------------------------------------------------------------

      2015-07-26 11:23:51,064 INFO  [stdout] (MSC service thread 1-4) GMS: address=host1:nesi-request-portal-server, cluster=ee, physical address=172.17.1.254:56900

      2015-07-26 11:23:51,064 INFO  [stdout] (MSC service thread 1-4) -------------------------------------------------------------------

      2015-07-26 11:23:51,512 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 60) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,512 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 62) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,525 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 62) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,526 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 60) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,527 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 61) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,530 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 61) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.

      2015-07-26 11:23:51,704 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000078: Starting JGroups channel ejb

      2015-07-26 11:23:51,693 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000078: Starting JGroups channel web

      2015-07-26 11:23:51,748 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000094: Received new cluster view for channel web: [host1:nesi-reference-server|2] (3) [host1:nesi-reference-server, host1:nesi-reference-server3, host1:nesi-request-portal-server]

      2015-07-26 11:23:51,768 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000094: Received new cluster view for channel ejb: [host1:nesi-reference-server|2] (3) [host1:nesi-reference-server, host1:nesi-reference-server3, host1:nesi-request-portal-server]

      2015-07-26 11:23:51,803 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000079: Channel ejb local address is host1:nesi-request-portal-server, physical addresses are [172.17.1.254:56900]

      2015-07-26 11:23:51,803 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 62) ISPN000079: Channel web local address is host1:nesi-request-portal-server, physical addresses are [172.17.1.254:56900]

      2015-07-26 11:23:51,865 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 60) ISPN000128: Infinispan version: Infinispan 'Insanely Bad Elf' 7.2.3.Final

      2015-07-26 11:23:51,872 WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (ServerService Thread Pool -- 62) ISPN000329: Unable to read rebalancing status from coordinator host1:nesi-reference-server

      2015-07-26 11:23:52,593 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0002: Started dist cache from ejb container

      2015-07-26 11:23:53,630 INFO  [org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher] (ServerService Thread Pool -- 65) Deploying javax.ws.rs.core.Application: class com.six_group.java_ee.common.lifecycle.war.admin.JaxRsActivator$Proxy$_$$_WeldClientProxy

      2015-07-26 11:23:53,643 INFO  [org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher] (ServerService Thread Pool -- 65) Adding singleton resource com.six_group.java_ee.common.lifecycle.war.admin.AdminRequestDelegator from Application class com.six_group.java_ee.common.lifecycle.war.admin.JaxRsActivator$Proxy$_$$_WeldClientProxy

      2015-07-26 11:23:53,660 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 65) WFLYUT0021: Registered web context: /java-ee-common-lifecycle-war-admin

      2015-07-26 11:23:53,790 WARN  [com.six_group.java_ee.common.api.util.PropertiesHandler] (ServerService Thread Pool -- 69) Can't extract property: com.six_group.java_ee.common.lifecycle.core.scheduler.events Use default:

      2015-07-26 11:23:53,827 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured ref server web context : /seni-ref-server-web

      2015-07-26 11:23:53,829 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured ref server node list   : edbe47631237:9580:host1,edbe47631237:9680:host2

      2015-07-26 11:23:53,829 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured connection timeout     : 10000

      2015-07-26 11:23:53,830 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured socket timeout         : 35000

      2015-07-26 11:23:53,830 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured polling interval       : 5000

      2015-07-26 11:23:53,831 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Configured max pooled connections : 100

      2015-07-26 11:23:53,833 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Adding to Reference Server URLs. Node: host1 Url: http://edbe47631237:9580/seni-ref-server-web

      2015-07-26 11:23:53,833 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (ServerService Thread Pool -- 60) Adding to Reference Server URLs. Node: host2 Url: http://edbe47631237:9680/seni-ref-server-web

      2015-07-26 11:23:53,830 INFO  [com.six_group.ssx.seni.request_portal.module.ejb.version.VersionBean] (ServerService Thread Pool -- 66) request-portal version: LATEST-20150724-279

      2015-07-26 11:23:53,842 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000078: Starting JGroups channel lifecycle-cache

      2015-07-26 11:23:53,843 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000094: Received new cluster view for channel lifecycle-cache: [host1:nesi-reference-server|2] (3) [host1:nesi-reference-server, host1:nesi-reference-server3, host1:nesi-request-portal-server]

      2015-07-26 11:23:53,844 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000079: Channel lifecycle-cache local address is host1:nesi-request-portal-server, physical addresses are [172.17.1.254:56900]

      2015-07-26 11:23:53,858 INFO  [com.six_group.java_ee.common.lifecycle.core.admin.AdminService] (ServerService Thread Pool -- 63) AdminServiceObjectName plugin found. Customized admin service object name is: com.six_group.ssx.seni.request_portal.dispatcher:service=AdminService

      2015-07-26 11:23:53,871 INFO  [com.six_group.java_ee.common.lifecycle.core.lifecycle.LifeCycleServiceBean] (ServerService Thread Pool -- 71) Registered Module  : HttpDispatcherLifecycleBean    InitPriority: 050 StopPriority: 050 Version: null

      2015-07-26 11:23:53,872 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0002: Started default cache from lifecycle-cache container

      2015-07-26 11:23:53,876 INFO  [com.six_group.java_ee.common.lifecycle.core.lifecycle.LifeCycleServiceBean] (ServerService Thread Pool -- 71) Cluster mode is manually set to false.

      2015-07-26 11:23:53,882 INFO  [com.six_group.java_ee.common.lifecycle.core.lifecycle.LifeCycleServiceBean] (ServerService Thread Pool -- 71) Registered Service : SchedulerBean                  InitPriority: 000 StopPriority: 000 Version: LATEST-20150421-97

      2015-07-26 11:23:53,890 INFO  [com.six_group.java_ee.common.lifecycle.core.lifecycle.LifeCycleServiceBean] (ServerService Thread Pool -- 71) No commands for recovery found!

      2015-07-26 11:24:04,711 INFO  [com.six_group.java_ee.common.lifecycle.core.ha.masternode.MasterNodeDetector] (EJB default - 1) New masternode detected: host1

      2015-07-26 11:24:04,732 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (EJB default - 1) New node event arrived: host1

      2015-07-26 11:24:04,733 INFO  [com.six_group.ssx.seni.request_portal.module.war.dispatcher.HttpClientCreatorBean] (EJB default - 1) Mapped node event: host1 to: http://edbe47631237:9580/seni-ref-server-web

      2015-07-26 11:24:52,452 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 62) MSC000001: Failed to start service jboss.infinispan.web.dist: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

              at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)

              at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850)

              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:599)

              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:554)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:424)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:438)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:118)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:109)

              at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:79)

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

              ... 4 more

      Caused by: org.infinispan.commons.CacheException: Bad response received from coordinator: CacheNotFoundResponse

              at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:525)

              at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:117)

              at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:110)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:497)

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

              ... 18 more

       

       

      2015-07-26 11:24:52,480 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 61) MSC000001: Failed to start service jboss.infinispan.web."request-portal-dispatcher-war-dispatcher-prod.war": org.jboss.msc.service.StartException in service jboss.infinispan.web."request-portal-dispatcher-war-dispatcher-prod.war": org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

              at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)

              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)

              at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:850)

              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:599)

              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:554)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:424)

              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:438)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:118)

              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:109)

              at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:79)

              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)

              ... 4 more

      Caused by: org.infinispan.commons.CacheException: Bad response received from coordinator: CacheNotFoundResponse

              at org.infinispan.topology.LocalTopologyManagerImpl.executeOnCoordinator(LocalTopologyManagerImpl.java:525)

              at org.infinispan.topology.LocalTopologyManagerImpl.join(LocalTopologyManagerImpl.java:117)

              at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:110)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:497)

              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)

              ... 18 more

       

      2015-07-26 11:24:52,493 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([

          ("subsystem" => "infinispan"),

          ("cache-container" => "web"),

          ("distributed-cache" => "dist")

      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.web.dist" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

          Caused by: org.infinispan.commons.CacheException: Bad response received from coordinator: CacheNotFoundResponse"}}

      2015-07-26 11:24:52,497 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "request-portal-dispatcher-war-dispatcher-prod.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.web.\"request-portal-dispatcher-war-dispatcher-prod.war\"" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.\"request-portal-dispatche:

      r-war-dispatcher-prod.war\": org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

          Caused by: org.infinispan.commons.CacheException: Bad response received from coordinator: CacheNotFoundResponse"}}

      2015-07-26 11:24:52,576 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "request-portal-dispatcher-war-dispatcher-prod.war" (runtime-name : "request-portal-dispatcher-war-dispatcher-prod.war")

      2015-07-26 11:24:52,577 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "java-ee-common-lifecycle-war-admin.war" (runtime-name : "java-ee-common-lifecycle-war-admin.war")

      2015-07-26 11:24:52,577 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "xadisk-1.2.1.rar" (runtime-name : "xadisk-1.2.1.rar")

      2015-07-26 11:24:52,577 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0010: Deployed "ojdbc6.jar" (runtime-name : "ojdbc6.jar")

      2015-07-26 11:24:52,589 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report

      WFLYCTL0186:   Services which failed to start:      service jboss.infinispan.web."request-portal-dispatcher-war-dispatcher-prod.war": org.jboss.msc.service.StartException in service jboss.infinispan.web."request-portal-dispatcher-war-dispatcher-prod.war": org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

            service jboss.infinispan.web.dist: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

       

       

      2015-07-26 11:24:52,781 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 9.0.1.Final (WildFly Core 1.0.1.Final) started (with errors) in 70544ms - Started 765 of 977 services (13 services failed or missing dependencies, 334 services are lazy, passive or on-demand)