7 Replies Latest reply on May 26, 2006 4:15 PM by brian.stansberry

    Repeating errors in log with session replication

    jvandusen

      We're running JBoss 3.2.6 in a clustered environment with two nodes. First issue is that I see an exception in the log when one of the nodes starts, but no problem with the other node. The log looks like this:

      2006-05-17 10:47:48,911 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Number of cluster members: 2
      2006-05-17 10:47:48,911 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Other members: 1
      2006-05-17 10:47:48,911 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Fetching state (will wait for 60000 milliseconds):
      2006-05-17 10:47:49,018 INFO [org.jboss.ha.jndi.HANamingService] Listening on /0.0.0.0:1100
      2006-05-17 10:47:49,023 WARN [org.jboss.ha.jndi.HANamingService] Failed to start AutomaticDiscovery
      java.net.SocketException: No such device
      at java.net.PlainDatagramSocketImpl.join(Native Method)
      at java.net.PlainDatagramSocketImpl.join(PlainDatagramSocketImpl.java:134)
      at java.net.MulticastSocket.joinGroup(MulticastSocket.java:274)
      at org.jboss.ha.jndi.DetachedHANamingService$AutomaticDiscovery.start(DetachedHANamingService.java:440)
      at org.jboss.ha.jndi.DetachedHANamingService.startService(DetachedHANamingService.java:243)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
      at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884)
      at $Proxy18.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:414)
      at org.jboss.system.ServiceController.start(ServiceController.java:436)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604)
      at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy8.deploy(Unknown Source)
      at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:304)
      at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:478)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:201)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:274)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
      at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:414)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:588)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy5.deploy(Unknown Source)
      at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407)
      at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311)
      at org.jboss.Main.boot(Main.java:145)
      at org.jboss.Main$1.run(Main.java:399)
      at java.lang.Thread.run(Thread.java:534)

      At the start of the log excerpt I pasted above, you can see that it appears to have recognized the other node, so I'm not sure if the exception is relevant, but it does concern me. The configuration appears identical on both servers, so I'm not sure why this happens on one and not the other.

      The other issue is an exception that appears about once every minute in the logs on both nodes. Here is the exception:

      2006-05-17 12:55:34,237 ERROR [org.jboss.web.tomcat.tc5.session.JBossCacheManager] processExpires: failed with exception: java.lang.NullPointerException
      2006-05-17 12:55:34,237 INFO [STDOUT] java.lang.NullPointerException
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService.getAttributes(JBossCacheService.java:236)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.AttributeBasedClusteredSession.populateAttributes(AttributeBasedClusteredSession.java:101)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.AttributeBasedClusteredSession.initAfterLoad(AttributeBasedClusteredSession.java:93)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.loadSession(JBossCacheManager.java:364)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.findSessions(JBossCacheManager.java:298)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processExpires(JBossCacheManager.java:431)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossManager.backgroundProcess(JBossManager.java:644)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:4647)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1619)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1628)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1628)
      2006-05-17 12:55:34,238 INFO [STDOUT] at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1608)
      2006-05-17 12:55:34,238 INFO [STDOUT] at java.lang.Thread.run(Thread.java:534)

      Any thoughts on this? Upgrading JBoss is not an immediate option for us, although we will consider it in the hopefully not-too-distant future.

        • 1. Re: Repeating errors in log with session replication
          jvandusen

          I should also mention that the repeating NullPointerException occurs in both our test and production environment, but the SocketException only occurs in production. The two environments are supposedly identical. Also, the NullPointerException occurs once every minute most of the time, but occasionally it just decides that it wants to take a break for a day or so. At this moment it is happening in production, but not in test, although it was happening in test yesterday, and nothing was changed in test to make it stop. I restarted JBoss a few times last night, and on most of those restarts the error continued, but then suddenly it just stopped. History shows that it will be back, though.

          • 2. Re: Repeating errors in log with session replication
            jvandusen

            On further research I've learned that the NullPointerException is caused by a bug in JBoss 3.2.6 which is fixed in 3.2.7 and above. I've copied the one-line bug-fix from 3.2.7 into the 3.2.6 code and recompiled, and we're testing that in our test environment.

            The SocketException is still a mystery.

            • 3. Re: Repeating errors in log with session replication
              brian.stansberry

              It's trying and failing to establish a multicast socket. By default this is on address 230.0.0.4; it's configured in the HA-JNDI section of cluster-service.xml. Is multicast working correctly on that server?

              • 4. Re: Repeating errors in log with session replication
                jvandusen

                At the moment I'm not able to test multicast on the production server, where this problem is occurring, but it appears to be working in the test environment. The cluster-service.xml is the same in prod as it is in test, which is basically just everything left at the default settings.

                I'm just figuring out this whole clustering/session replication business, and the documentation I've found so far hasn't said much about tweaking that file, so I've left it untouched. Should I be adjusting anything in there?

                • 5. Re: Repeating errors in log with session replication
                  brian.stansberry

                  You shouldn't need to tweak the cluster-service.xml file. The other odd thing is if you are pretty much using the default file, other clustering services are using multicast, so if multicast was broken on that machine, you'd be getting other errors.

                  Can you paste the HA-JNDI portion of cluster-service.xml from the failing server? Please use the

                  button to wrap it or it won't come through on the forum.

                  • 6. Re: Repeating errors in log with session replication
                    jvandusen

                    Here's the snippet.....

                    <!-- ==================================================================== -->
                     <!-- HA JNDI -->
                     <!-- ==================================================================== -->
                    
                     <mbean code="org.jboss.ha.jndi.HANamingService"
                     name="jboss:service=HAJNDI">
                     <depends>jboss:service=DefaultPartition</depends>
                     <!-- Name of the partition to which the service is linked -->
                     <attribute name="PartitionName">DefaultPartition</attribute>
                     <!-- bind address of HA JNDI RMI endpoint -->
                     <attribute name="BindAddress">${jboss.bind.address}</attribute>
                     <!-- RmiPort to be used by the HA-JNDI service
                     once bound. 0 => auto. -->
                     <attribute name="RmiPort">0</attribute>
                     <!-- Port on which the HA-JNDI stub is made available -->
                     <attribute name="Port">1100</attribute>
                     <!-- Backlog to be used for client-server RMI
                     invocations during JNDI queries -->
                     <attribute name="Backlog">50</attribute>
                    
                     <!-- Multicast Address and Group used for auto-discovery -->
                     <attribute name="AutoDiscoveryAddress">230.0.0.4</attribute>
                     <attribute name="AutoDiscoveryGroup">1102</attribute>
                    
                     <!-- IP Address to which should be bound: the Port, the RmiPort and
                     the AutoDiscovery multicast socket. -->
                     <!-- Client socket factory to be used for client-server
                     RMI invocations during JNDI queries -->
                     <!--attribute name="ClientSocketFactory">custom</attribute-->
                     <!-- Server socket factory to be used for client-server
                     RMI invocations during JNDI queries -->
                     <!--attribute name="ServerSocketFactory">custom</attribute-->
                     </mbean>
                    
                     <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
                     name="jboss:service=invoker,type=jrmpha">
                     <attribute name="ServerAddress">${jboss.bind.address}</attribute>
                     <!--
                     <attribute name="RMIObjectPort">0</attribute>
                     <attribute name="RMIClientSocketFactory">custom</attribute>
                     <attribute name="RMIServerSocketFactory">custom</attribute>
                     -->
                     </mbean>


                    • 7. Re: Repeating errors in log with session replication
                      brian.stansberry

                      Sorry for the slow response. Your file looks fine; at this poing I've no good idea what the problem could be.