1 Reply Latest reply on Nov 21, 2011 4:16 PM by ron_sigal

    Messaging server lock up issues

    vbs_jboss_accnt
    vbs_jboss_accnt Aug 31, 2011 8:47 PM

      We are having issues with a JBOSS Application server running with the following versions

      jboss messaging 1.4.6

      jboss remoting 2.5.2

      primarily being used for messaging purposes.

       

      The server often locks up  and the only way to get the clients to connect again is to restart the server.

       

      Looking at the stack trace ,multiple workerthreads are waiting on this following lock

       

       

      - waiting on <0x1990f6eb> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)

      java.lang.Object.wait(Native Method)

      java.lang.Object.wait(Object.java:485)

      EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)

      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2210)

      org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)

      org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)

      org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)

      org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)

      org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

      org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)

      sun.reflect.GeneratedMethodAccessor233.invoke(Unknown Source)

      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      java.lang.reflect.Method.invoke(Method.java:597)

      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)

      org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)

      org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)

      org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)

      org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)

      org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:897)

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:768)

      - locked <0x69b3f1c0> (a org.jboss.remoting.transport.socket.ServerThread)

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:721)

      org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:548)

      org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

       

       

      Attached is the settings from the remoting-bisocket-service.xml file

       

       

                     <!--              

                     <attribute name="writeTimeout" isParam="true">30000</attribute>

                     -->

       

                     <!-- End immutable parameters -->

                    

                     <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>

                    

                     <!-- Periodicity of client pings. Server window by default is twice this figure -->                              

                     <attribute name="clientLeasePeriod" isParam="true">10000</attribute>

                     <attribute name="validatorPingPeriod" isParam="true">10000</attribute>

                     <attribute name="validatorPingTimeout" isParam="true">5000</attribute>

       

                     <attribute name="failureDisconnectTimeout" isParam="true">0</attribute>

                     <attribute name="callbackErrorsAllowed">1</attribute>

                     <attribute name="registerCallbackListener">false</attribute>

                     <attribute name="useClientConnectionIdentity" isParam="true">true</attribute>

                           

                      <attribute name="timeout" isParam="true">0</attribute>

       

                     <!-- Max Number of connections in client pool. This should be significantly higher than

                          the max number of sessions/consumers you expect -->

                     <attribute name="JBM_clientMaxPoolSize" isParam="true">1000</attribute>

                    

                     <!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->

                     <attribute name="callbackTimeout">10000</attribute>

                    

       

       

      I am trying to find out if its a configuration issue or a code issue on our side ?

      Is it possible that one particular client is having problems and that in turn causes the server to lock up ?

       

       

      Please help.

      • 2007 Views
      • Tags:
        • 1. Re: Messaging server lock up issues
          ron_sigal
          ron_sigal Nov 21, 2011 4:16 PM (in response to vbs_jboss_accnt)

          Hi Vidhya,

           

          Sorry for the delay.  Are you still having this problem?

           

          Can you see in the log which thread is holding the lock at <0x1990f6eb>?  Note that the lock exists in the JBossMessaging code.  While it is possible that there is some relevent interaction between Remoting and JBossMessaging, it's more likely that the issue is confined to JBossMessaging.

           

          Have you posted the question to the JBossMessaging user forum?

           

          -Ron

            • Actions