5 Replies Latest reply on May 1, 2012 10:35 AM by jbertram

    Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11

    daokeeff

      Hello all,

      As part of my internal testing for our product against HornetQ 2.2.11 (embedded in JBoss AS 7.1.0.Final), I recently encountered a deadlock (see attached jstack, the deadlock is near the end, a snippet of which I've inlined below). Initially I thought it was a HornetQ problem, so I posted a query on the HornetQ forum, but the HornetQ devs think it is a JBoss AS 7 problem and suggested I should repost here (see here for my post on the HornetQ forum: https://community.jboss.org/message/733037#733037 ). Can any of you help to confirm whether it is indeed a JBoss AS 7 issue and create an appropriate defect?

       

      Found one Java-level deadlock:

      =============================

      "pool-5-thread-10":

        waiting to lock monitor 0x00002aaabc05d910 (object 0x00000007c69e5560, a org.jboss.remoting3.remote.InboundMessage),

        which is held by "Remoting "uxesplxbld11" read-1"

      "Remoting "uxesplxbld11" read-1":

        waiting to lock monitor 0x00002aaab06223a8 (object 0x00000007c69e5580, a org.xnio.streams.BufferPipeInputStream),

        which is held by "pool-5-thread-10"

       

      Java stack information for the threads listed above:

      ===================================================

      "pool-5-thread-10":

          at org.jboss.remoting3.remote.InboundMessage.openInboundWindow(InboundMessage.java:128)

          - waiting to lock <0x00000007c69e5560> (a org.jboss.remoting3.remote.InboundMessage)

          at org.jboss.remoting3.remote.InboundMessage$2.acknowledge(InboundMessage.java:63)

          at org.xnio.streams.BufferPipeInputStream.read(BufferPipeInputStream.java:140)

          - locked <0x00000007c69e5580> (a org.xnio.streams.BufferPipeInputStream)

          at org.jboss.remoting3.remote.InboundMessage$3.read(InboundMessage.java:82)

          at java.io.DataInputStream.readByte(DataInputStream.java:248)

          at org.jboss.naming.remote.protocol.v1.ReadUtil$1.read(ReadUtil.java:55)

          at java.io.InputStream.read(InputStream.java:163)

          at java.io.FilterInputStream.read(FilterInputStream.java:116)

          at java.io.FilterInputStream.read(FilterInputStream.java:90)

          at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:262)

          at org.jboss.marshalling.SimpleDataInput.readUnsignedByte(SimpleDataInput.java:224)

          at org.jboss.marshalling.river.RiverUnmarshaller.start(RiverUnmarshaller.java:1186)

          at org.jboss.naming.remote.protocol.v1.ReadUtil.prepareForUnMarshalling(ReadUtil.java:64)

          at org.jboss.naming.remote.protocol.v1.Protocol$1.handleServerMessage(Protocol.java:109)

          at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.run(RemoteNamingServerV1.java:70)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      "Remoting "uxesplxbld11" read-1":

          at org.xnio.streams.BufferPipeInputStream.pushEof(BufferPipeInputStream.java:112)

          - waiting to lock <0x00000007c69e5580> (a org.xnio.streams.BufferPipeInputStream)

          at org.jboss.remoting3.remote.InboundMessage.cancel(InboundMessage.java:186)

          - locked <0x00000007c69e5560> (a org.jboss.remoting3.remote.InboundMessage)

          at org.jboss.remoting3.remote.RemoteConnectionChannel.handleMessageData(RemoteConnectionChannel.java:424)

          at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:234)

          at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)

          at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)

          at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)

          at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)

          at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)

          at org.xnio.nio.NioHandle.run(NioHandle.java:90)

          at org.xnio.nio.WorkerThread.run(WorkerThread.java:184)

       

      Found 1 deadlock.

       

      Thanks,

      Dan.

        • 1. Re: Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11
          jaikiran

          There have been many deadlock/hangs fixed in JBoss Remoting project after 7.1.0.Final of AS7 was released. Infact, one was fixed just yesterday. Try out the latest nightly builds of AS7 https://community.jboss.org/thread/167590 where such issues have been addressed.

          • 2. Re: Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11
            bjs

            I'm also interested in this fix... but (funnily enough) investing considerable time testing out the latest (non stable) build 'just in case' this extremely serious server deadlock happens to have been fixed isn't really a practical option! Especially as even if it is fixed we can't recommend anyone to use a non-stable release in production.

             

            So it would really help us if you could confirm whether this specific deadlock has been fixed or not.

             

            If it has definitely been fixed already, we can stop worry about it and wait for the next stable release including the fix (and tell our customers to avoid JBoss 7.1.0.Final in the meantime), but if it hasn't been included in other recent fixes, presumably it's important not only to us, but also for the stability of JBoss itself that it gets filed as a defect so someone can look into fixing it? I had a look at the change list at https://ci.jboss.org/jenkins/job/JBoss-AS-7.x-latest/changes and didn't see any changes related to deadlocks at all, and certainly not a list of the specific deadlocks that may have already been closed.

            • 3. Re: Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11
              jbertram

              To be clear, this problem is not with HornetQ itself.  It is with JBoss Remoting 3 or XNIO or both.

               

              HornetQ doesn't use JBoss Remoting 3 or XNIO, but JNDI lookups do.  You could probably work-around the issue temporarily by using the HornetQ API and not doing JNDI look-ups.  For example:

               

              import org.hornetq.api.core.TransportConfiguration;
              import org.hornetq.api.jms.HornetQJMSClient;
              import org.hornetq.api.jms.JMSFactoryType;
              import org.hornetq.core.remoting.impl.netty.TransportConstants;
              ...
              Destination destination = HornetQJMSClient.createQueue("testQueue");  //this is NOT the JNDI name, it is the HornetQ name
              HashMap<String, Object> connectionParams = new HashMap<String, Object>();
              connectionParams.put(TransportConstants.HOST_PROP_NAME, "localhost");
              TransportConfiguration transportConfiguration = new TransportConfiguration("org.hornetq.core.remoting.impl.netty.NettyConnectorFactory", connectionParams);
              ConnectionFactory connectionFactory = (ConnectionFactory) HornetQJMSClient.createConnectionFactoryWithoutHA(JMSFactoryType.CF, transportConfiguration);
              
              1 of 1 people found this helpful
              • 4. Re: Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11
                daokeeff

                Hey Justin,

                Thanks for replying, that was helpful. Unfortunately we do need to support JNDI lookups, so we can't use the workaround you suggested.

                Dan

                • 5. Re: Deadlock in JBoss AS 7.1.0.Final / HornetQ 2.2.11
                  jbertram

                  Couple of points:

                  1. I believe the deadlock was fixed here.  This fix was made after 7.1.1.Final was released so it will only be available in a nightly build (that Jaikiran suggested).
                  2. If you need real support (not just forum support that might disappear when developers get busy) then I recommend a contract with Red Hat.
                  3. JBoss Remoting 3 and XNIO are both "3rd-party" projects which JBoss AS consumes.  Changes within these projects won't necessarily show up in a change-log for the AS itself.