1 2 Previous Next 23 Replies Latest reply on May 9, 2012 5:02 AM by ataylor

    Not able to add and fetch 100 000 message to JMS Queue. Exception ->Unable to get managed connection for java:/JmsXA

    varsha.g

      I am using jboss7.1.2.Final-SNAPSHOT build #824 from https://ci.jboss.org/hudson/view/All/job/JBoss-AS-7.x-latest/  and HornetQ Server version 2.2.13.Final.

      These are the settings in my standalone/configuration/standalone-full.xml

       

                    <address-settings>

                         <address-setting match="#">

       

                             <dead-letter-address>jms.queue.DLQ</dead-letter-address>

       

                             <expiry-address>jms.queue.ExpiryQueue</expiry-address>

       

                             <redelivery-delay>0</redelivery-delay>

       

                             <max-size-bytes>524288000</max-size-bytes>

       

                             <page-size-bytes>52428800</page-size-bytes>

       

                             <address-full-policy>PAGE</address-full-policy>       <message-counter-history-day-limit>10</message-counter-history-day-limit>

                         </address-setting>

                 </address-settings>

       

       

      <subsystem xmlns="urn:jboss:domain:messaging:1.2">

                 <hornetq-server>

                     <persistence-enabled>true</persistence-enabled>

                     <journal-type>ASYNCIO</journal-type>

                     <journal-file-size>10485760</journal-file-size>

                     <journal-min-files>2</journal-min-files>

                     <journal-max-io>500</journal-max-io>

       

      </hornetq-server>

      </subsystem>

       

       

      When I'm Testing with 100000 records; The Server hangs and starts again.  Everytime it hangs its a different error or warning, Following are the different errors/warnings :-

       

        -- 13:02:14,483 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-10 (HornetQ-client-global-threads-1868041274)) XA operation failed Cannot start, session is already doing work in a transaction XidImpl (557588165 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.68.-87.52.16.116.-56.-98.-29.79.-115.93.-68.0.0.34.47.0.0.0.0.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.68.-87.52.16.116.-56.-98.-29.79.-115.93.-68.0.0.34.46.49 code:-6 

       

        -- 13:04:52,783 ERROR [stderr] (Thread-10 (HornetQ-client-global-threads-1868041274))     at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636)

       

        -- 12:52:56,421 WARN  [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-10 (HornetQ-client-global-threads-1868041274)) Unable to call after delivery: javax.resource.spi.LocalTransactionException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.

       

        -- 12:53:26,474 WARN  [com.arjuna.ats.jta] (Thread-10 (HornetQ-client-global-threads-1868041274)) ARJUNA016086: TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff44a93410:74c89ee3:4f8d5dbc:222e, node_name=1, branch_uid=0:ffff44a93410:74c89ee3:4f8d5dbc:222f, subordinatenodename=null, eis_name=unknown eis name > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException

       

      -- 13:05:12,682 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

      13:05:12,685 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Complete Status:

          Managed Threads: 3

          Active Threads: 0

          Active Tasks:

          Pending Tasks:

              com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask@6a7ad0a8

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2518fc35

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@b560e9a

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ed8a646

      Pool thread stack traces:

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

       

      --  13:05:12,682 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

      13:05:12,685 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Complete Status:

          Managed Threads: 3

          Active Threads: 0

          Active Tasks:

          Pending Tasks:

              com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask@6a7ad0a8

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2518fc35

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@b560e9a

              com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ed8a646

      Pool thread stack traces:

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

          Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]

              java.lang.Object.wait(Native Method)

              com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

       

      Please Let me know the solution for the above

      thanks

        • 1. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
          ataylor

          maybe you are running out of memory, maye your code is wrong, you havent really given us much to go on

          • 2. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
            clebert.suconic

            This seems related to the duplicated XA records on the Transaction Manager. I just pushed a new release to AS7 which will possibly fix this.

             

            https://github.com/jbossas/jboss-as/pull/2061

             

             

            The pull request is still open as I write this due to tests.. if there are any failures we will fix and push another change.

             

            Are you creating new Connection Factories?

            • 3. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
              varsha.g

              I have a test servlet which sends messages to the JMS queue. And I have MDB which fetches the messages from teh queue.

              I have attached my TestServlet and MDB code. I have also attached my standalone-full.xml.

               

              These are the HornetQ settings for the file standalone-full.xml:

               

                  <subsystem xmlns="urn:jboss:domain:messaging:1.2">

                         <hornetq-server>

                             <persistence-enabled>true</persistence-enabled>

                             <journal-type>ASYNCIO</journal-type>

                             <journal-file-size>10485760</journal-file-size>

                             <journal-min-files>2</journal-min-files>

                         <journal-max-io>500</journal-max-io>

               

                  </hornetq-server>

                  </subsystem>

               

                   <address-settings>

                                 <address-setting match="#">

               

                                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>

               

                                     <expiry-address>jms.queue.ExpiryQueue</expiry-address>

               

                                     <redelivery-delay>0</redelivery-delay>

               

                                     <max-size-bytes>104857600</max-size-bytes>

               

                                     <page-size-bytes>10485760</page-size-bytes>

               

                                     <address-full-policy>PAGE</address-full-policy>       <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                 </address-setting>

                         </address-settings>

               

               

              When I test the JMS queue using JMeter by sending 100 000 messages to queue it throws the following exception on 2gb RAM :-

               

                 ---    19:30:23,747 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException

                  at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]

               

               

                 ---  19:30:23,430 ERROR [stderr] javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA

               

               

              But I tested the same on 4gb RAM. It works fine without any errors.

               

              As I'm writing the messages to the page, Why is there a RAM issue?

               

              Please let me know what can be the issue?

               

              Thanks.

              • 4. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                ataylor

                first thing i notice is that your servlet is connecting to localhost and you are using java:/JmsXA which should only be InVM.

                 

                Also the first stacktrace is not HornetQ.

                 

                regarding paging, if you are running out of memory you should lower it.

                1 of 1 people found this helpful
                • 5. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                  varsha.g

                  Thanks for the reply.

                   

                  Please can you elaborate on  :

                   

                     first thing i notice is that your servlet is connecting to localhost and you are using java:/JmsXA which should only be InVM.

                   

                   

                  And regarding paging,

                  I minimized as follows: max-size-bytes = 5242880  &  page-size-bytes = 524288    But I'm still getting the error.

                  • 6. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                    varsha.g

                    As per our chat on IRC, I set the following settings to producer in my code :

                     

                      - producer.setDeliveryMode(DeliveryMode.PERSISTENT)

                     

                    But I was still getting the error.

                     

                    As the following link states :

                    http://docs.oracle.com/javaee/1.3/api/javax/jms/MessageProducer.html

                     

                    Sets the producer's default delivery mode.

                    Delivery mode is set to PERSISTENT by default.

                    • 7. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                      ataylor

                      you will have to give us more infom like i say, stack traces and a test taht we can run if you can. if it works with 4GB then its a memory issue and you need to configure accordingly.

                      • 8. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                        varsha.g

                        Hi Andy

                         

                                I have attached the TestServlet and MDB on above my post.

                         

                                   In jboss7.1.2 there is several ConnectionFactories available in JNDI: 

                        • java:/JmsXA
                        • java:/ConnectionFactory

                         

                                       Which one is best to use if i want to implement the point-to-point Asynchronous Messaging queue. I've the MDB code on my same Application.

                         

                        Should I need to add the following jms-jca-provider

                         

                        <jms-jca-provider name="HornetQ"

                                                  adapter="jms-ra.rar"

                                                  connection-factory="java:/ConnectionFactory"

                                                  jndi-context-factory="org.jnp.interfaces.NamingContextFactory"

                                                  transacted="false">

                                    <property name="jcaActivationMapper" value="org.jboss.soa.esb.listeners.jca.JBossActivationMapper"/>

                                    <jms-bus busid="esb_jms_gateway">

                                        <jms-message-filter dest-name="queue/in.gw"

                                                            dest-type="QUEUE" transacted="false"/>

                                    </jms-bus>

                                    <jms-bus busid="esb_jms_master_bus">

                                        <jms-message-filter dest-name="queue/in"

                                                            dest-type="QUEUE" transacted="false"/>

                                    </jms-bus>

                                    <activation-config>

                                        <property name="useJNDI" value="true" />

                                        <property name="useLocalTx" value="true"/>

                                        <property name="dLQMaxResent" value="1"/>

                                    </activation-config>

                           </jms-jca-provider>

                         

                        If yes then what is the configuration setting for the jboss 7.1.2 ? and what is use of this? is thr any proper documentation for this?

                         

                        I've referred the following links --->

                         

                        http://docs.oracle.com/javaee/6/tutorial/doc/bncdq.html

                         

                        http://docs.oracle.com/javaee/1.3/api/javax/jms/MessageProducer.html

                         

                        https://community.jboss.org/thread/195612?tstart=0

                         

                         

                         

                        Thanks a lot.

                        • 9. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                          ataylor

                          Like i say, the jca connection factory, java:/JmsXA, is for invm conncetions, the other is a pure hornetq conenction factory that is used for external clients.

                           

                          refarding your MDB an dservlet, its not really a test, i need something i can run and reproduce your issue with, but like i saifd before if it works with 4GB then its a memory configuration issue and you need to tune your app to deal with it.

                          • 10. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                            varsha.g

                            Hi,

                                I've attached the my test app to test the MDB for 100 000 Messages.

                            • 11. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                              varsha.g

                              Hi Andy

                               

                                 Thanks for response.

                               

                               

                               

                                 I've attached the war file. and .jmx file which call the testServlet 100 000. And at a time it will add the one message to queue.

                               

                                 And zip of jmeter and  jmeter script  to excute.

                               

                              Steps to test the war.

                               

                               

                               

                              1) Download the Test.war file from attachment.

                               

                              2) Deploy the war on jboss7.1.2.

                               

                              3) unzip the jmeter.

                               

                              4) Download the test.jmx file from attachment.

                               

                              5) To excute below command firstly select the bin directory of jmeter

                               

                                   ./jmeter -n -t <path for jmx file>

                               

                              Example:->

                               

                              linux-t0cw:/home/jmeter/apache-jmeter-2.6/bin # ./jmeter -n -t <path for jmx file>

                               

                              On 2GB It throws the following exception under the server log of jboss AS

                               

                               

                               

                                              19:30:23,430 ERROR [stderr] javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA

                               

                                  And also I'm not able to add the whole  100 000 messages.

                               

                              is this  configuration issues?

                               

                              how to resolve this ?

                              • 12. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                                varsha.g

                                Hi Andy

                                 

                                  Have you run the above Application. and able reproduce this issue ?

                                         Is this the configuration issues?

                                 

                                Exactly where I'm going wrong?

                                • 13. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                                  ataylor

                                  no, I have been too busy, will try and look at it next week. just out of curiousity, how are you configuting the memory when you start the server?

                                  • 14. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
                                    varsha.g

                                    Thanks For reply

                                                              Okay. Whenever you have time please have look on it.

                                                       I've not did any extra memory configuration.

                                             

                                     

                                    This are the memory configuration when we start the server

                                        

                                    JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

                                     

                                    Thanks in advance.

                                    1 2 Previous Next