1 2 Previous Next 23 Replies Latest reply: May 9, 2012 5:02 AM by Andy Taylor RSS

Not able to add and fetch 100 000 message to JMS Queue. Exception ->Unable to get managed connection for java:/JmsXA

varsha g Newbie

I am using jboss7.1.2.Final-SNAPSHOT build #824 from https://ci.jboss.org/hudson/view/All/job/JBoss-AS-7.x-latest/  and HornetQ Server version 2.2.13.Final.

These are the settings in my standalone/configuration/standalone-full.xml

 

              <address-settings>

                   <address-setting match="#">

 

                       <dead-letter-address>jms.queue.DLQ</dead-letter-address>

 

                       <expiry-address>jms.queue.ExpiryQueue</expiry-address>

 

                       <redelivery-delay>0</redelivery-delay>

 

                       <max-size-bytes>524288000</max-size-bytes>

 

                       <page-size-bytes>52428800</page-size-bytes>

 

                       <address-full-policy>PAGE</address-full-policy>       <message-counter-history-day-limit>10</message-counter-history-day-limit>

                   </address-setting>

           </address-settings>

 

 

<subsystem xmlns="urn:jboss:domain:messaging:1.2">

           <hornetq-server>

               <persistence-enabled>true</persistence-enabled>

               <journal-type>ASYNCIO</journal-type>

               <journal-file-size>10485760</journal-file-size>

               <journal-min-files>2</journal-min-files>

               <journal-max-io>500</journal-max-io>

 

</hornetq-server>

</subsystem>

 

 

When I'm Testing with 100000 records; The Server hangs and starts again.  Everytime it hangs its a different error or warning, Following are the different errors/warnings :-

 

  -- 13:02:14,483 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-10 (HornetQ-client-global-threads-1868041274)) XA operation failed Cannot start, session is already doing work in a transaction XidImpl (557588165 bq:0.0.0.0.0.0.0.0.0.0.-1.-1.68.-87.52.16.116.-56.-98.-29.79.-115.93.-68.0.0.34.47.0.0.0.0.0.0.0.0 formatID:131077 gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.68.-87.52.16.116.-56.-98.-29.79.-115.93.-68.0.0.34.46.49 code:-6 

 

  -- 13:04:52,783 ERROR [stderr] (Thread-10 (HornetQ-client-global-threads-1868041274))     at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636)

 

  -- 12:52:56,421 WARN  [org.hornetq.ra.inflow.HornetQMessageHandler] (Thread-10 (HornetQ-client-global-threads-1868041274)) Unable to call after delivery: javax.resource.spi.LocalTransactionException: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.

 

  -- 12:53:26,474 WARN  [com.arjuna.ats.jta] (Thread-10 (HornetQ-client-global-threads-1868041274)) ARJUNA016086: TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff44a93410:74c89ee3:4f8d5dbc:222e, node_name=1, branch_uid=0:ffff44a93410:74c89ee3:4f8d5dbc:222f, subordinatenodename=null, eis_name=unknown eis name > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException

 

-- 13:05:12,682 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

13:05:12,685 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Complete Status:

    Managed Threads: 3

    Active Threads: 0

    Active Tasks:

    Pending Tasks:

        com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask@6a7ad0a8

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2518fc35

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@b560e9a

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ed8a646

Pool thread stack traces:

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

 

--  13:05:12,682 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

13:05:12,685 WARN  [com.mchange.v2.async.ThreadPoolAsynchronousRunner] (Timer-2) com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@5a32f187 -- APPARENT DEADLOCK!!! Complete Status:

    Managed Threads: 3

    Active Threads: 0

    Active Tasks:

    Pending Tasks:

        com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StmtAcquireTask@6a7ad0a8

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2518fc35

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@b560e9a

        com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ed8a646

Pool thread stack traces:

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

    Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]

        java.lang.Object.wait(Native Method)

        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

 

Please Let me know the solution for the above

thanks

  • 1. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Andy Taylor Master

    maybe you are running out of memory, maye your code is wrong, you havent really given us much to go on

  • 2. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Clebert Suconic Master

    This seems related to the duplicated XA records on the Transaction Manager. I just pushed a new release to AS7 which will possibly fix this.

     

    https://github.com/jbossas/jboss-as/pull/2061

     

     

    The pull request is still open as I write this due to tests.. if there are any failures we will fix and push another change.

     

    Are you creating new Connection Factories?

  • 3. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    I have a test servlet which sends messages to the JMS queue. And I have MDB which fetches the messages from teh queue.

    I have attached my TestServlet and MDB code. I have also attached my standalone-full.xml.

     

    These are the HornetQ settings for the file standalone-full.xml:

     

        <subsystem xmlns="urn:jboss:domain:messaging:1.2">

               <hornetq-server>

                   <persistence-enabled>true</persistence-enabled>

                   <journal-type>ASYNCIO</journal-type>

                   <journal-file-size>10485760</journal-file-size>

                   <journal-min-files>2</journal-min-files>

               <journal-max-io>500</journal-max-io>

     

        </hornetq-server>

        </subsystem>

     

         <address-settings>

                       <address-setting match="#">

     

                           <dead-letter-address>jms.queue.DLQ</dead-letter-address>

     

                           <expiry-address>jms.queue.ExpiryQueue</expiry-address>

     

                           <redelivery-delay>0</redelivery-delay>

     

                           <max-size-bytes>104857600</max-size-bytes>

     

                           <page-size-bytes>10485760</page-size-bytes>

     

                           <address-full-policy>PAGE</address-full-policy>       <message-counter-history-day-limit>10</message-counter-history-day-limit>

                       </address-setting>

               </address-settings>

     

     

    When I test the JMS queue using JMeter by sending 100 000 messages to queue it throws the following exception on 2gb RAM :-

     

       ---    19:30:23,747 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException

        at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]

     

     

       ---  19:30:23,430 ERROR [stderr] javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA

     

     

    But I tested the same on 4gb RAM. It works fine without any errors.

     

    As I'm writing the messages to the page, Why is there a RAM issue?

     

    Please let me know what can be the issue?

     

    Thanks.

  • 4. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Andy Taylor Master

    first thing i notice is that your servlet is connecting to localhost and you are using java:/JmsXA which should only be InVM.

     

    Also the first stacktrace is not HornetQ.

     

    regarding paging, if you are running out of memory you should lower it.

  • 5. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Thanks for the reply.

     

    Please can you elaborate on  :

     

       first thing i notice is that your servlet is connecting to localhost and you are using java:/JmsXA which should only be InVM.

     

     

    And regarding paging,

    I minimized as follows: max-size-bytes = 5242880  &  page-size-bytes = 524288    But I'm still getting the error.

  • 6. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    As per our chat on IRC, I set the following settings to producer in my code :

     

      - producer.setDeliveryMode(DeliveryMode.PERSISTENT)

     

    But I was still getting the error.

     

    As the following link states :

    http://docs.oracle.com/javaee/1.3/api/javax/jms/MessageProducer.html

     

    Sets the producer's default delivery mode.

    Delivery mode is set to PERSISTENT by default.

  • 7. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Andy Taylor Master

    you will have to give us more infom like i say, stack traces and a test taht we can run if you can. if it works with 4GB then its a memory issue and you need to configure accordingly.

  • 8. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Hi Andy

     

            I have attached the TestServlet and MDB on above my post.

     

               In jboss7.1.2 there is several ConnectionFactories available in JNDI: 

    • java:/JmsXA
    • java:/ConnectionFactory

     

                   Which one is best to use if i want to implement the point-to-point Asynchronous Messaging queue. I've the MDB code on my same Application.

     

    Should I need to add the following jms-jca-provider

     

    <jms-jca-provider name="HornetQ"

                              adapter="jms-ra.rar"

                              connection-factory="java:/ConnectionFactory"

                              jndi-context-factory="org.jnp.interfaces.NamingContextFactory"

                              transacted="false">

                <property name="jcaActivationMapper" value="org.jboss.soa.esb.listeners.jca.JBossActivationMapper"/>

                <jms-bus busid="esb_jms_gateway">

                    <jms-message-filter dest-name="queue/in.gw"

                                        dest-type="QUEUE" transacted="false"/>

                </jms-bus>

                <jms-bus busid="esb_jms_master_bus">

                    <jms-message-filter dest-name="queue/in"

                                        dest-type="QUEUE" transacted="false"/>

                </jms-bus>

                <activation-config>

                    <property name="useJNDI" value="true" />

                    <property name="useLocalTx" value="true"/>

                    <property name="dLQMaxResent" value="1"/>

                </activation-config>

       </jms-jca-provider>

     

    If yes then what is the configuration setting for the jboss 7.1.2 ? and what is use of this? is thr any proper documentation for this?

     

    I've referred the following links --->

     

    http://docs.oracle.com/javaee/6/tutorial/doc/bncdq.html

     

    http://docs.oracle.com/javaee/1.3/api/javax/jms/MessageProducer.html

     

    https://community.jboss.org/thread/195612?tstart=0

     

     

     

    Thanks a lot.

  • 9. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Andy Taylor Master

    Like i say, the jca connection factory, java:/JmsXA, is for invm conncetions, the other is a pure hornetq conenction factory that is used for external clients.

     

    refarding your MDB an dservlet, its not really a test, i need something i can run and reproduce your issue with, but like i saifd before if it works with 4GB then its a memory configuration issue and you need to tune your app to deal with it.

  • 10. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Hi,

        I've attached the my test app to test the MDB for 100 000 Messages.

  • 11. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Hi Andy

     

       Thanks for response.

     

     

     

       I've attached the war file. and .jmx file which call the testServlet 100 000. And at a time it will add the one message to queue.

     

       And zip of jmeter and  jmeter script  to excute.

     

    Steps to test the war.

     

     

     

    1) Download the Test.war file from attachment.

     

    2) Deploy the war on jboss7.1.2.

     

    3) unzip the jmeter.

     

    4) Download the test.jmx file from attachment.

     

    5) To excute below command firstly select the bin directory of jmeter

     

         ./jmeter -n -t <path for jmx file>

     

    Example:->

     

    linux-t0cw:/home/jmeter/apache-jmeter-2.6/bin # ./jmeter -n -t <path for jmx file>

     

    On 2GB It throws the following exception under the server log of jboss AS

     

     

     

                    19:30:23,430 ERROR [stderr] javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA

     

        And also I'm not able to add the whole  100 000 messages.

     

    is this  configuration issues?

     

    how to resolve this ?

  • 12. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Hi Andy

     

      Have you run the above Application. and able reproduce this issue ?

             Is this the configuration issues?

     

    Exactly where I'm going wrong?

  • 13. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    Andy Taylor Master

    no, I have been too busy, will try and look at it next week. just out of curiousity, how are you configuting the memory when you start the server?

  • 14. Re: APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
    varsha g Newbie

    Thanks For reply

                              Okay. Whenever you have time please have look on it.

                       I've not did any extra memory configuration.

             

     

    This are the memory configuration when we start the server

        

    JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

     

    Thanks in advance.

1 2 Previous Next