9 Replies Latest reply on May 2, 2012 8:49 AM by meenarc

    HornetQ backup server -Error message "Cursor 0 had already been created"

    meenarc

      We have a configured a Hornetq HA setup (CentOS 5, using NFS mount with HQ_2_2_10_EAP_GA_BUILD2,Java NIO) using a static cluster configuration. This setup worked for  a while. When I killed the live server recently, the backup server didn't get activated and I get the following error in the backup server log.

       

       

      16:38:00,635 WARN  [loggerI18N] [com.arjuna.ats.internal.jta.recovery.xarecovery1] Local XARecoveryModule.xaRecovery  got XA exception javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery, XAException.XAER_RMERR

      16:38:00,636 WARN  [HornetQXAResourceWrapper] Can't connect to any hornetq server on recovery [XARecoveryConfig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false], username=null, password=null]]

      16:38:02,024 WARN  [RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]

      16:38:02,813 SEVERE [HornetQServerImpl] Failure in initialisation

      java.lang.IllegalStateException: Cursor 0 had already been created

                at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.createSubscription(PageCursorProviderImpl.java:99)

                at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1564)

                at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1444)

                at org.hornetq.core.server.impl.HornetQServerImpl.access$1200(HornetQServerImpl.java:137)

                at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation.run(HornetQServerImpl.java:1950)

                at java.lang.Thread.run(Thread.java:619)

      16:40:10,665 WARN  [HornetQXAResourceWrapper] Can't connect to any hornetq server on recovery [XARecoveryConfig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false], username=null, password=null]]

      16:40:10,665 WARN  [loggerI18N] [com.arjuna.ats.internal.jta.recovery.xarecovery1] Local XARecoveryModule.xaRecovery  got XA exception javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery, XAException.XAER_RMERR

      16:40:10,666 WARN  [HornetQXAResourceWrapper] Can't connect to any hornetq server on recovery [XARecoveryConfig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false], username=null, password=null]]

      16:42:20,695 WARN  [HornetQXAResourceWrapper] Can't connect to any hornetq server on recovery [XARecoveryConfig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatchSize=1048576, transactionBatchSize=1048576, readOnly=false], username=null, password=null]]

      16:42:20,695 WARN  [loggerI18N] [com.arjuna.ats.internal.jta.recovery.xarecovery1] Local XARecoveryModule.xaRecovery  got XA exception javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery, XAException.XAER_RMERR

      16:42:20,696 WARN  [HornetQXAResourceWrapper] Can't connect to any hornetq server on recovery [XARecoveryConfig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatchSize

        • 1. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
          clebert.suconic

          What would happen if you restarted the same server? are you getting any issues?

           

          Maybe the backup node doesn't have access to the paged folder?

           

          The stack trace you pointed out would fail to reload at the same server (not just at the backup side).

           

          If you can restart the live node back, there's probably a misconfiguration on the page folder location on the backup.

           

           

           

          Since You're using EAP, maybe you could provide us the data if this has not been solved after this discussion.

          • 2. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
            meenarc

            Bounced live and backup and restarted live and backup still gave the same exception. The thing is when I kill both live and backup and bring up backup alone it is able to startup fine and read all the journal info from the shared location . It is only during a kill of the live , backup throws this exception

             

            Attached are 2 zips-containing the respective hornetq config and the server log.

            • 3. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
              clebert.suconic

              I need you first to analyze this outside of NFS.

               

              Our recommendation is always against NFS. You can use NFS if you disable caching.. and that still needs to be tested well.

               

              If you replicate this issue outside of a NFS setup (local file) we can then take a look.

              • 4. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                meenarc

                Clebert,

                 

                I get the same exception when I have 2 live/backup jboss nodes sharing a local FS. These 2 jboss nodes run on the same box that has 2 IP's

                Enabled AIO this time around and also turned on TRACE logging.

                 

                Attached are 2 zips-containing the respective hornetq config and the server log.

                • 5. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                  clebert.suconic

                  Do you have a way to reproduce it with clean data?

                  • 6. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                    meenarc

                    Clean data as in delete all the data in journa/paging/large messages or?

                    • 7. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                      clebert.suconic

                      Can you call Print-data on your data, and attanch the output here?

                       

                      Example:

                       

                      java -Xmx2G -cp /work/hornetq/trunk/build/jars/hornetq-logging.jar:/work/hornetq/trunk/build/jars/hornetq-core.jar:/work/hornetq/trunk/thirdparty/org/jboss/netty/lib/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

                       

                       

                      If you are concerned about posting data on the forums...  (case there's any real data), you may either mask the text from Print-data or send it directly to me case you are ok with private sharing but not on the public forum.

                      • 8. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                        meenarc

                        sudo java -cp ./hornetq-logging.jar:./hornetq-core.jar:./netty.jar org.hornetq.core.persistence.impl.journal.PrintData bindings journal

                        ********************************************

                        Server's ID=334c8dd1-93c9-11e1-b649-000c292e2a0d

                        ********************************************

                        ********************************************

                        B I N D I N G S  J O U R N A L

                        ********************************************

                        #JournalFileImpl: (hornetq-bindings-1.bindings id = 1, recordID = 1)

                        operation@AddRecord;recordID=2;userRecordType=24;isUpdate=false;IDCounterEncoding [id=2147483647]

                        operation@AddRecord;recordID=0;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=0, name=jms.queue.angel-logoutQueue, address=jms.queue.angel-logoutQueue, filterString=null]

                        operation@AddRecord;recordID=3;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=3, name=jms.queue.DLQ, address=jms.queue.DLQ, filterString=null]

                        operation@AddRecord;recordID=5;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=5, name=jms.queue.ExpiryQueue, address=jms.queue.ExpiryQueue, filterString=null]

                        #JournalFileImpl: (hornetq-bindings-2.bindings id = 3, recordID = 3)

                        #JournalFileImpl: (hornetq-bindings-4.bindings id = 4, recordID = 4)

                         

                        ### Surviving Records Summary ###

                        recordID=2;userRecordType=24;isUpdate=false;IDCounterEncoding [id=2147483647]

                        recordID=0;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=0, name=jms.queue.angel-logoutQueue, address=jms.queue.angel-logoutQueue, filterString=null]

                        recordID=3;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=3, name=jms.queue.DLQ, address=jms.queue.DLQ, filterString=null]

                        recordID=5;userRecordType=21;isUpdate=false;PersistentQueueBindingEncoding [id=5, name=jms.queue.ExpiryQueue, address=jms.queue.ExpiryQueue, filterString=null]

                         

                        ### Prepared TX ###

                         

                        ### Message Counts ###

                        message count=0

                        message reference count

                        prepared message count=0

                        ********************************************

                        M E S S A G E S   J O U R N A L

                        ********************************************

                        #JournalFileImpl: (hornetq-data-1.hq id = 1, recordID = 1)

                        #JournalFileImpl: (hornetq-data-2.hq id = 2, recordID = 2)

                        #JournalFileImpl: (hornetq-data-3.hq id = 3, recordID = 3)

                        #JournalFileImpl: (hornetq-data-4.hq id = 4, recordID = 4)

                        #JournalFileImpl: (hornetq-data-5.hq id = 5, recordID = 5)

                        #JournalFileImpl: (hornetq-data-6.hq id = 6, recordID = 6)

                        #JournalFileImpl: (hornetq-data-7.hq id = 7, recordID = 7)

                        #JournalFileImpl: (hornetq-data-8.hq id = 8, recordID = 8)

                        #JournalFileImpl: (hornetq-data-9.hq id = 9, recordID = 9)

                        #JournalFileImpl: (hornetq-data-10.hq id = 10, recordID = 10)

                        #JournalFileImpl: (hornetq-data-11.hq id = 11, recordID = 11)

                        #JournalFileImpl: (hornetq-data-12.hq id = 12, recordID = 12)

                         

                        ### Surviving Records Summary ###

                         

                        ### Prepared TX ###

                         

                        ### Message Counts ###

                        message count=0

                        message reference count

                        prepared message count=0

                        • 9. Re: HornetQ backup server -Error message "Cursor 0 had already been created"
                          meenarc

                          Clebert,

                           

                          I have uploaded the entire data directory to the jboss support ticket. I have a data dir with messages on it and one without messages

                           

                          Thanks