13 Replies Latest reply on Apr 26, 2012 10:01 PM by clebert.suconic

    Couldn't locate page transaction WARNING

    black_ice_spa

      Hello,

       

      I'm using HornetQ 2.2.2 server, on a REALLY BLOATED MACHINE (aka performance its -1, swap file has like 8gb used...), queues are using PAGING mode (i think HDD is becoming crazy, OS must be paging memory and hornetq too). MaxMemorySize its 10mb and pageSize its 5mb. My guess is that we should use everything-on-memory and let OS handle swapping... or better machine, but well, need to search alternatives for now.

       

      Server was restarted because queue had 21 pages, and even with timeouts of around 30minutes client with a receiver.receive(60000) said it had no elements (after 13minutes waiting).

      After a restart (i dont know which kind of restart, forced insta-kill or controlled), im getting many messages like this:


      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,099 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659849, ignoring message on position PagePositionImpl [pageNr=764, messageNr=0, recordID=0]

      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,130 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659857, ignoring message on position PagePositionImpl [pageNr=764, messageNr=1, recordID=0]

      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,130 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659865, ignoring message on position PagePositionImpl [pageNr=764, messageNr=2, recordID=0]

      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659873, ignoring message on position PagePositionImpl [pageNr=764, messageNr=3, recordID=0]

      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659881, ignoring message on position PagePositionImpl [pageNr=764, messageNr=4, recordID=0]

      [Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659889, ignoring message on position PagePositionImpl [pageNr=764, messageNr=5, recordID=0]....

       

       

      Someone could tell me what's the possible cause of that warning, and if i should care about it?

       

      Thanks,

        • 1. Re: Couldn't locate page transaction WARNING
          clebert.suconic

          if you killed the server with pending Transactions.. that could happen.

           

           

          There were a few fixes that would affect cleaning up of pages.. maybe that would fix it. We are about to upload a new release 2.2.14. maybe you would like to give it a try on that.

          • 2. Re: Couldn't locate page transaction WARNING
            black_ice_spa

            Thanks =)

             

             

            We are trying to get server moved to a better machine, for now we have to keep that version (if i could switch i would do, but...), but for future projects we'll update.

            • 3. Re: Couldn't locate page transaction WARNING
              black_ice_spa

              Hello again,

               

              I could get a copy of hornetq server files, indeed, there was a page file which was not removed (it was out of order, aka, 0000458.page, and after it, 00000461.page...) it would cause those messages in every receiveMessage() call, and server wouldnt return any item in that queue (same messages giving error everytime).

               

              I had to manually delete it and remove from server.

               

              We are gonna try 2.2.14 and check if its capable of removing those pages automatically,

               

              Edit: found it, but cant build, dependencies are missing i think .

               

                 [java] [INFO] Unable to find resource 'org.hornetq:hornetq-jms:jar:2.2.snap

              shot' in repository jboss (http://repository.jboss.org/nexus/content/groups/publ

              ic/)

                   [java] Downloading: http://repo1.maven.org/maven2/org/hornetq/hornetq-jms/2

              .2.snapshot/hornetq-jms-2.2.snapshot.jar

                   [java] [INFO] Unable to find resource 'org.hornetq:hornetq-jms:jar:2.2.snap

              shot' in repository central (http://repo1.maven.org/maven2)

               

              I could built 2.2.15 tho, its AS7, can it run standalone?

               

              Thanks,

              • 4. Re: Couldn't locate page transaction WARNING
                clebert.suconic

                We just release 2.2.14 today.. look for the announcement.

                1 of 1 people found this helpful
                • 5. Re: Couldn't locate page transaction WARNING
                  black_ice_spa

                  Oh thanks!, testing right now, My work time is very early , when i tried to build it there was not release yet.

                   

                  I tested it right now, same problem with new version, this is my current paging directory (remember, it comes from a really bloated machine, and yeah they probably restarted PC while it was writing-removing pages (it took like 15min to do that....)).

                   

                  Directorio de F:\Programas\hornetq-2.2.14.Final\data\paging\969c707e-0551-11e1-

                  aa13-005056985a84

                   

                  12/04/2012  14:53         5.242.771 000000458.page

                  18/04/2012  18:50         5.242.407 000000461.page

                  18/04/2012  18:55         5.242.480 000000462.page

                  18/04/2012  19:00         5.242.858 000000463.page

                  18/04/2012  19:05         5.242.203 000000464.page

                  18/04/2012  19:09         5.242.038 000000465.page

                  18/04/2012  19:13         5.242.165 000000466.page

                  18/04/2012  19:17         5.242.159 000000467.page

                  18/04/2012  19:21         5.242.851 000000468.page

                  18/04/2012  19:23         4.204.587 000000469.page

                  02/11/2011  14:52                24 address.txt

                   

                   

                   

                  If i execute server and run a client, client will receive null (aka, timeout call) or block forever.

                   

                  Server log:

                   

                  c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,336 WARNING [PageSubscriptionImpl]

                  Couldn't locate page transaction 23622387897, ignoring message on position PageP

                  ositionImpl [pageNr=458, messageNr=4543, recordID=0] on address=jms.queue.Cola_V

                  CBT_VC queue=jms.queue.Cola_VCBT_VC

                  .........

                   

                  * [Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=0c82e835-8b22-11e1-b2

                  c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,340 WARNING [PageSubscriptionImpl]

                  Couldn't locate page transaction 23622388185, ignoring message on position PageP

                  ositionImpl [pageNr=458, messageNr=4591, recordID=0] on address=jms.queue.Cola_V

                  CBT_VC queue=jms.queue.Cola_VCBT_VC

                   

                   

                  * [Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=0c82e835-8b22-11e1-b2

                  c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,340 WARNING [PageSubscriptionImpl]

                  Couldn't locate page transaction 23622388191, ignoring message on position PageP

                  ositionImpl [pageNr=458, messageNr=4592, recordID=0] on address=jms.queue.Cola_V

                  CBT_VC queue=jms.queue.Cola_VCBT_VC

                   

                  MessageNr=1...4592 (i think), at least 4543..4592

                   

                   

                  Restarting server does nothing (same error will be thrown after).

                  Queue do receive new objects tho (and page them at tail).

                   

                  If i remove 000000458.page file, it works perfectly.

                   

                  I can provide a sample data folder if needed.

                  • 6. Re: Couldn't locate page transaction WARNING
                    clebert.suconic

                    It looks like you just restarted the server (killed it) with pending Transactions?

                    • 7. Re: Couldn't locate page transaction WARNING
                      clebert.suconic

                      It should clean up itself if you consume all the messages.

                      • 8. Re: Couldn't locate page transaction WARNING
                        black_ice_spa

                        I cant consume messages, server will give that error EVERYTIME i try to consume (and return null after timeout), but i can still insert messages (i can see them consumed last after i fix the "error" manually).

                         

                        After i delete the .page file manually it does work fine, i can consume rest of messages etc.

                         

                        I dont control the server myself (i only got files from it) so i cant speak 100% sure, but its very possible they did kill it with pending transactions (or maybe later and transactions couldnt persist, idk, server is REALLY BLOATED, well now they increased RAM and it looks a little better).

                         

                         

                        I have a "dummy process which consumes from the queue". This is what happens.

                         

                        1- Execute dummy process --> Server will throw those messages and dummy process will receive nothing or "receive" null if there is a timeout.

                        2- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout.

                        3- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout. 

                        4- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout. 

                        ...

                        ...

                        5- Shutdown server

                        6- Remove 0000458.page (the old "stuck" one).

                        7- Start server

                        8- Execute dummy process --> Server will say nothing and dummy process will receive messages from queue in the right order (the ones who were in the other pages), working 100% right.

                         

                         

                         

                        PD: This what happens happens in the bloated server and in a "standalone machine" (quad core, nothing running etc....) where we copied server files.

                        • 9. Re: Couldn't locate page transaction WARNING
                          clebert.suconic

                          You are always having this situation when starting the data from scratch?

                           

                           

                          We need a replicator then... we have several (and really serious tests) and we don't have this kind of scenario on any of our tests, and this is the first time I was informed about it.

                           

                           

                          It could e some configuration issue there?

                          • 10. Re: Couldn't locate page transaction WARNING
                            black_ice_spa

                            No, we dont have it when starting the data from scratch (aka cleaning the data), and we were not capable to reproduce it in our test-server, the data which we got from the real (and bloated) server its already "corrupted". They increased RAM, removed the "broken pages" and its working fine, no problems on that matter.

                             

                            I can provide you a corrupted data, but cant reproduce the error from scratch.

                             

                            Apart from using transactions and 10mb queues with 5mb pages, i think all other configurations are default.

                            • 11. Re: Couldn't locate page transaction WARNING
                              clebert.suconic

                              We fixed the server to avoid this situation. If the data is already like that (i.e. a page ouf of sync) you just have to remove it manually.

                              1 of 1 people found this helpful
                              • 12. Re: Couldn't locate page transaction WARNING
                                black_ice_spa

                                Wow that was fast, thank you.I think we'll update to next release.

                                 

                                Much appreciated =).

                                • 13. Re: Couldn't locate page transaction WARNING
                                  clebert.suconic

                                  We had issues with cleaning paging in the past which were fixed.

                                   

                                  Software is a box full of surprises.. you may have found a new bug (I doubt on this case).. so if you're still seeing this happening, we need to know what's going on.

                                   

                                  I hope this won't happen based on the tests we (and our QA team) and other customers and community users have done.

                                   

                                   

                                  If you see this issue again with the new release, please let us know.