13 Replies Latest reply: Apr 26, 2012 10:01 PM by Clebert Suconic RSS

Couldn't locate page transaction WARNING

Tino Sainz Newbie

Hello,

 

I'm using HornetQ 2.2.2 server, on a REALLY BLOATED MACHINE (aka performance its -1, swap file has like 8gb used...), queues are using PAGING mode (i think HDD is becoming crazy, OS must be paging memory and hornetq too). MaxMemorySize its 10mb and pageSize its 5mb. My guess is that we should use everything-on-memory and let OS handle swapping... or better machine, but well, need to search alternatives for now.

 

Server was restarted because queue had 21 pages, and even with timeouts of around 30minutes client with a receiver.receive(60000) said it had no elements (after 13minutes waiting).

After a restart (i dont know which kind of restart, forced insta-kill or controlled), im getting many messages like this:


[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,099 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659849, ignoring message on position PagePositionImpl [pageNr=764, messageNr=0, recordID=0]

[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,130 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659857, ignoring message on position PagePositionImpl [pageNr=764, messageNr=1, recordID=0]

[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,130 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659865, ignoring message on position PagePositionImpl [pageNr=764, messageNr=2, recordID=0]

[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659873, ignoring message on position PagePositionImpl [pageNr=764, messageNr=3, recordID=0]

[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659881, ignoring message on position PagePositionImpl [pageNr=764, messageNr=4, recordID=0]

[Thread-18 (group:HornetQ-server-threads1921072065-493973408)] 18:43:50,146 WARNING [org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl]  Couldn't locate page transaction 19346659889, ignoring message on position PagePositionImpl [pageNr=764, messageNr=5, recordID=0]....

 

 

Someone could tell me what's the possible cause of that warning, and if i should care about it?

 

Thanks,

  • 1. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    if you killed the server with pending Transactions.. that could happen.

     

     

    There were a few fixes that would affect cleaning up of pages.. maybe that would fix it. We are about to upload a new release 2.2.14. maybe you would like to give it a try on that.

  • 2. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    Thanks =)

     

     

    We are trying to get server moved to a better machine, for now we have to keep that version (if i could switch i would do, but...), but for future projects we'll update.

  • 3. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    Hello again,

     

    I could get a copy of hornetq server files, indeed, there was a page file which was not removed (it was out of order, aka, 0000458.page, and after it, 00000461.page...) it would cause those messages in every receiveMessage() call, and server wouldnt return any item in that queue (same messages giving error everytime).

     

    I had to manually delete it and remove from server.

     

    We are gonna try 2.2.14 and check if its capable of removing those pages automatically,

     

    Edit: found it, but cant build, dependencies are missing i think .

     

       [java] [INFO] Unable to find resource 'org.hornetq:hornetq-jms:jar:2.2.snap

    shot' in repository jboss (http://repository.jboss.org/nexus/content/groups/publ

    ic/)

         [java] Downloading: http://repo1.maven.org/maven2/org/hornetq/hornetq-jms/2

    .2.snapshot/hornetq-jms-2.2.snapshot.jar

         [java] [INFO] Unable to find resource 'org.hornetq:hornetq-jms:jar:2.2.snap

    shot' in repository central (http://repo1.maven.org/maven2)

     

    I could built 2.2.15 tho, its AS7, can it run standalone?

     

    Thanks,

  • 4. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    We just release 2.2.14 today.. look for the announcement.

  • 5. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    Oh thanks!, testing right now, My work time is very early , when i tried to build it there was not release yet.

     

    I tested it right now, same problem with new version, this is my current paging directory (remember, it comes from a really bloated machine, and yeah they probably restarted PC while it was writing-removing pages (it took like 15min to do that....)).

     

    Directorio de F:\Programas\hornetq-2.2.14.Final\data\paging\969c707e-0551-11e1-

    aa13-005056985a84

     

    12/04/2012  14:53         5.242.771 000000458.page

    18/04/2012  18:50         5.242.407 000000461.page

    18/04/2012  18:55         5.242.480 000000462.page

    18/04/2012  19:00         5.242.858 000000463.page

    18/04/2012  19:05         5.242.203 000000464.page

    18/04/2012  19:09         5.242.038 000000465.page

    18/04/2012  19:13         5.242.165 000000466.page

    18/04/2012  19:17         5.242.159 000000467.page

    18/04/2012  19:21         5.242.851 000000468.page

    18/04/2012  19:23         4.204.587 000000469.page

    02/11/2011  14:52                24 address.txt

     

     

     

    If i execute server and run a client, client will receive null (aka, timeout call) or block forever.

     

    Server log:

     

    c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,336 WARNING [PageSubscriptionImpl]

    Couldn't locate page transaction 23622387897, ignoring message on position PageP

    ositionImpl [pageNr=458, messageNr=4543, recordID=0] on address=jms.queue.Cola_V

    CBT_VC queue=jms.queue.Cola_VCBT_VC

    .........

     

    * [Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=0c82e835-8b22-11e1-b2

    c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,340 WARNING [PageSubscriptionImpl]

    Couldn't locate page transaction 23622388185, ignoring message on position PageP

    ositionImpl [pageNr=458, messageNr=4591, recordID=0] on address=jms.queue.Cola_V

    CBT_VC queue=jms.queue.Cola_VCBT_VC

     

     

    * [Thread-23 (HornetQ-server-HornetQServerImpl::serverUUID=0c82e835-8b22-11e1-b2

    c1-89d2ecc225fe-153893925)] 20-Apr 21:51:48,340 WARNING [PageSubscriptionImpl]

    Couldn't locate page transaction 23622388191, ignoring message on position PageP

    ositionImpl [pageNr=458, messageNr=4592, recordID=0] on address=jms.queue.Cola_V

    CBT_VC queue=jms.queue.Cola_VCBT_VC

     

    MessageNr=1...4592 (i think), at least 4543..4592

     

     

    Restarting server does nothing (same error will be thrown after).

    Queue do receive new objects tho (and page them at tail).

     

    If i remove 000000458.page file, it works perfectly.

     

    I can provide a sample data folder if needed.

  • 6. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    It looks like you just restarted the server (killed it) with pending Transactions?

  • 7. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    It should clean up itself if you consume all the messages.

  • 8. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    I cant consume messages, server will give that error EVERYTIME i try to consume (and return null after timeout), but i can still insert messages (i can see them consumed last after i fix the "error" manually).

     

    After i delete the .page file manually it does work fine, i can consume rest of messages etc.

     

    I dont control the server myself (i only got files from it) so i cant speak 100% sure, but its very possible they did kill it with pending transactions (or maybe later and transactions couldnt persist, idk, server is REALLY BLOATED, well now they increased RAM and it looks a little better).

     

     

    I have a "dummy process which consumes from the queue". This is what happens.

     

    1- Execute dummy process --> Server will throw those messages and dummy process will receive nothing or "receive" null if there is a timeout.

    2- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout.

    3- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout. 

    4- Execute dummy process --> Server wont throw those messages but dummy process will receive nothing or "receive" null if there is a timeout. 

    ...

    ...

    5- Shutdown server

    6- Remove 0000458.page (the old "stuck" one).

    7- Start server

    8- Execute dummy process --> Server will say nothing and dummy process will receive messages from queue in the right order (the ones who were in the other pages), working 100% right.

     

     

     

    PD: This what happens happens in the bloated server and in a "standalone machine" (quad core, nothing running etc....) where we copied server files.

  • 9. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    You are always having this situation when starting the data from scratch?

     

     

    We need a replicator then... we have several (and really serious tests) and we don't have this kind of scenario on any of our tests, and this is the first time I was informed about it.

     

     

    It could e some configuration issue there?

  • 10. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    No, we dont have it when starting the data from scratch (aka cleaning the data), and we were not capable to reproduce it in our test-server, the data which we got from the real (and bloated) server its already "corrupted". They increased RAM, removed the "broken pages" and its working fine, no problems on that matter.

     

    I can provide you a corrupted data, but cant reproduce the error from scratch.

     

    Apart from using transactions and 10mb queues with 5mb pages, i think all other configurations are default.

  • 11. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    We fixed the server to avoid this situation. If the data is already like that (i.e. a page ouf of sync) you just have to remove it manually.

  • 12. Re: Couldn't locate page transaction WARNING
    Tino Sainz Newbie

    Wow that was fast, thank you.I think we'll update to next release.

     

    Much appreciated =).

  • 13. Re: Couldn't locate page transaction WARNING
    Clebert Suconic Master

    We had issues with cleaning paging in the past which were fixed.

     

    Software is a box full of surprises.. you may have found a new bug (I doubt on this case).. so if you're still seeing this happening, we need to know what's going on.

     

    I hope this won't happen based on the tests we (and our QA team) and other customers and community users have done.

     

     

    If you see this issue again with the new release, please let us know.