-
15. Re: Filter problem
Why don't you try your fix and see if it works? I would save us time in investigating as we are really maxed out right now.
-
16. Re: Filter problem
hi,
The fix works for the mix of consumers problem but it's not our main problem at this time ;-).
As I said our main problem is that sometimes some messages stay blocked in a queue and the thread consume 100% CPU.
This is what happen in the deliver :
if (holder.iter == null) { ref = messageReferences.removeFirst(); } else { ref = holder.iter.next(); }holder.iter is not null but holder.iter.next() is null
then the thread never consumes the message that is in messageReferences.
Maximilien
-
17. Re: Filter problem
I was just saying, if you could try and fix this it would save us some time, I doubt if it is a difficult fix, and we are very maxed right now.
-
18. Re: Filter problem
Hi,
After a little time of investigation i understand a little better how the queueImpl works.
So i can give more informations now :
Our problem appears when there is a call to messageReferences.addFirst, the cachedIterator is not updated and then the consumer that match doesn't see the message. Looking a bit more on how it is created :
/** * Returns an iterator over the elements in this deque in proper sequence. * The returned <tt>Iterator</tt> is a "weakly consistent" iterator that * will never throw {@link java.util.ConcurrentModificationException}, * and guarantees to traverse elements as they existed upon * construction of the iterator, and may (but is not guaranteed to) * reflect any modifications subsequent to construction. * * @return an iterator over the elements in this deque in proper sequence. */
and may (but is not guaranteed to) reflect any modifications subsequent to construction.Have i miss something or hornetq is using a cached iterator that is not reliable for filtered consumers?
Thanks,
Maximilien
-
19. Re: Filter problem
this is a small case:
- create and start 2 consumers with different filters
- create a message that match the first consumer and consume it with it
- create another message that match the first consumer
- the second consumer try to consume the message
- it doesn't match then creates its iterator and call messageReferences.addFirst
- it prompt another run of deliverRunner via the executor
- then the first consumer consumes it (still no iterator for it right now)
- create a message that match the second consumer
- the second consumer consumes the message
- create another message that match the second consumer
- the first consumer try to consume the message
- it doesn't match then creates its iterator and call messageReferences.addFirst. (here is the problem)
- it prompt another run of deliverRunner via the executor
- then the second consumer try to consume a message from its cached iterator but the message is not present then return null.
- it prompt another run of deliverRunner via the executor and then the loop start
-
-
21. Re: Filter problem
I've just created a JIRA https://jira.jboss.org/browse/HORNETQ-469 as it's a different problem than the first one
-
22. Re: Filter problem
Hello Maximilien,
your test case has been tested on our machine against 2.1.1.Final. Before the 4th message is received the Java process starts to produce heavy processor load (on a 4 core processor). Therefore your test case is reproducable on our machine.
The (legacy) application mentioned in
https://community.jboss.org/message/555343#555343
uses consumers with 5 different selectors on the same queue. We do not see the heavy cpu load but (randomly?) blocked messages that are delivered after reconnecting the clients that are Pojo's and MDB's.
Best regards,
Andreas
-
23. Re: Filter problem
Hi andreas,
Thanks for your feedback about my test case.
I doubt we are facing the same problem because my investigations shows me that my problem is due to consumers with filter only implementation and you don't seem to use them, right ?
Maximilien
-
24. Re: Filter problem
Ok, I am looking at this case today.
As an aside, it's usually better for performance reasons to use a topic with multiple subscriptions, each having a selector, rather than a queue with multiple selectors.
With a topic subscription, the selector is evaulated just once when the message is routed to the subscription.
With the queue the selector is evaluated every time delivery is attempted.
Using a topic should workaround this issue, while I fix it.
-
25. Re: Filter problem
Hello Tim and Maximilien,
we had a closer look into this issue and it may relate to our problem because the test case cpu-eating_2 can easily be modified to use MessageHandlers instead of synchronous message receiving, see the attachment. The same method QueueImpl.deliver is executed when using MessageHandlers as it has been observed while using a debugger. All of our the consumers process the messages - through the JMS interface - asynchronously.
Yes, using a topic with durable subscriptions is an interesting alternative, but
(A) 11 of the 12 Service-Units use the same message selector,
(B) and the set of messages that could be consumed by those 11 and the 12th selector are not disjunct
(The 12th is exclusively reserved to one kind of job with direct customer contact, the other 11 may handle those requests as well if they are idle.)
(C) and the application as it is needs the once and only once (to one consumer) feature.
Yes, we could redesign the application to use 5 queues (one for each type of selector-string) without a filter instead of 5 types of filters with one queue. And this may optimize the throughput of the queue, but this queue throughput is not the limiting factor of the application with very slow consumers.
But we would, without further actions, loose the feature (B). A decision for a such redesign shall be based on the knowledge that the usage of the filters is the one and only reason for the problems we have seen.
Best regards,
Andreas
-
26. Re: Filter problem
Hello Tim,
this email informs you that the problematic behavior of HornetQ in the case of
* consumers with different selectors
* and alternating messages that fit the selectors
could have been reproduced on the test system with our real application.
We
* changed the application so that the service-units they could process dummy messages of type A and B,
* configured one to accept only messages of type A and one to only accept messages of type B
* and then sent a bunch of pairs of messages A and B to the queue.
At one point of time the following is observed:
* A high processor load of the JBoss (near to 100%).
* A ConsumerCount that indicates that the consumers are connected.
* A DeliveryCount = 0
* A MessageCount >> 0
That is, the messages in the queue become frozen while the server is processing with high load.
If the client connection is stopped and started again, the consumer starts to processes some messages but
then the DeliveryCount becomes 0 again.
Therefore it is really not unlikely that the thread reduces to this point.
Best regards,
Andreas
-
27. Re: Filter problem
Hi,
Andreas : our production app uses jms message listeners too, the test case uses synchronous receivers for simplicity ;-)
Tim : thanks for the reply. I had already read your advice about using topics rather than queues in another discussion but it's not easy/possible for us because filtered consumers are created dynamically independently of message producing and may vary.
Modified : didn't read well your proposition about topic, as you propose create a topic with selectors the same way we use our queue it could be an option, i'll look at it, thank you
Maximilien
-
28. Re: Filter problem
using topics is not an option for us as we create subscribers dynamically and independently of message publishing, that may results in missed messages.
Maximilien
-
29. Re: Filter problem
wiktorowski maximilien wrote:
using topics is not an option for us as we create subscribers dynamically and independently of message publishing, that may results in missed messages.
Maximilien
Durable subscribers wouldn't miss messages.