2 Replies Latest reply on Jul 17, 2015 7:00 PM by seacuke23

    Consume messages not matching other consumer's selectors

    seacuke23
    seacuke23 Jul 15, 2015 6:49 PM

      We're considering an architecture that involves a single outgoing work queue with all messages put on the queue containing a property specifying a "worker name".  If a worker doesn't exist to satisfy the order (ie. there isn't a consumer whose message selector matches the "worker name") we would like it started up somewhere in the cluster (SE nodes) and to remain alive servicing orders until it sees fit to shut down.

       

      Is there a way to have a message selector that will consume only messages that other consumers wouldn't?  Is there another strategy to divert unconsumed messages to another queue?  We have considered setting a message expiry and using an expiry queue but that won't work because a message that meets the selection criteria of one of the consumers may expire before the consumer that would have handled it can get to it.  In that case we would like that message to remain on the queue until the consumer gets to it.

       

      If there isn't a way to do this, can you suggest a different architecture that would produce the same result?

       

      Also, is there a point (number of consumers using selection on a single queue) that this architecture becomes unfeasable or poorly performant?

        • 1. Re: Consume messages not matching other consumer's selectors
          gaohoward
          gaohoward Jul 15, 2015 8:52 PM (in response to seacuke23)

          I feel like you need to implement this part of logic yourself. for example you create a consumer that receive every message from the outgoing queue, examines the worker name and check if the worker exists or not. Starts the worker if needed.

           

          Then forward the message to another topic (which all workers subscribe to). Then commit the session to let the message be removed from the original queue.

           

          Or just directly dispatch the message to the worker (no topic needed).

          1 of 1 people found this helpful
            • Actions
          • 2. Re: Consume messages not matching other consumer's selectors
            seacuke23
            seacuke23 Jul 17, 2015 7:00 PM (in response to gaohoward)

            Thanks Yong.  I don't think we'd be able to do it quite like that.  One requirement we have is if we lose a node we'd want to recreate the workers on another node relatively quickly.  Here's our current plan.

             

            Three queues:

            1) order queue - orders put on this queue with worker name property used in selection by consumers (workers).

            2) active worker queue - coordinating system pushes out a message for each worker expected to be running.  Worker consumes and discards off this queue with selectors.  Messages not consumed within a second will be assumed not to have a worker running and will be expired to queue 3.

            3) worker launch queue - nodes consume off this queue and start the named worker

             

            So.....does this sound ludicrous?

              • Actions