7 Replies Latest reply: May 26, 2009 1:28 PM by Kevin Conner RSS

ContentBasedRouting painfully slow

Ryan Hochstetler Newbie

Recently, we began testing our ESB implementation against a constant, live data feed. One thing became immediately evident. The two ContentBasedRouters in the system were absolute bottlenecks. Thousands of messages were queuing up behind these routers, which were processing tens of messages per second, at best. Is this performance expected, or is there an issue? I did see a JIRA that is nearly an exact replica of my experience, but it appears to be written for Drools 4.0.x, whereas I'm using Drools 5 (JBESB-2152). Should I file another JIRA or do I just have unrealistic expectations of the CBR?

The particulars:
JBoss ESB 4.5, deployed in JBoss AS 4.2.3
reloadRules is OFF
# of rules: 20-ish per CBR
complexity of rules: the most simple rules you could imagine.

In the meantime, I've eliminated the CBRs and replaced them with purpose-built classes, but we don't want to give up on rules-based routing, since we do intend to use the complexity in the near future.

  • 1. Re: ContentBasedRouting painfully slow
    Daniel Bevenius Master

    Hi,

    would you be able to modify one of the quickstarts with your use case? I'd be happy to take a look at it.

    Thanks,

    /Daniel

  • 2. Re: ContentBasedRouting painfully slow
    Martin Vecera Novice

    Hello, according to our current results, the issue has been really fixed for 4.2.1.CP04 and the performance became 5 times better and it didn't appear again. Could you provide a sample, that would be the best.

  • 3. Re: ContentBasedRouting painfully slow
    Ryan Hochstetler Newbie

    As it turns out, the CBR itself was not to blame. I originally thought it was slow because it was performing tasks that were far less computationally complex than the rest of the system, but still backing up like the rest of the system, no matter how many threads we configured on the CBR service.

    I performed a threaddump and realized that there were entirely too many threads BLOCKED on the synchronization within the CachingRegistryInterceptor. The effect of installing that one choke point in the system basically put all of the services in lock step. Adding more threads to what we considered to be the slower slower services just created additional wait time for the quick CBR threads.

    Since our services and EPRs are primarily fixed at boot, I wrote a replacement registry cache by eliminating the native synchronization and using the ReentrantReadWriteLock. This has improved throughput quite a bit. Every thread can simultaneously query the registry now. Our system behaves as it should and our thread-tuning is actually responding as expected.

  • 4. Re: ContentBasedRouting painfully slow
    Kevin Conner Master

    https://jira.jboss.org/jira/browse/JBESB-2528

    Presumably your reason for going down this path is more to do with the registry performance, for which there are other issues logged (and fixed) against our version of juddi/scout.

    Kev

  • 5. Re: ContentBasedRouting painfully slow
    Ryan Hochstetler Newbie

    I wasn't attempting to improve registry performance. The CachingRegistryInterceptor was configured right out of the box in JBESB 4.5.

    I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...

  • 6. Re: ContentBasedRouting painfully slow
    Ryan Hochstetler Newbie

     

    "ryanhos" wrote:
    I wasn't attempting to improve registry performance. The CachingRegistryInterceptor was configured right out of the box in JBESB 4.5.

    I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...


    4.4, not 4.5.

  • 7. Re: ContentBasedRouting painfully slow
    Kevin Conner Master

     

    "ryanhos" wrote:
    I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...

    Apologies, I wasn't suggesting that you should do that.

    The only time this has come up recently has been after the caching interceptor was disabled and, in that instance, the real culprit turned out to be juddi/scout. We did address the synchronization as part of that investigation.

    You said earlier that you were using ESB 4.5 and, if true, would have meant that you had decided to re-enable the interceptor and so I had assumed, albeit incorrectly, that this would be to address the performance.

    All I was intending to do was highlight a related issue and point out that there may still be underlying issues.

    Kev