7 Replies Latest reply on May 26, 2009 1:28 PM by kconner

    ContentBasedRouting painfully slow

    ryanhos
    ryanhos Apr 15, 2009 3:35 PM

      Recently, we began testing our ESB implementation against a constant, live data feed. One thing became immediately evident. The two ContentBasedRouters in the system were absolute bottlenecks. Thousands of messages were queuing up behind these routers, which were processing tens of messages per second, at best. Is this performance expected, or is there an issue? I did see a JIRA that is nearly an exact replica of my experience, but it appears to be written for Drools 4.0.x, whereas I'm using Drools 5 (JBESB-2152). Should I file another JIRA or do I just have unrealistic expectations of the CBR?

      The particulars:
      JBoss ESB 4.5, deployed in JBoss AS 4.2.3
      reloadRules is OFF
      # of rules: 20-ish per CBR
      complexity of rules: the most simple rules you could imagine.

      In the meantime, I've eliminated the CBRs and replaced them with purpose-built classes, but we don't want to give up on rules-based routing, since we do intend to use the complexity in the near future.

      • 3691 Views
      • Tags:
        • 1. Re: ContentBasedRouting painfully slow
          beve
          beve Apr 15, 2009 4:13 PM (in response to ryanhos)

          Hi,

          would you be able to modify one of the quickstarts with your use case? I'd be happy to take a look at it.

          Thanks,

          /Daniel

            • Actions
          • 2. Re: ContentBasedRouting painfully slow
            mvecera
            mvecera Apr 16, 2009 9:58 AM (in response to ryanhos)

            Hello, according to our current results, the issue has been really fixed for 4.2.1.CP04 and the performance became 5 times better and it didn't appear again. Could you provide a sample, that would be the best.

              • Actions
            • 3. Re: ContentBasedRouting painfully slow
              ryanhos
              ryanhos May 26, 2009 11:29 AM (in response to ryanhos)

              As it turns out, the CBR itself was not to blame. I originally thought it was slow because it was performing tasks that were far less computationally complex than the rest of the system, but still backing up like the rest of the system, no matter how many threads we configured on the CBR service.

              I performed a threaddump and realized that there were entirely too many threads BLOCKED on the synchronization within the CachingRegistryInterceptor. The effect of installing that one choke point in the system basically put all of the services in lock step. Adding more threads to what we considered to be the slower slower services just created additional wait time for the quick CBR threads.

              Since our services and EPRs are primarily fixed at boot, I wrote a replacement registry cache by eliminating the native synchronization and using the ReentrantReadWriteLock. This has improved throughput quite a bit. Every thread can simultaneously query the registry now. Our system behaves as it should and our thread-tuning is actually responding as expected.

                • Actions
              • 4. Re: ContentBasedRouting painfully slow
                kconner
                kconner May 26, 2009 11:52 AM (in response to ryanhos)

                https://jira.jboss.org/jira/browse/JBESB-2528

                Presumably your reason for going down this path is more to do with the registry performance, for which there are other issues logged (and fixed) against our version of juddi/scout.

                Kev

                  • Actions
                • 5. Re: ContentBasedRouting painfully slow
                  ryanhos
                  ryanhos May 26, 2009 12:12 PM (in response to ryanhos)

                  I wasn't attempting to improve registry performance. The CachingRegistryInterceptor was configured right out of the box in JBESB 4.5.

                  I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...

                    • Actions
                  • 6. Re: ContentBasedRouting painfully slow
                    ryanhos
                    ryanhos May 26, 2009 12:19 PM (in response to ryanhos)

                     

                    "ryanhos" wrote:
                    I wasn't attempting to improve registry performance. The CachingRegistryInterceptor was configured right out of the box in JBESB 4.5.

                    I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...


                    4.4, not 4.5.

                      • Actions
                    • 7. Re: ContentBasedRouting painfully slow
                      kconner
                      kconner May 26, 2009 1:28 PM (in response to ryanhos)

                       

                      "ryanhos" wrote:
                      I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...

                      Apologies, I wasn't suggesting that you should do that.

                      The only time this has come up recently has been after the caching interceptor was disabled and, in that instance, the real culprit turned out to be juddi/scout. We did address the synchronization as part of that investigation.

                      You said earlier that you were using ESB 4.5 and, if true, would have meant that you had decided to re-enable the interceptor and so I had assumed, albeit incorrectly, that this would be to address the performance.

                      All I was intending to do was highlight a related issue and point out that there may still be underlying issues.

                      Kev

                        • Actions