Recently, we began testing our ESB implementation against a constant, live data feed. One thing became immediately evident. The two ContentBasedRouters in the system were absolute bottlenecks. Thousands of messages were queuing up behind these routers, which were processing tens of messages per second, at best. Is this performance expected, or is there an issue? I did see a JIRA that is nearly an exact replica of my experience, but it appears to be written for Drools 4.0.x, whereas I'm using Drools 5 (JBESB-2152). Should I file another JIRA or do I just have unrealistic expectations of the CBR?
JBoss ESB 4.5, deployed in JBoss AS 4.2.3
reloadRules is OFF
# of rules: 20-ish per CBR
complexity of rules: the most simple rules you could imagine.
In the meantime, I've eliminated the CBRs and replaced them with purpose-built classes, but we don't want to give up on rules-based routing, since we do intend to use the complexity in the near future.
As it turns out, the CBR itself was not to blame. I originally thought it was slow because it was performing tasks that were far less computationally complex than the rest of the system, but still backing up like the rest of the system, no matter how many threads we configured on the CBR service.
I performed a threaddump and realized that there were entirely too many threads BLOCKED on the synchronization within the CachingRegistryInterceptor. The effect of installing that one choke point in the system basically put all of the services in lock step. Adding more threads to what we considered to be the slower slower services just created additional wait time for the quick CBR threads.
Since our services and EPRs are primarily fixed at boot, I wrote a replacement registry cache by eliminating the native synchronization and using the ReentrantReadWriteLock. This has improved throughput quite a bit. Every thread can simultaneously query the registry now. Our system behaves as it should and our thread-tuning is actually responding as expected.
Presumably your reason for going down this path is more to do with the registry performance, for which there are other issues logged (and fixed) against our version of juddi/scout.
I suppose I'll spend tonight reading the last 500 JIRAs. Perhaps it will solve several of my other problems...
Apologies, I wasn't suggesting that you should do that.
The only time this has come up recently has been after the caching interceptor was disabled and, in that instance, the real culprit turned out to be juddi/scout. We did address the synchronization as part of that investigation.
You said earlier that you were using ESB 4.5 and, if true, would have meant that you had decided to re-enable the interceptor and so I had assumed, albeit incorrectly, that this would be to address the performance.
All I was intending to do was highlight a related issue and point out that there may still be underlying issues.