I have a project based on Errai 1.3.1.Final (I tried 1.3.2.Final but can't even compile the project due to big changes needed to be done in pom.xml which I can't setup to have it working correctly) and I noticed (by chance) that my application is working fluently until I have open up to 6 applications.
When I open even one more application, all open applications suddenly work very very slow in area of communication with server. All this happens when I use servlet org.jboss.errai.bus.server.servlet.DefaultBlockingServlet, but when my web.xml has servlet org.jboss.errai.bus.server.servlet.JBossCometServlet configured, instead of speed degradation, I receive error like this:
16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) Exception in thread "Dispatch Worker Thread" org.jboss.errai.bus.server.QueueUnavailableException: no queue available to send. (queue or session may have expired)
16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.server.ServerMessageBusImpl.getQueueByMessage(ServerMessageBusImpl.java:888)
16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.server.ServerMessageBusImpl.send(ServerMessageBusImpl.java:418)
16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.client.api.builder.ConversationMessageWrapper.sendNowWith(ConversationMessageWrapper.java:183)
16:40:20,215 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.client.api.builder.AbstractMessageBuilder$1.sendNowWith(AbstractMessageBuilder.java:59)
16:40:20,216 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:109)
16:40:20,216 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.client.util.ErrorHelper.sendClientError(ErrorHelper.java:79)
16:40:20,216 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.client.util.ErrorHelper.handleMessageDeliveryFailure(ErrorHelper.java:163)
16:40:20,216 ERROR [stderr] (Dispatch Worker Thread) at org.jboss.errai.bus.server.Worker.run(Worker.java:131)
16:40:20,220 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/dms].[ErraiServlet]] (http--127.0.0.1-8080-4) Servlet.service() for servlet ErraiServlet threw exception: java.lang.NullPointerException
at org.jboss.errai.bus.server.servlet.JBossCometServlet.event(JBossCometServlet.java:88) [errai-bus-1.3.1.Final.jar:]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilterEvent(ApplicationFilterChain.java:494) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.core.ApplicationFilterChain.doFilterEvent(ApplicationFilterChain.java:399) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:273) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:139) [jboss-as-web-7.0.2.Final.jar:7.0.2.Final]
at org.jboss.as.web.NamingValve.invoke(NamingValve.java:57) [jboss-as-web-7.0.2.Final.jar:7.0.2.Final]
at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:49) [jboss-as-jpa-7.0.2.Final.jar:7.0.2.Final]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:897) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:626) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2054) [jbossweb-7.0.1.Final.jar:7.0.2.Final]
at java.lang.Thread.run(Thread.java:722) [:1.7.0_01]
Interesting is, that after the stacktrace appears, some applications starting working well again, but some other still can't communicate with server.
I completely stuck...
What can be a reason and how to prevent this strange behaviour?
I add that when I have open more then 5 applications, and click on every app to start errai-bus communication with server, all are hanging some time and after that time all give the result dispite I clicked them in a bit different moments in time.
Another information is that when I close as many apps as needed to stay with 5 only, all remaining apps are working very fluentry again. It looks like 5 is a magic number.
Also, consider using the org.jboss.errai.bus.server.SimpleDispatcher dispatcher implementation in your code. Using the AsyncDispatcher can often be overkill for apps, and we'll probably not make it the default in the future.
Beware of a typo in the ErraiServices.properties file where the example SimpleDispatcher starts off with the package name: org.container.errai.* -- this is not correct.
I'm sorry to hear about these issues.
First, for the upgrade to 1.3.2.Final, there should be almost no changes necessary to your pom.xml. The update itself had only a handful of commits beyond 1.3.2.Final and they were just targeted at fixing specific bugs. If your Errai version is factored out into a variable like it is in the quickstarts, that should really be the only thing you need to change.
If you can tell us what problems you had with your build after the upgrade, we can help you make that work.
Second, the bigger issue of performance degradation with more than 5 clients connected. I'm going to create a simple app today to try and reproduce this on our end. Can you give me some more details so that my app is a reasonable approximation of yours?
- Are the messages generally pushed from the server, or is it more of a request-response communication pattern?
- Approximately how big are the messages coming from the clients?
- Approximately how big are the messages coming from the server?
- How frequently is a message sent from a client to the server?
- How frequently is a message sent from the server to a client?
- Are you using the Errai bus directly, through CDI, or both?
Thanks for your help in tracking down this problem.
Unfortunatelly, the property you suggested seems to have no impact on the issue. I changed from default 5 to 100 and this time 7th instance of the application started working very slow (6 previous also started working slow). So, it looks like sometine 5 and sometime 6 is the magic number :|
I tried also org.jboss.errai.bus.server.SimpleDispatcher distatcher implementation but with no effect - still 7th instance of the application starts working poorly.
That's my ErraiService.properties file:
# Request dispatcher implementation (default is SimpleDispatcher)
# Worker pool size. This is the number of threads the asynchronous worker pool should provide for processing
# incoming messages. This option is only valid when using the AsyncDispatcher implementation.
# Worker timeout (in seconds). This defines the time that a single asychronous process may run, before the worker pool
# terminates it and reclaims the thread. This option is only valid when using the AsyncDispatcher implementation.
I'll try to use Errai 2.0 and will see if there is the same issue.
I believe this issue is actually just a browser resource limit... it has nothing to do with the server. Try this:
- Start the app
- Keep opening tabs in your favourite browser until the app "hangs" (probably 3-6 tabs at most)
- Now open another browser (if you used Chrome in step 2, try Firefox now)
- See that the app still works fine in that other browser, although it remains hung in the first browser
- Close all but one of the tabs in the first browser, and observe that the app becomes responsive again
The effect you're observing is that your browser refuses to have more than a small number of open HTTP connections (say, 3-6) to the same server across all its tabs and windows. Each open tab in the same browser is going to have its own long-polling (comet) request. As soon as you use up that allotment, the tabs have to start trading them off with each other. This is not a good situation to be in.
Bottom line: to do scalability testing, you have to use separate browser instances, not just separate tabs in the same browser instance.