5 Replies Latest reply on Jan 29, 2013 8:09 AM by jonas.zuberbuehler

java.io.IOException: buffer overflow kills app

jonas.zuberbuehler Jan 23, 2013 5:40 AM

Once in a while (more or less once a week) our webapp won't load and the user is presented a blank screen. If we check the server log there are tons of the following exception:

22-01-2013 14:17:01,328 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr: java.io.IOException: buffer overflow
22-01-2013 14:17:01,329 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.errai.bus.server.io.buffers.TransmissionBuffer.checkOverflow(TransmissionBuffer.java:654)
22-01-2013 14:17:01,330 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.errai.bus.server.io.buffers.TransmissionBuffer.readWait(TransmissionBuffer.java:480)
22-01-2013 14:17:01,331 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.errai.bus.server.MessageQueueImpl.poll(MessageQueueImpl.java:119)
22-01-2013 14:17:01,332 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.pollForMessages(DefaultBlockingServlet.java:145)
22-01-2013 14:17:01,333 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doGet(DefaultBlockingServlet.java:97)
22-01-2013 14:17:01,334 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
22-01-2013 14:17:01,335 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
22-01-2013 14:17:01,335 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
22-01-2013 14:17:01,336 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
22-01-2013 14:17:01,337 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62)
22-01-2013 14:17:01,338 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
22-01-2013 14:17:01,339 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
22-01-2013 14:17:01,340 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at net.comerge.kcc.server.filter.LoggingFilter.doFilter(LoggingFilter.java:39)
22-01-2013 14:17:01,341 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
22-01-2013 14:17:01,342 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
22-01-2013 14:17:01,343 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
22-01-2013 14:17:01,344 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
22-01-2013 14:17:01,345 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:397)
22-01-2013 14:17:01,346 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50)
22-01-2013 14:17:01,347 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
22-01-2013 14:17:01,348 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
22-01-2013 14:17:01,348 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
22-01-2013 14:17:01,349 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
22-01-2013 14:17:01,350 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
22-01-2013 14:17:01,351 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505)
22-01-2013 14:17:01,352 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445)
22-01-2013 14:17:01,353 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
22-01-2013 14:17:01,354 [ajp--127.0.0.1-8009-19] (someuser) ERROR    stderr:         at java.lang.Thread.run(Thread.java:679)

The only solution after this happens is to kill JBoss-AS 7.1 (it won't shutdown normally) and start it once again. We looked through all our log files to find a usage pattern on how to reproduce the error but couldn't find any. It occured after a user was idle for two hours, it occured in the middle of a user action and so on.

Do you have idea what might cause that error and/or do you have any suggestions on how to debug to find the cause?

Any help is appreciated!

cheers

--jonas

1. Re: java.io.IOException: buffer overflow kills app

cbrock Jan 23, 2013 9:11 AM (in response to jonas.zuberbuehler)

The error is generally caused if the load of the application is too high to accomodate the rate of messages going through the bus. You can increase the size of the buffer (the default is 32MB) by using the following property in the ErraiService.properties file:

errai.bus.buffer_size=<size in MB>

Try increasing the size to 64MB

errai.bus.buffer_size=64

Is you application under high load?

Actually, an overflow in the buffer should only effect the one client that experienced it, not the entire bus. So this is something we need to investigage.
Actions
2. Re: java.io.IOException: buffer overflow kills app

jonas.zuberbuehler Jan 23, 2013 10:36 AM (in response to cbrock)

Thanks for your answer Mike.

Our application is not under high load, no.

We just did a load test (about 15k requests, in sum about 15MB) which provoked the buffer overflow for two different users (not the one we used for testing). JBoss went crazy (CPU usage around 80 to 90 percent) and the log was continously spammed with the above stacktrace for both users. This didn't kill the application it only slowed it down for around 10 minutes (the two users could continue to work). After that it seemed like errai was recovering, CPU load went normal and the exception didn't occur anymore.

Hope that helps. Will post later if increasing the buffer size helped.
Actions
3. Re: java.io.IOException: buffer overflow kills app

jfuerth Jan 25, 2013 10:49 AM (in response to jonas.zuberbuehler)

Hi Jonas,

We were able to reproduce this problem, and we committed a fix yeterday. Can you verify in your environment by trying again with the latest 2.2.0-SNAPSHOT builds?

[EDIT: I originally wrote 2.0-SNAPSHOT, which is not correct!]
Actions
4. Re: java.io.IOException: buffer overflow kills app

jonas.zuberbuehler Jan 25, 2013 11:42 AM (in response to jfuerth)

You guys rock! Thanks a lot for getting back to me so fast. I'll try it on Monday and let you know if the fix works.
Actions
5. Re: java.io.IOException: buffer overflow kills app

jonas.zuberbuehler Jan 29, 2013 8:09 AM (in response to jfuerth)

It looks good. We couldn't reproduce the bug so far on our test system. We will go live tomorrow bundled with Errai 2.2.0.Final and I'm positive that our issue is fixed.

Thanks again.
Actions

Go to original post