12 Replies Latest reply on Dec 19, 2014 4:18 AM by zhouyaguo

    Problem with Closed Connection jboss-as-7.1.1(2,3)

    khensel

      I have been struggling with this for awhile and I'm not sure if it's a problem with jboss or on the database side or the VPN.

       

      Our configuration is as such:

       

      We are connectin to a DB2 mainframe from Redhat Linux.  The Linux machine has a VPN in place that encrypts the data as it comes across.  We have DB2 Connect installed, but I am bypassing it by going directly to the MF through the VPN using type 4 DB2 driver. 

       

      I am running JBoss-as-7.1.3 (compiled from source).  I am running CXF Restful Web Services to get data to pass on to another application.  The error occurs in the DAOs.   Things will run OK for awhile after I clean start JBoss 7, then we intermittently get the following types of errors:

       

      10:21:58,719 ERROR [edu.berkeley.eas.enrollment.dao.classlist.TelebearsClassListDao] (http-/127.0.0.1:8380-74) +++ Exception occurred in stuprod.sect205f: CallableStatementCallback; SQL [{call stuprod.sect205f(?, ?, ?, ?, ?)}]; [jcc][t4][10335][10366][3.57.110] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003; nested exception is com.ibm.db2.jcc.am.io: [jcc][t4][10335][10366][3.57.110] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003: org.springframework.dao.DataAccessResourceFailureException: CallableStatementCallback; SQL [{call stuprod.sect205f(?, ?, ?, ?, ?)}]; [jcc][t4][10335][10366][3.57.110] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003; nested exception is com.ibm.db2.jcc.am.io: [jcc][t4][10335][10366][3.57.110] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003

      Caused by: com.ibm.db2.jcc.am.io: [jcc][t4][10335][10366][3.57.110] Invalid operation: Connection is closed. ERRORCODE=-4470, SQLSTATE=08003

       

      The errors happen more and more until all I get in the log are these errors.  They happen in every stored procedure call (DAO), so it's no specific one.

       

      I have attached the custom standalone.xml (enrollment-services.xml).  I have tried many combinations of connection pool, threads (jca) etc and it always happens again.   I am also attaching sample java classes for DAO and a service class.

       

      This has become a real problem as this morning my production log has 5 million lines in it.  Please help if you can.


      Let me know if you need any more information.

       

      Thanks, Ken

        • 1. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
          jaikiran
          1 of 1 people found this helpful
          • 2. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
            khensel

            Thanks for your answer.

             

            So, I just added the check-valid-connection-sql to my datasource.  I will be watching the logs today to see if it helps, we are at a much lower usage today, but should see something by tonight.  

             

            If I am correct, the valid-connection checker only checks if I have a valid connection, right?  What do I do about having the connection pool being dropped -- or whatever is happening?

             

            K

            • 3. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
              khensel

              Since adding check-valid-connection-sql, I am getting these errors (less of them but still a bunch):

               

              15:02:36,644 WARN  [org.jboss.jca.core.connectionmanager.listener.TxConnectionListener] (http-/127.0.0.1:8380-645) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5bdca59a[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7772e044 connection handles=0 lastUse=1359412149104 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@47147721 pool internal context=SemaphoreArrayListManagedConnectionPool@10993bfc[pool=DB2DS] xaResource=LocalXAResourceImpl@16e1cda0[connectionListener=5bdca59a connectionManager=3eaf7f90 warned=false currentXid=null] txSync=null]: com.ibm.db2.jcc.am.zn: [jcc][t4][2030][11211][3.57.110] A communication error occurred during operations on the connection's underlying socket, socket input stream,

              or socket output stream.  Error location: Reply.fill().  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001

               

              at com.ibm.db2.jcc.am.bd.a(bd.java:319)

                      at com.ibm.db2.jcc.t4.a.a(a.java:365)

                      at com.ibm.db2.jcc.t4.a.a(a.java:360)

                      at com.ibm.db2.jcc.t4.ab.b(ab.java:222)

                      at com.ibm.db2.jcc.t4.ab.c(ab.java:243)

                      at com.ibm.db2.jcc.t4.ab.c(ab.java:354)

                      at com.ibm.db2.jcc.t4.ab.v(ab.java:1139)

                      at com.ibm.db2.jcc.t4.db.a(db.java:38)

                      at com.ibm.db2.jcc.t4.t.a(t.java:32)

                      at com.ibm.db2.jcc.t4.sb.i(sb.java:135)

                      at com.ibm.db2.jcc.am.lm.eb(lm.java:1934)

                      at com.ibm.db2.jcc.am.lm.a(lm.java:2863)

                      at com.ibm.db2.jcc.am.lm.e(lm.java:985)

                      at com.ibm.db2.jcc.am.lm.execute(lm.java:969)

                      at org.jboss.jca.adapters.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:74)

                      at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:1097)

                      at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:461)

                      at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:464)

                      at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:282)

               

              at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397)

                      at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365)

                      at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)

                      at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)

                      at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)

                      at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)

                      at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc-3.0.6.RELEASE.jar:3.0.6.RELEASE]

                      at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc-3.0.6.RELEASE.jar:3.0.6.RELEASE]

                      at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:940) [spring-jdbc-3.0.6.RELEASE.jar:3.0.6.RELEASE]

                      at org.springframework.jdbc.core.JdbcTemplate.call(JdbcTemplate.java:1003) [spring-jdbc-3.0.6.RELEASE.jar:3.0.6.RELEASE]

                      at org.springframework.jdbc.object.StoredProcedure.execute(StoredProcedure.java:144) [spring-jdbc-3.0.6.RELEASE.jar:3.0.6.RELEASE]

                      at edu.berkeley.eas.enrollment.dao.studentprofile.TelebearsStudentProfileProc.execute(TelebearsStudentProfileProc.java:131) [classes:]

                      at edu.berkeley.eas.enrollment.dao.studentprofile.TelebearsStudentProfileDao.getTelebearsStudentProfile(TelebearsStudentProfileDao.java:93) [classes:]

                      at edu.berkeley.eas.enrollment.service.TelebearsStudentProfileService.getTelebearsStudentProfile(TelebearsStudentProfileService.java:66) [classes:]

                      at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source) [:1.6.0_33]

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_33]

                      at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_33]

                      at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:162) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [rt.jar:1.6.0_33]

                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [rt.jar:1.6.0_33]

                      at java.util.concurrent.FutureTask.run(FutureTask.java:138) [rt.jar:1.6.0_33]

                      at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:118) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:208) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:166) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:113) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:184) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:112) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.1.Final.jar:1.0.1.Final]

                      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:163) [cxf-bundle-jaxrs-2.4.0.jar:2.4.0]

                      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.17.Final.jar:]

                      at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.9.Final.jar:2012-08-06 19:12]

                      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.17.Final.jar:]

                      at edu.berkeley.eas.enrollment.filter.SimpleCsvAuthFilter.doFilter(SimpleCsvAuthFilter.java:85) [classes:]

                      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.17.Final.jar:]

                      at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:165) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

                      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:679) [jbossweb-7.0.17.Final.jar:]

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final.jar:]

                      at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_33]

               

              15:02:36,653 WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject] (http-/127.0.0.1:8380-645) IJ000612: Destroying connection that could not be successfully matched: org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@5bdca59a[state=DESTROYED managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@7772e044 connection handles=0 lastUse=1359412149104 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@47147721 pool internal context=SemaphoreArrayListManagedConnectionPool@10993bfc[pool=DB2DS] xaResource=LocalXAResourceImpl@16e1cda0[connectionListener=5bdca59a connectionManager=3eaf7f90 warned=false currentXid=null] txSync=null]

              • 4. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                jaikiran

                Those don't look like error messages and instead are WARN  messages which show you exactly why the connection was no longer considered valid when the validity check was done using the SQL that you configured. I believe the implementation just logs that WARN and hands out a new/different valid connection to the application. Are you seeing any errors?

                • 5. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                  jesper.pedersen

                  You likely need <allow-multiple-users/> - and you are missing the entire <validation> too....

                  • 6. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                    khensel

                    Jesper,

                     

                    Could you clarify please? 

                     

                    Here is my datasources configuration:

                    <subsystem xmlns="urn:jboss:domain:datasources:1.1">

                                <datasources>

                                    <datasource jndi-name="java:/EnrollmentServicesDS" pool-name="DB2DS" enabled="true" use-java-context="true" use-ccm="true">

                                        <connection-url>${db2.db}</connection-url>

                                        <connection-property name="autoCommit"> false</connection-property>

                                        <driver>db2jcc</driver>

                                        <pool>

                                            <min-pool-size>1</min-pool-size>

                                            <max-pool-size>200</max-pool-size>

                                            <use-strict-min>true</use-strict-min>

                                        </pool>

                                        <security>

                                            <security-domain>${db2.ds}</security-domain>

                                        </security>

                                        <validation>

                    <!--                        <check-valid-connection-sql>select 1 from sysibm.sysdummy1 with ur</check-valid-connection-sql> -->

                                            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2ValidConnectionChecker"/>

                                            <stale-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2StaleConnectionChecker"/>

                                            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.db2.DB2ExceptionSorter"/>

                                        </validation>

                                        <statement>

                                            <track-statements>true</track-statements>

                                        </statement>

                                        <timeout>

                                            <blocking-timeout-millis>5000</blocking-timeout-millis>

                                            <idle-timeout-minutes>0</idle-timeout-minutes>

                                        </timeout>

                                    </datasource>

                                    <drivers>

                                        <driver name="db2jcc" module="com.ibm.db2.jcc">

                                            <driver-class>com.ibm.db2.jcc.DB2Driver</driver-class>

                                        </driver>

                                    </drivers>

                                </datasources>

                     

                    Ther jca-subsystem:

                    <subsystem xmlns="urn:jboss:domain:jca:1.1">

                                <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>

                                <bean-validation enabled="false"/>

                                <default-workmanager>

                                    <short-running-threads>

                                        <core-threads count="100"/>

                                        <queue-length count="100"/>

                                        <max-threads count="100"/>

                                        <keepalive-time time="10" unit="seconds"/>

                                    </short-running-threads>

                                    <long-running-threads>

                                        <core-threads count="100"/>

                                        <queue-length count="100"/>

                                        <max-threads count="100"/>

                                        <keepalive-time time="50" unit="seconds"/>

                                    </long-running-threads>

                                </default-workmanager>

                                <cached-connection-manager debug="true" error="true"/>

                            </subsystem>

                     

                    Using this socket-binding: <socket-binding name="ajp" port="8009"/>

                     

                    through Apache.

                     

                    Log is set to DEBUG, with overrides for my application, spring and some others to INFO.

                     

                     

                     

                    Thanks, Ken

                    • 7. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                      jesper.pedersen

                      <pool>

                      • Add <prefill>true</prefill>
                      • Add <allow-multiple-users/>
                      • Remove <use-strict-min>

                       

                      <timeout>

                      • Remove <idle-timeout-minutes> or set it to a sane value

                       

                      You can play with <flush-strategy>IdleConnections</flush-strategy> also...

                       

                      :jca: isn't used (directly) for datasources.

                       

                      Google should help you to find datasource templates for DB2 - however, you owe me a beer now - so I'm happy

                      1 of 1 people found this helpful
                      • 8. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                        khensel

                        Thanks for your answer Jesper!

                         

                        i added allow-multiple-users, removed use-scrict-min and added prefill.  I had already found a datasource example here: http://docs.jboss.org/ironjacamar/userguide/1.1/en-US/html/ex_datasources.html, which is where I found the connection-checker stuff.

                         

                        I had tried prefill and various other settings over the past few months and the errors continued.  It seems the <validation> items are helping more than anything has so far.

                         

                        I'll keep you posted.


                        Thanks again,

                         

                        Ken

                        • 9. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                          khensel

                          One more question:

                           

                          Does using cache connection manager buy me anything?  I have use-ccm="true in the datasource definition.  I found this on Google.  That is why the jca subsystem has   <cached-connection-manager debug="true" error="true"/>.

                           

                           

                          If so, what does this buy me?  You said the datasource doesn't directly use jca:.  Should I take this stuff out or leave it in / change it?

                           

                          Thanks for all of your help,

                           

                          Ken

                          • 10. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                            jesper.pedersen

                            Support for lazy transaction enlistment, and WARN if you forget to close your connections. Remove error="true".

                            • 11. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                              khensel

                              This seems to be doing better since adding the connection validation to the configuration.  The system has not been under as heavy of a load as it was earlier in the week.


                              I'll keep you posted.

                               

                              Thanks, Ken

                              • 12. Re: Problem with Closed Connection jboss-as-7.1.1(2,3)
                                zhouyaguo

                                <check-valid-connection-sql> take effect only when <validate-on-match> is true