JBOSS6.1 crashes / AIX6.1 / Oracle_client11.2.0.3 / java6_64
Hi all,
We are using JBOSS6.1 Final version in our environment
AIX 1 6
java6_64
oracle 11.2.0.3 client
-------------------------------------
Every now and then the JVM crashes with the following javacore output. I doubt that the client is the culprit but not able to pinpoint the reason . Can you help or advise me as how to proceed further. javacore are also attached. If you guys need any more information then do let me know.
---------------------------------------
NULL ------------------------------------------------------------------------
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TISIGINFO Dump Event "gpf" (00002000) received
1TIDATETIME Date: 2014/04/08 at 10:35:15
1TIFILENAME Javacore filename: /hahome/phoenix/jboss6/jboss6_webgrensesnitt/bin/javacore.20140408.103450.14024892.0004.txt
1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x0
1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections
NULL ------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
NULL ================================
2XHOSLEVEL OS Level : AIX 6.1
2XHCPUS Processors -
3XHCPUARCH Architecture : ppc64
3XHNUMCPUS How Many : 16
3XHNUMASUP NUMA is either not supported or has been disabled by user
NULL
1XHEXCPCODE J9Generic_Signal_Number: 00000004
1XHEXCPCODE Signal_Number: 0000000B
1XHEXCPCODE Error_Value: 00000000
1XHEXCPCODE Signal_Code: 00000033
1XHEXCPCODE Handler1: 09001000A0AAAA20
1XHEXCPCODE Handler2: 09001000A0AA1E58
NULL
1XHEXCPMODULE Module: /progs/app/oracle/product/11203/client/lib/libclntsh.so
1XHEXCPMODULE Module_base_address: 0900000005684000
NULL
1XHREGISTERS Registers:
2XHREGISTER R0: 09000000058BFC10
2XHREGISTER R1: 00000001236DAC90
..
.
.
.
.
1XMCURTHDINFO Current thread
NULL ----------------------
3XMTHREADINFO "WorkerThread#0[10.179.120.84:47451]" J9VMThread:0x0000000119A3EB00, j9thread_t:0x0000000119998980, java/lang/T
hread:0x070000013DAA08C0, state:R, prio=5
3XMTHREADINFO1 (native thread ID:0x36D00C5, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at oracle/jdbc/driver/T2CStatement.t2cDefineExecuteFetch(Native Method)
4XESTACKTRACE at oracle/jdbc/driver/T2CPreparedStatement.doDefineExecuteFetch(T2CPreparedStatement.java:1138)
4XESTACKTRACE at oracle/jdbc/driver/T2CPreparedStatement.executeForRows(T2CPreparedStatement.java:725)
4XESTACKTRACE at oracle/jdbc/driver/OracleStatement.executeMaybeDescribe(OracleStatement.java:1199(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/OracleStatement.doExecuteWithTimeout(OracleStatement.java:1289(Compiled Code))
4XESTACKTRACE at oracle/jdbc/driver/OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3584)
4XESTACKTRACE at oracle/jdbc/driver/OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3628)
4XESTACKTRACE at oracle/jdbc/driver/OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java
:1493)
4XESTACKTRACE at steria/fap/as/dbaccess/SessionDbConnection.verify(SessionDbConnection.java:599)
4XESTACKTRACE at steria/fap/as/pool/SubPool.getPoolItem(SubPool.java:248)
4XESTACKTRACE at steria/fap/as/pool/SubPool.getFreePoolItem(SubPool.java:279)
4XESTACKTRACE at steria/fap/as/pool/Pool.getFreePoolItem(Pool.java:47)
4XESTACKTRACE at steria/fap/as/dbaccess/DBConnManager.getConnection(DBConnManager.java:101)
4XESTACKTRACE at steria/fap/as/dbaccess/DBConnManager.getFormsSDBAccess(DBConnManager.java:86)
4XESTACKTRACE at steria/fap/as/defmanager/DefManager.getApplicationObject(DefManager.java:1305)
4XESTACKTRACE at steria/fap/as/defmanager/DefManager.getApplicationObject(DefManager.java:1289)
4XESTACKTRACE at steria/fap/as/defmanager/DefManager.getApplicationObject(DefManager.java:1284)
4XESTACKTRACE at steria/fap/as/operations/Operations.openSearch(Operations.java:1169)
4XESTACKTRACE at steria/fap/as/FapAsBean.openSearch(FapAsBean.java:459)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method)
4XESTACKTRACE at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48(Compiled Code))
4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code
))
4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:600(Compiled Code))
4XESTACKTRACE at org/jboss/invocation/Invocation.performCall(Invocation.java:386)
4XESTACKTRACE at org/jboss/ejb/StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java
:233)
4XESTACKTRACE at org/jboss/resource/connectionmanager/CachedConnectionInterceptor.invoke(CachedConnectionIntercepto
r.java:156)
4XESTACKTRACE at org/jboss/ejb/plugins/StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceIntercept
or.java:173)
4XESTACKTRACE at org/jboss/ejb/plugins/CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
4XESTACKTRACE at org/jboss/ejb/plugins/AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
4XESTACKTRACE at org/jboss/ejb/plugins/TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
4XESTACKTRACE at org/jboss/ejb/plugins/TxInterceptorCMT.invoke(TxInterceptorCMT.java:181(Compiled Code))
4XESTACKTRACE at org/jboss/ejb/plugins/SecurityInterceptor.process(SecurityInterceptor.java:228)
4XESTACKTRACE at org/jboss/ejb/plugins/SecurityInterceptor.invoke(SecurityInterceptor.java:211)
4XESTACKTRACE at org/jboss/ejb/plugins/security/PreSecurityInterceptor.process(PreSecurityInterceptor.java:100)
4XESTACKTRACE at org/jboss/ejb/plugins/security/PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:84)
4XESTACKTRACE at org/jboss/ejb/plugins/LogInterceptor.invoke(LogInterceptor.java:205(Compiled Code))
4XESTACKTRACE at org/jboss/ejb/plugins/ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
4XESTACKTRACE at org/jboss/ejb/SessionContainer.internalInvoke(SessionContainer.java:650)
4XESTACKTRACE at org/jboss/ejb/Container.invoke(Container.java:1072)
4XESTACKTRACE at sun/reflect/GeneratedMethodAccessor288.invoke(Bytecode PC:40)
4XESTACKTRACE at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37(Compiled Code
))
4XESTACKTRACE at java/lang/reflect/Method.invoke(Method.java:600(Compiled Code))
4XESTACKTRACE at org/jboss/mx/interceptor/ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
4XESTACKTRACE at org/jboss/mx/server/Invocation.dispatch(Invocation.java:96(Compiled Code))
4XESTACKTRACE at org/jboss/mx/server/Invocation.invoke(Invocation.java:87(Compiled Code))
4XESTACKTRACE at org/jboss/mx/server/AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:200(Compiled Code))
4XESTACKTRACE at org/jboss/mx/server/MBeanServerImpl.invoke(MBeanServerImpl.java:670)
4XESTACKTRACE at org/jboss/invocation/unified/server/UnifiedInvoker.invoke(UnifiedInvoker.java:232)
4XESTACKTRACE at org/jboss/remoting/ServerInvoker.invoke(ServerInvoker.java:967)
4XESTACKTRACE at org/jboss/remoting/transport/socket/ServerThread.completeInvocation(ServerThread.java:791)
4XESTACKTRACE at org/jboss/remoting/transport/socket/ServerThread.processInvocation(ServerThread.java:744)
4XESTACKTRACE at org/jboss/remoting/transport/socket/ServerThread.dorun(ServerThread.java:586(Compiled Code))
4XESTACKTRACE at org/jboss/remoting/transport/socket/ServerThread.run(ServerThread.java:234)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK sdbgrfbibf_io_block_file+0x164 (0x09000000058B9C48 [libclntsh.so+0x235c48])
4XENATIVESTACK dbgrfrbf_read_block_file+0x218 (0x0900000005805ADC [libclntsh.so+0x181adc])
4XENATIVESTACK dbgrmflrp_read_page+0x88 (0x09000000059F140C [libclntsh.so+0x36d40c])
4XENATIVESTACK dbgrmblgmp_get_many_pages+0x14c (0x09000000059B9F10 [libclntsh.so+0x335f10])
4XENATIVESTACK dbgrmmdrrmd_read_relation_meta_data+0x150 (0x09000000059FAAD4 [libclntsh.so+0x376ad4])
4XENATIVESTACK dbgrmmdora_open_record_access_full+0xe34 (0x0900000005A06058 [libclntsh.so+0x382058])
4XENATIVESTACK dbgriporc_openrel_wcreate+0x134 (0x0900000005972D98 [libclntsh.so+0x2eed98])
4XENATIVESTACK dbgrip_open_relation_access+0x88 (0x090000000597310C [libclntsh.so+0x2ef10c])
4XENATIVESTACK dbgrip_start_iterator+0x184 (0x0900000005977FC8 [libclntsh.so+0x2f3fc8])
4XENATIVESTACK dbgrip_relation_iterator+0x15c (0x0900000005979620 [libclntsh.so+0x2f5620])
4XENATIVESTACK dbgruprac_read_adrctl+0xc8 (0x09000000059633CC [libclntsh.so+0x2df3cc])
4XENATIVESTACK dbgrsc_set_createid+0x240 (0x0900000005786584 [libclntsh.so+0x102584])
4XENATIVESTACK dbgrcas_create_adr_schema+0x34 (0x0900000005787FD8 [libclntsh.so+0x103fd8])
4XENATIVESTACK dbgraid_adr_init_disk+0x150 (0x0900000005787A94 [libclntsh.so+0x103a94])
4XENATIVESTACK dbgrcaod_check_adr_ondisk+0xc8 (0x090000000578994C [libclntsh.so+0x10594c])
4XENATIVESTACK dbgrlWriteAlertDetail_int+0x8c (0x09000000058951F0 [libclntsh.so+0x2111f0])
4XENATIVESTACK dbgrlWriteAlertDetail+0x150 (0x09000000058935D4 [libclntsh.so+0x20f5d4])
4XENATIVESTACK dbgrlWriteAlertText+0x24 (0x09000000058933E8 [libclntsh.so+0x20f3e8])
4XENATIVESTACK nlddwrtlog+0x114 (0x09000000056A5BB8 [libclntsh.so+0x21bb8])
4XENATIVESTACK nldsadrvfp+0x4c (0x0900000005F91B50 [libclntsh.so+0x90db50])
4XENATIVESTACK nldsfprintf+0xf4 (0x0900000005F90258 [libclntsh.so+0x90c258])
4XENATIVESTACK niqlce1+0xa0 (0x0900000006FE17E4 [libclntsh.so+0x195d7e4])
4XENATIVESTACK nioqer+0x9ec (0x0900000006FD67D0 [libclntsh.so+0x19527d0])
4XENATIVESTACK nioqrc+0x26b8 (0x0900000006FC527C [libclntsh.so+0x194127c])
4XENATIVESTACK ttcdrv+0x3dc (0x0900000005EE23C0 [libclntsh.so+0x85e3c0])
4XENATIVESTACK nioqwa+0x4c (0x0900000006FE1270 [libclntsh.so+0x195d270])
4XENATIVESTACK upirtrc+0x97c (0x0900000005EFA6C0 [libclntsh.so+0x8766c0])
4XENATIVESTACK kpurcsc+0x6c (0x0900000005EF8E90 [libclntsh.so+0x874e90])
4XENATIVESTACK kpuexec+0xc10 (0x0900000005EF4B34 [libclntsh.so+0x870b34])
4XENATIVESTACK OCIStmtExecute+0x20 (0x0900000006500E04 [libclntsh.so+0xe7ce04])
4XENATIVESTACK eooExecute+0x1788 (0x0900000003151F4C [libocijdbc11.so+0x9f4c])
4XENATIVESTACK Java_oracle_jdbc_driver_T2CStatement_t2cDefineExecuteFetch+0x2e8 (0x090000000315696C [libocijdbc11.so
+0xe96c])