ThreadLocal holding instance of BufferManagerImpl
nsabina Jan 21, 2013 4:47 AMWe are getting OOM and heap dumps (almost daily) and it shows a consistent memory tree (attached).
Heap analyzer identifies leak suspects as instances of ThreadLocal holding BufferManagerImpl.
It appears to be associated with connection problems with one of backends but this is not confirmed. Based on stack trace OOM is happening when we are trying to load object tree on UI request sent to this specific backend with connection problems. Our tree loading is calling SortUtility which seems to be allocation BufferManagerImpl, is the issue with returned tuples?
Can someone from Teiid team please review the snapshot of heap dump and stack trace and suggest what could be a failing piece here?
Thank you.
SELECT '__constant__' AS c_0, 'DB2 Performance' AS c_1, null AS c_2, null AS c_3, '__column******__' AS c_4, dsConf AS c_5, 'dsConf' AS c_6, null AS c_7, '__constant__' AS c_8, 'Non-Data Sharing Subsystems' AS c_9, null AS c_10, null AS c_11, '__column__' AS c_12, dsSystem AS c_13, 'dsSystem' AS c_14, null AS c_15, '__column__' AS c_16, dsSSID AS c_17, 'dsSSID' AS c_18, null AS c_19, '__constant__' AS c_20, 'Subsystem Performance' AS c_21, null AS c_22, null AS c_23, '__constant__' AS c_24, 'History' AS c_25, null AS c_26, null AS c_27, '__objectname__' AS c_28, '00118' AS c_29, 'DBMzDB2.IDB2_HSUDSQIC_Rbase_Vbase' AS c_30, null AS c_31, null AS c_32, null AS c_33, null AS c_34, null AS c_35 FROM DBMzDB2.IDB2_HSUDSQIC_Rbase_Vbase) AS u ORDER BY c_1, c_5, c_9, c_13, c_17, c_21, c_25, c_29, c_33
at com.ca.chorus.db.DbExecutor.handleSqlException(DbExecutor.java:423)
at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:337)
at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:267)
at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getFilterTreeWithoutRoot(ObjectsServiceImpl.java:214)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)
at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)
at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getTreeForSection(ObjectsServiceImpl.java:2424)
at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getTreeNodeChildren(ObjectsServiceImpl.java:2736)
at com.ca.mfui.chorusR2.server.service.PreloadUserTree$1$1.run(PreloadUserTree.java:70)
at com.ca.chorus.server.transaction.TransactionRunner$3.run(TransactionRunner.java:75)
at com.ca.chorus.server.transaction.TransactionRunner.invoke(TransactionRunner.java:43)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)
at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)
at com.ca.chorus.server.transaction.TransactionRunner$2.run(TransactionRunner.java:56)
at com.ca.mfui.chorusR2.server.service.ChorusThreadPool$2.call(ChorusThreadPool.java:83)
at com.google.inject.servlet.ServletScopes$3.call(ServletScopes.java:194)
at com.ca.chorus.server.transaction.TransactionRunner.invoke(TransactionRunner.java:23)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)
at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)
at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)
at com.ca.mfui.chorusR2.server.service.RequestScopedThreadPoolCallableDecorator.call(RequestScopedThreadPoolCallableDecorator.java:42)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1121)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
at java.lang.Thread.run(Thread.java:779)
Caused by: org.teiid.jdbc.TeiidSQLException: TEIID30504 performance_netmaster: Java heap space
at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:113)
at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:70)
at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:656)
at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:62)
at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:512)
at org.teiid.client.util.ResultsFuture.addCompletionListener(ResultsFuture.java:139)
at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:508)
at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:211)
at com.ca.chorus.db.LeakDetectingPreparedStatement.executeQuery(LeakDetectingPreparedStatement.java:52)
at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:296)
... 26 more
Caused by: org.teiid.core.TeiidProcessingException: TEIID30504 performance_netmaster: Java heap space
at org.teiid.dqp.internal.process.DataTierTupleSource.exceptionOccurred(DataTierTupleSource.java:502)
at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:289)
at org.teiid.dqp.internal.process.TupleSourceCache$BufferedTupleSource.nextTuple(TupleSourceCache.java:107)
at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:279)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.S****ortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)
at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)
at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)
at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)
at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)
at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)
at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)
at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)
at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)
at org.teiid.query.processor.relational.SortNode.getFinalBuffer(SortNode.java:191)
at org.teiid.query.processor.relational.RelationalPlan.getFinalBuffer(RelationalPlan.java:270)
at org.teiid.query.processor.QueryProcessor.getFinalBuffer(QueryProcessor.java:264)
at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:143)
at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:382)
at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:291)
at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)
at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:219)
at org.teiid.dqp.internal.process.DQPCore.executeRequest(DQPCore.java:386)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at org.teiid.logging.LogManager$LoggingProxy.invoke(LogManager.java:121)
at org.teiid.jboss.TransportService$2.invoke(TransportService.java:205)
at $Proxy18.executeRequest(Unknown Source)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at org.teiid.transport.LocalServerConnection$1$1.call(LocalServerConnection.java:131)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:177)
at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:249)
at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:233)
at org.teiid.transport.LocalServerConnection$1.invoke(LocalServerConnection.java:129)
at $Proxy18.executeRequest(Unknown Source)
at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:631)
at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:506)
... 29 more
Caused by: org.teiid.translator.TranslatorException: Java heap space
at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.handleError(ConnectorWorkItem.java:205)
at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:264)
at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:425)
at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:170)
at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:167)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)
at java.util.concurrent.FutureTask.run(FutureTask.java:177)
at org.teiid.dqp.internal.process.DQPCore$FutureWork.run(DQPCore.java:118)
at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:249)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:123)
at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:298)
... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.ca.chorus.teiid.connector.netmaster.NMConnection.read(NMConnection.java:919)
at com.ca.chorus.teiid.connector.netmaster.NMConnection.requestNetMaster(NMConnection.java:1048)
at com.ca.chorus.teiid.connector.netmaster.NMConnection.sendRequest(NMConnection.java:1198)