7 Replies Latest reply on Jan 25, 2013 12:57 PM by shawkins

    ThreadLocal holding instance of BufferManagerImpl

    nsabina

      We are getting OOM and heap dumps (almost daily) and it shows a consistent memory tree (attached).

      Heap analyzer identifies leak suspects as instances of ThreadLocal holding BufferManagerImpl.

      It appears to be associated with connection problems with one of backends but this is not confirmed. Based on stack trace OOM is happening when we are trying to load object tree on UI request sent to this specific backend with connection problems. Our tree loading is calling SortUtility which seems to be allocation BufferManagerImpl, is the issue with returned tuples?

       

      Can someone from Teiid team please review the snapshot of heap dump and stack trace and suggest what could be a failing piece here?

       

      Thank you.

      headump.jpg

       

      SELECT '__constant__' AS c_0, 'DB2 Performance' AS c_1, null AS c_2, null AS c_3, '__column******__' AS c_4, dsConf AS c_5, 'dsConf' AS c_6, null AS c_7, '__constant__' AS c_8, 'Non-Data Sharing Subsystems' AS c_9, null AS c_10, null AS c_11, '__column__' AS c_12, dsSystem AS c_13, 'dsSystem' AS c_14, null AS c_15, '__column__' AS c_16, dsSSID AS c_17, 'dsSSID' AS c_18, null AS c_19, '__constant__' AS c_20, 'Subsystem Performance' AS c_21, null AS c_22, null AS c_23, '__constant__' AS c_24, 'History' AS c_25, null AS c_26, null AS c_27, '__objectname__' AS c_28, '00118' AS c_29, 'DBMzDB2.IDB2_HSUDSQIC_Rbase_Vbase' AS c_30, null AS c_31, null AS c_32, null AS c_33, null AS c_34, null AS c_35 FROM DBMzDB2.IDB2_HSUDSQIC_Rbase_Vbase) AS u ORDER BY c_1, c_5, c_9, c_13, c_17, c_21, c_25, c_29, c_33

          at com.ca.chorus.db.DbExecutor.handleSqlException(DbExecutor.java:423)

          at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:337)

          at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:267)

          at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getFilterTreeWithoutRoot(ObjectsServiceImpl.java:214)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)

          at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)

          at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getTreeForSection(ObjectsServiceImpl.java:2424)

          at com.ca.mfui.chorusR2.server.service.ObjectsServiceImpl.getTreeNodeChildren(ObjectsServiceImpl.java:2736)

          at com.ca.mfui.chorusR2.server.service.PreloadUserTree$1$1.run(PreloadUserTree.java:70)

          at com.ca.chorus.server.transaction.TransactionRunner$3.run(TransactionRunner.java:75)

          at com.ca.chorus.server.transaction.TransactionRunner.invoke(TransactionRunner.java:43)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)

          at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)

          at com.ca.chorus.server.transaction.TransactionRunner$2.run(TransactionRunner.java:56)

          at com.ca.mfui.chorusR2.server.service.ChorusThreadPool$2.call(ChorusThreadPool.java:83)

          at com.google.inject.servlet.ServletScopes$3.call(ServletScopes.java:194)

          at com.ca.chorus.server.transaction.TransactionRunner.invoke(TransactionRunner.java:23)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:71)

          at com.ca.chorus.aop.guice.PerfTracer.trace(PerfTracer.java:88)

          at com.ca.chorus.server.transaction.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:31)

          at com.ca.mfui.chorusR2.server.service.RequestScopedThreadPoolCallableDecorator.call(RequestScopedThreadPoolCallableDecorator.java:42)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)

          at java.util.concurrent.FutureTask.run(FutureTask.java:177)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1121)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)

          at java.lang.Thread.run(Thread.java:779)

      Caused by: org.teiid.jdbc.TeiidSQLException: TEIID30504 performance_netmaster: Java heap space

          at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:113)

          at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:70)

          at org.teiid.jdbc.StatementImpl.postReceiveResults(StatementImpl.java:656)

          at org.teiid.jdbc.StatementImpl.access$100(StatementImpl.java:62)

          at org.teiid.jdbc.StatementImpl$2.onCompletion(StatementImpl.java:512)

          at org.teiid.client.util.ResultsFuture.addCompletionListener(ResultsFuture.java:139)

          at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:508)

          at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:211)

          at com.ca.chorus.db.LeakDetectingPreparedStatement.executeQuery(LeakDetectingPreparedStatement.java:52)

          at com.ca.chorus.db.DbExecutor.withResultSet(DbExecutor.java:296)

          ... 26 more

      Caused by: org.teiid.core.TeiidProcessingException: TEIID30504 performance_netmaster: Java heap space

          at org.teiid.dqp.internal.process.DataTierTupleSource.exceptionOccurred(DataTierTupleSource.java:502)

          at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:289)

          at org.teiid.dqp.internal.process.TupleSourceCache$BufferedTupleSource.nextTuple(TupleSourceCache.java:107)

          at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:279)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:146)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.S****ortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.nextBatchDirect(SortNode.java:91)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

          at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:279)

          at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

          at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

          at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

          at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

          at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:247)

          at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:184)

          at org.teiid.query.processor.relational.SortNode.sortPhase(SortNode.java:102)

          at org.teiid.query.processor.relational.SortNode.getFinalBuffer(SortNode.java:191)

          at org.teiid.query.processor.relational.RelationalPlan.getFinalBuffer(RelationalPlan.java:270)

          at org.teiid.query.processor.QueryProcessor.getFinalBuffer(QueryProcessor.java:264)

          at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:143)

          at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:382)

          at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:291)

          at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

          at org.teiid.dqp.internal.process.RequestWorkItem.run(RequestWorkItem.java:219)

          at org.teiid.dqp.internal.process.DQPCore.executeRequest(DQPCore.java:386)

          at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

          at java.lang.reflect.Method.invoke(Method.java:613)

          at org.teiid.logging.LogManager$LoggingProxy.invoke(LogManager.java:121)

          at org.teiid.jboss.TransportService$2.invoke(TransportService.java:205)

          at $Proxy18.executeRequest(Unknown Source)

          at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)

          at java.lang.reflect.Method.invoke(Method.java:613)

          at org.teiid.transport.LocalServerConnection$1$1.call(LocalServerConnection.java:131)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)

          at java.util.concurrent.FutureTask.run(FutureTask.java:177)

          at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:249)

          at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:233)

          at org.teiid.transport.LocalServerConnection$1.invoke(LocalServerConnection.java:129)

          at $Proxy18.executeRequest(Unknown Source)

          at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:631)

          at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:506)

          ... 29 more

      Caused by: org.teiid.translator.TranslatorException: Java heap space

          at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.handleError(ConnectorWorkItem.java:205)

          at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:264)

          at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:425)

          at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:170)

          at org.teiid.dqp.internal.process.DataTierTupleSource$1.call(DataTierTupleSource.java:167)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:345)

          at java.util.concurrent.FutureTask.run(FutureTask.java:177)

          at org.teiid.dqp.internal.process.DQPCore$FutureWork.run(DQPCore.java:118)

          at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:249)

          at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:123)

          at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:298)

          ... 3 more

      Caused by: java.lang.OutOfMemoryError: Java heap space

          at com.ca.chorus.teiid.connector.netmaster.NMConnection.read(NMConnection.java:919)

          at com.ca.chorus.teiid.connector.netmaster.NMConnection.requestNetMaster(NMConnection.java:1048)

          at com.ca.chorus.teiid.connector.netmaster.NMConnection.sendRequest(NMConnection.java:1198)

        • 1. Re: ThreadLocal holding instance of BufferManagerImpl
          shawkins

          > Heap analyzer identifies leak suspects as instances of ThreadLocal holding BufferManagerImpl.

           

          There is only a single BufferManagerImpl instance for all of Teiid, so having ThreadLocal references to that single instance is not an issue.  Are you seeing multiple distinct BufferManagerImpl(s)?  If yes are you running mulitple Teiid instances or has just a restart of Teiid (not the whole VM) been performed?  If it's the case that you're performing a restart of just Teiid, and using local connections then it is possible that one of the calling threads is still holding a reference to an old BufferManagerImpl instance.  Adding shutdown logic to the BufferManagerImpl would address that.

           

          If this is not the case, then there's more aspects to consider.  Are you explicitly sizing any of the BufferManager config properties, such as memory-buffer-space?

           

          The Teiid memory architecture is multi-layerd.  The BufferManager will directly hold references to batches/table pages based upon a guess of the memory footprint.  The objects held are live and on-heap - ideally for light loads we can then skip serialization entirely.  This memory region is roughly constrained by the max-reserve-kb setting (more on that in a second).  As batches are pushed out of that region, they are serialized into a shared memory buffer.  That is the large byte array held by the BlockByteBuffer thread local.  This array is fixed and its size is determined by the max-memory-space setting.  At the least this acts as a shared serialization buffer, but if enough space is available it acts as a second caching layer, such that values are only evicted to disk on demand.

           

          So generally having a single large byte array allocated is not an issue - in fact it is desirable as that memory usage is fixed/predictable and can be moved off-heap via the memory-buffer-off-heap.  The only thing you have to consider is that if you are explicitly setting it and leaving it on heap, is the size appropriate to allow all of the other processing operations. 

           

          It's more likely that you could be experiencing issues with the working memory area of the BufferMangager.  There are quite a few factors to consider here including the max-reserve-kb, max-processing-kb settings, whether large Object values are being used (that we aren't estimating the footprint correctly for), and if local connections are being used (which in conjunction with the max-processing-kb setting may be putting too much memory pressure since local work is not constrained by max-active-plans).

           

          > Our tree loading is calling SortUtility which seems to be allocation BufferManagerImpl, is the issue with returned tuples?

           

          The SortUtility will reserve working BufferManager memory based upon the schema of the data being sorted.  If the processing is held up due to a failing source or other issue post-execution issue, then that particular plan will still be holding that memory, recuding the amount of working memory available to other work.  Based upon the max-processing-kb setting and the number of concurrently executing plans, additional work may then be able to assume more memory than is available in the vm.

          1 of 1 people found this helpful
          • 2. Re: ThreadLocal holding instance of BufferManagerImpl
            nsabina

            Thank you for your suggestions Steve,

             

            Since OOM is consistently related to specific datasource connection problems I do have few more questions to ask.

             

            The only difference as far as this datasource translator is concerned that for every execution their connector/translator is sending us up to 8 warnings while most of other translators are rarely sending more than 2.

            Warnings as discussed here - https://community.jboss.org/thread/198360

             

            I understand that these warnings are added to Statement object by Teiid? We have a delegating  safety harness translator (outermost)  which intercepts exceptions from underlying datasource translator and adds them to execution payload.

             

            Right now:

              1) Some warnings are duplicates, we can eliminate adding duplicates to execution context but if I am not mistaken they will still be a part of Statement object, right? So until we release it memory wil be still consumed regardless of if we add to execution payload or not?

            2) The reason we are getting bunch of warnings per execution is because this is how translator/connector for this datasource is written.

            We are getting socket I/O exception + Java IOException + TranslatorException/ResourceException ... .

            What would be proper way to handle several exceptions in translator to report this type of datasource issues back to platform?

            3) If preparedStatement is not explicitely closed in finally block in [something]Execution.getNextResultSet() in respective datasource translator can it cause memory leak? According to one of our team members Teiid would close this resource even if it is not explicitly done in translator, can you please confirm?

             

            Thank you.

            • 3. Re: ThreadLocal holding instance of BufferManagerImpl
              shawkins

              > Since OOM is consistently related to specific datasource connection problems I do have few more questions to ask.

               

              It's not visible on the image, but the first thing to rule out is the residual BufferManagerImpl reference (something else is holding ~400 MB in the breakdown shown).  If one is erronously being kept alive after a Teiid restart, that is likely accounting for the bulk of the issue.

               

              > 1) Some warnings are duplicates, we can eliminate adding duplicates to execution context but if I am not mistaken they will still be a part of Statement object, right? So until we release it memory wil be still consumed regardless of if we add to execution payload or not?

               

              If you add a warning to the execution context it will be part of the Statement.  Once the Statement has the warnings it will retain them until clearWarnings or another execute is called.  I don't quite follow the second part.  What do you mean by release it? 

               

              > 2) The reason we are getting bunch of warnings per execution is because this is how translator/connector for this datasource is written.

              > We are getting socket I/O exception + Java IOException + TranslatorException/ResourceException ... .

              > What would be proper way to handle several exceptions in translator to report this type of datasource issues back to platform?

               

              Assuming this is a JDBC source, JDBCBaseExecution.addStatementWarnings controls how warnings are added to the ExecutionContext.  You could look at customizing that logic to better consolidate or filter warnings.  If there is something that seems generally applicable, such as the ability to configure the translator to not accumulate the warning, just log an issue.

               

              > 3) If preparedStatement is not explicitely closed in finally block in [something]Execution.getNextResultSet() in respective datasource translator can it cause memory leak? According to one of our team members Teiid would close this resource even if it is not explicitly done in translator, can you please confirm?

               

              Are you talking about the stock JDBC translator or some customization?  In the stock JDBC translator statements are closed when the execution is closed, which can happen prior to the client closing the statement as long as no lobs are returned or the translator is marked as having readable lobs after close.  Can you describe this a little more fully?

               

              Steve 

              1 of 1 people found this helpful
              • 4. Re: ThreadLocal holding instance of BufferManagerImpl
                nsabina

                Looks like I was confused about what addWarnings to context exactly do, now it is clear, thank you. We certanly need to eliminate dups there.

                As for your last question, this is custom translator written by other team and this specific Executions class containing  getNextResultSet() method implements org.teiid.translator.ResultSetExecution

                • 5. Re: ThreadLocal holding instance of BufferManagerImpl
                  nsabina

                  Warnings issue was solved however need some more clarification on the below

                  It turns out there are 2 instances of BufferManagerImpl, second one is taking very little memory and is referenced by top root class org.teiid.jboss.TeiidExtension.

                  Is not a valid instance? Can you tell from the snapshot what was it created for ?

                   

                  Thank you,

                   

                  second instance.jpg

                  • 6. Re: ThreadLocal holding instance of BufferManagerImpl
                    rareddy

                    The one created from the TeiidExtension is the boot strapped instance of BufferManager during startup of the Teiid server.  Where is other instance? Can you check the id on both so that they are referring to same instance?

                    • 7. Re: ThreadLocal holding instance of BufferManagerImpl
                      shawkins

                      https://issues.jboss.org/browse/TEIID-2359 ensures that the BufferManager resources will be cleaned up appropriately after a Teiid restart even when using local connections.

                       

                      Steve