9 Replies Latest reply on Apr 19, 2012 3:30 PM by rokhmanov

    Size limitations of addWarning in ExecutionContext

    rokhmanov

      Hi,

       

      We use ExecutionContext.addWarning() to pass messages from translators to the client (consumed by Statement.getWarnings() method).

      Is there any practical size limits when use this approach? Messages are wrapped into TeiidSQLWarning each, and when we attempt to pass some significant amount of them (less than a humdred, for example), we are getting the error below.

      (in another words we run ExecutionContext.addWarning() a hundred times per request):

       

      [SingleInstanceCommunicationException]data length too big: 55150548 (max: 33554432)

      1 [StreamCorruptedException]data length too big: 55150548 (max: 33554432)

       

      Unfortunatley we cannot provide more detailed exception dumps in the moment. Seems that number of available messages corellates with each message size. The bigger message size - the lesser a number of messages can be successfully passed. Our <100 limit is with messages about 150 characters each.

       

      A couple of times we got an OOM on Teiid with just a handful of big messages. We will attempt to limit a size of each message and a number of messages passed, but do you know what is this magic number 33554432 means, and is it possible to configure it?

       

      Thanks!

      Andriy

        • 1. Re: Size limitations of addWarning in ExecutionContext
          rareddy

          Andrily,

           

          Teiid allows max of 32 GB as the packet size, I see your sizes or well above. I suggest using a profiler and see why your message packets are so huge. Your estimate of (150 chars *16*100) is not even close.

           

          Also see related https://community.jboss.org/thread/196848?tstart=0

           

          Ramesh..

          • 2. Re: Size limitations of addWarning in ExecutionContext
            shawkins

            The serialization structure for exceptions is verbose see the org.teiid.client.util.ExceptionHolder class where we send both a byte form and a deconstructed form in case the child exception classes do not exist on the client.  It doesn't seem like the overhead should be that high though.  I can investigate some on that.

             

            As for the data legth value it is coming from org.teiid.net.socket.OioOjbectChannelFactory.MAX_OBJECT_SIZE, which can easily - and probably should - be converted into a configurable parameter.  There are already other settable values on the factory class.  The protocol sends the object payload size value ahead of the data so that the client does not risk going out of memory by reading too large of a result.  The size is appropriate for a stock installation of teiid, but may be too small in specific circumstances, such as with additional warnings and/or large batch sizes.  Can you log a JIRA?

             

            If you are getting an OOM on the server side, it would be good to analyze the heap dump and see if Teiid is a contributor to the problem.  In some cases source systems, especially stock mysql jdbc clients, are exceptional poor at handling large results and are a major cause of OOM errors.

             

            Steve

            • 3. Re: Size limitations of addWarning in ExecutionContext
              rokhmanov

              We just got a fresh OOM and I was able to preserve a good heapdump. Please see attached two reports I made - "Top Consumers" and "Leak Suspects". The heapdump itself is 36MB, too big to attach to discussion thread. I can upload it, let me know if you have any open ftp or drop place.

              The Leak Suspects shows that TeiidSQLWarnings points to one huge instance of CompactObjectOutputStream from Netty package. The thing I cannot explain yet is the fact that this OOM happened even after we put a limitation on number of addWarning() calls we are making (it was only 4 warnings added by us, each of the size of 4364 bytes, I see it in logs). The heapdump shows 116 instances of TeiidSQLWarning which occupy about 120MB of heap. Can teiid itself add warnings to ExecutionContext?

              • 4. Re: Size limitations of addWarning in ExecutionContext
                shawkins

                The serialized size of a typical Exception used as a warning will be around 10-20kb but could be much larger depending on the number of child exceptions and overall stacktrace depths.  So yes it is quite possible that too many warnings would be an issue with the 32 Mb message size limit.

                 

                For JDBC translators Teiid will add warnings to the ExecutionContext if they are present on the source Statement - see JDBCBaseExecution.addStatementWarnings.  What Execption instance is the warning you're adding?  A TeiidSQLWarning is only used by Teiid in our client so it's confusing to see them in the what is presumably the server VM.  Is this from the server side and are you making a socket connection to a Teiid instance?  CompactObjectOutputStreams are ephemeral, so all of the memory/references they hold are for the current out going messages and it certainly does look like the memory footprint and subsequent byte size of the TeiidSQLWarnings is much too large.

                 

                Steve

                • 5. Re: Size limitations of addWarning in ExecutionContext
                  markaddleman

                  We're adding a subclass of SQLWarning to the context.  There's no real reason for that, we could just as easily (probably more so) use a subclass of Exception.  The warning objects are typically nor very large and we never care about (and don't explicitly include) the stack trace but it looks like Teiid is using standard Java serialization which, of course, will serialize the stack trace.  Indeed, that could be the source of the problem.

                   

                  In the case where we're seeing the OOM problems, the translator is not a JDBCExecutionFactory, so I don't think the SQLWarning bundling behavior applies here.

                   

                  Teiid is running in the same JVM as the client (our server code).  We are connecting to teiid using TeiidDataSource and specifying "localhost" + port.  For various reasons, we want to end up with an embedded Teiid connection.  Is this not the right way?

                  • 6. Re: Size limitations of addWarning in ExecutionContext
                    shawkins

                    Mark,

                     

                    It doesn't matter which subclass it is, I just wanted to verify that is was still just the Teiid client logic creating the TeiidSQLWarning instances.  Confirming that you are making a socket connection implies a possible flow within the vm of

                     

                    (source transaltor generating warnings) -> (teiid engine) -> (teiid embedded or socket client) -> (jdbc translator ?) -> (teiid engine) -> (socket client)

                     

                    So while the original translator may not be JDBC, I'm guessing that you do have an intermediate JDBC translator that is automatically picking up of the source warnings that have been attached to the ExecutionContext and then delivering them over a socket client.  That would explain why TeiidSQLWarning instances are showing up in an output stream.  Is this an accurate picture?  Are both clients socket clients? 

                     

                    Yes, you do want to use an embedded connection within the vm.  That will save you quite a bit of serialization overhead.  However there is still a question as to why the intermediate TeiidSQLWarnings are so large.  I think part of the answer lies with JDBCBaseExecution.addStatementWarnings.  The logic there is attempting to flatten the warning chain, but the call to setNextException appends so what we end up with is that each of warnings will be serialized with all of the subsequent warnings still attached.  You can verify this at the end client by examining the TeiidSQLWarning cause and seeing it is a redundant warning chain.

                     

                    So I think there are two issues here.  Bad handling of JDBC warnings and the need to add configurablity of the max object size.  I'll log a JIRA to cover these if this seems to fit what is happening.

                     

                    Steve

                    • 7. Re: Size limitations of addWarning in ExecutionContext
                      markaddleman

                      This is embarrassing:  you are beginning to understand our application better than I do...  Its quite possible that we're going through our recursive translator in the oom case.  That translator is a Teiid execution factory, so your hypothesis mashes sense.  I'll double check once i'm in the office.

                       

                      Regarding the embedded connection, how do we ensure that we're using an embedded connection?

                      • 8. Re: Size limitations of addWarning in ExecutionContext
                        shawkins

                        See https://issues.jboss.org/browse/TEIID-2007 and https://issues.jboss.org/browse/TEIID-2008 for the respective issues.  In particular TEIID-2007 should address the OOM shown in the heap dump and remove the need to change the client setting from TEIID-2008.

                         

                        Embedded connections simply don't set host/ports.  See http://docs.jboss.org/teiid/7.7.0.Final/client-developers-guide/en-US/html/teiid_connection.html#local_connection

                         

                        Steve

                        • 9. Re: Size limitations of addWarning in ExecutionContext
                          rokhmanov

                          I'll verify the changes for TEIID-2007 and TEIID-2008 after they will be available. Thanks Steven.