2 Replies Latest reply on Jul 28, 2015 4:57 PM by shawkins

    memory footprint of teiid representation of records relative to rdbms

    rupeshsingh01

      Hi all,

       

      I am trying to evaluate the memory footprints of teiid embedded runtime representation of data from a rdbms source. It will help in capacity planning.

       

      I am doing as below:

       

      public static void main(String[] args) throws Exception {

             

        TeiidEmbeddedStopIt teiidEmbedded = new TeiidEmbeddedStopIt();

       

            DataSource ds = EmbeddedHelper.newDataSource("org.h2.Driver", "jdbc:h2:tcp://localhost/~/test", "sa", null);

                memory("PROGRAM STARTED ");

              EmbeddedServer server = new EmbeddedServer();

             H2ExecutionFactory executionFactory = new H2ExecutionFactory() ;

             executionFactory.setSupportsDirectQueryProcedure(true);

             executionFactory.start();

             server.addTranslator("translator-h2", executionFactory);

             server.addConnectionFactory("java:/example-ds", ds);

             EmbeddedConfiguration config = new EmbeddedConfiguration();

             config.setTransactionManager(EmbeddedHelper.getTransactionManager());

             server.start(config);

              memory("TEEID  SERVER STARTED ");

              server.deployVDB(new FileInputStream(new File("xxxx")));

              memory("TEEID  VDB DEPLOYED ");

              teiidEmbedded.queryHoldings(server);

        }

      public void queryHoldings(EmbeddedServer server )  throws Exception{

              long start = System.currentTimeMillis();

              memory("getting connection etc");

              MyDataSource source = new MyDataSource(server);

              JdbcTemplate template = new JdbcTemplate(source);

              memory("QUERY to be performed ");

              List<Map<String, Object>> listOfMap =  template.queryForList("select * from HoldingsData");

              memory("QUERY PERFORMED ");

              System.out.println("Time Taken in retrieving data = " + (System.currentTimeMillis()-start));

              System.out.println("no of records retrieved = " +listOfMap.size());

              System.out.println("records retrieved = " +listOfMap);

              counter++;

              System.out.println("counter = "+counter);

          }

       

             

      memory unit is MBmemory used when TEIID server startedconsumed memory in TEIID Embedded Runtime(after VDB deployment)size of runtime when query on vdb to be performedsize of runtime when query on vdb performedsize of data in databasememory representation of data in TEIID Runtime
      case 1 (with joins)222423240.001251
      case 2 (without join)222423881.138865
      case1:Note: 41 records are retrieved from database using simple joins among 4 tables. Each record is around 32 bytes.
      32*41=1312 bytes = 0.00125122 MB
      case 2:199028 object (each of size 6 bytes)

       

       

      inconsistent results are coming.


      Can anyone tell the memory representation of data in TEIID Runtime ? Is there any other way to evaluate it?  it would help in capacity planning. Any suggestion/pointer is welcome.

        • 1. Re: memory footprint of teiid representation of records relative to rdbms
          shawkins

          > Can anyone tell the memory representation of data in TEIID Runtime ?

           

          There are a lot of considerations here, including but not limited to:

           

          - The in-memory rows will be held as java objects, which will typically have a different footprint than their database counterparts (usually some overhead). 

          - Teiid uses a tiered memory architecture.  There is a variable heap portion, a fixed memory buffer (that can optionally be off heap), and spillover to disk.  Usage of each of the areas will depend upon the processing and data retention needs of the query being processed.  A largely streaming result will be processed a batch (with a prefetch) at a time from the source and consume very little memory resources.  The numbers you are showing don't seem to reflect the fixed memory buffer though as you would expect it to be larger than 22 MB.

          - You need to make sure that your measurement is based upon the retained heap size.  Accumulated garbage doesn't reflect what is held in memory.

          - The amount of memory consumption will be variable during processing.  Sort operations (especially large) can be memory intensive, but once the the final output has been produced (unless a non-forward only result set is requested) the memory consumption can steadily drop.

           

          > Is there any other way to evaluate it?

           

          For the most part you are looking for query performance rather as the end metric.  You want to make sure that you are seeing a desirable number of queries per second for a given scenario.  If there is any doubt then you'll likely want to validate the query plan to ensure it looks as efficient as possible.

           

          > it would help in capacity planning. Any suggestion/pointer is welcome.

           

          Also have a look at: https://docs.jboss.org/author/display/TEIID/Memory+Management

          1 of 1 people found this helpful
          • 2. Re: memory footprint of teiid representation of records relative to rdbms
            rupeshsingh01

            Thanks Steven for answering my queries !!!

            For information, I was running with 247 MB of Max memory of JVM, deployed simple vdb having 2 models only and query was "select * from HoldingsData".

            I was getting heap space OOM also when running the same query over 5000 times.  Will run my tests after going through Memory managements topics..

            thanks

            - Rupesh

            • 3. Re: memory footprint of teiid representation of records relative to rdbms
              shawkins

              > I was getting heap space OOM also when running the same query over 5000 times.

               

              That is a very constrained VM size, but you still wouldn't expect an OOM.  The first thing to do is to have heap dump on out of memory enabled for your vm.  With a VM that small it should be easy to see what is holding the memory.

              1 of 1 people found this helpful