1 2 3 Previous Next 36 Replies Latest reply on Aug 15, 2012 10:51 AM by shawkins Go to original post Branched to a new discussion.
      • 15. Re: Continuous request against Procedure Execution
        rokhmanov

        Steven Hawkins wrote:

         

        Coming back to the suggestion.  Without having a different DataNotAvailable mechanism (such as one based upon a do not call me until Date similar to the java concurrent logic) the execution may get called early.   That leaves the approximation mechanism as not relying on the polling delays (and using his own timer or whatever to have full control) or alternatively he can recalculate the polling delay with each attempt at an execution before rethrowing an exception.

         

        I followed suggestion to use combination of DataNotAvailableException.NO_POLLING + ExecutionContext.dataAvailable()  and java.util.Timer. I see the same behavior I observed earlier - the DataNotAvailableException.NO_POLLING does not work as it is supposed to be. Here is my sample implementation:

         

        1. My ReusableExecution translator makes only one row of data per every execution();
        2. The next() method delivers this row of data. After the row delivered, the row collection is set to null;
        3. If during the next() execution the row is null - throw DataNotAvailableException.NO_POLLING;
        4. A single instance of Timer gets created per translator and runs ExecutionContext.dataAvailable() with predefined poll interval;

         

        Here is my next() method:

             @Override
             public List next() throws TranslatorException, DataNotAvailableException {          
                  try {
                       return timeRow;                    
                  } finally {
                       if (null == timeRow)
                       {
                              throw DataNotAvailableException.NO_POLLING;
                       }
                       timeRow = null;
                  }
             }
        

        As you see, it is very simple. Behavior I observe:

        1. I expect the execute() method called after the Timer fires up the dataAvailable(). I  see no traces of execute() in logs or in debug, it runs only once;
        2. Per my implementation above, the DataNotAvailableException.NO_POLLING executed twice (as I see it in logs), both times - when next() returns the null. It is a long-time question for me why Teiid cannot be satisfied with a single null returned from next() and attempts to run next() again;
        3. If I artificially make my next() method return a null only once (see below) - the DataNotAvailable.NO_POLLING exception gets ignored, means no pause between returned rows for client - wrong behaviour for me;
         
             @Override
             public List next() throws TranslatorException, DataNotAvailableException {          
                  try {
                       return timeRow;                    
                  } finally {
                       if (null == timeRow) && awakeMode.get())
                       {
                            awakeMode.set(false);
                              throw DataNotAvailableException.NO_POLLING;
                       }
                       timeRow = null;
                  }
             }
        

        The "awakeMode" is AtomicBoolean, set to true by default, and during every Timer execution. The complete source of my translator (TimerExecution.java) is attached. Most important for understanding are execute(), next() and initTimer() methods, the rest can be omitted.

         

        I am still not sure if the behavior above is a defect, non-implemented feature, or my wrong implementation. I did not open any Jira yet, I'd like to get a comments from Teiid developers, please.

         

        We are quickly running out of time, and currently I am not confident if I can make Teiid to do what we need. Probably soon we will have to make a decision to abandon the Teiid continuous execution feature and implement needed functionality on application side (this is unpleasant, but I am currently out of ideas how to make this work).

         

        Thanks,

        Andriy

        • 16. Re: Continuous request against Procedure Execution
          rareddy

          I think there is no need for second null return from your next call, it needs to retun DataNotAvailableException.NO_POLLING. May be something like this works?

           

               @Override
               public List next() throws TranslatorException, DataNotAvailableException
               {          
                    try {
                         if (sleeping.get()){
                             throw DataNotAvailableException.NO_POLLING;
                         }
                         sleeping.set(true);
                         return timeRow;
                    }
               }
          
          
          private void initTimer()
          {
              if (null == timer)
              {
                    new Timer().schedule(new TimerTask(){
                      @Override
                      public void run() {
                          System.out.println("############## Fire!...");                        
                          executionContext.dataAvailable();
                          sleeping.set(false);
                      }                    
                  }, period, period);            
              }
          }
          
          @Override
          public void execute() throws TranslatorException {
              System.out.println("##### Execute...");
              if ( hasMoreRows){
                  timeRow = new ArrayList<Long>(3);
                  timeRow.add(start);
                  timeRow.add(start += period);
                  timeRow.add(period);        
                  sleeping.set(true);       
              } 
              else {
                  timeRow = null;
              }
          }
          
          

           

           

          Ramesh..

          • 17. Re: Continuous request against Procedure Execution
            rokhmanov

            Ramesh, yes, my second chunk of code from my message above does exactly this - only one null returned. Please take a look again on a list of behaviors, #3.

            • 18. Re: Continuous request against Procedure Execution
              rareddy

              Andrily,

               

              I am not sure why you say (3) only returns the null once, every time "next()" is called then either a non-null result or null  returned along with optional DataNotAvailableException.NO_POLLING. Unless I am missing something obvious, I am not even sure how a return and as well as throwing an exception from finally block on the same method call behave in a java program.

               

              Ramesh..

              • 19. Re: Continuous request against Procedure Execution
                rokhmanov

                Ramesh,

                 

                I do not see anything specific about throwing exceptions from final block, it is a perfectly valid construction in java. Please run attached TimerExecution.java.zip and let me know what you're getting?

                About returning null once: the next() method once called supposed to return a null or row of data. Or throw an exception. The sample of code you put above works this way: after a row of data delivered, the DataNotAvailableException will be thrown every subsequent next() call. As a result - the execute() method will be never called again, even if Timer fires up, exactly same behaviour as I explained.

                • 20. Re: Continuous request against Procedure Execution
                  rareddy

                  Now that I read the document https://docs.jboss.org/author/display/TEIID/Executing+Commands, see ReusableExecutions. I can make sense of it now little better. See if the below works.

                   

                  Ever execution cycle in continuous execution works the same as normal executions in ExecutionFactory, calls "execute" first, then call "next" until the final row is returned with null value, then repeat process starting with "reset" call.

                   

                   

                       @Override
                       public List next() throws TranslatorException, DataNotAvailableException
                       {         
                            try {
                                if (hasData && !sleeping.get()){
                                    if (timeRow == null){
                                        hasData = false;
                                    }
                                    List results = timeRow;
                                    timeRow = null;
                                    return timeRow;
                                }
                               throw DataNotAvailableException.NO_POLLING;
                            }
                       }
                  
                       private void initTimer() {
                            if (null == timer) {
                                 new Timer().schedule(new TimerTask() {
                                      @Override
                                      public void run() {
                                           System.out.println("############## Fire!...");
                                           executionContext.dataAvailable();
                                           sleeping.set(false);
                                      }
                                 }, period, period);
                            }
                       }
                  
                       @Override
                       public void execute() throws TranslatorException {
                            System.out.println("##### Execute...");
                            timeRow = new ArrayList(3);
                            timeRow.add(start);
                            timeRow.add(start += period);
                            timeRow.add(period);
                                 hasData = true;
                       }
                  
                       @Override
                       public void reset(Command c, ExecutionContext executionContext, Connection connection) {
                            System.out.println("############### reset");
                                  // also set initial value of sleeping to false.
                                  sleeping.set(true);
                       } 
                  
                  • 21. Re: Continuous request against Procedure Execution
                    rokhmanov

                    Ramesh, I think your next() method will never return a real data, isn't it?

                    • 22. Re: Continuous request against Procedure Execution
                      rareddy

                      Andriy,

                       

                      Actually, I have bad code it should have been..

                       

                       

                       

                           @Override
                           public List next() throws TranslatorException, DataNotAvailableException
                           {         
                                if (hasData && !sleeping.get()){
                                    if (timeRow == null){
                                        hasData = false;
                                    }
                                    List results = timeRow;
                                    timeRow = null;
                                    return results;
                                }
                                throw DataNotAvailableException.NO_POLLING;
                           }

                       

                      Ramesh..

                      • 23. Re: Continuous request against Procedure Execution
                        rokhmanov

                        I've added a "NO_POLLING" sysout right before the throw statement and run it with 2 seconds Timer interval.

                        The console output below. The execute() called once and then translator looping until the next Timer event, eating 100% CPU.

                         

                        12:21:14,774 INFO  [stdout] (Worker1_QueryProcessorQueue1) ########## init timer...
                        12:21:14,781 INFO  [stdout] (Worker1_QueryProcessorQueue1) ##### Execute...
                        12:21:14,789 INFO  [stdout] (Worker1_QueryProcessorQueue1) ############## return Results:[0, 2000, 2000]
                        12:21:14,792 INFO  [stdout] (Worker1_QueryProcessorQueue1) ############## return Results: null
                        12:21:14,831 INFO  [stdout] (Worker0_QueryProcessorQueue2) ### Retrieved row:[0, 2000, 2000]
                        12:21:14,850 INFO  [stdout] (Worker1_QueryProcessorQueue4) ############### close
                        12:21:14,858 INFO  [stdout] (Worker2_QueryProcessorQueue5) ############### reset
                        12:21:14,860 INFO  [stdout] (Worker2_QueryProcessorQueue5) ##### Execute...
                        12:21:14,862 INFO  [stdout] (Worker2_QueryProcessorQueue5) ############### NO_POLLING
                        12:21:14,871 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue5) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,875 INFO  [stdout] (Worker2_QueryProcessorQueue7) ############### NO_POLLING
                        12:21:14,876 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue7) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,880 INFO  [stdout] (Worker2_QueryProcessorQueue9) ############### NO_POLLING
                        12:21:14,884 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue9) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,888 INFO  [stdout] (Worker2_QueryProcessorQueue11) ############### NO_POLLING
                        12:21:14,890 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue11) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,898 INFO  [stdout] (Worker2_QueryProcessorQueue13) ############### NO_POLLING
                        12:21:14,899 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue13) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,903 INFO  [stdout] (Worker2_QueryProcessorQueue15) ############### NO_POLLING
                        12:21:14,904 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue15) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,917 INFO  [stdout] (Worker2_QueryProcessorQueue17) ############### NO_POLLING
                        12:21:14,919 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue17) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:14,923 INFO  [stdout] (Worker2_QueryProcessorQueue19) ############### NO_POLLING
                        12:21:14,925 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue19) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        ...
                        ... same looping skipped ...
                        ...
                        12:21:16,771 INFO  [stdout] (Worker3_QueryProcessorQueue671) ############### NO_POLLING
                        12:21:16,772 WARN  [org.teiid.CONNECTOR] (Worker3_QueryProcessorQueue671) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:16,776 INFO  [stdout] (Timer-1) ############## Fire!...
                        12:21:16,776 INFO  [stdout] (Worker3_QueryProcessorQueue673) ############## return Results:[2000, 4000, 2000]
                        12:21:16,777 INFO  [stdout] (Worker3_QueryProcessorQueue673) ############## return Results: null
                        12:21:16,781 INFO  [stdout] (Worker1_QueryProcessorQueue674) ### Retrieved row:[2000, 4000, 2000]
                        12:21:16,784 INFO  [stdout] (Worker2_QueryProcessorQueue676) ############### reset
                        12:21:16,786 INFO  [stdout] (Worker3_QueryProcessorQueue675) ############### close
                        12:21:16,789 INFO  [stdout] (Worker2_QueryProcessorQueue676) ##### Execute...
                        12:21:16,791 INFO  [stdout] (Worker2_QueryProcessorQueue676) ############### NO_POLLING
                        12:21:16,792 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue676) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:16,795 INFO  [stdout] (Worker2_QueryProcessorQueue679) ############### NO_POLLING
                        12:21:16,796 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue679) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:16,804 INFO  [stdout] (Worker2_QueryProcessorQueue681) ############### NO_POLLING
                        12:21:16,806 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue681) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        12:21:16,811 INFO  [stdout] (Worker2_QueryProcessorQueue683) ############### NO_POLLING
                        12:21:16,812 WARN  [org.teiid.CONNECTOR] (Worker2_QueryProcessorQueue683) TEIID30004 Connector returned a 0 row non-last batch: timerange1.
                        ...
                        ...skipped...
                        ...
                        • 24. Re: Continuous request against Procedure Execution
                          rareddy

                          Andriy,

                           

                          Ok, now the behavior is right out of the translator, but engine is not behaving the correctly and being too chatty and not waiting for "dataAvailable" signal before calling the "next" call. I will take look at the Teiid code and see what is happening now.

                           

                          Ramesh..

                          • 25. Re: Continuous request against Procedure Execution
                            rokhmanov

                            Thanks Ramesh, you give me hope. =)

                            Just in case, I attached a server.log with debug information from the latest execution.

                             

                            Andriy

                            • 26. Re: Continuous request against Procedure Execution
                              rareddy

                              Andriy,

                               

                              BTW, I logged https://issues.jboss.org/browse/TEIID-2130 and we are taking a look at it.

                               

                              Ramesh..

                              • 27. Re: Continuous request against Procedure Execution
                                rokhmanov

                                Ramesh,

                                 

                                I saw the TEIID-2130 was Done earlier today, so I made a new Teiid 8.1.1.CR1 build and tried the translator. Unfortunately I see a error message, which reproducible only in Teiid 8.1.1.CR1 (the previous Teiid8.1.0.Beta2 has no errors).

                                Error happens in the moment when I submit a query to translator (VDB is active, no new errors in log during bootstrap): "org.teiid.translator.TranslatorException: TEIID31108 Data Source java:/accounts-ds not accessible."

                                The "accounts-ds" is teiid embedded datasource, defined in standalone-teiid.xml, see the attachment (accounts-ds.txt). Did anything changed in teiid datasource format in 8.1.1.CR1 , any new parameters I am missing?

                                Interesting that my translator is very simple and do not use the datasource at all. My VDB model definition also included in attached accounts-ds.txt

                                 

                                Seems the problem happens only with embedded datasources under continuous execution, the regular h2 translator and h2 datasource works well (did not checked anything else). I also attache the server.log with debug output and error details.

                                • 28. Re: Continuous request against Procedure Execution
                                  shawkins

                                  Andriy,

                                   

                                  CR1/Beta2 corrects the behavior of translators marked as sourceRequired (which is the default).  If your translator does not need a source connection, then call ExecutionFactory.setSourceRequired(false).  Otherwise we'll validate that the connection factory exists during start-up (there should be a warning) and require that a connection is passed to your Execution.  From your config snippet, I see accounts-ds on the source, but accounts-local in standalone.xml, which should explain why you are getting an exception.

                                   

                                  Steve

                                  1 of 1 people found this helpful
                                  • 29. Re: Continuous request against Procedure Execution
                                    shawkins

                                    I should also add if you set source required to false and if there is a connection-jndi-name attribute on the source element, then we'll still attempt to provide that connection to your execution, but won't throw an error if it isn't available.