4 Replies Latest reply on Jan 20, 2012 9:56 PM by genman

    MeasurementFacet; can query take > 5 minutes?

    genman
    genman Jan 19, 2012 7:18 PM

      I am querying Hadoop using Hive and trying to return back data from a SELECT statement as a series of measurements.

       

      The select can take quite a long time, especially if the cluster is busy.

       

      I was thinking of using the 'org.rhq.plugins.database.CustomTableComponent' to perform the query, but the query may not be appropriate for the 'getValues' method. Is there a suggested approach?

       

      One would be to adjust the database plugin to store the results of the query in a separate timer thread. This could be enabled or disabled by configuration. Should I make this change to the database plugin?

       

      Thoughts?

        • 1. Re: MeasurementFacet; can query take > 5 minutes?
          mazz
          mazz Jan 20, 2012 9:07 AM (in response to genman)

          > One would be to adjust the database plugin to store the results of the query in a separate timer thread.

          > This could be enabled or disabled by configuration. Should I make this change to the database plugin?

           

          That's that way I would think you would have to do it. This is analogous to what we did with the asynchronous availability collector (we had the same problem when collecting availability - some resources took too long to even respond to what is supposed to be a quick avail check). See : http://rhq-project.org/display/RHQ/Design-Asynchronous+Availability+Collector and BZ 536173 for how we solved that. We wrote some utility class to do it (so any plugin can reuse the functionality) and plugins are free to incorporate it to pick up that ability to run avail checks asynchronously.

            • Actions
          • 2. Re: MeasurementFacet; can query take > 5 minutes?
            genman
            genman Jan 20, 2012 2:35 PM (in response to mazz)

            Looking at the implementation, I'm not sure why you are using Executor not ScheduledExecutorService for the availability checks. Basically you've reinvented the wheel (timer) here. It's probably not the end of the world to create a dozen or so threads but not really sensible either.

             

            The benefit with ScheduledExecutorService is that it can also perform thread interruption using FutureTask.cancel(true) which plugins will probably want to use.

             

            The biggest impact would be to the "ResourceContext" class constructor, but shouldn't impact plugins at all as ScheduledExecutorService extends Executor.

             

            AvailabilityCollectorRunnable should probably be changed to use this as well.

              • Actions
            • 3. Re: MeasurementFacet; can query take > 5 minutes?
              mazz
              mazz Jan 20, 2012 4:01 PM (in response to genman)

              feel free to open up a bugzilla and submit enhancement patches :)

              1 of 1 people found this helpful
                • Actions
              • 4. Re: MeasurementFacet; can query take > 5 minutes?
                genman
                genman Jan 20, 2012 9:56 PM (in response to mazz)

                See: https://bugzilla.redhat.com/show_bug.cgi?id=783603

                 

                Before I send in my patch, I gotta get my changes approved for release so it can take a while. They are pretty straightforward changes, though.

                  • Actions