4 Replies Latest reply: Feb 9, 2012 11:57 AM by Elias Ross RSS

Bug with child resources never appearing

Elias Ross Master

I have a resource (basically a JMXServerComponent) where RHQ seems to 'discover' the MBean components but they never appear in the Inventory.

 

First some logs:

 

DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Discovering MBean resources with object name query template: hadoop:service=NameNode,na
me=FSNamesystemState
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Found [1] mbeans for query [hadoop:service=NameNode,name=FSNamesystemState].
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- [1] services have been added
DEBUG [InventoryManager.discovery-1] (inventory.InventoryManager)- Discovery for [FSNamesystemState] resources completed in [2] ms
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Parent [Resource[id=0, type=FSNamesystemState, key=hadoop:name=FSNamesystemState,service=NameNode, name
=NameNode Filesystem State, parent=NameNode]] must be imported/commited before service scan can run.
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Running service scan on parent resource [Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop
-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com]] looking for children of type [ResourceType[id=0, category=Service, name=NameNodeActivity, plugin=hadoop]]
DEBUG [InventoryManager.discovery-1] (inventory.InventoryManager)- Executing discovery for [NameNodeActivity] Resources...
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Discovering MBean resources with object name query template: hadoop:service=NameNode,na
me=NameNodeActivity
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Found [1] mbeans for query [hadoop:service=NameNode,name=NameNodeActivity].
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- [1] services have been added
DEBUG [InventoryManager.discovery-1] (inventory.InventoryManager)- Discovery for [NameNodeActivity] resources completed in [3] ms
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Parent [Resource[id=0, type=NameNodeActivity, key=hadoop:name=NameNodeActivity,service=NameNode, name=h
adoop:name=NameNodeActivity,service=NameNode, parent=NameNode]] must be imported/commited before service scan can run.
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Running service scan on parent resource [Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop
-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com]] looking for children of type [ResourceType[id=0, category=Service, name=NameNodeInfo, plugin=hadoop]]
DEBUG [InventoryManager.discovery-1] (inventory.InventoryManager)- Executing discovery for [NameNodeInfo] Resources...
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Discovering MBean resources with object name query template: hadoop:service=NameNode,na
me=NameNodeInfo
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- Found [0] mbeans for query [hadoop:service=NameNode,name=NameNodeInfo].
DEBUG [ResourceDiscoveryComponent.invoker.daemon-3] (jmx.MBeanResourceDiscoveryComponent)- [0] services have been added
DEBUG [InventoryManager.discovery-1] (inventory.InventoryManager)- Discovery for [NameNodeInfo] resources completed in [1] ms
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Parent [Resource[id=0, type=NameNodeActivity, key=hadoop:name=NameNodeActivity,service=NameNode, name=h
adoop:name=NameNodeActivity,service=NameNode, parent=NameNode]] must be imported/commited before service scan can run.
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Parent [Resource[id=0, type=FSNamesystemState, key=hadoop:name=FSNamesystemState,service=NameNode, name
=NameNode Filesystem State, parent=NameNode]] must be imported/commited before service scan can run.
DEBUG [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Runtime discovery scan took 592 ms.
INFO  [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Scanned [1] servers and found [2] total descendant Resources.
TRACE [InventoryManager.discovery-1] (inventory.RuntimeDiscoveryExecutor)- Runtime report contains 3 Resources with a size of 594461 bytes

 

You can see the agent sees them and sends to the server.

 

I don't see anything logged on the server as an error or warning. I don't see anything added to the database.

 

I also observe the following: I changed my plugin definition for the JMXServer to 'singleton' but the Inventory pane shows a group of resources still. This is probably unrelated, but I'd like to fix that. How is singleton treated or stored internally?

 

I guess I need to know where to look before filling out a bug report or fixing the issue.

 

Any ideas?

  • 1. Re: Bug with child resources never appearing
    Elias Ross Master

    Okay, responding to myself. It seems like the plugin on the server is no longer valid, though I do see it listed.

    2012-02-09 09:43:26,702 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=FSNamesystemState, key=hadoop:name=FSNamesystemState,service=NameNode, name=NameNode Filesystem State, parent=NameNode]] has an unknown type [ResourceType[id=0, category=Service, name=FSNamesystemState, plugin=hadoop]]. The Agent most likely has a plugin named 'hadoop' installed that is not installed on the Server. Resource will be ignored...

    2012-02-09 09:43:26,704 ERROR [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Reported resource [Resource[id=0, type=NameNodeActivity, key=hadoop:name=NameNodeActivity,service=NameNode, name=hadoop:name=NameNodeActivity,service=NameNode, parent=NameNode]] has an unknown type [ResourceType[id=0, category=Service, name=NameNodeActivity, plugin=hadoop]]. The Agent most likely has a plugin named 'hadoop' installed that is not installed on the Server. Resource will be ignored...

    2012-02-09 09:43:26,704 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] getExistingResource processing for [Resource[id=18521, type=Linux, key=vg61l01ad-hadoop002.apple.com, name=vg61l01ad-hadoop002.apple.com, parent=<null>, version=Linux 2.6.18-164.15.1.0.1.el5]]

    2012-02-09 09:43:26,704 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] id=18521: Agent claims resource is already in inventory.

    2012-02-09 09:43:26,706 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] id=18521: Found resource already in inventory with specified id

    2012-02-09 09:43:27,471 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Merging [Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com]]...

    2012-02-09 09:43:27,471 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] getExistingResource processing for [Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com]]

    2012-02-09 09:43:27,471 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] id=18523: Agent claims resource is already in inventory.

    2012-02-09 09:43:27,471 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] id=18523: Found resource already in inventory with specified id

    2012-02-09 09:43:27,473 DEBUG [org.rhq.enterprise.server.discovery.DiscoveryBossBean] Existing Resource Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com] has invalid parent type (ResourceType[id=10008, category=Platform, name=Linux, plugin=Platforms]) and so does plugin-reported Resource Resource[id=18523, type=NameNode, key=NameNode:/usr/lib/hadoop-0.20, name=NameNode, parent=vg61l01ad-hadoop002.apple.com] (ResourceType[id=0, category=Platform, name=Linux, plugin=Platforms]) - valid parent types are [[ResourceType[id=10211, category=Server, name=Hadoop, plugin=hadoop]]].

    How do I ensure the server is using the right plugin and has the right version? It may be picking up the old version somehow... Is there a way in my plugin to ensure the versions match up?

  • 2. Re: Bug with child resources never appearing
    Lukas Krejci Novice

    This is weird and I'm not sure how you got into that situation. But when things do work correctly, the server is the authoritative source of the plugins and the agent checks that it has the exact sames plugins deployed as there are on the server. it does periodical checks of this during its lifetime to pick up plugin changes on the server. You can force this check by running the "plugins update" command on the agent's command line (or you can just restart the agent, the check is done right at the startup, too).

     

    What I find strange though is that you say that you see the plugin listed but at the same time it looks like the server doesn't know about it. By "listed" you mean it's visible in the server GUI in the list of plugins or in the agent by running "plugins info" command?

  • 3. Re: Bug with child resources never appearing
    Elias Ross Master

    What I believe is going on:

    • I deployed the old 'hadoop' plugin that comes with RHQ. When deployed, the server created resources in the database with some information indicating parent/child relationships. In the old plugin, NameNode belonged to "Hadoop" server. In the new plugin NameNode belongs to the platform. (As it should be, IMHO.)
    • I deployed my new plugin that uses the same resource names, though not entirely, and in fact the same name of the plugin. Although the server and client get the new version of the code, the server thinks the resource heirarchy must not have changed. From its point of view NameNode V1 and V2 have the same child/parent relationships but this is not the case.
    • RHQ maps new versions of resources to the old version assuming the name hasn't changed. Notice the resource IDs: The NameNode resource is 'low' compared to the new MBeans.

     

    So the fix seems to be:

    • When the server detects a new version of the plugin and the resource names are the same, check to see that the parent relationships are the same.
    • ...update the parent/child relationship. If this is hard, then delete the old resource IDs.
    • Or at least document a way to deal with this condition. I assume "purging" the plugin works here. I am about to check this.

     

    Anyway, this feels like a fairly obscure bug.