2 Replies Latest reply on Apr 2, 2012 6:27 PM by aureliocalegari

    RHQ 4.3.0 not discovering JVM on Solaris?

    aureliocalegari

      Hello,

       

      I'm trying to make the JMX Agent recognise a our server in a Solaris 10 box. I have no problem with the very same configuration in a windows box. I can connect though JVisualVM and JConsole but the agent is failing to add to the inventory.

       

      Heres the techs:

      - RHQ 4.3.0

      - Agent running on a JVM 1.6 (update 31) for Solaris

      - Servers I want to monitor are based on OC4J and run on JVM 5. I have added as part of the properties the JMX remoting (can connect to them using other tools as above). The agent is throwing the following:

       

      2012-03-30 11:04:12,175 WARN  [RHQ Agent Prompt Input Thread] (rhq.core.pc.util.DiscoveryComponentProxyFactory)- The discovery component for resource type [ResourceType[id=0, name=JMX Server, plugin=JMX, category=Server]] has been blacklisted

      2012-03-30 11:04:12,183 WARN  [RHQ Agent Prompt Input Thread] (rhq.core.pc.inventory.InventoryManager)- Discovery for Resources of [ResourceType[id=0, name=JMX Server, plugin=JMX, category=Server]] has been running for more than 300000 milliseconds. This may be a plugin bug.

      org.rhq.core.pc.inventory.TimeoutException: Call to [org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources()] with args [[org.rhq.core.pluginapi.inventory.ResourceDiscoveryContext@3fb547]] timed out. Invocation thread will be interrupted.

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread(DiscoveryComponentProxyFactory.java:243)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke(DiscoveryComponentProxyFactory.java:207)

          at $Proxy54.discoverResources(Unknown Source)

          at org.rhq.core.pc.inventory.InventoryManager.invokeDiscoveryComponent(InventoryManager.java:321)

          at org.rhq.enterprise.agent.promptcmd.DiscoveryPromptCommand.discoveryForSingleResourceType(DiscoveryPromptCommand.java:362)

          at org.rhq.enterprise.agent.promptcmd.DiscoveryPromptCommand.discovery(DiscoveryPromptCommand.java:310)

          at org.rhq.enterprise.agent.promptcmd.DiscoveryPromptCommand.processCommand(DiscoveryPromptCommand.java:218)

          at org.rhq.enterprise.agent.promptcmd.DiscoveryPromptCommand.execute(DiscoveryPromptCommand.java:95)

          at org.rhq.enterprise.agent.AgentMain.executePromptCommand(AgentMain.java:2878)

          at org.rhq.enterprise.agent.AgentMain$5.run(AgentMain.java:2786)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: java.lang.Exception: Thread[ResourceDiscoveryComponent.invoker.daemon-1,5,main] with id [24] is hung. This exception contains its stack trace.

          at org.hyperic.sigar.Sigar.getProcPort(Native Method)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.getServerSockets(JMXDiscoveryComponent.java:494)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResourceDetails(JMXDiscoveryComponent.java:382)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources(JMXDiscoveryComponent.java:139)

          at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:292)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          ... 1 more

      2012-03-30 11:04:12,274 WARN  [InventoryManager.discovery-1] (rhq.core.pc.util.DiscoveryComponentProxyFactory)- The discovery component for resource type [ResourceType[id=0, name=JMX Server, plugin=JMX, category=Server]] has been blacklisted

      2012-03-30 11:04:12,277 WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Discovery for Resources of [ResourceType[id=0, name=JMX Server, plugin=JMX, category=Server]] has been running for more than 300000 milliseconds. This may be a plugin bug.

      org.rhq.core.pc.inventory.TimeoutException: Call to [org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources()] with args [[org.rhq.core.pluginapi.inventory.ResourceDiscoveryContext@1389244]] timed out. Invocation thread will be interrupted.

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invokeInNewThread(DiscoveryComponentProxyFactory.java:243)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ResourceDiscoveryComponentInvocationHandler.invoke(DiscoveryComponentProxyFactory.java:207)

          at $Proxy54.discoverResources(Unknown Source)

          at org.rhq.core.pc.inventory.InventoryManager.invokeDiscoveryComponent(InventoryManager.java:321)

          at org.rhq.core.pc.inventory.InventoryManager.executeComponentDiscovery(InventoryManager.java:2348)

          at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.pluginDiscovery(AutoDiscoveryExecutor.java:185)

          at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:104)

          at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: java.lang.Exception: Thread[ResourceDiscoveryComponent.invoker.daemon-2,5,main] with id [41] is hung. This exception contains its stack trace.

          at org.hyperic.sigar.Sigar.getProcPort(Native Method)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.getServerSockets(JMXDiscoveryComponent.java:494)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResourceDetails(JMXDiscoveryComponent.java:382)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources(JMXDiscoveryComponent.java:139)

          at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:292)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          ... 3 more

      2012-03-30 11:04:12,281 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Discovery for [JMX Server] resources took [300009] ms

      2012-03-30 11:04:12,374 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Detected new Server [Resource[id=0, type=RHQ Agent, key=atsapdod05 RHQ Agent, name=RHQ Agent, parent=<null>, version=4.3.0]] - adding to local inventory...

      2012-03-30 11:04:12,766 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Detected new Server [Resource[id=0, type=Samba Server, key=/etc/samba/smb.conf, name=atsapdod05, parent=<null>]] - adding to local inventory...

      2012-03-30 11:04:12,776 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Detected new Server [Resource[id=0, type=Postfix Server, key=/etc/postfix/main.cf, name=atsapdod05, parent=<null>]] - adding to local inventory...

      2012-03-30 11:04:12,861 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Detected new Server [Resource[id=0, type=Cron, key=Cron, name=Cron, parent=<null>]] - adding to local inventory...

      2012-03-30 11:04:12,926 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=httpd.*,process|basename|nomatch|parent=httpd.*], name=[HttpdOnUnix]], discovered-process=[process: pid=[11641], name=[/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker], ppid=[26668]]

      2012-03-30 11:04:13,848 WARN  [ResourceDiscoveryComponent.invoker.daemon-3] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- Discovery of Apache process [process: pid=[11641], name=[/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker], ppid=[26668]] failed: '/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker' is not a valid Apache executable (java.lang.Exception: Unable to find 'Apache/' in: /usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker).

      2012-03-30 11:04:13,849 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [server] inventory report to Server...

      2012-03-30 11:04:13,887 WARN  [ResourceDiscoveryComponent.invoker.daemon-3] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- Discovery of Apache process [process: pid=[11641], name=[/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker], ppid=[26668]] failed: Apache null is not suppported.

      2012-03-30 11:04:15,354 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Syncing local inventory with Server inventory...

      2012-03-30 11:04:15,378 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Found 1 servers.

      2012-03-30 11:04:15,378 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running runtime discovery scan rooted at [platform]

      2012-03-30 11:04:15,380 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Scanned [0] servers and found [0] total descendant Resources.

      2012-03-30 11:04:15,381 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [runtime] inventory report to Server...

      2012-03-30 11:04:15,419 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Syncing local inventory with Server inventory...

      2012-03-30 11:04:20,384 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Running runtime discovery scan rooted at [platform]

      2012-03-30 11:04:20,386 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Scanned [0] servers and found [0] total descendant Resources.

      2012-03-30 11:04:20,386 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [runtime] inventory report to Server...

      2012-03-30 11:04:20,427 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Syncing local inventory with Server inventory...

      2012-03-30 11:04:28,314 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333065800887][March 30, 2012 11:03:20 AM EST], Agent=[1333065868306][March 30, 2012 11:04:28 AM EST]

      2012-03-30 11:05:28,348 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333065860933][March 30, 2012 11:04:20 AM EST], Agent=[1333065928345][March 30, 2012 11:05:28 AM EST]

      2012-03-30 11:06:28,425 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333065920970][March 30, 2012 11:05:20 AM EST], Agent=[1333065988422][March 30, 2012 11:06:28 AM EST]

      2012-03-30 11:07:28,488 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333065981073][March 30, 2012 11:06:21 AM EST], Agent=[1333066048485][March 30, 2012 11:07:28 AM EST]

      2012-03-30 11:08:28,735 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066041319][March 30, 2012 11:07:21 AM EST], Agent=[1333066108731][March 30, 2012 11:08:28 AM EST]

      2012-03-30 11:09:28,811 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066101395][March 30, 2012 11:08:21 AM EST], Agent=[1333066168807][March 30, 2012 11:09:28 AM EST]

      2012-03-30 11:10:28,939 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066161516][March 30, 2012 11:09:21 AM EST], Agent=[1333066228936][March 30, 2012 11:10:28 AM EST]

      2012-03-30 11:11:28,978 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066221563][March 30, 2012 11:10:21 AM EST], Agent=[1333066288975][March 30, 2012 11:11:28 AM EST]

      2012-03-30 11:12:29,036 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066281616][March 30, 2012 11:11:21 AM EST], Agent=[1333066349032][March 30, 2012 11:12:29 AM EST]

      2012-03-30 11:13:29,107 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333066341691][March 30, 2012 11:12:21 AM EST], Agent=[1333066409103][March 30, 2012 11:13:29 AM EST]

      2012-03-30 11:13:36,995 ERROR [ResourceDiscoveryComponent.invoker.daemon-1] (org.rhq.plugins.jmx.JMXDiscoveryComponent)- Failed to determine JVM version.

      java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted]

          at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)

          at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)

          at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.buildResourceDetails(JMXDiscoveryComponent.java:430)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResourceDetails(JMXDiscoveryComponent.java:402)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources(JMXDiscoveryComponent.java:139)

          at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:292)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted]

          at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)

          at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)

          at javax.naming.InitialContext.lookup(InitialContext.java:392)

          at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)

          at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)

          at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)

          ... 14 more

      Caused by: java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted

          at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:614)

          at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)

          at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)

          at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)

          at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)

          at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)

          ... 19 more

      Caused by: java.io.InterruptedIOException: operation interrupted

          at java.net.PlainSocketImpl.socketConnect(Native Method)

          at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

          at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

          at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

          at java.net.Socket.connect(Socket.java:529)

          at java.net.Socket.connect(Socket.java:478)

          at java.net.Socket.<init>(Socket.java:375)

          at java.net.Socket.<init>(Socket.java:189)

          at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)

          at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)

          at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)

          ... 24 more

      2012-03-30 11:13:49,787 ERROR [ResourceDiscoveryComponent.invoker.daemon-2] (org.rhq.plugins.jmx.JMXDiscoveryComponent)- Failed to determine JVM version.

      java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted]

          at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)

          at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)

          at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.buildResourceDetails(JMXDiscoveryComponent.java:430)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResourceDetails(JMXDiscoveryComponent.java:402)

          at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources(JMXDiscoveryComponent.java:139)

          at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:292)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted]

          at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)

          at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)

          at javax.naming.InitialContext.lookup(InitialContext.java:392)

          at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)

          at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)

          at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)

          ... 14 more

      Caused by: java.rmi.ConnectIOException: Exception creating connection to: 127.0.0.1; nested exception is:

          java.io.InterruptedIOException: operation interrupted

          at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:614)

          at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)

          at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)

          at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)

          at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)

          at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)

          ... 19 more

      Caused by: java.io.InterruptedIOException: operation interrupted

          at java.net.PlainSocketImpl.socketConnect(Native Method)

          at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

          at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

          at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

          at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

          at java.net.Socket.connect(Socket.java:529)

          at java.net.Socket.connect(Socket.java:478)

          at java.net.Socket.<init>(Socket.java:375)

          at java.net.Socket.<init>(Socket.java:189)

          at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)

          at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)

          at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)

          ... 24 more

       

      And when I try to add manually I get the following:

       

      add failed for resource of type [JMX Server] and parent resource id [11611]

      org.rhq.core.clientapi.agent.PluginContainerException: The [JMX] plugin does not properly support manual addition of [JMX Server] Resources.

          at org.rhq.core.pc.inventory.InventoryManager.manuallyAddResource(InventoryManager.java:694)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)

          at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)

          at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)

          at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)

          at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)

          at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)

          at $Proxy0.execute(Unknown Source)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)

          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)

          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)

          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)

          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)

      2012-03-30 11:48:56,361 INFO  [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementSenderRunner)- Measurement collection for [49] metrics took 119ms - sending report to Server...

      2012-03-30 11:49:01,823 INFO  [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Sending availability report to Server...

      2012-03-30 11:49:16,856 ERROR [RHQ Server Polling Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.time-not-synced}The server and agent clocks are not in sync. Server=[1333068489440][March 30, 2012 11:48:09 AM EST], Agent=[1333068556853][March 30, 2012 11:49:16 AM EST]

      2012-03-30 11:49:18,659 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Executing server discovery scan...

      2012-03-30 11:49:19,137 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[7878], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,137 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[26532], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[14664]]

      2012-03-30 11:49:19,137 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28526], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,138 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28525], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,138 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28520], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,138 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28521], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,139 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28531], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,139 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[28003], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,139 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[24284], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,139 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[7548], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,140 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[24560], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk/bin/java], ppid=[1]]

      2012-03-30 11:49:19,140 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[7790], name=[/usr/local/oracle/ATSQA02/j2ee_q03_10.1.3/jdk1.6.0_17/bin/java], ppid=[1]]

      2012-03-30 11:49:19,140 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=^java.*], name=[java]], discovered-process=[process: pid=[13524], name=[/users/atsjq03/rhq-enterprise-agent/jdk1.6.0_31/bin/java], ppid=[13497]]

      2012-03-30 11:49:19,343 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Process scan auto-detected new server resource: scan=[ProcessScan: query=[process|basename|match=httpd.*,process|basename|nomatch|parent=httpd.*], name=[HttpdOnUnix]], discovered-process=[process: pid=[11641], name=[/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker], ppid=[26668]]

      2012-03-30 11:49:19,376 WARN  [ResourceDiscoveryComponent.invoker.daemon-8] (org.rhq.plugins.apache.ApacheServerDiscoveryComponent)- Discovery of Apache process [process: pid=[11641], name=[/usr/local/oracle/ATSQA02/ohsj_q03_10.1.3/ohs/bin/httpd.worker], ppid=[26668]] failed: Apache null is not suppported.

      2012-03-30 11:49:19,376 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [server] inventory report to Server...

      2012-03-30 11:49:19,487 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Syncing local inventory with Server inventory...

      2012-03-30 11:49:19,510 INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Found 0 servers.

      2012-03-30 11:49:29,540 INFO  [WorkerThread#0[10.35.12.74:41748]] (rhq.core.pc.inventory.InventoryManager)- Plugin Error: During manual add, discovery component method [org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResource()] returned null (either the Resource type was blacklisted or the plugin developer did not implement support for manually discovered Resources correctly).

      2012-03-30 11:49:29,549 ERROR [WorkerThread#0[10.35.12.74:41748]] (rhq.core.pc.inventory.InventoryManager)- Manual add failed for resource of type [JMX Server] and parent resource id [11611]

      org.rhq.core.clientapi.agent.PluginContainerException: The [JMX] plugin does not properly support manual addition of [JMX Server] Resources.

          at org.rhq.core.pc.inventory.InventoryManager.manuallyAddResource(InventoryManager.java:694)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)

          at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)

          at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)

          at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)

          at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)

          at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)

          at $Proxy0.execute(Unknown Source)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)

          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)

          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)

          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)

          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)

      2012-03-30 11:49:53,602 INFO  [WorkerThread#0[10.35.12.74:41748]] (rhq.core.pc.inventory.InventoryManager)- Plugin Error: During manual add, discovery component method [org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResource()] returned null (either the Resource type was blacklisted or the plugin developer did not implement support for manually discovered Resources correctly).

      2012-03-30 11:49:53,602 ERROR [WorkerThread#0[10.35.12.74:41748]] (rhq.core.pc.inventory.InventoryManager)- Manual add failed for resource of type [JMX Server] and parent resource id [11611]

      org.rhq.core.clientapi.agent.PluginContainerException: The [JMX] plugin does not properly support manual addition of [JMX Server] Resources.

          at org.rhq.core.pc.inventory.InventoryManager.manuallyAddResource(InventoryManager.java:694)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)

          at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)

          at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)

          at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)

          at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)

          at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)

          at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)

          at $Proxy0.execute(Unknown Source)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290)

          at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184)

          at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)

          at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)

          at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)

          at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)

        • 1. Re: RHQ 4.3.0 not discovering JVM on Solaris?
          ips

          From your logs, it looks like the root case is that Sigar.getProcPort() is hanging and causing the discovery thread to be timed out by the plugin container:     

           at org.hyperic.sigar.Sigar.getProcPort(Native Method)       
           at org.rhq.plugins.jmx.JMXDiscoveryComponent.getServerSockets(JMXDiscoveryComponent.java:494)      
           at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResourceDetails(JMXDiscoveryComponent.java:382)      
           at org.rhq.plugins.jmx.JMXDiscoveryComponent.discoverResources(JMXDiscoveryComponent.java:139)  
          

          I'm guessing there might be a bug in Sigar.getProcPort() on Solaris.  In any case, the getServerSockets() method has been completely removed in RHQ master, so there's a good chance you will no longer see this issue in RHQ 4.4 (or if you build RHQ yourself from master).

          • 2. Re: RHQ 4.3.0 not discovering JVM on Solaris?
            aureliocalegari

            Thanks Ian! Once I checkout and build, do I need to go for a plain vanila new installation or I can just replace the server binaries and restart it? Is there a specific jar I could replace to avoid the hassle of a new installation?

             

            Cheers

            Aurelio