1 2 3 Previous Next 32 Replies Latest reply on Jul 15, 2009 5:30 PM by mazz

    How to monitor a MBean on WebSphere

    rafaelri

      Hi all,

      I am trying to monitor a custom MBean on WebSphere. I've followed the steps described at http://management-platform.blogspot.com/2008/11/monitoring-custom-jmx-mbeans-with-jopr.html but it seems like discovery is not working neither the manual addition of the service.
      I've packaged the rhq-plugin.xml file inside a jar (named: rhq-gife-jmx-1.0.jar) and deployed it inside rhq-plugins folder. As I mentioned, it is loaded fine but does not show on resources tab.
      I even tried to run a manual update and discovery on agent console but with no success.
      I am pasting here the contents of my rhq-plugin.xml:

      <?xml version="1.0" encoding="UTF-8" ?>
      
      <plugin name="GifeJmx"
       displayName="Gife JMX Plugin"
       version="1.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns="urn:xmlns:rhq-plugin"
       xmlns:c="urn:xmlns:rhq-configuration">
      
       <depends plugin="JMX"/>
      
       <service name="GifeManagement"
       description="GifeManagement"
       discovery="org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent"
       class="org.rhq.plugins.jmx.MBeanResourceComponent"
       supportsManualAdd="true">
      
       <runs-inside>
       <parent-resource-type name="JMX Server" plugin="JMX"/>
       </runs-inside>
      
       <plugin-configuration>
       <c:simple-property name="objectName" readOnly="true" default="gife:cell=%cell%,name=PontuacaoManagement,type=PontuacaoManagement,node=%node%,process=%process%"/>
       <c:simple-property name="nameTemplate" default="PontuacaoManagement on #{server}"/>
       <c:simple-property name="descriptionTemplate" default="PontuacaoManagement on #{server}"/>
       <c:simple-property name="cell" type="string" description="Cell of the PontuacaoManagement"/>
       <c:simple-property name="server" type="string" description="Server of the PontuacaoManagement"/>
       <c:simple-property name="node" type="string" description="Server of the PontuacaoManagement"/>
       </plugin-configuration>
      
       <operation name="atualizarUltimaPontuacao"
       displayName="atualizarUltimaPontuacao"
       description="atualizarUltimaPontuacao">
       <parameters>
       <c:simple-property name="chave" type="string" required="true"/>
       <c:simple-property name="fila" type="long" required="true"/>
       <c:simple-property name="pontuacao" type="integer" required="true"/>
       </parameters>
       </operation>
       <operation name="obterUltimaPontuacao"
       displayName="obterUltimaPontuacao"
       description="obterUltimaPontuacao">
       <parameters>
       <c:simple-property name="chave" type="string" required="true"/>
       <c:simple-property name="fila" type="long" required="true"/>
       </parameters>
       <results>
       <c:simple-property name="operationResult" type="integer"/>
       </results>
       </operation>
      
       <metric property="NumeroPontuacoes"
       displayName="NumeroPontuacoes"
       defaultOn="true"
       displayType="summary"
       defaultInterval="30000"
       description="Numero Aleatorio"/>
       </service>
      
      
      </plugin>
      


      Am I missing some configuration or some entry on the xml file for the plugin? I've opened the JMX plugin file and saw an entry related to the WebSphere template but I couldn't figure out what I am missing, but I really feel like I am missing something since I've not specified the JMX url.

      best regards,
      Rafael Ribeiro

        • 1. Re: How to monitor a MBean on WebSphere
          mazz

          You do not need to specify the JMX URL in the plugin descriptor because you are reusing the connection metadata info through the use of injection into the JMX plugin resource type heirarchy (that's what the <runs-inside> does).

          The "JMX Server" server resource type provides the metadata for the connection URL.

          Now, its probably the case that the JMX plugin canNOT automatically discovery WebLogic servers. I don't know this part of the code well, but I suspect that is what is happening (typically when people say, "my agent can't auto-discovery my running application", the answer is usually, "its because the plugin doesn't know how to auto-detect that kind of application")

          But you should be able to manually add it. I see in your plugin metadata, you specify, "supportsManualAdd="true"" which is good - you are telling the system that this kind of resource can be added explicitly by the user (this feature is used for this exact use-case - when you know you have a resource that can be monitored, but the plugin has no way of auto-discovering it - you are essentially giving a hint to the agent plugin as to the location of the resource).

          Watch the end of this demo:

          http://support.rhq-project.org/display/RHQ/Plugins+-+Demos+-+Monitoring+Hibernate

          and you will see that I manually add a JMX Server - you will see the drop down includes WebLogic - this information is predefined for you in the JMX plugin metadata...which you can see here:

          http://svn.rhq-project.org/repos/rhq/trunk/modules/plugins/jmx/src/main/resources/META-INF/rhq-plugin.xml

          That's why your plugin metadata doesn't need to specify it, because you are extending the JMX plugin and it already has the info.

          So:

          1) in the UI, traverse to your platform resource's Inventory tab
          2) Select the drop down value "JMX Server" in the manual add option area
          3) Continue on from there - add JMX Server, select WebLogic, blah blah

          You may have to go into your new manually added resource's Inventory tab and configure ITS plugin configuration - especially if your application is requiring authentication credentials.

          • 2. Re: How to monitor a MBean on WebSphere
            mazz

            Just noticed you mentioned WebSphere, not WebLogic.

            Unsure if the JMX plugin is configured to work with WebSphere or if the underlying EMS library can connect to a WebSphere JMX MBeanServer.

            Take a look at the plugin descriptor and code for the JMX plugin and see.

            Feel free to contribute to the RHQ project with WebSphere support if it isn't supported today :)

            • 3. Re: How to monitor a MBean on WebSphere
              rafaelri

              Mazz,

              I am about to connect with my custom plugin. As soon as I get it running I'll post everything over here.
              What I'll risk saying for now is that indeed there is already support and it seems to be working. ;)

              • 4. Re: How to monitor a MBean on WebSphere
                rafaelri

                First thing I had to do although it seems to me like a hack... I had to place ws_runtime jar inside lib folder of Jopr.... I'll try to find a more elegant solution later on.
                Now it is throwing a

                An unexpected error occurred in the Agent. Failed to add resource with type [GifeManagement] and parent resource id [500550]. - Cause: java.lang.NullPointerException:null

                Before it was complaining about some missing WAS classes on classpath.

                • 5. Re: How to monitor a MBean on WebSphere
                  rafaelri

                  Full stacktrace for this NPE:

                  2008-12-04 16:09:40,585 ERROR [WorkerThread#0[10.202.70.173:1693]] (rhq.core.pc.inventory.InventoryManager)- Manual add failed for resource of type 'GifeManagement' and parent resource id [500550].
                  java.lang.NullPointerException
                   at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:129)
                   at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:81)
                   at org.rhq.core.pc.inventory.InventoryManager.manuallyAddResource(InventoryManager.java:394)
                   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:585)
                   at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
                   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:585)
                   at com.sun.jmx.mbeanserver.StandardMetaDataImpl.invoke(StandardMetaDataImpl.java:414)
                   at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
                   at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
                   at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
                   at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
                   at $Proxy1.execute(Unknown Source)
                   at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:358)
                   at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:253)
                   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
                   at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:560)
                   at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:383)
                   at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:165)
                  2008-12-04 16:09:47,711 INFO [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Collection of measurements for [org.rhq.plugins.platform.WindowsPlatformComponent@129bc5b] took [3501ms]
                  2008-12-04 16:09:47,711 INFO [MeasurementManager.sender-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Measurement collection for [5] metrics took 3501ms - sending report to Server...
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/lib/*] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/com.ibm.etools.jsse/ibmjsse.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ext/mail.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ibmcertpathprovider.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/java/jre/lib/ext/ibmjceprovider.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/org.apache.xerces_4.0.13/xercesImpl.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Connection library dependancy [AppServer/deploytool/itp/plugins/org.apache.xerces_4.0.13/xmlParserAPIs.jar] not found in directory: c:\ibm\WebSphere\AppServer\plugins
                  2008-12-04 16:09:59,384 INFO [ResourceContainer.invoker.daemon-9] (org.mc4j.ems.connection.ConnectionFactory)- Discovered libraries in 0 ms
                  


                  • 6. Re: How to monitor a MBean on WebSphere
                    mazz

                     

                    "rafaelri" wrote:
                    First thing I had to do although it seems to me like a hack... I had to place ws_runtime jar inside lib folder of Jopr.... I'll try to find a more elegant solution later on.
                    Now it is throwing a
                    An unexpected error occurred in the Agent. Failed to add resource with type [GifeManagement] and parent resource id [500550]. - Cause: java.lang.NullPointerException:null

                    Before it was complaining about some missing WAS classes on classpath.


                    This is why I added this new config setting to the JMX plugin:

                    http://jira.rhq-project.org/browse/RHQ-1094

                    This is in trunk. Now, you can just point to the client .jar file that is found in your managed product (analogous to the client jars you find in JBossAS deployments). I demonstrate this feature in the hibernate demo. Hibernate had the same issue - it uses this JMX plugin to get to the hibernate statistics MBean, but I need the hibernate.jar classes because Hibernate's Statistics class is only found there (and could be different depending on which hibernate I'm monitoring).

                    In the JBossAS case, the EMS libraries knows which client jars it needs - but I'm suspecting it doesn't know about which WebSphere ones it needs, so you have to tell it yourself. That's where my new configuration setting comes in.


                    • 7. Re: How to monitor a MBean on WebSphere
                      rafaelri

                      I had to create a custom AbstractConnectionTypeDescriptor called Websphere61ConnectionTypeDescriptor. WebsphereConnectionTypeDescriptor that comes with MC4J uses old jar in its classpath. Another problem I faced was that I had to repackage rhq-jmx-plugin-1.1.0.GA.jar so every jar that was inside org-mc4j-ems-1.2.5.jar is now on lib folder of rhq-jmx-plugin-1.1.0.GA.jar.
                      After all those changes, it seems like it is connecting now but it is throwing another weird NPE.
                      Ahh... those changes to packaging were necessary to avoid NoClassDefFound Errors... since the classloader that loaded the MC4J jars were unable to see WAS classes. Actually, the one that loaded org-mc4j-ems-impl.jar that was oddly placed inside org-mc4j-ems-1.2.5.jar.

                      • 8. Re: How to monitor a MBean on WebSphere
                        rafaelri

                        First thing I noticed... WebsphereConnectionProvider wasnt able of handling a secure WAS installation. So I did some changes to this class, follows the new code:

                        package org.mc4j.ems.impl.jmx.connection.support.providers;
                        
                        import java.lang.reflect.Field;
                        import java.lang.reflect.Method;
                        import java.net.URI;
                        import java.util.Properties;
                        import javax.management.MBeanServer;
                        import javax.management.j2ee.Management;
                        import javax.naming.Context;
                        import javax.naming.NamingException;
                        import org.mc4j.ems.connection.settings.ConnectionSettings;
                        import org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy;
                        
                        public class WebsphereConnectionProvider extends AbstractConnectionProvider
                        {
                        
                         public WebsphereConnectionProvider()
                         {
                         }
                        
                         protected void doConnect()
                         throws Exception
                         {
                         ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
                         try {
                         Context ctx = null;
                         Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
                         Class adminClientClass = Class.forName("com.ibm.websphere.management.AdminClient", true, getClass().getClassLoader());
                         Class adminClientFactoryClass = Class.forName("com.ibm.websphere.management.AdminClientFactory");
                         Properties props = new Properties();
                         URI serverUrl = new URI(connectionSettings.getServerUrl());
                         if((serverUrl.getScheme().equalsIgnoreCase("http")) || (serverUrl.getScheme().equalsIgnoreCase("https")))
                         {
                         System.setProperty("javax.net.debug", "ssl,handshake,data,trustmanager");
                         System.setProperty("java.protocol.handler.pkgs", "com.sun.net.ssl.internal.www.protocol");
                         props.put(getConstant(adminClientClass, "CONNECTOR_TYPE"), getConstant(adminClientClass, "CONNECTOR_TYPE_SOAP"));
                         } else
                         {
                         props.put(getConstant(adminClientClass, "CONNECTOR_TYPE"), getConstant(adminClientClass, "CONNECTOR_TYPE_RMI"));
                         }
                         String username = connectionSettings.getPrincipal();
                         String password = connectionSettings.getCredentials();
                         boolean security = ((username != null) && (!"".equals(username)));
                         props.setProperty(getConstant(adminClientClass, "CONNECTOR_SECURITY_ENABLED"), Boolean.toString(security));
                         if (security) {
                         props.put(getConstant(adminClientClass, "USERNAME"), username);
                         props.put(getConstant(adminClientClass, "PASSWORD"), password);
                         }
                         props.put(getConstant(adminClientClass, "CONNECTOR_HOST"), serverUrl.getHost());
                         props.put(getConstant(adminClientClass, "CONNECTOR_PORT"), String.valueOf(serverUrl.getPort()));
                         Method createMethod = adminClientFactoryClass.getMethod("createAdminClient", new Class[] {
                         java.util.Properties.class
                         });
                         Object adminClient = createMethod.invoke(null, new Object[] {
                         props
                         });
                         statsProxy = new GenericMBeanServerProxy(adminClient);
                         mbeanServer = statsProxy.buildServerProxy();
                         } finally {
                         Thread.currentThread().setContextClassLoader(contextClassLoader);
                         }
                         }
                        
                         public String getConstant(Class clazz, String name)
                         throws Exception
                         {
                         Field field = clazz.getField(name);
                         return (String)field.get(null);
                         }
                        
                         private Management retrieveMEJB(Context ic)
                         {
                         try
                         {
                         Object objref = ic.lookup("ejb/mgmt/MEJB");
                         return mejb;
                         }
                         catch(NamingException ne) { }
                         catch(Exception ce) { }
                         return null;
                         }
                        
                         public MBeanServer getMBeanServer()
                         {
                         return mbeanServer;
                         }
                        
                         public long getRoundTrips()
                         {
                         return statsProxy.getRoundTrips();
                         }
                        
                         public long getFailures()
                         {
                         return statsProxy.getFailures();
                         }
                        
                         protected GenericMBeanServerProxy statsProxy;
                         protected MBeanServer mbeanServer;
                         private Management mejb;
                         private static final String MEJB_JNDI = "ejb/mgmt/MEJB";
                        }
                        
                        
                        


                        But anyway... I guess (only a guess) jopr is not properly setting connectionSettings object.
                        connectionSettings.getPrincipal() always returns null.

                        Apart from this I had to place several classes on lib folder under agent install dir:

                        AppServer/lib/bootstrap.jar
                        AppServer/deploytool/itp/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar
                        AppServer/runtimes/com.ibm.ws.admin.client_6.1.0.jar
                        appserver/plugins/com.ibm.ws.security.crypto_6.1.0.jar

                        This solved NoClassDefFoundErrors (probably the new feature you cited could solve my issue) but for now I did like this.
                        Apart from this, I did (but I am not sure whether this is mandatory) a new ConnectionTypeDescriptor ...
                        follows the code as well:
                        package org.mc4j.ems.connection.support.metadata;
                        
                        public class Websphere61ConnectionTypeDescriptor extends AbstractConnectionTypeDescriptor {
                        
                         public Websphere61ConnectionTypeDescriptor()
                         {
                         }
                        
                         public boolean isMEJBCompliant()
                         {
                         return true;
                         }
                        
                         public String getDefaultServerUrl()
                         {
                         return "http://localhost:8880";
                         }
                        
                         public String getDefaultPrincipal()
                         {
                         return "wasadmin";
                         }
                        
                         public String getDefaultCredentials()
                         {
                         return "";
                         }
                        
                         public String getDefaultJndiName()
                         {
                         return null;
                         }
                        
                         public String getDefaultInitialContext()
                         {
                         return null;
                         }
                        
                         public String getConnectionType()
                         {
                         return "WebSphere6.1";
                         }
                        
                         public String getConnectionMessage()
                         {
                         return "Websphere 6.1 Connection";
                         }
                        
                         public String[] getConnectionClasspathEntries()
                         {
                         return (new String[] {
                         "AppServer/deploytool/itp/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar","AppServer/deploytool/itp/plugins/com.ibm.websphere.webservice.rt.v5.1.1_6.1.2.v200803090003/lib/commons-logging-api.jar"
                         });
                         }
                        
                         public String getConnectionNodeClassName()
                         {
                         return "org.mc4j.ems.impl.jmx.connection.support.providers.WebsphereConnectionProvider";
                         }
                        
                         public String getDisplayName()
                         {
                         return "WebSphere 6.1";
                         }
                        
                         public String getRecongnitionPath()
                         {
                         return "AppServer/bin/ProfileManagement/plugins/com.ibm.websphere.v61_6.1.200/ws_runtime.jar";
                         }
                        
                        }
                        


                        Also changed rhq-plugin.xml for JMX plugin to add support for this new ConnectionTypeDescriptor.

                        I guess I am almost making it work...
                        But Mazz... do you have any records of any issue related to the principal and credentials not set on connectionSettings?

                        regards,

                        • 9. Re: How to monitor a MBean on WebSphere
                          mazz

                          The only odd credentials problem is related when EMS uses the JBossAS JNP client code and when running two JBossAS servers on the same box but have different creds between them.

                          But this doesn't involve you because you aren't going to JBossAS.

                          You can submit bugs to the MC4J project if you are seeing things in EMS wrt WebSphere connectivity. But unsure how much time Greg would have to devote to fixing them (plus I don't think any of us here have access to a WebSphere installation). But if you have patches for the WebSphere connector code that are fully functional that work for you, I'm sure he would be happy to take those and merge them into the code base.

                          • 10. Re: How to monitor a MBean on WebSphere
                            rafaelri

                            For sure I'll send this to MC4J later on.
                            But how can I open something like a bug report for this part related to jopr not properly setting the credentials and password?

                            • 11. Re: How to monitor a MBean on WebSphere
                              rafaelri

                              For sure I'll send this to MC4J later on.
                              But how can I open something like a bug report for this part related to jopr not properly setting the credentials and password?

                              • 12. Re: How to monitor a MBean on WebSphere
                                mazz

                                https://jira.jboss.org/jira/browse/JOPR

                                I know the JMX plugin properly sets the credentials for the JBossAS EMS connector - that's how we handle authenticated connections to JBossAS via JNP. So, unsure how that is different from the stuff you are seeing.

                                • 13. Re: How to monitor a MBean on WebSphere
                                  mazz

                                  Actually, since this involves the JMX plugin, that's a plugin that comes from the upstream RHQ project, its JIRA is:

                                  http://jira.rhq-project.org/

                                  • 14. Re: How to monitor a MBean on WebSphere
                                    rafaelri

                                    Mazz,

                                    I had posted a message on MC4J forum but had no reply. I saw that Greg is usually here on Jopr forum, do you suggest any other way for me to contact him and provide the patch for MC4J EMS?

                                    best regards

                                    1 2 3 Previous Next