9 Replies Latest reply on Jan 31, 2011 8:30 AM by mazz

    rhq-agent-wrapper.log file not limited

    guanshuiwang

      I start the agent using "sh rhq-agent-wrapper.sh start".

      But I found the log file "rhq-agent-wrapper.log" increased unlimited.  And the log file would increase to several G. It's too large.

       

      What logs in the file?

      How can I limit it ?

       

      Thanks!

        • 1. rhq-agent-wrapper.log file not limited
          mazz

          That should not have gigs worth of logs in it.  This is the standard out output of the agent - and the agent prints very little to stdout. The only thing I can think of is:

           

          a) you wrote and deployed your own custom agent plugin and it calls System.out or otherwise prints/logs messages to stdout

          b) you are getting massive amounts of system exceptions within the agent and its dumping those to stdout (I can't think where we'd be dumping exceptions to stdout, but I'm sure its possible for rare exception messages we'd print it to the console/stdout to make sure users are aware its happening).

           

          What are some of these messages that you see in this log file?

          • 2. rhq-agent-wrapper.log file not limited
            guanshuiwang

            Thanks.

            There are lots of messages in it like:

             

            org.mc4j.ems.connection.EmsException: Could not load attribute value java.rmi.ConnectException: Connection refused to host: mwm-24.gre.hp.com; nested excepti

            on is:

                    java.net.ConnectException: Connection refused

                    at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:235)

                    at org.jbosson.plugins.jbossesb.ESBMessageEventPoller.poll(ESBMessageEventPoller.java:123)

                    at org.rhq.core.pc.event.EventPollerRunner.run(EventPollerRunner.java:49)

                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

                    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

                    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                    at java.lang.Thread.run(Thread.java:636)

            Caused by: org.mc4j.ems.connection.EmsConnectException: java.rmi.ConnectException: Connection refused to host: mwm-24.gre.hp.com; nested exception is:

                    java.net.ConnectException: Connection refused

                    at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:137)

                    at $Proxy49.getAttribute(Unknown Source)

                    at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:199)

                    ... 10 more

            Caused by: java.rmi.ConnectException: Connection refused to host: mwm-24.gre.hp.com; nested exception is:

                    java.net.ConnectException: Connection refused

                    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)

                    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)

                    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)

                    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)

                    at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)

                    at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)

                    at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)

                    at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)

                    at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66)

                    at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)

                    at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)

                    at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

                    at $Proxy48.getAttribute(Unknown Source)

                    at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                    at java.lang.reflect.Method.invoke(Method.java:616)

                    at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)

                    ... 12 more

             

            Hope your reply!

            Thanks.

            • 3. rhq-agent-wrapper.log file not limited
              mazz

                      java.net.ConnectException: Connection refused

                      at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:235)

                      at org.jbosson.plugins.jbossesb.ESBMessageEventPoller.poll(ESBMessageEventPoller.java:123)

               

              Looks like you are using the JBossON ESB plugin, which is causing the problem you are reporting.

               

              The ESB plugin's ESBMessageEventPoller class is doing a throwable.printStackTrace() - which it should not be. Plugins should never output to stdout - they should log those messages (via log4j for example). See the ESBMessageEventPoller source code, specifically line 138 for an example where it is doing this.

               

              Since it looks like you are using the official JON product, I recommend you open a support case against the ESB plugin and see what they can about patching that plugin for you. It will be simple - they just need to remove all instances of t.printStackTrace() and replace with logging. You could alternatively  write up a JIRA bug report - the JIRA project is https://issues.jboss.org/browse/JBESB ... However, if you are an official customer and have a support contract for the ESB product and its plugin, you are entitled to submitting a support case and track the problem using that official mechanism.

              • 4. rhq-agent-wrapper.log file not limited
                guanshuiwang

                Thanks very much! I will try it.

                • 5. rhq-agent-wrapper.log file not limited
                  guanshuiwang

                  How to close the log to rhq-agent-wrapper.log?

                  Thanks very much!

                  • 6. rhq-agent-wrapper.log file not limited
                    mazz

                    the only way to stop that log from being generated is to manually edit the rhq-agent-wrapper.sh and remove the stdout redirection (i.e. remove the ">" and everythere after it from the line:

                     

                        eval "$RHQ_AGENT_START_COMMAND > \"${RHQ_AGENT_HOME}/logs/rhq-agent-wrapper.log\" 2>&1"

                     

                    You may also be able to have a cron job run periodically (perhaps every hour?) to clear that file out.

                     

                    But the real solution to your problem is to get that plugin to stop dumping large amounts of data to stdout. Did you submit a JIRA on the problem to the ESB development team?  Or were you able to open a support ticket with Red Hat?

                    • 7. rhq-agent-wrapper.log file not limited
                      mazz

                      Sorry, I had a typo in that last message and it probably didn't make sense. I was just saying you can remove the redirections entirely. I guess you could also redirect to /dev/null. So, if you want to manually edit the rhq-agent-wrapper.sh, and use /dev/null, just change that line that redirects to the log file so it looks like:

                       

                          eval "$RHQ_AGENT_START_COMMAND > /dev/null 2>&1"

                       

                      Of course, you will lose all record of the error messages (which, again, is why I recommend getting the plugin to change its implementation so it logs the messages to a log4j log file).

                      • 8. rhq-agent-wrapper.log file not limited
                        guanshuiwang

                        Thanks!

                        • 9. rhq-agent-wrapper.log file not limited
                          mazz