9 Replies Latest reply: Jan 24, 2012 2:51 PM by Elias Ross RSS

Rhq-agent 4.3.0 memory leak

Damiano Scaramuzza Newbie

Hi,

I've compiled the last git release (or close to last one ,2b26d4412f3fbc28f4131efa7732f3f5e31e4955) and is running now on 4 clients (3 rhel5, 1 Debian 6.0).

I've experienced a slow system memory leak (faster in rhel5 and slower in Debian 6.0 , 15MB per hour in rhel5 and 5Mb per hour in Debian 6.0).

It's possible some unclosed handle using augeas or sigar ?

  • 1. Re: Rhq-agent 4.3.0 memory leak
    Damiano Scaramuzza Newbie

    I can add a sample from rhq server used as monitor for resourcesleak.JPG

  • 2. Re: Rhq-agent 4.3.0 memory leak
    Ian Springer Master

    What types of and how many server Resources are the Agents managing? (e.g. 2 JBoss EAP 4.3 servers and 2 Apache 2.2 servers per Agent)

     

    Also, hava you tried using a profiler such as JProfiler to pinpoint what is leaking?

  • 3. Re: Rhq-agent 4.3.0 memory leak
    Julio Barreto Newbie

    Good afternoon

    How do I download page for version RHQ 4.3

    thank you

    Julio

  • 4. Re: Rhq-agent 4.3.0 memory leak
    Ian Springer Master

    4.3 hasn't been released yet. 4.2 is still the latest release. If you want to try out the very latest code, you will have to build from source.

  • 5. Re: Rhq-agent 4.3.0 memory leak
    Damiano Scaramuzza Newbie

    Hi Ian,

    both are monitoring a single instance of jboss as 7 and typical OS stuffs (filesystem,sshd, cpu.. the standard) . The rhel5 is monitoring a postgres 8.4 instance.

    The jpg attached is taken from rhq server interface but obviously I see the memory leaks directly from OS too. Tipically after 3-4 days I've a 2.4G swap file used and I have to restart rhq-agent.

    I'll use Jprofile to pinpoint the problem a little better

  • 6. Re: Rhq-agent 4.3.0 memory leak
    Elias Ross Master

    There is definitely a native memory leak.

     

    I updated my hosts from 4.0 to 4.1 and I see the process looks like this:

     

      PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM


    15   0 11.0g  10g  13m S  0.0 33.0  26:03.99 java                                                                                                            

     

    That's 11g of memory usage. jhat and the like show quite a lot less in the JVM itself. Here is a histogram:

     

    num #instances     #bytes  class name
       1:     39295    5184840  <constMethodKlass>
       2:     39295    4725784  <methodKlass>
       3:      4091    4414496  <constantPoolKlass>
       4:     43604    3832328  [C
       5:     64747    3511200  <symbolKlass>
       6:      4712    3101592  [I
       7:      4091    2994248  <instanceKlassKlass>
       8:      3706    2709792  <constantPoolCacheKlass>
       9:     23291    2499224  [Ljava.lang.Object;
      10:     12682    2042320  [Ljava.util.HashMap$Entry;

     

    $ /usr/java/jdk1.6.0_17/bin/jmap -heap 28675 
    Attaching to process ID 28675, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 14.3-b01
    
    using thread-local object allocation.
    Parallel GC with 8 thread(s)
    
    Heap Configuration:
       MinHeapFreeRatio = 40
       MaxHeapFreeRatio = 70
       MaxHeapSize      = 134217728 (128.0MB)
       NewSize          = 2686976 (2.5625MB)
       MaxNewSize       = 17592186044415 MB
       OldSize          = 5439488 (5.1875MB)
       NewRatio         = 2
       SurvivorRatio    = 8
       PermSize         = 21757952 (20.75MB)
       MaxPermSize      = 88080384 (84.0MB)
    
    I'm guessing the plugin update/upgrade doesn't update the shared libraries as well?
    
    Or the new versions have leaks?
    
    Here is what I have (notice the dates):
    
    $ ls -l /usr/local/rhq-agent/lib/augeas/lib/
    total 412
    -rw-r--r-- 1 root root 260432 Jul 26 20:19 libaugeas.so
    -rw-r--r-- 1 root root  72404 Jul 26 20:19 libfa.so
    -rw-r--r-- 1 root root  72404 Jul 26 20:19 libfa.so.1
    

     

    Added formatting

  • 7. Re: Rhq-agent 4.3.0 memory leak
    mazz Master

    Did you write up a BZ? If not, can you do so and document all the relevent info (OS version, hardware, etc. etc.). Also indicate what resources you have in inventory and are monitoring. Don't think we've had any reports of their agent process growing to 11gig. Its either gonna be sigar or jna/augeas. My money would be on jna/augeas but that's just a WAG.

  • 8. Re: Rhq-agent 4.3.0 memory leak
    Elias Ross Master
    Bug 766959 has been added to the database

    I added information on Inventory. It's nothing special (no weird resources, I think.)
  • 9. Re: Rhq-agent 4.3.0 memory leak
    Elias Ross Master

    I found out leaks in the Grub and OpenSSH plugins. If you disable them, then you shouldn't see any more leaks.