5 Replies Latest reply: Mar 14, 2012 10:47 AM by Peter Johnson RSS

JBoss/JVM process RAM usage increasing steadily

Nam nam Newbie

On checking RAM usage for our JBoss/Java process using 'ps aux' command, RSS value= 12161968KB, and it is increasing steadily.

We have set the Max heap size to 2GB while starting the process. We are checking heap size using 'jvisualvm' and it is below 2GB.

(-Xss1m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:NewRatio=2 -Xms2048m -Xmx2048m -XX:MaxPermSize=256m).

Environment:

OS: Red Hat Linux

JVM: 1.6.0_30

App Server: jboss-5.1.0.GA

 

Need help on trouble shooting this issue.

  • 1. Re: JBoss/JVM process RAM usage increasing steadily
    Peter Johnson Master

    RAM for a Java app is consumed by more than just heap (RAM usage for the heap will never exceed 2048MB in your case).

     

    See this for a list of everything that can use up RAM:

    http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running#560894

     

    How many threads are running? Have the number of threads increased? If so, that could account for the additonal RAM usage.

  • 2. Re: JBoss/JVM process RAM usage increasing steadily
    Nam nam Newbie

    We have tried to replicate this issue on Test bed with Heap size set to 1GB.

    On test bed as soon as JBoss is started we find that there are a lot of threads (JBoss System Thread(1) - around 300 threads) that are created.

    I have no idea what these threads are for, any information on this will be helpful.

    This is even before any request is made to the servers. But this is not observer on Production.

     

    Thread count is tabulated below. Though number of threads has decreased the RSS value is increasing.

     

    Date

    Time

    Total Thread

    Daemon Thread

    Non-Daemon Thread

    JBoss System Thread(1)

    http Thread

    VSZ

    RSS

    SZ

     

    2012-03-12

    17:55:30

    431

    403

    28

    300

    61

    2343840

    1028664

    585960

    mapped: 1835679K writeable/private: 1803527K shared: 0K

    2012-03-12

    18:22:42

    428

    399

    28

    300

    61

    2409304

    1500656

    602326

    mapped: 1835679K writeable/private: 1803527K shared: 0K

    2012-03-13

    09:52:52

    423

    395

    28

    300

    61

    2409304

    1501628

    602326

    mapped: 1835679K writeable/private: 1803527K shared: 0K

  • 3. Re: JBoss/JVM process RAM usage increasing steadily
    Peter Johnson Master

    You have the heap set to 2GB. The JVM will reserve that memory immediately, but will not start using it until it needs it. That would expalin the RSS increasing  - as the app runs and it uses the heap the JVM will start to make use of some of that memory that it has allocated and the OS will make it resident. Notice that after the app has been runnign for a while that the RSS rmains fairly constant.

     

    Note that the VSZ remain fairly constant. The increase after the initial size can be attributed to the perm gen increasing as classes are loaded.

     

    I don't see anything here that would cause me alarm.

     

    As far as what each of the threads are doing, you could take a JVM thread dump and see.

  • 4. Re: JBoss/JVM process RAM usage increasing steadily
    Nam nam Newbie

    Values mentioned in the above post were of server where Heap size was configured as 1GB.

     

     

    Below are values from the server where Heap is configured to 2GB but RSS=5.5GB and this keeps increasing till it uses up the entire RAM available.

    Also the thread count is almost constant for over 24hrs but RSS value is increasing.

     

    Date

    Time

    Total

    threads

    Daemon

    threads

    Non daemon

    threads

    ‘JBoss

    System threads’

    HTTP

    threads

    VSZ

    RSS

    SZ

    mapped:

    writeable/private:

    shared:

    2012/3/13

    11:50:48

    277

    245

    32

    2

    208

    3644988

    3026036

    911247

    mapped: 3653180K

    writeable/private: 3403160K

    shared: 3272K

    2012/3/13

    14:32:03

    276

    244

    32

    2

    208

    3817224

    3186660

    954306

    mapped: 3825416K

    writeable/private: 3545084K

    shared: 3272K

    2012/3/13

    18:00:03

    276

    244

    32

    2

    208

    4175848

    3614728

    1043962

    mapped: 4184040K

    writeable/private: 3950492K

    shared: 3272K

    2012/3/14

    00:00:03

    276

    244

    32

    2

    208

    4696012

    4165484

    1174003

    mapped: 4704204K

    writeable/private: 4480168K

    shared: 3272K

    2012/3/14

    06:00:03

    276

    244

    32

    2

    208

    5168076

    4573556

    1292019

    mapped: 5176268K

    writeable/private: 4884224K

    shared: 3272K

    2012/3/14

    12:00:04

    276

    244

    32

    2

    208

    5953416

    5349432

    1488354

    mapped: 5961608K

    writeable/private: 5643848K

    shared: 3272K

    2012/3/14

    15:00:03

    276

    244

    32

    2

    208

    6349600

    5778032

    1587400

    mapped: 6357792K

    writeable/private: 6077580K

    shared: 3272K

     

  • 5. Re: JBoss/JVM process RAM usage increasing steadily
    Peter Johnson Master

    Which JVM are you using? OpenJDK? Or the one from Oracle? If from Oracle, revert to version 1.6.0_26 and see if that helps (we have noticed various issues with versions more recent than that). If that doesn't help, you'll have to contact Oracle to see if there is a memory leak issue with the JVM itself.

     

    This almost sounds as if a system resource (such as a file handle or a socket connection) is not being released. But JBoss AS itself maintains pools of resources and reuses them, but who knows what the native JVM code is doing (only Oracle would know).