-
15. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Nov 27, 2012 12:10 PM (in response to sfcoy)I don't think it's the case. The slow down occurs after some time the server was started. For example now, I have this war file with my code posted in the first message here. The Jboss is running for several days and it's really slow. Even if I undeploy the main application, the server remains slow. If I try to deploy it back, the deploy takes very long time. However, if I just shut down the JBoss and start it again, it will be fast.
I wouldn't say that whole virtual machine is running slow. If I run the code above in seperate JVM (just compile it with javac and run it) it's fast. This is bizarre.
-
16. Re: JBoss 7.1.1.Final/JVM slows over time
sfcoy Nov 27, 2012 9:00 PM (in response to lukascz)These memory options:
{quote}
-Xmx2048m -XX:MaxPermSize=2048{quote}
mean that you need at least 8G of real memory allocated to your VM, otherwise it will be subjected to disk swapping either by the guest OS or the host OS.
Java increases the sizes specified by 30% for 64bit JVMs.
-
17. Re: JBoss 7.1.1.Final/JVM slows over time
nickarls Nov 28, 2012 1:32 AM (in response to sfcoy)I'm still not buying the OOM thing. I understood that there was no issue and even if it would, it should not affect a CPU-only based test. Even if there was trashing it would not show up as CPU load, it would show up as I/O waits, resulting in that the CPU would not run at 100% during the test. And the test doesn't even use I/O.
My guess it's virtual server related somehow. The kernel gets confused in the virtual host somehow and does some re-nicing on the AS java process over time. Or something. I'm out of good explanations.
-
18. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Nov 29, 2012 3:29 PM (in response to nickarls)Here's yet another observation: when the server is after fresh start and I run the bubble sort, It utilizes 100% of one processor core. It never switches to another core or drops utilization below 95%. However after some time the server is running and I'm experiencing the performance problems, the method above is utilizing CPU core usually 100%, however I just found out from htop that this task is being switched very often to other cores. That is, at the beginning it's running on core #1, after say 2 seconds it's running on #5 then after say 2 seconds #8 etc. Furthermore, the utilization is not kept at 100% at the core but sometimes drops to 80% or even lower.
For the server after fresh start, even though If I simulate a load, it never switches the task to another core.
Is this a symptom of something?
-
19. Re: JBoss 7.1.1.Final/JVM slows over time
ctomc Nov 29, 2012 3:33 PM (in response to lukascz)Yes it is
virutalization swaps out cpu/ram to other machines after load decresses. (boot)
i think if you would run intensive algoritem that would use all cores in VM for long enough, hypervisor would relocate resource back to your VM after some time
--
tomaz
-
20. Re: JBoss 7.1.1.Final/JVM slows over time
nickarls Nov 30, 2012 1:48 AM (in response to ctomc)Still it is surprising that performance would go from 22 -> 330 secs from this...
-
21. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Nov 30, 2012 12:51 PM (in response to ctomc)It doesn't seem that that's the case. If I run the bubble sort in separate JVM (by running java PerformanceTest), it's behaving normally, it's fast, it's not switching processors. If there is a problem with VM, it should be a global problem. But this is issue only with Jboss 7 JVM. I was looking at the VMWare data with system administrator today and it doesn't show any issue. RAM/CPU are basically dedicated only to the machine where JBoss is running. It doesn't use swap either.
What about this data from top command for jboss 7 process?
VIRT 9093m, RES 2.6g, SHR 20m, %MEM 16.8
I believe this is normal.
-
22. Re: JBoss 7.1.1.Final/JVM slows over time
nickarls Dec 3, 2012 2:21 AM (in response to lukascz)Not sure if it really shows anything but what happens if you loop a Thread.sleep(1000) and then output something. Does it appear at one second intervals (wall clock time)?
-
23. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Dec 3, 2012 9:50 AM (in response to nickarls)Yes, it does. It's working correctly
-
24. Re: JBoss 7.1.1.Final/JVM slows over time
ctomc Dec 3, 2012 9:54 AM (in response to lukascz)That is strange.
can you post thread dump (jstack <pid-of-jboss-jvm>)
we might find something funny going on.
also what are the paramters you started jvm with?
--
tomaz
-
25. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Dec 3, 2012 9:56 AM (in response to ctomc)Here are JVM parameters;
-Xms1280m -Xmx2048m -XX:MaxPermSize=2048m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Duser.timezone=UTC -Djboss.server.default.config=standalone-full.xml -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
-
26. Re: JBoss 7.1.1.Final/JVM slows over time
ctomc Dec 3, 2012 10:34 AM (in response to lukascz)Hi,
i would change that to:
-server -Xms1280m -Xmx2048m -Xmn128m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Duser.timezone=UTC -Djboss.server.default.config=standalone-full.xml
note the -server & different PermGenSize as i doubt you need 2G PermGen, also i removed debugging part of the config.
--
tomaz
-
27. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Dec 3, 2012 11:45 AM (in response to ctomc)Thanks, I will do that.
Here is the thread dump: https://dl.dropbox.com/u/51364356/threadDump.txt
The code is executed in blah.blahhh.data.PerformanceTest.bubbleSort not that the thread is either in IN_VM state or IN_JAVA, it changes over time.
-
28. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Jan 14, 2013 11:48 AM (in response to lukascz)1 of 1 people found this helpfulSo we are upragded application server from JBoss 7.1.1.Final to SNAPSHOT of JBoss AS 7.1.4 and that issue didn't show up again. We are running it for a month right now. So the problem was with JBoss AS 7.1.1.Final.
Thanks for your help
-
29. Re: JBoss 7.1.1.Final/JVM slows over time
maderhan Jan 15, 2013 12:44 AM (in response to lukascz)Thanks for the information, we have similary problems.