I have a very powerful machine, with 80 cores and 32GB RAM, but when there is a traffic peak, it goes down. It hosts Apacje/JBoss/Postgres with deployed Seam web applications and EJB 3 for app. logic.
In thread dumps I see a lot of:
Thread: ajp-0.0.0.0-8009-14 : priority:5, demon:true, threadId:115, threadState:BLOCKED -
waiting on 0x7406ce55 (a java.util.WeakHashMap) org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:114)
Thread: ajp-0.0.0.0-8009-2 : priority:5, demon:true, threadId:103, threadState:BLOCKED
- waiting on 0x672e12af (a java.util.HashMap) org.jboss.metadata.web.jboss.JBossWebMetaData.getRunAsIdentity(JBossWebMetaData.java:839)
Thread: ajp-0.0.0.0-8009-14 : priority:5, demon:true, threadId:115, threadState:BLOCKED
- waiting on 0x7406ce55 (a java.util.WeakHashMap) org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:114)
As I understand, there is too many invocations of methods trying to access naming service. In Tomcat, I have set maxthreads on 8009 connector to 1200, however the dumps shows the problems starts when it's over 600-700.
I believe that setting up cluster should help, as the traffic will be distributed to several jboss instances. However, as I afraid cluster like a fire - is there anything more I can do?
great analysis and very good catch. This reminds me of my issue back in 2010 where I analyzed the synchronization in ENC factory is suboptimal. Could you have a look at the issue https://issues.jboss.org/browse/JBPAPP-4787 whether it is what you are seeing too?
As you see, these problems are not really fixable and havent been completely fixed.
Lets get to solution then. Really, solving this with "cluster" as in fully replicated states and etc does not make much sense. Clusters provide high availability but since you would be running all instances in the same machine, the failure of one component or network or whatnot of that particular machine makes your entire "cluster" unavailable. Therefore, I propose a solution to use multiple standalone servers (well, depends on your app if thats doable). You can either do this via different port binding or create new virtual network interfaces and bind servers to them.
Yes, this is exactly what I would like to try - create two standalone jbosses and loadbalance them using Apache, so the visitors traffic is divided between two of them.
Regarding your issues, I think it's same as mine:
Thread: ajp-0.0.0.0-8009-813 : priority:5, demon:true, threadId:1431, threadState:RUNNABLE
- locked <0x7406ce55> (a java.util.WeakHashMap)
and other threads waiting for weakhashmap. BTW, I am using JBoss 5.1 standalone, not EAP.
I suppose this solution will help (for a while) I have had a similar configuration using EJB.
For this we setup multiple instances on one power system. It will solve the 'sync' stuff but if the JBoss app scales you might bring the DB in behind under heavy load .
Will say as more performance you have as more you will need