I am deploying 5 wars in standalone mode on rhel 5.5/5.6. We have 6 rhel environments: local (windows), dev, int, perf, stage, and prod. perf-prod use three load balanced servers. Local, dev, int, and perf have been running these wars for months with no issue. However, one of the modules failed to deploy in the stage environment on one server. We set it up as a single instance, and it still failed. we copied the entire jboss directory to the other server, and it still failed.

The underlying cause is a class conflict with an entity manager calling a method from a remote class with the incorrect version.

2015-09-24 19:25:40,539 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/myApp]] (ServerService Thread Pool -- 58) JBWEB000287: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor#0' defined in class path resource [META-INF/spring/lmyApp-db-dao-context.xml]: Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [META-INF/spring/myApp-db-dao-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalAccessError: tried to access field org.hibernate.engine.spi.CascadeStyle.STYLES from class org.hibernate.ejb.engine.spi.EJB3CascadeStyle

I rooted through the pom and found that we had included dependencies for hibernate-core 4.2.0 and org.springsource.hibernate.4.1.0.Final. Manually removing the 4.1.0 jar from the war allowed the app to deploy successfully and run without issue. I have since reworked the dependencies to get rid of the org.springsource jars and am able to run in my local windows env without issue.

So I have two real questions:

Q1. why did this happen now?

Q2. how confident can I be that it won't happen again?

Q1.detail

I went through all the startup attributes for jboss and the only differences were the os.version and the user.timezone. There were a few properties

successful:    

failed:

Deploying on servers running the same OS as the failed server result in the exact same failure. From what I can tell, in the 5.5 environment, the classloader for entitymanager and EJB3CascadeStyle are the same, but for 5.6 they are always different.

Q2..detail

we have 5 wars being deployed and the hibernate version is not consistent across all 5. I have corrected the problem war so it only has the 4.2 jars, but the other wars have either 4.1.0 or 4.2. There are no direct dependencies between the wars, but they do use an interapp jar to pass rest messages. I can rework the other jars as well, but I am trying to minimize impact so we won't have to run a month of regression tests. Is there a testing strategy specific to hibernate beyond simply running a lot of tests?

My research

I have read extensively about class loading, but not enough to explain why/how this happened. I know that jboss has the 4.2 hibernate module, and it seems that the wars should be prioritizing the static module rather than loading the lib jars. The parent pom for each war inherits from a common pom which specifies the 4.2.0.final hibernate jars as provided. I thought that was just version control until I read about the static modules. I now suspect that we should have used scope-test for IDE deployments and the wars would be using the jboss provided classes. I know war classes and libs are supposed to be isolated from each other, so I am thinking that there is an annotation somewhere allowing the classes involved in the error to load from either the static module or an included dependency. In the 5.5 env, they load one way, but in the 5.6 env they load another.

I have tried to enable logging on the class loader, but I have been unable to determine what class to enable logging on. Where is the class loader defined? Does the war provide its own class loader, and where would I find that if it does? I have tried to modify the logging of other entries so that it will append the classpath to the class reference, but I have not been able to find the filter pattern-syntax, either.