Deployment Issues on JBOSS 7.1.1
I currently am having a weird issue with my JBoss Deployment. I had a system running on Redhat 5.6, Java 1.6, with JBoss 7.1.1 which runs without problem. I made some changes to our code and we moved to Redhat 6.4, Java 1.7, and stayed with JBoss 7.1.1. After the move, I am seeing random errors occasionally when JBOSS is deploying our application. I set up a script which reboots our system every 7 minutes and saves the log file if JBoss fails to deploy our application. Out of the 200 times it ran, JBoss did not deploy our application 27 times. Of those 27 failures, there errors boil down into 2 categories, ZipExceptions and ClassNotFoundExceptions.
From what I've been able to discern ZipExceptions generally occur when there is something wrong with the War file, but since it only failed 27 out of 200 times this seems unlikely to me? There are 3 different zip exceptions that happen on different occassions:
Exception 1:
08:40:57,722 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-10) MSC00001: Failed to start service jboss.deployment.unit."application.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."application.war".STRUCTURE: Failed to process phase STRUCTURE of deployment "application.war"
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:119) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: Failed to mount deployment content
at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:91) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:113) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
... 5 more
Caused by: java.util.zip.ZipException: invalid LOC header (bad signature)
at java.util.zip.ZipFile.read(Native Method) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.ZipFile.access$1400(ZipFile.java:56) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:677) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:413) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) [rt.jar:1.7.0_09-icedtea]
at java.io.FilterInputStream.read(FilterInputStream.java:107) [rt.jar:1.7.0_09-icedtea]
at org.jboss.vfs.VFSUtils.copyStream(VFSUtils.java:442) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFSUtils.copyStream(VFSUtils.java:422) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFSUtils.unzip(VFSUtils.java:872) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:536) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:567) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.as.server.deployment.DeploymentMountProvider$Factory$ServerDeploymentRepositoryImpl.mountDeploymentContent(DeploymentMountProvider.java:97) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:87) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
... 6 more
Exception 2:
07:15:34,270 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-13) MSC00001: Failed to start service jboss.deployment.unit."application.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."application.war".STRUCTURE: Failed to process phase STRUCTURE of deployment "application.war"
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:119) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: Failed to mount deployment content
at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:91) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:113) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
... 5 more
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:411) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158) [rt.jar:1.7.0_09-icedtea]
at java.io.FilterInputStream.read(FilterInputStream.java:107) [rt.jar:1.7.0_09-icedtea]
at org.jboss.vfs.VFSUtils.copyStream(VFSUtils.java:442) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFSUtils.copyStream(VFSUtils.java:422) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFSUtils.unzip(VFSUtils.java:872) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:536) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZipExpanded(VFS.java:567) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.as.server.deployment.DeploymentMountProvider$Factory$ServerDeploymentRepositoryImpl.mountDeploymentContent(DeploymentMountProvider.java:97) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.as.server.deployment.module.DeploymentRootMountProcessor.deploy(DeploymentRootMountProcessor.java:87) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
... 6 more
Exception 3:
10:12:25,116 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC00001: Failed to start service jboss.deployment.unit."application.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."application.war".STRUCTURE: Failed to process phase STRUCTURE of deployment "application.war"
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:119) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09-icedtea]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09-icedtea]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS018037: Failed to process WEB-INF/lib: "/content/BullfinchWeb.war/WEB-INF/lib/mongo-java-driver-2.12.0-rc2.jar"
at org.jboss.as.web.deployment.WarStructureDeploymentProcessor.deploy(WarStructureDeploymentProcessor.java:120)
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:113) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
... 5 more
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS018037: Failed to process WEB-INF/lib: "/content/BullfinchWeb.war/WEB-INF/lib/mongo-java-driver-2.12.0-rc2.jar"
at org.jboss.as.web.deployment.WarStructureDeploymentProcessor.createWebInfLibResources(WarStructureDeploymentProcessor.java:175)
at org.jboss.as.web.deployment.WarStructureDeploymentProcessor.createResourceRoots(WarStructureDeploymentProcessor.java:153)
at org.jboss.as.web.deployment.WarStructureDeploymentProcessor.deploy(WarStructureDeploymentProcessor.java:114)
... 6 more
Caused by: java.util.zip.ZipException: invalid CEN header (bad signature)
at java.util.zip.ZipFile.open(Native Method) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.ZipFile.<init>(ZipFile.java:214) [rt.jar:1.7.0_09-icedtea]
at java.util.zip.ZipFile.<init>(ZipFile.java:144) [rt.jar:1.7.0_09-icedtea]
at java.util.jar.JarFile.<init>(JarFile.java:152) [rt.jar:1.7.0_09-icedtea]
at java.util.jar.JarFile.<init>(JarFile.java:116) [rt.jar:1.7.0_09-icedtea]
at org.jboss.vfs.spi.JavaZipFileSystem.<init>(JavaZipFileSystem.java:97) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.spi.JavaZipFileSystem.<init>(JavaZipFileSystem.java:83) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZip(VFS.java:408) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.vfs.VFS.mountZip(VFS.java:434) [jboss-vfs-3.1.0.Final.jar:3.1.0.Final]
at org.jboss.as.web.deployment.WarStructureDeploymentProcessor.createWebInfLibResources(WarStructureDeploymentProcessor.java:170)
... 8 more
The same is true for the classnotfoundexceptions from what I've found online they occur when the code is missing a dependency. However wouldn't the deployment fail all the time if that was the case? The errors for these exceptions are much longer, so I'm not going to post them unless they are needed.
My suspicion is that moving from Redhat 5.6, Java 1.6 to Redhat 6.4, Java 1.7 has caused some sort of issue. However I wasn't able to find any similar issues people had with Redhat 6.4, Java 1.7. Any ideas of what could be going on? Let me know if any more information is needed.
Thanks,
Chris