HotDeployment confusion
epeters Jul 2, 2015 4:58 PMLooking today at a fresh JVM crash file, from Widlfly 8.2-Final, I'm finding myself scratching my head...
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00002b43e6f73c19, pid=29437, tid=47571906619712
#
# JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build 1.7.0_79-mockbuild_2015_04_11_03_39-b00)
# Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.5.5
# Distribution: Built on Red Hat Enterprise Linux Server release 5.11 (Tikanga) (Sat Apr 11 03:39:43 EDT 2015)
# Problematic frame:
# C [libzip.so+0x4c19] ZIP_GetEntry+0x79
...
From the JVM frames, it's clear it was attempting a hot-deployment:
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 172 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002b43e7241e38 [0x00002b43e7241de0+0x58]
J 1126 C2 org.jboss.modules.JarFileResourceLoader.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; (52 bytes) @ 0x00002b43e7539184 [0x00002b43e75390a0+0xe4]
J 2264 C2 org.jboss.modules.JarFileResourceLoader.getResource(Ljava/lang/String;)Lorg/jboss/modules/Resource; (207 bytes) @ 0x00002b43e787e228 [0x00002b43e787e180+0xa8]
J 6399 C2 org.jboss.modules.ModuleClassLoader.loadResourceLocal(Ljava/lang/String;)Ljava/util/List; (128 bytes) @ 0x00002b43e84a51e8 [0x00002b43e84a4b20+0x6c8]
J 6118 C2 org.jboss.modules.Module.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (298 bytes) @ 0x00002b43e83ed12c [0x00002b43e83ecb00+0x62c]
j org.jboss.modules.ModuleClassLoader.findResources(Ljava/lang/String;Z)Ljava/util/Enumeration;+5
J 6321 C2 org.jboss.modules.ConcurrentClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration; (65 bytes) @ 0x00002b43e7da9a74 [0x00002b43e7da9a00+0x74]
j java.lang.ClassLoader.getResources(Ljava/lang/String;)Ljava/util/Enumeration;+19
J 6860 C2 org.jboss.seam.deployment.SeamDeploymentProperties.getResources()Ljava/util/Enumeration; (63 bytes) @ 0x00002b43e86551cc [0x00002b43e8654fc0+0x20c]
J 7037 C2 org.jboss.seam.deployment.SeamDeploymentProperties.addPropertiesFromResourceBundle(Ljava/lang/String;Ljava/util/List;)V (95 bytes) @ 0x00002b43e86fce40 [0x00002b43e86fce00+0x40]
j org.jboss.seam.deployment.SeamDeploymentProperties.getPropertyValues(Ljava/lang/String;)Ljava/util/List;+17
j org.jboss.seam.deployment.DeploymentStrategy.initDeploymentHandlers()V+26
j org.jboss.seam.deployment.DeploymentStrategy.getDeploymentHandlers()Ljava/util/Map;+8
j org.jboss.seam.deployment.HotDeploymentStrategy.<init>(Ljava/lang/ClassLoader;Ljava/io/File;Ljavax/servlet/ServletContext;Z)V+53
j org.jboss.seam.init.Initialization.createHotDeployment(Ljava/lang/ClassLoader;Z)Lorg/jboss/seam/deployment/HotDeploymentStrategy;+60
j org.jboss.seam.init.Initialization.redeploy(Ljavax/servlet/http/HttpServletRequest;Lorg/jboss/seam/core/Init;)V+22
j org.jboss.seam.web.HotDeployFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V+442
...
From that context, the error makes some level of sense -- digging into into libzip.so related crashes, I find that libzip.so will trigger a crash if the zip file is being modified while the library tries to read from it.
If a hot-deploy gets its timing wrong, I could see it reading from a zip still being written, stepping into this error.
Except, our deployment scanner is set to only scan at startup, which happened thirteen hours before the crash - this leaves open question of "What was being hot-deployed when the JVM crashed?" I don't believe it was our application. The management console might, but we make minimal use of the management console (nearly zero), and I'm seeing no indication of deployments triggered there. If I stretch my imagination, I could see New Relic playing a role in this, but the redeploy request was triggered through Undertow. I'd be a little surprised if New Relic used that interface any.
Does anyone have suggestions on where to go looking for root cause in this?
//EricP