7 Replies Latest reply on Jul 20, 2011 9:13 AM by jaikiran

    JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"

    tskorka

      Hi,

      I'm trying to build JBoss from source code, but I face a problem with build hanging at "JBoss Application Server EmbeddedAS TestSuite":

       

      [INFO] ------------------------------------------------------------------------
      [INFO] Building JBoss Application Server EmbeddedAS TestSuite 6.1.0-SNAPSHOT
      [INFO] ------------------------------------------------------------------------
      [INFO]
      [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-java-version) @ jboss-as-embedded ---
      [INFO]
      [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (enforce-maven-version) @ jboss-as-embedded ---
      [INFO]
      [INFO] --- maven-enforcer-plugin:1.0-beta-1:enforce (ban-bad-dependencies) @ jboss-as-embedded ---
      [INFO]
      [INFO] --- buildnumber-maven-plugin:1.0-beta-3:create (set-build-properties) @ jboss-as-embedded ---
      [INFO]
      [INFO] --- build-helper-maven-plugin:1.4:maven-version (default) @ jboss-as-embedded ---
      [INFO]
      [INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ jboss-as-embedded ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] skip non existing resourceDirectory C:\Users\Tomasz\Documents\JBoss\jbossas-trunk\embedded\src\main\resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ jboss-as-embedded ---
      [INFO] No sources to compile
      [INFO]
      [INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ jboss-as-embedded ---
      [INFO] Using 'UTF-8' encoding to copy filtered resources.
      [INFO] Copying 8 resources
      [INFO]
      [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ jboss-as-embedded ---
      [INFO] Compiling 20 source files to C:\Users\Tomasz\Documents\JBoss\jbossas-trunk\embedded\target\test-classes
      [INFO]
      [INFO] --- maven-surefire-plugin:2.5:test (default-test) @ jboss-as-embedded ---
      [INFO] Surefire report directory: C:\Users\Tomasz\Documents\JBoss\jbossas-trunk\embedded\target\surefire-reports
      
      -------------------------------------------------------
       T E S T S
      -------------------------------------------------------
      Running org.jboss.jbossas.embedded.testsuite.ServerIntegrationTest
      

       

      In Windows task manager, I can see that new java process (for embedded JBoss) starts and utilizes ~50% CPU, but after ~10sec CPU utilization goes to 0%. Then nothing happens - build just hangs. When I kill that process, build for this component ends with status FAILED.

       

      Please help to solve this problem.

       

      I'm using Java 1.6.0_26. Sources has been fetched yesterday and updated today (r111650).

        • 1. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
          jaikiran

          It's under investigation. Some recent upgrade caused this. I'll talk to Shelly to figure out which commit caused this and whether to roll it back if we don't have a fix.

          • 2. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
            tskorka

            Thanks. Just did some experiments and here's what I can see:

             

            1. I didn't have command line SVN - I've installed it (SilkSVN), added it's 'bin' dir to %PATH% and it seems to have solved the problem. 3 out of 3 builds were successful!

             

            2. Until I've found above solution, I've been experimenting with removing contents of sunfire-reports dir under jboss embedded, combined with using '-e' switch - in most cases it worked and build did not hang.

             

            Anyways, hope this will help you

            • 3. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
              jaikiran

              It deadlocks for me:

               

              2011-06-20 15:37:04
              Full thread dump Java HotSpot(TM) Server VM (17.0-b16 mixed mode):
              
              "Attach Listener" daemon prio=10 tid=0x09140000 nid=0x4526 runnable [0x00000000]
                 java.lang.Thread.State: RUNNABLE
              
              "JBossWS-CXF-DefaultBus-Init" daemon prio=10 tid=0x7f310c00 nid=0x4450 waiting for monitor entry [0x7fa63000]
                 java.lang.Thread.State: BLOCKED (on object monitor)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)
                  - waiting to lock <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.net.URL.getURLStreamHandler(URL.java:1144)
                  at java.net.URL.<init>(URL.java:393)
                  at java.net.URL.<init>(URL.java:283)
                  at java.net.URL.<init>(URL.java:306)
                  at sun.net.www.protocol.file.Handler.openConnection(Handler.java:74)
                  - locked <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at sun.net.www.protocol.file.Handler.openConnection(Handler.java:55)
                  - locked <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at java.net.URL.openConnection(URL.java:945)
                  at sun.net.www.protocol.jar.JarURLConnection.<init>(JarURLConnection.java:66)
                  at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:24)
                  at java.net.URL.openConnection(URL.java:945)
                  at org.apache.cxf.common.logging.JDKBugHacks.doHacks(JDKBugHacks.java:70)
                  at org.apache.cxf.common.logging.Log4jLogger.<clinit>(Log4jLogger.java:55)
                  at java.lang.Class.forName0(Native Method)
                  at java.lang.Class.forName(Class.java:247)
                  at org.apache.cxf.common.logging.LogUtils.<clinit>(LogUtils.java:99)
                  at org.apache.cxf.BusFactory.<clinit>(BusFactory.java:63)
                  at org.jboss.wsf.stack.cxf.config.CXFInitializer$DefautBusInitializer.run(CXFInitializer.java:87)
                  at java.lang.Thread.run(Thread.java:619)
              
              "AuthenticationCacheFlushThread" prio=10 tid=0x813c0800 nid=0x444f waiting on condition [0x8049c000]
                 java.lang.Thread.State: TIMED_WAITING (sleeping)
                  at java.lang.Thread.sleep(Native Method)
                  at org.jboss.security.auth.AuthenticationCacheFlushThread.run(AuthenticationCacheFlushThread.java:83)
              
              "Thread-1" prio=10 tid=0x80957c00 nid=0x444e waiting for monitor entry [0x80da1000]
                 java.lang.Thread.State: BLOCKED (on object monitor)
                  at java.net.URLStreamHandler.getHostAddress(URLStreamHandler.java:412)
                  - waiting to lock <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at java.net.URLStreamHandler.hostsEqual(URLStreamHandler.java:439)
                  at sun.net.www.protocol.file.Handler.hostsEqual(Handler.java:117)
                  at java.net.URLStreamHandler.sameFile(URLStreamHandler.java:396)
                  at java.net.URLStreamHandler.equals(URLStreamHandler.java:316)
                  at java.net.URL.equals(URL.java:842)
                  at java.security.CodeSource.equals(CodeSource.java:135)
                  at java.util.HashMap.get(HashMap.java:305)
                  at java.security.SecureClassLoader.getProtectionDomain(SecureClassLoader.java:233)
                  - locked <0x93c97b30> (a java.util.HashMap)
                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
                  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.lang.ClassLoader.defineClass1(Native Method)
                  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
                  at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
                  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.lang.Class.getDeclaredConstructors0(Native Method)
                  at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
                  at java.lang.Class.getDeclaredConstructors(Class.java:1836)
                  at org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getDeclaredConstructors(IntrospectionTypeInfoFactoryImpl.java:576)
                  at org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getConstructors(IntrospectionTypeInfoFactoryImpl.java:134)
                  at org.jboss.reflect.plugins.ClassInfoImpl.getDeclaredConstructors(ClassInfoImpl.java:446)
                  at org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getConstructors(AbstractBeanInfoFactory.java:218)
                  at org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getBeanInfo(AbstractBeanInfoFactory.java:163)
                  - locked <0x948271c8> (a java.util.WeakHashMap)
                  at org.jboss.config.plugins.AbstractConfiguration.getBeanInfo(AbstractConfiguration.java:87)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfig.getBeanInfo(AbstractKernelConfig.java:80)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:78)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:97)
                  at org.jboss.kernel.plugins.dependency.PreInstallAction.installActionInternal(PreInstallAction.java:88)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
                  at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
                  at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
                  at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:894)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:641)
                  at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:182)
                  at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:58)
                  at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:63)
                  at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)
                  at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1571)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076)
                  at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679)
                  at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106)
                  at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143)
                  at org.jboss.profileservice.dependency.ProfileDeployAction.deploy(ProfileDeployAction.java:151)
                  at org.jboss.profileservice.dependency.ProfileDeployAction.installActionInternal(ProfileDeployAction.java:94)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
                  at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
                  at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
                  at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)
                  at org.jboss.profileservice.dependency.ProfileActivationWrapper$BasicProfileActivation.start(ProfileActivationWrapper.java:190)
                  at org.jboss.profileservice.dependency.ProfileActivationWrapper.start(ProfileActivationWrapper.java:87)
                  at org.jboss.profileservice.dependency.ProfileActivationService.activateProfile(ProfileActivationService.java:215)
                  at org.jboss.profileservice.dependency.ProfileActivationService.activate(ProfileActivationService.java:159)
                  at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.activate(AbstractProfileServiceBootstrap.java:112)
                  at org.jboss.profileservice.resolver.BasicResolverFactory$ProfileResolverFacade.deploy(BasicResolverFactory.java:87)
                  at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.start(AbstractProfileServiceBootstrap.java:91)
                  at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:132)
                  at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56)
                  at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827)
                  at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417)
                  - locked <0x94180890> (a org.jboss.embedded.core.server.JBossASEmbeddedServerImpl)
                  at java.lang.Thread.run(Thread.java:619)
              
              "Low Memory Detector" daemon prio=10 tid=0x8348d000 nid=0x444c runnable [0x00000000]
                 java.lang.Thread.State: RUNNABLE
              
              "CompilerThread1" daemon prio=10 tid=0x8348b000 nid=0x444b waiting on condition [0x00000000]
                 java.lang.Thread.State: RUNNABLE
              
              "CompilerThread0" daemon prio=10 tid=0x83489000 nid=0x444a waiting on condition [0x00000000]
                 java.lang.Thread.State: RUNNABLE
              
              "Signal Dispatcher" daemon prio=10 tid=0x83487800 nid=0x4449 runnable [0x00000000]
                 java.lang.Thread.State: RUNNABLE
              
              "Finalizer" daemon prio=10 tid=0x83478400 nid=0x4448 in Object.wait() [0x83195000]
                 java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  - waiting on <0x9398caa8> (a java.lang.ref.ReferenceQueue$Lock)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
                  - locked <0x9398caa8> (a java.lang.ref.ReferenceQueue$Lock)
                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
                  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
              
              "Reference Handler" daemon prio=10 tid=0x83476c00 nid=0x4447 in Object.wait() [0x831e6000]
                 java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  - waiting on <0x93f2fa68> (a java.lang.ref.Reference$Lock)
                  at java.lang.Object.wait(Object.java:485)
                  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
                  - locked <0x93f2fa68> (a java.lang.ref.Reference$Lock)
              
              "main" prio=10 tid=0x08d93000 nid=0x4443 in Object.wait() [0xb6ab1000]
                 java.lang.Thread.State: WAITING (on object monitor)
                  at java.lang.Object.wait(Native Method)
                  - waiting on <0x9426a740> (a java.lang.Thread)
                  at java.lang.Thread.join(Thread.java:1143)
                  - locked <0x9426a740> (a java.lang.Thread)
                  at java.lang.Thread.join(Thread.java:1196)
                  at org.jboss.bootstrap.impl.base.server.AbstractServer.start(AbstractServer.java:348)
                  at org.jboss.bootstrap.impl.mc.server.AbstractMCServerBase.start(AbstractMCServerBase.java:248)
                  at org.jboss.bootstrap.impl.as.server.AbstractJBossASServerBase.start(AbstractJBossASServerBase.java:402)
                  at org.jboss.jbossas.embedded.testsuite.ServerIntegrationTest.startEmbedddedASAndSetNamingContext(ServerIntegrationTest.java:235)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
                  at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
                  at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
                  at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
                  at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
                  at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
                  at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
                  at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
                  at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:140)
                  at org.apache.maven.surefire.Surefire.run(Surefire.java:109)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                  at java.lang.reflect.Method.invoke(Method.java:597)
                  at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:290)
                  at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1017)
              
              "VM Thread" prio=10 tid=0x83474400 nid=0x4446 runnable 
              
              "GC task thread#0 (ParallelGC)" prio=10 tid=0x08d9cc00 nid=0x4444 runnable 
              
              "GC task thread#1 (ParallelGC)" prio=10 tid=0x08d9e400 nid=0x4445 runnable 
              
              "VM Periodic Task Thread" prio=10 tid=0x8348ec00 nid=0x444d waiting on condition 
              
              JNI global references: 1615
              
              
              Found one Java-level deadlock:
              =============================
              "JBossWS-CXF-DefaultBus-Init":
                waiting to lock monitor 0x08da279c (object 0x93c964f8, a sun.misc.Launcher$AppClassLoader),
                which is held by "Thread-1"
              "Thread-1":
                waiting to lock monitor 0x08da26d4 (object 0x93c964b0, a sun.net.www.protocol.file.Handler),
                which is held by "JBossWS-CXF-DefaultBus-Init"
              
              Java stack information for the threads listed above:
              ===================================================
              "JBossWS-CXF-DefaultBus-Init":
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)
                  - waiting to lock <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.net.URL.getURLStreamHandler(URL.java:1144)
                  at java.net.URL.<init>(URL.java:393)
                  at java.net.URL.<init>(URL.java:283)
                  at java.net.URL.<init>(URL.java:306)
                  at sun.net.www.protocol.file.Handler.openConnection(Handler.java:74)
                  - locked <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at sun.net.www.protocol.file.Handler.openConnection(Handler.java:55)
                  - locked <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at java.net.URL.openConnection(URL.java:945)
                  at sun.net.www.protocol.jar.JarURLConnection.<init>(JarURLConnection.java:66)
                  at sun.net.www.protocol.jar.Handler.openConnection(Handler.java:24)
                  at java.net.URL.openConnection(URL.java:945)
                  at org.apache.cxf.common.logging.JDKBugHacks.doHacks(JDKBugHacks.java:70)
                  at org.apache.cxf.common.logging.Log4jLogger.<clinit>(Log4jLogger.java:55)
                  at java.lang.Class.forName0(Native Method)
                  at java.lang.Class.forName(Class.java:247)
                  at org.apache.cxf.common.logging.LogUtils.<clinit>(LogUtils.java:99)
                  at org.apache.cxf.BusFactory.<clinit>(BusFactory.java:63)
                  at org.jboss.wsf.stack.cxf.config.CXFInitializer$DefautBusInitializer.run(CXFInitializer.java:87)
                  at java.lang.Thread.run(Thread.java:619)
              "Thread-1":
                  at java.net.URLStreamHandler.getHostAddress(URLStreamHandler.java:412)
                  - waiting to lock <0x93c964b0> (a sun.net.www.protocol.file.Handler)
                  at java.net.URLStreamHandler.hostsEqual(URLStreamHandler.java:439)
                  at sun.net.www.protocol.file.Handler.hostsEqual(Handler.java:117)
                  at java.net.URLStreamHandler.sameFile(URLStreamHandler.java:396)
                  at java.net.URLStreamHandler.equals(URLStreamHandler.java:316)
                  at java.net.URL.equals(URL.java:842)
                  at java.security.CodeSource.equals(CodeSource.java:135)
                  at java.util.HashMap.get(HashMap.java:305)
                  at java.security.SecureClassLoader.getProtectionDomain(SecureClassLoader.java:233)
                  - locked <0x93c97b30> (a java.util.HashMap)
                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
                  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.lang.ClassLoader.defineClass1(Native Method)
                  at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
                  at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
                  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
                  at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
                  at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  - locked <0x93c964f8> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  at java.lang.Class.getDeclaredConstructors0(Native Method)
                  at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
                  at java.lang.Class.getDeclaredConstructors(Class.java:1836)
                  at org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getDeclaredConstructors(IntrospectionTypeInfoFactoryImpl.java:576)
                  at org.jboss.reflect.plugins.introspection.IntrospectionTypeInfoFactoryImpl.getConstructors(IntrospectionTypeInfoFactoryImpl.java:134)
                  at org.jboss.reflect.plugins.ClassInfoImpl.getDeclaredConstructors(ClassInfoImpl.java:446)
                  at org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getConstructors(AbstractBeanInfoFactory.java:218)
                  at org.jboss.beans.info.plugins.AbstractBeanInfoFactory.getBeanInfo(AbstractBeanInfoFactory.java:163)
                  - locked <0x948271c8> (a java.util.WeakHashMap)
                  at org.jboss.config.plugins.AbstractConfiguration.getBeanInfo(AbstractConfiguration.java:87)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfig.getBeanInfo(AbstractKernelConfig.java:80)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:78)
                  at org.jboss.kernel.plugins.config.AbstractKernelConfigurator.getBeanInfo(AbstractKernelConfigurator.java:97)
                  at org.jboss.kernel.plugins.dependency.PreInstallAction.installActionInternal(PreInstallAction.java:88)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
                  at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
                  at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
                  at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:894)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:641)
                  at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:182)
                  at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:58)
                  at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:63)
                  at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)
                  at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1571)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983)
                  at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076)
                  at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679)
                  at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106)
                  at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143)
                  at org.jboss.profileservice.dependency.ProfileDeployAction.deploy(ProfileDeployAction.java:151)
                  at org.jboss.profileservice.dependency.ProfileDeployAction.installActionInternal(ProfileDeployAction.java:94)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
                  at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
                  at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
                  at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
                  at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
                  at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)
                  at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)
                  at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)
                  at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)
                  at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)
                  at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)
                  at org.jboss.profileservice.dependency.ProfileActivationWrapper$BasicProfileActivation.start(ProfileActivationWrapper.java:190)
                  at org.jboss.profileservice.dependency.ProfileActivationWrapper.start(ProfileActivationWrapper.java:87)
                  at org.jboss.profileservice.dependency.ProfileActivationService.activateProfile(ProfileActivationService.java:215)
                  at org.jboss.profileservice.dependency.ProfileActivationService.activate(ProfileActivationService.java:159)
                  at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.activate(AbstractProfileServiceBootstrap.java:112)
                  at org.jboss.profileservice.resolver.BasicResolverFactory$ProfileResolverFacade.deploy(BasicResolverFactory.java:87)
                  at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.start(AbstractProfileServiceBootstrap.java:91)
                  at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:132)
                  at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56)
                  at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827)
                  at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417)
                  - locked <0x94180890> (a org.jboss.embedded.core.server.JBossASEmbeddedServerImpl)
                  at java.lang.Thread.run(Thread.java:619)
              
              Found 1 deadlock.
              
              • 4. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
                asoldano

                I can't really say why this has started happening only recently, in any case I've digged a bit into the problem, as yesterday a pretty much equivalent deadlock popped up on the jbossws hudson with jbossws-cxf trunk against AS 7 current master. Below is my analysis.

                The deadlock is between the system classloader and sun.net.www.protocol.file.Handler, due to their synchronized loadClass() and openConnection()/getHostAddress() methods. During initialization, Apache CXF performs some hacks, including disabling url caching; to achieve that, it opens a new connection as JDK does not have a static method for setting that, despite the flag being actually static. The process of opening a connection goes throught the Handler's syncronized openConnection method and in there ends up needing the system classloader for loading a class. Unfortunately, the lock on the system classloader might be taken by another thread invoking the synchronized loadClass() on it and also needing to use the CodeSource for locating the class to be loaded. That eventually ends up in needing the URLStreamHandler / Handler for comparing URL instances.

                The URLStreamHandler is created using the configured URLStreamHandler factory, which can be configured setting the java.protocol.handler.pkgs system property. AFAIK we do that for properly dealing with VFS stuff and the factory we configure basically return an handler that is a singleton. Hence the deadlock.

                AFAIK, the default JDK URLStreamHandler factory sun.misc.Launcher$Factory creates a new handler each time its createURLStreamHandler method is invoked.

                 

                I've just committed a patch on CXF side (https://issues.apache.org/jira/browse/CXF-3634) to avoid actually opening the connection during the initialization (it should be enought to create an instance of URLConnection and call the setDefaultUseCaches on it) and that seems to solve the problem I could see on jbossws hudson instance.

                However, we should probably evaluate the possible drawbacks of using a singleton URLStreamHandler (users opening connections still have a chance to cause deadlocks).

                • 5. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
                  dmlloyd

                  The only difference I can think of offhand which might cause this is that VFS no longer registers itself automatically as the sole URL handler factory.  There's some blob in common-core that should be handling this, iirc.  We might have to add in something to AS main() to wedge it in early on.

                  • 6. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
                    alesj

                    The only difference I can think of offhand which might cause this is that VFS no longer registers itself automatically as the sole URL handler factory.  There's some blob in common-core that should be handling this, iirc.  We might have to add in something to AS main() to wedge it in early on.

                    That's what I already did -- see latest AS trunk; org.jboss.Main class in particular.

                     

                    The actual code that does this should actually live in Bootstrap project,

                    so that Embedded AS works the same way as well.

                     

                    Imo the newly added code to AS does the same what VFS did,

                    it probably even registers URLStreamHandlerFactory before VFS did,

                    but this still hangs the server, hence no idea why we didn't see this before.

                    • 7. Re: JBoss 6.1 build hangs at "JBoss Application Server EmbeddedAS TestSuite"
                      jaikiran

                      This is now passing with the bootstrap fix which ALR added https://issues.jboss.org/browse/JBAS-9417