12 Replies Latest reply: Nov 19, 2013 10:53 AM by Anton Hughes RSS

Building 2.4 beta1 in Jenkins hangs

Anton Hughes Novice

Hi all

 

We have upgraded to 2.4b1, however we cannot complete the build in Jenkins.

 

It hangs at:

 

[INFO] INFO [Reflections] Reflections took 24382 ms to scan 200 urls, producing 12203 keys and 53008 values

[INFO] INFO [IOCBootstrapGenerator] generating IOC bootstrapping class...

[INFO] WARN [EnvUtil] reachability analysis disabled. errai may generate unnecessary code.

[INFO] WARN [EnvUtil] enable reachability analysis with -Derrai.compile.perf.perform_reachability_analysis=true

 

 

We have tried adding " -Dgwt.extraJvmArgs="-Xmx1024m" ", however, this did not help.

 

Any futher ideas?

  • 1. Re: Building 2.4 beta1 in Jenkins hangs
    Christian Sadilek Master

    Hi Anton,

     

    I have never seen this behaviour. The only advice I have is to take a look at the memory usage and overall system load when that happens.

     

    What platform/OS is your Jenkins build running on? I assume your build passes locally. What differences are there between your local environment and Jenkins (e.g. JDK versions etc.)?

     

    Cheers,

    Christian

  • 2. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    Hi Christian

     

    Yes, it builds and runs locally.

     

    The build sever is an amazon linux server, running the latest oracle 64bit jdk.

     

    I build, using the same jdk on both ubuntu and windows.

     

    This is our same application, that builds fine on 2.3, on the same server, but after upgrading hangs. And the memory usage seems fine (I will get our sys admin to confirm this).

     

    Is there any other advice you can give, or areas that we should look into?

     

    We are having to maintain our 2.3 branch because of this issue.

  • 3. Re: Building 2.4 beta1 in Jenkins hangs
    Christian Sadilek Master

    Hi Anton,

     

    Another thing that's definitely worth trying is to give the JVM that maven runs in more memory:

     

    Add this to your .profile or run it before starting the build

    export MAVEN_OPTS=-Xmx2g

     

    Cheers,

    Christian

  • 4. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    Hi Christian

     

    I tried as you recommended, but the build still gets stuck.

     

    This is the last output, where it hangs. We are now attempting with 2.4 snapshot.

     

    [INFO] 2013-09-04 14:04:13 INFO Reflections:253 - Reflections took 1129 ms to scan 181 urls, producing 0 keys and 0 values

    [INFO] Scanning for additional dependencies: jar:file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jaxrs-client/2.4.0-SNAPSHOT/errai-jaxrs-client-2.4.0-SNAPSHOT.jar!/org/jboss/errai/enterprise/client/jaxrs/JaxrsModule.java

    [INFO] Computing all possible rebind results for 'org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader'

    [INFO] Rebinding org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader

    [INFO] Invoking generator org.jboss.errai.enterprise.rebind.JaxrsProxyLoaderGenerator

    [INFO] [WARN] You are using the deprecated statingPage place change to role = DefaultPage.class

    [INFO] 2013-09-04 14:04:13 INFO IOCBootstrapGenerator:139 - generating IOC bootstrapping class...

    [INFO] 2013-09-04 14:04:13 WARN EnvUtil:436 - reachability analysis disabled. errai may generate unnecessary code.

    [INFO] 2013-09-04 14:04:13 WARN EnvUtil:437 - enable reachability analysis with -Derrai.compile.perf.perform_reachability_analysis=true

    [INFO] Checking ErraiApp.properties for configured types ...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-bus/2.4.0-SNAPSHOT/errai-bus-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jpa-datasync/2.4.0-SNAPSHOT/errai-jpa-datasync-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-navigation/2.4.0-SNAPSHOT/errai-navigation-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ioc/2.4.0-SNAPSHOT/errai-ioc-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ioc-bus-support/2.4.0-SNAPSHOT/errai-ioc-bus-support-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ui/2.4.0-SNAPSHOT/errai-ui-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking /var/lib/jenkins/workspace/app.ui%202.4%20upgrade%20test/src/main/resources/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jaxrs-client/2.4.0-SNAPSHOT/errai-jaxrs-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-validation/2.4.0-SNAPSHOT/errai-validation-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-data-binding/2.4.0-SNAPSHOT/errai-data-binding-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jpa-client/2.4.0-SNAPSHOT/errai-jpa-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-weld-integration/2.4.0-SNAPSHOT/errai-weld-integration-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-marshalling/2.4.0-SNAPSHOT/errai-marshalling-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-tools/2.4.0-SNAPSHOT/errai-tools-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-cdi-client/2.4.0-SNAPSHOT/errai-cdi-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...

    [INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking /var/lib/jenkins/workspace/app.ui%202.4%20upgrade%20test/src/main/webapp/WEB-INF/classes/ErraiApp.properties for bindable types...

  • 5. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    Does anyone have any ideas.

     

    We are still very much stuck due to this problem.

     

    The only difference between the jenkins job that does build and the one that doesnt is upgrading to 2.4.

  • 6. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    And I tried using 3.0-SNAPSHOT and get exactly the same result.

  • 7. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    Can you give us some idea of what has changed from 2.3 to 2.4 that might be causing this problem?

  • 8. Re: Building 2.4 beta1 in Jenkins hangs
    Jonathan Fuerth Master

    Hi Anton,

     

    As Christian said already, we haven't seen any Errai builds stalling, so we can't really make a reasonable guess about what might be causing this on your end.

     

    The best I can do is to offer some generic troubleshooting advice:

     

    1. When the build stops making progress, examine the thread stacks to find out what it's stuck on. There are many ways to do this:
      1. send the process a SIGQUIT (kill -QUIT <pid>), or
      2. use the jstack command, or
      3. use the VisualVM tool that comes with the JDK, or
      4. attach an IDE debugger
    2. Monitor the system calls the JVM is making. Watch for failed calls and calls that never return. Does it stop after a filesystem error? Is it hanging on a network connection? A DNS lookup? Depending on which flavour of Unix you're on, the command for doing this will be strace (Linux), ktrace (BSD/OS X), truss (Solaris), or dtrace (Solaris, FreeBSD, OS X). On Windows, you can use Sysinternals Process Monitor.
    3. If all of the above fails, clone the Errai codebase from github and use git bisect to find the troublesome changeset. The procedure (run all the shell commands on the Jenkins machine):
      1. Set up a Jenkins build for your project that depends on Errai 2.4.0-SNAPSHOT
      2. git clone https://github.com/errai/errai
      3. cd errai
      4. git checkout 2.4.0.Final
      5. git bisect start
      6. git bisect bad
      7. git bisect good 2.3.2.Final [git will now check out a revision between the 'good' and 'bad' revisions]
      8. mvn version:set -DnewVersion=2.4.0-SNAPSHOT
      9. mvn clean install -Dgwt.compiler.skip=true -Dmvn.test.skip=true
      10. run the Jenkins build. if it fails on import statements, you might need to compensate by doing an "organize imports" on the project
      11. mvn version:rollback [or git reset --hard; the point of this step is to eliminate any changed files that would make git refuse to switch to the next revision in the bisect]
      12. if the Jenkins build hangs, do a 'git bisect bad'; if it completes, do a 'git bisect good'
      13. goto 8

     

    Hopefully the stack dumps or syscall tracing will be enough to track down the cause of the problem. But if not, it's kind of fun to watch git bisect do its thing, and it's a great skill to add to your troubleshooting toolbox.

     

    -Jonathan

  • 9. Re: Building 2.4 beta1 in Jenkins hangs
    Cristian Miranda Newbie

    Hi Jonathan, Christian,

     

    We're having the same issue with 2.4.2.Final version of Errai.

     

    I've also tried with two different sample apps using Errai 2.4.2.Final and both builds hang in Jenkins at this point.

     

    INFO] INFO [IOCBootstrapGenerator] generating IOC bootstrapping class...

    [INFO] WARN [EnvUtil] reachability analysis disabled. errai may generate unnecessary code.

    [INFO] WARN [EnvUtil] enable reachability analysis with -Derrai.compile.perf.perform_reachability_analysis=true

    [INFO] Computing all possible rebind results for 'org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader'

    [INFO] Rebinding org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader

    [INFO] Invoking generator org.jboss.errai.enterprise.rebind.JaxrsProxyLoaderGenerator

    [INFO] Checking ErraiApp.properties for configured types ...

     

    Any ideas?


  • 10. Re: Building 2.4 beta1 in Jenkins hangs
    Jonathan Fuerth Master

    Hi Cristian,

     

    Can you post a stack dump of the stuck JVM? I posted instructions on how to obtain one in this forum thread on October 8, 2013.

     

    -Jonathan

  • 11. Re: Building 2.4 beta1 in Jenkins hangs
    Christian Sadilek Master

    Hi Anton and Cristian,

     

    Our co-worker Mauricio Salatino was able to reproduce this issue on machines that only have a single CPU available. I just pushed a fix for this and published new 3.0-SNAPSHOTs and 2.4.3-SNAPSHOTs. It's very likely that you've been facing the same issue. So, it's worth giving the latest snapshots a try.

     

    Here's the JIRA: [ERRAI-671] Build/Compilation hangs when running with only one CPU - JBoss Issue Tracker

     

    Cheers,

    Christian

  • 12. Re: Building 2.4 beta1 in Jenkins hangs
    Anton Hughes Novice

    HI Christian

     

    Thanks for the update.

     

    Unfortunately it came 1 day too late. We setup a new jenkins, on a new server, and am now basking in the warm glow of Errai 2.4.2.

     

    Still, its good to here that there is a fix for this issue.

     

    Thanks for looking into it.