-
1. Re: Building 2.4 beta1 in Jenkins hangs
csa Jul 24, 2013 6:32 PM (in response to magick93)Hi Anton,
I have never seen this behaviour. The only advice I have is to take a look at the memory usage and overall system load when that happens.
What platform/OS is your Jenkins build running on? I assume your build passes locally. What differences are there between your local environment and Jenkins (e.g. JDK versions etc.)?
Cheers,
Christian
-
2. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Jul 25, 2013 4:07 AM (in response to csa)Hi Christian
Yes, it builds and runs locally.
The build sever is an amazon linux server, running the latest oracle 64bit jdk.
I build, using the same jdk on both ubuntu and windows.
This is our same application, that builds fine on 2.3, on the same server, but after upgrading hangs. And the memory usage seems fine (I will get our sys admin to confirm this).
Is there any other advice you can give, or areas that we should look into?
We are having to maintain our 2.3 branch because of this issue.
-
3. Re: Building 2.4 beta1 in Jenkins hangs
csa Jul 31, 2013 5:09 PM (in response to magick93)Hi Anton,
Another thing that's definitely worth trying is to give the JVM that maven runs in more memory:
Add this to your .profile or run it before starting the build
export MAVEN_OPTS=-Xmx2g
Cheers,
Christian
-
4. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Sep 4, 2013 10:24 AM (in response to csa)Hi Christian
I tried as you recommended, but the build still gets stuck.
This is the last output, where it hangs. We are now attempting with 2.4 snapshot.
[INFO] 2013-09-04 14:04:13 INFO Reflections:253 - Reflections took 1129 ms to scan 181 urls, producing 0 keys and 0 values
[INFO] Scanning for additional dependencies: jar:file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jaxrs-client/2.4.0-SNAPSHOT/errai-jaxrs-client-2.4.0-SNAPSHOT.jar!/org/jboss/errai/enterprise/client/jaxrs/JaxrsModule.java
[INFO] Computing all possible rebind results for 'org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader'
[INFO] Rebinding org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader
[INFO] Invoking generator org.jboss.errai.enterprise.rebind.JaxrsProxyLoaderGenerator
[INFO] [WARN] You are using the deprecated statingPage place change to role = DefaultPage.class
[INFO] 2013-09-04 14:04:13 INFO IOCBootstrapGenerator:139 - generating IOC bootstrapping class...
[INFO] 2013-09-04 14:04:13 WARN EnvUtil:436 - reachability analysis disabled. errai may generate unnecessary code.
[INFO] 2013-09-04 14:04:13 WARN EnvUtil:437 - enable reachability analysis with -Derrai.compile.perf.perform_reachability_analysis=true
[INFO] Checking ErraiApp.properties for configured types ...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-bus/2.4.0-SNAPSHOT/errai-bus-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jpa-datasync/2.4.0-SNAPSHOT/errai-jpa-datasync-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-navigation/2.4.0-SNAPSHOT/errai-navigation-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ioc/2.4.0-SNAPSHOT/errai-ioc-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ioc-bus-support/2.4.0-SNAPSHOT/errai-ioc-bus-support-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-ui/2.4.0-SNAPSHOT/errai-ui-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking /var/lib/jenkins/workspace/app.ui%202.4%20upgrade%20test/src/main/resources/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jaxrs-client/2.4.0-SNAPSHOT/errai-jaxrs-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-validation/2.4.0-SNAPSHOT/errai-validation-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-data-binding/2.4.0-SNAPSHOT/errai-data-binding-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-jpa-client/2.4.0-SNAPSHOT/errai-jpa-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-weld-integration/2.4.0-SNAPSHOT/errai-weld-integration-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-marshalling/2.4.0-SNAPSHOT/errai-marshalling-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-tools/2.4.0-SNAPSHOT/errai-tools-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking file:/var/lib/jenkins/.m2/repository/org/jboss/errai/errai-cdi-client/2.4.0-SNAPSHOT/errai-cdi-client-2.4.0-SNAPSHOT.jar!/ErraiApp.properties for bindable types...
[INFO] 2013-09-04 14:04:15 DEBUG DataBindingUtil:293 - Checking /var/lib/jenkins/workspace/app.ui%202.4%20upgrade%20test/src/main/webapp/WEB-INF/classes/ErraiApp.properties for bindable types...
-
5. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Oct 3, 2013 11:46 AM (in response to magick93)Does anyone have any ideas.
We are still very much stuck due to this problem.
The only difference between the jenkins job that does build and the one that doesnt is upgrading to 2.4.
-
6. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Oct 3, 2013 12:06 PM (in response to magick93)And I tried using 3.0-SNAPSHOT and get exactly the same result.
-
7. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Oct 8, 2013 7:38 AM (in response to magick93)Can you give us some idea of what has changed from 2.3 to 2.4 that might be causing this problem?
-
8. Re: Building 2.4 beta1 in Jenkins hangs
jfuerth Oct 8, 2013 10:19 AM (in response to magick93)Hi Anton,
As Christian said already, we haven't seen any Errai builds stalling, so we can't really make a reasonable guess about what might be causing this on your end.
The best I can do is to offer some generic troubleshooting advice:
- When the build stops making progress, examine the thread stacks to find out what it's stuck on. There are many ways to do this:
- send the process a SIGQUIT (kill -QUIT <pid>), or
- use the jstack command, or
- use the VisualVM tool that comes with the JDK, or
- attach an IDE debugger
- Monitor the system calls the JVM is making. Watch for failed calls and calls that never return. Does it stop after a filesystem error? Is it hanging on a network connection? A DNS lookup? Depending on which flavour of Unix you're on, the command for doing this will be strace (Linux), ktrace (BSD/OS X), truss (Solaris), or dtrace (Solaris, FreeBSD, OS X). On Windows, you can use Sysinternals Process Monitor.
- If all of the above fails, clone the Errai codebase from github and use git bisect to find the troublesome changeset. The procedure (run all the shell commands on the Jenkins machine):
- Set up a Jenkins build for your project that depends on Errai 2.4.0-SNAPSHOT
- git clone https://github.com/errai/errai
- cd errai
- git checkout 2.4.0.Final
- git bisect start
- git bisect bad
- git bisect good 2.3.2.Final [git will now check out a revision between the 'good' and 'bad' revisions]
- mvn version:set -DnewVersion=2.4.0-SNAPSHOT
- mvn clean install -Dgwt.compiler.skip=true -Dmvn.test.skip=true
- run the Jenkins build. if it fails on import statements, you might need to compensate by doing an "organize imports" on the project
- mvn version:rollback [or git reset --hard; the point of this step is to eliminate any changed files that would make git refuse to switch to the next revision in the bisect]
- if the Jenkins build hangs, do a 'git bisect bad'; if it completes, do a 'git bisect good'
- goto 8
Hopefully the stack dumps or syscall tracing will be enough to track down the cause of the problem. But if not, it's kind of fun to watch git bisect do its thing, and it's a great skill to add to your troubleshooting toolbox.
-Jonathan
- When the build stops making progress, examine the thread stacks to find out what it's stuck on. There are many ways to do this:
-
9. Re: Building 2.4 beta1 in Jenkins hangs
cristianmiranda Nov 13, 2013 2:17 PM (in response to jfuerth)Hi Jonathan, Christian,
We're having the same issue with 2.4.2.Final version of Errai.
I've also tried with two different sample apps using Errai 2.4.2.Final and both builds hang in Jenkins at this point.
INFO] INFO [IOCBootstrapGenerator] generating IOC bootstrapping class...
[INFO] WARN [EnvUtil] reachability analysis disabled. errai may generate unnecessary code.
[INFO] WARN [EnvUtil] enable reachability analysis with -Derrai.compile.perf.perform_reachability_analysis=true
[INFO] Computing all possible rebind results for 'org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader'
[INFO] Rebinding org.jboss.errai.enterprise.client.jaxrs.JaxrsProxyLoader
[INFO] Invoking generator org.jboss.errai.enterprise.rebind.JaxrsProxyLoaderGenerator
[INFO] Checking ErraiApp.properties for configured types ...
Any ideas?
-
10. Re: Building 2.4 beta1 in Jenkins hangs
jfuerth Nov 13, 2013 2:25 PM (in response to cristianmiranda)Hi Cristian,
Can you post a stack dump of the stuck JVM? I posted instructions on how to obtain one in this forum thread on October 8, 2013.
-Jonathan
-
11. Re: Building 2.4 beta1 in Jenkins hangs
csa Nov 18, 2013 7:02 PM (in response to jfuerth)1 of 1 people found this helpfulHi Anton and Cristian,
Our co-worker Mauricio Salatino was able to reproduce this issue on machines that only have a single CPU available. I just pushed a fix for this and published new 3.0-SNAPSHOTs and 2.4.3-SNAPSHOTs. It's very likely that you've been facing the same issue. So, it's worth giving the latest snapshots a try.
Here's the JIRA: [ERRAI-671] Build/Compilation hangs when running with only one CPU - JBoss Issue Tracker
Cheers,
Christian
-
12. Re: Building 2.4 beta1 in Jenkins hangs
magick93 Nov 19, 2013 10:53 AM (in response to csa)HI Christian
Thanks for the update.
Unfortunately it came 1 day too late. We setup a new jenkins, on a new server, and am now basking in the warm glow of Errai 2.4.2.
Still, its good to here that there is a fix for this issue.
Thanks for looking into it.