When I report issue in bug tracking system about nightly build's JUnit test error I usually get simple answer that it is supposed to be working  because it is working on developer's workstation. After that routine conversation starts and it turns out that tests were running from development environment under Eclipse. Here I usually have to explain again and again that's not the same running tests from development environment and in build.

The right way to make yourself sure your tests will work in most cases without errors in nightly build is to start tests the same way as nightly build does. It was not easy for JBoss Tools tests until we created experimental branch and switched to Maven Tycho project. That means it is fairly easy to run tests now. Basically you need to change current directory and execute maven install goal. If it runs in development environment and in maven your tests are good and in most cases it should be fine in nightly build. Problems begin if it runs in development environment but it doesn't in maven. In this scenario you need to debug tests running in Tycho somehow and fix it. Fortunately it can be done using Java remote debugging support.

First of all you need to be sure you have built your sources you're going to debug and there is no differences between .java and  .class files. If you're going find problem from previous build just get right tagged version and build it before debugging session.

There are two options to use remote debugger

1. Simple use of -DdebugPort=8001 or what ever port you would like to use
2. Add full argLine for remote debugging configuration im pom.xml

Use debugPort system property (Recommended)

To use this just add -DdebugPort=<portNumber> to your maven command line replacing <portNumber> for desired port and make sure you have the same port configured in your Remote Java Application configuration in Eclipse Debug Configuration dialog as it explained for second option below.

Or use Full argLine for Remote Debugging Configuration in pom.xml (Deprecated)

Open pom.xml for your Eclipse Test Plug-in and add Java VM arguments like it shown below (actual port numbers, server names and  other parameters may be different, it depends from your environment)

<build>
<plugins>
<plugin>
    <groupid>org.sonatype.tycho</groupid>
       <artifactid>maven-osgi-test-plugin</artifactid>
       <version>${tycho-version}</version>
       <configuration>
            <argLine>-Xdebug -Xrunjdwp:transport=dt_socket,address=8001,server=y,suspend=y</argLine>
       </configuration>
    </plugin>
</plugins>
</build>

This snipped configured for remote debugging in OpenJDK 6, if your is different you might need to read [1] and configure it right for your JVM version.

Configure java projects with sources you're going to debug in Eclipse and create Remote Java Application configuration in Eclipse Debug Configuration dialog. Fill 'host' and 'port' fields with the same values from pom.xml argLine element. Press Apply button to save your changes and start your test plug-in from terminal like

$mvn install

It will go through build process and finally you ll see something like

[INFO] Expected eclipse log file: /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/data/.metadata/.log
[INFO] Command line:
 /bin/sh -c cd /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test && /usr/lib/jvm/java-6-openjdk/jre/bin/java -Dosgi.noShutdown=false -Dosgi.os=linux -Dosgi.ws=gtk -Dosgi.arch=x86 -agentlib:jdwp=transport=dt_socket,address=8001,server=y,suspend=y -jar /home/eskimo/.m2/repository/p2/osgi/bundle/org.eclipse.equinox.launcher/1.0.201.R35x_v20090715/org.eclipse.equinox.launcher-1.0.201.R35x_v20090715.jar -data /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/data -dev file:/home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/dev.properties -install /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work -configuration /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/work/configuration -application org.codehaus.tycho.surefire.osgibooter.uitest -testproperties /home/eskimo/Projects/jbt-modular/jst/tests/org.jboss.tools.jst.web.kb.test/target/surefire.properties
Listening for transport dt_socket at address: 8001

At this point build is waiting for you to attach remote debugger to the process because of using

suspend=y

in argLine element of your pom.xml file. So you need to return to eclipse and hit 'Debug' button in dialog opened before.

Build will continue at this point and stop on your break points so you can find out what is wrong with tests during nightly build.

[1] Java  Platform Debugger Architecture -  http://java.sun.com/javase/technologies/core/toolsapis/jpda/#Invocation

Words of Warning

When building and debugging your code locally, be aware that your eclipse environment or target platform inside the running eclipse should match the environment used to launch your tests. When running locally via the mvn clean verify command, tycho will default to using a MINIMUM configuration for both building AND testing your code. To ensure the code is compiled, and tests are launched, against a SPECIFIC target platform, make sure to use the -Pminimum or  -Pmaximum flags in your build command.

When debugging from within eclipse, if your current environment does NOT match the environment tycho uses to launch your tests, you may experience many types of confusing behavior. The most obvious will be if the source code lines do not match up to what is being executed, but other problems such as inaccessible or unhit breakpoints, or illogical code flows, may confuse you for a very long time. If you are running your tests with a -Pminimum flag, ensure your eclipse is set up to use that configuration. Similarly, If you are running your tests with a -Pmaximum flag, make sure you are using an environment which matches that target platform.

This will save you many hours