8 Replies Latest reply on Dec 22, 2011 8:26 PM by dan.j.allen

    Proposal: Filtering out tests not compatible with target container

    dan.j.allen

      I'd like to introduce into Arquillian the concept of execution environments (e.g., Java EE 6, EJB, CDI, OSGi, etc), the containers which provide them and a declarative way for a test to indicate an execution environment as a prerequisite. Using this information, the Arquillian test runner could automatically filter out tests that are not compatible with the target container. This is common requirement when mixing integration levels in a test suite. We already have this requirement in the Weld test suite, the JSR-299 TCK and the Arquillian examples.

       

      Currently, we are using include and exclude filters in the build script (e.g., the Maven surefire plugin configuration) to select which test classes will run when targeting a given container. This pushes information about how the test functions into external configuration, something we've tried to get away from in modern programming, particularly in the Java EE 6 programming model. Without that external configuration, you lose part of the logic that belongs to the test. Not only does this make the test dependent on the build, it also makes it difficult for new team members to pick up how the test suite operates. In short, it's voodoo magic. We want this information kept with the test. And we want the test suite to be executed intelligently.

       

      This feature request is tracked in the issue ARQ-287. I'm going to introduce two proposals in this thread, which I've prototyped in matching branches (ARQ-287-proposalA and ARQ-287-proposalB) my Arquillian github fork. In both branches, the Arquillian JUnit examples no longer have any include/exclude filters in the Maven surefire configuration!

       

      Before getting into the how, let's take a look at an example scenario.

       

      Example scenario

       

      Let's assume we want to write a test that uses an EJB that in turn uses a JMS resource (information not evident from looking at the test case). Such a test would not work unless the target container provides Java EE 6. An embedded EJB container is not sufficient. Thus, we need to declare that the test requires a Java EE 6 environment to communicate this information (and the intent of the test).

       

      The end result will look something like this:

       

      @RunWith(Arquillian.class)
      @RequiresJavaEE6
      public class MyServiceTestCase {
         @Deployment
         public static void createDeployment() { ... }
      
         @EJB
         MyService service;
      
         @Test
         public void shouldPutMessageOntoQueue() { ... } 
      }

       

      In another Java package, we may have a test that only uses CDI. In this case, an embedded CDI runtime is sufficient, though a full Java EE 6 container would certainly work as well.

       

      @RunWith(Arquillian.class)
      @RequiresCDI
      public class BeanManagerTestCase {
         @Deployment
         public static void createDeployment() { ... }
      
         @Inject
         BeanManager beanManager;
      
         @Test
         public void shouldRegisterMyBean() { ... } 
      } 

       

      Now we should be able to run our test suite without any special configuration (other than the Arquillian container integration) and Arquillian would select the tests to execute based on the target container (skipping the ones that are irrelevant).

       

      To accommodate this feature, we need to provide a way to:

       

      1. declare an execution environment
      2. associate execution environments with containers (and broader execution environments) and
      3. filter out the tests that require an execution environment which the target container doesn't provide

       

      Due to the extensible nature of Arquillian, we can't think of every possible execution environment, nor can we cover every Arquillian container integration that exists, present or future. Therefore, the way execution environments are defined must be extensible as well. My idea is to provide an API, define the most popular environments and allow the user to extend the built-in definitions or create their own. The filter logic should be able to accommodate any definition.

       

      The public API revolves around a meta-annotation design. I have two proposals, one which makes the execution environment a type, the other which is more CDI-ish by making more liberal use of meta-annotations. (Both APIs suffer a bit from the the fact that an annotation cannot implement an interface).

       

      Proposal A

       

      Propotype branch: http://github.com/mojavelinux/arquillian/tree/ARQ-287-proposalA

       

      In proposal A, execution environments implement a common interface, shown below, and are annotated with information that links them to containers (via @ProvidedBy).

       

      package org.jboss.arquillian.api.environment;
      
      public interface ExecutionEnvironment {} 

       

      Here's the execution enviornment specification for Java EE 6:

       

      package org.jboss.arquillian.api.environment.spec;
      
      import org.jboss.arquillian.api.environment.ExecutionEnvironment;
      import org.jboss.arquillian.api.environment.ProvidedBy;
      
      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.glassfish\\.(managed|remote|embedded)_3(_[0-9]+)*",
            "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_6(_[0-9]+)*"
         }
      )
      public final class JavaEE6Environment implements ExecutionEnvironment {} 

       

      An execution environment isn't much use without a way to indicate which runtimes provide it. Thus, we introduce the @ProvidedBy annotation to make the link between an execution environment and a collection of containers. If the container identifier is prefixed with "~ ", it's treated as a regular expression to allow containers to be match more broadly.

       

      The somewhat kludgey part here is the way a container is identified. The Arquillian container configuration (arquillian.xml) uses the name of the container implementation's package to identify a container, an association which I've adopted here.

       

      Note: I would prefer that we come up with a more robust mechanism for identifying a container that is not linked to the organization of the source code. Perhaps to be discussed in a seperate discussion thread, I'd like for containers to declare a unique identifier, a common name, vendor and the supported version range. This will also benefit the container configuration feature.

       

      The @ProvidedBy annotation can also specify other execution environments which provide the execution environment being defined. For instance, CDI is provided by embedded runtimes as well as any Java EE 6 container:

       

      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*",
            "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*"
         },
         environments = JavaEE6Environment.class
      )
      public final class CDIEnvironment implements ExecutionEnvironment {}
      

       

      The link to other environments is not only a way to define execution environment hierarchies, but also a way for the developer to add additional containers missed by the built-in execution environments. Developers also have the option of combining ExecutionEnvironment and @ProvidedBy to define a custom execution environment. It's totally extensible.

       

      For instance, some of the Arquillian JUnit examples can only be run on JBoss AS, so we define it as an execution environment:

       

      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_5(_[0-9]+)*"
         }
      )
      public final class JBossAS6Container implements ExecutionEnvironment {}
      

       

      Moving on, we need to address how a developer declares the environment required by a test. For that, we introduce the @RequiresEnvironment annotation. This can be applied to the test class in one of two ways. The first would be to use this annotation directly on the test class:

       

      @RunWith(Arquillian.class)
      @RequiresEnvironment(JavaEE6Environment.class)
      public class MyServiceTestCase { ... }
      

       

      However, that's kind of a lot of typing and it's not very elegant. The second, preferred approach would be to define a rule annotation that uses @RequiresEnvironment as a meta-annotation:

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.TYPE)
      @RequiresEnvironment(JavaEE6Environment.class)
      public @interface RequiresJavaEE6 {}
      

       

      This annotation is declared on the test class, hiding the link to the execution environment Java type as an implementation detail:

       

      @RunWith(Arquillian.class)
      @RequiresJavaEE6
      public class MyServiceTestCase { ... }
      

       

      I'll leave the choice open to the developer because honestly it's not much extra work to support both.

       

      We now move on to how the filtering is done. We need to hook into the test runner and mark a test class as skipped if the target container does not match the @ProvidedBy definition of the required execution environment. The approach I've taken is to introduce a method to the Arquillian TestRunnerAdaptor SPI that checks whether the test class is compatible:

       

      public interface TestRunnerAdaptor {
         ...
         /**
          * Checks whether the test class about to be executed is compatible
          * with the execution environment of the test (e.g., the target container)
          */
         boolean isCompatible(Class testClass);
      }
      

       

      When this is method is invoked is going to be specific to the test framework integration. At this point, only JUnit exposes the capability of skipping a test from the test listener programmtically. (Equivalent support is coming to TestNG soon). Here's the logic for JUnit:

       

      @Override
      public void run(RunNotifier notifier)
      {
         if(!deployableTest.get().isCompatible(getTestClass().getJavaClass()))
         {
            ignore = true;
            notifier.fireTestIgnored(getDescription());
            logger.info(testClass.getName() + " ignored since target container " +
               "does not provide required execution environment");
            return;
         }
         ...
      }
      

       

      Another kludgey is part is determining what container is being used and how to identify it. Again, I see this more of a general design consideration in Arquillian which really needs to be addressed to support the feature being proposed in this thread.

       

      I've prepared a second proposal which only really differs in how the execution environment annotations are defined.

       

      Proposal B

       

      Propotype branch: http://github.com/mojavelinux/arquillian/tree/ARQ-287-proposalB

       

      JSR-299 and Bean Validation really demonstrated how suitable annotations are as binding identifiers. I've tried to capture that design in proposal B by turning the execution environment interface into an annotation.

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.ANNOTATION_TYPE)
      @Documented
      public @interface ExecutionEnvironment {}

       

      Execution environment implementations would also be defined as annotations:

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.ANNOTATION_TYPE)
      @ExecutionEnvironment
      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.glassfish\\.(managed|remote|embedded)_3(_[0-9]+)*",
            "~ org\\.jboss\\.arquillian\\.container\\.jbossas\\.(managed|remote|embedded)_6(_[0-9]+)*"
         }
      )
      public @interface JavaEE6Environment {}

       

      An execution environment hierarchy would be defined in one of two ways. The first (implemented in the proposal B branch) mimics proposal A by specifying them in the @ProvidedBy annotation:

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.ANNOTATION_TYPE)
      @ExecutionEnvironment
      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*",
            "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*"
         },
         environments = JavaEE6Environment.class
      )
      public @interface CDIEnvironment {}

       

      The other option (not implemented) would be to use the parent execution environment as a meta-annotation:

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.ANNOTATION_TYPE)
      @ExecutionEnvironment
      @ProvidedBy(
         containers =
         {
            "~ org\\.jboss\\.arquillian\\.container\\.weld\\..*",
            "~ org\\.jboss\\.arquillian\\.container\\.openwebbeans\\..*"
         }
      )
      @JavaEE6Environment
      public @interface CDIEnvironment {}

       

      The required environment annotations differ from proposal A as well. Following the meta-annotation design, instead of using @RequiresEnvironment, we mark a required execution environment as a type of stereotype (to use CDI terminology) using the @RequiredEnvironmentStereotype annotation as a sibling of the execution environment annotation:

       

      @Retention(RetentionPolicy.RUNTIME)
      @Target(ElementType.TYPE)
      @JavaEE6Environment
      @RequiredEnvironmentStereotype
      public @interface RequiresJavaEE6 {}

       

      This required execution environment would be applied to the test class in the same way as proposal A.

       

      Conclusion

       

      I'm not sure whether proposal A or B is better. It may not even really matter as the end result (e.g., @RequiresJavaEE6) is the same. The best part is that the Arquillian JUnit examples no longer have any include/exclude filters in the Maven surefire configuration!

       

      I should note that I strongly oppose doing any of this in XML-based configuration. I want the information about what the test requires in the test class. I am open to considering other ways in which containers are linked to execution environments.

       

      I've focused on avoiding annotation scanning to keep overhead low and want to provide common execution environments out of the box to limit the work the developer has to do to use this feature.

        • 1. Re: Proposal: Filtering out tests not compatible with target container
          aslak

          Grouping and the execution environment of a TestClass goes beyond the Container Type, the ContainerConfiguration is a part of the 'key'. Especially when we move over to support for multiple server deployment/container in the same run.

           

          TestA Run on JBoss AS 6 against Server A

          TestB Run on JBoss AS 6 against Server B

           

          Being based on code/packages/layout, the current proposal does not support this.

           

          <jboss:container name="Server A">

               <jboss:remoteAddress />

          </jboss:container>

          <jboss:container name="Server B">

               <jboss:remoteAddress />

          </jboss:container>

          • 2. Re: Proposal: Filtering out tests not compatible with target container
            dan.j.allen

            Point taken. Allow me to provide some more explanation about this proposal.

             

            In general (not necessarily a response to your post), I see execution environment filtering and test case grouping as orthogonal concerns. The first is an absolute. Certain integration tests simply won't function unless inside a particular execution environment. That is information that belongs in the test class and a test should not be included when the target container doesn't provide that execution environment (and this information should not be mixed up with group classifiers).

             

            Developers may then decide to group tests based on speed, domain, etc, which can be achieved using TestNG groups or JUnit categories. (In fact, one of the nice features of Arquillian is that it does not stand in your way from using native features of the underlying test framework). Execution environment filtering would still need to be applied to a group/category test run.

             

            I do understand your point, though, that an execution environment may be defined as a specific configuration of a container (for instance, a container running with a certain JPA provider or running on a specific server). That's not too much of a stretch from the current proposal.

             

            As I mentioned in the first post, I don't really want the container association to be based on the packaging structure of the container implementation class. My hands are just tied at the moment because that's currently the only way to identify a container. On top of that, as you pointed out, we don't support multiple configurations of the same container anyway, so I couldn't code support for that in just yet.

             

            I was just thinking about the container association tonight, and I determined that if we added some metadata to the container implementation classes, it would help us build a more robust reference to the target container. I'm thinking something along the lines of:

             

            @ContainerAdapter(
               id = "jbossas-remote-6",
               profile = REMOTE,
               container = @ContainerIdentifier(
                  id = "jbossas",
                  name = "JBoss AS",
                  vendor = @VendorIdentifier(
                     id = "jboss",
                     name = "JBoss"
                  )
               ),
               supportedVersions = @VersionRange(min = "6.0.0.M4", UNBOUNDED)
            )
            public class JBossASRemoteContainer implements DeployableContainer { ... }

             

            (If there is a way to make constants out of the ContainerIdentifier and VendorIdentifier, we can trim some of the common information here).

             

            Then the container association would become more elegant:

             

            @ProvidedBy(
               containers =
               {
                  @ContainerMatcher(id = "jbossas", profile = ALL, minVersion = "6.0.0.M4"),
                  @ContainerMatcher(id = "glassfish", profile = ALL, minVersion = "3.0"),
               }
            )
            

             

            We could also introduce a configuration id property to the criteria to match only certain configurations of a container:

             

             

            @ProvidedBy(
               containers =
               {
                  @ContainerMatcher(id = "jbossas", profile = ALL, minVersion = "6.0.0.M4",
                     configurations = "Server A"),
                  @ContainerMatcher(id = "glassfish", profile = ALL, minVersion = "3.0"),
               }
            )
            

             

            (The annotation and property names are just suggestions and can be tuned as needed).

             

            I don't really see a way to avoid putting the container matchers on the execution environment itself without introducing annotation scanning or XML-based configuration. Besides, there really isn't any harm done following this approach. We are merely providing the most common execution environments, linked to the containers we support that provide them, as a convenience and model to follow. The developer could start all over and define his/her own specs and rules. We could also consider pushing these common specs and rules out into a separate artifact (arquillian-common-execution-environments), to keep from cluttering up the API.

             

            To really keep the door open, we should make sure to allow the user can plug in custom filtering logic (overriding/extending the behavior of the isCompatible() method), in case they want to change how Arquillian selects which test classes to skip. I've just hard coded an implementation into my prototype for the time being.

            • 3. Re: Proposal: Filtering out tests not compatible with target container
              dan.j.allen

              As a side note, Arquillian makes it really difficult to lookup the identifier of the target container. This information should be more readily accessible.

              • 4. Re: Proposal: Filtering out tests not compatible with target container
                dan.j.allen

                Ah! And to make the built-in specs more extensible, we can combine a configuration with an inherited runtime:

                 

                @ExecutionEnvironment
                @ProvidedBy(
                   environments = @EnvironmentMatcher(configuration = "Server A", type = JavaEE6Environment.class)
                )
                public @interface JavaEE6ContainerOnServerA {}
                

                 

                That would effectively cross the configuration with the environment spec, and any specs it inherits.

                 

                (I haven't decided whether or not the configuration() property should be plural in the matchers).

                 

                We would need to switch the type of environments() to EnviromentMatcher, though that would open up the door to adding an exact() property to match only the referenced environment and not any of its parents. For instance:

                 

                 

                @ExecutionEnvironment
                @ProvidedBy(
                   environments = @EnvironmentMatcher(type = CDIEnvironment.class, exact = true)
                )
                public @interface CDIStandaloneEnvironment {}
                

                 

                Just an idea.

                • 5. Re: Proposal: Filtering out tests not compatible with target container
                  dan.j.allen

                  Other names for @ContainerMatcher and @EnvironmentMatcher could be:

                   

                  • @ContainerProvider and @EnvironmentProvider
                  • @ContainerReference and @EnvironmentReference

                   

                  We may also want to keep the configuration keys out of the container matcher and only allow them in the environment matcher. That would just keep things DRY.

                  • 6. Re: Proposal: Filtering out tests not compatible with target container
                    dan.j.allen

                    I also realized that by having container adapter metadata, we can link environments to a container rather than to the Arquillian container adapter. Let's start with a simpler scheme for the container adapter metadata:

                     

                    @ContainerAdapter(
                       id = "jbossas-remote-6",  // substring of adapter artifactId
                       profile = ContainerProfile.REMOTE, // adapter profile 
                       version = 6, // generalized container version adapter targets
                       vendor = "jboss", // who created the adapter (not necessarily container vendor)
                       container = @ContainerIdentifier(
                          family = "jbossas", // the family of the container, referenced in adapter id
                          name = "JBoss AS", // formal name of container (optional)
                          vendorId = "jboss", // who creates the container, for grouping
                          vendorName = "JBoss", // formal name of container (optional)
                          versions = @VersionRange(min = @VersionDescriptor(major = 6)) // supported versions of container
                       )
                    )

                     

                    @VersionDescriptor is meant to break down a version string into major, minor and point release so that we can more easily compare them. It's hard to compare two version strings (unless you know it follows this scheme). We could also support version qualifiers (GA, Final, SP1) if we had an enum or something, though that might be too detailed. Most of the time major and minor are sufficient. (We could also have @VersionRange support an explicit list of versions via an explicit() property).

                     

                    Now when we defined the execution environment, the @ProvidedBy uses a selector based on the attributes of the container, not the adapter implementation. We define a @ContainerProvider annotation that is a selector for a container and an adapter that supports it:

                     

                    @ExecutionEnvironment
                    @ProvidedBy(
                       containers =
                       {
                          @ContainerProvider(family = "jbossas", versions =
                             @VersionRange(min = @VersionDescriptor(major = 6))),
                          ...
                       }
                    )
                    public @interface JavaEE6Environment {}
                    

                     

                    Here's an example with a more specific version range:

                     

                     

                    @ExecutionEnvironment
                    @ProvidedBy(
                       containers =
                       {
                          @ContainerProvider(family = "jbossas", versions = @VersionRange(
                             min = @VersionDescriptor(major = 5)), max = @VersionDescriptor(major = 5, minor = 1)),
                          ...
                       }
                    )
                    public @interface JavaEE5OnlyEnvironment {}
                    

                     

                    If we want only a remote Java EE 6 container, we could add a profile to the selector:

                     

                    @ExecutionEnvironment
                    @ProvidedBy(
                       containers =
                       {
                          @ContainerProvider(family = "jbossas", profiles = ContainerProfile.REMOTE,
                             versions = @VersionRange(min = @VersionDescriptor(major = 6))),
                          ...
                       }
                    )
                    public @interface RemoteJavaEE6Environment {}
                    

                     

                    We could then extend this to link to a specific configuration as I suggested previously:

                     

                    @ExecutionEnvironment
                    @ProvidedBy(
                       environments =
                       {
                          @EnvironmentProvider(type = JavaEE6Environment.class, configuration = "Server A")
                       }
                    )
                    public @interface RemoteJavaEE6Environment {}
                    

                     

                    Or a completely custom environment w/ a configuration:

                     

                    @ExecutionEnvironment
                    @ProvidedBy(
                       containers =
                       {
                          @ContainerProvider(family = "jbossas", profiles = ContainerProfile.REMOTE,
                             versions = @VersionRange(min = @VersionDescriptor(major = 6)),
                             configuration = "Server A")
                       }
                    )
                    public @interface JBossAS6ServerAContainer {}
                    

                     

                    The key is we aren't bound to the adapter implementation in any way. We are just specifying that an adapter which fits this criteria (perhaps even running under a given configuration) should be required. In a sense we have achieved the "binding annotation" style of CDI, just without the perfect type safety (because we need this to be more wide open).

                    • 7. Re: Proposal: Filtering out tests not compatible with target container
                      dan.j.allen

                      After a hike in the Santa Fe National Forest, I realized that I was introducing the configuration qualifier at the wrong level. A configuration does not define an execution environment, but is rather layered on top of one. So the proper place to specify it is in the @RequiredEnvironment annotation. Here's an example of the "Server A" configuration being applied to a new required environment annotation:

                       

                      @Retention(RetentionPolicy.RUNTIME)
                      @Target(ElementType.TYPE)
                      @RequiresEnvironment(value = JavaEE6Environment.class, configurations = "Server A")
                      public @interface RequiresServerAJavaEE6 {}

                       

                      The configuration identifier refers to a specific configuration of a container in arquillian.xml (not supported at the time of this posting).

                       

                      (You can put @RequiresEnvironment directly on the test class if you don't like the extra step of defining a custom annotation).

                      • 8. Re: Proposal: Filtering out tests not compatible with target container
                        dan.j.allen

                        This feature request is now being discussed on the DeltaSpike mailinglist in the thread basic decisions: test setup. It came up in the context of a discussion about how to organize tests that require different levels of specification compliance (CDI runtime vs Java EE 6 Web container). I've posted an updated proposal that reflects discussions that Aslak and I have had about how to keep the compliance stereotypes and the binding to container implementations in the user's space.