7 Replies Latest reply on Apr 18, 2012 3:54 PM by bmajsak

    Feedback from PAX 2011

    dan.j.allen

      I presented an Arquillian workshop and attended a few talks at the Project Automation Experience in Dec 2011. I'd like to share some of the feedback I collected from that event.

       

      First, there is absolutely no question that developers are facing serious challenges with integration/functional/acceptance testing. One developer told me that his company just gave up on tests because it was just too much of a headache. Most of the attendees in the Arquillian workshop agreed that anything but unit tests are just cryptic, voodoo machine setups.

       

      ...so it's nice to know that these are real problems...something we've picked up from talks, but in a workshop it's cool to have a conversation with people.

       

      As I began to present Arquillian, the first two questions that came up were:

       

      • isn't the test archive going to be inconsistent with the application archive generated by the build tool?
      • does it support Spring? (a question that always comes up with a NFJS audience)

       

      To the first question, I think it's an indicator that we need to put out more clear information about the value of microdeployments and to clarify that they don't preclude using the application archive for end-to-end acceptance testing. Basically, people are looking for both guidance and assurance. Hopefully the Arquilian guides can cover this subject. If you have advice, please feel free to add it to a guide or write up a new guide.

       

      To the second question, it's partially answered by the Seam Spring module, which let's CDI drive Spring, hence making Spring immediately available in Arquillian. However, we *really*, *really*, *really* need to get going on the native Spring archive processor and test enricher (done as an extension). We could grab tons of users by making ourselves relevant to Spring developers. I consider this a high priority for the project. Perhaps a good GSoC idea. In the meantime, we should put out a short blog on using the Seam Spring module as a means to test Spring with Arquillian. Any takers?

       

      One attendee asked what was going on with the JSFUnit extension. I see that we finally have some movement on that again (see https://community.jboss.org/wiki/200FinalRoadmapPlanningCommunityMeeting2March2012)

       

      One of the guys in the class is developing with Websphere and was asking about the adapter. He said he would be willing to help test it and give feedback (see the wiki page for building the WAS container adapter).

       

      The audience remained very engaged the whole time, I think. They most appreciated the recent extensions:

       

      • DBunit extension
      • RESTEasy extension (still living in the Arquillian Showcase)
      • Drone (of course, always blows people away)
      • integration-style mockito (mock objects that are produced as CDI beans, still living in the Arquillian Showcase)
      • Spock testrunner

       

      I wasn't able to get to:

       

      • jacoco
      • Arquillian Extensions plugin for Forge (dang!)
      • hacking on a spring enricher

       

      It would be pretty cool if during the talk or workshop, we could setup Jenkins and get the tests running in the background...that will really drive home a point. We could use OpenShift to avoid having to configure Jenkins manually during the talk.

       

      The other thing I'm picking up on is that we need to start establishing some resources for testing best practices (hence the JBoss Testing SIG). I'm hearing at the conference a lot about the testing pyramid. We should be advising how people should use Arquillian in their shops.

       

      I received some additional feedback the following day.

       

      First, when we show the CDI example, we need to pause and clearly introduce that CDI is dependency injection and that it's the standard in the Java EE platform. We can then draw on a loose association w/ Spring for those purely Spring developers. Apparently we lose some people in this example and subsequent ones since we use CDI frequently.

       

      Tying into a point I made earlier, we need to explain better how Arquillian fits in with QA, esp existing QA operations. From the beginning we have appealed to developers, so we need to get inside the head of QA. (Thankfully, we have lots of insight from our own QE team, though getting into some other testing conferences may gain us even more perspective).

       

      We also need to get into the JavaScript game. There was a lot of focus on it at RWX / PAX, both client and serverside. One person asked if phantom (browser controller) and qunit could be used together. The speaker was like "that would be a good idea". Yeah, hello, Arquillian

       

      Here are some of the frameworks that came up in talks:

       

      • JsTestDriver
      • Geb
      • PhantomJS
      • Testling
      • Jasmine
      • Cucumber
      • Mocha
      • Watir

       

      Some of these tools are JavaScript, some of them Ruby, all of them worth exploring for ideas.

       

      I also saw examples of the following testing techniques & tools:

       

      • Auto-flag slow tests, either for quality or for categorization
      • Test categories by regex (or an idea I had to use Drools)
      • Growl / libnotify support for failed tests

       

      People love Arquillian, no doubt, but there are many more people still on the fence. It's going to be a fun challenge to being them over, and watch Arquillian grow and mature as a result.

        • 1. Re: Feedback from PAX 2011
          kpiwko

          Hi Dan,

           

          to answer: isn't the test archive going to be inconsistent with the application archive generated by the build tool?

           

          Yes, for functional tests it might happen. However, a part of ShrinkWrap Maven 2.0.0 resolver, https://issues.jboss.org/browse/SHRINKRES-18 would give you 1:1 match for build tool result, given that you use Maven. Still I think that we should promote microdeployments even for functional tests, because it speeds up the deployment or might show bad architectural design of application as a side-effect.

           

          We expect a Shrinkres guide to happen next month.

           

          It would be pretty cool if during the talk or workshop, we could setup Jenkins and get the tests running in the background.

           

          This won't work for Drone, because OpenShift CI has no Xserver available. You might investigate configuring Drone to use SauceLabs from OpenShift Jenkins though.

           

          As for the Spring extension and GSoC, I've discussed this with Marius Boegovici, which is our Spring expert, they way how it can be implemented is pretty straighforward. If he's willing to mentor GSoC, it would be great, if not I might be able to do that.

           

          Show them Android and Drone extensions combined next time as well

           

           

           

           

           

           


          • 2. Re: Feedback from PAX 2011
            dan.j.allen

            Karel Piwko wrote:

             

            to answer: isn't the test archive going to be inconsistent with the application archive generated by the build tool?

             

            Yes, for functional tests it might happen. However, a part of ShrinkWrap Maven 2.0.0 resolver, https://issues.jboss.org/browse/SHRINKRES-18 would give you 1:1 match for build tool result, given that you use Maven. Still I think that we should promote microdeployments even for functional tests, because it speeds up the deployment or might show bad architectural design of application as a side-effect.

             

            +1 Well said. I think that giving them the piece of mind that it is possible to achieve a 1:1 match when they need it will open their minds to using microdeployments and experiencing the productivity boost. It's one of those "I'm holding onto the wall at the ice skating rink until I'm sure the wall will still be there when I'm about to crash" sort of thing.

             

            Karel Piwko wrote:

             

            We expect a Shrinkres guide to happen next month.

             

            Fantastic! As I mentioned in the meeting, you can either write it in the context of the Arquillian website guides (using Textile), or, if you want it standalone, perhaps the information I provided about AsciiDoc will help you get started. (Forget what I said before about reStructuredText, I decided I'm not a fan).

             

            Karel Piwko wrote:

             

            It would be pretty cool if during the talk or workshop, we could setup Jenkins and get the tests running in the background.

             

            This won't work for Drone, because OpenShift CI has no Xserver available. You might investigate configuring Drone to use SauceLabs from OpenShift Jenkins though.

             

            Ah. Well, I suppose setting up Jenkins on the local machine (or another laptop in the room) isn't such a touch challenge anyway. The important thing is to show the audience that CI *is* possible. Seeing is believing.

             

            Karel Piwko wrote:

             

            As for the Spring extension and GSoC, I've discussed this with Marius Boegovici, which is our Spring expert, they way how it can be implemented is pretty straighforward. If he's willing to mentor GSoC, it would be great, if not I might be able to do that.

             

            I've added it to the GSoC proposal page. Feel free to revise.

             

            Karel Piwko wrote:

             

            Show them Android and Drone extensions combined next time as well

             

            +1000 I can't *wait* to play with that extension. I'm drooling over it. When will the weekend be here??? hahahaha

            • 3. Re: Feedback from PAX 2011
              vineet.reynolds

              Dan Allen wrote:

               

              ....

               

              Here are some of the frameworks that came up in talks:

               

              • Cucumber

               

              Is this Cucumber or Cucumber-JVM? I did consider writing an integration for Cucumber-JVM at the time I came up with the JBehave integration (Hello, I need feedback there ), and if there's interest in this area I could help someone write it or investigate writing one.

               

              I'm putting the following section, because I didn't want to post this somewhere where it would be forgotten. Writing BDD/ATDD integrations for Arquillian is generally a PITA for one or several of the following:

              • Absence of interceptors in the TestRunners of these frameworks (should we be forced to use their testrunners). While some have Test listeners, these aren't sufficient, especially when you want to intercept an event context and delegate the execution to Arquillian.
              • Related to the above, but in the inverse way. If Arquillian were to be the one driving the ATDD/BDD framework, then a framework with a decoupled design is more amenable to integration.
              • Impedance mismatches. These are usually in the concurrent execution of tests. Arquillian relies on ThreadLocals, which kind of impedes a straightforward integration with a framework that like to spawn off threads, or one that supports parallel execution of tests (using the Maven Surefire plugin).
              • Reporting, for in-container tests. ATDD/BDD frameworks that generate reports as a separate Maven goal, are more amenable to integration. With the ones that do not perform this, you have to pray that the developers/Arquillian users have some means to configure the report generation path (for e.g., to a NFS share), since the APIs used internally could be the File APIs and not the Stream APIs.

               

              If someone would like a Cucumber-JVM integration, I think the above checklist would be a good start.

              • 4. Re: Feedback from PAX 2011
                vineet.reynolds

                Karel Piwko wrote:

                 

                Hi Dan,

                 

                to answer: isn't the test archive going to be inconsistent with the application archive generated by the build tool?

                 

                Yes, for functional tests it might happen. However, a part of ShrinkWrap Maven 2.0.0 resolver, https://issues.jboss.org/browse/SHRINKRES-18 would give you 1:1 match for build tool result, given that you use Maven. Still I think that we should promote microdeployments even for functional tests, because it speeds up the deployment or might show bad architectural design of application as a side-effect.

                 

                Well, I skeptical, but I'd like to be proven wrong. Generally speaking, I think there is a vaccuum here on how to write a test suite (not one-off tests) with Arquillian. Something on the lines of xUnit Patterns, would help - especially in knowing if micro-deployments are the best way in all, most or some cases.

                • 5. Re: Feedback from PAX 2011
                  aslak

                  Vineet Reynolds wrote:

                  • Absence of interceptors in the TestRunners of these frameworks (should we be forced to use their testrunners). While some have Test listeners, these aren't sufficient, especially when you want to intercept an event context and delegate the execution to Arquillian.

                  We should try to engage with the upstream projects to resolve these types of issue. We could have the wrong idea of how to integrate with them, and they could have missed some extension points.

                  • 6. Re: Feedback from PAX 2011
                    vineet.reynolds

                    Aslak Knutsen wrote:

                     

                    Vineet Reynolds wrote:

                    • Absence of interceptors in the TestRunners of these frameworks (should we be forced to use their testrunners). While some have Test listeners, these aren't sufficient, especially when you want to intercept an event context and delegate the execution to Arquillian.

                    We should try to engage with the upstream projects to resolve these types of issue. We could have the wrong idea of how to integrate with them, and they could have missed some extension points.

                    Yes, maybe I should have worded it differently - BDD/ATDD test framework integration with Arquillian is not an "API stitching" exercise, but will also involve opening up extension points for integration or creating new ones.

                     

                    Edit: And yeah, therefore it makes perfect sense to get in touch with the upstream maintainers.

                    • 7. Re: Feedback from PAX 2011
                      bmajsak

                      Ghost Driver is an implementation of the Remote WebDriver Wire protocol, using PhantomJS as back-end

                      https://github.com/detro/ghostdriver

                      Might be sth worth to look at.