7 Replies Latest reply on Apr 30, 2013 6:05 AM by kpiwko

    GSoC Arquillian Drone tasks

    kpiwko

      This discussion thread is created to share resources needed to develop Arquillian Drone task.

       

      Let me introduce Aleksey Shilin, which is interested in improving Arquillian Drone in Google Summer of Code 2012!

      Aleksey has experience with:

      about 6 month with android

      about 1.5 years with java

      in previous GSoG I developing selenium runner

       

      Welcome Aleksey to the Arquillian Development team!

      Picking up Robotium integration into Arquillian Drone, here are some hints:

      This project can devided into two distinct phases. The first one focuses on decoupling InstrumentationTestRunner from Android SDK and integrating into Arquillian. This will allow running any Android integration tests directly in Arquillian. The second phase is about adding Robotium into Arquillian Drone as a supported browser.

       

      So, the for the first phase:

      General instruction how to work with Arquillian Project are listed here: https://community.jboss.org/wiki/ArquillianDevelopmentAndContribution

      Arquillian Core code is here https://github.com/arquillian/arquillian-core

      Arquillian SPI getting started https://github.com/aslakknutsen/arquillian.github.com/blob/spi_guide/guides/getting_started_spi.textile

      Arquillian Android code is here https://github.com/arquillian/arquillian-extension-android

       

      The latter phase can be specified precisely later on.

       

      In my point of view, it would be ideal to enrich currently existing Android extension. Create a stack activated by an annotation which will mimics stuff what is done by InstrumentationTestRunner using basic Arquillian runner. This will allow people to easily combine Arquillian and Android tests together once extension is on classpath.

       

      Should you have any questions, feel free to ask.

       

      Thanks,

       

      Karel

        • 1. Re: GSoC Arquillian Drone tasks
          toseter

          Unfortunately I don't passed into last GSoC. Year passed, and now I have new ideas about this project.

           

          The base idea was to make test runnable through Arquillian, as shown on picture bellow.

          Like Selenium Remote driver.

          Extension.png

          In 2012 Google give us UI Testing on Android ( http://developer.android.com/tools/testing/testing_ui.html ) . So now we have to "backends" for UI testing : Robotium and uiautomator.

          Perhaps I should use both?

           

          And I have several own ideas:

           

          * Package test results ( screenshots , logs , etc )

          Reason: After making screenshots with Robotium, they saved on device. So to see screenshots, we need copy them first.

           

          * Parallel execution

          Reason: We can have several devices connected

           

          * Remote test execution over network

          Reason: Real device connected to other computer. Or situation when developer's computer is too slow to run several AVD's.

          This feature should be similar to Selenium Grid.

          device_manager.png

          * Device and AVD ( with pre-defined popular configurations of real devices )  managment

           

          * Device selection by capabilities ( screen size and etc )

          Maybe this feature is don't need?

           

          All this features need to simplify integration tests on Android.

          • 2. Re: GSoC Arquillian Drone tasks
            smikloso

            Hi Aleksey,

             

            nice to hear from you back with your ideas.If you are interested, I would like to express my opinions about them.

             

            1) Package test results (screenshots, logs etc)

            This is indeed quite nice idea. There is already an extension which is doing this thing. It takes screenshots before and after @Test and it even takes whole video of the test suite. The "problem" is that it takes screenshots of the whole screen display, not only emulator / device. So e.g. your desktop backgroud is visible as well (or recorded as well). Taking these screenshots inside device would be nice but it is questionable how much it is needed since we have already this kind of solution. It is not hard thing to do at all. Just one observer and event fired.

             

            https://github.com/qa/arquillian-screen-recorder

             

            I dont know about logs, what would you like to log?

             

            Edit March 2014

             

            This is done arquillian/arquillian-recorder · GitHub (1)

             

            2) Parallel execution that you can have several devices connected

             

            I do not know what exactly "parallel" means but with Arquillian Android container you can have several devices in your test suite and you can mix them easilly, one per test. You just register them in arquillian.xml as any other container adapter. See more here: https://github.com/smiklosovic/arquillian-container-android#basic-usage

             

            If you mean trully parallel execution, that actuall tests are executed in parallel, this is quite interesting but it would be hard to do since Arquillian is not capable of doing of such thing right now. You would need to abstract the whole test life cycle and invoke it "twice" so you would have one jvm for test suite.

             

            Edit March 2014 - this still holds (2)

             

            3) Remote test execution over network

             

            Very interesting, so you would basically have some device controller (aka device manager) which just acts as a proxy for your tests, that device manager would act something like a forwarder for your tests if I took it right. Hmmm, it is indeed needed feature but I have a feeling this functionality is going to be implemented on the container abstraction for upcomming Arquillian 2.0 so Arquillian would be capable of running (starting, stopping) of container remotely so I guess appropriate API (and SPI) how this can be implemented and modified will be available hopefuly.

             

            Karel, what do you think?

             

            4) Device and AVD with pre-defined popular configurations of real devices

             

            Does not Arquillian Android container do that? Could you elaborate more on this please?

             

            5) Device selection by capabilities

             

            see #4

             

            Edit March 2014 - this can still be done (3,4,5)

             

            Regards

            • 3. Re: GSoC Arquillian Drone tasks
              toseter

              Hi Stefan,

              1) Package test results (screenshots, logs etc)

               

              This is indeed quite nice idea. There is already an extension which is doing this thing. It takes screenshots before and after @Test and it even takes whole video of the test suite. The "problem" is that it takes screenshots of the whole screen display, not only emulator / device. So e.g. your desktop backgroud is visible as well (or recorded as well). Taking these screenshots inside device would be nice but it is questionable how much it is needed since we have already this kind of solution. It is not hard thing to do at all. Just one observer and event fired.

               

              https://github.com/qa/arquillian-screen-recorder

               

              I dont know about logs, what would you like to log?

              This extension makes screenshot and videos on developer computer, in case when test runs on real device it would be useless.

              There are alredy implemented mechanism to make screenshots with Robotium. https://github.com/jayway/robotium/blob/master/robotium-solo/src/main/java/com/jayway/android/robotium/solo/Solo.java#L2283

              Screenshot will include only application that under test. But to see this screenshot you have to copy it on developer machine with adb.

               

              About logs, http://developer.android.com/tools/help/logcat.html

               

              Other: application can use sqlite db for caching , and for debug purposes it can be packaged into test result, or other file on device.

               

              So for each AVD or real device would be 1 package, that includes screenshots(logs and etc) from this device.

              2) Parallel execution that you can have several devices connected

               

               

              I do not know what exactly "parallel" means but with Arquillian Android container you can have several devices in your test suite and you can mix them easilly, one per test. You just register them in arquillian.xml as any other container adapter. See more here: https://github.com/smiklosovic/arquillian-container-android#basic-usage

               

              If you mean trully parallel execution, that actuall tests are executed in parallel, this is quite interesting but it would be hard to do since Arquillian is not capable of doing of such thing right now. You would need to abstract the whole test life cycle and invoke it "twice" so you would have one jvm for test suite.

              I think there should be way to run all tests on all devices. Because of fragmentation , it would be useful for test UI L&F.

              4) Device and AVD with pre-defined popular configurations of real devices

               

               

              Does not Arquillian Android container do that? Could you elaborate more on this please?

               

              Yes, Arquillian Android container can do that. I think it should include popular pre-defined configurations(screen size, density, etc) of real devices e.g. : Samsung Galaxy S3, Galaxy Nexus etc.

              5) Device selection by capabilities

               

              see #4

               

              Example: We have pool of connected real devices, this feature give us ability to select devices by their capabilities( screen size, bluethooth, gps, camera etc), and we don't need to know their actual serial ids. It could be usefull with "Device Manager" and pre-defined device configurations.

               

              Regards.

              • 4. Re: GSoC Arquillian Drone tasks
                smikloso

                Hi again

                 

                1) I see now Regarding implementation, It is completely up to you but personally I would go for standalone extension (arquillian-android-logger (?)) which is capable of doing things you said. That extension would be just plugin for Arquillian Android container. Whenever I think about that, it is only matter of event fires and observers which are at the right place in the right time. You can easilly inject AndroidDevice there and extend core AndroidDevice API  so you could take screenshots directly via adb shell and just collect data and send them to client in prepared package when test suite is about to finish.

                 

                2) I see again, but, again personally, I consider this to be more complex then needed. When you have pom.xml, you can define profiles, every profile would go for every device. You can specify properties in profile which are filtered into arquillian.xml and so all to do in order to be able to test three devices one after another (with the same tests) is to execute maven three times, everytime with different profile. I am curious how you want to implement this otherwise.

                 

                3)

                 

                4) Ah so you mean that you would just specify in arquillian.xml "test this in android container which would mimic Samsung Galaxy S3" and under the roof, Android container would "know" exactly which kind of container it should start?

                 

                5) Yeah I understand now but to make this idea very useful you would have to have one hundred devices (you know this is just a number to get the idea) which have completely different configuration so you would be lost in finding out which AVD you should start, doing it manually. When you have 5 devices it is not hard to check that. But as you wish

                • 5. Re: GSoC Arquillian Drone tasks
                  kpiwko

                  Heya,

                   

                  my comments:

                   

                  ad 1) This would be indeed useful if some nice reports could be written. E.g. something like produced by Thucidides.

                   

                  ad 2) I agree with Stefan. This is about running the same set of tests on multiple configurations. This can easily be managed externally (Maven, Jenkins, etc...)

                   

                  ad 4) and 5) Looks good. How will be database kept up to date?

                   

                  Karel

                  • 6. Re: GSoC Arquillian Drone tasks
                    toseter

                    1) Then I should create standalone extension for this.

                    4) and 5) I think database should be in xml format, so we can create micro-extension with this database. Other way is load hosted xml once a week.

                     

                    What about 3) idea?

                    • 7. Re: GSoC Arquillian Drone tasks
                      kpiwko

                      ad 3) this is really the most interesting one. Actually, thinking about that ability to remotely execute something on a different machine is something many Arquillian extensions would profit from. Yes, this somehow planned for Arquillian 2.0, see https://github.com/arquillian/arquillian-organization/wiki/Future:-performance-testing, but nobody is working on it right now.