10 Replies Latest reply on Jan 23, 2014 9:34 AM by pmensik

    Idea for Arquillian logging extension

    pmensik

      Hey guys,

       

      I've got an idea for new extension which is supposed to improve logging in Arquillian because this so far really poor (I'd like to see test names, duration of the tests, class names and so on - all of this during runtime). I've achieved something like this through TestListenerAdapter but this is not portable to JUnit hence harder to maintain. It could also produce more statistics at the end of the suite, maybe graphs and other interesting stuff (however some of these are already covered by Surefire, TestNG and JUnit so I am not sure how much space is there for improvements). So is this even worth it?Any other ideas what this extension could cover?

        • 1. Re: Idea for Arquillian logging extension
          aslak

          The TestRunner(junit, surefire etc) will normally report this(at least class names / times / result) during and after the test suite exceution.

           

          What are you trying to achive? and Where?

           

          See test execution incontainer in the server.log?

          See test execution cient side in console?

           

          I know we've been discussing earlier how to consume server side logs, transfere them to the client and have them be part of / split up pr test class console log

          • 2. Re: Idea for Arquillian logging extension
            pmensik

            I ment something like (classic JUnit test)

             

            screen1.jpg

            Test with custom listener

             

            screen2.jpg

             

            My point was that you could get much more information during runtime (you can see which test is currently running and how long), this seems to me much more user-friendly (but maybe it's just me).

             

            And yes, server, console and selenium server logs capturing would be part of this extension, so you could get a more complex report about what really happened in your tests. But that's for another discussion how it should precisely look like.

            • 3. Re: Idea for Arquillian logging extension
              pmensik

              Hey guys,

               

              I finally got time to work on this extension so here is the results. It's going to produce a HTML report based on Twitter Bootstrap CSS library, it currently shows the exception which occured during test run, exception which appeared in the server log during test run (if any) and a small summary of whole test suite. Here is also project which you can use to try this out, just clone the arquillian-logging, build it and run tests in this project and see the report:) Note that this is just a version which I made after few days of playing around, so any tips and ideas for improvements are very welcome.

               

              My current plans for future version would be

              • better design
              • remove bugs (it's really not perfect now)
              • add log from Selenium
              • add an option to store results of previously run tests and compare to last run
              • add more info about this extension and Arquillian in general

               

              So do you think that is worth it to develop this project further?

              • 4. Re: Idea for Arquillian logging extension
                smikloso

                Hi Petr, I was looking on the code, what would be really cool is that when I put it on class path with Arquillian Droidium and user takes screenshots like this: https://github.com/arquillian/arquillian-droidium/tree/master/tests/droidium-screenshooter-01 right now they end up in some common directory and you are expecting to browse it manually. What would be totally cool is that there would be very simple gallery which would just process these images in some cool form. Like panel on the left, screenshot on the right, panel would be something like a tree where nodes would be test methods and lists would be taken screenshots. I would see it for some javascript + html in resources which would generate that tree on the fly. What do you think? (this actually reminds me our effort to make screenshot taking unified accross arquillian universe)

                • 5. Re: Idea for Arquillian logging extension
                  pmensik

                  Hey Stefan,

                   

                  the idea of putting these two extensions together (or at least make them work with each other) is really great! Making some kind of gallery from the taken screenshots is really not a problem since I am already using JQuery in this extension. I have a RHCSA training this week but we can discuss this in person next week, ok?

                  • 6. Re: Idea for Arquillian logging extension
                    smikloso

                    awesome, do not hesitate to catch me, by any means, thank you!

                    • 7. Re: Re: Idea for Arquillian logging extension
                      jhuska

                      Here are couple of notes we have collected during an offline meeting.

                       

                      Reporting extension

                      What it should do, and is currently doing:

                       

                      1. screenshots in the report - already has proof of concept
                        1. reporting extension would use smiklosovic/arquillian-unified-recorder · GitHub
                        2. Configuration of screenshots part will be done by the above extension. Reporting extension would listen for ScreenshoterConfigured, and will read its configuration to setup this feature.
                      2. retrieving log from application server for a particular test - POC
                      3. retrieving a Selenium log for a particular test - not done
                        1. Need to setup the path for it when selenium-server is started by an user
                        2. Need to clean a little bit the Graphene logging here, it pollutes it with logging the content of whole javascript files which are installed to the page.
                        3. Question is how useful is this ? Does anybody use it ?
                      4. Diff to the previous test results - not done
                        1. there would be a directory where previous reports are stored. It will show in a concise way whether the failing is a regression or not.
                      5. Browser JS logs - not done
                        1. We are wondering whether there is some way to retrieve e.g. JS errors which occured in the browser. At least some hack.
                      6. improvement for visual state of the generated report - POC with help of bootstrap
                        1. All of this report pieces should be reported per test in a readable way
                      7. Runtime logging of the test runners - POC
                        1. When tests are exedcuted, it is often handy to see in the loggs which test started and wheter it passed or not.
                        2. It is currently achieved by testng listeners. However, those listeners often mix-up with Arquillian life-cycle, and it is quite hard to maintain them.
                        3. There for an extension would help here.
                      8. Exporting the report to different formats - not done
                        1. e.g. to JSON - it can enable further processing of the results by other tools
                        2. or maybe such a extension deserves good API which can be used to plug in other tools

                       

                      Just ideas
                      • What about providing to the user a way how to filter the logs conveniently ? One would be able to setup which parts of the e.g. server log, or selenium log will be reported by entering some kind of regex ?
                      • provide the user a way to make logging based on WebDriver actions - intercept all call to the WebDriver objects: WebElements, Page Fragments and do some specified action (take screenshot, make a log somewhere)

                       

                      Your input would be much appreciated!
                      • 8. Re: Idea for Arquillian logging extension
                        huangdihu

                        Hi Petr,

                         

                        I want to log Arquillian test in html report, arquillian-logging is what I want, thank you.

                         

                        I use the maven dependency to fetch the artifact, but maven build complains cannot resolve the dependency, where did you publish arquillian-logging? Thank you~

                         

                        <groupId>org.jboss.arquillian.extension</groupId>
                        <artifactId>arquillian-logging</artifactId>

                        <version>1.0.0.Alpha1-SNAPSHOT</version>



                        • 9. Re: Idea for Arquillian logging extension
                          smikloso

                          Hi,

                           

                          I think it is not published anywhere but on Peter's github. You need to fork that repo (1) and build and install it yourself, then you can use it.

                           

                          AFAIK standard procedure with installing this one is just to mvn clean install.

                           

                          Please note that we are developing one unified screenshooter, video recorder, logger and reporter. This project is taking systematic approach how to do these things, it will consists of few standalone and very subtle extensions which would do what you need. It is in (2) and it is under development but we would like to publish first alpha pretty much soon.

                           

                          (1) pmensik/arquillian-logging · GitHub

                          (2) smiklosovic/arquillian-unified-recorder · GitHub

                          • 10. Re: Re: Idea for Arquillian logging extension
                            pmensik

                            Hi MINNA HU,


                            Stefan is right, the project is currently more like proof of concept than stable extension, so you have to build it on your own. But you are more than welcome to try it and propose feature requests!Project is currently assigned as a bachelor thesis to the student under my supervision so I hope we can expect some results soon:-)