In the RichFaces team, we are strong believers in testing approaches, which can be seen with our test suite written using jsf-test framework and with our outstanding functional tests suite written in Ajocado/Selenium.
However we would like to deepen a test coverage even more, especially by employing new testing techniques and leveraging existing tools.
The initiative should lead into better JSF UI testability, but not only for framework developers, but also for framework users.
Your feedback is a critical part of this discussion to ensure the design and usability will be of the highest level - these approaches are intended to evolve into regular testing tools and methodologies which could simply be grabbed and would fit into a bigger testing picture.
Arquillian Containers / Test Enrichment
First of all, we would like to support Arquillian project and leverage their outstanding container support and test enrichment capability, which would improve our ability to test on certain containers.
It would open new possibilities in space of a back-end coverage, especially regarding servlets.
ShrinkWrap should allow us use unified approach for test packaging.
Its extension ShrinkWrap Descriptors should enable one to test the framework in different configurations of JSF (like a partial state saving, a client state saving, etc.). Some improvements such as support for JSF specific descriptors may be necessary here.
When speaking about extensibility, we are going to leverage JSFUnit, a tool which enables to verify both, client and server, in one test method, making JSF integration testing a breeze, without need for mocking the JSF API.
Some extensions to JSFUnit were identified as crucial points for completeness and effectiveness:
- enable usage of Selenium 2
- verify state at arbitrary phases of the JSF lifecycle
- introduce spying on the JSF API calls
Selenium 2 / Graphene 2
Selenium 2 was identified as a critical tool for JSF test development, since it allows one to easily write front-end tests and provides outstanding support for multiple browsers through a unified API.
Graphene is then the project which helps to test AJAX applications by filling in the gaps of Selenium.
Summary of these efforts can be found in Arquillian Dev space.
Testable Data Model
One approach which could be potentially really useful in end-user testing is automation and it’s cost/effectiveness.
To this end, the RichFaces QE team is working on providing a high-level API for testing UI components which is built on top of Selenium (as low-level API).
I’m starting discussions in many places to share our thoughts with other open source teams in order to make sure this initiative really fits not only us, but the community as the whole!
Let’s make JSF test-driven development a breeze!
Why current situation is not enough?
jsf-unit tests does not completely reflect real environments (application servers, real browsers).
There are also expensive to develop and maintain because of usage of HtmlUnit, which have limited debuggability.
Metamer tests are expensive to be written and even more expensive to be run. We have too many functional tests with inadequate impact on quality.
Therefore we don't have enough tests to control quality of framework in real environment which could support agile technique of development, supporting confidence in making radical changes.
Proposal for new test structure:
Component testing should be covered on several levels:
- Java unit tests
- renderer tests
- functional (feature) tests
- integration tests
- regression tests
- sample tests
Java unit tests should verify any logic which is stand-alone and does not need any integration with servlet or JSF container or any other part. Should not require mocking at all.
Renderer tests are focused on verification of rendering logic as a whole. Component should be rendered to the page in its basic state and HTML output should be verified against pre-defined XML pattern (XMLUnit). This should be done for each of the rendering types for given component (e.g. Tooltip does have several rendering strategies for different layours - div, table and list).
Integration tests are written in Arquillian, any JSF interaction is then tested with Warp JSF extension. These tests should verify every interaction where components influences state of the FacesContext or triggers events or where life-cycle behavior is specific for given component. Integration tests also covers aspects where components integrates with each other.
Regression tests verifies important aspects which can’t be easily added to any other group and typically are created for reproducing and verifying found issues. They are typically written with Graphene and Warp.
Sample tests covers logic of sample applications, quickstarts, etc. They are written using Graphene, Warp and other Arquillian extensions. For testing component logic, Graphene Component Objects pattern is used.
Tests are written using Arquillian and Drone, thus it is possible to achieve high-level of portability.
This has some drawbacks:
- test execution in real containers is slow
- test execution in real browsers is slow
- test execution in real browsers is restricted to environments with display
In order to allow agile development (fast continuous integration, commit/pull request verification), but do not lose possibility to run tests in real environment, mocked containers and browsers might be used.
However it is also connected with some risks:
- portability and debugging on mocked browsers is paintful
Possibilities to improve execution time in continuous integration:
- alternative mocked browsers (GhostDriver)
- mocked servlet environment (jsf-test staging servlet feature)
Development & Maintanance:
Developers are responsible for writing basic set of tests for each component and each type of tests.
Quallity Engineers are responsible for running those tests on periodical basis (continuous integration), keeping portability (across browsers and containers). Quality Engineers are encouraged to extend basic tests, analyse them and incorporate improvements as required.
Very well said Lukas! Once all the pieces are inplace, RichFaces and the JSF community as a whole will surely benefit! thanks for all your efforts in this regard!
I'm curious to hear more details about the QUnit testing. I'll be sure to follow up on the status of tthis at our next community meeting.
My company is aiming for migrating from Richfaces 3 to Richfaces 4. However during this preliminary analysis I found that JSF 2.2 is going to be out soon. So I was think about the current support for Richfaces 4. What would happen if JSF 2.2 is not supported . Will the components changes heavily like from RF 3 to RF 4. Please reply soon.
Regardless of when JSF 2.2 comes around, it will be our top priority to support it with the subsequent RichFaces minor release.
Also, I expect the changes/improvement in JSF 2.2 to be mostly additive, and so will not be as difficult to adopt into the project (as opposed to the JSF 1.x to 2.0 migration which was a huge undertaking).
Thank you for the quick response. I have been scouring the web for both Richfaces and JSF development. Anyways, I found that JSF 2.2 has made its milestone release . So I started looking for a good strategy. Your assurance for support of the current Richfaces Project is what I sought for. As of now all my pages are in XHTML so I was thinking of what new problems and features HTML5 would. bring.
Thank you for your quick response. It has been very helpful.