Background
Jumping Elephants was founded in 2011 to provide IT consulting services primarily to Canadian government agencies. They specialize in the application of evidence-based and Agile methodologies to develop best-in-class digital solutions. Their service offerings include:
- Transformative digital systems design, development, testing and maintenance
- Service design and usability research, testing and optimization
Consulting veteran and cofounder, Jean Paul (JP) Simbandumwe, places strong emphasis on usability, advocating for extensive testing before and during the design, development, and evolution of any online system.
In early 2020, Jumping Elephants was engaged to create an online case management solution for a Canadian government agency. “We decided to standardize on a test automation product for internal development projects, “ reported JP. The goal was to find a testing solution to use in an Agile development process covering:
- Unit testing and as part of a continuous integration development process
- Automated functional testing for specific repetitive testing scenarios
- Performance testing to optimize system performance under expected load
“Additionally, we needed a tool to test both the user interface (portal) and the admin interface (back-end) and it needed to work for 12 different applications that we were creating for the agency.”
Selection Process
A preliminary features review was used to narrow down 18 candidates to 6 finalists. Thereafter, the following key criteria were applied to select the winning solution:
- Support for end-to-end testing between Adobe AEM and MS Dynamics
- Capability for continuous integration, regression, and performance testing
- Ease of use with minimal time and effort to learn and develop proficiency
- On-prem and SaaS deployment options, support for desktop, mobile, and web applications, and strong vendor support including training, troubleshooting, maintaining, and enhancing the solution
- Flexible and predictable pricing
Outcome
We selected Subject7 and have been using the platform intensively since mid-2020. We have seen several operational benefits:
- Improved confidence in code quality. Integrating automated tests into the development pipeline has increased overall code quality by helping to ensure that no new bugs are introduced to the official “checked in” build
- Enhanced Agile development through frequent testing. Jumping elephants has been able to support building and testing complex user interfaces (multi-page dynamic forms), in an Agile process. This required frequent builds and many rounds of testing to consistently validate the behavior of thousands of functionality points in each form. “The client did not have to come up with “the big book” of complete and concise requirements and could work with us to incrementally evolve the solution. We did not incur a “testing debt” and could utilize non-developer testing resources.”
- More efficient use of our client’s time during User Acceptance Testing (UAT). We provided video recordings of successful test scenarios for all UAT builds, allowing our clients to focus on exploratory and exception case testing
- Significantly reduced the time for input validation testing. Input validation on this project was an ideal candidate for automation. There were multiple forms with hundreds of fields that shared common or closely-related validation rules. Automating the validation to quicky and frequently test the combinations saved both time and money
- Supported performance tuning process. We conducted performance testing benchmarking, incremental code, and architectural refactoring to validate a specific concurrent user level. Performance tuning provided confidence that the application would support the expected user volume, avoiding a scenario where actual users provided the first stress test.
Conclusion
“Over the course of our project with the government agency, we conducted over 15,000 test executions, which enabled us to assure the software quality of 12 applications continuously. After a year of using Subject7 intensely, we are delighted with the solution’s technical capabilities and the support provided by Subject7. We have fully adopted Subject7 as part of our testing practices and client delivery.”
