Open Source is Not Free

There is a misconception in the software industry that open source software is free. While it is true that many test automation tools (Selenium, Cucumber, Katalan, etc.) are freely available, these tools are not without cost to the end user. These tools merely provide mechanisms to drive browsers, organize test suites, and enable parallel execution of automated tests, among other uses. However, in order to use these open source products, one has to dedicate one, or sometimes several, developers to write code to automate the browser, build the test suite, and build an automation framework. Costs in developing test automation from open source tools include time spent in learning and creating frameworks from scratch. Additionally, open source test tools require learning extra languages that may fall outside of the main product. As the size of a test framework grows, more people are required to maintain the framework, causing distraction from the organization’s core goals as developers and testers spend time building out a bespoke testing product.

In addition to trying to get internal resources to learn to use open source tools, finding highly skilled automation developers and architects is a very expensive and time-consuming venture. Once these resources are on board, replacing them is not only difficult, but any attrition puts the automation framework creation project at significant risk, as the development of automated tests is typically performed alongside writing of production code. Time lost in this manner is difficult to make up by orders of magnitude. Further, skilled automation developers and architects have significantly more costs to the program than regular manual testers. This cost variance is in the range of additional $20-$40 on an hourly basis. Even with the extra $20/hour for a single skilled automation developer, the annual cost of this additional single resource to the development team is $20 over 2080 hours, or roughly $40,000 per year.

So, what should companies expect from their automation software? An enterprise-level test automation platform should provide the following capabilities out of the box to enable maximum productivity right away:

• Should be able to deal intelligently with XPath: Some browser plugins provide facilities to generate XPath, which some automation testers use in their test scripts. These generated XPath are often absolute (i.e. very dependent on the existing structure of the page and hence fragile as pages change) and as a result not robust. This means automated tests will break easily, and require a lot of maintenance from sprint to sprint. Another issue for automation engineers is the need to create a relative XPath.
• No-coding: Testers should spend time testing, not writing code, and not using substandard record and playback tools.
• End to end automation capabilities: An enterprise automation solution will offer solutions to enterprise testing problems out of the box, whether it’s native mobile apps, testing REST APIs, working with databases, or even desktop, so that users don’t have to learn dozens of technologies just to test their software.
• Cross-browser execution analysis and robustness of execution within each browser: As an organization’s regression tests are executed over and over as part of their continuous integration paradigm, making sense of the information from each test execution is a daunting task. Making sense of that data over time can be overwhelming. A mature test automation platform should summarize the trends of execution of a test case over time, within each browser and across browsers. Testers can use this detailed summarization of each test case to make intelligent adjustments to the test suite, enabling informed testing that truly makes use of the wealth of information available from each case in every browser on every run of the suite.
• Parallel execution out of the box: Enterprise products often have complex testing requirements and automation suites that take time to run. Parallel test execution allows for tests to be run concurrently, saving time while getting results.
• Minimal configuration or no configuration: Configuration is a hassle, especially when running various tools over multiple browsers to perform different kinds of testing. Getting each person in the development and product teams set to run tests can be a virtual nightmare.
• Continuous-Integration-ready: A scalable, enterprise test automation solution needs to be ready to run. Continuous integration and configuring the build system to run automated tests with every new build is a requirement of modern software development.
• Video and screen-shot of test failures: An enterprise automation solution should offer video and screen-shots of test failures to enable ease in debugging and troubleshooting any issues.

Creating a robust automation platform with all of the mature, enterprise level features would require thousands of requirements, architecture, development, and testing hours and would be another code-base to maintain for a business. Subject7 test automation platform has all of the characteristics mentioned above, and is a true no-code platform. It has been leveraged at various federal and commercial projects at the enterprise level, and has matured over time by incorporating feedback from various clients.

A benefit of Subject7 is that software teams can use their existing staff to create and maintain automated tests as robust as those built by highly skilled developers with open source products such as Selenium. From a business perspective, the subscription costs for Subject7 are incredibly low compared to the costs of hiring developers, building a test solution from scratch, maintaining that solution, and trying to keep up with any changes in the open-source software being used. Even the least technical members of the project team can use the Subject7 solution and provide value, with a low barrier to entry and little or no configuration.