Will Record and Play test automation work for you?

There are plenty of exceptions to deciding what type of testing is best. We’ll provide some additional commentary to help you decide if, for example, the application is simple but the organization makes it complex.

Additional Factors

Reality has a way of not quite fitting into an easy two-by-two decision matrix. Here are a few additional factors that might tip the scales to one side or another.

Number of overlapping users. A single user can create and run tests rather easily. This is a scenario where Record and Play can be the most useful. As we introduce multiple users trying to modify the same “test,” we run into challenges. Under the hood, Record and Play tools are still code, and the tools need the capability to maintain two or more different user spaces, then reconcile those user spaces through version control. Software engineers use tools like Git to accomplish this. As a result, Record and Play users typically “step on each other” as they are making changes, or else develop an informal tradeoff system of who will work on which system. Combine that with code that is version controlled and branched in yet another system, we can run into a real version control problem. Version control operations are usually complex. The simple “easy to use” properties of Record and Play quickly diminish as users share assets, which leads to version conflicts.

Team Size/Number of teams. As we discussed earlier, if more than one person is testing the front-end, this can lead to conflicts when sharing Record and Play systems. Most of these systems really only allow one person to modify a scenario at a time; otherwise, the two users will “step” on each other when they save. Even if people are working in different scenarios, a good record playback tool will store the shared objects in an object repository. That way, when the object changes the expected changes only need to be made once. This invariably leads to redundant objects, version control conflicts, or both. Scale this up to multiple teams without clearly defined boundaries and the problem only increases.

Setup and deploy efforts. Test automation needs software to run against, but also data to run against. For example, in eCommerce, search tests need to have known seeded test data. As of today, a book search on Amazon for “how to reduce the cost of software testing” returns four books. Hard code four “4 results for” into the expected results and the test will fail in production when a new book comes out, or in test when new test data is copied from production.

Verification points. Record and Play needs to do more than click buttons and type numbers. It needs to check expected results and lots of them. In some cases, the results might not always be predictable. An example using invoicing software shows that if the terms are due in thirty days, the tester may need to write a utility to calculate the system date plus thirty to do the compare. So-called “zero code” tools rarely are. As a result, test tools require code, but lack re-use. Testers end up cutting and pasting verification code. When that code needs to change, it might need to change in a half-dozen different places. The extent to which a Record and Play tool supports reuse can greatly impact the long-term effort involved.

Interaction Points. Many test strategies require interacting with objects outside of the browser. This can include database setup, calling APIs directly, security, performance or load testing. Even something as simple as simulating multiple user interactions, to test for race conditions, may be outside the capability of simple Record/Playback tools. Requirements like these make the problem domain more complex, pushing the solution toward advanced or adaptive tools.

Silos. If the user interface is organized in clean elements, that can actually make testing much easier. For example, an ERP system might have an HR module, a manufacturing module, and a finance module, all of which can be built and tested separately.

Parallelism. Some teams speed up test runs by running several scenarios at the same time, also called “in parallel.” The newest version of Selenium IDE does have a way to run in parallel in the cloud. In practice, coordinating these types of tests creates extra overhead, confusion, and timing issues.

Our Recommendations

Over the years, we’ve seen a series of practices work for Record and Play. These are strategies that add value almost immediately, do not age easily, and will not require extensive training or dedicated assets.

If you want to chase the Record and Play dragon, here’s your playbook.

Quick Smoke Tests. Create a series of independent scenarios that test the core functionality of the application. No scenario should run in more than five minutes, the entire suite needs to take less than an hour. The smoke test demonstrates that the programmers probably haven’t broken anything big functionality wise.

Automate Setup of Data. If you can’t do setup through the backend, and it is required, you can create a Record and Play scenario to do setup, then call it with different variables.

User Interface (UI) elements that are stable. Create scenarios around a stable UI—or at least stable enough. If the picture is changing but the workflow is still the same, then try to use static IDs and elements that are predictable. Avoid long connections to elements based on their locations in the web page (XPath or CSS).

First-pass to export into another system. If the Record and Play system has an export feature, record a first pass then export it to a more robust test system for long-term work.

Jump out once you have to add code. When you start adding logic in a tool not really designed for variables or loops, then it might be time to move to something more robust. This is especially true if that code needs to be manually copied to other scenarios.

Track and limit scope. Forget ideas like 100% coverage. Instead, create a scale for how deep to go testing features. Publish the depth and feature map so the depth of your testing is well understood, along with the depth of your exposure. 100% is not a realistic goal, 20-40% might be…

Evaluate carefully with an eye toward ROI. Plan the investment of time and energy, compare that to the results in terms of defects found and actual time investment. Consider how that time would be spent without the tool – is it saving time? Enabling faster release? Catching bugs? Worst of all, could “passing” scenarios be creating a false sense of complacency?