Exploring the Importance of Exploratory Testing

Posted in Functional Testing

The software development lifecycle, including software testing, is heavily dependent on formalized processes, methods, planning, documentation, and traceability. Software would not be as stable or successful without these aspects, as defects would almost certainly reoccur in certain areas, new defects would go unnoticed, and teams could never truly assess the risk of releasing the software. It is also paramount that teams look at failures of previous projects and take actions to correct them for future projects.

However, the efficiency of these formalized processes should not cause us to overlook the importance of unscripted testing. Time is of the essence when projects are running behind, so sometimes it may be necessary to postpone test case creation and let testing commence based on the testers' knowledge of the requirements and software. There will also be occasions where specifications for functionality are incomplete. Since full test coverage is an impossible task for complex software, it is essential to perform some testing "off-script" in order to uncover defects. This kind of testing is known as exploratory testing.

For many people in the industry, exploratory testing is often conflated with ad hoc testing, but there is a distinction between the two. Ad hoc testing is informal and improvisational testing, executed without planning or documentation, with an emphasis on finding defects as quickly as possible. Even testers with little prior knowledge of the system can perform ad hoc testing. Exploratory testing is also informal, but is a more involved test approach for exploring how the software is behaving, with goals of learning about the software, test designing simultaneously while testing, uncovering bugs, and documenting the findings. Software testers are allowed to use their experience and creativity to test in regions of the software likely to contain bugs. Ideally, the tester would have expertise in testing the software, allowing for focus to be placed on riskier areas as well as quicker verification.

Exploratory testing does not involve documenting test strategy in a test plan or initially writing test cases to follow in testing the software. The best practice would be to design test cases as exploratory testing is performed. A possible way of doing this is through pair testing, where one tester executes while the other tester documents the steps taken. Documentation is extremely important, especially since defects found in previous projects influence the development and test coverage of software in future projects. While it would be difficult to properly document the entirety of exploratory testing, the tester can summarize their paths as testing is taking place and also write defect reports containing the steps to recreate issues. Once defects are uncovered in the course of exploratory testing, new test cases can be written and included in future test coverage to ensure the area under test is more likely to be defect-free.

How effective is exploratory testing? In an article written by James Bach, he explains that "in some situations, [exploratory testing] can be orders of magnitude more productive than scripted testing. I haven't found a tester yet who didn't, at least unconsciously, perform exploratory testing at one time or another." Even the most detailed requirements and specifications documents cannot indicate every intended behavior of the software. Thus, even in scripted test execution, software testers tend to notice functionality not covered in previous documentation and nuances about the system. This experience can inflame the fires of imagination and analytical thinking needed for effective exploratory testing and detecting defects that would have otherwise been undiscovered in scripted testing.

Given the importance of exploratory testing, how much emphasis should it receive relative to scripted testing? Earlier I mentioned that exploratory testing is especially helpful when testing time is limited. When deadlines are approaching, exploratory testing can be performed at a fast rate and touch on key aspects of the software. There is no magical percentage that should be allotted for exploratory testing and scripted testing, as each project has its own nuances and constraints. If ample time is available for testing, though, testers may want to apply the 80/20 rule.

In this case, 80 percent of testing could be scripted: the approach determined in test planning, the paths of testing formulated in test cases, the test case executions planned in a certain order, defects logged according to correct procedures, and documentation taking place at the necessary points. 20% of the testing effort can then be devoted to exploratory testing, using experience and creativity to test in new ways, unencumbered by the restrictive nature of scripted tests. If shown to be useful, these test ideas can be implemented in future test coverage in more formal test procedures. As time becomes more limited for the scope of testing, these 80/20 percentages can shift to accommodate remaining testing time. Exploratory and scripted testing are both hugely important to the software development lifecycle, and should be utilized in helping to build software of the highest quality.