Exploratory Testing vs. Scripted Testing: A Battle of Testing Heavyweights

Let’s talk about the benefits of exploratory testing vs scripted testing, and the ‘ad hoc’ debate. In the context of software testing it’s interesting to parse the 16th century Latin roots of this modern idiom. Today, the phrase ad hoc is often synonymous with sloppy and careless work, but in fact it simply means ‘for a specific purpose’. In software testing, both test methods have a ‘specific purpose’ as their goal. One method writes tests with purpose, the other explores the app with purpose. Perhaps the debate over which method is best is simply rhetorical? Still, the different results of ‘ad-hoc’ exploratory testing and session-based or ‘scripted testing’ in both processes and results bears investigation.

Software Defect Detection – An Overview

Software testing is defined as a process encompassing the design, generation, selection, and optimization of a set of test cases for testing a certain system or functionality.

Scripted testing (ST) follows a written path of documented test cases and test steps, known as a test script. Scripted software testing doesn’t allow for deviation from the path defined by the script. The tester’s job in a scripted environment is to color within the lines, follow each instruction exactly, and report the results.

In the late 90s and early ‘aughts, test heavyweight James Bach birthed the idea of defect detection through exploration, rather than scripted test cases. Exploratory testing (ET), he argued, would allow a tester to work within a larger sphere and increase defect detection. Exploratory testing is a manual method of exploring an app without test scripts or predetermined input definitions. In layman’s terms, it could be said that exploratory testing allows for testers to learn on the fly, adjust their tests accordingly and, in that process, uncover software issues that are often unanticipated. The result would be insight into the software that better represented the user’s experience versus that of those who created it. Still, many old-school testers derided exploratory testing as ‘ad hoc’ and continued to favor the rigors of scripted testing.

How Software Testing Has Changed

During this time, most research was focused on techniques for test case design, selection and prioritization, as well as on enhancing the efficiency of automated testing. In other words, researchers explored optimization of scripted testing. However, once the idea of exploratory testing spread, researchers started to ask this question: “Do testers performing manual functional testing with predesigned test cases find more or different defects compared to testers working without predesigned test cases?”

In 2007, one of the initial articles to address this question was published in the Institute of Electrical and Electronics Engineers journal by Juha Itkonen, et al. The research results presented are stunning. Essentially, the researchers found that: “our data showed no benefit in terms of defect detection efficiency of using predesigned test cases in comparison to an exploratory testing approach…[and] data indicates that test case-based testing produces more false defect reports.”

Wow. The idea that research done over a decade ago points to ideological and rhetorical differences as the reason for maintaining the silo between exploratory and scripted testing creates a dilemma. When it comes time to decide how to test our deliverables, which method is best? To help answer this question, I’ve bulleted the major takeaways from this study:

Exploratory Software Testing: Pros and Cons

Exploratory Testing Pros

• Mirrors user experience
• Tends to find usability and UI issues
• Relies on testers creativity, experience, and skill (also a potential con!)

Exploratory Testing Cons

• “Oracle” mistake (prior bias, tester finding what they expect)
• Time intensive
• Potential for inadequate or sloppy documentation

Scripted Software Testing: Pros and Cons

Scripted Testing Pros

• Simple or repetitive tasks that are well-suited for automation
• Tends to find functional defects
• Standardized documentation benefits repeatability and tracking
• Less time intensive

Scripted Testing Cons

• Non-standard results across individual testers
• Finds fewer defects
• Guides testers focus to limited areas, misses obvious defects
• Averages 2x as many defect reports, but many are false positives

Know When to Utilize Exploratory Testing and Scripted Testing

In the end, the modern view is that all testing is exploratory testing. In 2015, Bach wrote: “Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.”. Bach’s appeal to recognize all testing as a form of exploration, and to discontinue the rhetorical divide created by applying different labels to different forms of testing, makes perfect sense in conjunction with today’s shift-left methodologies. Often maligned as ‘as hoc testing’, research into exploratory vs scripted defect detection outcomes proves exploratory testing is a vital part of developing products to meet the UX gold-standards today’s consumer expects.

Both exploratory testing and scripted testing have a time and place, and both test methods are needed to create a quality product. Different types of tests produce different results…we know that. Here, however, it’s important to understand your purpose to determine which method will meet your current need.

Understanding the statistically proven results of each method, in conjunction with strategic deployment of both resources and competencies allows us to fine-tune test strategy. Automate when possible, script when necessary, and allow your testers to explore.

Contact Us