Testing APIs in eCommerce Sites Part 2: Sanity Checks
Sanity Checks – Find Problems Before They Find You
The quality of the code in an eCommerce API tends to suffer from a time crunch-driven perception that there isn’t enough time or money to do a proper job of Software Quality Assurance (SQA), so the engineers will just have to fix the code problems later—often when a defect becomes a critical failure. In the case of the API, this can produce a failure cascade out of proportion to the actual defect that causes the problem. An element of software quality assurance that can help alleviate some of the pressure on API maintenance is the sanity check.
One of the main problems in maintaining an eCommerce system’s code base is unintentional collateral breakage caused by feature upgrades and attempts to fix bugs. With daily pressure to change and fix things quickly, this problem can cause cascading failures where apparently unrelated segments of the system crash with each new modification. The cure for this is to test each aspect of the system to verify that it still retains its basic functionality. This is the basic concept behind a sanity check.
High-Level Sanity Checks for Quick Code Fixes
A sanity check tests as much as possible as quickly as possible. In an eCommerce environment, being able to check the entire system in two to four hours is critical. These checks focus on business processes, so each test verifies the basic operation of one critical site function. They run at a high level so that as much code as possible is exercised with each test. This won’t find every low-level defect, but it will guard against the loss of a critical system function going unnoticed until the tech support operation is flooded with calls about it.
Since the API is the critical juncture of data and function, any sanity check needs to test a list of the individual API calls, as well as having a set of purpose-directed tests that check API interactions. The individual call list will quickly point up a function that has been damaged by a change that directly modifies its operation. Testing API calls grouped by business functions points up more subtle failures caused by changes that, in and of themselves are innocuous enough, but in concert cause unwanted operational defects.
For maximum effectiveness, sanity checks should be run on a regular basis and especially just before a new version is rolled out on the live site. It is also important that the sanity check is run on the same QA mirror of the live site that is used to debug new fixes. This site is used solely to verify code modifications before they go live so it will be the first place that unintended consequences of those modifications appear.