Establishing Test Automation in the SDLC: A Guide for Not-Quite-Beginners
What We Cooked Today?
(Editor’s note: We hope you get the “Cooking with Cajun” references. Oh, and don’t forget to feed your test team.)
Today, we’re cooking up a brief guide to creating test automation inside an SDLC from scratch, what it takes, and how to get to a place where we have something effective.
We’re going to need to gather some ingredients, consider the tools, and then see if we can bring them all together. (Spoiler alert: we can, and it’s good.)
Gather Those Test Automation Ingredients
If you’ve never done this kind of thing before and you’ve been asked to make test automation a reality in your SDLC, there might be some critical foundations you need to consider before you begin.
First, let’s consider a thing or two about a thing or three to determine if test automation makes sense.
Documentation
Test automation is going to require a source of truth for those working on it. If you don’t have good test documentation or the features for your product are poorly defined, getting that groundwork done first is going to be very beneficial.
Good test documentation is going to provide traceability and metrics, as well as inform the timeline that is going to be top of mind when creating the test automation. It is also important to look at tests with a critical mindset, ensuring there is no ambiguity and “hidden” steps.
In brief, watch for steps that ask for vague assertions like “ensure the page is correct”, or “open each item in the list and ensure they work.”
Budget
At first glance, test automation is by no means the “cheap” solution. If you buy a SaaS solution, hire QA engineers, or even try to complete the work with existing resources, there is a bill to pay. This plays back into the “good documentation” point, as that drives the level of effort, and the level of effort drives the cost.
Effectiveness
If you have a small development team, a QA person who is doing a fantastic job, the SDLC is running on time, and bug rates aren’t increasing, you have to ask, “Why change it if it isn’t broken?” Test automation has value, but it also has to improve the SDLC or solve a problem that exists in the SDLC.
Urgency
If we assume there is a problem to solve, how quickly does it need to be solved?
This ties back into budget, but classical test automation takes a significant amount of time to build, so if there is an urgent need, you will have to consider that. Going fast means more resources, or a mechanism to accelerate (more on this last one in a bit).
What is the normal time for building tests?
We have seen typical build rates for a test to be around 1 to 4 hours when hand coding a test in an established framework. That is a broad average mind you. Talk to a good automation engineer and they will talk about how they can code a test every 10 to 20 minutes. That might be true for an hour or two, but then they may run into a buzz saw of a test that takes many hours to solution.
Anyway, if we take one to four hours per test, that means your suite of 200 manual cases could take from 200 to 800 hours to create. As the framework has to also be put in place, I’d recommend you lean towards the 800 number!
Staffing
If you don’t have the skills in-house, you need to decide if you are hiring, going to a QA company (like QualityLogic), selecting a no-code / low-code SaaS solution to enable your manual testers (more on this when we talk about tools), or if you’d be turning on recruiting to hire in-house engineers.
This comes back to urgency, as well as budget. In the United States, a highly experienced QA Engineer can command more than $100K in salary in some regions, so keep that in mind. Also don’t forget that what you pay the person isn’t the person’s cost. The actual cost of the average employee can be 1.3 x the salary of that employee!
Okay, that was a lot, but let’s say you’ve worked down that checklist, you’re still here, and the ingredient list is complete. You’ve got solid test cases, there is a problem with the QA keeping up, developers want test results faster, and you’ve got budget.
Now we need to get the tools we’re going to need to cook this thing up.
We’re Almost Cookin’ on Gas!
What are we going to use to make this test automation change from a desire into reality? We have options. Many options in fact! Let’s walk through some of them.
No-code / Low-code
While this popular option is the path that many have taken, it is also a path that many have struggled with later. I can’t say that all struggle, but I’ve talked to many clients who have.
The problem statement is this: The solution typically is SaaS-based, has a monthly subscription model, and rarely any ability to discontinue and hold on to your automation. A lot of clients evolve from where they want to mature into a more code-driven test automation solution and find themselves in the position that they simply have to walk away from what they built.
That can be a bitter pill to swallow, so consider this carefully before you go this route. Some solutions also make test maintenance challenging, and a change in your UI could mean manually changing a large number of tests to make them work again.
Record / Playback
Using tools that record and playback test automation have been around for years. They tend to be quick to use, but the tests they create are very far from best practice. There tends to be no abstraction, and the assertion logic can be poorly implemented. These recorded tests tend to be a nightmare to maintain, and I’d only recommend folks going down this path if they enjoy frustration and a loss of sleep.
Hand-coded
This has been the choice of many companies for the last couple of decades, and I doubt this is going to go away anytime soon.
The beauty of this type of solution is the freedom to have complete control over the way automation works. Using frameworks like Selenium and Playwright, your automation team can abstract code, create helper functions, and truly architect the solution to be how they want it to be.
The pain point comes in with the skill level to use this type of solution correctly. Hand-coded automation done incorrectly can be maddening, and if the QA team doesn’t have the right skills your budget can be eaten up simply maintaining the tests.
Those are the traditional options, though I have one more that I’m saving for later. Who doesn’t like dessert?
Cooking Time…
As I touched on earlier, the cooking time can certainly vary depending on tool selection, and it will certainly be impacted by maintenance time, because tests need to be updated as the product evolves.
Let’s walk through a classic hand-coded test automation solution, and what might it cost to actually build and code this in-house. I’ll work the numbers based on a few assumptions:
- 500 tests to build
- Average skill level of engineers
- Base salary of $80K, burdened up to $104K by taxes and benefits
- Annual costs for an ongoing program
- 30% maintenance
- 10% new feature growth month-by-month
- Average time to build a test: 2 hours
- Average time to maintain tests, report bugs, and triage issues: 1 hour (relates back to 30% maintenance)
As you can see, this is not trivial, so think carefully about the cooking time if you have to expectation set your leadership on when the meal will be ready!
An Alternative to Cooking at Home!
Let’s get the obvious out of the way. We’re a QA company and we can provide some alternatives to doing your QA in-house. One we’ve offered for years is coding the solution for you. We have coded a large number of programs over the years and have created innumerable tests and test harnesses.
There is a lot of reassurance in coming to experts who can both help you collect those ingredients and then cook the meal for you. We don’t promise to be as cheap as the superficial cooking at home route, but don’t forget those additional costs of recruitment as well as managing the resources!
What If I Want to Have My Cake and Eat It?!
Well, today I can say we have an option for this as well. A year or so ago I pondered a way to create test automation differently while still keeping that hand-coded feel.
From that thought our TestNitro service was born. Through internal tooling our team can:
- Build tests at a rate of roughly one every 10 minutes, and in many cases, even faster
- We can get a typical client to 100% of automatable tests completed in a month
- We take away the headache of maintaining the cases
- And did I mention at a price that is cheaper than cooking at home?
It sounds too good to be true, and I grant that the effectiveness of the solution we’ve built exceeded my hopes when it came to efficiency. But we’ve seen the data, and it holds true.
One of the other amazing aspects of TestNitro is that this solution doesn’t lock you into a contract. If after a few months, the client wants to discontinue? No problem. The test automation code lives in the client’s code repository, and it just means they now have the onus of maintaining it. The code uses abstraction to make it maintainable, giving you the best of both worlds. Costs that aren’t so different from low-code / no-code solutions, with fewer headaches, and the freedom to move on when you’re ready.
If that sounds good to you, let us know. You can even find out for yourself with a free proof-of-concept!
