Maintaining and Scaling Automation: Avoid Test Rot & Flaky Suites

Test automation – what a pain in the…!

You can choose your own ending on that one, but really I’m talking about the bane of any test automation professional’s existence: maintenance. Products change over time, and the automation has to change with it. The problem comes in that small functional changes can have quite a bit of blast radius in the test suite, and I’ve penned this piece with some thoughts on ways to manage this.

Observations of a [self-proclaimed] geezer

I’ve been around the block a few times in this QA world I find myself in. Over 25 years I’ve seen test automation suites abandoned more times than I can reasonably count. Why were they abandoned? The reasons are typically simple and fall into two categories:

• The automation was started, deprioritized, and all existing tests are now completely out of sync with the product
• The automation was overly confusing and when the person doing it left, nobody could pick it up again

In both cases, over times the tests became rotten, kaput, invalid, and defunct. If it were a parrot, it would be an “ex-parrot”! (egregious Monty Python reference).

In smaller companies, test automation is often an afterthought, or a pet-project of an aspiring QA team member. The efficiency promise is there so many companies try and run with it to capitalize on the automation dream: faster test execution and more reliable results.

That is all true until things change. People can’t keep up, they get pulled on to other priorities, or the talent leaves for greener pastures, and the automation dies over time.

Is my test automation effort doomed?

Well, no, but like anything some careful consideration needs to be made about how to implement test automation, and more importantly how to maintain it long term. The planning should involve the development team as well as QA, as developers often get tasked with expanding coverage in a shift-left SDLC.

We run into very few web or mobile apps that were created with developers giving strong consideration to how to enable test automation long term. That means giving test automation something reliable to interact with. QA can build the cases, but when it comes to locating the things they need to have the automation framework interact with, they are often left using brittle locators that inevitably fail over time.

So what locator strategies should we leverage? There are many options, but classic choices include:

• DataTestid – an html attribute specifically for testing things
• Unique IDs and names – attributes that can allow easy and reliable direct access to an element
• Accessibility roles – by changing accessibility roles, it is possible to get a reasonably reliable locator
• CSS – Little better than using the old-school xpath, these tend to be brittle, yet they are all too often the only practical option

There are more ways to locate of course, but these are the major ones. Ultimately getting the development team, or granting the QA team the rights to add in some of these locators directly into the product code can make a massive difference in maintainability.

Tangent rant on architecture!

One of the other things that our development brothers and sisters can help with is stop over-architecting solutions! Test automation professionals run into architecture on a regular basis that simply cause test automation headaches.

Some architecture issues we run into frequently are:

• Nested shadow DOM: While not being a huge fan of shadow DOM in the first place, we’ve run into several cases where shadow DOM has been nested up to 3 to 5 levels deep. I still can’t comprehend why such a design choice was made, but it was! Happily, not by one of our clients, but it was by a very well known SAAS provider who shall remain nameless
• iFrames: In some rare cases I can understand iFrames. Secure checkouts being provided by a payment processor is a good example of “I get it.” We however see them frequently just injecting UI into a page from local resources, and that makes a guy scratch his head. Some test automation frameworks struggle with iFrames at all, see Cypress for example!
• Dynamic content: Okay this one isn’t so fair, as the web is a dynamic place and sometimes this just needs to happen. With that said, test automation ultimately boils down to true / false, so this is where developers need to truly help QA with those locators, and QA then needs to expectation set the team about what they can test. That might be as little as verifying that a container has content

Obviously, the code has to do what the product needs, but I’ll always advocate for the simple solution as that tends to be more accessible, and honestly more maintainable over time.

What else should I know about test automation?

The simplest concept of automation is for the test engineers to help themselves. Through simple abstraction and framework architecture, their lives can be simplified. Page Object Model (POM) is hardly a new concept, and it isn’t the only answer, but it has stuck around for a reason. Abstracting locators is the most pragmatic way to maintain tests, to where a single edit in a locator method can revitalize a large number of cases, without risking find / replace errors.

Now I grant a good IDE has truly made find / replace a much more elegant solution, but I’ll still contend that abstracted code is easier to manage and read.

What about AI in test automation?

Sure, everyone is talking about it, and I’ll tell you where I trust it right now, on the 21^st day of October, 2025. I trust AI to provide good suggestions for DAMP names when enough context is provided, give good prompt for code blocks when creating cases using something like GitHub Copilot, and even creating POM architecture.

The key is to go small with your unintelligent AI buddy. Test automation cases are best when atomic, and your AI assistant should be treated the same way. The more leeway it has to invent, the more risk you have of exposure. I think we’ve all seen it. If you ask AI to describe anything verbosely, there will be a ton of aggregated data which is correct, and some elements or nuances which aren’t. Your job is to spot what isn’t right.

What that means in the world of test automation, is that your engineers MUST verify everything created by AI with a fine-tooth comb, and this is even more critical if AI has a part in the development side. There is also growing concern about security flaws in products built with AI, so be extremely cautious on giving AI complete freedom when it comes to automated test case creation or maintenance.

These tools can bring a great deal of improved efficiency, but if the tool is being operated by the unskilled, I for one will not trust the results.

You haven’t talked that much about actually maintaining the tests!

Well, this is true, because it is very pragmatic. You need the resources to maintain them, the skillset to maintain them, and ultimately the budget. There needs to be enough benefit from the automation to where everyone in the SDLC depends on it working correctly. Failures have to be triaged, defects reported, and tests updated as the product changes. We’ve covered a lot of ground on how to get ahead of this with future planning, rather than reactive solutioning.

Can you help with our test automation efforts?

Well, we invented TestNitro as a service to help with this entire situation. It streamlines the entire process, getting you fully automatable – and maintainable – coverage in weeks. And in circumstances where TestNitro may not be the best fit, we’ve still got our team of QA professionals building automation every day, either with or without the help of AI.

If you haven’t heard about TestNitro, it is a service that we created at QualityLogic to accelerate the build of test automation. It is deterministic and isn’t driven by AI to ensure that the coverage created is exactly what is needed. We’re building tests at a rate sometimes as fast as a case every 5 minutes, and the service takes care of the maintenance as well. If that sounds interesting, let us know.

If you are just needing someone to make this problem go away, we can help you regardless of the tools we bring to solution the problem.

Anyhoo! Enough from this guy right now, as always, leave a comment or hit us up if you want to provide feedback or need help.