How to Build End-to-End Test Automation Into a CI/CD Pipeline

Teams are shipping faster than manual regression can keep up, and AI writing more code widens the gap. In Sonar’s 2026 State of Code survey, developers said AI accounts for 42% of all committed code, yet only 48% always check AI-assisted code before committing it. Building test automation into the pipeline is how they keep releasing without breaking what already works.

Build test automation into a CI/CD pipeline in stages: unit tests on every pull request, unit tests plus a limited front-end smoke pass on merge, and a full regression run before release. If you release multiple times a day, add a nightly build to catch regressions. Before any of that, get your test data under control, because everything in test automation is predicated on results being repeatable and predictable.

If AI is writing the code, the tests need to be built separately and audited separately, or you lose the traceability that tells you the feature was tested at all.

Key takeaways:

• Run unit tests on every pull request (five minutes max), add a smoke pass on merge (10 minutes max), and run full end-to-end regression overnight or before release.
• Stage test data through an API or database query and isolate it per run. Unpredictable data is the most common cause of flaky tests.
• Retry a failing test up to three runs before it blocks a release, then triage the flake with one of four actions: quarantine, retire, refactor, or fix the cause.
• Use Playwright for web, and frameworks that act directly on the platform for mobile (XCTest, Espresso, Detox).
• When AI writes the code, tie it to detailed acceptance criteria and build the tests separately so the coverage stays auditable.

What Has to Be in Place Before You Automate?

Two things: controlled test data and a stable environment.

The graphic shows controlled test data as the base layer, staged by API or database query and isolated per run. Above it sit the three legs of environment stability: adequate resources, controlled releases to QA, and the product’s own stability.

Controlled Test Data

“Some of this is product specific, but the biggest bane of existence for test automation, and for manual QA as well, is data management,” says Paul Morris, General Manager of Government Services and Engineering at QualityLogic. “Everything about test automation is predicated on results being repeatable and predictable. To get there, you have to control the data.”

In a manual process, that might mean creating an account in the product, setting various states, and then validating against that user so you get the outputs you expect. That often happens in test automation as well. But the more you rely on the user interface to set data states, the more brittle the automation becomes, and the more laborious it is to set the users or the data up.

Data should be established programmatically, quickly and efficiently, ideally via an API or a database query. Then when you run your test through the user interface, you have predictable data. You don’t have to worry about whether the data is correct, and you don’t have to worry about whether other people are using your data set, because you keep it isolated and configured by the automation at runtime.

“Data is the number one need for test automation to have real repeatability and less flakiness,” Morris says.

Teams running E2E at scale land in the same place. Will Brock, who runs more than 200,000 E2E tests a day, describes each test setting up its own data through an API with a new user per test, plus a fresh site and database for each branch.

A Stable Environment

Stability is the second piece, and there is more than one way to think of it. There are three legs to the stability stool:

• Resources. The environment is under-resourced, so a database or an API times out instead of returning the expected result. That can often be solved by increasing resources. It usually becomes evident when you run test automation, because test automation simulates load.
• A moving target. The development team is pushing code to the QA environment five, 10, or 15 times a day. The QA person is trying to hit a moving target. They don’t know if the behavior they saw five minutes ago is still there. You report a defect, you can’t reproduce it, and it’s because a dev pushed code you didn’t know about. A stable environment is a controlled environment: who releases, when they release, and how they release.
• The product itself. The product has bugs of its own, where you see unstable behavior, flaky behavior, periodic bugs, due to architectural issues inside the product.

If you have the first two cracked, an adequately resourced environment that is controlled so QA doesn’t collide with other users pushing code or changing data, you have a foundation where you can start eliminating the issues in the product itself. If you don’t solve the other two, it’s hard to triage whether the stability issue you’re seeing is in the product, the environment, or the behavior of the users of the environment.

Pre-Automation Checklist

Before you automate, check that:

• Test data is created programmatically, via API or database query, instead of through the UI
• Each test run gets its own isolated data set, configured at runtime
• The environment has enough resources that databases and APIs don’t time out under test load
• Releases to the QA environment are controlled: who, when, and how
• The people building the automation know the tools for the job

How Do You Build E2E Automation Into the Pipeline, Step by Step?

Once you control data and the environment, the build-out follows a consistent order. Each step links to the section that covers it in detail.

1. Stage test data programmatically and isolate it per run (see the prerequisites above).
2. Control releases into the QA environment so tests aren’t chasing a moving target.
3. Pick a framework that acts directly on your platform: Playwright for web, native frameworks for mobile.
4. Write atomic E2E tests mapped to acceptance criteria, one positive and one negative case per criterion.
5. Attach each test type to a pipeline trigger with a time budget: pull request, merge, nightly, and pre-release.
6. Make the run a real gate. The test job should exit non-zero on failure, and your repository should require that check to pass before merge. GitHub calls this a required status check on a protected branch.
7. Run suites in parallel so run time stays inside each stage’s budget.
8. Add retries and a written flaky-test policy so flakes get triaged without holding releases.
9. Keep the artifacts (traces, screenshots, logs, and reports) from every run for triage and audit.
10. Name an owner for the suite, so the automation survives when people move on.

This is what steps five through nine can look like in a GitHub Actions workflow with Playwright. The same structure works in Jenkins, GitLab CI, or whatever CI server your team already runs.

# Illustrative example: one workflow, three triggers on:   pull_request:            # unit tests on every PR   push:     branches: [main]       # unit tests + smoke pass on merge   schedule:     – cron: “0 6 * * *”    # nightly full regression (UTC) jobs:   unit:     runs-on: ubuntu-latest     timeout-minutes: 5     steps:       – uses: actions/checkout@v4       – run: npm ci && npm test   smoke:     if: github.event_name == ‘push’     needs: unit     runs-on: ubuntu-latest     timeout-minutes: 10     steps:       – uses: actions/checkout@v4       – run: npm ci && npx playwright install –with-deps       – run: npx playwright test –grep @smoke –retries=2   regression:     if: github.event_name == ‘schedule’     runs-on: ubuntu-latest     strategy:       matrix:         shard: [1, 2, 3, 4]     steps:       – uses: actions/checkout@v4       – run: npm ci && npx playwright install –with-deps       – run: npx playwright test –shard=${{ matrix.shard }}/4 –retries=2       – uses: actions/upload-artifact@v4         if: failure()         with:           name: report-${{ matrix.shard }}           path: playwright-report/

A few details in that file do the work. timeout-minutes enforces the five- and 10-minute budgets. –retries=2 gives a failing test three runs in total. –shard splits the regression suite across four parallel machines, and the artifact upload keeps the report when something fails. Playwright’s CI guide covers the setup for other CI servers.

Which Tests Run at Each Stage of the Pipeline?

We recommend running unit tests on every pull request, adding a smoke pass on merge, and saving the full regression for release.

Stage	Tests	Target Run Time
Every Pull Request	Unit tests	No more than 5 minutes
Merge	Unit tests plus a limited front-end smoke pass	No more than 10 minutes
Before Release	Full regression	Overnight if it needs it
Nightly, If You Release Several Times a Day	Full regression, to catch what slipped through	Overnight

The graphic lays out the four stages left to right, with the test type and target run time for each.

That structure must be adjusted to the delivery modes the SDLC demands. With AI, we’re seeing more Kanban-style delivery, where releases happen multiple times a day. If the full test run can’t happen in a timely manner for each release, the nightly build catches the regressions. In practice, you tune it to your own situation, because every context is unique.

How Big Should the Smoke Pass Be?

Asked how big a smoke pass should be, “How big is the ocean?” Morris says. “A smoke pass could be 10 tests or 200 tests, and run time depends on resources. I like to see unit tests take no more than five minutes and smoke tests no more than ten. Regression can do its own thing overnight, so I rarely worry about duration there.”

Pick smoke tests by risk. Cover the workflows that would stop customers or revenue if they broke (sign-up, login, checkout, or your product’s core transaction) before anything else.

What Happens When a Regression Test Fails?

When a regression run fails, QA typically triages it. We check why and isolate it to either a flaky test or incorrect code in the test or the app. We typically introduce retries into the harness so a failing test runs three times. It only blocks the release if it still fails after the retries. A flake may well pass on retry, and then we can triage the flake and make the test more stable without holding the release. If you have good data, flakes are normally about steps being too long, or instability in the app or the environment.

The flowchart shows the two outcomes after retries: a consistent failure blocks the release, and a flake gets triaged while the release continues.

How Should You Handle a Flaky Test?

Flaky tests cost real time at scale. Atlassian’s engineering team reported that flaky tests caused as much as 21% of master build failures in its Jira Frontend repository, and that reruns waste more than 150,000 hours of developer time a year.

For a flaky test: triage it, identify the root cause, then act. The action is one of four:

• Quarantine: Take it out of the run during triage.
• Retire: The functionality has changed to the point where the test should be replaced instead of refactored.
• Refactor: Fix the test to make it more stable.
• Fix the code or the environment: If the root cause is the product under test or the environment, fix that and leave the test alone.

Two habits keep the quarantine list short. Give every quarantined test an owner and a date to come back out. And hold new tests to a higher bar: Will Brock’s rule is that a new test has to pass without retries before it merges, so retries never hide a test that was flaky from day one.

How Do You Keep the Pipeline Fast as the Suite Grows?

Keeping the pipeline fast as the suite grows comes down to resources and parallel runs. Run time is dependent on resources, so when the suite slows down, that is the first place to look. Configure parallel testing and run suites concurrently to cut test-run time. With enough processing power behind it, an entire suite can run fast enough to go on every PR instead of only on merge.

Why Do Unit Tests Run on Every Pull Request?

“Unit tests are the most required test and often the least utilized,” Morris says.

A unit test exercises a function: some small, isolated piece of code where the function takes something in, does something, and puts something out. If you have a unit test that governs what can and cannot go into that function, you have a solid foundation to build upon. If somebody else uses that function and tries to pass the wrong thing in, it’s going to blow up, because you’ve already thoroughly exercised it from the unit perspective.

They are quick to write, and AI is good at writing them, which is low cost with big reward. They run fast. You get reports back within seconds, because they’re working at the code level and aren’t focused on user experience. They just make sure the foundational code does what it says on the label.

Typically, those are added by developers. QA can add them, but it should always have been the case that the developer adds unit tests for their scope of work. Even where it’s enforced, developers will often add one happy path unit test and won’t do all the negative cases. With AI, there is no reason now for developers not to be adding unit tests as they produce code. It is so fast.

How Should End-to-End Tests Be Structured?

Atomic and chained together.

End-to-end is where you replicate the user experience. That’s a slower test run, and not right for every PR. But that slower run gets you to a state where you’re confident the workflow can be completed by the end user, because you’re doing it from the front end: opening the app, logging in, doing the workflow, and producing the results.

Make those tests atomic where you can, then chain them together, so that breakpoints are easy to identify when something goes wrong. You don’t want a test that has 900 steps. The chance of that flaking out is high. If each test does something small and atomic and reports back, that’s the ideal state. Sometimes you have to compromise, but that’s the ideal state to follow.

Peer-reviewed work backs this up. In an ICEIS 2025 industrial case study of 155 Cypress E2E tests in a CI pipeline, the biggest pain points were wait-time synchronization, dependencies between tests, and data cleanup. Atomic tests with their own data avoid the last two.

Because of the way we run a QA lifecycle, those test automation cases can be in place by the time the developers are finishing their work, so they can be combined and run as soon as the build is ready to deploy into QA for a manual pass and user acceptance testing.

How Does AI-Generated Code Change Test Automation?

It comes down to discipline, for the code and for the tests.

The data shows the gap. In Sonar’s survey, 96% of developers said they don’t fully trust that AI-generated code is functionally correct, and 38% said reviewing it takes more effort than reviewing code written by people. QualityLogic calls the backlog this creates verification debt.

If the code is adequately reviewed at the PR stage by the developers, and the developers understand what the code is and what it does, things stay consistent with how it’s always been. If there’s less discipline in the SDLC and more rubber-stamping of what the AI produced, you can end up with a massive amount of drift from what the original feature specification said to what the AI produced.

If you want this to work well in the AI era, you need thorough review of the spec from a QA perspective, to narrow down the acceptance criteria and validate what’s there. Then when the devs are building code, their AI is working to meet those detailed acceptance criteria, which are far tighter than the subjective prose that forms the feature story. If you have 50 acceptance criteria tied to 50 or 100 test cases, one positive and one negative, you have a much more solid foundation of truth, which is far less ambiguous. It gives the AI less decision-making capability from a “how should it be” perspective, and drives it down the straight line of “this is what I need to build.”

We’ve seen the same thing with professional development. Some developers are more literal than others, and if they don’t have good acceptance criteria, they get more freedom to create. They can invent a solution. That discipline is even more critical with AI because of the nature of generative unpredictability. You’re going to end up with more unpredictable results than with a good, skilled, intuitive developer.

The same applies to the tests. Having the AI build the test automation is a strategy, but it increases the risk around traceability. It’s the potential black box in AI, where AI builds its own tests unchecked. There’s a trend to okay that work, and then you lose audit traceability. “Test automation needs to be built separately and audited separately, so you can validate what the testing is and how it’s being done,” Morris says. There has to be a separate cycle to validate that coverage is complete and effective.

That separate cycle takes real time. In mabl’s 2026 report, teams said they spend about 20% of their working week manually verifying AI-generated tests and code.

What Does This Look Like on a Real Project?

For a leading community and creator platform, mobile automated testing lacked an integrated delivery pipeline. We built the test cases, built the Jenkins hooks, advised on the infrastructure, configured the bare metal Mac machines for the pipeline, configured parallel testing, and ran suites concurrently to reduce the time of test runs.

The plan was for tests to run on each merge. But with the processing power in the bare metal Macs, the entire suites didn’t take long to run, so they ran on each PR instead. That took a long time to get going, but it was pre-AI, so no great surprise there.

On another engagement, a client starting with zero internal QA went from a manual foundation to about 3,000 end-to-end tests integrated into a Jenkins CI/CD pipeline, with parallel runs and defects logged automatically on failure.

Which Test Automation Framework Should You Use?

Playwright for web. For mobile, whichever framework acts directly on the platform.

Platform	Recommended Tool	Why
Web	Playwright	Reliable, fast, modern languages, usually the language you already build in, easy to read
React Native	Detox	The go-to for that platform; not used outside it
Flutter	Flutter’s own framework	Built for the platform
iOS Native (Swift)	XCTest	Built into Xcode, acts directly on the app with no middleman
Android Native (Kotlin)	Espresso	Acts directly on the app with no middleman
Fallback	Appium	The app version of Selenium; a middleman layer, so only when you have to

Playwright is increasingly preferred for web. It’s reliable, fast, efficient, and written in modern languages. It’s a solid, mature approach to test automation, and that’s why it’s become the gold standard. People understand it. It’s often in the language they’re building the apps in. You can write it in almost anything, but there’s comfort in it being a language you’re familiar with. It’s easy to read, easy to understand, and a fast entry point into good, clean test automation.

Moving an existing suite to Playwright is faster than it used to be. In one QualityLogic migration from Protractor, the team converted 18,000+ lines of test code in 15 hours, against an estimated 460 hours by hand.

For mobile, Detox is a highly specific framework used in React Native. Flutter has its own. For native apps, Swift has XCTest and Kotlin has Espresso. Appium is still around, the app version of Selenium, and sometimes that’s where you have to go. But for a true native app, go as native as you can. The tools built into Xcode and Espresso act at a level where there’s no middleman. Appium runs the test through its own codebase, then connects to the mobile app and does a translation. There’s a level of potential risk and instability there, because it’s not acting directly.

QualityLogic adapts to whichever framework a client wants, uses it, and stays current with it. But from an advisory point of view, we’ll always try to use the right tool for the problem we’re solving. For native and cross-platform apps, see QualityLogic’s mobile app testing.

What Are the Most Common Test Automation Mistakes?

Three come up most:

1. The wrong tool for the job: Either someone doesn’t have the experience and has made a best effort, which is laudable, but without subject matter knowledge it’s easy to choose the wrong tool. Or they’ve selected a tool they’re comfortable with but aren’t staying current. Say they choose Appium because they’ve built Appium before, or they know Selenium, so Appium isn’t a big lift. That may get them some automation, but it isn’t the right tool if they could do the work at the grey box level, interacting directly with the app instead of going through a middleman.
2. The builder gets pulled away: A person is given the go-ahead to start building, but they immediately get pulled onto something else. They have some test automation knowledge; they start it, but they get pulled back into manual because there are releases and everything’s on fire. They can’t dedicate to it, and it dies on the vine. That’s the typical 90-day problem.
3. The one person who understood it leaves: This isn’t necessarily a 90-day problem, but it happens a lot. Somebody competent came in and built a strong foundation. When they left the company, nobody else knew how to deal with it, and it died.

Test maintenance is where many of these efforts stall. In mabl’s 2026 report, 41% of respondents named test maintenance their top pain point. Paul Morris’s earlier post on avoiding test rot and flaky suites covers how to keep a suite healthy once it’s built.

How QualityLogic Handles These Risks

QualityLogic assigns a dedicated team to the automation, so the work doesn’t stop when releases heat up, and no single person holds all the knowledge. Every test script and artifact QualityLogic builds is delivered to your repository, so you own the suite and your team can run it after the engagement. Tool choice starts from your platform and goals, and QualityLogic’s engineers stay current across web and mobile frameworks.

If your automation effort has stalled or never started, see how QualityLogic’s test automation services build suites into your pipeline.

How Should You Resource a Test Automation Effort?

Start with people who know the tools and a data staging process you can follow.

• People who know the tools: Make sure the people doing the work know the tools for the job.
• A data staging process: You need a clean, easy-to-follow way to get data staged for QA, both manual and automation. That sometimes means making sure APIs exist where you can stage data. It may be database refreshes or stored procedure calls. Whatever it is, get a good data strategy in place. “If I’m a CTO, I’m pounding on that all day long,” Morris says, “because it slows everything down if it isn’t true and it increases the risk of the whole effort failing.” Unpredictable data equals flaky tests equals being driven insane by false failures. Every time that happens, people have to triage it, figure out what happened and why, and there’s a lot of burn time in that triage. It’s getting better with automation tied into AI, which can sometimes troubleshoot faster, but a lot of the time it doesn’t have the context, so it guesses, and that creates even more friction.
• Resourcing and a parity target: Decide what resources you can apply, how much time you have, and what your goal is. When do you reach one-to-one parity, meaning for every feature developed there are at least positive and negative test cases for every acceptance criterion? When should you target that state, and how do you resource it? You can do a lot more with AI than you used to.
• Traceability, if you’re audited: If you’re in an audit-sensitive environment, SOC 2 or some other auditable standard, you’ve also got to be thinking about traceability. How do you know this feature was adequately tested? What artifacts prove it? By the time you get audited, you want to point the auditor at any one of your stories or features and say: here’s how it was tested, here’s the entire lifecycle from start to finish, and here’s all the evidence collected along the way.

Which Metrics Show the Automation Is Working?

Track a short list from the start, so you can tell whether the pipeline is helping or just adding noise:

• Run time per stage against its budget (five minutes for PRs, 10 for merge)
• Flake rate: tests that fail and then pass on retry, as a share of all runs
• Escaped defects: bugs found in production that a test should have caught
• Acceptance-criteria parity: the share of criteria with both a positive and a negative test
• Quarantine age: how long flaky tests sit outside the run

Where to Start If Your Pipeline Has No E2E Tests Yet

Start with the data. Get test data staged by API or database query and isolated per run, then lock down who releases to the QA environment. With that in place, add unit tests to every pull request, a short smoke pass on merge, and a full regression overnight, and hold each stage to its time budget.

If you’d like a second set of eyes on your pipeline, QualityLogic’s QA consulting and strategy team can review where you are today. Or talk to QualityLogic’s client engagement team about your stack, your release cadence, and where automation would help first.

Frequently Asked Questions

Where Should End-to-End Tests Run in a CI/CD Pipeline?

Run a small E2E smoke pass on merge and the full E2E regression before release. Teams that release several times a day add a nightly full run. Unit tests cover every pull request.

Should End-to-End Tests Run on Every Pull Request?

Full end-to-end suites usually run before release, or in a nightly build if you release several times a day, because they’re a slower run. With enough processing power behind parallel runs, a full suite can be fast enough to run on every PR.

How Long Should Each Pipeline Stage Take?

Unit tests no more than five minutes, smoke tests no more than 10. Regression can run overnight.

How Does End-to-End Testing Differ From Integration Testing?

Integration tests check that two or more components (a service and its database, or two APIs) work together. End-to-end tests run a complete user workflow through the front end, across every layer of the system. E2E tests are slower and catch problems integration tests miss, so pipelines run fewer of them.

What Happens When a Regression Test Fails?

QA triages it. A failing test is retried until it has run three times, and if it still fails, it blocks the release. A flake that passes on retry gets triaged without holding the release.

What Do You Do With a Flaky Test?

Triage it and find the root cause. Then quarantine it, retire it, refactor it, or fix the product or environment and leave the test alone.

Why Are Automated Tests Flaky?

Most often because the data wasn’t staged correctly. With good data, flakes come down to steps that are too long or instability in the app or the environment.

What Test Data Strategy Works for E2E Tests in CI?

Create test data programmatically through an API or a database query, give each run its own isolated data set, and configure it at runtime. Setting data up through the UI makes tests slower and more brittle.

Who Should Write Unit Tests?

Developers, for their own scope of work. QA can add them, but with AI there’s no reason for developers not to.

Is Playwright Better Than Selenium?

For web, Playwright is now preferred over Selenium. It’s faster, written in modern languages, and usually in the language you’re already building in.

Should You Use Appium for Mobile?

Use Appium only when you have to. Appium is a middleman layer between the test and the app, so for native apps use XCTest or Espresso, and for React Native use Detox.

Can AI Write Your Test Automation?

Yes, and it’s good at unit tests. But AI-built tests need to be built and audited separately, or you lose the traceability that proves a feature was tested.

How Long Does It Take to Build Test Automation Into a Pipeline?

It depends on access and tooling. A full mobile pipeline on bare-metal Macs took a long time pre-AI. With GitHub Actions and current tooling, Paul Morris has put test suites in place in a few days for his own experiments.