Testing AI-Generated Code: Treat the Spec as a Budget, Not a Ledger

The solution to every technical challenge I’ve faced in my long career has been to find a conceptual framework for viewing the problem I’m trying to solve. Once I can visualize that framework, the solution’s details begin to fall into place. Over the past few weeks, a series of blog posts and customer discussions helped me realize I didn’t have a clear framework for understanding the practical complexities of testing AI-generated code. What follows is the journey I took to reach that framework.

Consensus Wisdom

The QA community has no shortage of diverse opinions on testing AI-generated code. However, some consensus [a, b, c] is emerging around four things:

1. Providing AI with a clear, detailed specification (spec) is the most effective way to ensure that AI-generated code meets its intended use case. We don’t want AI making assumptions.
2. Given the velocity and volume at which AI generates code, detailed human review of AI-generated source code is impractical.
3. AI can play a useful interactive role in identifying spec gaps and explaining what the implemented code actually does.
4. The strongest final safeguard is a rigorous, vetted test suite that is mapped directly to documented requirements. Some argue that these tests effectively serve as the spec.

There are lots of nuances around these statements, but they ring true to me.

How Good Is That Spec?

So, the next question I asked myself is: how do you verify that you have a good spec? Are its requirements necessary, appropriate, unambiguous, complete, singular, feasible, verifiable, correct, and conforming? It turns out there are three complementary tiers of techniques for finding out:

• Tier 1 – Check the language. Engineering tools can validate the linguistic clarity of requirements against standards like ISO/IEC/IEEE 29148, the INCOSE Guide to Writing Requirements, and the EARS templates. QVscribe, Jama, and RQA are well established in aerospace and defense, and spec-driven AI coding tools have their own passes, such as /speckit.clarify and AWS’s Kiro’s “analyze requirements” step. Using AI as an oracle can also help surface requirements you may have missed.
• Tier 2 – Sample the interpretation. Hand the same spec to different LLMs and compare what comes back. Generate code with two models and see where their behavior diverges. Have two models write tests and see where the suites disagree. Inject faults and measure how many the generated tests catch. Count the clarifying questions the AI asks. None of these is proof, but any divergence is a red flag that a requirement is open to interpretation.
• Tier 3 – Make it executable or provable. Property-based testing, which Kiro generates directly from EARS requirements, checks that a stated property holds across thousands of generated inputs. Verification languages such as Dafny go further and prove that it holds for all inputs.

After digesting the myriad ways to ensure your spec is complete, I shook my head and reflected on a prior blog post [d] where I wrote about how rare it was in my career to have a profound understanding of the requirements before starting to write code. Many of the requirements evolve during the coding journey. So, the tools above do something valuable but narrow: they sharpen the requirements you can state up front. On most projects, those will be a minority of the requirements you end up with. That is the first practical complexity our framework has to address.

How Do I Know What the Code Does If I Don’t Review It?

Let’s start with why reviewing AI-generated code is hard. First, there is a lot of code, perhaps thousands of lines over dozens of files, that took the AI just a few minutes to generate. Second, the code may use programming languages or algorithms you are unfamiliar with. Third, you may lack the domain knowledge to effectively “see” gaps in the implementation. Often, that domain knowledge comes from the coding process, which the human no longer does.

So, what are the alternatives? One approach is to treat the AI-generated code as a black box and focus all your effort on building a thorough set of tests (with AI help) that validate the requirements. But once the code is generated, how do you determine the delta between the requirements you wrote and the domain-specific decisions the AI made while writing the code? You need this information to build a set of tests strong enough to validate the black box. A few ideas:

• Force the agent that generates the code to write down its decisions, then map those decisions against your requirements to find the gaps.
• Use an LLM different from the one that wrote the code to review the code and the existing test suite, assessing whether the tests validate all current requirements and whether the code contains decisions that are neither documented in the requirements nor covered by a test.
• Break verbose AI-generated requirements documents into human-understandable atomic requirements. We’re experimenting with using AI to do this, then using the results to assess test coverage.

This is the second practical complexity our framework must address. Whether we like it or not, we’ll need to reverse-engineer the AI’s decisions as the code evolves. The techniques above are imperfect, but they are how the requirements nobody wrote down get written down, and that is the only way to build tests robust enough to validate the black box.

The Framework: A Requirements Ledger

Conventional wisdom for spec-driven development is that the spec is an input. For most real projects, it’s actually more of an output. What you need in a regulated industry is not a spec that was complete on day one, but a reconciled set of requirements with evidence, however the requirements were learned.

Before going further, four terms, used precisely from here on:

• Spec – The document you write before the AI starts coding.
• Requirement – A single, testable statement of behavior, regardless of where it came from.
• Ledger – The set of accepted requirements, along with the evidence that each one was tested.
• Ledger entry – One requirement plus everything an auditor needs to trust it.

The inspiration for the framework came from finance, where nobody expects a budget to be right, but everyone expects every transaction to be recorded, approved, and reconciled. The spec is the budget: your best guess at what is required. The ledger is the books: the set of artifacts at the end of the development process that is ready for the auditor to audit.

In this ledger model, every requirement gets there by one of three routes:

• Stated – Known and written in the spec before the AI started coding.
• Decided – A choice made during the build, typically by the AI, that determines behavior the spec didn’t consider.
• Discovered – A behavior found in the code that nobody wrote down.

The two practical complexities map directly onto these routes. The spec-quality tools sharpen the Stated requirements. The gap-finding techniques are how Decided and Discovered requirements get into the ledger at all. And in my experience, Decided and Discovered will outnumber Stated, which is exactly why the budget metaphor fits.

Regardless of the route, every ledger entry needs the same things: a unique identifier, the requirement and its origin, an independent test with execution evidence, and an acceptor. The auditor does not care when you learned the requirement. They care that it is recorded, accepted, tested, and evidenced. This single rule is what lets you stop pretending that the up-front spec was complete.

In this framework, I envision four actors, some AI and some human, each with a distinct role. The names matter less than the separation between them: the Extractor and the Verifier must not share context with the Builder, and the Acceptor is where human judgment enters.

• Builder – An AI agent that writes the code, proposes tests, and documents its decisions. Its tests are developer tests, not independent verification. Its edits to the spec are candidates, not requirements.
• Extractor – An AI agent from a different model family, with no access to the Builder’s context, that identifies gaps between the code and the ledger. A code behavior with no ledger entry is a candidate Discovered requirement. A ledger entry with no corresponding implementation is a coverage gap. A conflict between an entry and the code needs reconciliation. What the Extractor cannot find is a requirement that appears in neither the code nor the ledger. Intent still has to come from product owners, risk analysis, and exploratory testing. The ledger records it. It doesn’t source it.
• Acceptor – A named human for high-risk entries, an AI oracle for low-risk ones. Accepts or rejects candidate entries, resolves conflicts, and signs off on test results. High risk is defined up front (money movement, personal or health data, authentication, regulatory controls) and always goes to a human. Everything else can go to the oracle, with a human sampling its decisions and accountable for them.
• Verifier – A distinct AI agent or a human, with no access to the code, that derives tests from the ledger alone. Running those tests and injecting faults is mechanical work the pipeline can do. The restriction is on how tests are derived, not on who runs them.

This framework is a continuous loop during development rather than an archaeology project at the end. The process is seeded with the clearest spec feasible, using some of the tools noted earlier in this article. That spec becomes the first set of entries in the ledger. From there, the loop unfolds as follows:

• The Builder generates code and records its decisions – It is instructed to write down every choice the spec didn’t determine. Each of these is a candidate Decided entry. Agents don’t log everything, so treat this step as a convenience, not a safeguard.
• The Extractor compares the code to the ledger – It reports three things: behaviors in the code that no entry covers (candidate Discovered entries), entries the code doesn’t appear to implement, and entries that conflict with the code or with each other. This step is the backstop.
• The Acceptor rules on every candidate – Each one becomes a new ledger entry, is rejected as a defect (behavior the code should not have), or is noted as an implementation detail that doesn’t change behavior and doesn’t need a test. Observed behavior is a candidate for investigation, not evidence of what the product should do.
• The Verifier writes tests from the ledger, not from the code – New and changed entries get tests derived from the requirement text alone. Tests the Builder wrote are useful developer tests, but they are not independent evidence.
• The Verifier’s tests run, evidence is attached, and faults are injected – Execution results are recorded against the entry they tested. Then known faults are injected into the code to confirm the tests fail when they should. A test that passes against a broken implementation proves nothing.
• Repeat at every increment – Every merge, iteration, or release candidate runs the loop again. The ledger is always current, with open candidates visible, so the release gate is a final reconciliation and sign-off rather than a reconstruction.

A simple example shows the difference between recording behavior and accepting it. The spec says: process a card payment. The Builder, reasonably, adds automatic retries on a network timeout and logs the decision. The Extractor flags the retry as behavior with no ledger entry. The Acceptor doesn’t just record it. What happens when the first attempt succeeded but the acknowledgment was lost? A requirement is added: no duplicate charges on retry. The Verifier, never having seen the code, writes a test for it. The retry was a candidate. The duplicate-charge protection is the requirement.

Wrap-Up

This framework is imperfect, but it gives you a way to live with the two practical complexities I started with: you are unlikely to have a complete spec up front, and you won’t be able to review every line of code the AI generates. The ledger doesn’t require either. It requires that every requirement, however you learned it, ends up recorded, accepted, tested, and evidenced.

We’re still testing parts of this framework on real projects, including the AI-driven requirements deconstruction. If you want to see how it holds up, subscribe to our State of AI QA Newsletter.

References

---