The State of AI QA Newsletter: November 2026
Who’s Liable When Nobody Reads the AI’s Code?
You can also sign up to receive this newsletter in your inbox each month.
Jim Zuber, co-founder and CTO
What’s Inside
Top Stories
This month has been rich with stories that change your perspective on AI. Here are a few that I particularly enjoyed.
- The Toughest Bugs: How much does an AI coding agent need to know before it can track down a stubborn bug? Aya Mahmoud recently explored that question through Meta’s SWE-sweep benchmark. Researchers reintroduced bugs into open-source projects that human reviewers had originally missed for more than a year. Then they challenged AI models to find and fix them. With no guidance about what was wrong or where to look, the models resolved just 1% to 5% of the bugs. Give them issue descriptions, file names, and line numbers, however, and the results changed dramatically, reaching as high as 87% on one subset of the benchmark. Mahmoud compares working without those clues to asking a mechanic to diagnose a noise they can’t hear.
Takeaway: In my lifetime of coding experience, finding the bug was always 90% of the solution. Fixing it wasn’t the big deal. As Mahmoud points out, for tough bugs, a well-written bug report is what tips AI performance from useless to extraordinary. A hopeful sign that humans still have a place in the bug-fixing world!
- AI Liability: If we stop reading AI-generated code, who takes responsibility when something goes wrong? That question runs through a recent post by software design author and consultant Mark Seemann. He was responding to Agile Manifesto co-author Robert C. Martin, who says he now relies on extensive testing, metrics, and QA rather than reading the code his agents produce. Seemann points out that human review has always missed bugs, so expecting AI review and testing to catch everything hardly seems fair. His analogy is self-driving cars: improvement over human performance is the relevant standard, even if perfection remains out of reach. Still, better testing won’t eliminate failures in released software. A misplaced label or an incorrectly sorted list is one thing. A car accelerating when the driver hits the brakes is quite another. Seemann expects the legal consequences to land with a person or company.
Takeaway: It will take quite some time for society’s notion of human responsibility and liability to catch up with the role that AI is playing in product development. In the meantime, we will face the dilemma of releasing AI-generated software we barely understand while carrying personal liability for the mistakes AI makes that we don’t catch.
- Capability Spikes: An AI agent can pull off something astonishing and then stumble over a task that seems easy. Nate Jones, an AI analyst with a large following, recently described an agent that recreated a complex application by watching it work but couldn’t successfully customize a simple software library. That combination challenges an assumption we often make about intelligence: exceptional ability in one area probably signals strength in others. With AI, the connection may be much less dependable. Jones suggests that future models could become both smarter and more uneven, with their best capabilities advancing much faster than their weakest. The weaker capabilities might even improve considerably while the gap keeps growing.
Takeaway: It’s a mistake to apply our human notion of generalized intelligence to AI. One foolish mistake shouldn’t cause us to dismiss everything an AI can do, but more importantly, one remarkable success shouldn’t persuade us to trust it with everything else. Only time will tell whether the capability gap is widening or narrowing as AI evolves.
Two Clocks
Why faster coding doesn’t always mean faster releases
One of the more confusing consequences of AI coding is that a team can become substantially more productive without getting software out the door much faster. The developers are delivering more code. The tests may be running faster too. Yet the work waiting for release keeps piling up. It is tempting to conclude that QA needs more automation. Sometimes it does. But that may not be the problem holding things up.
In regulated industries, development and approval respond to different pressures. A team can adopt a better coding model as soon as it proves useful. The people responsible for approving the software still need to establish that it behaves as intended and that the conclusion is supported by evidence they can defend. Their obligations do not change with each model release. If anything, each new model widens the gap: more code arrives, less of it comes with a history anyone can vouch for, and someone still has to sign off on it. These are the two clocks: the pace at which technology makes something possible and the pace at which institutions are prepared to accept it.
The distinction matters because it changes where you look for improvements. If testing is slow because someone is manually repeating a well-defined procedure, automation can help. If testing is slow because nobody can agree on what the correct result should be, automation can multiply the uncertainty. And if the software has been thoroughly tested but the supporting evidence is scattered across tickets, conversations, and test logs, the delay is in demonstrating what the team already knows.
Those are different problems, and spending more on the same tools will not solve all of them.
A useful place to start is with a release that took longer than expected. Look at where the time actually went. How much was spent executing checks? How much went into resolving questions about intended behavior? How much went into assembling an explanation that someone else could review and approve? That exercise can tell you whether your next investment belongs in automation, clearer requirements, or a better way to capture evidence during development.
It also makes the budget conversation more honest. AI can reduce the effort required for individual tasks while increasing the amount of software that needs verification. Savings in one part of the process do not tell you what the whole process will cost. Nor does a growing review backlog necessarily mean the reviewers have become less efficient.
My view is that teams need to plan for both clocks. Use AI where it improves the work and make the requirements for acceptance part of that work from the beginning. The more quickly code arrives, the less practical it becomes to sort out its purpose and reconstruct its verification history just before release.
The full Two Clocks article on the QualityLogic blog explores why this mismatch persists, what earlier technology transitions can teach us, and how our experience at QualityLogic has shaped an approach to verification that addresses both speed and institutional acceptance. It also includes three questions engineering and QA leaders can use to examine their own teams.
Testing AI-Generated Code: Ledger vs Budget
Keeping track of what you actually agreed to build
An AI agent finishes a feature, the tests pass, and the team moves on. Somewhere in that process, the agent may have made a dozen decisions nobody asked it to make. How long should a session remain active? What happens to an incomplete submission? Which information belongs in an error log? The software needs answers even when the specification doesn’t provide them. So, the agent supplies its own.
Those answers may be reasonable. That does not mean they are the answers your product needs.
This is one of the difficulties I keep coming back to with AI-generated code. We tell teams to write better specifications, and that is good advice. But I have rarely worked on a project where we understood every requirement before development started. Building the software helped us discover what we had missed. With AI doing more of the building, that discovery can happen without the people responsible for the product being aware of it.
Consider a feature that lets users download a report. The specification describes the report’s contents but says nothing about how long the download remains available or who can use the link. The AI implements a working feature, and its tests confirm that the report downloads successfully. Everything appears fine. Yet the team still has a decision to make about access to that report. A passing test cannot make that decision for them.
A useful exercise is to take one recently generated feature and ask which choices it required that the specification never addressed. Have the coding agent explain its decisions, then use a separate review to look for omissions. Treat the results as questions for the team. Some choices will be acceptable, some will need to change, and some will expose requirements nobody had considered.
The important step is what happens next. If a choice affects product behavior, accepting it should create a testable requirement. Rejecting it should lead to a correction. Simply documenting what the code happens to do risks turning an unexamined assumption into an approved feature.
This is where the distinction between a budget and a ledger becomes useful. Your initial specification resembles a budget: it guides the work using what you know at the time. As development teaches you more, you need a dependable account of the decisions you have accepted and the evidence supporting them. That is the ledger. For each requirement, you should be able to establish where it came from, who or what approved it, and which independent test results demonstrate that it has been met.
The practical change is to make that reconciliation part of development. When a new behavior comes to light, resolve it while the context is still available. Waiting until release turns a manageable series of decisions into a reconstruction project.
There is a limit worth keeping in mind. Comparing the code with the requirements can expose disagreements between them. It cannot reliably reveal something both have omitted. Product knowledge, risk analysis, and exploratory testing still have a job to do.
The full requirements ledger article on the QualityLogic blog develops these concepts into a working framework, including how to separate code generation from independent verification, which decisions need human acceptance, and how to check whether your tests actually detect faulty behavior. It also examines ways to strengthen the initial specification without assuming it will be the last word.
Final Reflections
This month’s issue was built around a single idea: trust the model less and verify the system more. The Two Clocks article explains why that burden does not shrink as the models improve. In regulated industries, the accountability for verification cannot be automated, and the gap between what AI can do and what institutions will accept widens with every model release. The Ledger article is my attempt to carry that burden into practice.
I will leave you with two things from my ham radio hobby that caught my attention this month. Neither is a big story. Both are the fast clock running faster than I expected:
- I purchased some alternate firmware (GoldenRadio) for an ATS25 SDR radio receiver recently. The firmware’s features were great, but what struck me was that a very small team (a few people at most) was releasing major new features daily. These are complex radio communication capabilities. Not trivial. I don’t see how that release velocity happens without AI. For me, the rapid feature releases ARE the value proposition, not the initial feature set.
- I just upgraded a radio transceiver to new open-source firmware (OpenX6X00). What was interesting was the backstory. A solo developer had been open-sourcing some great work on this firmware but saw an opportunity to link a commercial product to it and abruptly went closed source (free, but not open source). In less than a week, another community member took the last open-source release and, with AI doing the heavy lifting, shipped an open-source version that replicated much of what the original author planned to offer commercially.
To me, both are canaries in the coal mine. If a handful of hobbyists can ship at that pace, your developers can too, and your release gate will feel it first. The fast clock is not slowing down. Running verification at that pace, without trusting the model to check its own work, is exactly what this issue was about.
Thanks for reading,
Jim Zuber Co-Founder and CTO at QualityLogic
