Trust the Model Less, Verify the System More: Four AI Insights
While I review almost 1,000 AI-related articles each month, few profoundly shift my perspective. This month, however, has been rich with such articles, and I’d like to share some of those new perspectives with you.
The Toughest Bugs
Aya Mahmoud recently shared the results of a fascinating Meta benchmark (SWE-sweep) that tests how well AI handles truly elusive bugs. Meta took bugs from open-source projects that had slipped past human reviewers for over a year and reinjected them into the code to see if AI models could find them.
Left completely in the dark with zero hints, the AI models flopped, resolving only 1% to 5% of the bugs. But as researchers fed the models better clues (like file names, line numbers, and issue descriptions), success rates skyrocketed, topping out at 87%.
Mahmoud likens the initial blind test to a mechanic trying to fix a noise they can’t hear. The punchline? For the toughest bugs, a well-written bug report separates useless from extraordinary AI performance. It’s a striking, quantifiable reminder of where human expertise can still make or break AI effectiveness.
AI Liability
Mark Seemann, a well-known software design author and consultant, recently commented on a post by Robert C. Martin, co-author of the Agile Manifesto, who said he no longer reads the code his AI agents write. Instead, he surrounds them with rigorous tests, metrics, and QA procedures. Seemann makes the following observations.
Human code review was never foolproof for finding bugs. Demanding perfection from LLM-based code review and testing sets a bar that human programmers never cleared. As with self-driving cars, the system doesn’t have to be perfect, just better than humans.
Even if AI testing exceeds human testing, bugs will still slip into the field. Text in the wrong place, lists out of order, and the like. Most of these are benign. But what if a bug causes real harm, like a car that accelerates when you step on the brakes? Who is liable?
Seemann says a company or person will most likely bear the legal consequences. He acknowledges that fully automated software engineering may be coming, but the human notion of liability and responsibility may not keep up. His closing question, captures the article’s essence: “Are you ready to go to prison for an AI?”
Capability Spikes
Nate Jones, an AI analyst with a huge following, recently wrote in his Substack about an AI agent that could reproduce a complex application just by observing how it worked, yet failed to customize a simple software library. We tend to assume that a human who is extraordinarily smart in one area is probably smart in most other areas. However, it may not be safe to assume that dramatic gains in AI capabilities in one area automatically translate into broader improvements across all AI capabilities.
Jones hypothesizes that as AI gets dramatically smarter, it may also get “spikier,” with the gaps between its strongest and weakest abilities growing wider. Even its weaker abilities could improve while falling further behind its strongest ones.
His article is long, but the key takeaway for me was this: if AI makes a dumb mistake, that doesn’t mean it’s useless in other areas. Conversely, if it succeeds spectacularly in one area, we shouldn’t automatically assume similar success elsewhere. We shouldn’t apply our human notion of generalized intelligence to AI until we have more evidence about whether those gaps are widening or narrowing as AI evolves.
Engineering AI Safety
Medium author Noah Byteforge did a wonderful job drilling into this recent quote by Microsoft CEO Satya Nadella: “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.”
That statement initially baffled me, but Byteforge explains the logic behind it. Traditional software systems are governable because specific behaviors trace to specific code paths, which we can then edit to change the behavior. AI models don’t have this property. They function on statistical reasoning and model weights, meaning we have no way to verify that an AI’s output resulted from a trustworthy process.
However, Byteforge points out that backend software systems have always depended on unpredictable external resources, like third-party APIs, and engineers already have a whole toolbox of techniques to manage that uncertainty. Those exact same techniques (model diversity, observation, independent controls, containment, and verifiability) can be used to manage the trustworthiness of AI.
At its core, Byteforge argues that Nadella’s post proposes that we apply the principles we’ve used for decades to build reliable distributed systems to this new category of non-deterministic AI systems.
Trust Less, Verify More
My takeaway from these four insights is a single rule: trust the model less and verify the system more. Meta showed how much AI depends on what we tell it, Jones showed how unevenly it performs, Byteforge showed how to engineer around that, and Seemann showed why we don’t have a choice. The models will keep getting better. The burden of AI verification stays with us.
Most months, only a handful of the nearly 1,000 AI articles I read change how I think about testing. Those are the ones I write about in our State of AI QA Newsletter.
