How Our Founder, Jim Zuber, Solves for Shipping AI Code Faster
Ask the co-founder of a software testing company whether quality is worth slowing down for, and you expect an easy yes. But you won’t get that from Jim Zuber. In a competitive market, he says, the months you spend testing every path before release can cost you the sale (and he’s not convinced exhaustive testing is always worth it).
It’s a contrarian stance from a lifelong quality expert.
Jim has spent nearly 50 years in and around software quality. He co-founded QualityLogic and its predecessor, Genoa Technology, and built test tools that became de facto standards in imaging, fax, telephony, and smart grid. Few people have watched testing change for as long, or from as close. We asked him where he thinks the industry is headed, and what it means for the teams building software with AI.
You’ve watched software testing change for decades. What are the biggest shifts you’ve seen?
When I started, testing wasn’t about software at all. I ran quality assurance at Dataproducts, the largest independent printer manufacturer in the world at the time, and the work was mostly electromechanical. What struck me when software testing die arrive was how wasteful it was. Teams wrote elaborate test cases for every release in low-level code, then threw almost all of it away and started over on the next one. That seemed insane to me, so my partner Gary James and I built one of the first reusable, standardized test languages. That company eventually became QualityLogic.
For years after that, we worked shoulder to shoulder with the HPs, Intels, and Microsofts of the world, building the tests that turned their technology into industry standards.
And over time, we saw a gradual shift in two dimensions. First, we got less access to the engineers building the products. Companies began to put us in a box with other vendors. Then, open-source testing tools spread, and standards bodies started building their own test suites. Our work shifted into certification test suites, where these days we’re often the first to implement a brand-new standard as part of our test development, and we end up finding the holes in an emerging standard.
So, every few years, I’ve had a front-row seat as testing reinvents itself. But AI is the biggest reinvention yet.
Why is AI a bigger shift than the others?
Speed. Engineers are adopting AI fast, and two-week sprints have become two-day sprints. QA functions were never built to move at that pace, so a gap opens between how fast you can generate code and how fast you can test it.
Then there’s the code itself. You can describe an app in a sentence, and the AI will build it. But it does that by making hundreds of assumptions that you don’t see, and some of those assumptions will be incorrect.
So, you build up what we call verification debt: the cost of never going back to check whether the AI’s assumptions were right.
What caught me off guard is that the AI can’t be trusted to test its own work. On a recent project the AI wrote 500 test cases in two days: unit tests, integration tests, UI tests, a mountain of them. Impressive, until I looked more closely. The implementation of all those test cases was less than ideal.
For example, it didn’t want to test the interface at all. It kept reading the code to try to reason whether things worked, instead of opening the app and using it. When I asked why it wasn’t interacting with the application, it said, more or less, “I could, I suppose. I do have access to the tools for that.” Then it turned the tables and handed me a list of ten things to go test myself. I asked why it wasn’t running them. “Well, I could do that, I suppose.”
Another thing I’ve watched it do: when a test needs a service that isn’t available, the AI writes a mock stand-in, lets the mock pass, and marks the feature tested even without running the real test.
One tool wrote a beautiful test suite into its plan, then decided on its own to skip running most of it to save tokens, until I told it to run everything, every time. The bottom line is that it often takes shortcuts, and you can’t assume it will do what you need it to do. So, you have to work through these idiosyncrasies to ensure the AI-developed code is tested adequately.
Where does that leave software companies coding with AI?
Software quality is about managing risk involving limited time, limited money, and hard choices about where to spend. It always has been. AI just raised the stakes.
Right now, there are two camps. On one side is the “dark factory” idea: forget about testing and just trust the AI, or the developers, to cover quality. On the other is the old-school conviction that testing is fundamental and never really changes (AI might make it faster, but the same work still has to get done). I sit in neither camp. There’s a sweet spot somewhere in the middle, and every business has to find its own.
And here’s the part that gets me in trouble: in a competitive market, code velocity can be life or death. I’m not convinced, not one iota, that exhaustive testing beats shipping fast. I remember when companies spent a year testing before a release, and now we ship updates to our phones every day and fix problems in five minutes.
The right amount of up-front testing isn’t always the same. How much you can flex depends on your market, your speed, and how much risk your business can absorb to stay competitive. So, we don’t want to be the person in the corner saying, “slow down,” because that’s the kiss of death. What these companies need right now is someone to help them go fast, show them the risk on each path, and help them make the right choices.
Does that mean companies need fewer testers or different ones?
Different ones. The job is moving from writing test scripts by hand to supervising the AI that writes them and catching it when it cuts corners. That requires people who can read a requirement, challenge it, and think about what could go wrong, which happens to be the thing AI is worst at.
The better AI gets at writing code, the less of the value is in the coding itself. What’s left is the hard part: saying exactly what you want, knowing what “good” looks like, and being able to tell whether the AI delivered it. That’s taste and judgment, which only highly experienced people have.
None of this is new for us. Running a testing engagement has always been the same move: a test leader walks into a situation they don’t know yet, figures out what needs to happen, writes it down clearly enough to hand off, then makes sure the job was done right. AI can’t do this yet. What’s changed is who you hand the job over to. For years that was a human tester. Now it might be a human tester, or it might be AI. But the required skill of defining precisely what you want, then checking you got it, is exactly the same.
I believe the spec is becoming the real asset, and the code is just what gets generated from it. You can see it already in how you fix a bug: you update the spec and let the AI regenerate, rather than patching the code by hand.
So, what should companies do about it?
Long term, you need to rethink where quality fits in your process. Right now, it usually comes at the end—after the AI has written the code, when someone’s trying to make sense of what got built.
If you already have a pile of AI-generated code nobody is quite sure about, that’s a situation we can help with directly: we put proven testing methods to work, sped up with AI, to stabilize the code and surface the assumptions buried inside it.
But the better move is to build quality into the front end. Make sure the AI isn’t making a pile of hidden assumptions nobody checks, that it documents what it’s doing, and that it does a real job of testing its own work. That’s the idea behind spec-driven development: you write a solid specification before you let the AI generate a line of code and you use it to guide future testing.
The AI also needs something like a quality assurance overlay (help validating its own work), so what it produces is actually good and not just good-looking. Getting that right is harder than cleaning up after the fact, and most teams aren’t set up to do it alone.
Maintaining Quality at AI Speed
That’s what we do for teams building with AI. Our testers work from what the software is supposed to do, not the code the AI wrote, so the AI isn’t the one deciding whether the AI passed. The calls that AI can’t make on its own, like whether the product works for a real person, stay with a human.
Jim has watched testing reinvent itself every few years for nearly half a century. This is the next reinvention, and the teams that come out ahead will be the ones that keep quality high while they ship fast. Blind trust in AI won’t get them there. Neither will slowing everything down.
If your team is shipping faster than it can test, fill out the form below to connect with us to talk about how quality should fit in your AI workflow.
This is a topic we’ll keep returning to as the tools, practices, and pitfalls continue to evolve. If you want our take as things unfold, subscribe to the Quality Trail newsletter.
Let’s build the best program for your needs
Tell us about your program challenges and gaps and we’ll get right back to you.