What AI Hasn’t Changed: Ten Quality Lessons from 50 Years in the Field
I’ve spent most of my adult life in one kind of quality leadership role or another. In the 1970s, I was Director of Quality for Dataproducts, at the time the largest independent printer manufacturer in the world. In the 1980s, I was a founder and CTO of Blue Chip Software, a consumer software company that at one point had three of the ten best-selling software titles in the country. From the 1990s to today, I’ve been CTO of QualityLogic, providing software testing products and services to many of the largest high-tech companies in the world.
Over that time, I’ve learned countless hard lessons, and some have proven more enduring than others. Software development, and the role of quality in particular, is now going through profound change with the arrival of AI. In this post, I’ll share ten of those lessons that still seem to hold true, even in the middle of such a fundamental shift in how software is built and tested.
Lesson 1 – Hiring Great People
Recent articles have discussed how perplexed technical managers are about how to interview and vet software developers now that writing code is barely a job requirement. Understanding how programs are put together still matters, of course, even with AI, as do architectural skills and the ability to break a complex problem down into granular implementation steps.
But in my experience, the single most reliable indicator of a great technical hire has been the quality of the questions they ask. Do their questions reveal a deep conceptual understanding of the subject, or some unusual insight? I find it hard to describe exactly what I’m listening for, but I know it when I hear it: I get rocked back on my heels, and the thought runs through my head, “holy crap, this is a seriously smart person I’m talking to.”
And this is exactly the kind of skill requirement that AI doesn’t change. When the machine can generate the code, what you’re really hiring is the quality of someone’s thinking, and that shows up in the quality of the questions they ask.
Lesson 2 – Process Is the Hard Part
One of the hardest lessons I learned in a lifetime in quality is that building reliable, high-quality processes is really hard. I remember being asked at Dataproducts to develop a gravity-feed paper tray. How hard could it be? Paper comes out of the printer, flips over, and lands in a tray. Very hard, as it turned out. I quickly learned that humidity, temperature, paper composition, paper velocity, and countless other variables all affected whether it actually worked. It was anything but easy.
In a higher-stakes example, one of my QA responsibilities at Dataproducts was a magnet bonding process for the hammer actuators that struck a revolving band of characters to print each character on the page. These printers were used on naval ships and submarines as part of tactical command-and-control systems. Out in the field, the magnet bonds began to fail, producing missing characters in mission-critical communication documents. That is about as serious a quality problem as you can get. It took us nearly a month to isolate the process fault, with factories idled, ships returning to port for replacement printers, and military readiness potentially compromised. Developing processes that drive quality and mitigate risks is challenging.
The AI world puts the model at the center of attention. But there is a growing recognition that the process around the model, often called harness engineering, is a profound determiner of what an AI solution can actually do. That harness is the scaffolding, tooling, context, and guardrails built around the model, and getting it right is its own hard discipline. The moral is the same one those printers taught me decades ago: process is the hard part. Invest your time and effort in getting that harness right.
Lesson 3 – Don’t Be Afraid of Change
I’ve been an early technology adopter my whole life. I bought the first 6502 microprocessor development board, the KIM-1, and wrote a football simulation in machine language that rendered the field across a half-dozen seven-segment LEDs. Those were the days. And I bought the IBM PC the day it hit the market, with a whopping 16K of RAM, using money my wife’s grandmother had given us from the $100,000 she’d won in the lottery a few days before.
One of my earliest forays into new technology was the Texas Instruments programmable calculator. I snatched one up the minute it came out and taught myself to program it. At the time, I was working at Dataproducts as a mechanical parts inspector, a job that involved a lot of complex calculations. I pre-programmed the calculator to do many of them and was excited beyond words to show it off at work.
My boss was appalled. He insisted you couldn’t trust these newfangled devices, and that the only reliable way to run the numbers was with a traditional calculator. I was banned from using the programmable one for my work. Profoundly disappointed, I moved on and did the best job I could. Within a short time, I was promoted out of the department.
Six months later, there was a reorganization, and as fate would have it, my old boss, Mr. No Calculator, was now reporting to me. I was a good sport about it, though he did retire shortly thereafter.
The programmable calculator was the AI of its day: a new tool the establishment dismissed as untrustworthy, right up until it became indispensable.
The moral, as it applies to AI, is simple. Embrace it, or you may find yourself where my old boss ended up, reporting to a former employee, or worse, out of a job.
Lesson 4 – Don’t Take Shortcuts
When I was at Blue Chip Software, a game I had written called Millionaire was on a rocket ride. It had been number one on the charts for 15 weeks, and we were getting ready to ship a major new release. Distributors had pre-ordered more than 20,000 copies, and they were pressuring us hard to ship. We had full-page ads queued up in all the major computer magazines.
There was just one problem: an intermittent bug during gameplay that we couldn’t isolate. I hired a dozen college kids and had them work around the clock trying to reproduce it. We finally tracked it down, fixed it, and cut a master copy to send to the duplicator, who would in turn ship the disks to distributors inserted in our packaging. Things were looking up.
As I was putting the master disk into a FedEx package, I booted it up one last time, and to my horror I saw a glaring misspelling on the game’s splash screen. I am a poor speller, and that kind of mistake embarrasses me personally. In desperation, I pulled out a hex editor, hacked the executable to fix the misspelling, dropped the disk into the FedEx package, and shipped it off to the duplicator.
Well, as you might guess, 20,000 duplicated disks went out to the distributors, then out to customers, and every one of them crashed. My hack to fix the spelling had introduced a fatal error. It nearly killed the company. We recovered only through an epic effort to make things right for customers, and thanks to some remarkably forgiving distributors.
Shortcuts have always been dangerous from a quality standpoint, and they are even more so in today’s world of AI-generated code. The AI does all the work in seconds. It feels good, it looks good, it works on your machine. That is exactly the trap. The more finished a change looks, the greater the temptation to skip the testing that would catch the fatal flaw underneath. My hex edit looked like a harmless one-second fix too. Whether it’s a hand-typed hack or a block of AI-generated code you never verified, the change that seems too small to test is the one that nearly kills the company.
Lesson 5 – Expect the Unexpected
For most of my life as a software tester, sending a test case to a device under test was the most routine thing in the world. Send the test, look at the results, decide whether it passed. But twice in my career I sent a test to a device and got a result I could never have imagined.
When I first started Genoa Technology, the company that would later become QualityLogic, we had the chance to help Dataproducts, my former employer, test their new solid-ink printer. This novel machine used liquefied crayons as its print medium, held in heated vats that rode along with the print head as it traversed the page. There were only three prototypes in existence, each costing more than a quarter of a million dollars, and one of them was sent to our lab to test the printer’s command language. We carefully set it up, prepared our test suite, and sent the first test. To our horror, as that first test was processed, the print head began to oscillate back and forth, faster and faster, until the entire printer was lurching from one side to the other. Eventually the print head collided with the side of the housing, and all of the melted ink vomited over the interior electronics, entombing them like Pompeii. Needless to say, the printer was a total loss.
Even more dramatically, Epson once shipped us one of its prototype printers from Japan to test. We carefully unpacked it, sent our first test, and the printer burst into a ball of fire. We scrambled for fire extinguishers. The good news is we got it out; the bad news is the printer was another total loss, most likely from an electrical short of some kind. The moral is simple: expect the unexpected. And it has never been more relevant than it is today, as AI agents escape their sandboxes, reach into systems they were never meant to touch, and take actions no one anticipated. You send what looks like a routine instruction and get back something you could never have imagined. In testing, that isn’t the exception. It’s the job.
Lesson 6 – Build for Reuse
After selling Blue Chip Software to Britannica Software, I was bound by a long non-compete, so the game business was off the table. At the time, Dataproducts was releasing its first page printer with multiple printer-language emulations, and they asked me to come back and create the company’s first software quality testing lab, built around that upcoming release. It was an exciting time: hiring staff, working with Toshiba, who we had licensed the print engine from, and wringing the bugs out of the printer emulations.
When the printer finally shipped, I reflected on the testing process and realized something that bothered me. All the test code our engineering group had written, in some low-level language, would most likely be useless for the next release. We would have to build it all again from scratch. I asked around the engineering team, and no one but me seemed concerned about the wasted effort. That nagging realization became the inspiration for Genoa Technology’s first product, a printer-emulation-specific programming language and development environment, built so that printer tests could be written once and then reused and repurposed from one release to the next.
The lesson applies directly to AI, and a lot of teams are learning it the hard way right now. Don’t treat an AI implementation as a one-off. The prompts, the evaluation suites, the test harnesses, and the tooling you build for one project are expensive to create and easy to throw away, and throwing them away is exactly what happens when no one designs for reuse. Abstract the work so it can be leveraged across projects instead of requiring a rebuild every time. The teams getting durable value from AI are the ones treating that scaffolding as an asset, not as a one-and-done.
Lesson 7 – Don’t Get Stuck in the Ruts
Founding Genoa Technology taught me a second lesson, and this one nearly put us out of business. I started the company with my co-worker and good friend Gary James, and we set off on a one-year coding effort to create “PC3,” our new printer testing tool. Then we spent the next year trying to sell it. Printer companies loved the idea but basically didn’t want to write their own tests. Neither did we, so round and round we went, quarter after quarter, convinced that the next sales pitch would finally break things open.
It didn’t. As bankruptcy approached, the head of quality at Printronix asked us to develop a full test suite using the tool we were hoping to sell, rather than just buy the tool from us and develop the tests with it themselves. We finally said yes, and that one change turned everything around. The rest is history: our business took off like a rocket once we stopped trying to get the customers to build their own tests with the tool we created.
The lesson for AI work is to recognize when you are stuck in the wagon-wheel ruts in the road. If your AI implementation isn’t used or adopted in the way you expect, the answer is rarely to keep hammering at the same approach with more force. Step back and rethink the framing. Our breakthrough didn’t come from a better sales pitch; it came from abandoning the assumption that someone else would write the tests. With AI, the breakthrough usually comes the same way, not from grinding harder on an approach that isn’t working, but from stepping back and changing it to match the way people actually want to interact with it.
Lesson 8 – The 80% Solution
Thoroughly testing a software implementation can be hard. In fact, sometimes the burden is so great that companies simply abandon the effort and roll the dice. Yet some of our most successful products came from neither giving up on testing nor testing it all: instead, we focused instead on what I call the 80% solution.
Here’s an example. In the 2000s, printer manufacturers began adding the ability to render PDF files sent directly to the printer. From an interoperability testing standpoint, the challenge was vast. The PDF specification ran to a thousand pages, there were millions of PDF files available to download, and many clone PDF producers each generated files with their own distinct fingerprint. Test groups didn’t know where to start. Some ran a few representative files and called it good enough; others ran enormous suites that took weeks to complete.
Our approach was different. We harvested a large number of files from across the internet and ran them through a profiler that identified the key implementation characteristics in each one. We could then analyze that dataset to find the smallest set of PDF files that, together, exercised at least one instance of every characteristic we cared about. The resulting test set did not cover everything a PDF might do, but it was a rational solution to an almost impossible testing problem. The product was very successful.
My most memorable takeaway from that effort came during a presentation I gave to the U.S. division of Panasonic on our PDF testing solution. Afterward, I was asked whether I could guarantee our tests would catch every problem they might encounter with PDF printing interoperability. My honest, from-the-gut answer was, “We’re not that smart.” From the back of the room, one of the Panasonic engineers said, “Finally, someone who isn’t full of shit.”
The same dilemma is everywhere in AI. Agent behavior can be nondeterministic, and the volume of code AI can generate is so vast that checking every line is impossible. You can’t test everything, but you can manage risk by focusing your effort where it matters most, given your skills, your budget, and your time.
It’s worth being clear that this is not the shortcut I warned about in Lesson 4; it is the opposite. A shortcut is skipping a test you know you need, the way I did with that one-line hex edit. The 80% solution is deciding deliberately, up front, which tests will buy you the most protection for the effort you can afford. What you can’t do is pretend the problem away. Rolling the dice is not a solution.
Lesson 9 – Leveraging Partnerships
QualityLogic’s formation 25 years ago was the result of a merger of three companies: Genoa Technology, which Gary James and I founded; Revision Labs, founded by James Mater; and GenRev, a joint venture owned by both companies. The creation of GenRev is a lesson in itself.
In the mid-2000s, our primary testing focus was printer image-rendering software (PDF, PostScript, HP-GL, and the like), and our largest customer at the time was Hewlett-Packard. They purchased our test suites, we tested their printers in our labs, we had staff on site at HP facilities, and we provided training to their test engineers.
During an onsite visit to HP in Boise, I met with one of the key executives in the printer division. Over lunch in the company cafeteria, he pointed out the window at an empty lot across the way and said, “It sure would be nice if you were across the street from us.” On the flight home, two thoughts kept circling through my mind: “This is opportunity knocking,” and “The financial risk could be significant.”
After some lengthy internal debate, we decided the risk was too great to take on alone, so we set out to find a joint-venture partner to share it. Revision Labs, in the Portland/Seattle area, had a great reputation, was about our size, and although we occasionally competed, there wasn’t significant overlap in our customers. Over a series of negotiations, we became convinced they would be a great partner, and the deciding factor was that they shared our values about how you treat employees, vendors, customers, and every other stakeholder. The joint venture, GenRev, was formed and became a smashing success, leading three years later to the merger of all three companies into QualityLogic.
Few companies possess all the skills to fully leverage AI. Your partners may be at arm’s length, such as when you are buying tools or licensing access only to a large language model, or far more integrated into your day-to-day operations. It matters that they bring the right skills, adequate depth of resources, and reasonable pricing, but that isn’t the whole story. I can only tell you, from a lifetime of experience, that shared values belong at the top of your partner-selection criteria. That is one of the reasons we are moving much of our AI business to Anthropic; they have shown that their values are more than just words.
Lesson 10 – Avoiding Shiny Objects and Finding the Real Deal
A few years ago, I was asked by the Chinese government’s major energy research center in Beijing to teach a class on a message exchange protocol and to certify a testing lab they had built. It was an exciting opportunity to visit mainland China for the first time, though I had made countless trips to Hong Kong back in the Dataproducts days.
I was impressed by the quality of the city’s infrastructure, though I did get a good sampling of Beijing’s infamous smog. My hosts at the research center were gracious, and the training was going well. At the end of the first day, I let my host know that I was tired from the time change and the long day and would prefer to head back to my hotel. I was a bit mystified when he led me instead to a van loaded with most of the class, and before I knew it we were parking in front of a five-star restaurant. Not wanting to make a scene, I went with the flow. The next day followed the same pattern: my polite request to go back to the hotel landed me at yet another five-star restaurant. A bit slow on the uptake, I finally realized that I was the meal ticket. As long as I was present, their expense-account limits had no bounds. From then on I was a good sport about eating at a fine restaurant each night instead of resting in my room.
On the third and final day, though, I beseeched them to just take me to a simple local restaurant, which they did. After dinner, as we waited for dessert, I asked whether energy drinks were popular in China, explaining my addiction to Red Bull and my craving for one since arriving. To my great surprise, one of my hosts jumped up and ran out of the restaurant, returning a few minutes later with a grin on his face and a Red Bull he had bought from a street vendor across the way. It was a wonderful gesture, and I didn’t have the heart to tell him that the Red Bull he’d handed me was counterfeit. The can was slightly the wrong color, and the logo was off just a bit. I grinned, popped the tab, and took my chances with the mystery drink.
The moral, as it pertains to AI, is that there are a lot of test-tool vendors and consultants making wild claims about their AI-enabled offerings, and most don’t live up to the hype. You may be desperate for a tool or capability, just as I craved that Red Bull. But unless you are careful and do your due diligence, you may find that shiny new AI-enabled tool about as satisfying as my “mystery drink” Red Bull.
Concluding Thoughts
While it may seem like AI is turning the world upside down, I hope that some of these war stories and the lessons I’ve drawn from them help you see that many of the core principles from our past business lives can still guide us as so much of our professional lives get challenged and disrupted by AI.
