Automation and Artificial Intelligence
AI in everything. That is how it feels these days. It seems like you can’t read anything without that two-letter acronym popping up, and I think it’s about time I offered up an opinion on the topic. I don’t want to be left out of the cool kid’s crowd!
The problems with pragmatism
I describe myself as a pragmatist and I think my peers would likely agree. They’d also describe me as cynical, skeptical, possibly humorous, and likely a critical thinker (or possibly just critical). In other words, I’m pretty much your stereotypical quality assurance professional. Naturally that means I have thoughts about where to use AI, and where not to.
The first quibble
First and foremost, I’d say that the “I” in “AI” is marketing rather than fact. Actually, let’s ask the artificially intelligent system for its own opinion!
Prompt: In 20 words or less, is this system actually intelligent?
Response: No. It applies learned patterns and rules to generate responses but lacks self-awareness or independent reasoning. Confidence: 98%.
Note that last part about confidence. I insist on it telling me how confident it is on any response. I don’t necessarily believe in its confidence mind you, but the system seems pretty definitive about this topic.
Prompt: In 20 words or less describe what this system actually is.
Response: A large-scale statistical language model that predicts text using patterns from vast datasets without true understanding or consciousness. Confidence: 99%.
Where are you going with this?
Where I’m going is to the heart of the matter, and then we’ll go on to how it applies to the topic of this blog. Artificial Intelligence, as it exists today, is NOT intelligent. It is in fact definitively unintelligent. What this thing is, is an incredibly sophisticated dataset mixed with some advanced probability math that can produce text that will be most likely to satisfy the prompt. The people who did this are incredibly intelligent, and I give them nothing but credit for it. The math involved in this thing is vastly above my ability to grasp.
The problem though is this. Critical thinking is a huge part of being good QA, and this system, doesn’t think.
Get back to the point!
Fine, I’ll stop pointing out something that I think is being mischaracterized, possibly to the detriment of a lot of organizations. This tool has its place in our QA world, and I’ll happily sing its praises for what I know it can do.
Test automation is a great example of where this artificial intelligence can be hugely beneficial (look, no snark, I didn’t even “intelligence”… wait, I just did. Onwards!). I am not talking about some of these systems that claim you can just point at your product and turn the QA over to the tool. Maybe some can truly do a good job, but I have my doubts. No, what I think this tool can be extremely useful for, is the hand-creation of test automation code.
Why is that? Simple.
The large language model is trained on a vast dataset of text, including the immense historical knowledge of Stack exchange and other sites. What this means is it is amazingly effective at predicting what comes next in code, especially straightforward code like test automation. I can, with good confidence, get AI to produce simple applications in a language such as Python, and it will, for the most part, do what I want.
Let’s take a basic automation test case structure.
First, we arrange the test, and set the preconditions, then we perform an action, and finally we wrap up with an assertion. The code needed for this is very straightforward for the most part. Find an element, click on it, or type into it. The heavy lifting is being done by the test framework. This type of activity is very much in the AI wheelhouse.
Hold on now!
I’m not saying it is good practice to let the AI choose what to test.
The QA world revolves around deterministic behavior and critical thinking. The only way we can be confident that these skills aren’t lost is by keeping the people doing what they do best. Let AI write code faster? Absolutely in the case of test automation.
You might want to be a bit more careful in your development side mind you because AI can provide some very convincing code, but is it truly what you want? Maybe not. In test automation though, if you know the locator, and the code created clicks on the thing… well it can’t go far wrong. It is tightly constrained. I think it could also be quite useful in the world of writing unit tests.
Remember that the narrower the scope is, the better odds are that the AI will produce effective code.
Is that it?
Certainly not.
Consider the concept of assessing acceptance criteria. AI is a fine tool for that second opinion. If you provide a feature description and the existing acceptance criteria, it will likely do a fine job of pointing out things that might have been missed.
Here is an example:
Prompt: As a user, I want to be able to enter a username into the username field.
A/C: The username needs to be a valid email.
What acceptance criteria should I add?
Response: Here are the additional acceptance criteria you should add to make this feature testable and unambiguous:
Additional Acceptance Criteria
- Format Validation
- The username must match a valid email format (e.g., localpart@domain.tld) using a defined regex.
- The validation must occur on both client-side and server-side.
- Character Rules
- Only standard email characters are allowed (letters, digits, periods, underscores, hyphens, plus signs in the local part).
- The domain must contain at least one period and no illegal characters.
- Length Limits
- Minimum length: X characters (define exact number).
- Maximum length: 254 characters (per RFC email length standard).
- Case Sensitivity
- Validation is case-insensitive.
- Error Handling
- If invalid, display a specific error message: “Please enter a valid email address.”
- Error message must be visible to screen readers.
- Whitespace Handling
- Leading and trailing spaces are automatically trimmed before validation.
- Empty Field
- Submitting with an empty username must trigger an error message.
- Security
- The field must reject scripts or HTML to prevent injection attacks.
See what I mean? Not too bad, and as an aid to a person wanting a second opinion, entirely useful and helpful. I’d have expected good QA to have caught all these things and now have very little to do, but we get tired and we sometimes forget, and this type of tool can help offset that human frailty.
Are you guys using it?
We certainly are where it makes sense, and only when our clients approve. The AI thing is a sensitive area and some organizations are vehemently opposed to these tools, and we have to respect that. While we only use systems which explicitly claim not to use our input data for use in their models, these same companies face a lot of litigation from people who feel that their copyrights were massively abused. I’m not weighing in on that subject, but I’m a cautious guy who has to constantly consider risk.
We’re using this tooling, where permitted, in line with the content of this blog. We use it where it makes sense, and we’re not replacing humans. We might expedite some of the work, and we might use it as an augment, but that is what the tool is truly for.
We are likely to add this on to our TestNitro offering (our fully managed test automation service) because it is a great help in designing DAMP names for locators, and that is something we can definitely use help with when we create code as fast as TestNitro can!
Terminator! The end of the world as we know it?!
Well no, it is pretty much the end of this article though! I’ll take this back to the beginning where I mentioned critical thinking. In QA we question everything about how a thing should work. We don’t predict how it should work. We think laterally, as well as linearly, and that isn’t in the AI model. We are the safety net for the client, and I for one can’t turn that over to something that literally, by its own declaration, does not THINK! So, think about it.
Humans will use what is being produced, humans better sign off that it works. At least for now!
