Posted on

Diversity in Software Test: How a Diverse Workforce Impacts Revenue and Software End-User Satisfaction

Diversity shapes our world. As individuals we have different tastes, experiences, strengths, preferences, ideas and dreams. Much like biodiversity benefits the environment, or a diversified portfolio benefits the investor, diversity benefits us as a society — and on a smaller scale, it benefits the workplace.

Workplace diversity is a hot topic, and for good reason. There are multiple studies that show companies with higher rates of diversity have higher earnings. A recent study by the Boston Consulting Group (BCG) found that companies that have more diverse management teams report 19% higher revenues due to innovation. This finding is huge for tech companies, start-ups, and any industry where innovation and effective change management is the key to growth.

The study illustrates that diversity is not just a metric to strive for, it should be an integral part of a successful revenue generating business. Reaping the bottom-line benefit of diversity doesn’t end with diverse leadership, it arises when diversity is practiced at all organizational levels.

What is Diversity?

Diversity is not simply about gender or even ethnicity, although those are the high-level metrics we tend to measure by. Diversity is also about our different abilities, our sexual orientation, our faith, our age, our socioeconomic status and our origins. These factors represent our individual embodied experience and constitute a wealth of skill, knowledge and potential when leveraged by wise companies.

diverse group of people in software test

In the workplace, diversity should be more than a metric asking us to hire more of this or that group. Simply hiring more women does not change workplace attitudes or how we value (or devalue) our coworkers. To truly embrace diversity we have to understand, appreciate, and actively engage with our differences. Doing this will allow teams to develop creative solutions and innovative new ideas. Of course, the business world is ruled by numbers, so we turn to numbers to judge whether our organization is diverse enough.

Why is Diversity Important?

Let’s look at areas other than business revenue where diversity has been shown to create long-term health and abundance. For many years now, science has understood the importance of environmental biodiversity. Studies show us that the reintroduction of wolves to Yellowstone Park benefited the entire ecosystem. The inclusion of this single element of diversity is credited with saving multiple species like beaver, deer, fox, and mice, as well as repairing an ecosystem plagued by defoliation, erosion and imbalance — to the point of restoring native river flows. [1] This effect is called a ‘trophic cascade’ and, if indicative of the power of inclusion, we should all want to see a trophic cascade in our organizations

There’s also diversity in investments.The investment that promises year-over-year growth with moderate risk, has always been the diversified stock portfolio. From international to cross-industry representation, we’re better off putting our money into a cross-section of businesses that, in aggregate, are bound to do well. [2]

If diversity reaps such immense benefits for both our investments and our environment, why aren’t we more successful at creating diversity in the workforce?

Maybe you’re a hiring manager, a PM, or a C-level who wants more than ideology to support diversity initiatives? Diversity and Inclusion (D&I) initiatives are all the rage, but what is the evidence to support these initiatives? Don’t worry, I’ve got your back here. Let’s look at some statistics.

The Business Impact of Diversity

  • Racially diverse teams outperform non-diverse ones by 35% [3]
    • One of the biggest things stopping managers from implementing diversity is the fear that introducing people who ‘are different’ and ‘don’t understand’ each other will hamper productivity.
  • A cognitive intelligence study done by MIT researchers found that successful teams had three things in common: they gave one another roughly equal time to talk, they were sensitive towards each other (even in awkward situations), and they included more women. [4]
    • In other words, having different types of people on the same team can help us look at problems more carefully while also being more innovative, creative and inclusive about our solutions.
  • Ethnically diverse companies are 35% more likely to earn above-average revenue in their industry [5]
    • Ethnicity is just one aspect of diversity, albeit an important one. If we used D&I metrics that accounted for more than just ethnicity, metrics that accounted for all the other variable that impact human diversity, what might this percentage grow to be?
Above average diversity for higher innovation revenues
Image credit:

Fostering Workplace Diversity

“Fewer large companies are run by women than by men named John”

When a single name outnumbers an entire gender, it reveals a staggering problem.

It’s been argued that correlation doesn’t equal causation (greater gender and ethnic diversity in corporate leadership doesn’t automatically translate into more profit), and this argument is true. If we hire simply to meet a D&I metric without actively engaging in the work of shifting our attitudes and perceptions we’re not being effective at leveraging diversity’s potential.

One example of shifting our perceptions is by using ‘resilient language’, a phrase coined by Ash Coleman, QA engineer and founder of QualityINClusive, a diversity consulting firm. She points out that engaging diversity and inclusion in the workplace often means getting clear with the ways we ideate our biases in response to the type of language used. We infer who is included, and who is excluded, from any given situation through the language used to describe it. Working within the underlying corporate culture to change foundational ideologies is where correlation begins to become causation.

The truth is, the more diverse your company is, the better positioned the company is to win top talent, increase employee satisfaction, and improve customer retention. All of this leads to a cycle of increasing returns. And this cycle suggests that other kinds of diversity — in age, sexual orientation, and experience — are also likely to bring some level of competitive advantage for companies that can attract and retain such diverse talent.

Why Hiring for “Culture Fit” Fails Diversity Initiatives

There’s no shortage of advice on how companies can become more innovative. The catch is that most of that advice is based on anecdotal evidence. But there’s one step companies can take that does have some data behind it. Hire for potential. Hire for skill. Hire for experience. Whatever you do, don’t hire for ‘culture fit’.

Companies often hire candidates based on ‘culture fit’. Whatever the ‘company culture’ is — be it pub nights, spartan races, or a dog-friendly office, the sheer fact of hiring to fit a predefined role is akin to stereotyping. Think of ‘culture-fit’ in another way. By seeking only those candidates who ‘fit’, you’re limiting your resources. When resources are artificially limited by culture-fit constraints, it’s likely you’ll see an employee mix that isn’t really a mix. It’s a somewhat homogeneous blob of people who all like the same things — and likely use very similar processes to solve problems.

Diversity and hiring for culture fit

It’s not uncommon to see tech recruiting posts that advertise pub and party nights as a ‘perk’ of employment. This innocuous sounding statement, baked into you job posting, sets the stage for some prospective applicants to feel excluded. Perhaps they’re gluten-intolerant, prefer wine, have a family to go home to, are in recovery, or have chosen not to imbibe for one of many other reasons.

Crafting language that describes your company’s persona as a ‘pub culture’ excludes those for whom the ‘pub culture’ paradigm doesn’t fit. Further, use of such language creates a space where those humans who don’t see themselves as a ‘fit’ are likely to eliminate themselves from consideration in the first place. This may well cause your company to miss out on highly qualified and valuable employees.

Benefits of Diversity in Software Test

By bringing together our individual backgrounds, skills, and experiences, businesses are better able to breed the type of innovative and creative solutions needed to succeed in an increasingly competitive economy. Diverse people have diverse ideas and diverse ideas allow us to out-think our competitors, a boon in the fast-paced world of software QA!

In the software world, embracing and leveraging workplace diversity is truly a game changer. Now that user experience (UX) plays a vital role in the success of a piece of software, we want people who can ideate from a broad range of perspectives. Software test is no longer simply ‘data functional’, it’s now ‘people functional’. Software testing should always encompass how real people use products in the real world.

Creating useful and usable interfaces for real people hinges on being able to understand our users and approach user problems creatively. Unless you’re targeting a truly narrow and specialized market, it’s highly unlikely that a homogeneous team will be able to imagine the various embodied experiences of our users without embodying at least some of those experiences, as well.

We can look at the benefits of diversity in QA two ways:

  • Diversity enables us to understand and design for our users (increasing adoption rates, and with it, revenue)
  • Diversity enables us to hire the best candidate for the role from the largest pool of potential (increasing effectiveness, and with it, revenue).

Diversity combines optimized skill-sets and increased user empathy in a win-win for everyone.

How to Hire Diverse Employees in Technology

The face of the computer science graduate is changing. More women and underrepresented people, especially those who are younger, are entering the tech field and will begin to make up a significant portion of the hiring pool.

Diversity in Software Test and Technology Growing
Statistics show a rapid rise in the number of female students and underrepresented minorities taking the AP Computer Science exam. ( Graphic)

The first step in recruiting for diversity is opening the door to underrepresented applicants. However, there’s more to consider. Following are some ways we can ensure applicants aren’t just another metric to tick off our ‘interview diversity’ checklist:

Check Your Recruiters’ Unconscious Bias

  • Unconscious bias might feel like a buzz phrase these days, but it’s real. Bias starts with a name on a resume. Make sure you give equal consideration to John Smith, Jane Smith, Juan Rodriguez, Kanisha Browne, Samir Nasri and Mi-Ling Chan.

Establish A Clear Organizational Commitment

  • Have a clearly written organizational commitment to diversity and share that commitment both internally and externally, then make sure operational practices align with your commitment.

Leverage Your Internal Networks

  • Organize employee resource groups (ERGs) that leverage their expertise to serve on hiring panels, involve them in talent management activities and create opportunity for them to interact with the executive team.

Walk the Talk

  • If you want a diverse workforce, you want a diverse sourcing strategy. Look for diverse sourcing strategies such as college recruiting, community associations, social media platforms, and industry specific groups.

Make Diversity a Part of Every Hiring Conversation

  • Does your benefits plan cover same-sex marriages? Share that! Let you applicants know any ways you advocate for social causes that advance diversity.

Set Direction and Parameters for Talent Needs

  • HR leaders can increase diversity of talent by setting the direction, parameters and constraints of hiring needs. What missing variables can improve a cross-functional team?

For more information on hiring challenges in the technology industry check out our recent blog post on hiring for potential.

[1] Staff, “Wolf Reintroduction Changes Ecosystem,” My Yellowstone Park, 21-Jun-2011. [Online]. Available: [Accessed:18-Oct-2018]
[2] S. Moore, “How Much Diversification Is Enough?,” Forbes, 12-May-2018. [Online]. Available: [Accessed: 18-Oct-2018].
[3] J. Wolfers, “Fewer Women Run Big Companies Than Men Named John,” The New York Times, 02-Mar-2015. [Online]. Available: [Accessed: 18-Oct-2018].
[4] R. Tulshyan, “Racially Diverse Companies Outperform Industry Norms by 35%,” Forbes, 13-Feb-2015. [Online]. Available: [Accessed: 18-Oct-2018].
[5] “The secret ingredient that makes some teams better than others,”, 07-Dec-2015. [Online]. Available: [Accessed: 18-Oct-2018].
Posted on

Developing Your Testing Superpower: Why You Should Cultivate Empathy in Software Test

“User experience focuses on understanding and designing experiences tailored to human behavior” ~ Dr. Carly Finseth, Teach Like a Gamer: Adapting the Instructional Design of Digital Role-Playing Games (Studies in Gaming)

In the hubbub of developing and testing your software product, leveraging the value-add of empathy in software test often gets overlooked. The fundamental role of software testing is as a search for quality-related information. We use this information to close the gap between what we know and what we don’t know about our products. This search for information is a rich intellectual activity requiring us to question, study, model, explore, experiment, deduce and extrapolate. These activities ask us to both analyze—and empathize.

Empathy, defined as the ability to understand and share the feelings of another, is the star of today’s human-centered software design and test. It’s no longer enough to create an app that doesn’t suck. Unless your software is a true unicorn, there’s at least a handful of other apps that do the same thing yours does. And if there isn’t yet, you can bet your lucky unicorn there will be soon! With so many choices, a customer has no reason to get stuck using an interface they don’t like—or that doesn’t seem to like them.

IBM stated in their write-up on User-Centered Design that “every dollar invested in ease of use returns $10 to $100.” How? Let’s look at statistics for two categories, websites and apps.

For websites, the rise of the mobile device means:people using mobile phones

  • 73% of website users will abandon your site if it’s not optimized for mobile
  • 52% of users who have a poor experience with a mobile site will not engage with the company—on any platform—PCs included
  • 60% use mobile exclusively to make purchase decisions

The average smartphone user interacts with their device 47 times a day. This translates to:

  • Current number of Android apps in the market: 2,746,701
  • New Android apps in the market during Sept 2018: 45,306
  • Percentage of apps introduced in September listed as ‘low-quality’ by AppBrain: 52%!

What do these statistics show? We can learn that one of the best times to engage empathy is at the beginning of the development process! Historically, websites and apps have been designed for early adopters, then adapted for a more general audience. Now the shift to mobile devices asks us to shift this thinking to a ‘mobile-first’ approach. Instead of starting with a website or app full of feature-creep, empathize with your users and where they’re using your software. Use the constraints of a small screen and an attention divided user to make the hard decisions about reducing unnecessary complexity and providing important information first.

There’s an astounding amount of digital use data available online. What it boils down to is this: there’s a lot of ‘bad UX’ in the digital world—and today’s customers won’t use platforms with negative emotional UX.

Empathy in Software Test – The Human Touch

We’ve all used sites and apps that felt like the product owners were from a different planet. Think of the site with strange user paths to find info, or the app so complicated it seems to require a PhD to use. Both of these scenarios leave our users with an unpleasant residue of negative emotions. It might seem silly to write that a compilation of code is ‘unfriendly’ or ‘built for someone not like me’, but that’s exactly the message we send our customers when our testing lacks empathy.

Today’s software is assumed to work but is adopted when it offers a pleasing experience. To be truly empathetic software testers we have to feel the pain—and the pleasure—of our users. We must ‘walk a mile in their shoes’. If we don’t allow for the emotions of our customers during test, we end up with something that’s not human-centered test, it’s ‘task-focused’ test.

How Empathy Uncovers Design Flaws

Websites and apps are created for human users, (unless you’re a cat, then you get apps of your own). What do we humans have in common? A complex and vibrant palette of emotions. These emotions affect our perception of product quality and value, and our assessment of a company’s ethos. Cheap wine tastes better in fancy glasses. Sales of Macintosh computers soared when Apple introduced the colorful iMac.

Empathy in Software testIn Emotional Design, Why We Love or Hate Everyday Things cognitive scientist Don Norman shows us that research into emotion and cognition proves attractive things ‘work better’. A successful product not only eliminates confusion, frustration, and irritation, it is also attractive, pleasurable, and fun. Hey, we’re humans, right?

Norman’s research into why we use the products we do divides our emotional responses into three categories: visceral, behavioral and reflective. At the visceral level, physical features—look, feel, and sound—dominate. This means the sculpted curves of a heavy wine glass make our cheap wine taste better.

The reflective level is where we interact with the product because of its meaning. Because it tells us something about ourselves, or our culture. Reflective design is about self-image and the message the product sends to others. This is why teenagers want whatever the cool new kicks are.

The behavioral level is about use. Look doesn’t matter, performance does. Those sneakers might look cool, but are they comfortable? The behavioral aspect is where most software design and testing are focused, often under the guise of User Experience (UX). UX begins with ensuring our products fulfill a user need through functionality and usability, but Norman’s research teaches us that behavioral design addresses only a small fraction of the emotional palette.

Empathy in Software Testing is Vital to End-User Satisfaction

UX is the hot test-and-design method of the year, and for good reason. According to the 2018-2019 World Quality Report ensuring end-user satisfaction is the top objective for QA and testing functions.

UX stems from design thinking, an approach that begins and ends with people. Beyond a test focus of how the software works through functional testing, UX testing focuses on how the software makes us feel when it’s working. Empathy allows us to address not only the behavioral design aspect, but the visceral and reflective levels as well. This distinction—task completion vs emotional fulfillment—is why great software test engineers have a healthy sense of empathy. They feel their users’ pain and pleasure. They think beyond task completion to emotional fulfillment.

Empathetic testers walk that mile in a users’ shoes. They may even actually talk to their users. They think about what their user is trying to do, and how it feels to do it. After all, how likely is it that a 28-year old guy will able to design software for a 70-year old woman without understanding her journey? And that understanding starts with empathy.

The need for empathy doesn’t begin or end with UX, or even with software testing. Empathy is a value-add during functional, performance, and exploratory testing and a boon for developers, also. Think of it this way, even if you’re working on a standard business application, fundamentally you want your users to enjoy—or at least not hate—using your software.

Using Empathy to Help Users Complete Tasks and Love the Experience

You might have caught articles touting “Good UX focuses on helping the user complete their task and minimizes frustration.” I’ve written the same many times, and in a basic sense the statement is true. Users want to ‘do the thing’ with a minimum of hassle. But it’s easy to forget that there’s more to life than ‘getting it done’. So, I ask myself: “Why are the current UX testing methods still primarily ‘task’ focused and why, when user emotions are considered, are the negative emotions like frustration emphasized?”

Interactive technologies are meant for human users. It makes sense to engage in our test activities from the perspective of getting a task done Good UX equals a happy userwithout frustration, but also to build interaction that supports feelings of accomplishment, belonging, self-fulfillment, self-esteem, family, satisfaction, security, delight, empowerment, and emancipation. In short, we should design and test our software offerings with user well-being in mind.

Dr. Marc Hassenzahl, professor of Ubiquitous Design/Experience and Interaction at the University of Siegen, Germany recently published A Personal Journey Through User Experience in the Journal of Usability Studies. In his article, Hassenzahl shares the way Human-Computer Interaction theory and his understanding of what UX is has shifted over two decades. He takes us from “A sharp distinction was made between sober and elegant ‘tools’ and gimmicky ‘toys’—the latter supposedly bloated with useless functionality and ornament.” to “cognitive science-imbued approaches… that embrace beauty, emotion, experiences, persuasion, and many more concepts.”

Much like Normans research into the three types of emotional response (visceral, reflective, and behavioral) Hassenzahl makes a persuasive argument for us to recognize “the experiential as a central and explicit objective of design”. In software, this means we shift from “a narrow concern about whether a product can be understood and operated to a much broader concern about desired and undesired experiential consequences of prolonged product use.”

This sort of human-centered design-thinking is a critical tool for everyone involved in the creation of a software product, and is the fundamental value-add of empathy in tech.

Software Testing Skills for the Empathetic Tester

Create empathy-based test scenarios

  • How do our customers feel when using our products?
  • What are our users seeing and hearing when using our software?
  • What types of positive or negative thoughts are they thinking while using our apps?

Focus on user goals beyond task completion

  • Using a product isn’t a goal in and of itself. Dig deeper and see what needs your product is meeting will meet.
  • For example, “use social media” is not a user goal, but “connect with my friends” or “let the world know what I’m doing”, or even “stay in touch during emergencies.”

Dissolve the IT intra-industry empathy gap

  • There is often a lack of empathy between people working together on projects. When we’re under pressure to deliver a new website or application, it is all too easy to be critical of our colleagues without understanding their experience of the situation.
  • Don’t be the Tester who points at Dev and says, “poor quality” or the Dev who looks at Business and mutters “changing requirements”.
  • Instead, ask “what are they experiencing right now?” and “how can I help?”

Use Empathy to Release Better Software Products

Every step of your customers’ journey, every touch point, every interaction, every detail of your brand and your software, should reflect empathy for their experience. The myriad digital offerings available today make user experience and empathy the initial, and often only, driving force behind success.

The first step we in the software industry should take, I think, is to embrace empathy. We must put ourselves in everyone’s shoes and not just the ones that fit best.

Check out our usability testing services

Posted on

The Tech Talent Shortage: Why Your Recruitment Strategy May be to Blame

No matter who’s list you look at (like this one, or this one) STEM programs dominate the list of ‘most useful’ college degrees, with disciplines like web development, software engineering, computer science and data analytics hogging the top spots. Code Camps, both free and paid, promise to turn anyone into a front-end or back-end developer in nine months. Classrooms from university to grade-school promote STEM initiatives.


Because, two things. First, that ‘T’ in STEM stands for technology. Without technology, progress in the S, E, and M aren’t possible. Sure, we could still do long division with pen and paper, but who wants to? Tech not only enables scientific development, it also allows us to shop online, call an Uber, research a topic, and even have an easy ‘out’ when it comes to performing arithmetic. Technology permeates every industry; successful companies recognize this fact.

STEM Tech Talent Shortage

In 2017, Business Insider quoted Dr. Andrew Chamberlain, Glassdoor’s Chief Economist, as saying “Any organization today with a mobile app, web presence, or digitized data are struggling to fill jobs like data scientists, software engineers, and mobile developers.” Companies across all industries, including healthcare, finance, and retail, are now ‘tech companies’, whether they realize it or not.

Second, this fact: A shortage of tech workers is already happening and is predicted to reach almost a million unfilled positions by 2030 if current growth continues. The need for software developers alone is expected to rise 30% by 2026. Hiring experts are already sounding the alarm. A survey by stated “Almost 9 in 10 respondents (86%) said they find it challenging to find and hire technical talent, with over a third (36%) saying they find it “very” challenging.”

What Does the Future Look Like for Employers in High-tech Industries?

Ask any recruiter what roles are hardest to fill in 2018 and you won’t be shocked. Tech workers top the list, with Software Developers, Web Designers, Data Scientists, Solutions Architects, and QA Engineers in high demand. Yet, finding the right candidate is a challenge. Why? Because adhering to the outdated paradigm of seeking candidates with experience, rather than potential, is limiting.

Tech evolves daily, which means that what was relevant last week might be less so next week. Hands-on, current experience matters more than technologies that were studied ten years ago. Expectations of ‘experience’ translate into hiring failure when organizations hold fast to requirements such as “x years of experience with z platform”. Talk about limiting your search to applicants stuck in the past!

Outdated hiring processes

What about the recent college graduate (millennials, if you must) who don’t have baked-in, preconceived notions about how things are supposed to work? Recent graduates, with a plethora of theoretical knowledge but very little dogma, might just be your ticket to finding employees who are more malleable and more willing to break barriers than the experienced workers. I don’t know about you, but I’d rather work with someone who’s demonstrated adaptability to changing circumstances than one who’s constantly shouting “BUT WE’VE ALWAYS DONE IT LIKE THIS!”

How Will Companies Have to Adapt to the Changing Values of a New Workforce?

In 2017, millennials became the largest generation in the labor force, surpassing GenXers by 3 million. As a result, organizations who want to benefit from this uniquely motivated cohort should understand their perspective. Millennial attitudes have been shaped by tremendous technological advances and a turbulent political and social climate.

In opposition to commonly held negative views of millennials, here are a few (and maybe surprising) traits of the generation as revealed by research:

  1. Millennials are curious, and they’re excited to learn new skills. Toiling away at a job just to gain a promotion isn’t a path these folks are interested in. Companies can leverage the millennial desire to expand skill-sets by creating career-development programs offering access to job-related ‘experiences’, rather than a simple ladder to the top.
  2. Millennials are the digital generation. They grew up with cell phones, the internet and personal computers. Because of this, they prefer to work with companies who are technological innovators. They also learn new technology very quickly so keep that in mind when you’re writing a job description that requires x years of experience with a particular software program. Chances are, most millennials can pick up and master a new software program in a matter of months or even weeks.
  3. Millennials are collaborative. Provide them with the opportunity to co-create with others in the organization and watch the ideas flow! Equally important to millennial collaboration is organizational transparency. Developing a transparent work environment where employees are aware of work happening throughout the organization encourages this generation to remain vested in organizational success.
  4. Millennials like feedback, even if it’s not praise. Because they’re curious, and want to improve themselves professionally, they view regular feedback as an important component of professional growth. Try scheduling short weekly or bi-weekly meetings with managers to mentor and provide feedback.

A team of millennial employees

How to Recruit Talented, Adaptable Applicants Regardless of Experience

Anyone can follow steps to solve a problem, but the greatest asset to any business is true agility. Yes, it’s important to find a candidate who can code in a specific language, but finding one who can also learn, adapt and evolve if other digital skills are needed should be the goal of any tech worker recruiting effort. Tech recruiting is hindered by a narrow definition of what a ‘quality-applicant’ looks like. If we only consider metrics like university attended, degree held, and years of experience, we’re missing out on a massive pool of candidates.

Then, there are the non-traditional applicants. These individuals often have the chops to succeed but are disqualified due to ‘circumstance’. Recently, a senior-level manager with our company had the chance to ask a local tech CEO about hiring for potential. The CEO shared a story about one of their best employees, a computer savant who was caught hacking into their college database. They met at a local pizza bistro, where the ‘hacker’ was stuck waiting tables due to their criminal record. The CEO saw potential and took a chance by hiring someone conventionally considered unhirable and was glad they did. That “hacker” is now a top-performer at the company and well-liked by clients.

Non-traditional applicants, from mid-life career switchers with code camp experience to those with skills garnered through decidedly unconventional methods, are an often overlooked and underutilized resource. These people, much like millennials, want to explore new ideas and bring value to their organizations. They can be much more valuable than hiring the applicant who wants to ride out tenure for promotion and retirement.

Admittedly, it can be difficult to define candidate metrics when recruiting for potential, so here are a few ideas:

Test for Skill

  • Code challenges are a tool that recruiters can use to evaluate non-traditional candidates for technical prowess and willingness to learn. A code challenge, or some other mock scenario, allows you to get a firsthand look at problem-solving skills, adaptability, and poise under pressure. Even if a candidate doesn’t produce the ‘right’ answer, testing for skill allows recruiters to empirically learn how someone would perform on the job.

Don’t be Afraid of ‘Job-Hoppers’

  • Frequent job changes used to be seen as the mark of an unreliable candidate, but the younger generation sees job changes as a way to grow skill-sets. Candidates with a modest diversity of experience are more likely to be adaptable and having diverse experiences can help with problem-solving.

Internally Assess What the Job Entails:

  • Take a hard look at the positions’ requirements. Are they nice-to-haves, or must-haves? Clearly define the position’s responsibilities, then look outside the box for ways a person could gain these skills. Think about our pizza-serving hacker example! Interviewers who understand the context of the position in relation to the candidate’s experience can then tailor the interview to explore the candidate’s potential.

Leverage the Value of Training Programs and Mentorship Opportunities

  • Formal education is a great foundation, but the application of education to workplace problem-solving is a skill learned through experience and mentorship. While most companies prefer candidates who can hit the ground running, the unfortunate truth is universities often don’t teach to the complexities of modern tech stacks. Recruit emerging talent that can be mentored and cultivated, then provide mentoring programs either internally or externally. Incentivize the desire to learn.

What Does the Tech Talent Shortage Mean for Software Testing?

One of the largest challenges faced by the software quality assurance sector is the deconstruction of traditional development silos. Collaboration among developers, IT professionals and operation engineers has become the norm as methodologies like continuous development and Agile sweep the industry. These development methods call for team members with cross-functional skills, yet at the same time advances in specialized test sectors like automation and security are creating hard-to-fill niche specialties. According to a panel of QA recruiters from a recent Denver SQuAD meeting, employers are looking for testers who have highly technical skills and specialize in a particular discipline such as security, performance, or automation implementation.

What does the tech talent shortage mean for individuals wanting to advance their career as a test engineer? First, commit to staying abreast of QA industry developments. This could mean honing your programming skills, learning a new machine language, or getting experience in automation tools like Selenium. Testers, indeed anyone working in a technical field, can’t afford to bury their heads in the sand and ignore the modern-day tools and techniques being used.

always learn

There are plenty of non-traditional, non-university ways to do this. Check out courses by Udemy, Coursera, CodeAcademy, and Treehouse. These sites offer training in everything from basic code and manual testing to specialties in Cybersecurity, Agile testing, and Automation. Manual testing will always be the foundation of quality assurance but moving your capabilities towards fluency in specialized skill-sets is a proven QA career boost.

Companies can support employee development through long-term initiatives with short-term benefits like increased loyalty and improved performance and engagement. Employees want to feel their managers genuinely care and are committed to supporting their professional advancement and even their personal growth. Millennial’s, in particular, desire support, coaching and paths to advancement. Initiatives could include:

  • professional training
  • in-house coaching and mentoring
  • cross-departmental training and
  • ‘soft-skill’ or personal skill development.

One thing is certain, the rapidly changing digital world we live in offers a wider range of opportunities than ever before. Whether you’re a business owner, an employee, or a job-seeker, we now have more ways to learn and to do than ever before. Why get stuck in the past?

If your company is having a hard time finding qualified test automation engineers, test technicians, or test case writers contact us about outsourcing your QA functions.

Contact Us Today

Posted on

Women in Test: Fiona Charles

Women in Software Test - Women in Technology

The Women in Test Series

Women in Test is a series focused on women in the software testing world, and the ways they advocate for inclusivity and diversity in the discipline. The software industry has traditionally been a homogeneous field, but as we know, times are changing. Leadership and employees alike have begun to recognize the value of diversity, even as they often struggle to practice it. Speaking to an industry undergoing intense change in its makeup, our series participants share their stories, thoughts and advice as advocates for inclusion and diversity in test.

An Interview with Fiona Charles

Fiona Charles has seen it all. With 40 years in software development, Fiona’s had a front row seat to the evolution of the digital age and the associated ideologies and methodologies. Fiona now heads Quality Intelligence, an independent consulting company offering her expertise in software testing and quality management. Helping companies align their quality strategies with their business challenges is Fiona’s current focus, but her introduction to the tech world began as a technical writer in her university’s library automation company.

Her introduction to the world of tech came during her years at the University of Toronto. While she was a student there she was hired to work for a summer as the pioneer technical writer at UTLAS, the University of Toronto Library Automation Systems. UTLAS had never employed a technical writer before, so Fiona’s challenge was to learn how to be a technical writer and invent her job. Her first day at work she was shown to her desk. When her boss showed up, he said, “I hope you’ve brought your library card so you can go to the Computer Science Library and figure out how to do this job.” (She didn’t find anything useful there.) He also handed her a book by computer scientist Jerry Weinberg, The Psychology of Computer Programming, saying “You might want to read this if you want to understand the people you’ll be working with.”  Weinberg’s work as author and teacher of the psychology and anthropology of computer software development was groundbreaking when it was published in 1971, and is considered foundational to Human Computer Interaction (HCI) theory today.

The book and that first job were the genesis of Fiona’s lifelong approach to software development, testing and consulting,

“Software development is all about people.”

QualityLogic: What were the early years of tech like for software developers? Was testing a defined role?

Fiona: When I started testing, I didn’t meet any other testers. I was fortunate to work closely with programmers who mentored me; they taught me how to test.

QualityLogic: You’ve been involved in the evolution of the software industry. Can you speak to any differences in the treatment of women in the field over the years?

Fiona: When I started at UTLAS in 1978, we didn’t have gender balance exactly, but a large number of the programmers and senior technical staff were women. Nobody there questioned that or thought it was odd. There was certainly sexism in operation, but the tech culture I experienced was nothing like an explicit bro culture, and I think the people I worked with would have been revolted by the idea.

The gender mix at UTLAS was common in software development then. That began changing in the 1980’s towards a male-dominated model. (Many people have speculated on the reasons for this, but that’s not really important to this discussion.)

“The past is the past. We now have a software industry that is heavily male-dominated, and in some subcultures, toxic for women in technical roles. To me, the interesting and important question is what can we do to make the present and future better?”

QualityLogicI love that you’re focused on positive change. Let’s talk about ways to ‘make the present and future better’.

Fiona: If we want to increase gender diversity in software development—and I happen to believe that would be good for the world generally, as well as specifically for women’s aspirations—then we need to help women get into positions of prominence so other women can see that it’s possible for them too. That applies to management roles, and also to conference platforms, blogs, articles, books: everywhere authoritative voices are heard.

Since conference speakers represent thought-leaders in our industry, focusing on change in this area is one way to re-engage women in tech, including testing. Think about this, if you go to a conference and all the authoritative figures are male, then what does that say to a young female audience member, about perceptions of her value and her potential contribution to the industry?

“I think it’s essential for women to see other women being full participants at conferences, and that means delivering keynotes, conducting workshops and tutorials and delivering track talks.”

QualityLogicYou and Anne-Marie Charrett co-founded Speak Easy, a technical conference diversity initiative. I understand you’ve passed the torch to new leadership, but could you tell us a bit about how the program works?

Fiona: Speak Easy operates through forming alliances with conferences and pairing new speakers with mentors to help them succeed with their talks. It focuses on new voices from the testing community to bridge the gender gap represented at conferences, in particular with the number of women speakers. This idea of pairing mentors with young women in test isn’t only about seeing more diversity on conference stages, but can also help with other areas like writing proposals, or articulating value and making presentations at work.

QualityLogic: You focus on diversity in conference speakers as a way to create diversity in the industry yet there seems to be a large amount of diversity in other platforms, like female software developers and testers on Twitter. Why do you think there’s a disparity in representation between the two?

Fiona: I’m not sure there really is greater genuine diversity on Twitter, though there may be comparable numbers of men and women with accounts. Twitter is a poor medium for the exchange of ideas. It’s built for clever one-liners, not for conversation. That makes it inherently competitive, rather than collaborative. That can put women off, because female communication tends to be more collaborative than competitive, and you’ll see often women duck out of an exchange when they see it becoming competitive because that’s really not how most of us work.

“The most productive way to generate ideas is through conversations, and where else do you get a large group of individuals together who to converse about a specific topic than at conferences? Conferences done right genuinely encourage people learning from one another.”

QualityLogicWhy did you choose to go into consulting? What’s the draw?

Fiona: The biggest draw is that it’s a people-focused job. And I love to create order out of chaos! That’s one reason I love testing, too.

QualityLogic: You told me about attending Jerry Weinberg’s residential course focused on problem solving in leadership. Can you expand on the ways this helped you with consulting?

Fiona: That would take a book! The class was PSL, Jerry Weinberg’s Problem Solving Leadership workshop, which I attended in 2001. Many people have found PSL to be life-changing. For me, it was an expansion of a journey I’d been on for a long time. The things I learned there have had impacts on everything I have done since, in life as well as in my work. It’s difficult to pinpoint exactly what those were, but PSL certainly liberated me to be more myself at work and it taught me a heightened awareness and understanding of my own and other people’s behaviors. There were more specific learnings, too, like models and techniques for approaching and solving problems. All of those definable and indefinable learnings have been assets in my consulting work, and in fact I’m still learning from them.

QualityLogic:What advice would you give someone wanting to become a consultant like yourself?

Fiona: “To be an effective consultant you have to have something to consult about. There must be a body of experience to draw from. You also have to be personally comfortable living and working in conditions of uncertainty, while understanding that most clients and other people you work with are probably not.”

Fiona was careful to point out that consulting isn’t simply deploying “a big box of processes”. That type of rigid thinking rarely works. Rather, she said, “I don’t consult to a foregone conclusion. I don’t arrive with a box of processes tucked under my arm—I come with questions. I don’t believe in “best practices” or one right answer. I usually want to start by knowing how much software risk a company’s carrying and how they’re dealing with it. That can help expose the underlying dysfunctions and then we can start looking at ways to address those.”

Fiona’s a pragmatist. She boiled down her consulting ideology to this,

“Stop being hung up on the religion of processes. Focus on what will work.”

Posted on

Women in Test: Gem Hill

Women in Software Test - Women in Technology

The Women in Test Series

Women in Test is a series focused on women in the software testing world, and the ways they advocate for inclusivity and diversity in the discipline. The software industry has traditionally been a homogeneous field, but as we know, times are changing. Leadership and employees alike have begun to recognize the value of diversity, even as they often struggle to practice it. Speaking to an industry undergoing intense change in its makeup, our series participants share their stories, thoughts and advice as advocates for inclusion and diversity in test.

An Interview with Gem Hill

Podcast legend Gem Hill uses “Let’s Talk About Tests, Baby” to chat about software test topics with a variety of influencers, and about the health challenges faced by people in what’s often a high stress environment. If you’ve worked in test, or tech, you likely know how important self-care is in what’s often a high-stakes and decidedly time-crunched world. Gem shines a light on mental hygiene with a non-judgmental, ‘been there’ understanding of how depression, anxiety, and imposter syndrome can affect us at work and beyond.

Gem graduated from university with a degree in biochemistry. From there she went into pharmacy where she found the job appeal lackluster, at best. Her heart was in science, but she found research to be “a slog”. After spending time learning the ropes by working for digital agencies, she landed a dream job as a senior tester in The BBC’s Voice and AI department. Gem’s other dream job, podcasting, gives her the perfect opportunity to create a safe space to talk about mental health issues in the software test world.

I was introduced to Gem via podcast #95 titled “You don’t need client approval, you just need self-approval.” This 2018 talk took on the topic context switching, or the issues that arise when our focus is divided among many projects or clients, with fellow tester James Sheasby Thomas. Gem’s intro blurb for the podcast reads, “Before I moved to the BBC, I worked in a couple of agencies and after talking to a few non-agency testers I realized a lot of people don’t know what it’s like working in an agency.” Having spent half a year doing agency work myself, I realized she was right. Until you’ve worked as a hired gun in an agency you have no idea how the gig economy, where temporary jobs are common, and companies tend toward hiring independent contractors and freelancers instead of full-time employees, impacts life.

QualityLogic: Agency work requires constant focus switching. That must be a challenge. It probably feels like there’s a point of diminishing returns, where it becomes a detriment to all your projects. How did you handle it?

Gem: Gem clarified by saying “I loved my time in agency. It taught me a lot. You’re on so many different projects you can learn a lot in a very short space of time.” “But,” she said, “I think there is a detriment in that you can’t absorb some of those things properly. They’re gone the minute you’re doing something else.”

Her advice? Get serious about structuring your time! Gem told me she was lucky that a lot of clients she worked with were still sort of ‘waterfall-y” so she had an idea of what was in the pipeline week by week, or even day by day “in this sometimes-chaotic environment”.

QualityLogic: How do you structure time when you’ve got different clients with different demands?

Gem: Gem told me that no matter what, it’s always going to be hard to juggle the multiple client scenario. But, “With test, knowing I had some sort of space planned for a project gave me calmness in the moment to deal with what was in front of me.” One technique Gem suggested is to split user stories into the smallest chunks possible. Doing this helps because it allows us to schedule smaller chunks of time for each point.

QualityLogic: What’s your overall take on the ‘gig economy’?

Gem: “I worry about how it seems to encourage a race to the bottom” Gem said bluntly. “It’s suddenly ‘how cheap can you do this; how quick can you do this?’ Then, paying employees a decent amount of money and offering decent working conditions just goes out the window chasing those contracts. This leads to terrible projects and the whole cycle of awfulness continues.”

Still, Gem’s take on agency work isn’t all cautionary. “I think it can be done well. I’ve seen agencies do it well, where they consider basic mental health and they encourage physical health instead of just buying people beer and pizza.” Gem’s frankly over the beer-and-pizza paradigm! “Just give me a salad, please!”

QualityLogic: You’ve often talked about mental health in your podcasts. Anxiety, depression, and imposter syndrome seem to be common in test.

Gem: “Sure, I love to talk about mental health because everyone has it, but some people don’t think about it. It’s nice to think about mental health before you absolutely have to.” Gem told me that she had been one who ended up ‘having’ to think about her mental health when she reached a point where it was “basically look after myself or lose my job.” She realized depression and anxiety had crept up on her without realizing it. “Depression and anxiety happen. They’re just part of life. Life’s going to give you a kickin’, sometimes, no matter who you are, but it’s important to realize when it’s been going on for too long and you’ve actually slipped into a mental illness.”

Gem believes that raising awareness of mental health issues helps everyone.

“It’s like accessible design. It helps people with disabilities, but it helps everyone else at the same time. The same applies to thinking about mental and physical health in the work environment. It helps everybody.”

QualityLogic: So, how can we recognize ‘the creep’? How can we, as employees, tell when we’ve crossed the line from just having a down day to mental illness?

Gem: Gem suggests we monitor creep by assessing the different levels of self-care we practice. “So, you’ve got your basic stuff, showering, brushing your teeth, making sure you’re eating properly and drinking something besides coffee, Red Bull, and alcohol. Just making yourself suitable for public consumption. Then you have the house that you live in, which is the next level up, because now you’ve got to look after something that isn’t you. So, you’ve got to hoover and wash the bed sheets. Then, you’ve got doing things that you enjoy.”

Gem said that when we find ourselves not having the energy to do the things we enjoy it’s time to start thinking about what’s going on in our lives.

“You might just be going through a bad time, but maybe keep an eye on that.” The real warning sign is “If you don’t want to do housework, and not because it’s the most boring thing in the world, but because you cannot gather the energy to care that you haven’t hoovered the carpet in a month and there’s hair everywhere, that’s when I’d start to be worried.”

Gem’s premise is this, that tracking how much effort we can put into self-care and the care of our surroundings, translates directly to our mental state. Monitoring when we don’t have the energy for fun stuff, or tidying the house gives us the opportunity to reach out for support and course correct before rock-bottom happens.

QualityLogic: What can employers do to raise awareness of mental health?

Gem: “The BBC offers a program called “Mental Health First Aid” training.” This is a training program to help employees support employees in crisis. “The idea is teaching employees how to spot danger signs and how to respond to someone if they need help.” Employees who’ve taken this training wear different colored lanyards so that people who are feeling anxious or depressed can identify graduates of the program as a safe person to seek help from.

“We can look around and see the lanyards and know that person knows what I’m going through. When mental health issues aren’t stigmatized it’s much easier to ask, “Can we have a chat?”

In Gem Hill’s 2017 TestBash Manchester talk she says, “marking yourself as someone that is safe to talk to…and engaging with those that look like they are having a bad day” can be a huge lifesaver.

At any given time 1 in 5 people are dealing with a mental health issue. Even if you don’t have a mental health issue, someone around you likely does. You can be a mental health ally by educating yourself about the issues and regularly checking in on the people you work with. Simply asking someone about their day may be what that person needs to feel seen and valued.

Note: If you feel like you don’t have anyone at work to talk to there is an awesome Slack channel you can join. It’s a place where you can check in and chat with people all over the globe who may be experiencing the same things you are.

Posted on

Women in Test: Melissa Eaden

Women in Software Test - Women in TechnologyThe Women in Test Series

Women in Test is a series focused on women in the software testing world, and the ways they advocate for inclusivity and diversity in the discipline. The software industry has traditionally been a homogeneous field, but as we know, times are changing. Leadership and employees alike have begun to recognize the value of diversity, even as they often struggle to practice it. Speaking to an industry undergoing intense change in its makeup, our series participants share their stories, thoughts and advice as advocates for inclusion and diversity in test.

An Interview with Melissa Eaden

In addition to nine years in testing, Melissa is also a technical writer and EditorBoss at the Ministry of Testing. She’s spent two years as a consultant for Thoughtworks, a software consulting company comprised of “a community of passionate individuals whose purpose is to revolutionize software design, creation and delivery, while advocating for positive social change.” She’s also a blogger, canine enthusiast, and a frequent speaker at test conferences around the world. Melissa is passionate about quality, in more ways than one. And, she’s highly likely to wield her metaphorical sword if you suggest testing is a non-technical role!

QualityLogic: Why is the technical vs. non-technical debate a bad thing?

Melissa: “Think about this, the person who fixes your car used to be called a mechanic. Now, they’re automotive technicians. But does this vocabulary change create more respect, more value, for the profession? It shouldn’t have ever mattered…not everyone can fix a car.” I got her point. Somehow, the use of the word technical confers a higher status. Fixing a car is highly technical, no matter what we call it.

To Mel, the automotive technician is no different than the support employee who, in order to help a customer, probably knows more about the interface than the developers do. Or, the software tester who might not have written the code but is intimately familiar with it.

“Too often, pay is based on ‘technical ability’, yet everyone plays a role in creating and maintaining a successful product.”

Mel speaks from experience here. The first testing job she was offered came with a pay rate lower than she made in support—at the same company! Mel had this to say about the experience: “It was my first testing job. The value of the testing role was considered less than my support position. I had been in support roles for several years at this point. My support manager offered to give me a $2 raise if I didn’t change jobs because I was that good. The HR department said that the testing role wasn’t equivalent and that it was $5 less an hour than an entry level support position. Although I had a degree, and regardless of my experience with support, according to the company, the testing role was not considered a technical role while my support role was. Due to my tenure at the company, I was allowed to keep the pay I started with in my support role when I transitioned to testing.”

QualityLogic: Ok, so you’re talking about more than just a shift in vocabulary? This is a shift in perception?

Melissa: “Artists, Bakers, Chefs…they all have technical skills. We can’t devalue someone’s contribution based on an arbitrary definition of technical.”

The sword was out! Mel told me that by defining some tech positions as technical and others as not, it opens the door to exclusion and inequality.

When a company considers testers non-technical, they are inadvertently making the decision to not train people, and to disable/disconnect them from learning possibilities. I’ve overheard managers, while working with clients as a consultant, say that their testers were not technical and if they wanted to learn automation, they would have already. Never mind that they basically eliminated a chance for them to learn by not offering training in the first place.

In short, Mel said, viewing testing as non-technical or telling a tester they’re non-technical is discrimination, and affects everything from pay rate to self-value. “Do not accept anyone calling you non-technical. Do not accept a narrative someone else has created for you”; the closing line of her 99 Second talk at TestBash Philly 2017.

QualityLogic: What’s one way to advocate for the value of test…and the technical expertise of the tester?

Melissa: Mel prefers to “move pebbles, not mountains” This means taking the small steps that add up to big change. “We need to change the visibility of testing,” Mel told me, “and one way to do that is by being our own advocates.”

Mel put this idea into practice early in her test career. As a ‘visibility’ experiment, she shared her test notes, screenshots and workflow videos to the development team’s storyboard. By submitting a body of proof, her social engineering plan turned the organizational tides. Communicating what she did as a tester helped create value with all the project stakeholders. Mel also found she’d created organizational “test allies”, others who would also act as advocates for test. And so, a few pebbles moved a mountain.

QualityLogic: Why do you think there’s a divide between testers and developers?

Melissa: “Developers think about the solution, testers think about the problem.” This often feels like an us-vs-them approach that becomes more complicated as testers tend to fall into an “empathy trap”. Testers wants everyone to be happy, from the customer to the developers. But, Mel pointed out, testing is inherently destructive.

Communicating test results relies on good communication skills and a shared understanding of value to produce the best results. The best way to bridge the gap is to recognize where blind spots in communication are happening. Software is more than just writing code, it’s a network of interconnected players who need to talk about (and often show) the value of their contributions.

QualityLogic: What advice would you give the C-Suite?

Melissa: Not hesitating a bit, Mel said, “Pay equally, publish the pay rates, and level the playing field.” For a summary of gender inequity in pay rates, Mel points to, a global nonprofit whose mission is to build workplaces that work for women. This page at Catalyst details pay differences by gender, age, and skin-tone around the globe.

“This [gender pay discrepancy] is a historical fact. Women and minorities are paid less. Women also undervalue themselves and don’t ask for what they are worth on a regular basis. They are less likely to negotiate a higher pay rate or talk about pay needs when it comes to raises and evaluations. Women and minorities are also more harshly judged in evaluations causing them to see less improvement in pay over time. A lot of women have been taught not to stick up for themselves (I believe it’s the same for minorities, but I hesitate to speak from that point of view as I am not a person of color and do not have that perspective). I believe this is changing, but change is slow.”

QualityLogic: What advice would you give to an underrepresented person in test?

Melissa: “Women and minorities tend to undervalue themselves, then, the company takes advantage of this,” Mel said.

Women and minorities are more likely to enter into IT through testing and support roles. If these roles are seen as non-technical, it’s not far-fetched to believe that the moniker of ‘non-technical’ is a subtle discrimination.

“If I’m in a technical realm and I’m considered non-technical, I’m a lesser valued employee/person. Once I’m considered ‘technical’ and able to handle technical situations and problem solving, my experience completely changes. I get more access to tools, tech stacks, developers, ideas, opinions, and working styles. In some instances – my title didn’t change; my own skill set didn’t really change – the culture or the company culture changed around me to allow me to be considered ‘technical’.”

“I was never considered non-technical in my customer support roles. It baffled me that I was considered a non-technical employee in my testing role when many of the same skills I used in customer support were vital to my testing position. That is where I call bullshit every time.”

Posted on

Women in Test: Jenny Bramble

Women in Software Test - Women in Technology

The Women in Test Series

Women in Test is a series focused on women in the software testing world, and the ways they advocate for inclusivity and diversity in the discipline. The software industry has traditionally been a homogeneous field, but as we know, times are changing. Leadership and employees alike have begun to recognize the value of diversity, even as they often struggle to practice it. Speaking to an industry undergoing intense change in its makeup, our series participants share their stories, thoughts and advice as advocates for inclusion and diversity in test.

An Interview with Jenny Bramble

Jenny Bramble’s Twitter tagline says she “Tests code, pets cats, speaks at events, drinks coffee, goes to shows, wears boots, acts theatrical, does things.” This may be the understatement of the year, because as you’re about to read, Jenny does things…a LOT of things!

Jenny’s done it all, from support to DevOps to test. Currently, Jenny calls WillowTree Apps home where she’s a software test engineer specializing in Risk-Based Testing (RBT). Jenny is also a passionate advocate for diversity, both in test and in the world.

My introduction to Jenny’s work came while researching a piece on RBT. I found this YouTube video showcasing her knowledge of risk-based testing, her personality, and her cat Dante. It seems felines are the perfect creature to help illustrate risk-based thinking in software!

I spent a fun hour with Jenny and Dante (and Dante’s new kitten, Dax) via Hangout. We talked about RBT and the great things that happen when we include people who interface differently with the world in our software testing process.

The day before Jenny and I chatted the Twitter test community blew up over a DZone tweet. DZone, “one of the world’s largest online communities and leading publisher of knowledge resources for software developers.” (their words, not mine) had just tweeted a Top 10 list of ‘Test Influencers’ who were all men, and 90% white.

Jenny called them out, because she does things, right?

DZone’s response? “We missed some great people on this list, and we apologize for that. While we believe everyone in our original post is worthy of being on this list, we’d like to make this an opportunity to hear from technical experts like yourselves to build more inclusive lists in the future.”

Unfortunately, DZone’s response left me a little underwhelmed, so I decided to start our chat by getting Jenny’s take on their response.

QualityLogic: What’s up with DZone’s ‘Top Automated Testing Influencers’ list? What do you think of their response?

Jenny: Members of the Twitter test scene kicked the list around on Slack and realized no one could find a common thread pulling the ‘Top 10” together. Other than being men, these weren’t prolific speakers, writers, or tweeters. They were simply men in test.

“We were just talking about it on Slack and couldn’t figure out why they picked those humans. I am not sure anyone asked what they actually used to compile the list. Which, you know, goes to show these things are dang arbitrary anyway. Really, they had no good answer. If a respected voice in testing is going to put something like this on social media, then it needs to represent the true test world.”

Jenny told me she sees a huge amount of diversity in the audience at conferences like TestBash and AgileDays and wanted to make sure DZone recognized how their list promotes exclusion. She also pointed out that lists like these often represent the unconscious bias of the list maker.

QualityLogic: What does that mean, the unconscious bias of the list maker? And, how can we begin to identify our biases?

Jenny: We tend to seek out people we identify with. This feels true, right? We can be uncomfortable around people who are different than us, and usually feel more comfortable around people with similar interests, ideologies, and appearances. It’s part of being human.

Jenny bases her advocacy on this idea, that the trick to making diversity comfortable is to first understand we humans do this thing, then make a conscious effort to embrace the differences. Only then can we see the benefits of a non-homogenized world, especially in software testing.

QualityLogic: What value does diversity in test bring?

Jenny: “Users are all different, and we need to represent and evaluate from these differences to create successful products.”

Jenny pointed out that factors like socioeconomic status, education level, personal experience, brilliantly colored hair or a love of funky boots, in addition to skin hue, gender presentation or differently-abled bodies are all ways a person can be diverse. It’s up to the software tester to represent this spectrum.

Jenny leverages her work in risk-based testing to help her understand and advocate for the value of diversity.

“People see risk differently”, she said, “this is why diversity, the people who interface differently, are so important.”

In risk-based testing we need to leverage varied viewpoints to evaluate and prioritize software risks. If we do this, says Jenny, then we’re able to create products to “address the needs of the Silo of Humanity” rather than silos of user groups. (Side note: Jenny’s wit shines when she’s speaking to her passions. Silo of Humanity is prime Jenny, an impromptu phrase that prodded me to think about my unconscious bases!)

QualityLogic: So, diversity of ‘interface’ causes users to see risk differently? Why does this matter?

Jenny: Jenny offered an example of a music app she tested. For some reason, there were three different ways to download music, and this caused conflicts that broke the app. The thing was, no one realized the third way existed, until Jenny ran UX tests. Lo and behold, they found a user who took the road less traveled!

“You never know what path a user finds intuitive, but you have to anticipate the risk they’ll find a different path. What will that path mean for functionality? Evaluate it, does it matter? Yes? Then be like a cowboy.”

QualityLogic: Like a cowboy? Jenny, you’re going to have to explain this…

Jenny: Like a cowboy herding cattle.

“You’re not trying to get your users to walk single file, you’re trying to get the majority of them to head in the same direction.”

If you’ve ever seen a cattle drive you know Jenny speaks truth. Cattle aren’t hemmed in single file. Instead they’re all meandering in the same general direction and will eventually get where they’re going. Not that Jenny thinks users are cattle, oh no! Rather, it’s an apt metaphor for reducing the risk your user will find, and take, the path less traveled. That would be more like herding cats.

QualityLogic: What mentoring advice do you give underrepresented individuals?

Jenny: “One great way to know whether a company is giving lip service or truly practices diversity is to look at the interview panel. It is usually comprised of ‘genres’ of people they want to hire. Is there diversity in this organizational presentation? If not, it may mean they don’t understand the value of diverse thought. They may be seeking presentation related skills, rather than performance related.”

At one point, Jenny was the only female with non-traditional attire on a panel interviewing a developer. Afterwards, this person told Jenny she was the only reason they accepted the position. They had concerns that the homogeneity of the rest of the panel was representative of the company culture but decided that Jenny’s presence balanced the scales. A happy ending for the interviewee, and a great tip for judging company culture.

One thing Jenny wants to eliminate for underrepresented groups in software testing is imposter syndrome. We’ve all probably felt this before, but it can be a huge barrier for those who interface differently. Jenny suggests keeping a notebook on your desk and noting each time you have a growth interaction, did a new thing, mentored someone or got mentored, asked an insightful question, got or gave constructive feedback, or had a positive social interaction.

Keeping a journal detailing personal growth moments and social interactions will help “build an accurate picture of how others see you. Neither your picture or their picture are actually accurate–the truth is somewhere in the middle. But, by seeing their picture and your picture, you can often find a common ground that is more true than either separately.

Jenny pointed out an added benefit of keeping a growth journal, they’re handy tools when it’s time to “articulate your value and your skills to others and be a means to clarify your worth to yourself. Then, you can act with confidence!”

Posted on

Women in Test: Ash Coleman

The Women in Test Series

Women in Test is a series focused on women in the software testing world, and the ways they advocate for inclusivity and diversity in the discipline. The software industry has traditionally been a homogeneous field, but as we know, times are changing. Leadership and employees alike have begun to recognize the value of diversity, even as they often struggle to practice it. Speaking to an industry undergoing intense change in its makeup, our series participants share their stories, thoughts and advice as advocates for inclusion and diversity in test.

An Interview with Ash Coleman

Ash Coleman is insightful, ambitious, and animated. A skilled QA engineer, tester, manager, and consultant who led a past life as a chef and now presents internationally on the topic of diversity in software test. In short, Ash is the proverbial inspirational story we want our children to see. Meeting via Hangout, Ash served up her take on diversity in software testing, how to use resilient and inclusive language, and how she uses her experience to help others transform. Along the way I met Bijou, Ash’s feline companion, and felt right at home as she gently pried my eyes open to the real concerns faced by real people in the software testing world.

QualityLogic: Let’s start with your background. For those who aren’t familiar with your work, how did you get into testing?

Ash: Almost 10 years ago I headed to New York City hoping to work my way to the top of New York’s food scene. I quickly found that a person with years of experience in the kitchen elsewhere could spend years bussing tables while waiting for a chance to prove themselves in a NY kitchen. Rather than demoting myself to satisfy the hierarchy of the New York food scene, I decided to forge another career path.

Ultimately, I started working a customer service position for a software company. There, management quickly recognized that my natural curiosity and probing questions would translate perfectly to a test role. In test, I found my experience in the culinary arts to be an excellent platform for my new role.

“Hospitality is 100% about human satisfaction. This is true of test, as well.”

Since that serendipitous day, Ash has become a skilled QA engineer and has founded the consulting firm, QualityINClusive, dedicated to organizational and individual transformation through diversity coaching for hire.

QualityLogic: Tell us more about QualityINClusive. What led you to start consulting?

Ash: When confronted with complacency around the issue of diversity, I will often ask “Why?” As in, why isn’t diversity important? This one simple question often results in an “ah-ha” moment. Asking people to articulate a response to their devaluation of diversity is an eye-opener. They start examining their unconscious bias.

Of course, getting an honest answer requires people, especially people with power in an organization, “to understand and be comfortable with inclusive and resilient language”. According to Ash, this means that we ideate our biases in response to the type of language used. We infer who is included, and who is excluded, from any given situation through the language used to describe it.

This idea, that the language we use matters, is especially important in software testing. For example, it’s not uncommon to see tech recruiting posts that advertise pub and party nights as a ‘perk’ of employment.

This innocuous sounding statement sets the stage for some prospective applicants to feel excluded. Perhaps they’re gluten-intolerant, prefer wine, have a family to go home to, are in recovery, or have chosen not to imbibe for one of many other reasons.

Crafting language that describes your company’s persona as a ‘pub culture’ excludes those for whom the ‘pub culture’ paradigm doesn’t fit. Further, use of such language creates a space where those humans who don’t see themselves as a ‘fit’ are likely to eliminate themselves from consideration in the first place. This may well cause your company to miss out on highly qualified and valuable employees.

The abstract from Ash’s Agile2017 workshop, “The Things We Don’t Say, How Biased Language Crafts Culture” lays it out like this:

“While we claim to support the evolution of resilient autonomous teams, a desire to define the culture in explicit marketable terms can create a barrier to entry. Are you really creating culture and fostering an environment for agility, or are you creating exclusive spaces? A lot can be derived from the specific words you use to describe your team, culture and collaboration schemes.”

Next, Ash brought up the retention rate of underrepresented people in test. She pointed to Tech Leavers [1], a 2017 first of its kind study, that found workplace culture drives turnover, and significantly affects the retention of underrepresented groups—while costing the tech industry more than $16 billion each year.

While this may feel like an obvious statement (although the dollar impact is staggering), what isn’t so obvious is what that number means for the people whose lives are impacted by organizational culture. Company profits are one thing. Personal well-being is entirely another. Here’s a short-list of Tech Leavers results you might be surprised by:

  • LGBT employees were the most likely to be bullied (20%) and experience public humiliation (24%) and 64% said it contributed to their decision to leave.
  • People from underrepresented groups, such as African Americans, Latinos and Native Americans, were most likely to leave due to unfairness (40%).
  • People of color were significantly more likely to cite unfairness as a reason for leaving than white and Asian women (36% versus 28%).
  • Nearly one-quarter of persons of color experienced stereotyping, twice the rate of white and Asian persons.
  • Nearly one-third of persons of color were passed over for promotion, more than any other group.

You may be wondering, as I did, how these statistics caused a $16 billion loss in tech industries as cited by Tech Leavers. Ash pointed me to the book Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. Accelerate cites research showing that “teams with more diversity with regard to gender or underrepresented minorities are smarter (Rock and Grant 2016), achieve better team performance (Deloitte 2013), and achieve better business outcomes (Hunt et al. 2015).”

Echoing Ash’s call for inclusion as the foundation for diversity, Accelerate’s authors state “It is also important to note that diversity is not enough. Teams and organizations must also be inclusive. An inclusive organization is one where “all organizational members feel welcome and valued for who they are and what they ’bring to the table.’ All stakeholders share a high sense of belonging and fulfilled mutual purpose” (Smith and Lindsay 2014) [2]

The intersection of these two topics, the way language shapes culture and the abysmal retention rate for those outside the homogeneous norm in test, is the genesis of QualityINClusive.

QualityLogic: Your keynote at Agile Testing Days with Keith Klain was titled “Culture Is More Than A Mindset.” What does it mean for culture to be more than a mindset?

Ash: The way we craft our language, both what is said and what isn’t said, often creates an exclusionary atmosphere. And, once we recognize the ways in which we’ve used language to create exclusionary spaces, we can find ourselves in a mightily uncomfortable place. In that uncomfortable space it helps to remember that not everything worth doing is easy.

“Language used frivolously, without thought and attention, can disenfranchise a team member or reinforce a stereotype. The ‘ask’ of diversity is to be cognizant of the words you’re using. The things you say cannot be redacted and can be harmful to others. For the busy executive trying to staff their company this takes more time, more energy, all those things no one thinks they have time for. But using resilient and inclusive language pays off.”

Ash believes a better way to use language is to recognize its power to shape culture. “Inclusive language starts with recognizing that the way we speak either creates a divide or bridges a chasm”. This means we understand inclusive language as an awareness of how we shape our surroundings with words. And, we understand resilient language to be the use of words that clearly express “what we stand by, who we are, and what we expect.”

QualityLogic: You mentioned resilient language? What’s that?

Ash: Resilient language is simply language that is steadfast and transparent. Resiliency means using language that expresses what we stand by, who we are, and what we expect and used traditional job postings as an example.

Too often job postings are intentionally vague. They might include a laundry list of nice-to-have but not necessary skills or be worded in a way that offers no solid metric to judge applicant suitability by. These postings then become more a means of screening for those who fit the corporate mold than a way of leveraging applicants’ current and beneficial skill-sets.

Essentially, this kind of language creates the opportunity for personal or organizational bias—instead of knowledge or potential— to be the deciding factor in hiring.

QualityLogic: How do you coach an HR manager or talent recruiter in crafting inclusive job postings?

Ash: Craft the job posting to convey exactly what is expected and include only what’s necessary for the position This strategy benefits both applicant and management.

Resilient language allows an applicant to use job criteria to map out their career path, instead of wondering if they have enough skill (in the case of laundry list job postings) to apply. This also allows management to look at an applicant’s potential and current skill sets, rather than get bogged down in hiring to meet diversity quota or not hiring because the applicant doesn’t fit the ‘company persona’.

QualityLogic: Why are organizations still struggling with this shift towards resilient and inclusive language? Do some companies seem to give ‘lip service’ to diversity?

Ash: Change is hard. Organizational transformation is hard. All involved have to be comfortable with using resilient language, or they’re likely to disengage from the conversation (be it a job posting, and interview, or an underrepresented person asking for a promotion.) As humans, we can find it difficult to engage in new and often uncomfortable spaces. This is where QualityINClusive steps in. We can do this. Change is hard, but the outcome outweighs the difficulty.

By outcome, she’s referring to both bottom line benefits1, and to the individual satisfaction of a career well-lived. Ash makes her theory of transformation coaching resilient with this statement:

“Diversity isn’t about replacing people at the table, it’s about building a bigger table. The goal of diversity isn’t to exclude the previously included, it’s to expand resources through acknowledging the fact that diversity of people = diversity of thought.”

When truly practiced, this concept alone advantages SQA teams to produce superior products.

QualityLogic: How so? How does diversity benefit an organization’s software testing efforts?

Ash: Diversity of employees generates increased empathy in software testing.

Ash asks us to think of this scenario: Imagine you’re testing a financial app. You’re a middle-aged, middle-class white man with a graduate degree in computer science and an undergrad in finance. You’ve never had to worry about your bank balance, let alone how to feed your children. Now, imagine your user is a single mother paying her way through school while working two jobs. Her goal may be to track every penny spent and build her family’s future.

“Whatever your perspective,” Ash said, “it isn’t the same as anyone else’s.”

Ash’s scenario drives home the need to utilize diversity to better serve our customers.

QualityLogic: We’ve been talking about how diversity benefits organizations, how can people empower themselves to promote that diversity?

Ash: My plan with QualityINClusive is to focus on coaching women, and other underrepresented groups in test, to “leverage their package and take a place at the table to create organizational change.

If the statistics from Tech Leavers are any indication, Ash’s work in advocacy and coaching is just the thing needed to shape the testing workplace into “a place for people to thrive.”

Ash’s message to company executives is simple, yet powerful.

“We can do this! It may be hard to venture through the discomfort of the unknown, but the outcome outweighs the difficulty.”

QualityLogic: If you could say one thing to C-suite execs, what would it be?

Ash: “It’s still not a pipeline issue!”

Frankly, this is my favorite Ash Coleman quote, and a powerful push-back to organizations who claim they lack diversity because it’s hard to find.

“There are plenty of diverse people to include, it’s their [the organizations] mindset that’s impeding an inclusive culture. There is no pipeline here, it’s really just a matter of perspective.”


[1] Kapor Center. (2018). Tech Leavers. [online] Available at:
[2] N. Forgson PhD, J. Humble, G. Kim. (2018). Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press; 1st edition


1 Studies analyzing the impact of diversity on the bottom line tend to focus on binary gender. There’s plenty of research linking the presence of women in leadership positions to higher financial performance (McGregor 2014), stock market performance (Covert July 2014), and hedge fund returns (Covert January 2014). Furthermore, a study conducted by Anita Woolley and Thomas W. Malone measured group intelligence and found that teams with more women tended to fall above average on the collective intelligence scale (Woolley and Malone 2011)

Posted on

Maximizing Load and Performance Testing Efficiency in an Agile Environment

A given of the Waterfall development methodology was that load and performance testing were done at the end of the development cycle. Both required the entire system to be integrated and its functionality verified in order to produce meaningful results.

Issues in performance times for a feature or glitches produced by operational degradation at less than the designed maximum number of users would start test/fix cycling until operational performance met requirements. These activities were considered to be part and parcel of User Acceptance Testing (UAT). But, that was then, and this is now. Agile has changed this game substantially.

Why Load and Performance Testing is Different in Agile

The reduction of the development release cycle from months to a week or two has pushed the demand for quick performance test verificationLoad and performance testing in agile to the wall. If it is to be done at all, there has to be a highly automated, quick turn-around testing process that puts a minimum stretch on the release schedule. And notice the first part of the previous sentence.

With the endless delivery crunch of Agile, performance and, especially, load testing are often pushed right off the end of that schedule and don’t happen at all. This is a prescription for disaster on the user review front.

In tandem with creating both automated and manual functional tests while those functions are being committed to code, load and performance tests must be produced as well.

Load and Performance Testing Problems in the Agile Environment

The load test suite is typically created early in the product’s life cycle when its user interface is first operational. It will need maintenance and special situations where the user interface cannot be used must be supported during Agile sprints. This speaks specifically to exercising third party APIs at the levels of expected use.

Performance is typically dependent on the exchange of data between code modules of various descriptions. But to test it, all those interfaces have to be up and running and a body of use-relevant test data has to be ready to exercise them. Once again, the test of performance gets pushed back because the elements necessary to support it are not going to be available until the end of the development effort. This is especially true in this era of Software as a Service (SaaS).

The rapid spread of the SaaS concept has pushed the issue of unavailable system functions back into development as well.

One organization QualityLogic worked with estimated that some 40 percent of the development group’s working time was spent in trying to find work-arounds for missing software services. Building the code for a function was dependent on access to code that was being built for use by several other functions.

Developers began writing what were called ‘stubs’ to provide fixed responses that somewhat simulated the performance of the missing service code. This worked well enough that it grew into the concept and reality of Service Virtualization (SV).

Service Virtualization, Help or Diversion?

Service Virtualization is used to help enterprises reduce their code development inter-dependencies. Neither the testers nor the developers have time to create an endless array of micro-environment circumstances. To quote Steve Anderson at Clutch,

“You can instead create the service virtualization environments that are going to mimic the calls going back and forth. And you just treat them as another variation of systematic elements and micro-services.”

Agile teams can use Service Virtualization to leverage virtual services instead of pooling in resources from production that could be disrupted by the test process itself. This gives a boost to testing (especially load and performance) and development when key components are not available from the new system architecture. SV emulates the behavior of code-based services that will eventually be present in the final production system. As Agile requirements evolve, these virtual services can evolve with them.

A schedule-ridden development manager might balk at the idea of diverting precious resources to creating SV assets but that would be short-sighted.

SV service emulations can become an essential part of a reusable test and development infrastructure that greatly reduces time lost waiting for a service to be delivered by another team. Service Virtualization can make software assets available to everyone on-demand.

When It All Works

Service Virtualization can support robust application development by creating virtual SaaS assets that can be used to test system performance, load management and operational functionality in a near real time scenario. Third-party APIs are a high risk area for performance defects and load issues. This makes it especially useful for identifying and characterizing defects at the API layer where access to the actual service is either spotty or non-existent. Leverage service virtualization to save time, cost and escaped issues in the released system.

Enterprises are using Service Oriented Architecture (SOA) and Software as a Service (SaaS) to create robust applications and shorter times-to-market. Service Virtualization can make these assets available anytime, anywhere.

Load & Performance Testing Services

Posted on

Outsourced Software QA Testing – 4 Rules to Guide Your Management Strategy

Whether you are working as part of an outsourced team or you’re in-house managing an outsourced resource there is one aspect that rings true about both experiences. That is, poorly defined expectations for outsourced partners will derail your project. And, the communication skills of all parties involved will make or break your venture.

A Case Study in Poor Outsourcing Management

I was part of an outsourced web design team tasked with migrating online content for an international client. Expectations were set, but they were vague. We agreed to deliver XYZ but the client failed to communicate what their vision of success looked like and how we would be fitting into their process.

The client gave us a drop-dead date, access to their documentation and proprietary CMS and not much else. Communications were bottle-necked by the client’s Project Manager who rarely bothered to read email, and when they did, never seemed to fully credit the validity of the team’s questions, concerns, or suggestions. Expectations like development processes and deadlines fluxed constantly. Resulting in a constant scramble to understand where and how we could best help our partner.Outsourced Software Testing

These issues turned what should have been a straightforward project into a nightmare. As the drop-dead date rolled around it was clear the wheels had fallen off the project and the engine wasn’t far behind. Everyone was forced to scramble for solutions. We worked weeks of overtime to fix what shouldn’t have happened in the first place—a project off the rails—due to communication missteps and unclear expectations.

Despite our team’s willingness to communicate, set expectations, or simply find out which project aspect was the focus at any one time, the client’s lack of planning and trust led to lack of communication, which led to failure. This conflict broke the migration and our team’s spirits.

This tale of outsourced woe is cautionary, and frankly, an outlier. More often, I’ve enjoyed collaboration as both the outsourcer and outsourcee to achieve product innovation, align with organizational strategy, allow each partner to focus on core competencies, and to leverage core competencies into product assets. The benefits of outsourcing software testing are enormous, but finessing the details requires strategy.

Why Look Into Outsourced Software QA Testing?

Today’s digital leaders face rapid shifts as innovations in IoT, VR/AR, Big Data, and SaaS drive market evolution and increase user expectations. These changes, combined with the large variety of digital interfaces, have created an environment where knowledge silos don’t work and trust underlies productive outsourcing partnerships. The decision to outsource software testing has shifted from a cost-driven strategy to one that benefits by strategically leveraging partnerships. For a more detailed look the business benefits of outsourced qa testing, read this article.

Outsourced Relationship Management

You’ve picked a software testing company you’d like to partner with but the decision to outsource doesn’t end there. Outsourced relationships are complex. They rarely flow from contract to process to finished product in a linear manner.

Communication and expectations are the key to any successful working relationship but become especially important when you are managing an outsourcing partner. We know that software test management is a dynamic practice. Developing and testing software asks all parties to adapt to shifting input ranging from device configurations to user feedback. Give your outsourced test partner clear direction on project goals and partner roles to eliminate confusion and enable them to add the business value you hired them for.

Outsourced Software QA Testing – 4 Rules to Guide Your Management Strategy

Choosing to outsource your software testing is a difficult choice. One that is far too often a made based on cost cutting or a frantic attempt to meet production deadlines. Outsourcing should be a strategic decision. One that is made with care and with a solid plan in place for mutual goals, deliverables, and communication.

1. Fully Develop, Then Share Your Action Plan

Think of your management plan as a roadmap to how the relationship can achieve best value for all parties. The first step to create this plan should be a thorough review of what’s needed and why. Involve your stakeholders, fine-tune the details, then create a document detailing the what and why. This document should have the objectives required of the outsourced partner, what constitutes contractual ‘good behavior’ from both parties, and how disputes will be handled. Use clear and concise language for your action plan and make it available to all parties.

2. Eliminate Ambiguity

Ambiguity sucks. Unclear or imprecise directions have sent more than one team off on an unproductive tangent. Stop ambiguity before it starts by defining:

  • Who will do what? This eliminates the potential for duplication of effort.Open and Honest Communication to outsourcing partner
  • Who is responsible? Mapping out specifics such as who will create test scenarios, how bug-reporting will flow, and what the expected performance markers are early in the process will save time and trouble down the road.

3. Create Opportunities for Open and Honest Communication

As with any relationship, it can be difficult to gauge one’s place in the larger picture without open and honest communication. The first part of effective communication with your outsourced partner is to simply remember to communicate! This builds rapport, trust, and allows you to catch issues as they arise. Often, a 10-minute video chat is all it takes to clarify who is doing what, what has been done, and what is needed to move forward.

Keep in mind that open and honest also means giving and receiving feedback. It could be that your outsourced software test company isn’t meeting your expectations. Be forthright yet objective with this discussion, bring examples, and be ready to detail the improvements you’d like to see. Of course, feedback can flow the other way. It may be that your outsourced partner has the advantage of a fresh perspective on your business processes. Be willing to consider suggestions for process improvements. After all, they’re vested in seeing you succeed.

4. Be Realistic

Your outsourced software QA team are humans, just like your internal team. The outsourced company is a company just like yours. It’s essential you understand the abilities and restrictions of your outsourced software QA partner. You’re asking them to commit to your schedule, your project, and your success…as defined by you. There’s rarely a good time to break out the whip, but when it happens make sure you haven’t driven your outsourced partner into over-promising through ambiguous expectations or unclear communication.

Posted on

Test Script Maintenance is Mission Critical for Your Test Automation Program

What happens to old tests? In short, they don’t just fade away. A major challenge of test coverage management is making sure that system verification test suites don’t get clogged with redundant or worse, obsolete tests.

Automated Test Script Maintenance
Automated tests tend to raft up like old clothes stuffed into the back of a large closet.

Manual test operations tend to rapidly cull test cases due to the continuous review they receive from testers. But automated tests tend to raft up like old clothes stuffed into the back of a large closet.

The relatively recent advent of test automation tools that generate test scripts by recording processes and GUI activity has aggravated the issue of test script management. Advertised as engines for quick, accurate test script generation, the code produced by these tools usually must be altered before useful testing can occur.

This tends to consume exactly the maintenance resources they are intended to free up. If used extensively, test recorders can cause a great many marginally useful and very inflexible test scripts to be amassed in a very short period.

Take the output of an auto-generating test tool, combine it with the test scripts created specifically by code developers, then add in tests written by QA engineers to plug the gaps and the body of test scripts can get out of hand quickly. Now salt heavily with functional changes brought about by defect fixes with undesired feature interactions and you have a test array that rapidly goes from unmanageable to unusable.

Automated Test Script Maintenance Is Crucial

A great deal of consideration and concern is applied to the questions of which tests should be automated and which shouldn’t. Test automation works well with those functions and features that are least subject to change, but this has its down side as well. Test automation scripts written for stable code are not reviewed very often, if at all.

This makes them subject to unexpected problems when an underlying code function is changed, modifying the operation of a tested feature. The code changed and the test script didn’t and now it complains with false positives. These erroneous flags clog the results review process and require continuous examination to be sure that the flagged errors are really from bogus tests.

If this goes on long enough, the entire automated test process will be called into question. Management will begin to see test automation as an expensive waste of time and resources instead of the useful product verification tool that it is.

This is why test script maintenance must be included in the initial plans for any test automation project. The cost in resources and schedule time should be part of the plan and be wrapped into the velocity expectations for each Agile sprint as well.

Test script maintenance must be conducted from a clearly explained plan that has the buy-in of management, development and quality, a plan that keeps the script array informative and vital.

Keeping the Test Array Fresh

An effective automated test script maintenance strategy is key to preserving the validity of the test script array. Such a plan grows out of carefully documenting the inception and retirement criteria for automated as well as manual tests. This will encourage a disciplined approach to maintaining the entire system validation process by tying a thorough review of scripts for specific features to a parallel review of their manual tests.

Test Creation Criteria

  • Test creation should be predicated on clear criteria for why it is necessary and when it is to be implemented. At the other end of the process, formal test retirement criteria should govern a regular audit of the existing tests. With the pressures to create them, test retirement becomes critical. A firmly grounded retirement plan keeps test suites from growing out of control.

Regression Test Control

  • Regression tests are another source of rapid test suite growth. Retirement of obsolete scripts will help keep regression suites under control. Understand that full regression testing and the data sets it entails may simply take longer than the development schedules permit. “Regression test everything” sounds impressive but it doesn’t scale and can undermine your automated testing effort.

Test Script Maintenance Saves Time

New functionality and the business case that it supports are the basis for test inception criteria. When new code creates test coverage gaps, an immediate decision is required to expand the test array or log the gap for future action.

When functionality and/or the code that supports it changes, the tests that verify that functionality need to be rewritten or retired. Test script fragility tends to vary inversely with the manual effort put into writing it. But, that said, any change to a UI screen or the workflow to implement a business process can break all the scripts associated with it.

Broken tests cost the one resource that is always in shortest supply, time. Every release verification should push out any non-functional test scripts that it reveals. The code release process itself should incorporate and depend on this activity.

Posted on

Software QA in an Agile World: Achieving Agile Testing


Software Quality Assurance (SQA) — a planned and systematic approach to the evaluation of the quality of and adherence to software product standards, processes, and procedures.

Agile development is the concept of churning out a continuous stream of product releases, most of which are expected to enter production. QA being ‘planned and systematic’ as stated above flies in direct contradiction to the prime motivations of Agile. Two of the main challenges of testing and QA management under the Agile methodology are:

  • Agile teams are supposed to be cross-functional and self-organizing which means that there is to be a minimal level of specialization among the members. Like a start-up company, members of an Agile team should all be able to do anything required to get the product out.
  • The Agile methodology is based on incremental development. So, every sprint creates a deliverable version that is a release candidate. This doesn’t make testing go away, it means that testing must be just as continuous as development through the entire life-cycle. Nor does regression testing disappear from the agenda. If anything, it becomes even more intense and vies for QA resources right alongside functionality verification.

So if QA is so antithetical to Agile, what is the point of having a QA team in the first place? Why not simply have the developers test their code as they create it? After all, they wrote it so they should have the best grasp of both the code and the functions it is intended to provide. To quote the bard, ‘therein lies the rub.’

Pushing testing responsibilities onto the developers does not make them accomplished testers. Instead, the lack of well integrated QA test personnel makes for frazzled, overworked developers. Code development and code testing require different mindsets and very different approaches.

The developer knows what the code is supposed to do and approaches testing it to demonstrate that it does exactly that. The tester approaches the code from the expectation that it won’t do what it is supposed to and looks to try every possible way to prove that it doesn’t.

It is no leap of imagination to realize that this tends to put developers and testers at odds with each other to speak well of the situation. Hold that thought for a moment and then try to imagine compressing both these perspectives into one person.

What Does Agile Testing Really Look Like?

Before Agile, there was the Waterfall methodology. Code updates were often referred to as being “thrown over the wall” to QA for testing. This phrase embodies the perspective of the day that prescribed a complete separation of development and testing. Engineering and QA were separate, divided functions and never the twain should meet. While this worked for a months-long development cycle, it is totally impractical for a days-long sprint cycle.

Agile testing - Agile developmentAgile requires that testing and development proceed in parallel and, as much as possible, that they overlap each other. This means that developers take on white box module testing from their perspective of code awareness to see that all the observable workings of each code module function as that developer expects. It also means that QA oriented testers take over with black box testing just as soon as service stubs and minimal system integration of the new code allow.

There is no longer a sharp dividing line between development and testing that supports or even allows the ‘throw it over the wall’ attitude. QA personnel begin to work with the new code as soon as it is white box tested and then turn around and work with development to help generate test automation scripts that will follow the new code through its release into production.

Note that development and testing are working hand-in-glove with each other. Development is striving to get the maximum conformance of the functionality to the specified new feature or bug fix. Quality is approaching from the business process side to make sure that development’s translation of the functional specs actually results in what the customer (and by definition product marketing) really wants.

How Do We Get to Agile Testing from Here?

Reducing a complex undertaking to a simple statement, QA has to be merged into the entire Agile SCRUM team. In a bit more detail, SQA must be integrated into:

Product Management

When management is planning product feature additions/changes, QA should be involved in identifying the necessary elements to be specified to development to get the desired results translated into working code. Specifying Agile stories, their scope, their interdependency and the use cases to be satisfied are all prime areas for QA participation as they have spent the most time with the system as a whole and have an innate understanding of its structure.

Sprint Planning

From epics to stories, QA provides oversight of plan viability and QA resource requirements for each effort. Often sprints are planned against development effort points. QA needs to add velocity points for the QA effort as well, especially where that effort may be greater than anyone other than QA realizes.

Sprint Execution

QA works directly with planning to make sure that resources and time are allocated for necessary testing activities. They also assure that there is daily input at the stand-up meetings about QA status and they participate in retrospectives to improve development and test processes.


Agile has very nearly turned documentation into a dirty word. It diverts hours from other ‘productive’ activities and, besides, everything is going to change too quickly to document in any event. Code documentation has devolved to no more than the comments blocks that developers put into their modules. But, when someone has to go back into the code to work on it, QA’s documentation of test cases becomes an invaluable resource to reconstruct what each piece was supposed to do. And user guides are still necessary. The technical writer’s right hand resource is still the QA tester who worked all the way through the project.

User Acceptance Testing

QA acts as the gate keeper when the code exits development and enters the User Acceptance Testing phase. This is where QA stands in for the customer to make sure that features and their controls perform intended tasks and don’t have unexpected glitches (no it’s not a feature). This is also where regression testing must be performed to verify that this sprint’s fixes work without unfortunate side effects and that past sprint’s fixes are still in place.pixelated lion

Lions and Tigers and Software QA Oh My!

If all this has started to sound somewhat overwhelming, consider this possibility. QualityLogic has been working with software development organizations for over 30 years and has seen the rise and organizational angst of dealing with many Agile implementations.

Our test technicians and engineers work directly with our clients’ development organizations to consult on the transformational effort of merging them with QA. In many instances, we have provided a significant portion of the QA staffing required. Those same engineers and techs can work directly with Agile sprints to perform all the merged QA efforts described above and more.