QualityLogic

The power and fatal weakness of software is its ultimate malleability. It can be quickly and easily changed by drastic degrees. This one inescapable aspect would be good cause to abandon software entirely if it weren’t for its extreme power to make our machines do our bidding. In this day and age, software runs nearly everything of any importance in our world.

The elemental issue with software is its ephemeral quality. Physical products have a substance which can be

examined by all our five senses to see that they meet the measure of their requirements and that the requirements themselves make sense. Software has no such constraints.

Software is composed of millions of lines of text that must be written with ultimate precision. The slightest coding mistake can have massive repercussions when the system is used. Back in the 1960s, a satellite launch was turned into a fiery explosion by the absence of a comma in a line of code.

Something so impactful must be carefully designed, but design work consumes time without yielding the appearance of project progress. Numerous ‘software failures’

can be laid at the doorstep of poor or insufficient design. So, given all this, software development should be a rigorous, careful, deliberate, tightly controlled process… right?

Producing Quality Software in an Agile World

The preceding sentence loosely describes what was called the waterfall software development method. Features were described in detail by marketing, turned into designs and code structures by engineering, verified by quality, fixed by engineering, and then released every couple of years. Literally, code releases came at intervals of 12 to 24 months and, even at this pace, defect-free code was a fond desire rather than a reality.

Today, the Agile methodology has won the hearts and minds of modern software development organizations. As commerce moved onto the Internet, web software became the gateway to sales of just about everything. At the very least, a current, eye-catching web site is required to announce the existence of the product or service to its prospective customer base.

This puts marketing on a bubble of making continuous changes to site code to show off new products and sales campaigns. This push to beat the competition to the web is what has propelled Agile into becoming the premier development method. Agile is predicated on releasing operational code every one to two weeks. Reducing the scope and quantity of these changes down to no more than four or five per release is supposed to allow for a series of small feature additions/bug fixes to be propelled into the functioning system code very quickly.

Sounds good doesn’t it?

The reality is that the constant push toward what is called Continuous Release — releasing each change as it is made — has resulted in the abrogation of most of the controls on software development. The intrinsic re-usability of code has led to the creation of what are called frameworks which are simply tool boxes of software modules that can be tied together to perform various common software processes. These frameworks are so useful that development organizations have standardized them, and they have become commercially available development tools.

So, we have a process that focuses tightly on a few system aspects to release incremental changes rapidly using pre-fabricated code building blocks as much as possible. What could go wrong?

Sanity Checks for Quality Software

Design

It is not enough for the code to follow the design, the design itself must be carefully considered and explained to the programmers. It is vital for the product managers to attend the Agile planning meetings to make sure that both development and quality understand what the intention of the new feature is and, especially, how it should work for the user.

Schedule

Market windows may wait for no one, but there still has to be time allocated for the entire development process. Non-negotiable deadlines are the arch enemies of good design, thorough documentation, and careful verification. Being first to market with a product or service that doesn’t work is not a win.

Budget

Software development takes longer and costs more than anyone thinks it should. This should be a given in the planning of any software development project. Low cost estimates or quotes for any project activity or tool set should be instantly suspect. If the project budget is so tight that it has no flexibility, perhaps the project itself needs to be reconsidered.

Documentation

Unfortunately, documentation has become the bane of the Agile methodology. It is typically viewed as useless history, the creation of which consumes time and resources better used for development work. It is still a necessity to leave a trail of what was done and why in order to facilitate the next time that code has to be modified. In particular, defect reports have to be detailed and contain exact descriptions of how to replicate symptoms.

Management

Quality used to be the conduit of information flow between product management and code development. Agile has shifted this responsibility to product management. Those responsible for the final product have to stay on top of getting the design right, communicated to engineering, and properly understood by those testing the results. With Agile, the opportunity for corrective guidance is a very narrow window that management must seize and use to fullest effect.

Verification

A perception exists that QA is an expense rather than a revenue-generating investment. That may be the accounting take but sacrificing testing to meet a market window is a prescription for disaster. With Agile, quality has truly become everyone’s responsibility. As such, it must be planned for in each sprint and any test plan, automated test script or sanity check must be updated in order to maintain its relevance. Development and management must become the direct allies of quality working together to see that the release meets the design, performs the function fully, doesn’t damage other parts of the system, and passes use acceptance as a working addition to the system.

Yes, Virginia, Software Fails But All is Not Lost

Software is a mutable solution to a vast array of challenges. It is easy to create and easy to change. But because of its power, it requires attention to detail like nothing else in our world. Development organizations need to pay close attention to process as they plunge into the break-neck pace of Agile and Continuous Release.

Where unit tests are deeply involved in the functionality of individual code modules, the integration test process is focused on the system as a whole and the complete combination of those modules. It is deeply concerned with how they communicate with each other. As its name implies, integration is also the first place where the system is tested as a functional entity. It is where the code has to live up to global operational requirements that are beyond the simple functionality contained in the module specs.

What Do Integration Tests, Test?

Integration testing is the polar opposite to unit testing. Having said that, they are still complementary and the sins of one can sometimes be redeemed in the other. In addition to verifying module interactivity, it looks at the system from the user interface to the back end’s data management functions. It is the first place where the tests begin to use the code as if it were actually the intended product.

One argument that suggests itself is that, if unit testing is properly performed, why is integration testing necessary at all?

via GIPHY

For starters, the system’s essential modules are typically written by a staff of programmers. Programmers are highly skilled and very creative and they won’t all read the design specs or approach coding features the same way. This can and will lead to disconnects and glitches when their modules must talk to each other. This combines with the fact that the specs themselves are likely to change while the modules are being written, making some aspects of what the programmers wrote in the first place unnecessary.

This is reason enough for integration testing but, as the television commercial exhorts, ‘Wait, there’s more!’ Unit tests work with inter-module communications using software stubs that simulate the rest of the system. These are make-do workarounds and, when the system’s modules have to communicate: data formatting, error trapping, hardware interfaces and third-party service interfaces tend to have unforeseen issues that were not observable at the module level.

Integration testing isn’t the end of the verification process but it is a necessary step in preparing the system code for acceptance testing where it must actually perform its functions for real users and produce the desired and expected results.

Integration Test Insights

Integration test cases should focus on the data interfaces and flows between the code modules of the system. A typical case would test a data entry function in one module to see that it was correctly reflected in the data entered into a database record by another module. When successfully completed, the test has verified the process of accepting data, transmitting it and storing it across a string of code modules.

While completion of the module unit tests should be the gating condition for testing the integration of those modules, unit tests will not catch all the functional issues of their modules. Some of these will only be apparent when the module is exercised in the context of the system itself and that means that the integration tester has to be on the lookout for functional issues. As stated above, integration and unit testing are complementary but not completely independent of each other.

Integration Testing for The Agile Era

Integration testing was once a relatively isolated province of the waterfall development model. The code for a release was assembled into a system and integration tests were performed to see that it worked well enough for verification tests against the marketing product specifications. That was then, this is now.

Agile development to support Continuous Release means that modules are plugged into the system and changed on an as needed basis. This requires that integration testing be done incrementally and that it strikes a balance between the changes to related modules. Often, drivers to call specific functionality and service stubs to simulate other parts of the system are necessary components of integration as well as module testing. Once again, these two parts of the software quality process wind up being more intertwined than their definitions suggest.

Integration Testing Planning and Approach

The planning and approach to integration testing are both crucial. Test plans not only specify what is being tested, they must pointedly call out what is not tested. Listing test exclusions is vital to preparing the next phase of user acceptance testing for where it needs to look most closely. The test environment has to be carefully constructed with its facilities grown from the development system that the code was written on in order to subject the stubs and drivers that were used to re-evaluation by verifying that they stimulate the modules correctly.

Last but far from least, defect documentation is extremely critical for integration. Defects have to fully explain the context in which they were seen, the steps to reproduce them and the impact they have on system operation. These reports must go back to the programmers for bug fix work and forward to user acceptance to facilitate regression tests that make sure the fixes stay in place and work as expected.

Where the Software Meets the Road

Integration tests are designed to catch the defects that would only appear when the code is interacting with real world operating scenarios. They are critical to the success of your software and the success of your users!

Risk-based testing (RBT) is exactly what it sounds like, testing your software based on risk. We know from last week’s blog that circular definitions like this annoy me, so let’s break RBT down to some measurable components.

The first and most obvious question is: what is a ‘risk’? Once we clarify what a risk is in the software environment, we can discuss how to measure it and what value risk-based software testing offers your company’s QA efforts.

Jenny Bramble, risk-based testing guru and accomplished conference speaker, defines software risk as a combination of the impact of something happening (a feature failure), and the probability of that something happening. Here’s an example: You’ve developed an app that requires a login to access. If the login flow has the hiccups, the risk is your user won’t login, therefore impacting user experience and business needs (like revenue generation). Yet the other half of the RBT equation asks: what’s the likelihood of this risk, the login issue, happening? Well, both impact and probability can feel subjective, so let’s back up a step…

From my brief example you can see that defining risk based on impact and probability requires a combination of intuition, experience, and data. Defining risk, and risk-based testing in software quality assurance, may be easier with some background on the concept. So, where did RBT originate?

The Evolution of Risk-Based Testing

RBT has its origins in Risk Management, one of 10 areas in which project managers should hold competency in according to the 2018 International Organization for Standardization’s (ISO) guidelines. Risk management is useful across a broad range of industries, although methods, definitions and goals vary widely according to whether the risk management method is in the context of project management, security, engineering, industrial processes, financial portfolios, actuarial assessments, or public health and safety. Despite industry differences, all risk management strives to identify, evaluate and prioritize risk to allow a coordinated and cost-effective use of resources in controlling or eliminating the risk.

Risk-based testing is then a project management mindset rooted in the drive to understand what risks are going to have the greatest impact on a project and what risks are most likely to happen, then creating a test plan around both. RBT is used in many industries, but it’s especially useful in software because, let’s face it, there’s simply no way to test every function or code aspect for every eventuality. Any experienced PM worth their weight knows a project will have limited resources with a finite amount of time, bodies, attention and expertise. In software QA, focusing those resources where they will optimize ROI is the focus of risk-based testing. As Bramble says: “Even if we had unlimited resources, testing everything is ridiculous, impossible, and a terrible use of resources.”

Benefits of Risk-Based Software Testing

In the software quality assurance realm, risk-based testing presents multiple benefits. RBT first asks a team to define terms. This creates the framework to talk about risk and communicate cross-functionally. When all stakeholders understand terms the same way, dialogue surrounding decisions and needed resources becomes much more clear and concise (miscommunication is also a ‘risk’, right?)

Once everyone’s on the same page with terms, be they management, developers, or external users, it becomes easier to identify the risks each group perceives. This also creates engagement from all the stakeholders in a project. (and we know not having ‘buy-in’ at all levels is risky, too…)

Aligning stakeholders and creating shared terminology is just the first step. Remember that combination of intuition, experience, and data I touched briefly on? These aspects are at play in what I’ll call ‘RBT Phase 2’. In phase 2, defining and prioritizing risks is the goal. Here, the work of making a — very — educated guess about the probability of failure begins. A good place to start the process of identifying risk is by answering the following questions:

What features do users interact with the most?
How do the users interact with those features?
How likely is it the features will be used the ‘wrong’ way?
What part of the code is the most fragile?
Where have you seen failures in the past?
Are there outside influences that impact risk?
Overall, what is most likely to break?

You can see that answering these questions is one-part intuition, one-part experience, and one-part data. That’s a lot of parts for one person to offer input on! This is where phase 1 of RBT, defining terms and engaging stakeholders, pays off. It’s likely different stakeholder groups will have different answers to the above questions, generated through different experiences with either the process or product. Using these different perceptions during risk analysis will ensure that the dreaded ‘intangible risk’ — the risk never identified — doesn’t bomb your project. And, you’ll have project-peace-of-mind that those risks with the highest probability and the greatest impact have been eliminated.

Risk Analysis Tools

There are as many methodologies for defining risk, and creating an RBT strategy, as there are industries practicing risk management. A favorite in the SQA industry is the risk matrix. Creating a risk matrix is straightforward. First, determine a rating system. You could use colors or numbers, happy and sad faces, rainbows and lightning bolts. It doesn’t matter as long as you are consistent. Then, using that combination of gut-feeling, experience, and data, start at the feature level and assign each perceived case a rating.

From there you can complexify the process as much as needed to refine each risk to individual user stories, then develop test cases addressing each risk. The risk matrix can also be used to identify other risks like those arising from code fragility, integrations, and the external environment. If the matrix conveys the right information, it can be as complex or simple as you like.

Risk-based Software Testing is All About Prioritizing

Software, apps, and website all have one thing in common, they’re notorious for breaking when that one user does that one thing. Code is a complex (and often elegant) environmental interplay of man and machine, developer and user, hardware and software wherein testing for every eventuality is impossible. But, you can focus your team’s efforts to high-risk, high-impact areas through risk-based testing.

RBT offers a framework to talk about what needs tested, and what needs prioritized. Then, your team can answer other important questions like: Where do we test to get the most bang for our buck? What bits of code needs the most attention and what needs the least? What tests are a good candidate for automation? And what should we test during each stage of the SDLC?

In short, risk-based software testing provides your team with clarity and focus to derive a thoughtful testing program, leading to a better software product.

For more information on risk-based testing, follow this link.

Businesses, especially software-centric businesses, have dramatically increased the velocity of their operations. Software development itself has gone from bi-annual version updates to weekly releases via the Agile methodology. Managing this eye-watering rate of change has brought every business function to the necessity of monitoring Key Performance Indicators (KPIs) to keep these processes under control.

The principle concept here is the idea of an indicator. A value that, in and of itself, may mean little but has a direct correlation to the operation of an essential business function. As the indicator value changes, it indicates the stability and movement direction of the process. In software development, these indicators are called metrics because they are used to ascribe a measurable value to a process aspect that is subjective enough to make a simple direct numeric evaluation difficult.

Software Quality Metric Controversy

The controversial aspect of metrics is their empirical nature. The metric itself may measure something as apparently irrelevant as the number of lines of code in a module. As long as the module works, who cares how many lines of code it took to create it?

Yet, by tracking the ease of maintenance (hours to fix a bug), the incidence of code defects and their effects on the overall system, it is possible to directly see a correlation between how large a module is and how prone to defect issues it will be. This makes the number of lines of code per module a useful metric.

Software Quality Metrics You Should be Tracking

Their simple, sheer usefulness has made metrics indispensable as both quality and business management tools. In the realm of software development, metrics tend to fall into one of two groups. Detailed code development metrics are direct measurements of the system code itself and provide valuable insights into the design practices of the development group. System metrics are more likely to be of interest to management as they are overall process indicators of how well the business is converting work hours into revenue generating products.

The following will address some important system metrics and leave detailed development metrics for another post.

1. Completed Stories

Agile sprints operate on the basis of ‘stories’ that are actually code modification descriptions. They are written up from the standpoint of the desired effect that the described change will have on system operation. This change is parsed into a code development effort and a QA effort to verify that the story’s intention was accomplished without any disruption to other system facilities. It is also given a ‘velocity’ rating of the necessary resources it will consume.

A vital measure of an Agile SCRUM’s effectiveness is its ability to estimate these required efforts and complete its self-assigned tasks. Stories completed as a percentage of stories attempted in each sprint is a primary metric for this measurement. Better than a simple velocity number, this will tell how well the sprint was organized in terms of taking on those stories that could be accomplished and were actually completed in the assigned time frame.

Management can use the completion percentage to detect bottlenecks in the overall mixture of the development and quality efforts. It points directly to the granularity of stories and the effectiveness of the group working on them.

2. Test Case Successful Completion Rate

Test completion steps back to a broader perspective than story completion in that it looks at the process of verifying the system’s changes and overall operation. Successful test completion means that the code changes are being verified and that the verification process itself is working to validate those changes.

Here management is looking to determine how well the system is faring over time. Are the feature additions/defect corrections proceeding quickly and are they working as desired? A warning sign of excessive sprint velocity is the reduction of the test completion rate and/or a rise in the number of test failures. Of particular concern is test case abandonment indicating that the quality process isn’t keeping up with development.

3. Escape Rate

Microsoft coined the concept term, escape rate. It refers to the number of bugs that are found in each release after it goes to production. This is one of the most critical measurements of a software organization as it talks directly to the efficiency of both the development and quality efforts.

Development must be both rapid and methodical to keep up with Agile release rates. Quality has to stay on top of verifying every code addition and change to keep product and company market reputations intact. A rising escape rate is a blaring warning that one or both of these operations needs immediate management attention.

4. User Reviews

User satisfaction ratings are right next to the escape rate in their importance to managing software development. Escaped defects are sometimes simply ignored by users because the value provided by the product outshines the detriment of having to work around a system flaw. By contrast, some of the most trivial defects will annoy the user base to the point of mass abandonment of the product in which they occur.

Management needs to carefully monitor user reviews for both their overall satisfaction levels and their specific complaints. User reviews are the most effective way of spotting a management beloved ‘feature’ that users consider an annoyance or even a defect.

Align Quality Assurance Metrics with Business Goals

Measuring everything can become an obsession with a detrimental effect on both development and QA. Choose metrics by direct observation of their relevance to your business process. Choose at least four or five to guard against variations in their relevance to your operations. A metric that works only some of the time is worse than none at all. Above all, be transparent with the working staff about what is being measured and why. When they see the positive results gained from these insights they will begin working toward goals that increase what these metrics measure.

Often, requirements-based software testing is defined something like this: “a testing approach in which test cases, conditions and data are derived from requirements.” This sort of circular explanation is annoying — and unhelpful.

What are Software Requirements?

Google defines requirements as “a thing that is needed or wanted”. With this definition in mind, we can apply the concept of ‘requirements testing’ to everyday life. Say you’re holding a dinner party and deciding what to serve. It’s likely you considered your guest list armed with the knowledge that Ann is gluten-free, and Dan is vegan, but you’re not sure about Dan’s partner… you think he may be kosher? You made menu decisions based on your guests’ requirements, and executed your cooking tasks accordingly.

We make decisions every day based on requirements, and software development is no different. Here, requirements are business deliverables — the ‘what’ that provides value — when delivered by a product ‘how’. The software value, then, become the ‘requirements’ and we define tests designed to meet business and user needs by exercising specific software functions or probing nonfunctional attributes, such as reliability or usability. Properly articulated requirements are the starting point for everything: project scoping, cost estimation, scheduling, coding, testing, and release management. Starting a project without a proper analysis of requirements is a recipe for disaster — it’s like building a house without a blueprint.

Requirements-Based Testing

Poorly defined requirements can lead to developing the wrong features, or the right features the wrong way. There’s no denying it feels great to get that code flowing but taking the time to develop clearly articulated and specifically testable requirements prior to development will pay off.

Why? Because this: Studies have shown two things. First, that most defects have their root cause in poorly defined requirements. Second, that the cost of fixing an error is cheaper the earlier it is found. Defects found in the requirements phase via ambiguity reviews cost ~$25-45 to fix. However, if a defect discovered in integration or systems testing has its roots in poor requirements definition, the cost to fix jumps to ~$750-3,000 per defect.

A study by HP and IBM estimated the cost of fixing defects discovered in production to be an astounding $10,000+! Poorly defined requirements lead to a nightmare of re-do; of the requirements, the design, the code, the tests, the user documentation, and the training materials. All this “re-do” work can send a project over budget and over schedule. Taking the time to understand, define, refine, and document requirements throughout the software development life cycle pays off, in a big way.

Defining Requirements for Requirements-Based Software Testing

The distinction between ‘what’ and ‘how’ is often a source of push-back against requirements-based testing. Software design is ‘how’, but the requirements are the ‘what’. The software design must meet stakeholder requirements.

Here’s an example scenario:

Your team is tasked with building an eCommerce site. To be successful, the site has many jobs to do. From the business perspective it needs to generate revenue through sales, secure user data, and allow monetary transactions while also updating inventory and coordinating shipping. From the user perspective, the platform should allow a user to create an account; to search for, find, and compare items; offer a shopping ‘cart’ in which to place their items; and end with a ‘checkout’ stage.

All these tasks should flow seamlessly and be intuitive. Sure, this is a simplified scenario, but you can see that by defining the requirements of the ‘what’ (it should DO) you’ll get to the ‘how’ (it should do the thing). This is exactly why time spent documenting and testing requirements isn’t paralysis by analysis, but a large measure of forward-thinking in the development process…

Testing Requirements: Best-Practice Tips

It might feel like paralysis by analysis to do the hard work of defining requirements, so let’s look at three easy maximums to remember when developing requirements.

1. Requirements Definition Begins with Clear Communication:

Poorly defined initial requirements, and lack of clarity and depth to the original specification document, are frequently the primary cause of requirements creep. Project Managers must be able to clearly communicate to developers the requirements of the software product being developed. To do this, clearly defined business objectives for the project need to be in place — from the conceptual phase through delivery.

2. Eliminate Ambiguity in Requirements Documentation:

Write requirements in a consistent style that allows all users to achieve the same understanding of the requirements. Documentation should be explicit and unambiguous. Each requirement should have a logical cause and effect detailing the expected outcome of a specific test action. Bonus tip: well written requirements can be reused in future projects!

3. Best Practice Requirements Design:

Each requirement should measure a specific ‘what’, should play well with other requirements, and (importantly) not contradict other requirements. Validate requirements against objectives. Without the guidance provided by tracing requirements back to business and project objectives, project teams often rely on their own judgment about whether to include individual requirements. These ‘out-of-scope’ requirements lead to time and budget overrun.

Requirements Based Software Testing – the Takeaways

Defining requirements begins with the first phase of software development where the largest portion of defects have their root cause and the correction of errors is the least costly.

Well defined and documented requirements can then be communicated clearly, concisely and consistently across the project team to develop requirements-based tests focused on meeting project objectives (and staying in scope), while crafting a product that delights your users.

Finally, if you find your project constantly struggling with creep or direction shift, it’s a sign that requirements weren’t developed fully or communicated clearly. Step back, define and refine your requirements using the guidelines above, and take that knowledge to your next project.

We can help with requirements based testing. Schedule a consultation here.

Test automation holds the promise of enormous benefits for improving product and service quality. It applies computational power to what it does best, the repetitive exercise of software systems. And, like any other powerful tool, misusing it is fraught with risk.

Software development has progressed from the Waterfall method that allowed months and even years for creating, assembling and testing a software release to Agile and Continuous Integration that crank out releases in a week or two. Along the way, software quality assurance had to keep up or get run over. This has created a glut of software products that have been lightly touched by testing only to be slammed by customer reviews.

The rapid release of features and defect fixes in software products and services means that test verification has had to change dramatically to keep up.

Software Testing Processes of the Past

In the past, software testing was the province of a department staffed by manual testers whose job it was to take marketing’s feature lists and verify that they were faithfully replicated in the products. Some of these people were code engineers who understood the system well enough to perform white box testing on it and ferret out deeply buried bugs. Most were techs who could follow test plans and use the system user interface to exercise all that it was supposed to do.

Any test response that looked inappropriate was logged in a defect management system and broadcast to the rest of the organization in a periodic bug report. These reports would be discussed and turned into prioritized action items at meetings of engineering, marketing and quality management. Testing and verification moved in lock-step with a glacial development methodology.

Automated Testing Necessary in an Agile World

Now, quality has been moved onto the front lines. Testers and code developers are expected to work in tandem fixing bugs as quickly as they are found. The Agile SCRUM has stand-up meetings daily in which the ‘stories’ selected for the current two-week sprint are discussed in terms of the day’s progress in both code development and operational verification. Once a fix or feature is integrated into the system, it is expected to be verified and released in a day or two at the most.

This has made automated testing a necessity. It has also made it appear to be the cure-all for getting verification done at or beyond the speed of development. While it offers vast benefits for testing system functionality quickly and repetitively, it is not the cure-all for test development. Test automation needs to be a tool in quality’s kit rather than the default test mode.

Test Automation Gone Awry

As with product code development, the test process to be automated must first be fully understood. Attempting to automate an ill-defined array of tests results in chaotic test results that rapidly become meaningless. This commonly happens when an attempt is made to automate manual exploratory tests which are guided more by the experience of the tester than a fixed test case.

Test automation is a matter of taking tests that are well defined with clearly defined result requirements and making them machine-executable. Automation will not fix problems with misunderstood code functions and product features for which detailed, workable testing has yet to be developed.

Automated Test Script Maintenance is Key!

Automated test scripts can be easily generated by test frameworks that will record UI actions and replay them. Unfortunately, such scripts are brittle in the extreme and, if anything changes in the code, the script will yield nothing but false errors. Over-use of poorly designed or maintained test scripts eventually leads to the automation process being generally rejected as being more trouble than it’s worth. Automation requires the commitment of scarce code development talent to script maintenance and these people will be constantly pulled off to write revenue generating code instead of working on tests.

Both of the above are going to be aggravated by the concept of ‘if a little automation is good, a lot is better.’ Attempting to automate testing of system aspects that change either on a regular or erratic basis will cause the scripts to become either obsolete or perceived as code talent sink holes. While automation can do wonders with scripting some test processes, it is a disaster with others. It is not a one-size-fits-all.

Navigating a Safer Path to Test Automation

What to do? Test automation is both valuable and necessary and will provide the boost that the QA process needs if a few simple guidelines are implemented.

1. Management Buy-in

Get management’s buy-in to test automation as a planned process effort with an adequate budget. Test automation is going to require script maintenance and it will be expensive. The payoff from a well-designed and maintained array of automation scripts is worth its price but worth nothing if it is abandoned because the budget dried up.

2. Choose the Right Automated Test Tools and Frameworks

Determine before starting that the tools/frameworks to be used are suitable to what is to be automated and how that part of the system under test works. The framework on which tests scripts are built must provide the flexibility and depth of functionality to stimulate the system under test and correctly record the results that it is expected to see. And expect to spend money for staff training on the automation tools as it will pay for itself many times over.

3. Carefully Determine What Tests to Automate

Layout which tests are to be automated because they are repetitive and lend themselves to this type of testing. Avoid the idea that all manual testing can be replaced with automation. This is how automation projects turn into costly failures. Automating user interface tests means that the scripts must change each time the interface is changed which will make the scripts as expensive as the UI code. The opposite end of this is that automating tests of API calls is an excellent use that puts the computer testing another computer.

A simple spectrum of automation viability is:

Test automation is a powerful tool, but it won’t do everything. Plan it, manage it and fund it properly and it will be a major help in navigating the Agile development crush.

The standard argument for outsourcing has been done to death. As in, “We all get it. ‘Nuff said.” The term itself originated in the 80’s to describe contracting with external entities to provide “an exchange of services, expertise, and payments”. Outsourcing, in its most basic interpretation, is a business maneuver to save money. By ‘saving money’ I’m referring to companies who outsource manufacturing by locating their facilities in areas with fewer restrictions or a lower employment wages with the goal of increasing revenue. This definition casts a dark shadow on any discussion of the idea, and unfortunately, such limited understanding only serves to exacerbate the divide between in-house and external teams. Outsourced software testing is a means of expanding and enhancing a product offering, not a ‘make-it-cheaper’ ideology.

What if we understood outsourced software testing services to be a method which facilitates an organization’s focus on core competencies, while fueling innovation through external input and mitigating a shortage of skill or expertise in build-specific areas? Augmenting your software test team externally might fill your need to address a gap in your in-house expertise, or to test a product on platforms it doesn’t make sense to purchase.

Outsourced software testing services aren’t a ‘replacement’ for your employees. Even in this view, outsourcing can save your organization time and money by focusing SDLC actions to best utilize individual or team skill-sets!

What If Your In-house QA Team Gives Outsourced QA the Evil-Eye?

Here’s the thing: if you are a project manager with a high-level view of business requirements, you are responsible not only for SDLC processes, but those small details — like the build budget. You ask yourself questions like: “Does it make sense to invest in several mobile devices to use in testing…or could you confidently outsource this to a company with the hardware (and experience!) to complete the tests?” Further, say you decided your project budget allowed for the purchase of several different mobile devices. Now you ask yourself, will we ever use this hardware again? (Think about the bride’s wedding dress…or the groom’s tux. Who spends that much money on a one-and-done? And don’t give me that argument about saving the attire for your kids. Fashion changes fast! Just like hardware does…)

Today’s business reasons for outsourcing QA go beyond the original ‘cost saving’ paradigm. Outsourced software testing allows smart companies to provide stronger releases, at a quicker pace and your employees need to know this! In-house teams are vested in both their work and their jobs. It’s safe to assume they’d like to do right by their employer and to keep their paychecks rolling in! A difference in understanding business strategy can translate into your tribe regarding outsourced QA as a threat. Here’s where you, as a manager, can help alleviate employee concern about outsourcing.

Outsourced QA Tip #1 – Communicate

Internally

Outsourcing can confuse employees who don’t understand why you are doing so and add challenges to the daily workflow. Confusion (or resentment) creates a less-than-positive work culture, which can severely impact productivity. The best way to deal with this is prevent it from happening. How?

Provide transparent communication to your team about the reasons for outsourcing. Frame outsourcing as a strategic business decision meant to achieve objectives through targeted use of outside resources with appropriate skill sets. Position outside resources as partners (NOT competitors!) and an extension of the existing team. Highlight your internal team’s areas of expertise and remind them that outsourcing will allow them to focus on what they do best!

Externally

To facilitate partnership, it’s best to establish workflow and communication protocols early in the collaboration. It’s true that outsourcing often brings workflow challenges, and the area most likely to be a workflow pain-point is cross-team communication. Clumsy communication protocols can hinder progress, create redundancy of effort, and lead to lack of clarity. These three issues have the potential to negate each and every expected benefit of outsourcing. Create protocols for information sharing, define workflows (who does what), and build a shared strategy, then lay it all out on the table. Communicate (and enforce!) these details to both teams with honesty and confidence.

QA Outsourcing Tip #2 – Scrum!

I’ve worked with teams as both the outside resource, and the internal team member. I’ve seen Slack channels get garbled and email threads get lost. I’ve felt the frustration of seeking clarification through multiple points of contact and getting different answers. I’ve had to re-do days worth of work because an ‘inside’ contact went on vacation and didn’t leave process updates for the external team. (Yes, really — and oh so frustrating!)

From this experience the biggest piece of advice I have is:

A daily Scrum via Hangouts or Skype is the key to cross-team communication!

Yeah, I said it. But seriously, dear reader, if you find yourself in the position of managing a project with outsourced talent, please schedule daily or weekly video conferences. Just like an internal Scrum, an external video conference strips away the communication clutter. A 15-minute chat resolves issues far quicker (and with less misunderstanding) than a 2-hour Slack conversation and putting voices and faces to names will help integrate both teams. It’s much harder to feel threatened by someone when they’re no longer a faceless name in an email thread.

Final Thoughts on Outsourced Software Testing

One area with a large impact on communication and workflow is whether you’ve outsourced onshore or offshore. Offshore software QA will complicate communications in additional ways including differences in time and culture. If your partner is on the other side of the globe, Scrums become much harder to coordinate. How will you address this?

Cultural differences can be un-examined barriers to progress and often ask employees to address unconscious biases or modify unconscious behavior. You may find Professor Hofstede’s research into the 6 Dimensions of National Culture a useful tool when prepping for offshore collaboration.

To conclude, the most helpful thing management can do to remedy both internal and external concerns in outsourced QA is communicate. Yes, really. My suggestion might feel like something out of ‘one of those’ magazines, but maybe they’re not off the mark. Communication is a vital part of any relationship and nowhere more so than in outsourced software testing. Build your business relationships on a solid foundation of targeted communication protocols and you’ll realize the power of outsourcing!

Interested in Outsourced Software Test Services? Contact Us Today to Alleviate Your QA Headaches.

Writing software test cases isn’t something to take lightly. Aside from coding and testing your app, writing test cases is the third ingredient to consider in the ‘successful release’ triumvirate. Does this sound like a lot of pressure to place on a task often regarded as mere ‘writing’? It should — because it is. Poorly written test cases directly impact testing, and therefore revenue. This holds true whether we’re discussing a customer-facing or internal platform. Either type of software will suffer if test cases are designed by someone who doesn’t understand the value test cases bring to the testing process. In short, writing test cases is an adjunct to stellar development, not an afterthought.

What is a Software Test Case?

The surface level definition of a test case is quite simple. A software test case determines a set of conditions in which the software is checked to ensure requirements are satisfied, and that it functions as specified. A test case is a set of instructions covering a single instance of how and what to test. Test cases include: a title, a description, exact test steps, expected results, and observed results. Straightforward, right? Well, yes — and no. Designing test cases like a pro is more than simply writing “Check compatibility with all browsers”!

Who Writes Software Test Cases?

Other than unit tests, test cases should be written by the quality assurance (QA) team. While quality is everyone’s job, it actually is the job of QA to manage the quality process from idea to release. Pro-level test case design asks the writer to take a deep dive into the application under test (AUT) through both the business and user perspective. This sort of deep-dive creates extremely fine-grained attention to detail. Test case writers ask themselves: What is the test goal? To validate what the software is supposed to do? To validate what it’s not supposed to do? To find out if the software breaks at a certain point?

To adequately test more than the ‘happy-path’ imagined by the developers, test cases must be designed to reflect the way non-developers interact with digital offerings.

Still, simply creating test scenarios with an in-depth understanding of the AUT and its users isn’t enough. The pro test case writer will acknowledge their audience, purpose, and goal — and compose accordingly.

One method of translating test requirements into accessible ‘everyday speak’ for testers is embodied by the ‘plain-language’ movement. Plain language is writing designed to ensure the reader understands as quickly, easily, and completely as possible with a goal of being easy to read, understand, and use. Plain language avoids verbose or convoluted language, and technical jargon. This means test case writers shouldn’t feel trapped by the linguistic expectations of the developer discourse community, but also not feel compelled to ‘dumb-down’ their writing. After all, if our tester gets lost in the jargon, what good is the test case? Here, the optimal test case writer is the QA teams’ technical writer who understands the UAT, the development process, and the audience.

The Value of Test Case Design

Test cases add downstream value by enabling anyone to go in and retest using the written case. Well-written test cases can mean the difference between making or breaking functionality and user satisfaction when releasing updates or adding new features. Test cases are powerful artifacts for the future, as well as acting as a repository of information about the way a system or feature works.

Summing the value of test cases, we see the following:

• Ensures comprehensive test coverage by addressing all aspects of functionality, including business requirements, user needs, hardware variances, and out-of-spec user actions
• Properly documented test cases are reusable, allowing anyone to reference them and execute the test.
• Test case documentation acts as a knowledge-base for platform details

Now that we’ve looked at the ways well-written test cases impact software development, and the skill-sets a test case writer should hold, let’s explore a few high-level steps you can take to ensure your team writes quality test cases.

6 Tips for Writing Software Test Cases

Use a Strong Title

• A good test case starts with a strong title. Best practice indicates naming the test case to explain the module you’re testing. Example: When testing browser compatibility, a title might read “Steps to test application to browser compatibility”. Titling test cases with relevant keywords also allows for quick search referencing in test case databases.

Include a Detailed Test Description

• The description should tell the tester what they’re going to test and include any other pertinent information such as the test environment, test data, and preconditions that must be met before the test is executed (such as an OS or browser specific version being loaded)

Include Assumptions

• Include assumptions that apply to the test. This might include where the user starts the test at (order page? search page?)

Create Clear and Concise Test Steps

• Test steps should include the data and information needed to execute the test. Nothing less, and nothing more. Don’t leave out necessary details, but keep them clear and concise (see: plain language!)

Include the Expected Result

• This tells the tester what they should experience as a result of the test steps and determines if the test case is a “pass” or “fail”.

Make it Reusable

• The pro level test case is reusable and provides long-term value to both the software development and test team. When writing a test case, keep this in mind. Save time — and money — in future iterations by re-using the test case instead of rewriting it.

Final Thoughts on Designing Test Cases

Anticipating user needs, and integrating those needs with business requirements, is crucial to finessing your app before deployment. This is a job for a professional! Don’t relegate test case writing to an afterthought or view it as ‘just another administrative function’. Test case design has the potential to impact development, whether that impact is positive or negative depends on the skill of the test case writer.  Let your QA team handle test case development, and rest assured the right people are doing the ‘right things’. Oh, and don’t forget to use plain language! 🙂

Need help designing test cases? We can help!