Test Automation Tools: Open Source or Proprietary?
eCommerce, and business conducted over the Internet in general, has to keep abreast of an entire web ecosystem of access devices and software. From desktop/laptop computers to smartphones and browsers to apps to hybrids, the increasingly diverse array of gear and the code it runs present a singular problem of excess.
The QA group that is trying to continuously maintain its focus on getting test coverage on the most important combinations of these is slogging uphill against a stiff wind. The ongoing demand from product management is to cover as much as possible as thoroughly as possible while keeping up with the changing demographics of device/software usage. And, trying to be helpful, the CTO suggests ‘why don’t you just automate the bulk of the test work?’
Which Test Automation Tool Should You Use?
That’s a great idea. Now, which of the plethora of test automation tools will yield the best return on your department’s investment of time and money for the effort of configuring it and writing test scripts for it? Current test tools are improved continuously, new tools are appearing constantly, and many of them have an Achilles heel of one sort or another and are withering away. A common issue in this selection process is that it often takes about a year for a tool to show signs of being a failure.
What tool to use very much depends on what you are trying to accomplish. Are you looking to create a fast-running smoke test as opposed to a comprehensive system regression test? What is the time frame in which you have to produce results? What kind of budget is available for hardware resources to test on and software resources to test with? The way you deploy your tests and the resources to accomplish them also depends on the goals of the product and development teams. Does the IT group use continuous integration, a version of the Agile development methodology, or some more traditional model?
In an ongoing series of articles, we will try to help you sort through some of these questions and provide some guidelines for answering them. We will note tools that have special capabilities, unfortunate pitfalls, and those that are undergoing renovation to better address the current state of mobile test automation as a genre of QA interest and effort. Stay with us. Yesterday’s silver bullet may quickly become tomorrow’s lead slug.
In working within our clients’ test requirements, QualityLogic has continuously analyzed the available array of test automation tools, and we continue to do so. One issue that has to be addressed for each test situation is whether it makes more sense to use an open source tool set or a proprietary test tool system. It can be well argued both ways.
The Pros of Open Source Test Automation Tools
Open source tools have the near incalculable advantage of being no-cost acquisitions. They are literally available for the effort of downloading them from the .org websites that promote them. The more widely accepted tools are supported by large networks of developers and some are maintained by companies that sell ancillary services that facilitate the tool’s use and deployment.
Many of the more successful tools have plugins and associated toolkits or frameworks. These can come in the form of bridges to different technologies, such as making a tool focused on iOS testing functional with Android devices, as well. And they can be collections of test code scripts and modules that are focused on particular aspects of device/app operation. This latter add-on is particularly valuable as these scripts can be used as building blocks to create complex test suites without having to create all the working bits from scratch.
The Cons of Open Source Automation Tools
On the down side, open source tools are ever a work in progress. Some are operated by command line interfaces and require considerable programming experience to get more than cursory results from them. Others address very narrow aspects of test automation and require a number of other associated tools and modules in order to produce any results at all. Figuring out exactly what is needed and how to connect and configure all the pieces is an exercise left to the user. This worthy may or may not be able to get needed operational answers from the forums associated with most open source tools.
What’s Great about Proprietary Automation Tools
Hmmm… well what about proprietary tools? They tend to be much more polished software systems. The companies that make and sell these products are vying for your business as a potential customer, so they make systems of their tool sets that are typically unified under a single graphical user interface.
Out of the box, they will include the capabilities to record and edit test scripts, queue and execute them, and then analyze the returned results from a number of sophisticated perspectives. While they don’t typically have the array of plugins and tool kits seen with open source, they make up for this lack with a level of usability that takes much of the time and pain out of setting up complex test systems. In addition, most of them will use test scripts written in a variety of languages, with Perl, Python, Java, Ruby and C# typically supported. Another large plus is that most all of them come with detailed documentation on configuration and usage, along with an array of classes to train both beginning and advanced users.
And…The Not So Great of Proprietary Test Automation Tools
That said, all of this shiny goodness comes at a cost. Proprietary test systems are typically sold via seat/installation licenses. They may constrain use of the product to a particular host computer, a particular user, or both, and their prices will vary accordingly. They usually sell for around $3,000 per seat per year (yes, you have to pay for them every year) for an installation license that is tied to a host. The price can go to as much as $10,000 to $15,000 per seat for a ‘floating’ license that may be used by anyone on any system.
Another consideration is that, unlike open source tools which have a following of developers, proprietary tools are usually the product of a single company. If you have invested considerable resource expenditures in configuring, deploying and scripting with a proprietary tool and its parent company goes out of business, you may be stuck with an expensive system that slowly fades into disuse for lack of support and development to follow the march of new technologies. For this very reason, IT groups that invest in these products may also demand code escrow arrangements so they can take up maintenance and expansion of the product in the event of the failure of the business that they originally purchased it from.
So, How Do You Finally Decide Which Test Automation Tool to Use?
In selecting between open source and proprietary, look to your QA staff and assess your needs according to their strengths. If you have a crew of ex-programmers who are ready to take on an open source tool and write whatever code is necessary to make it perform, your path is clear. If you are looking for a tool that can be used by anyone familiar with your own product or site to create quickly usable tests, you may want to look closely at purchasing a proprietary tool that helps them through that process.
In future articles we will look in depth at various open source and proprietary tools to help you make that decision.