Total efficiency often is a state that is difficult to attain; but once a company and its processes are streamlined within a loop of continuous improvement, it has proven significant to both the company and its customers. Quality covers all business areas and processes; guaranteeing quality guarantees efficiency since rework is not required and waste eliminated, or at least reduced to a minimum. Within software engineering, software test automation is a technique that cuts costs and reduces software product time to market through maximizing test coverage, but there are important questions to ask before investing in QA tools.
The benefit of software test automation is obvious: at the end of the day, products and services will ship fewer bugs and users will be happier. However, in spite of the benefit, software testing automation is not universal. Before investing in and implementing automated testing, companies need to answer three important questions:
- Why automate?
- What to automate?
- Free or license?
Which tool is the “right” tool for QA automation depends on the answers to those three questions. Let’s take a closer look at the considerations.
Why automate in the first place?
This question is one of the most common questions overlooked when embarking on an automation testing process. But it’s a crucial question; without a solid answer, the business, financial and technical groups within a company won’t be aligned. This can cause operational problems later. It is important to have solid, logical arguments supporting the process. These include:
- Reduction of test case execution time: One of the big advantages of automation is to reduce the test case execution time. By implementing correctly automation execution time can be reduced up to 800 percent.
- Maintainability of the tests: In a well-designed automation testing framework it is easy to perform changes in the tests, add new features, new steps and create new kinds of test result reports.
- Fast and easy error catching throughout the software project life cycle: By implementing different techniques for automation testing such as unit testing and functional testing combined with continuous integration, errors are easily identified, guaranteeing that each commitment to the code repository has the expected structure and meets the expected functionality. If integrated with continuous integration, automated tests guarantee traceability, so it is easy to know exactly where the defect is and who is responsible. This will eliminate additional waste of time.
What to automate?
Deciding what needs to be automated has a huge impact on long-term results and costs. Additionally, the automation tools that should be utilized depend on what actually will be automated. Here are several key considerations.
1. Platforms (cross-browsing or device), technology and type: Usually one of the most important aspects to consider at the very beginning of an automation process is what platforms are going to be supported. This is mostly because the available tools and the tester skills are different based on two areas: browser testing and mobile device testing.
With cross-browser tests there is a set of tools that can help in creating reusable scripts for different browsers in order to gain an easy way to maintain the tests and also guarantee a certain degree of coverage.
On the other hand, with mobile device testing the first step is to define what kind of devices are going to be tested since the tools available in the market are targeted at different platforms. Choosing the one that best matches the particular need is key.
2. Tester skills: multi-language or recording: Automation is all about writing scripts; a person with development/programming skills is required, especially one with strong object-oriented applied knowledge as well as experience in a specific development technology/language.
Object-oriented and development patterns will help to create maintainable testing frameworks. On the other hand, recording tools don’t require the same technical skills because the process is all about “recording” the steps that will be automatically reproduced later by the tool. In this case what is needed are testers who are able to identify possible scenarios and steps to record them.
These types of tests are not as maintainable or reusable since they become obsolete once something in the user interface changes.
3. Testing types to automate: Functional? UI? Regression? When looking for an automation tool, both the type of tests and the time available to create and execute tests are key factors. It is important to consider that not all the test types can be automated by the same kind of tool. And, even though multiple tools need to be used, there are some that are very compatible with each other. This is because they share development languages and development techniques that can help the team reduce unnecessary extra work to train resources for different development technologies.
4. Change management: Change management is one of the most important areas to keep in mind when creating test scripts. Creating reusable scripts will help to avoid the duplication of code. Test scripts are small pieces of software, small programs; as a result the standard software development best practices apply to these automated scripts.
Applying design patterns, following naming and coding conventions, properly commenting out code, and loose coupling are among the important aspects of creating maintainable, flexible and well-structured test scripts.
5. Technical support, documentation, manuals and examples: A good documentation and change management methodology can help achieve a better understanding of the test scripts. Having this information will help the rest of the team perform changes in an easier way; it will also help new team members understand the scripts quicker. Good documentation reduces ramp-up time and time wasted in answering questions. Examples and manuals are great input to teach new users how the testing software works.
6. Integration with systems (unit testing), APIs and build software (continuous integration): There are many ways to automate different test techniques such as unit testing, automated functional testing and continuous integration tools. These combined tools and techniques help to ensure high levels of quality in software projects. Here’s why:
- Code is tested in different perspectives.
- Continuous integration tools provide information about errors found in the test execution.
- Possibility to create stable builds to perform deploys after a rigorous automation testing process and a strong coverage.
Continuous integration and automation not only help reduce testing execution time but also help to address errors found in the builds faster. With continuous integration and automation testing the team can identify very important aspects of a new issue introduced because of a commit (such as the developer/tester that committed the code that broke the build, exactly which code or module was affected and exactly what is the issue). Having this data available makes it easier to fix the issue or issues in a short time compared to doing manual testing and code reviews only.
7. Proprietary automation frameworks: Creating your own automation scripts has many advantages; one of them is the customization of specific testing actions. A proprietary framework allows the team to incorporate new features to their testing process, add value to the current tools that are on the market and also utilize different types of tools or technologies as complements to create a robust framework.
For this kind of framework construction the most recommended testing tools are the developer-oriented tools or frameworks — especially open source — because they are easily customizable.
When creating a framework it is very important to use design patterns and good coding practices since it will deliver a very stable and robust framework. It also will be easy to maintain.
8. Results management and logging support: Results management is a very important feature of an automation framework or a test script since it is the way the QA team lets the rest of the team know what is going on with the software under construction. The more details you add to the results of the execution of the scripts the faster the whole team can:
- Take action on different situations
- Deploy stable builds
- Incorporate customized error messages
- Take screenshots at the exact moment an issue is detected.
9. Test data: Using developer-oriented testing tools helps the QA team to create scripts that interact with the application database being tested. Creating testing scenarios with clean data guarantees the correct execution of the tests and ensures that the tests were not adversely affected by other tests that had been executed. Also, these tools enable the team to extract data that is preloaded in the database in case they need it.
Free or license?
One of the big concerns that companies have is the cost of testing tool licenses. A key consideration is not whether the tool is commercial or free but, rather, if it will adequately suit the company’s needs. Usually the advantage that licensed tools provide vs. free tools is customer support. Also, licensed tools provide more tool options that help save time in the script-creation process.
On the other hand, free tools do not usually provide customer support. However, many have very strong development and support communities, as well as documentation that provide users with helpful information.
A key consideration is that, in many cases, the use of a free tool causes the team to spend time creating or configuring different features/options that are already developed or configured with licensed tools. This is why a team needs to do a proper assessment between costs vs. needs vs. available functionality when deciding what tool will best fulfill the specific requirements. The cost/benefit analysis is extremely important when selecting a tool!
Conclusions
- Automation tools are a great way to save time and money for long-term projects because they reduce the time expended in regression testing and other kinds of testing.
- Automation testing guarantees that builds will be more stable in less time. When there is no automation in place, the software being developed is released to QA before a thorough inspection, not ensuring that the build is 100 percent stable and/or fully functional.
- Choosing the correct tool for the QA automation testing ensures that the QA team will create the scripts with the expected quality and in the expected time.
- A tool cannot be defined as a good tool or bad tool only by its cost. The tool’s quality depends mostly on what the team expects to do regarding software automation.
- Developer-oriented tools are great for automation because they are very flexible in terms of creating new functionality or incorporating other frameworks or tools to support robust frameworks that create and execute test cases.
- How to present the results of each execution is a vital point to take into account when selecting an automation tool because this is where the team is going to see the information needed to define if a build is stable or not. Also, in case the build has issues, with a good reporting feature the team can find and fix the issues in less time, adding additional value to the project.
- A very good choice when a company is looking for an automation tool is the support it has — how difficult it is to find information about how to use it, how to fix problems, or how to improve it. A good support community or customer support process is a very important point to take into account.
- A tool that offers an easy change management methodology within the automation scripts is an excellent choice, specifically if the QA team is working on projects that call for Agile methodologies or if the requirements change periodically.
- Tools that share technological aspects between test techniques are great to have because they save time and money in training and in test creation and maintenance time as well.
Francisco Barquero is a quality assurance engineer at nearshore software engineering services company Avantica Technologies. He has multiple years of experience in software engineering industry including QA activities focused on utilizing industry standards and applying both manual and automated testing techniques. Together with QA engineers located in the company’s software engineering centers in Costa Rica and Peru, Francisco collaborates with North American, Latin American and European customers utilizing both traditional and Agile methodologies.