The software development environment has shifted significantly in the recent years with the introduction of Agile and Devops. This new focus on higher speed and the shift to smaller, more frequent releases, has a great influence on testing. In the last year the average percentage of test automation has increased from 28 percent to 45 percent, according to the World Quality Report 2015-2016 . But what is test automation actually? Is it the Holy Grail?
What is test automation actually? Test automation or ‘automatic testing’ is an automated test process where in predefined and set values are controlled by a ‘robot’ in a particular information system or subsystem. This obviously isn’t a real ‘robot’, but a piece of software that allows another piece of software to check the defined values. To create a visual, the term ‘Robot’ best fits the description.
The testrobot has many advantages as long as the instructions and data sets given to the robot are correct. A robot is better, faster and more capable to find mistakes, it can compare millions (conversion data) of lines without ever needing a cup of coffee, it can run unlimited regression test on labor-intensive processes, it can quickly compare installations with each other and we can still keep going on from there. Test automation offers many advantages.
This robot also comes with it’s disadvantages. First, all instructions that should be given to the robot should be concretely identified and ‘explained’ to that robot. Mapping of what is being tested and how it is being tested is a profession on its own. Now imagine all the input, process and output variables and conditions have to be written as well. And as these instructions and conditions are conceived and organized, the robot still needs to be ‘trained’ to do the right comparisons. This is people’s work. The field of artificial intelligence is growing rapidly, but it will certainly not take those activities out of the hands of humans in the near future. Test automation is expensive, as a result of which a good cost/benefit analysis with clear trade-offs is needed, to make a clear distinction between which parts will be performed by the robot and manually by humans.
‘Test Automation is therefore perfect for comparing variables, and most preferably in large quantities, but for the time being will not exceed the intuition and the implicit observation of a human being.’
Secondly, the robot can make high-speed comparisons, but during manual testing the scope beyond test execution is more broad. The human perspective is much more aware and intuitive and therefore ‘automatically’ looks beyond the test case. Think of connections between input, illogical acts, spelling mistakes and so forth. The robot has no intuition, it does absolutely nothing more or less than to compare variables and therefore has no implicit observation. Precisely because of the implicit observation a human can detect coincidences. Whereby an error is discovered only at a specific combination of variables or while having knowledge of daily operational problems, which still can change in the course of time. Test Automation is therefore perfect for comparing variables, and most preferably in large quantities, but for the time being will not exceed the intuition and the implicit observation of a human being. As an example: exploratory testing, which is very valuable to quickly find faults, but a robot won’t be able to do this in the near future.
Thirdly, it is often overlooked that some integrated processes aren’t automatable. These include financial transactions to be processed by batch through another institution before permanently booking these in your financial systems. In addition, maintenance is of the instruction and data set of the robot is more work than initially thought. Because the context (both business processes and IT) of organizations are constant in motion, the robot must be trained again with new instructions and custom data. In addition, the software which is automatically tested is also changing. All new functionalities, new interfaces and the like. Will first be evaluated and tested by humans on functionality, but as well in the organizations specific setting. Before the new instructions can be given tot he robot. Test Automation is ideal for stable conditions, in which the frequency of changes is limited.
Fourth and final point, In our field of user/acceptance testing it’s difficult to carry these out by a robot. The definition of acceptance tests is to establish that the organization is ready for the use of the software. Thereby it is not only determined if it works, but also whether the user can do their work with it. Will the robot determine if the organization is ready to use it?
In addition, the review and analysis of the test result is people’s work. At the moment that 10 users review 10 test cases I get a 100 test results. What if 4 users say that it is good, and 6 says it is an error. At that point a dialogue should start on the interpretation of the data and the larger context in which the good- or error-finding is reached. You can not have a robot do this.
In our vision all test results must be managed from one central location; the test results of the human, but as well the test results of the robot. Because also the test results of the robot must be evaluated and analyzed on truth and context. From there on issues are created and shared with stakeholders. These aren’t only the software vendors, but also the organization itself. Test automation is a nice addition in the entire test spectrum, but all test results in the end need to be reviewed by humans.
Test automation has many advantages in the context of efficiency and effectiveness during the implementation, but it has its share of disadvantages. Focussing on Agile-like software development and deployments test automation is essential to manage the quality and speed of increments.
However: Testing is a profession and testing is thinking. Therefore it is crucial to test manually, with all it’s intuitions, implicit observations and determination if an organization is actually ready to use the software within a healthy balance with automated testing. testing might involve several roles, the QA-manager and the IT-manager, the test manager or the release manager. And of course the end users as user testers.