Test Automation and Automated Testing Explained

You say to-mah-to, I say to-may-to. Can we call the whole thing off? Of course we can. Because no matter how we each pronounce the word, both you and I are referring to the same thing – a plump red fruit that tastes good in salads and on sandwiches. There’s no real dispute here – we’ve just got different accents, and so we enunciate certain words differently. Where we might genuinely disagree as we tuck into our lunch, however, is on the differences between test automation and automated testing. In some software development circles, these two terms get bandied about as if they can be used interchangeably. They can’t. Like tomatoes and apples, there are differences between the two – and differences are important. A tomato is a round red fruit with seeds inside. So is an apple. But no chef worth her salt would use them interchangeably when making soup or ketchup.

Continuous Testing

Before we get into the distinctions between test automation and automated testing, we first need some context – and that context is continuous testing. Continuous testing is a DevOps practice of evaluating quality at every stage of the software development and delivery process. It is an overarching software testing strategy that involves testing early, testing often, testing everywhere, and – importantly – automation.

The primary purpose of continuous testing in the software development and delivery pipeline is to obtain feedback on the business risks associated with a software release candidate (i.e., a software change or update) as quickly as possible. Continuous testing is crucial to modern business practices for the simple reason that most companies are under constant pressure to continuously adapt software in line with user feedback, shifts in the market, and changes to business strategy.

This has given rise to the continuous delivery model – i.e., where software is constantly in development and must always be ready for deployment. However, in the digital economy where websites, apps, and other software have become the primary interface between business and customer, an application failure is a business failure. Even a seemingly minor glitch can have severe repercussions if it impacts the user/customer experience (customer experience, of course, is fast becoming the number one brand differentiator). As such, continuous testing is required to discover problems as early as possible in the software development process, and thereby mitigate business risks when releasing software to production.

(Image source: tricentis.com)

Continuous testing differs from traditional methods of software testing in the very sense that it is continuous – i.e., undisrupted and carried out on a persistent basis. In traditional environments, testing only gets completed in one big bang at the end of a software development cycle. However, as continuous delivery models have gained prominence in the rush to ensure releases go to market as quickly as possible, running tests late in the software development process exposes teams to the risk of only discovering problems at a very late stage. This can be a total game-changer – the team may be forced to go back to the drawing board at the last minute, causing a massive roadblock in the release of new developments, and defeating the very purpose of the continuous delivery model.

Continuous testing is the answer. By making testing continuous and integrating it into every stage of the build process, issues are identified almost immediately and can, therefore, be rectified much sooner – closing the circle to ensure fast and successful continuous delivery.

In addition, it must also be remembered that as each new release adds new features to the software (which must be continuously tested), they also cause changes that can break the software’s existing functionality. So, testing existing functionality (i.e., regression testing) has to be continuous, too.

However, many companies are working with limited resources and limited time to execute all of these tests – and at some point, they are bound to fall behind. And this is where both automated testing and test automation come in.

Automated Testing vs. Test Automation

For testing to be continuous with limited resources, some tests (as many as possible) need to be automated – and there also needs to be a way to track and manage all the different tests that are being performed (both manually and via automation).

And here is where we get the distinction between automated testing and test automation.

Automated testing is the process of conducting specific tests – such as regression tests or unit tests– using automation tools rather than doing them manually. Test automation, on the other hand, refers to automating the process of tracking, managing, and initiating the different tests. In this way, test automation doesn’t always refer to the tests themselves (be they automated or otherwise), but rather how software development professionals manage these tests throughout the continuous delivery pipeline.

Automated Testing

The main benefit of automated testing is that it simplifies – or even removes – as much of the manual effort of testing as possible, while also speeding up the process. Unit testing and in particular, regression testing are often considered good candidates for automated testing because they tend to consume large amounts of a software development team’s time and resources. In fact, some software professionals think of automated testing as “automated regression testing” because they don’t consider regression testing real testing at all – rather the laborious and repetitive process of rechecking existing functionality.

Automated testing can boost a software development team’s efficiency. Automated tests can run repeatedly – even continuously – at any time of day and result in higher accuracy (less chance of human error – particularly as testing the same functionality over and over again can lead to routine-blindness), greater coverage and earlier bug detection. These tests can also be performed in a faster timeframe than a human tester, enabling staff to focus on other project priorities.

Test Automation

Naturally, managing all of the testing needs in a continuous testing environment is no easy undertaking. It requires extensive efforts in terms of communication between stakeholders to keep track of where new code has been deployed, when each piece needs testing, and how it all feeds back into the constantly moving process of continuous delivery.

Test automation is the answer. By automating the tracking and managing of all testing requirements – including the extent of existing testing coverage and what other types of testing might still be required to increase that coverage – test automation ensures that testing is managed effectively and that software development teams maintain a high standard of quality at all stages of the continuous delivery pipeline.

In essence, test automation is about generating test cases – i.e., what to test and when to test it – automatically, and then scheduling test runs to execute those tests. Test automation tools are designed to flag up a list of items for which test cases need to be created – bringing new needs to software testers’ attention automatically – and can also actually initiate an automated test. In addition, these tools track the progress of each test case to completion, and automatically separate and categorize test cases to make it easy for users to ensure there is proper coverage across each stage of the development lifecycle.

Final Thoughts

To summarize, with continuous testing, software development teams aim to reduce business risk by testing early, testing faster, testing often, and automating. To be continuous, tests must integrate seamlessly into the continuous software delivery pipeline to provide actionable feedback which is appropriate for each stage of development. Test automation tools are designed to automatically trigger code testing at each stage. As such, the term doesn’t refer to the tests themselves – rather, the process of managing, tracking, and initiating the various tests. Actual tests can be either manual or automated – and when they are automated, the process is referred to as automated testing.