October | 2023
Across industry verticals, enterprises are moving towards test automation. It has been observed that there are a lot of investments into test automation but not every initiative has been able to provide the desired benefits. How many times have we heard the project sponsor say – “I have invested into automation, but I don’t see the benefits and automation has not been able to add significant value.” What moved the benefits away? The answer lies on the Automation Checkerboard.
Test automation has characteristics similar to the news we read in a newspaper or watch on television. It keeps changing every day, and it needs to be refreshed regularly, lest it loses its intrinsic value. Usually, test automation is an afterthought largely driven due to schedule slippages in upstream development. The typical approach to automation is an ROI computation that compares the investments against the effort savings brought in by automation. Developing an automated test upstream is cheaper than developing the same script much later in the development lifecycle. Due to this dynamism, the traditional ROI model will not predict the benefits correctly.
Every enterprise believes that it can derive significant benefits from automation. However, there is no structured model or approach, which defines where to start and when to end the investments into automation. The complexity of the problem is aggravated due to the fact that test automation activity is highly dynamic in nature. Any task that gets automated will always be affected due to a business change, process change or technology change. Hence a simple ROI model will always fail, and will be inadequate to drive the strategy for automation.
Understanding the problems with test automation
Let us look deeper into the problems of test automation at an enterprise level. Here is the story of a test team – Team GodSea which is performing the testing of a suite of applications in a leading investment bank. The classical problems with test automation are brought out.
GodSea - the test team has been writing regression tests and executing them manually across many releases. The regression pack has grown, and the test assets have changed many hands. It has reached a stage where the regression pack is large, and cannot be executed entirely for the next release as manual manpower is not sufficient. The team selects a subset of tests for regression based on the risk and priorities. The regression pack is not fully executed, and the test coverage is limited. The natural fallout is that some bugs slip into production.
GodSea test team figures that the best way to solve the problem is to automate the regression test pack. Performing an automation assessment yields the following results
A new test automation team MountainGod is setup to address the issues in testing. Their objective is to bring in cost benefits along with increased quality. Once the automation team starts, it faces the following challenges:-
A periodic review of the MountainGod team reveals that the automation is far behind expectations and the confidence of the application teams on the overall testing is low. Additional efforts have gone in to make the automation work, making the automation more expensive and the ROI weaker. The automation scripts that have been built so far are being A new test automation team MountainGod is setup to address the issues in testing. Their objective is to bring in cost benefits along with increased quality. Once the automation team starts, it faces the following challenges:- subject to continuous maintenance. The GodSea team is scampering as testing schedules are crunched and they have to finish up in the reduced timeframe. The benefits from automation are being questioned.
This is the classical story with every enterprise on an automation journey.
The Solution- Automation Checkerboard
The benefits derived from automation for an enterprise needs to be looked at in two dimensions. The first dimension is the ‘Quantum of Automation’ and this drives the classical ROI model. The quantum of automation achieved resulting in effort savings compared against the absolute manual effort determines the cost benefit achieved through automation.
The second dimension often neglected is the ‘Effectiveness of Automation’. In the purest sense, effectiveness of automation is a misnomer, as it tends to define the effectiveness of the tests itself than the automation per se. However, automation without this relationship to the defined tests is quite meaningless and hence it is imperative to keep this dimension into consideration.
Automation has evolved from being an efficiency lever into a two dimensional plane where the effectiveness of automation also needs to be measured. This can be best visualized as a simple two dimensional checkerboard.
The starting point on the grid is the current state of automation in the project. If a project has moved from 20 to 80% automation achievement (X axis), it is highly possible that due to maintenance considerations, the effectiveness can be near zero (Y axis). Using the two dimensional automation checkerboard Automation Checkerboard gives a clear answer to the question – Where and how the benefits move.
The Automation Checkerboard is a useful tool to define the automation strategy and roadmap for an enterprise – the plane on the x axis defines the quantum and the y axis defines the effectiveness. The right moves by an enterprise will deliver on both planes – efficiency and effectiveness, and hence deliver maximum value from automation. To be able to effectively use the Automation Checkerboard, it is important to understand the positioning and the moves towards realizing the benefits. The aim would be to move from the current position to the top right corner to derive the maximum benefits.
Starting on the grid
The first step for any project is to position itself on the checkerboard. This is based on the current state of the project. If the project is just about starting and in its infancy, then the left most corner is the starting position. However, if the project is already in progress, and some percentage of automation has already been developed, then the squares on the right would be the starting position.
Knight’s move
At the start of the automation initiative, it is very intuitive to start building on the quantum of automation for early benefits. At this level, we need to address the aspects of technology stack based on the needs of the test suite. The right choice of the automation toolset as well as the framework is required to implement automation. A good automation framework provides an abstraction layer that removes the technical nuances of the automation tool and provides with higher productivity of automation. This move will attempt to move the effectiveness north by one position while the quantum of automation produced will move it east by multiple positions. A classical Knight’s move delivers the horsepower to gain on the pace of automation development and deliver effectiveness even with smaller percentage of automation development.
Rook’s move
The design of the components and their reusability is the key to success. Each test scenario needs to be analyzed and the business process components need to be defined. Doing this early in the testing lifecycle enables proactive and progressive automation. The right design and underlying framework enables development of automation scripts even while the application development is in progress. This is most beneficial in Agile projects and is probably a true implementation of ‘Shift left’. By designing the business process components well, each component can be automated once, and reused across multiple test scenarios as well as across multiple phases of the testing lifecycle, increasing the overall effectiveness with the same quantum of automation development. A classical Rook’s move looks at moving north by improving the maturity of automation and delivering higher benefits.
Bishop’s move
The Bishop’s move is to move diagonally and bring in continuous improvements. This move is required to improve the performance and speed of execution. Improving the speed of execution and enabling unattended execution helps in continuous refresh of the regression pack. This requires integration with the test management tool, typically HP’s ALM. The other aspect that needs to be taken care of is the preconditions for each test execution to be automated and creation/organization of test data required for execution. In this way, the tests can be scheduled to run automatically at a predetermined time, and the results of the execution are auto-populated back into ALM. Continuous integration into the development environment helps improve the effectiveness, right from code build to smoke and regression tests done in an automated and unattended way.
Queen’s move
The Queen’s move is the most critical action towards the success of automation. It is used to address regular maintenance. The automated test suite should be run regularly (nightly, unattended) and analyzed for failures. By running the automated pack regularly, the maintenance needs are caught much early and provide sufficient time for the team to make updates to the automated regression pack. This is most beneficial in an agile environment when things can change on a daily basis and the automated executions can catch issues very early in the lifecycle.
Having a continuous running regression engine will enable tests to be pushed in and out, and the results continuously monitored. The outdated tests can be purged out on a regular basis and newer tests should be defined, automated and added to the existing pack. This will ensure that the automated regression pack is at all times up-to-date and most-effective.
Conclusion
Automation is dynamic by nature. It can get outdated in no time. For it to deliver benefits, it needs to be constantly monitored and refreshed. Design of components ensures reusability. Choice of tools and frameworks enables speed and agility. Continuous integration helps increase the benefits and at the same time keeps the automation relevant. Ensuring early maintenance gives the desired long-term benefits.
The automation checkerboard provides a structured mechanism and gives a two dimensional view towards deriving the maximum benefits from the investments into automation. The checkerboard and starting point is representative of the current state of the project and at what time the investments into automation have begun. The different moves on the checkerboard are metaphors for levers that need to be engaged to derive maximum benefits.
There are multiple openings for a game. It’s the right set of moves that make you win.
Girish Kulkarni is the Global Practice Head for Test Automation, Testing Services. With two decades of experience in Software development and testing, he has played various leadership roles at Wipro. A business leader with successful results in delivering projects for global Financial Services customers, he has rich and diverse experience – from driving transformation programs through automation to Program management, Product development, Business enablement and Global Account management.
Being an avid reader, creative thinker, painter, sculptor, cook and a gardener, he likes to indulge in various hobbies and activities.
The speedy SaaS momentum can not only be attributed to a tight economy but also to the associated low Total Cost of Ownership
DevOps enables the IT organization to break out of silos, embrace collaboration, and inject that elixir of agility to drive business success.
With OSS becoming a complicated and critical function for CSPs, the benefits that can be realized by migrating OSS on the cloud has also significantly amplified.
