Is manual testing dead?

Personal disclaimer - I work in a team where every developer follows TDD and is capable of writing browser automation tests in BDD style using Cucumber/Ruby. We have a manual tester on team and no automation tester. Other teams around me that I closely interact with have not manual testers on team but only automation testers.

Update: 03/05/2014

Recently Uncle Bob Martin wrote a post titled When TDD does not work. Uncle Bob is a veteran in the field of TDD and his article mostly revolves around unit testing. But the ideas he expresses can be extrapolated to automataion in general. It is ressuring that my thoughts here are inline to a great degree with what is trying to convey in his article.

TL;DR; - The world of programming is very fast moving towards automating everything under the sun. But IMHO we are far from calling manual testing dead. 

The objective of the article is not to reason about/against the above argument. Automation is a big investment and is good investment on most counts but it may not pay off all the time. It can be useful understand in what situations that investment does not pay off. There is no thumb rule to these situations, these are purely based on my experience working with software I have built over the years. But before we get into that, lets try to understand what does it cost to automate a test (or use case, user journey, scenario, whatever you name it). Formally this is called  "Total Cost of Ownership" of automating a test. This consists of following

  1. Cost of automating a test
  2. Maintaining the automated test over time as your code changes
  3. Cost of manually testing edge cases not covered by the automated test

This inherently indicates that manual testing is still not our of order, but lets talk on that later. The first point is interesting - cost of automating a test. If you are automating simple tests through unit tests or integration tests, then this cost is very minimal. The investment here pays off very well and you can afford to automate a large number of tests. The reduced cost is a direct outcome of maturity of modern IDEs and testing tools. Today, you can hit the ground running with a unit or integration test in no time. It does not take long time to build and run tests. It is becoming more and more easier to repeatedly run these tests on more than one machines (thus making continuous integration cheaper). These tests are almost entirely written in the same language that the software under test (SUT) is written in. That makes it even easier. Maintaining these tests as SUT changes is not hard. 

Things become little difficult when we move towards browser automation tests that let you automate complete user journey. Agreed, this field is no more new and lot of advancement has happened in last few years in this area. We now have very stable tools like Waitr and Selenium WebDriver that make programatically interacting with browsers a breeze. But there are still some challenges 
  1. A different set of skills are required to build these tests. If your team is new to this kind of testing then they would take longer time in the beginning to automate simple user journeys
  2. If the SUT changes, then it takes longer to make changes to these tests
  3. In my experience, people prefer using Ruby for automation. So if you are a .NET or Java developer then you need to invest in cost of learning a new language
  4. These tests are not as reliable as unit tests are. They may be flaky and lot of times their reliability depends on factors beyond just tests e.g. load on the machine, response of the web pages etc.

By now, you would have some clarity on what I mean by TCO. A test might be cheap to automate in the first place but maintaining it over time may be expensive.

Another important point of view when it comes to automated tests is how is the faring in detecting defects or how badly the test is hiding defects. As your code changes, your tests either do not change, change or become completely irrelevant. If your tests are not changing then it is possible that they are failing to detect defects or hiding defects. If a test becomes completely irrelevant after a code change then the investment you made in automating the test ceases. From this standpoint, it would be worthwhile to ask yourself, how frequently the code being tested is going to change. You might need to talk to BAs and product owners to get a clear answer to this but if you are confident that code is likely to change a lot in near future then you need to be judgmental about how far you go with automation (unit tests are exempt from this. I always write unit tests no matter what)

Having said all of the above, I feel, there are situations when we come to a conclusion that manual testing is a better option. Below is a list of such situations from my experience

  1. Testing long running batch jobs - It is difficult to completely automate the testing of long running batch jobs. The core logic of the batch job can be unit tested but automating a single end-to-end run of a batch job, though possible, is not feasible and not worth the investment. Even if attempted, in my experience, it does not give enough confidence about the quality of the software and we resort to some level of manual testing in the end
  2. Testing email content and delivery - Email delivery can be tested to some extent by writing emails to disk instead of delivering them but then you are not really testing delivery. Also testing content of the email and how they render in browser based email clients vs. desktop based emails clients is something that cannot be automated reliably enough. Moreover, there is no one standard when it comes to desktop and mobile phone clients. Combinations are just too many and investment in automating this with tools available today is not worth it.
  3. Testing of user interfaces - Browser automation tools confirm the present of a particular element on page but they cannot verify the look and feel of the element. There is some movement happening in this area and people are experimenting with different tools. But again, given the maturity of these tools, automating such testing is not reliable. It is best left for human eyes. Here is a list of tools that let you automate testing of visual aspects of a web page by the way.
  4. Browser compatibility testing - Gone are the days when everyone on the planet used IE6. People now use tens of different types of browsers and every product owner wants his product to work on any browser out there in the market. They do not want to loose revenue because we did not support a browser that a prospective client was using. So testing software on vast range of browsers is inevitable. If you have got some browser automation tests then you can go a step further an run you test in multiple browsers. There are tools like BrowserStack that let you do exactly that. But this approach is not scalable given the time it would take to run all your tests on all possible browser combinations. And usually, the number of user journeys automated are not large enough to build confidence that software works in all browsers. 
  5. Testing responsiveness of website - "Responsive" is the buzzword today. Every website you build has to be responsive (well, not every site, but most sites).  Testing responsiveness of a website is very difficult to automates for the fact that you would need to render the site on different devices in order to see how your pages are scaling. You can use websites that render your site in different device resolutions to show you how your pages scales but automating that is not very easy. This kind of activity is mostly preferred to be done manually on real devices.
  6. Testing user journeys spanning multiple systems - Enterprise software usually have more than one component intricately communicating with each other. In order to reliably tests such systems, data flowing through one component into another needs to be verified. It is possible to break the problem down into logical units and test each unit with support of unit tests. But again, you feel the need of one test that runs an end-to-end user journey and ensures that everything is where it belongs. Automating such tests would be time consuming. Such tests would also take a long time to run.
Each of the above situations is something I have experienced. For some, I have attempted automating tests, frustrated and given up. For others, I had to rely on manually testing after getting issues from production users. All this makes me feel that manual testing is not dead. Rather role of manual testing has become more important. Automation has made it difficult for simple defects to creep in by mistake. What is there in the software are the defects that are not easy to find. So manual testers of these days have a huge responsibility of finding defects that automated tests have hidden. 

Suhas Chatekar

Discussion

  • Comment with Disqus
  • Comment with Facebook
comments powered by Disqus