Is manual testing dead?
Personal disclaimer - I work in a team where every developer follows TDD and is capable of writing browser automation tests in BDD style using Cucumber/Ruby. We have a manual tester on team and no automation tester. Other teams around me that I closely interact with have not manual testers on team but only automation testers.
Recently Uncle Bob Martin wrote a post titled When TDD does not work. Uncle Bob is a veteran in the field of TDD and his article mostly revolves around unit testing. But the ideas he expresses can be extrapolated to automataion in general. It is ressuring that my thoughts here are inline to a great degree with what is trying to convey in his article.
TL;DR; - The world of programming is very fast moving towards automating everything under the sun. But IMHO we are far from calling manual testing dead.
- Cost of automating a test
- Maintaining the automated test over time as your code changes
- Cost of manually testing edge cases not covered by the automated test
- A different set of skills are required to build these tests. If your team is new to this kind of testing then they would take longer time in the beginning to automate simple user journeys
- If the SUT changes, then it takes longer to make changes to these tests
- In my experience, people prefer using Ruby for automation. So if you are a .NET or Java developer then you need to invest in cost of learning a new language
- These tests are not as reliable as unit tests are. They may be flaky and lot of times their reliability depends on factors beyond just tests e.g. load on the machine, response of the web pages etc.
Another important point of view when it comes to automated tests is how is the faring in detecting defects or how badly the test is hiding defects. As your code changes, your tests either do not change, change or become completely irrelevant. If your tests are not changing then it is possible that they are failing to detect defects or hiding defects. If a test becomes completely irrelevant after a code change then the investment you made in automating the test ceases. From this standpoint, it would be worthwhile to ask yourself, how frequently the code being tested is going to change. You might need to talk to BAs and product owners to get a clear answer to this but if you are confident that code is likely to change a lot in near future then you need to be judgmental about how far you go with automation (unit tests are exempt from this. I always write unit tests no matter what)
- Testing long running batch jobs - It is difficult to completely automate the testing of long running batch jobs. The core logic of the batch job can be unit tested but automating a single end-to-end run of a batch job, though possible, is not feasible and not worth the investment. Even if attempted, in my experience, it does not give enough confidence about the quality of the software and we resort to some level of manual testing in the end
- Testing email content and delivery - Email delivery can be tested to some extent by writing emails to disk instead of delivering them but then you are not really testing delivery. Also testing content of the email and how they render in browser based email clients vs. desktop based emails clients is something that cannot be automated reliably enough. Moreover, there is no one standard when it comes to desktop and mobile phone clients. Combinations are just too many and investment in automating this with tools available today is not worth it.
- Testing of user interfaces - Browser automation tools confirm the present of a particular element on page but they cannot verify the look and feel of the element. There is some movement happening in this area and people are experimenting with different tools. But again, given the maturity of these tools, automating such testing is not reliable. It is best left for human eyes. Here is a list of tools that let you automate testing of visual aspects of a web page by the way.
- Browser compatibility testing - Gone are the days when everyone on the planet used IE6. People now use tens of different types of browsers and every product owner wants his product to work on any browser out there in the market. They do not want to loose revenue because we did not support a browser that a prospective client was using. So testing software on vast range of browsers is inevitable. If you have got some browser automation tests then you can go a step further an run you test in multiple browsers. There are tools like BrowserStack that let you do exactly that. But this approach is not scalable given the time it would take to run all your tests on all possible browser combinations. And usually, the number of user journeys automated are not large enough to build confidence that software works in all browsers.
- Testing responsiveness of website - "Responsive" is the buzzword today. Every website you build has to be responsive (well, not every site, but most sites). Testing responsiveness of a website is very difficult to automates for the fact that you would need to render the site on different devices in order to see how your pages are scaling. You can use websites that render your site in different device resolutions to show you how your pages scales but automating that is not very easy. This kind of activity is mostly preferred to be done manually on real devices.
- Testing user journeys spanning multiple systems - Enterprise software usually have more than one component intricately communicating with each other. In order to reliably tests such systems, data flowing through one component into another needs to be verified. It is possible to break the problem down into logical units and test each unit with support of unit tests. But again, you feel the need of one test that runs an end-to-end user journey and ensures that everything is where it belongs. Automating such tests would be time consuming. Such tests would also take a long time to run.