Should you use more than one automated accessibility testing tool?
If you’re already aware of Betteridge’s Law, then you know the answer already.
There are some that would argue that you need to use multiple tools because automated accessibility tools can’t find everything and because each tool takes its own approach to testing – including what they specifically test for. This sounds spot on, but misses the point about automated testing entirely.
I believe the primary benefit to automated testing is the efficiency and repeatability they add to the test process. Using multiple tools for testing will only add work, thereby reducing the efficiency benefit of using the tool. In addition, irrespective of the quality & accuracy of each tool, the other problem that arises is differing guidance. Even if they found the same things, each tool will use different words to describe the issues. For instance Tenon will say “This form field is missing a label” and AMP will say “Provide explicit labels for form fields”. While they’ve both found the same issue the user now needs to interpret the messages to determine if they are pointing out the same thing. Add the fact that each tool may find different things, may apply different severities, and may provide different guidance and now you’re really losing efficiency because now you need to determine which tool is right and where. Now you are adding a lot of work for very little benefit.
We already know that there’s only so much that automated testing tools can find. Automated testing isn’t the end but rather the beginning of the process. Tools aren’t a replacement for expert review but rather a supplement to it. Using > 1 tool doesn’t close that gap effectively and instead adds unnecessary work that’s probably time better spent with manual testing. The best approach is to find a tool you like, become an expert user of it, become familiar with how it works (including its shortcomings) and use it.