In my 10 year career doing usability & accessibility, I’ve used tools all the way back to Bobby back before CAST sold it to Watchfire. I’ve used all the big expensive tools, all the free tools and toolbars. In fact, it was once my job to know what each tool could and could not do. With my change in employers two months ago, I’m no longer employed by a company that makes or sells tools and my new employer has no pecuniary interest in any company which does. I’d like to take an opportunity now for a rant. Specifically, I’d like to rant about the current breed of automated accessibility testing tools, be they free or paid. This is not, in any way, aimed at any specific vendor including any of my former employers. If you make an automated tool, this rant’s for you. Apologies in advance for the change-in-tone compared to my normal posts.
Five Simple Criteria
I am frequently asked by clients, potential clients, or people in the accessibility community which tools I’d recommend for them. My answer, historically, has usually been a very diplomatic “well, it depends on what your needs are”. That’s still the answer, but from now on, I will base my direct recommendations based on the following 5 simple criteria.
Test the Browser DOM
I can’t say this often enough or loudly enough: If your tool doesn’t test web documents as rendered in a browser, it isn’t testing the right thing. No, using jDOM or PHP’s various DOM extensions or anything that “makes” a DOM isn’t the same. The product must reside in the browser or use a headless browser and must use the information from the browser DOM to do its testing. As a tool vendor you should know why these two are not the same thing and why the browser DOM is vital. If you don’t know why this is important or refuse to acknowledge it, get out of the tool business. I absolutely won’t recommend an automated testing tool which doesn’t test the DOM.
Provide Fast, Accurate, and Reliable Results
This is why people buy your tool: They want to do testing and get fast, accurate, and reliable results. They don’t want to sift through false positives and they don’t want to have to “scrub” the reports to filter out repetitive, vague, inaccurate, irrelevant, erroneous, or misleading results. Unfortunately, this is what happens with most tools. Over time this has gotten better, but it still sometimes feels like tool vendors want to offload the work to the user. This is inconvenient, at best, and in my experience is the most often cited reason why tools become abandonware.
Understand this: Absent any real expertise on the part of the user of the tool, they will explicitly trust the results they get from a tool. If I can’t trust that the tool will give my client accurate and reliable results, I won’t put my name on a recommendation.
Make the Tool User-Friendly
This should go without saying: Tools which are difficult to use won’t get used. In sales, the cost & time of gaining a new customer is always higher than getting repeat business from an existing customer. So if your business model is based on license sales, it would seem smart to make a tool that the customer wants to renew their license for. Surprisingly it seems most tool vendors don’t quite get this. Here’s a hint: If your users are complaining that the tool is hard to use, then you need to fix it. The customers aren’t stupid. They don’t need more training, they need a tool that is easy to use. If I don’t have confidence that my customers will be able to use the tool successfully, I won’t recommend it.
Bug-free Features that Work as Stated
No software is ever perfect. Anyone who expects perfection is unrealistic. In fact, that’s what these tools are meant to find out, right? So it is understandable that at times tools will have bugs. But if specific features do not work as intended – or don’t work at all – that calls into question your entire approach to quality. This is especially true in cases where blatant bugs to core features go ignored. Not caring about quality is akin to not caring about your customers, in my opinion.
Responsive Customer Support
In cases where a customer needs help, is experiencing problems using or understanding the product, or has a bug, how do you handle it? Given the above – that no software is perfect – how you support customers with problems says a lot. If I recommend a tool to someone and they’re given poor customer support, that would make me look bad.
Ultimately Quality is Key
All five of these criteria are aimed at one thing: Quality. For me to give out a recommendation for any product – not just an automated accessibility testing tool – I have to feel comfortable that I’m steering people to a vendor and a product which has a firm commitment to quality, not just for their product but for their related services and support. I’m not going to vouch for anything less. That’s my promise to my customers, potential customers, peers, and members of the accessibility community.