Before you read this, I want you to spend a few moments looking around where you are right now. Ponder, for a moment, the furniture you’re sitting on, the room you’re sitting in, and the computer you’re reading this on. All of it is the result of technological innovations that have a history that goes back almost 3 million years. That’s the point (as far as we know) that hominids began creating tools.  About 1.8 million years ago hominids began making structures that can be called houses.

Innovation happens to be a logarithmic thing. 1.8 million years is a long time.  But anatomically modern humans (homo sapiens) emerged around 300,000 years ago. We started exhibiting “modern behavior” around 150,000 years ago. We began creating art 40,000 years ago and began fishing around the same time. Agriculture began around 10,000 years ago, math around 8,000 years ago, and our first civilizations appeared 6,000 years ago.

The organization of humans into civilizations kicked off a massive number of innovations in increasingly short (relative) time periods:

  • Wheel: 4000 BC
  • Writing systems: 3500 BC
  • Copper: 3200 BC
  • Bronze: 2500 BC
  • Salt: 2500 BC
  • Chariot: 2000 BC
  • Iron: 1500 BC
  • Sundial: 800 BC
  • Glass: 500 BC
  • Cast iron: 400 BC
  • And so on

As time progresses, the period between major advances in science and technology shortens. It took humans 20,000 years to progress from fishing nets to fishing hooks, but 58 years to go from the first successful airplane flight to putting a human into space.

Tools have always been about accomplishing tasks – either doing things that we couldn’t do before or doing things that we could do much more efficiently.  Sure, we could remove the hide from an animal with our hands before eating it, but a sharpened piece of flint made it far easier.  Axes and spears were effective at hunting, but a bow and arrow or harpoon was better. Sure, a person could grind grain using stones to make bread for their family, or we could grind the grain at scale using waterpower to make bread for our entire community.

If you’re reading this article, it means you’re using a computer connected to the WorldWide Web – something that didn’t exist until 1989 and now drives the global economy in one way or another.   In 34 years, the Web has gone from non-existent to completely revolutionizing the entire world. All aspects of life have been changed due to the Web, even in under-developed nations.

What does this have to do with Accessibility?

Automated systems arguably go back to a few hundred years BC. Mechanical feedback-based controllers were critical components for the machines that drove the Industrial Revolution and the use of automation has been vital in performing operations at scale and with reliable quality. Automation reduces human effort when performing time-intensive, monotonous tasks. Automation increases the predictability of output. Automation increases quality and accuracy.

Again, take a look around you right now and ponder the things around you:

  • In 1877, telephone calls were routed by hand, but automated in 1892
  • Chemicals for medicines were mixed by hand, but automated in 1930
  • Your drinking glass would have been made by hand were the process not automated in 1905

But perhaps most relevant to our discussion are the clothes that you’re currently wearing. The weaving of fabric and cloth has existed for a few thousand years with the tools evolving along the way, with loom technology really taking off during the Industrial Revolution. In 1804, Joseph Marie Jacquard patented a loom that was controlled by a “a number of punched cards laced together into a continuous sequence. Multiple rows of holes were punched on each card, with one complete card corresponding to one row of the design.” This use of replaceable punched cards to control a sequence of operations served as inspiration for Charles Babbage’s Analytical Engine.

The proliferation of these automated weaving machines led to skilled textile workers rising up in protest, going so far as destroying automated textile machines. These protesters called themselves Luddites. The Luddites feared that the time spent learning the skills of their craft would go to waste, as machines would replace their role in the industry.

All of this takes me to the sentiment from some in the Web Accessibility field suggesting that “Automated testing won’t solve web accessibility”.  The evidence purported to support such a claim is a blog post that demonstrates that a handful of free automated testing tools are not able to find every possible accessibility problem.

Automation is the future, and it always has been

It is true that there are a lot of shortcomings when it comes to automation of something like accessibility testing. In my talk titled “There is no such thing as a normal user” I make the argument that the number of combinations of operating systems, browsers, input devices, output devices, assistive technologies and possible user impairments make it impossible to make too many concrete assumptions about the user. This also means that fully relying on automated testing is a bad idea.

But the statement “automated testing won’t solve web accessibility” is, frankly, absurd. On one hand, it is almost too obvious to bother saying. But on the other hand, it is absurd because it implies that automated testing is bad and should not be used. I would argue that the opposite is true: it is already good and we’d better hope that automated testing gets better and its use more prolific.

By some estimates, there are 1.5 billion websites on the Web and the Web is becoming the singularly dominant platform across ICT systems.  Developers on the most popular sites on the Web commit tens-of-thousands of code changes per day. Currently there are around 4400 members of the International Association of Accessibility Professionals (IAAP).  Even if the real number of people doing accessibility testing across the world is 100x that, there are still over 18.5 million developers committing code that must be tested. This is a job that cannot be done manually.  Not with the current talent pool, and likely not with any eventual talent pool. If there was ever a job for automation, this is it.  Not only should we embrace it, we should celebrate it.

Let’s contrast manual vs. automated evaluation, in terms of time. Prior to our acquisition by Level Access, Tenon, LLC also performed accessibility audits. The testing of each component reliably ranged from 1.5 to 2.5 hours.  In that time period, Tenon’s testers utilized an extremely robust testing methodology that included a checklist of around 250 distinct check items. Put another way, each of Tenon’s testing staff could cover just under 3 accessibility checks per minute (2.78, to be exact)

With a reasonable utilization target of 70%, you can expect a worker to have around 5.5 hours of productive time per day, or 330 minutes.  This means that they can do around 917 accessibility checks per day.

Contrast that with the capabilities of an automatic testing tool. Tenon.io, for example had 189 tests that evaluated over 2600 distinct failure conditions. Across millions of pages, it averaged about 4 seconds to test each page – and Tenon is widely regarded as being slow. It could still evaluate 20 full web pages in the same time that it took a human to test one component. Not only could it evaluate that page in 4 seconds, but it could also log and report an average of 60 issues per page and calculate robust statistics on those findings.

Sadly, it bears mentioning that no automated accessibility testing tool on the market today can fully test for every possible accessibility challenge. In fact, some WCAG Success Criterion cannot be tested for using any automated means available now. That said, it is still irresponsible to act as though automated testing is bad and should be avoided.

A while ago, a competing vendor started making the claim that their tool could find over 70% of accessibility issues, then made hand-wavy explanations of how they came to that conclusion. I decided to do my own research. What I found, based on Tenon’s data, is that my own findings from my blog post “What can be tested and how” from 2012 is still accurate – but that there’s a pretty big difference between “possible” and “likely”, and Tenon.io was able to find 59% of likely accessibility issues.

To arrive at this conclusion, Tenon’s staff helped me map the individual tests from Tenon.io to the checkpoints in Mortise (our manual testing platform). After performing that mapping, we looked at the things that were/ were not found by most systems in the wild. What we found was that of the data logged in Mortise, 59% of them could be directly mapped to a Tenon test. No matter how you slice the data, anywhere from 1/2 to 2/3 of likely accessibility issues can be found using entirely automated means. Put together, the efficiency and coverage of automated testing means it is nigh irresponsible not to use automated testing.  Not only that, but every single time you perform automated testing on the same system, the positive benefits from efficiency multiply. This makes automated testing vital for the long-term management of accessibility in an organization.

The future is coming sooner than you think

One recent comment I received on a LinkedIn post said:

Yeah, the automatic testing revolution will happen in “a few years”. As it was foretold every year for the last 20 years.

This statement conveniently ignores piles of innovation in this space over the last 28 or so years:

  • Free web-based testing tool, Bobby, came out around 1995 providing quick accessibility testing a page at a time.
  • Desktop applications, such as APrompt (2002), AccVerify(2003), and InFocus (2003) are released, which can spider and test entire websites
  • WAVE, a single page tester from WebAIM is released in 2002 and is significantly better than Bobby
  • Web Accessibility Toolbar (WAT) provides in-browser testing for IE in 2005
  • WAVE toolbar is released in 2008.
  • In 2009, SSB BART Group merged InFocus with its internal auditing tool to create the Accessibility Management Platform (AMP), which directly merged InFocus’s automated results into manual audit projects
  • In 2010 Deque releases FireEyes, a plugin for Firebug that synchronizes results with Deque’s flagship product, Worldspace, and includes the ability to create Selenium-style scripts to trigger accessibility testing along the way.
  • In 2014, Tenon is released to private beta. It is the first SaaS product in accessibility. Having an externally facing API it was also the first to:
    • Have a Node module
    • Have Grunt & Gulp plugins
    • Have a WordPress Plugin
    • Have a Drupal Plugin
    • And more
  • In 2015, Deque releases aXe into Open Source. It is later incorporated into Google’s Lighthouse and Microsoft’s Accessibility Insights and has plugins to work with numerous testing and linting products.
  • In 2020 Evinced launches its tool, which uses AI for portions of its testing.

And along the way everyone, collectively, got better and better at testing.

The skepticism and negativity in the statement “Yeah, the automatic testing revolution will happen in “a few years”. As it was foretold every year for the last 20 years” ignores how far we’ve come.

  • We’ve gone from testing document source as a string to analyzing the rendered DOM
  • We’ve gone from testing a single page at a time to testing entire websites
  • We’ve gone from static analysis to dynamic analysis
  • We’ve gone from automatic testing on its own to having automatic testing accompany manual testing.
  • We’ve gone from being a separate tool that developers use after release to integrating into the tools they use while developing.
  • We’ve gone from being a tool that can only test after content is published to being able to prevent publishing of inaccessible content in the first place

And we continue to push further.

What the future holds

The LinkedIn status I posted which inspired the skeptical response above was this:

It is a common belief that automated testing for accessibility isn’t enough. At present, there’s a near universal belief that manual testing is needed to fully test for accessibility. I believe that to be the case still, but I also believe that we’re only a few years away from being able to significantly reduce the amount of manual testing needed.”

I stand by this statement because I believe that innovations in web accessibility testing will follow the similar logarithmic pattern that technology, in general, does.  This is especially true because the companies creating automated accessibility testing tools are bigger than they’ve ever been and can innovate more rapidly.

In the next few years, I predict:

  • We will start seeing testing happening during design, before code is even written
  • We will start seeing real-time feedback in developer IDEs and content management systems
  • While it is already possible to integrate with testing frameworks and build pipelines, this integration will be much easier
  • While it is already possible to track duplicate issues across a website, we will start seeing pattern recognition that allows you to see the actual source of the issue. We will even be able to predict what manual issues will occur based on the automatically discovered issues
  • While it is already possible to (somewhat) predict what manual tests are relevant based on the automatic testing, this will get much better and save manual testers a ton of time by not having them test irrelevant stuff

Finally, because of the last two bullets above, we will be able to not only gain higher coverage automatically but do so to such a degree that it will make manual testing so much more efficient that it’ll be better described as semi-automated.

While I currently cannot foresee a time when there’s no longer a need for skilled manual testing and other skilled professionals throughout the design & development process, I do foresee a time that everyone’s job becomes much easier and faster as a result of the advancement of tools.  That’s something we should all embrace.

Note: Significant portions of this post’s discussion around the history of tools and automation (especially the dates) are heavily borrowed from Wikipedia.. If that kind of thing interests you, I also heartily recommend Bill Bryson’s “A short history of nearly everything“, a book I re-read quite often.

My company, AFixt offers full accessibility services from testing, training, consulting, and remediation. If you need help, get in touch with me now!