Skip to content

We need more choices in automated testing

For most of the past decade, one project has been at the center of automated accessibility testing. It made accessibility checks a normal part of continuous integration, brought useful findings into developer tools, and established a reporting format that developers, auditors, and procurement teams immediately recognize.

Its contribution is hard to overstate. The web is more accessible because of it, and nearly everyone working in digital accessibility today has benefited from that work. I want to make that clear at the outset, because this is not a complaint about what that project has done. It is an argument for what the field needs next.

We need more choices

When one engine becomes dominant, its capabilities can start to define the boundaries of the practice around it. The checks it performs well are treated as the important ones. Everything else gets put in the "manual testing" bucket. Before long, coverage is judged against what the tool can detect rather than what the underlying accessibility standard requires.

That is a problem because automated tools still detect only a fraction of the issues involved in a real conformance evaluation. Many requirements depend on context or intent. Others require someone to understand what the content is supposed to communicate, compare the result with the purpose of the design, or simply use judgment. No rule running inside a web page can handle all of that. But a healthy ecosystem would include several serious efforts to narrow the gap between what our tools currently test and what accessibility standards actually ask of us. Right now, there are very few.

The lack of competition has also made some obvious limitations feel normal. One example is evidence. Most tools identify an element, give it a selector, and link to an explanation of the rule. That is useful for debugging, but it is not enough for an auditor, regulator, or designer who needs to review or challenge the finding. A stronger result would include a screenshot, a stable identifier, a structured reference to the relevant standard, and a record that can be compared across multiple runs. Those details would make it possible to measure progress instead of repeatedly discovering the same problems. They would also make disagreements about a finding far more productive.

Remediation is another weak spot. A report may explain why an element failed without showing what a passing version would look like. Developers need more than a description of the problem. They need practical guidance they can apply. Remediation should be a core part of the result, not an afterthought in the documentation.

Our handling of standards could also be much better. WCAG, Section 508, EN 301 549, RGAA, BITV, JIS, and similar standards have real structure. They contain principles, guidelines, criteria, levels, and versions. Yet many testing systems reduce all of that to a collection of string tags. That approach may have been convenient initially, but it now limits what teams can do with the data. If engines represented standards as structured information, reporting systems and dashboards could work with that information directly. Teams responding to procurement requirements would not have to begin by parsing naming conventions.

The division between automated and manual testing is similarly artificial. Accessibility practitioners do not experience these as unrelated activities. Automated checks, guided reviews, AI-assisted analysis, and human judgment are all parts of the same evaluation process. They should be able to share the same metadata, examples, references, and result format. Instead, practitioners often have to assemble a workflow from several unrelated products, each with its own terminology and data model.

There is also more to a reliable testing system than running rules against a page. Browser management, queues, duplicate detection, baselines, and result storage all affect whether the findings can be trusted. Engines have traditionally left those responsibilities to the applications built around them. That was understandable when the engines were smaller and implementations were highly customized. It is less convincing now.

The same is true of AI. The useful question is not whether an accessibility engine should “use AI.” The important question is how that capability is designed. There is a major difference between an engine that happens to call a particular model and one that defines a contract any suitable model can fulfill. A good design would also support deterministic replay during testing and preserve the evidence needed for a person to review the model’s conclusion. That is a concrete engineering problem, and the field would benefit from treating it as one.

None of this means the dominant project has failed. It became successful for good reasons: it is focused, fast, permissively licensed, widely localized, and careful about false positives. Those qualities are difficult to reproduce, and they deserve more credit than they sometimes receive. The issue is not that one project should have solved every problem. It is that we should not expect one project to do so. More competition would give the field room to test different ideas about coverage, evidence, remediation, standards, workflow, and judgment. It would encourage tools to address more of the requirements that currently fall outside automated testing. It could also help accessibility results become useful not only as developer feedback, but as durable evidence for audits, compliance work, and procurement.

Competition would be good for the established tools as well. A monoculture may be comfortable for a while, but it rarely serves the incumbent – or its users – well in the long term. The project that brought automated accessibility testing this far deserves real credit. At the same time, the people building digital products, and the people who rely on those products being accessible, deserve more than a market built around a single engine. There is room for several serious projects with different ideas about what accessibility testing should cover and how its results should be used. It is time for more of them.

Published inAccessibilityTesting