The Accessibility Conformance Report (ACR) based on the ITI VPAT® is the leading global reporting format for assisting buyers and sellers in identifying information and communications technology (ICT) products and services with accessibility features. Version 2 of the VPAT was expanded to include the leading ICT accessibility standards: Section 508 (U.S.), EN 301 549 (EU), and W3C/WAI WCAG.
(https://www.itic.org/policy/accessibility/vpat)
For the most part, the VPAT is used by Government Agencies in the Executive Branch of the US Federal Government as a way to verify the accessibility of ICT products and services they purchase. Such agencies are required to do so under Section 508 of the Rehabilitation Act and so, in the process of doing their market research, they may ask for documentation from vendors to self-declare their level of conformance.
Unfortunately, the VPAT format is lacking one important trait: ease of use. Through no fault of ITI, the standards reported on in the VPAT are now much more robust than earlier versions and, therefore, the testing and reporting conformance with those standards is now more complicated. A new organization called Standard Accessibility Reporting (SAR) has been formed to work on a new reporting methodology that’s easier to understand, but for now the VPAT is the go-to format.
I’ve put together this guide to reviewing a VPAT ACR for those who might be new to reviewing such documents to (help) determine a product’s accessibility. This is not, however, a definitive guide, but is mostly a way to quickly determine if things are amiss with the ACR before taking the time to review the product’s accessibility yourself.
Review the metadata
Often the first signs that things are not right will show up right in the various metadata at the beginning of the document.
Document Version
The first thing to check is what version of the template is being used. As of this writing, the current VPAT version is 2.4 which was released in March 2022. While there are no critical differences between the 2.x versions, an older version might indicate that the information in the ACR is out of date.
Document Date
Like the above, you should check the date on the ACR. Websites, web-based software, desktop software, and mobile applications rarely go unchanged for long, and you should expect an ACR to reflect the current state of conformance for the product it covers. If you’re given an ACR document more than a year old, I recommend asking the supplier to update it or give a very solid reason why the document is so old. E-learning content and hardware can often go unchanged for a long time, but almost no other ICT product does.
Product Description
Does the product description match the information of the product that you are reviewing? Beyond the obvious – that you want to make sure the ACR is for the right product – you will also want to make sure it matches the final features and configuration of the product that you’re buying. In some cases, suppliers will limit the scope of the VPAT to only include certain product features or configuration options. That might be listed here, or it might be listed in “Notes and Exceptions”, below. If the description does not match the thing you’re buying, ask the supplier for an ACR that does cover what you’re buying.
Contact Information
Verify that the contact information is complete and correct. You should also make sure that the information provided is for a person who can provide more information for the product’s accessibility, because it is very likely that you will need to ask this person questions later. Often this information is for someone in sales or marketing, who are the last people that you’ll want to ask about accessibility. However, for bigger vendors it can sometimes be their internal legal counsel. There’s not much you can do about that since accessibility is a compliance concern.
Note and Exceptions
If there’s any content provided under “Notes and Exceptions”, read it very carefully. As I mentioned earlier, you want to make sure that they’re not exempting anything from the ACR. In some cases, they will disclose things here which may be important to keep in mind as you review the ACR. For example, they might state that certain parts of the product are not covered by the ACR. In that case, you should make a strong request for a new ACR that covers those portions as well.
Evaluation Methods
Under evaluation methods used, make sure that the content in this section accurately conveys that a robust methodology was used in evaluating the product. Naturally what determines “a robust methodology” depends on the type of product but at the very least the narrative described here should give you confidence that they’ve understood the accessibility requirements for that type of product and described the mechanisms used to test the product’s conformance to those requirements.
Review the Conformance Declarations
Once you have verified the quality of the metadata, you can transition to looking into the individual conformance declarations. Those are the long, scary, detailed tables in the ACR in which the supplier will declare their product’s level of conformance to each individual requirement of the chosen standard(s). As the saying goes, “the Devil is in the details” and it is very important to ensure that this information is clear, informative, and honest. Unfortunately, many ACRs suffer on all 3 points.
Completeness
The first concern, when reviewing the conformance declarations, is whether the information is complete. This means:
- There’s an entry in the column for “Conformance Level” for every provision/ success criteria of the standard(s) they’re reporting conformance on. That entry must be only one of: Supports, Partially Supports, Does not support, or Not Applicable.
- There’s content within every cell in the “Remarks and Explanations” column for every provision/ success criteria of the standard(s) they’re reporting conformance on.
It isn’t enough to provide content in the “Conformance Level” column, the ACR must also describe how or why that Conformance Level was chosen. This is exactly why “Remarks and Explanations” exists.
Disclosure
The “Remarks and Explanations” column exists to provide the supplier with an opportunity to give honest disclosure to the reasoning behind answer provided within “Conformance Level”. Within “Remarks and Explanations” you want to look for two things:
1. Does the content in that cell accurately reflect an understanding of what the success criteria is about?
2. Does the content in that cell honestly disclose the nature and extent of any shortcomings or specifics about how they support the success criteria?
This can be your best indicator of whether the product is accessible or not, because if this information does not clearly reflect an understanding of the goals of the success criteria, that can also mean they probably did not accurately test the product.
Use these 4 criteria for determining each of the Remarks and Explanations:
- When they say that their product supports a specific success criterion, have they accurately conveyed the ways that it supports that success criteria?
- When they say it partially supports a success criterion do they list specific and easy-to-understand information about what exceptions exist for that success criteria?
- When they say it does not support the success criteria, do they accurately say why it does not support it and the extent to which it does not support?
- When they say that a success criterion is not applicable, do they accurately and informatively describe why it is not applicable?
The content within the Remarks and Explanations column need not be exhaustive but should instead be a concise narrative that explains the justification for their stated Conformance Level.
Harmony
Diving a little deeper, you should verify that no information is in conflict. For example, some Section 508 conformance criteria and EN 301 549 conformance criteria map to one or more WCAG Success Criterion. In cases where they do not map to WCAG, they might still map to each other. Ensure that the content in “Remarks and Explanations” matches in those cases where they do map to each other.
Their Disclaimers
Be sure to review any legal disclaimers provided in the ACR document very carefully. This is an area where companies will attempt to disclaim any level of accessibility and will also attempt to avoid making any claims or warranties around the product. What this, means in laypersons terms, is that they are not standing by the information that they’ve provided in the ACR.
As a buyer, you want the ability to compel them to stand by the claims in their ACR. If they’ve stated that their product is conformant, but you later find out they failed to disclose some accessibility issues, you want to be able to point to the statements made in the ACR. Having heavy-handed disclaimers in the ACR will prevent you from doing that.
Conclusion
An ACR is one of the fastest ways to gain insight into a product’s conformance with major industry standards for accessibility. Due to the high-stakes nature of compliance with laws that reference those standards, you should never simply trust vendor’s claims outright. That said, an accurate, clear, and transparent ACR is a valuable tool when reviewing potential ICT purchases. Unfortunately, ACRs can be difficult to fill out and difficult to review. Hopefully the information above can be helpful in doing a careful review of ACRs prior to spending the time reviewing the product yourself.