There has been much discussion, and some arguments, about how to determine the accessibility of websites. Unfortunately, this is often polarised around two simplistic choices: A compliance/conformance based approach that usually involves a checklist of criteria; or, some form of user testing by people who have different disabilities and/or who rely on different assistive technologies. Both approaches have their strength and limitations, and neither can provide a reliable declaration about the accessibility of a site on its own.

For the individual web user, the accessibility of a site depends on many factors and how they interrelate. The obvious starting point are the personal barriers to web access that the user faces; these might for example, be technological or environmental limitations, or a physical disability that might necessitate the use of an assist device, or cognitive, learning or language problems that make the content of a page hard to understand.

Next, we need to consider the actual quality of the website code as well as the ability of the user-agents (such as browsers and assistive technologies) to present the content of the page in a way that the user can perceive. And finally, how skilful the user is in using the browser and/or assistive technology they rely on to access the web. With regard to this last point, it is often erroneously assumed that most people know how to use the accessibility features of a browser or computer operating system, and all assistive technology users are expert users of their technology.

Guidelines

Over the years, the W3C Web Accessibility Initiative (WAI) have developed sets of guidelines to help codify what is required to produce and render accessible web content, including:

• Web Content Accessibility Guidelines (WCAG 2.0), which is concerned with the preparation of website content
• Authoring Tool Accessibility Guidelines (ATAG 1.0), which is concerned with the authoring tools (including CMSs) that can be used to prepare content
• User Agent Accessibility Guidelines (UAAG), which is concerned with how the browser, media players, assistive technologies etc interact with the content

Many web developers are aware of WCAG and some strive to produce content that complies with these guidelines. However, few are aware of ATAG and more importantly UAAG. Since conformance with the UAAG is largely beyond the control of developers, even well meaning and very dedicated developers cannot guarantee the content they produce will be fully accessible to all.

User-agents like screen readers rely on accessibility APIs, for example Microsoft Active Accessibility (MSAA), to expose objects, roles and states within the content, for example to identify a checkbox and whether or not it has been checked. However, this only works when the accessibility API is recognised by the user-agent(s) and this is not always the case.

This problem has been compounded by the rapid advances in web content technologies and techniques over the last few years and relative slowness by user-agents to keep up with these advances. Consider for a moment the increasing adoption of WAI-ARIA (Accessible Rich Internet Applications) and HTML 5, both of which offer some exciting accessibility features. The dedicated, and some might say valiant, work by Steve Faulkner from the Paciello Group over the years has highlighted the variable support by browsers and screen readers for some of these advanced features. For example,

• ARIA Role Support (March, 2009) outlines how MSAA exposes ARIA roles by different browsers
• HTML5 Browser Support (July, 2011) contains a table, which is regularly updated, that provides a good indication of how well new HTML 5 features are accessibility supported by browsers
• JAWS Support for ARIA (October, 2010) documents how JAWS (10+) supports ARIA.

Evaluation tools

There are a wide range of free and not-so-free tools that can help determine compliance with accessibility guidelines by individual web pages or a collection of page. For example, and in no particular order of preference:

• Web Accessibility Toolbar (WAT) contains a wide range of tools to aid manual examination of web pages for a variety of aspects of accessibility.
• WebAim WAVE online tool for use with single pages. It is quick to use and provides results which are easy to understand
• HiSoftware Compliance Sheriff contains an Accessibility Module that enables automated monitoring for site wide compliance
• Deque Worldspace FireEyes for accessibility compliance testing of static and dynamic web content. An Enterprise version is also available
• Total Validator online tool for validating pages against accessibility guidelines a Pro version is available for site wide testing

Although all these tools are useful and I use some of them regularly, in my opinion none can be replied up alone to reliably indicate either the degree of compliance with a specific set of guidelines or the overall accessibility of web page(s). The results obtained by automated testing tools like these need to be interpreted and confirmed by human evaluators.

Testing times

The risk of litigation, combined with political and moral pressure, has focussed increasing attention on the importance of ensuring websites are accessible. As a result, site owners, designers and developers now face the tasks of deciding what is the most efficient and reliable way of evaluating the accessibility of their sites.

As mentioned earlier, there are two general approaches for determining website accessibility: conformance review and user-testing. Ideally, any thorough accessibility evaluation should involve both approaches, but the constraints of budgets and time often mean that this is not possible. One feature common to both approaches however, is the importance of using an experienced accessibility evaluator who has an understanding of the potential barriers that people with disabilities might face and how these can be addressed. I now want to briefly consider the pros and cons of the two approaches.

Conformance review

Conformance reviews are the most common way of assessing the accessibility of websites. In general, this involves someone with expert knowledge checking whether the site as a whole, or more commonly a selection of pages, comply with a predetermined checklist of criteria such as WCAG 2.0. The assessment process is sometimes also referred to as a ‘manual inspection’ or ‘expert review’.

The selection of pages to evaluate is very important. Giorgio Brajnik and others from Universita di Udine, Italy reported in the paper “Effects of Sampling Methods on Web Accessibility Evaluations” (PDF) on a study that showed the use of predefined pages (e.g. home, contact, site map etc) may result in an inaccurate compliance result for 38% of checkpoints.

PROS

• Able to identify a large and diverse range of non-compliance issues that might cause problems for a variety of potential end users and/or technologies.
• The checklist items often provide a clear indication of what is required to rectify non-compliance.
• Easy to incorporate into the different phases of the site development process and this can be particularly useful in an agile or iterative development environment.
• Relatively quick and easy to implement when compared to user testing.

CONS

• Totally dependent on the quality of the checklists or guidelines used. In my opinion however WCAG 2 is pretty good in this regard.
• Tendency to view accessibility from just the perspective of whether or not a site passes or fails a number of checklist items, and may fail to adequately consider how easily or effectively the site might be for people with disabilities to use.
• Particular difficulty with issues that blur the boundary between usability and accessibility, for example site structure (e.g. is it shallow or deep), which can be particularly relevant to older web users or those with cognitive disorders.
• Does not involve real users doing real tasks in real time.

User testing

User testing usually involves a group of users with different disabilities, and different levels of skill in using the internet and their required assistive technology, undertaking a series of typical website tasks. The actions of the test participants are observed (and recorded) by the evaluator with the aim of identifying the accessibility barriers that maybe encountered.

Although task-based user testing for usability and accessibility share some common techniques, there are significant differences. For example, context is likely to be more important when evaluating accessibility since web content often has to go through transformation processes in order to be accessible. The output of these transformations (e.g. text alternatives for images, text to speech via a screen reader, speech to text as captions, magnification, extraction of links and headings on the page etc) has to be rendered in a way that retains the meaning and integrity of the original content while also meeting the needs of a diverse user base.  In the article “Beyond Conformance” (PDF) Giorgio Brajnik from Universita di Udine, Italy, explores the differences between usability and accessibility and why context is crucial when evaluating accessibility:

“Context is more crucial for accessibility than it is for usability. Besides being dependent on users’ experiences, goals and physical environment, accessibility of a website depends also on the platform that’s being used. It is the engine of a transformation process that is not under the control of the developer. In fact, accessibility of digital media requires a number of transformations to occur automatically, concerning the expression of the content of the website”

PROS

• Evaluator is able to observe people encountering (and hopefully overcoming) real usability/accessibility problems in real time.
• Accurately identify problems that actually prevent specific groups of people from accessing web content.
• Test participants are able to rate the severity of the problems they encounter and identify those that are likely to be catastrophic.
• Likely to generate highly valid results for people who have the same disabilities.

CONS

• Difficult to recruit test participants with different disabilities.
• Hard to obtain a test cohort that is large enough to canvas the range of assistive technologies and participant skill levels in using those technologies
• Depends greatly on developing test scenarios (scripts) which are appropriate for test participants with different requirements.
• Difficult to correlate and prioritise the problems encountered by a diverse group of people with different requirements who use different technologies
• The testing process is expensive and time consuming

How inaccessible is it?

Many governments and organisations now require websites to be accessible. In most cases, compliance with this requirement is determined by conformance, either formally or informally, with predetermined guidelines/rules such as WCAG 2.0 or the US Section 508. In Australia for example, the Australian Government Information Management Office (AGIMO), requires all government agencies to comply with WCAG 2 at level AA by the end of 2014, and the Australian Human Rights Commission has indicated that it will use WCAG 2 when considering the validity of a complaint made under the Commonwealth Disability Discrimination Act. I looked in more detail at the adoption of WCAG 2 by various countries in an earlier post, “Adopting WCAG 2” (June, 2009).

WCAG 2 provides for three levels of compliance, Level A, Level AA and Level AAA, with the recommendation not to require Level AAA conformance “as a general policy for entire sites because it is not possible to satisfy all Level AAA Success Criteria for some content”. As a result, most jurisdictions that use WCAG 2 require websites to conform at either Level A or Level AA.

It appears that many regulators appear to adopt a pass or fail approach to the use of Guidelines, Checkpoints or Success Criteria and don’t factor in the potential severity of non-compliance with individual criterion. Nor is the likely impact of non-compliance with different criteria compared. To take an extreme example: A site which fails to provide text alternatives for all images, fails to comply with the Level A Success Criteria 1.1.1, as does a site with just one or two missing image alts on say one page.

Is a site which fails to comply with two Success Criteria any less inaccessible than one that fails to comply with five? And, what about a site that fails badly to comply with the Level AA Success Criteria 1.4.3 (e.g. has a contrast minimum of 1.5:1 for navigation items), should we consider this to be more or less accessible than another site that contains minor infringements of just a couple of Level A Success Criteria?

In a future article, I plan to look at how both the incidence and likely impact (severity) of accessibility barriers might be incorporated into the accessibility conformance review process.