Measuring accessibility

There has been much discussion, and some arguments, about how to determine the accessibility of websites. Unfortunately, this is often polarised around two simplistic choices: A compliance/conformance based approach that usually involves a checklist of criteria; or, some form of user testing by people who have different disabilities and/or who rely on different assistive technologies. Both approaches have their strength and limitations, and neither can provide a reliable declaration about the accessibility of a site on its own.

For the individual web user, the accessibility of a site depends on many factors and how they interrelate. The obvious starting point are the personal barriers to web access that the user faces; these might for example, be technological or environmental limitations, or a physical disability that might necessitate the use of an assist device, or cognitive, learning or language problems that make the content of a page hard to understand.

Next, we need to consider the actual quality of the website code as well as the ability of the user-agents (such as browsers and assistive technologies) to present the content of the page in a way that the user can perceive. And finally, how skilful the user is in using the browser and/or assistive technology they rely on to access the web. With regard to this last point, it is often erroneously assumed that most people know how to use the accessibility features of a browser or computer operating system, and all assistive technology users are expert users of their technology.


Over the years, the W3C Web Accessibility Initiative (WAI) have developed sets of guidelines to help codify what is required to produce and render accessible web content, including:

Many web developers are aware of WCAG and some strive to produce content that complies with these guidelines. However, few are aware of ATAG and more importantly UAAG. Since conformance with the UAAG is largely beyond the control of developers, even well meaning and very dedicated developers cannot guarantee the content they produce will be fully accessible to all.

User-agents like screen readers rely on accessibility APIs, for example Microsoft Active Accessibility (MSAA), to expose objects, roles and states within the content, for example to identify a checkbox and whether or not it has been checked. However, this only works when the accessibility API is recognised by the user-agent(s) and this is not always the case.

This problem has been compounded by the rapid advances in web content technologies and techniques over the last few years and relative slowness by user-agents to keep up with these advances. Consider for a moment the increasing adoption of WAI-ARIA (Accessible Rich Internet Applications) and HTML 5, both of which offer some exciting accessibility features. The dedicated, and some might say valiant, work by Steve Faulkner from the Paciello Group over the years has highlighted the variable support by browsers and screen readers for some of these advanced features. For example,

  • ARIA Role Support (March, 2009) outlines how MSAA exposes ARIA roles by different browsers
  • HTML5 Browser Support (July, 2011) contains a table, which is regularly updated, that provides a good indication of how well new HTML 5 features are accessibility supported by browsers
  • JAWS Support for ARIA (October, 2010) documents how JAWS (10+) supports ARIA.

Evaluation tools

There are a wide range of free and not-so-free tools that can help determine compliance with accessibility guidelines by individual web pages or a collection of page. For example, and in no particular order of preference:

  • Web Accessibility Toolbar (WAT) contains a wide range of tools to aid manual examination of web pages for a variety of aspects of accessibility.
  • WebAim WAVE online tool for use with single pages. It is quick to use and provides results which are easy to understand
  • HiSoftware Compliance Sheriff contains an Accessibility Module that enables automated monitoring for site wide compliance
  • Deque Worldspace FireEyes for accessibility compliance testing of static and dynamic web content. An Enterprise version is also available
  • Total Validator online tool for validating pages against accessibility guidelines a Pro version is available for site wide testing

Although all these tools are useful and I use some of them regularly, in my opinion none can be replied up alone to reliably indicate either the degree of compliance with a specific set of guidelines or the overall accessibility of web page(s). The results obtained by automated testing tools like these need to be interpreted and confirmed by human evaluators.

Testing times

The risk of litigation, combined with political and moral pressure, has focussed increasing attention on the importance of ensuring websites are accessible. As a result, site owners, designers and developers now face the tasks of deciding what is the most efficient and reliable way of evaluating the accessibility of their sites.

As mentioned earlier, there are two general approaches for determining website accessibility: conformance review and user-testing. Ideally, any thorough accessibility evaluation should involve both approaches, but the constraints of budgets and time often mean that this is not possible. One feature common to both approaches however, is the importance of using an experienced accessibility evaluator who has an understanding of the potential barriers that people with disabilities might face and how these can be addressed. I now want to briefly consider the pros and cons of the two approaches.

Conformance review

Conformance reviews are the most common way of assessing the accessibility of websites. In general, this involves someone with expert knowledge checking whether the site as a whole, or more commonly a selection of pages, comply with a predetermined checklist of criteria such as WCAG 2.0. The assessment process is sometimes also referred to as a ‘manual inspection’ or ‘expert review’.

The selection of pages to evaluate is very important. Giorgio Brajnik and others from Universita di Udine, Italy reported in the paper “Effects of Sampling Methods on Web Accessibility Evaluations” (PDF) on a study that showed the use of predefined pages (e.g. home, contact, site map etc) may result in an inaccurate compliance result for 38% of checkpoints.


  • Able to identify a large and diverse range of non-compliance issues that might cause problems for a variety of potential end users and/or technologies.
  • The checklist items often provide a clear indication of what is required to rectify non-compliance.
  • Easy to incorporate into the different phases of the site development process and this can be particularly useful in an agile or iterative development environment.
  • Relatively quick and easy to implement when compared to user testing.


  • Totally dependent on the quality of the checklists or guidelines used. In my opinion however WCAG 2 is pretty good in this regard.
  • Tendency to view accessibility from just the perspective of whether or not a site passes or fails a number of checklist items, and may fail to adequately consider how easily or effectively the site might be for people with disabilities to use.
  • Particular difficulty with issues that blur the boundary between usability and accessibility, for example site structure (e.g. is it shallow or deep), which can be particularly relevant to older web users or those with cognitive disorders.
  • Does not involve real users doing real tasks in real time.

User testing

User testing usually involves a group of users with different disabilities, and different levels of skill in using the internet and their required assistive technology, undertaking a series of typical website tasks. The actions of the test participants are observed (and recorded) by the evaluator with the aim of identifying the accessibility barriers that maybe encountered.

Although task-based user testing for usability and accessibility share some common techniques, there are significant differences. For example, context is likely to be more important when evaluating accessibility since web content often has to go through transformation processes in order to be accessible. The output of these transformations (e.g. text alternatives for images, text to speech via a screen reader, speech to text as captions, magnification, extraction of links and headings on the page etc) has to be rendered in a way that retains the meaning and integrity of the original content while also meeting the needs of a diverse user base.  In the article “Beyond Conformance” (PDF) Giorgio Brajnik from Universita di Udine, Italy, explores the differences between usability and accessibility and why context is crucial when evaluating accessibility:

“Context is more crucial for accessibility than it is for usability. Besides being dependent on users’ experiences, goals and physical environment, accessibility of a website depends also on the platform that’s being used. It is the engine of a transformation process that is not under the control of the developer. In fact, accessibility of digital media requires a number of transformations to occur automatically, concerning the expression of the content of the website”


  • Evaluator is able to observe people encountering (and hopefully overcoming) real usability/accessibility problems in real time.
  • Accurately identify problems that actually prevent specific groups of people from accessing web content.
  • Test participants are able to rate the severity of the problems they encounter and identify those that are likely to be catastrophic.
  • Likely to generate highly valid results for people who have the same disabilities.


  • Difficult to recruit test participants with different disabilities.
  • Hard to obtain a test cohort that is large enough to canvas the range of assistive technologies and participant skill levels in using those technologies
  • Depends greatly on developing test scenarios (scripts) which are appropriate for test participants with different requirements.
  • Difficult to correlate and prioritise the problems encountered by a diverse group of people with different requirements who use different technologies
  • The testing process is expensive and time consuming

How inaccessible is it?

Many governments and organisations now require websites to be accessible. In most cases, compliance with this requirement is determined by conformance, either formally or informally, with predetermined guidelines/rules such as WCAG 2.0 or the US Section 508. In Australia for example, the Australian Government Information Management Office (AGIMO), requires all government agencies to comply with WCAG 2 at level AA by the end of 2014, and the Australian Human Rights Commission has indicated that it will use WCAG 2 when considering the validity of a complaint made under the Commonwealth Disability Discrimination Act. I looked in more detail at the adoption of WCAG 2 by various countries in an earlier post, “Adopting WCAG 2” (June, 2009).

WCAG 2 provides for three levels of compliance, Level A, Level AA and Level AAA, with the recommendation not to require Level AAA conformance “as a general policy for entire sites because it is not possible to satisfy all Level AAA Success Criteria for some content”. As a result, most jurisdictions that use WCAG 2 require websites to conform at either Level A or Level AA.

It appears that many regulators appear to adopt a pass or fail approach to the use of Guidelines, Checkpoints or Success Criteria and don’t factor in the potential severity of non-compliance with individual criterion. Nor is the likely impact of non-compliance with different criteria compared. To take an extreme example: A site which fails to provide text alternatives for all images, fails to comply with the Level A Success Criteria 1.1.1, as does a site with just one or two missing image alts on say one page.

Is a site which fails to comply with two Success Criteria any less inaccessible than one that fails to comply with five? And, what about a site that fails badly to comply with the Level AA Success Criteria 1.4.3 (e.g. has a contrast minimum of 1.5:1 for navigation items), should we consider this to be more or less accessible than another site that contains minor infringements of just a couple of Level A Success Criteria?

In a future article, I plan to look at how both the incidence and likely impact (severity) of accessibility barriers might be incorporated into the accessibility conformance review process.


  1. I can’t help but think that your post is based on a HTML-centric view of the Web. WCAG 2.0 made it clear that it was format-neutral and so there is a need to decouple Web accessibility from accessibility of HTML and related W3C standards.

    Take, for example institutional repositories which host large numbers of PDFs of research papers (739,000+ across UK repositories). How should one ‘measure’ the accessibility of such services, which play an important role in many universities?

    The UK has developed the BS 8878 code of practice for Web Accessibility which focusses on the processes surrounding the development, deployment and maintenance of Web sites which affect accessibility issues. I feel that this provides a more realistic approach to enhancing the accessibility of Web services. Note that myself and David Sloan have just submitted an abstract to the WAI Metrics online symposium in which we describe how metrics can be used in the context of BS 8878 which explores approaches described in a blog post on Web Accessibility, Institutional Repositories and BS 8878 and expanded further in a post on Metrics, This Time For Web Accessibility.

    If brief, I agree with Giorgio Brajnik’s comment that “Context is more crucial for accessibility than it is for usability“. WCAG fails to provide a contextual approach so there is a need for a standardised approach which can address such contextual issues. As described in a post on BS 8878: Applying a Level of Redirection to Web AccessibilityThere isn’t a problem in computer science which can’t be solved by adding a level of redirection“. I feel that approaches such as BS8878 can provide that level of redirection, and build on WCAG’s strengths whilst allowing its limitations to be addressed.

    • Thanks for the comment Brian, but I am not sure why you feel that I am trying to promote a HTML-centric view of the web. Like you, I agree one of the strengths of WCAG 2 is technological neutrality – see an earlier post “WCAG 2 and Accessibility Supported“. My main aim was to look at the way discussions of how to determine the accessibility of web content often revolves around two approaches, and I know these are not the only ways of assessing accessibility but they are very commonly used. Many thanks for the links in your comment to some informative and useful resources.

  2. Conformance review strikes me as a subset of Expert Assessment. An expert isn’t limited to following checkpoints and guidelines in doing an assessment. So I’m curious about why the limit expert assessment to conformance reviews and then flag that limitation as a con.

    I tend to use guidelines as a suggested starting point, and evaluating the site as a whole while working through the core use-cases the site is intended to support. That way we can spot accessibility issues or potential headaches that a set of guidelines can easily miss. So using guidelines to inform, not limit the assessment. That’s given me, and my colleagues, a deeper and more meaningful understanding of accessibility issues.

    • Isofarro, thanks for the comment. I agree conformance review should be a subset of an expert accessibility assessment. However this is often not the case, and when it isn’t, sole reliance on whether or not something complies with a selection of predetermined checklist criteria can present problems. This is why I suggest we shouldn’t rely on either of the two approaches I describe alone and why I raised a couple of questions at the end of the article.

      I have seen sites prepared by highly competent developers being declared “inaccessible” solely on the basis of one or two relatively minor (and largely inconsequential) instances where a WCAG 2 criterion has not been complied with. On the other hand, I have also heard developers and site owners declare a site to be fully “accessible” just because one person didn’t encounter any problems when using it with a screen reader.

  3. Thanks for your response – and for your link to your “WCAG 2 and Accessibility Supported“ post. I remember reading that when it was published :-)

    I guess my concern is that your post describes a resource-oriented view on accessibility, with the implication that accessibility issues can be addressed by fixing the resource (whether its an HTML resource, PDF, Flash, …). Note I’m not saying that’s what you think, rather that promoting this view can lead to policy-makers focussing soley on WCAG-conformance (I feel this is a concern as I described in a post on Government Web Sites MUST Be WCAG AA Compliant!)

    The accessibility papers published by several accessibility researchers and practitioners primarily in the UK are based on the notion that accessibility is caused primarily by a mismatch between the user and the user’s environment and addressing just one access of the user’s environment (the digital resource) is not necessarily the most appropriate response.

    Our work began in 2004 in exploring e-learning accessibility. We argued that the need was for accessibility learning outcomes, and not necessarily accessibility e-learning resources. We later extended our work to address the accessibility of cultural resources – what dos it mean to “understand” a Salvadore Dali painting, for example.

    We feel that BS 8878 may provide a solution by addressing a bigger picture, and going beyond the resource.

    Our papers are available at Note that the paper on Reflections on the Development of a Holistic Approach to Web Accessibility summarised developments offer ideas from 2004-2008.

    In addition my blogs posts on Web accessibility are available at

  4. Roger, this is exactly the topic I have been mulling over lately. Missing an ALT attribute in one case can be so trivial it isn’t even annoying, but in another case it can be an absolute barrier to accomplishing a task. That is is the core problem with basing the definition of accessibility on adherence to items on a checklists. On the other hand, doing a series of tests using assistive technologies can miss important issues, such as when the AT “guesses” labels when they are not included.

    The approach I have been taking has been to identify defects, according to checklists, and then use AT to determine the severity. Severity is a combination of impact and frequency of occurrence. I then get a reality check from users on the severity score. This approach not only gives you a good overall view of the usability or the site or application, but also generates a prioritized list of what is most important to fix.

    I am looking forward to the next article!

  5. I always perform both. I fancy it gives me a pretty good picture of a system’s accessibility. My challenge now is how to convey that information to my audience: government customer and IT. In both cases it is very hard to convey the fact that it is a shaded picture; that it is not a simple pass/fail situation; that if you focus on the goals, there are many solutions to the same problem; etc. This is not what they like to hear, and the nuance may be taken as a license to do nothing. IO would welcome any discussion or advice in this area…

  6. I think there is a lot of common sense ioevlvnd with making Web sites the right way BUT unfortunately its not that easy. We really do need guidance from people who are professionals in that area.Have you seen the latest draft of WCAG 2.0? Joe Clark’s article was based on the WCAG 2.0 Draft from last year. Joe has said he was impressed with how much the working group listened to his comments. Plus the thoughts of the development community have been pretty positive over the latest draft.

Leave a Reply

Your email address will not be published. Required fields are marked *