Research shows adhering to WCAG doesn't solve blind users' problems

Accessibility
1

http://dl.acm.org/citation.cfm?id=2207736 there’s a PDF in there, about 10 pages (but only 7 have what you want to read).

So, some researchers got together and were all like, “If more websites conform to WCAG, does this actually decrease the number of problems blind users run into on them?” and went to find out. Turns out it didn’t seem to matter. Or, it mattered sometimes, but didn’t stop a lot of problems because they aren’t problems even mentioned in the WCAG.

First, the study:
They used 32 blind volunteers to test. These people did not represent the whole swath of blind users: instead I’d say they prolly cluster near the middle of the Gaussian curve. They were majority

  • JAWS users
  • IE users
  • regular Internets users (not n00bs or grandmas)

A group of sites were put together for testing*. The researchers had trouble finding AAA pages so they just wanted to know, what’s the difference in problems the blind testers ran into on non-conforming (to either WCAG1 or WCAG2) pages vs pages that technically conform? Now, there were fewer of certain types of problems on conforming pages. Pages who conformed more, had fewer of certain types of problems which were covered in the WCAG and the sites had conformed to those guidelines.

*List of tested sites:
www.lflegal.com (Lainey Feingold’s site)
www.green-beast.com (Mike Cherim’s site, though I’m not sure he still updates it)
www.york.gov.uk
www.nhsnss.org
www.copac.ac.uk
www.theaa.com
www.dh.gov.uk
www.digizen.org.uk
www.jisc.ac.uk
www.royalmail.com
www.pret.co.uk
www.tuc.org.uk
www.britishmuseum.org
www.nhsdirect.nhs.uk
www.ford.co.uk
www.ticketmaster.co.uk

Some sites were conforming to WCAG, some weren’t. They’re listed in order of most-conforming to Fails.

But as a web developer, you’d want to look specifically at Table 2, which lists most of the main problems people had.

What are those problems?

  1. Content found in pages where not expected by users
    Total user problems: 99
    Covered by WCAG2? No.

  2. Content not found in pages where expected by users
    Total user problems: 88
    Covered by WCAG2? No.

  3. Pages too slow to load
    Total user problems: 27
    Covered by WCAG2? No.

  4. No alternative to document format (e.g. PDF)
    Total user problems: 17
    Covered by WCAG2? No.

  5. Information architecture too complex (e.g. too many steps to find pages)
    Total user problems: 15
    Covered by WCAG2? No.

  6. Broken links
    Total user problems: 10
    Covered by WCAG2? No.

  7. Functionality does not work (as expected)
    Total user problems: 50
    Number Covered by WCAG2: 13

  8. Expected functionality not present
    Total user problems: 29
    Number Covered by WCAG2: 10

  9. Organisation of content is inconsistent with web conventions/common sense
    Total user problems: 39
    Number Covered by WCAG2: 14

  10. Irrelevant content before task content
    Total user problems: 86
    Number Covered by WCAG2: 39

  11. Users cannot make sense of content
    Total user problems: 65
    Number Covered by WCAG2: 44

  12. No/insufficient feedback to inform that actions has had an effect
    Total user problems: 72
    Number Covered by WCAG2: 49

In other words, a majority of problems faced by these disabled volunteer testers weren’t specifically accessibility-related: they were usability problems, which hit everyone (though people with strong disabilities get hit harder and it takes them longer and more effort to overcome the problems). The majority of these problems seem they could have been fixed not with someone testing the pages for accessibility and blind adherence to the WCAG spec, but by user testing. And they could have hit most of these problems testing “regular” users; that is, you don’t need to be able to find 30 blind people to find these problems: your grandmother and her bridge club would’ve found them just as easily.

Example of one of the problems (too much crap nobody cares about before the content the user’s actually seeking):

I remember we had this problem on many of our insurance pages. Where did the BS text come from? It was someone’s attempt at having “content” for the freakin search engines. Oh I could rage for days about the crap people throw on websites “for the googles”. I’m going to start saying this hurts your websites. Don’t listen to the SEO craptastic advice about what’s good for search engines. Getting people onto your page via SEs only for them to leave because your page is crappy is… a waste of everyone’s time. And electricity. And bandwidth. And the blood of wee orphan children.

There was one disability-related issue that the study mentioned at length: when there was a video, the blind testers preferred a separate audio track describing what went on in the videos over a text description. Personal preference, one which both the developer and the website budget would have to keep in mind. Having a text description fulfilled a WCAG requirement, but users had (in this study) clear preference of one fulfilling method over another.

Anyway the point of the study was to see if the number of problems people encountered went down as sites went from non-conforming to WCAG1 conforming, or from WCAG1 to WCAG2 (which was supposed to be clearer to developers on what to do… testing on developers with little accessibility knowledge seems to show WCAG2 is almost as confusing and mis-interpreted as WCAG1).

They also then state their opinion,

WCAG is about accessibility and the people who wrote it have a generally narrow definition of “accessible”. Or, if they’re going to add more usability stuff to it, the name should reflect it and the definition of “disabled” needs to be broadend. I agree that you can’t really call something “accessible” if it’s not usable. Kinda defeats the whole purpose of building something in the first place. It’s like spending time making broken toys.

What I would rather see happen: that web developers, bosses, and budgets get off their duffs and actually bother doing real user testing on their sites and applications. Yup, it’s no easy automated test a machine can do, but damn, it’s practically guaranteed to increase profits if that’s what your site or application does. Sheesh.

And what’s easier? Getting grandma to try our your page/application, or reading the WCAG? If you have limited time and have to choose between one or the other, which one will bring you greater results and the most improvements, for all users? I’ll claim the grandma. Ideally you’d do both, but nobody lives in an ideal world, we live in a crap world and it’s just going to suck either way.

What do you think?

2

I guess a bit of an update/extras:
Brian Kelly believes WCAG “needs a wrapper which provides contextualisation” and goes further in [url=http://ukwebfocus.wordpress.com/2012/05/09/introducing-bs8878-on-global-accessibility-awareness-day-gaad/]a post on UK Web Focus. He mentions [url=http://www.hassellinclusion.com/bs8878/]BS 8878 as an example of how web developers can make good use of WCAG2 guidelines.

3

All this seems a bit excessive to me.

Remember our talk about how to convey to screen reader users a message, where I started devising “impressive” tech solutions like the use of strong in an out of context manner?

You said: why not simply say what you have to say. And I couldn’t agree more. Simple as that.

<hr>

Going back to the issue: why not simply doing it, instead of preparing technical solutions prone to fail?

Why not simply have a built in section for accessibility. Literally. Like the <nav> section. In it, describe the page and put navigation for it, for accessibility. An accessible short presentation and ToC. Accessible by default in UAs by ESC key. Your accessibility safe heaven in every website.

Further more, why not Google helping with accessibility in its SERPs? Doesn’t it care for this market share? Forget SEO algorithms! Shouldn’t it provide data about each link and about each website in accessibility terms too? And I don’t mean the level of accessibility the website has employ, but actual data helping special needs users specifically to choose from and make sense of SERPs.

Why not a SE for special needs users alone? A branch in each Google, Yahoo, Bing, perhaps, if not. Like m.* only acc*. Are web devs the only ones that have to comply? What about UAs forgetting about -webkit and implementing -accessibility?

4

I don’t think that’s true. Web accessibility, like accessibility in general, is about providing access that is equal or equivalent to that provided to people who do not have a disability.

Poor usability is only a web accessibility issue if it discriminates against people with a disability. WCAG is not trying to address bad design principles and development techniques per se, just those that place people with a disability at a disadvantage. A site that is AAA compliant with WCAG is by no means guaranteed to be free of usability problems, or even to be particularly well-designed.

It’s quite possible that site visitors without impaired vision have as much trouble with these sites - and sites in general - as blind people do. Perhaps compliance has only brought these sites’ usability issues to the attention of blind people, whereas the same sites in non-compliant mode might have been totally inaccessible.

1 Like
5

Ricky, Poes didn’t write that quoted section that was an “article” citation. :slight_smile:

What she actually said was she thought that term the [article] used was too “narrow” and that real human testing should also be done - if possible. Hence the hypothetical “granny” with age related problems or possible lack of computer knowledge regardless hence a perceived barrier.

6

I don’t think that’s true:

What is Web Accessibility
Web accessibility means that people with disabilities can use the Web.

7

The authors of the study believe WCAG should cover usability more, since bad usability affects the disabled (and generally harder than non-disabled).

I’m not sure if (more) usability should be in the WCAG (maybe we need a WCUG instead), but I don’t believe unusable sites can ever be considered “accessible.” They’re broken.

Correct. Simple mechanical adherence to WCAG does not ensure your site is usable, or therefore accessible, to real human beings.

Since the researchers were surprised that the results showed that sites going from non-compliance to WCAG, or from WCAG1 to WCAG2, did not have fewer problems, this tells me that they expected the majority of problems encountered by specifically the blind to be WCAG-covering accessibility issues. It turned out here the problems were at least half basic usability and outside the scope of the current WCAG. And, things a company/organisation could probably find with a grandma and her bridge club (it’s pretty easy for a company to claim they can’t afford to find 30 blind people for testing, and it would indeed be difficult and expensive. But cheaper usability testing with almost as much value can be had with the example grandma.

Yup. Exactly my point. I too am frustrated with pages that don’t get to the point, broken links and crappy information architecture. And I do not rely on a screen reader.

In general the blind don’t bother splitting hairs. They try to use a site. The site doesn’t work. If they’re not WCAG members or web accessibility people, they don’t care if the problems they hit are in the WCAG or not. It doesn’t matter. Broken is broken.
On a mailing list I’m on, people will post websites and ask where the problem is: is it something on the page, or is it the browser, or is it the screen reader?

8

In agreement on all of that, Slimme Poes (stom ben je niet).

Having read the article thoroughly, I do think the authors have made a few baseless assumptions, held expectations that were unwarranted and drawn some dodgy conclusions.

My point is that WCAG is not meant to address usability issues, and there’s no reason to expect a site that is WCAG-compliant to be especially usable. itmitică’s quote from the W3C only points out the shortsightedness (ahem) of such definitions.

I think WCUG would be very helpful indeed, but we’d still need WCAG to address disability-specific accessibility issues.

All of this does portend very well what I expect to happen, which is that owners, designers and developers will expect WCAG-compliant sites to pass usability muster - and they will be disappointed.

9

There’s some more discussion about the study over at the WebAIM mailing list …and it seems the study’s PDF is now behind a paywall, for those who wanted to read it now.

10

There does seem to be this mentality in all of web development where people think that simply following all the rules makes them accessible; AND at the same time we have the opposite where you have the folks that use articles like this one to throw it all in the trash.

It’s a bit like fundamentalists vs. science really. In the same way Asimov said “It is not so much that I have confidence in scientists being right, but that I have so much in nonscientists being wrong.” – it comes down to those who follow the WCAG are more likely to come up with a good page than those who ignore it.

But fairytale-land pseudo-science – or in our case bad web design, is still rubbish regardless of what rules you followed to get there. Following rules, guidelines, and other ideas put together by hard-thinkers simply makes you more likely to succeed at the task; it’s NOT a guarantee.

That said, a lot of those sites are absolute rubbish; and broken here since many of them are fixed widths with fixed height elements using dynamic fonts. /FAIL/ hard… of course with many of them being turdpress templates this is hardly a shocker, given what a total accessibility train wreck that is in the first place!

First three I looked at of those sites had broken layouts, paragraphs around non-paragraph elements, endless nesting DIV for christmas only knows what reason, improper heading orders, willy-nilly/nonsensical use of HR, and on the whole were poster-children for bad development techniques. All of those mean bad accessibility, and the WCAG covers exactly NONE of that.

11

Detlev Fischer has written an article outlining problems with the study’s methodology and metrics (comments welcome) and stating that, with the methods of the study being unclear much of the time, the conclusion that WCAG-compliant sites do not reduce problems encountered by the blind may itself be suspect (also that of course that the blind in general cannot represent all disabled users, while the WCAG does).

He also states that the WCAG may indeed have something to say regarding some of the problems listed as “Not covered by WCAG”. For example, problems #1 and #2 (Content found in pages where not expected by users, and content not found in pages where expected by users) could be covered by sections 2.4.4 and 2.4.6 (that the links users clicked told them correctly what the next page would have on it, and that the headings of these pages would correctly head and outline the page content, leaving few surprises for users).

Quote from the conclusion:

12

This reflects the current thrust of disability rights advocacy which is definitely towards “equivalent access” and this concept is now enshrined in the UN Convention on the Rights of Persons with Disabilities, the Telecoms Package and other recent policy/regulation/legislation developments. The concept is not completely simple, however. To use an analogy, what is “equivalent access” to a building for a person in a wheelchair? It implies at least that they can get in to all the rooms, use all the facilities, operate all the light switches, etc. that everyone else can. But does it imply that they have to be able to do this as quickly or as comfortably as non-wheelchair users? Is that even possible (without artificially slowing down the non-wheelchair users)?

In the case of websites, equivalent access would not seem to require that blind users should find the information architecture simple, that the functionality should work as they expect it or that there should be no irrelevant (to them) marketing bumf on a page. But bear in mind that web access is generally a lot slower for a blind person anyway, and that they will be further slowed down by these barriers, possibly past the critical point of the whole task being too difficult to be worth the effort. In which case, the effect of the usability barriers does become disproportional.

That may be true and I would like to know, did the research test for this? If not, then it certainly wasn’t testing for equivalence of access. But again, even if vision impaired visitors only came across the same issues that everyone else does, the effect (especially the net effect when barriers multiply) may make the site unusable. This would not necessarily be revealed by an analysis of individual problems in isolation, however. It is quite a complex subject and it’s good that these researchers are looking at the wider issues of usability, but I’m with ronsman. The goal should be “equivalent access” as a concept, even though the definition of precisely what that means is a little unclear to me!

13

As much as I’m a advocate of accessibility, and as much as I hate on mobile devices, screen readers and other methods of accessing pages … the MOMENT people start talking about “equivalent access” or even “equality”, I kneejerk into “what is this BS” mode.

Reason being, I don’t believe in ‘equality’. Let’s be fair to the crippies, they’re not equal. PERIOD. They cannot function equally with equal treatment, and trying to say otherwise is unfair and outright insulting.

Though of course, that’s so non-PC all the limp wristed tofu loving namby pampy feel-good California drum circle types get their panties in a twist the moment you mention it.

It ranks right up there with this whole “entitlement” BS.

14

[FONT=Verdana]Sorry - I have to take issue with you there. Equal access means I should be able to make my own dentist’s appointment, like everybody else. As long as my dentist keeps insisting the only way to make an appointment is by telephone, I can’t do that and have to rely on somebody else to do it for me. Let me book by e-mail or letter and then I have equal access.

And please don’t use offensive terms to label people that you have decided are not equal. Not equal to whom? Just because I don’t use the internet the same way as the next guy, does that give anybody the right to say I’m not “equal”? I don’t think so. [/FONT]

15

I feel pretty entitled to being able to do things other grownups can do when it’s pretty darn reasonable for me to be able to do that, and that’s the matter.

This isn’t “we should be able to play Crysis on an Atari”, this is “the internet allows me to do things for myself, so why is it okay for some limp-wristed whiney webdesigner to make that impossible for me through his/her own incompetence?”
At least, that’s how I see it. When you have to ask your children to translate your doctor telling you that you have cancer because you can’t hear or understand him… when you have to ask your spouse to pick you up and bring you into the store… when you have to wait for a friend to show up to help you do something incredibly silly like check why your mail is silent…
(and Crusty knows what some of this is like, so I don’t think that’s where his argument is coming from…)

This is why I care about accessibility. It’s all about frustration, and not from “oh man I’m paralysed but I should be able to walk!” but from “this is something I expected to be able to do all by myself as an adult human being” when, due not to impossibility but silliness, is not the case.

(When I worked in hospital we had a case of Polish parents having their 12-year-old daughter translate from the doctor that the mom had an aggressive cancer, because there was no Polish translator available on that day, despite a large Polish community. I imagine Deaf parents go through this sadly often.)

I identify very well with frustration. Unnecessary frustration on the web is why I bother.

Damned straight.

I think Crusty’s point here is that, if the disabled were equal, they would not need additional help/software/time/etc to do similar tasks, compared to the non-disabled. Similarly with the on-going gedoe over females in tech, there’s no point in saying men and women for example are equal when they are different (and comparing things that are different is difficult to decide what “equal” means).
But if his point is also that “well we can’t expect the blind to be able to watch YouTube videos the same way sighted people do”, well no, they’re not going to watch YouTube videos the same way sighted people do, but that’s not what “equal access” means here, does it?

I believe people have a right to socialise with friends and family over the internet, using the same software everyone else is using. There’s no reason for this to be difficult. Having all your friends and family install TTY devices at home is difficult. Facewaste and twitter and LinkedIn and e-mail and forums don’t have this excuse.

I believe people have the same right to information and knowledge, whatever that information may be: learning what your government is doing to you, learning health information, communicating with your doctors, kids’ teachers, insurers, bankers, commercial companies, etc.

I believe they have the same right to be entertained and seek entertainment, as human beings seem to like that kind of thing.
I believe the internet has made bringing these things to a wider group of people, especially the disabled, actually doable without going into the “playing Crysis on an Atari” argument. It now costs less and means it’s now feasible. Putting lifts and ramps on every tiny old building is very often not feasible, for example. Equal access therefore is harder. Web sites? Jebus on a stick, it takes some work but it’s not something to throw up our hands on and say “bleh!”

And I believe that because if this, it’s now a responsibility web developers and technology innovators have.

16
Off Topic:

Thanks, @Stomme_poes. That’s pretty much what I was trying to say, but so much better put. :slight_smile: I just get so mad at having to state the (to me) blindingly obvious at all, that I get incoherent very quickly. :frowning:

17

Read the comments with interest as I volunteered for years with a challenged horse back riding program. Also a friend convinced my dad, who was a home builder, to implement 2 changes that make all the difference in accessible housing. 3’0 doors and hallways (width and entry/exit points). These two items are very expensive and hard to retro fit yet make such a big difference for accessibility, and they are items everyone enjoys :slight_smile:

Seems to me websites are the same, if a few items can be identified, those few items that if they are built in, make that big difference, and would lean toward they are probably the same items that many users appreciate.

I can appreciate a short checklist, with maybe a sentence on why for each item. Use it for myself, show it to customers :slight_smile:

18

This is an interesting area of web design that is often overlooked. I’ve never even considered optimizing my site for this demographic before. How would one determine if they have visually impaired traffic on their site?

19

Tanya: you don’t. You’ll never know how abled or disabled a visitor to your site is unless they bother contacting you to complain : ) Similar to how you’ll never know their race or height or shoe-size.

This is because web visitors using Accessibility Technology are using it on top of regular web browsers, and it’s the web browsers who give request information to your server (and the browser does not know if there’s AT running above it, and if it did it still won’t tell the server).

So you instead build with the assumption that you want your site to reach as many people as possible, within reason, and build with regular best practices in mind. I built a whole insurance site front-end with screen reader testing… though I’m pretty sure not a single blind person has ever visited that site. Still, should one do so, the site will Just Work, and people like stuff that Just Works. Other than learning how to do it, and then some testing, it didn’t cost anything to build that way.

As techmichelle mentioned about the wider doors and little wheel ramps when there are height differences between rooms, it’s usually pretty easy to just build things that way in the beginning, and often very costly and difficult to add them in later.

Usually building accessibly is simply building semantically and well anyway. If you look around the web for things like “accessible web forms” most of the recommendations are the things we should have been doing all along, like “use labels with your inputs”. The difference then is, when your design visually doesn’t want to show a label, instead of omitting the label, you keep it in the HTML like normal and pull it offscreen using CSS (pulling offscreen rather than display: none/visibility: hidden means AT considers the element on screen and so offers it to users).

Another example, from recent work I’ve done: we’re using Twitter Bootstrap for a client. Bootstrap’s code expects this for lists with headings above them:

<ul class=“someclass”>
<li class=“nav-header”>Some “heading”</li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
</ul>

The CSS then expects to style the fake header to look like a real heading, and assumes the other items underneath belong to that heading.
Obviously styling one list item does not make it heading at all. The best practices thing to do is to make a real heading:

<h3>Some heading</h3>
<ul class=“someclass”>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
<li><a href=“#”>regular link</a></li>
</ul>

Now everyone will know it’s a heading, not just people who happen to be able to see Bootstrap’s CSS.

20

I’m not sure quite what you mean when you talk about WCAG
techniques for other formats. I believe there are already techniques
related to other formats such as Silverlight, Flash, and PDF.

<snip />