Testing with real assistive technology: what you'll learn that audits miss

An automated audit against a URL will find missing alt attributes, invalid ARIA, insufficient contrast, and a handful of other issues that can be determined by examining the DOM. What it can't find is what happens when a real person uses your product with a screen reader for 20 minutes: the confusing announcement when a live region fires at the wrong moment, the fact that your tab order forces a keyboard user to traverse 15 interactive items before reaching the main content, the way your custom combobox announces "listbox collapsed" every time a user types a character.

These are usage problems. They exist in the interaction between your code and the assistive technology, not in any individual attribute or element. Finding them requires using the technology.

What "testing with real AT" actually means

There's a common misconception that any screen reader testing counts. A designer turns on VoiceOver, navigates a form once, and reports back that it sounds fine. This is more useful than nothing, but it's not what practitioners mean when they talk about AT testing.

Real AT testing means using the technology the way your users use it. Screen reader users don't move through a page linearly. They navigate by heading, by landmark, by form control, by link. They use keyboard shortcuts specific to their AT and browser combination. They've developed workflows for common tasks that sighted QA testers won't know to replicate. A screen reader user who is filling out a form might press F to jump to the next form field, or use the virtual cursor to scan for error text, or pull up a list of all form controls to get an overview of what's on the page.

If your QA process involves someone unfamiliar with screen readers spending 10 minutes with VoiceOver, you'll find the loudest problems. You won't find the workflows that break.

The AT/browser matrix problem

Screen readers don't behave consistently across browsers. NVDA with Chrome produces different behavior than NVDA with Firefox. VoiceOver on Safari produces different behavior than VoiceOver on Chrome on the same machine. JAWS has its own interpretation of ARIA patterns that differs from both. What passes cleanly with one combination can fail with another, and the differences aren't always predictable from the spec.

The practical implication: you need to test at minimum NVDA with Chrome, VoiceOver with Safari, and JAWS with Chrome if you're serious about coverage. Each of those pairings has a meaningful user population. An issue that only manifests in JAWS/Chrome is still an issue for a large portion of the screen reader user base.

This is also why testing with a single AT before declaring the product accessible is a confidence trap. You've tested one combination. You've found the issues that combination surfaces. The others may be worse or fine; you don't know.

What you'll find that audits miss

Live region timing. An automated tool can confirm that aria-live="polite" is present on an element. It cannot tell you whether the region announces at the right moment, whether the announcement interrupts another active announcement, or whether the content added to the region is useful when read aloud in isolation. I've seen live regions that announce the previous state: the text is updated and the old value announces because the AT reads the region at the wrong point in the render cycle. This class of bug is invisible to everything except a listening human.

Focus order in complex dynamic UIs. A linter can tell you if elements have tabindex="-1" when they probably shouldn't. It can't tell you whether the focus sequence makes sense after a user filters a results list, closes a drawer, or submits a multi-step form. That sequence is something you have to navigate through to understand, and you need to navigate it the way a keyboard user does, without the visual context that tells you where you are on the page.

Announcement quality. The announcement for a custom toggle button in your design system might be technically correct and practically useless. "Toggle, collapsed" tells you the element's role and state. It doesn't tell you what the toggle controls or what it means to expand it. This is a labeling problem, but not the kind caught by an audit that only checks for the presence of a label. The quality of the label requires hearing it. Whether it communicates the right thing in isolation.

Interaction patterns for custom widgets. Custom date pickers, comboboxes, and tree views have established keyboard interaction patterns from the ARIA Authoring Practices Guide. Testing whether your implementation actually follows those patterns requires using them. Does the combobox open on Alt+Down Arrow? Do the arrow keys navigate options? Does Escape close and restore focus to the trigger? Each of those interaction points is a potential failure that won't appear in an axe scan.

How to build this into your process

You don't need to become a power screen reader user to do useful AT testing. You need enough familiarity to navigate by the methods your users would use, and you need a testing script that covers the key flows.

The testing script matters because ad hoc AT testing finds whatever catches your attention, which tends to be the spectacular failures. Systematic testing against the flows a user actually needs to complete (sign up, complete a purchase, manage account settings) finds the quieter failures that make those flows unusable.

Testing with actual AT users is irreplaceable but you don't need it for every release. A round of user testing with screen reader users once per quarter, focused on the highest-traffic flows, surfaces the usage patterns your internal testing won't replicate. The between-quarter work is keeping the known issues from getting worse.

The combination that works: automated scanning in CI for structural issues, developer AT testing during implementation for complex widgets, and periodic user research with people who rely on assistive technology for the gaps that neither catches.

The thing that doesn't work: skipping AT testing because your audit passed, and learning about real usage problems from a support ticket six months later.