We publish the runs, including the ones that went wrong

Four annual financial reports, a 155-page accessibility guide and a financial-aid form, all published by US universities. Nothing we wrote ourselves. Checked in veraPDF and by hand in PAC 2024 across four rounds.

Of the 624 that remain, 331 are images awaiting a human sentence and 293 are fonts the source never embedded. Neither can be fixed by re-tagging a document. Content, reading order, tables, lists, language and metadata pass on all six, with zero PAC warnings.

What this is not. Six documents chosen by us, not sampled at random. They were picked as the hardest in a 25-document corpus, which makes them a stress test and not an estimate of your backlog. Measured 7–9 September 2026.

Four real defects surfaced during those rounds — and PAC caught every one of them while veraPDF reported none.

Inline elements never nested in a block

Links, form fields and annotations are inline-level elements; every one we emitted sat directly under a section. 721 warnings across five documents. Found only by reading PAC's actual message after two theories derived from counting turned out to be wrong.

An optional-content dictionary inherited from the source

Missing a name, carrying a forbidden auto-state array. Passed straight through re-tagging because we never touched it.

Titles truncated to fragments

Three documents got titles like TABLE OF and Annual — a wrapped cover heading read one visual line at a time.

Descriptions written twice

An annotation's description mirrored onto the structure element as /Alt, where it replaces the element's own text rather than adding to it — shadowing the link text a reader needed.

veraPDF is the ISO reference implementation. PAC 2024 is what accessibility offices actually run. On the same files, against the same standard, they do not agree — and neither is simply stricter.

Conformance is legally binding under ADA Title II and EN 301 549. Which tool was run changes the answer, and we have not found this published anywhere.

Six documents is not a survey. This is a real observation on a small sample, not a systematic study of validator agreement. It is the finding we would most like to be wrong about, and the one we would most like help testing at scale.

Firefox 130 generates alt text for PDF images on-device with a 180M-parameter model. It is the only shipped product doing the same job, so we benchmarked against it on 40 figures from real institutional documents.

29 of 40 figures produced a single character. On most of the rest it invented something: a target-size example came back as “A sign that says, 'Don't drawer the war' and a red and white arrow.”

This is a distribution result, not a rigged test. Their model works: on its own example photograph it is fine, and on photographs from our own corpus it scored 6 of 6. It was trained on COCO, Flickr30k and Pexels — photographs — and institutional PDF figures are charts, screenshots and logos. The honest reading is that the distribution accessibility remediation faces is not the one general captioners are trained for. It is also 4.6× faster than ours, which matters for a browser captioning inline while someone waits.

“How much human work is left” is the number this whole product is judged on, so we checked our own. A third of it was our bug.

The queue read 485 images awaiting a description. One of those documents paints a decorative gradient on all 155 of its pages, and each one was being counted as a separate figure demanding its own sentence — 158 phantom review items, a third of the total, because the artifact rule had only ever looked at text and never at images. Real queue: 331.

Of what remains, roughly 200 are charts we will not let a model guess at, and roughly 120 are photographs and logos a model can draft for a person to confirm. Screenshots — which a blunt gate had been sending to humans as opaque graphics — are now separated by reading the text inside them.

Everyone in this space quotes a human-effort number. Nobody validates the denominator. Ours was wrong by a third until we counted it properly.

The prompt had been written once against the WCAG guidance and never compared with anything. Six variants, scored on 164 human-written reference descriptions and 40 annotated charts, said we had it wrong in two ways.

The eight carefully written rules were over-compressing: against a human reference of “Accessible Electronic Document Community of Practice (AED COP) Logo” the rules returned “AED COP Community Of Practice” — dropping the institution's name. The worked examples carried the quality and the rules hurt.

The sweep also caught a live bug: one variant answered DECORATIVE on 35% of figures, and nothing downstream handled it — the literal word would have been written into the document and read aloud to a blind user.

And charts want the opposite. Only a prompt that explicitly asks for the relationship a chart shows gets past naming the chart type — and it still reaches the most informative description level on 8% of charts against a published human baseline of about 14%. That gap is why the chart gate stays shut.