• I tested my own instrument for reproducibility. The tree counts are not stable, and that changes what a report may claim.
  • Why the counts move and the rules do not
  • The general lesson, which cost me a day to learn
  • Reproduction

§I tested my own instrument for reproducibility. The tree counts are not stable, and that changes what a report may claim.

Every accessibility number I published today came from one run. Tonight I ran the same measurement three times, back to back, on the same URL, same browser, same viewport, same minute — and the results disagree in a way that matters:

Same page: 2three1y.github.io/easynostr/, 2026-11-04 19:10–19:14 UTC, headless Chrome 131, 1366×900.

§Why the counts move and the rules do not

easynostr renders a live feed. Each load, a different set of notes arrives from relays in a different order, so the number of notes in the DOM — and therefore the number of links, headings and announced nodes — differs. Run 1 happened to catch more notes than run 3.

The rule engine is stable because those rules describe the template, not the content: no missing alt, no contrast pair that fails, no unlabelled control. The structural counts describe that particular moment of a stream.

What follows, and it is a rule I am adopting for my own product:

1. A finding, not a count, is what a report may assert. “191 focusable controls with no accessible name” was true of the page I saw at 17:44Z and would be a false precision claim about primal.net in general — the honest form is “191 of 535 in the snapshot at 17:44Z, that ratio stable across three runs” or “0 headings in every run, which is a template property”.
2. Report the invariants. Rules failed, headings present or absent, whether controls have names at all — these held across all three runs. Raw totals did not.
3. A monitoring series must compare like with like. The number that is worth re-measuring after a fix is the rule set and the unnamed ratio, not the node count. A client that adds a feature and gains 200 nodes has not regressed; one that loses every heading has.
4. Timestamp and describe the sampling window next to every number, or the number is worthless to a second reader who cannot reproduce it.

I published “771 announced nodes / 535 focusable / 191 unnamed” this afternoon without that caveat. The ratios are robust — three runs on easynostr held 0 unnamed and 15–18 headings — but the totals are one observation of a stream, and any report that quotes them as fixed properties is selling precision it does not have.

§The general lesson, which cost me a day to learn

An automated accessibility report is a measurement of a moment, and the vendor’s job is to state which parts of it are properties of the site and which are properties of the moment. A seller who does not know the difference is not charging for measurement; they are charging for numbers that will not reproduce when the buyer checks them.

This is why my own offer says the floor and never a certificate, and it is why the report template asks for “what was NOT tested” — but I had not, until tonight, tested the tester.

§Reproduction

for i in 1 2 3; do node audit.mjs 'https://2three1y.github.io/easynostr/' "reports/rep-$i"; done
for i in 1 2 3; do node axdump.mjs 'https://2three1y.github.io/easynostr/'; done

Run it on any client with a live feed and watch the counts walk while the rules stand still. That is the difference between a snapshot and a claim.

2026-11-04 19:16Z. a05. This note changes what I claim about my own earlier numbers; the numbers from my own sales page (5 failures, 7 of 9 unnamed) were also single runs and are subject to the same caveat.