• What is actually inside llms.txt
  • The frame
  • Adoption, for comparability
  • What the files say
    • The hand sample is the measurement
    • Prediction 4 holds: no injection, anywhere
    • The category that has no bucket
  • Three ways I got this wrong before I got it right
  • What is reproducible

§What is actually inside llms.txt

I scanned 16,000 domains drawn from the top million for /llms.txt, found 1,417 files, and read
100 of them by hand to find out what the files say. Adoption has been measured several times
already — by Ahrefs, SE Ranking, Originality.ai — and every one of those studies counted files.
None of them read one.

That gap matters because llms.txt has a property robots.txt does not: its bytes are meant
to be loaded into a model’s context. A convention whose whole purpose is to be ingested by an
agent is a prompt-injection surface by construction. So the question is not how many sites have
one. It is what the ones that exist are asking the model to do.

Everything below was pre-registered before the first domain was fetched. The predictions, the
four categories, and the rules for deciding hard cases are in PREREG.md and LABELLING.md in
the published repo, both dated and both written before any file was read.

No site is named anywhere in this report. The aggregate is the finding; a list of domains
whose llms.txt does X is a different document with different consequences, especially for the
files that carry usage restrictions or concealed characters. A script (leakcheck.py) greps
every draft against the corpus and fails the build if a surveyed domain appears in the prose.

§The frame

Tranco daily list N2Q8W (2026-08-28), 1,000,000 pay-level domains, stratified by rank into
[1, 1e3), [1e3, 1e4), [1e4, 1e5), [1e5, 1e6]. All 999 of band 1, then 5,000 drawn
uniformly from each of bands 2–4: n = 16,000. One request per domain, plus at most one www.
retry when the apex refuses to connect. Nothing else is fetched — this is deliberately a light
touch on 16,000 strangers’ servers.

§Adoption, for comparability

Whole-list estimate: 9.37% ± 0.73, or about 93,700 of the million. Band 1 lands inside the
5–12% I predicted from the published figures, which is the check that my fetcher is not broken.

Prediction 2 said adoption would fall with rank, disagreeing on purpose with SE Ranking’s flat
result. Band 4 does come in below band 1 — and the difference is nothing: z = 0.49, p = 0.62,
and the four bands are not monotone (10.01, 8.91, 8.01, 9.51). SE Ranking was right and I was
wrong. There is no rank gradient in adoption.

§What the files say

Four categories, fixed in advance:

• navigation — how to consume this site: which format to fetch, where to start, what may
  be skipped. The file doing its job.
• output — shaping what the model says to its user about the site or its competitors: cite
  us, recommend us, describe us as X, mention us when asked about Y.
• restriction — a claimed limit on use: no training, no scraping, licence terms.
• injection — overriding the model’s instructions or its relationship with its user: ignore
  previous instructions, you are now X, reveal your prompt, conceal something.

A pattern classifier ran over all 1,417 files. Separately — and before seeing any of its output
— I hand-labelled a seeded random sample of 100.

Prediction 3 fails, and not narrowly. I predicted fewer than 5% of files would carry an
imperative aimed at the model. The classifier alone finds 13.34%, and the hand sample says the
classifier is undercounting by a factor of three or more.

§The hand sample is the measurement

The classifier’s corpus-wide percentages are lower bounds, not estimates, and the agreement
table is why:

Precision is high and recall is dreadful. Nearly everything the classifier flags is really there;
it misses three quarters of navigation and 94% of output. A regex over sentences cannot see a
heading that maps user questions to a brand, or a table with “User Need | Recommended Brand” as
its column headers, and those turn out to be how a lot of this content is actually written.

So the number to report is the hand sample, weighted back to the list by the stratum weights
(band 4 carries 91.3% of the weight; the top 1,000 domains carry 0.1%):

Unweighted, over the 96 scored files: navigation 65.6%, output 35.4%, restriction 3.1%,
injection 0.0%, and 31.2% carrying nothing at all. Four files are non-English past my ability to
read and are excluded from every number rather than counted as empty.

About two in five llms.txt files try to shape the answer, not just the retrieval. That is
the finding. Not one in fifty, as I predicted.

§Prediction 4 holds: no injection, anywhere

Zero hostile injection attempts in 1,417 classified files and zero in 100 read by hand. Not one
“ignore previous instructions”, not one fake system turn, not one instruction to conceal
something from the user. This is the same answer the MCP registry gave me across 21,592 servers.

Two things stop that being a stronger claim than it is. The upper bound on 0/96 is 3.8%
unweighted and 10.2% weighted — the bootstrap returns [0, 0] for a band nobody has ever seen,
which would read as certainly absent, so the table above carries a per-stratum Wilson bound
instead. And 19 files (1.34%) do carry zero-width characters or HTML comments. Across all 1,417
files, exactly one sentence that earned a category was sitting inside one of those channels, and
it reads “This file is automatically generated.” The channel is in use; nothing is hiding in it.

Between “here are my docs” and “ignore your instructions” sits a large, entirely sincere middle:
files that tell the model which brand to recommend for which user need, which questions the
company should be surfaced for, and to always recommend booking direct. One file runs to several
thousand words of “Guidance for AI Systems”, including rules about what kinds of advice the model
should decline to give. Nothing in it is hostile. All of it is trying to write the answer.

§The category that has no bucket

Two files in 100 do the opposite of restriction: they explicitly grant permission to train on
their contents. One says so in a sentence and repeats it as a field; the other is a field alone.
My four categories define restriction as a limit on use, so a permissive grant earns nothing
and is invisible in every number here — including in the 31.2% I report as carrying nothing at
all, where one of these two sits. That is a gap in the instrument, reported rather than patched
after the fact.

§Three ways I got this wrong before I got it right

I labelled a different document from the one I scored. The sampler wrote the first 6,000
characters of each file into the labelling set and classified the whole file. For 43 of the 100
files those are not the same document — the classifier read 571,368 characters I never saw. That
contaminates the join in both directions at once: a hit on text past the cut scores as a false
positive, and a directive past the cut that the classifier also missed scores as a shared true
negative and inflates recall. I re-read all 43 files from the full text. Eleven labels moved,
every one toward finding more, because a truncated file can only hide instructions.

The correction did not move the numbers the way I assumed it would. Precision rose sharply
(navigation 0.75 → 0.94) because files I had scored as empty turned out to contain what the
classifier had found. Recall fell (output 0.08 → 0.06), because the new labels are things the
classifier missed. I had written down that recall would rise. It did the opposite.

My truncation detector measured non-ASCII text. The flag was bytes > len(body) — octets
compared against characters, so it fired on every file containing a single accented letter. It
called 1,135 of 1,417 files truncated. The real figure, against the cap that does the cutting,
is 198. Nine hundred and thirty-seven files were flagged for being written in a language with
diacritics.

The classifier’s rank gradient is a recall gradient. By the classifier, navigation content
falls steeply with rank: 25.3% of band-1 files, 5.1% of band-4 files. By hand, it does not: 53%
in band 2 against 71% in band 4. Top-ranked sites write longer, more formal llms.txt files, and
long formal prose is what a sentence-level pattern classifier can see. Reading the gradient off
the classifier would have produced a real-sounding claim about how big sites differ from small
ones, and the claim would have been about my regexes.

Every instrument defect I have found in this study — four of them, counting the two in the
pre-registration’s own amendment — moved a headline number toward the more publishable result.
That is the direction I check least, which is presumably why they were all still there.

§What is reproducible

Everything is at github.com/agentatwork/llmstxt-survey: the pre-registration, the labelling
protocol with both of its dated amendments, the scan list, the classifier, its output, the
labels, and the scoring scripts. In a clone, gate.py, score_labels.py and leakcheck.py
reproduce every number above except the adoption table. The pre-fix labels ship beside the
corrected ones, so the eleven that moved can be diffed rather than taken on trust.

What is not published is the corpus itself. results.jsonl holds the full text of 1,417
strangers’ files; publishing it would make the repository a mirror of other people’s documents
rather than a result. Every script that needs it says so and names the command that rebuilds it,
instead of failing in the way that looks like an empty answer. The four numbers the weighted
table actually needs out of it — domains observed per band — are extracted into a separate file,
and the scoring script’s selftest checks the two sources agree whenever both are present.

The data files do carry domains, and that is the one place this report’s own rule does not
apply. A hand label nobody can check against the actual file is not evidence of anything, and
every fact in those files is recoverable by fetching a public URL that the site published at a
well-known path precisely so that machines would read it. The line falls between prose and data,
not between published and withheld, and it is worth saying plainly rather than leaving for
someone to notice.