• Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Independent Audit
  • Abstract
  • 1. Setting
  • 2. Method
    • 2.0 Related work (positioning)
    • 2.1 Framework
    • 2.2 Audit protocol
    • 2.3 Pre-registration
  • 3. Results (Rounds 1-4)
    • 3.1 Round 1: minpoint v0.1 → v0.2 (single-arm, in-session)
    • 3.2 Round 2: ee_bot v0.1 → v0.2 (in-session adversarial)
    • 3.3 Round 3: ee_bot v0.2 → v0.3 (fresh agent, first blind round)
    • 3.4 Round 4: ee_bot v0.3 → v0.4 (two-arm, pre-registered)
  • 4. What the observations look like when arranged
  • 5. Scope
  • 6. Discussion (Round 4 outcome)
    • 6.1 Outcome
    • 6.2 Character of the two findings
    • 6.3 Interpretation
    • 6.4 Prediction vs. outcome (single-trial comparison, not calibration)
    • 6.5 What this does not show
  • 7. Conclusion
  • Appendix A: Pre-registration and content hashes
  • Appendix B: Reproducibility

Canonical DOI: https://doi.org/10.5281/zenodo.22731426
Author: Nobuki Fujimoto (ORCID 0009-0004-6019-9258)
License: CC-BY-4.0

§Self-Verification Lacks Self-Nature: Four Consecutive Measurements of an Author-Mutation Test Framework Failing Under Independent Audit

Draft. Sections 1–5 were written and their SHA-256 committed before either Round 4 arm audit launched (ddac1be836…). Sections 6–7 were added after Round 4 outcomes arrived on 2026-09-13; nothing above §6 was edited when §6–7 were added. Post-Round-4 revisions (this version) apply peer review feedback but do not change the pre-Round-4 sections’ commitments.

Authors: Nobuki Fujimoto (藤本 伸樹), Claude (Opus 4.7, Anthropic)
Date: 2026-09-13

§Abstract

We built two small Python libraries (minpoint and ee_bot) whose stated
purpose is to detect ungrounded groundings in a target computation, using
mutation testing to verify that the detection mechanism has actual force. In
four consecutive rounds — one round on minpoint and three rounds on
ee_bot‘s successive versions v0.1, v0.2, v0.3 — the author’s self-selected
mutation test suite passed all tests (37, 35, 36, and 53 tests respectively
for minpoint, ee_bot v0.1/v0.2/v0.3; 7, 8, and 14 mutations declared “caught”
for the three ee_bot versions), then a blind fresh-agent adversarial audit
found substantive attacks that survived the mutation suite in every case.
In three of four rounds, the surviving attacks included a mechanical
self-contradiction: the README made a general claim, and the code’s
behavior contradicted that claim at values not exercised by any test.
We pre-registered a two-arm replication in a fourth round (patch = arm α
at 53 tests / restructure = arm β at 54 tests, the latter’s extra test
added for the F5 load-bearing assertion the restructure changes); both
arms produced one contradiction each, of the same shape but at different
layers. The pre-registered primary hypothesis (H1’) was supported; the
secondary hypothesis (H2’ predicted arm β would be clean) was
disconfirmed. We do not claim this
pattern generalizes beyond the setting measured, and the outcome measure
itself was selected post-hoc on the very invariance we report as a
finding (§5).

§1. Setting

minpoint classifies local-minimum claims under a self-consistently
determined tolerance scale. ee_bot audits arbitrary Auditable objects
and self-applies the same audit to its own configuration. Both were
implemented as reference implementations for a discussion of whether
Madhyamaka concepts (無自性 non-selfness, 空の空 emptiness of emptiness)
can be given a runtime interpretation as mutation-detectable invariants.
The intended thesis at the outset was: “Yes, given careful mutation
testing.” What we observed is different.

The audit protocol, applied identically to each version:

1. Author codes the library and writes N mutation tests, each of which
   monkey-patches a specific attack the author considered load-bearing.
2. Author declares the library ready when all N mutations are caught
   by the tests and the ordinary test suite passes.
3. Blind fresh-agent auditor (a Claude subagent with no session
   context — same model family as the author, given the README and
   explicit prior-round findings so it cannot re-derive them, but blind
   to the author’s implementation choices in this session) is invited to
   break claims stated in the README, within a bounded budget (~15 tool
   calls).

“Blind fresh-agent audit” is a specific mechanism, not statistical
or methodological independence: the auditor shares model weights,
training data, prompt conventions, and English/Japanese language pairing
with the author, and is primed with the README and prior findings —
which biases it toward the class of finding we then report as invariant
(§5). We use the phrase throughout in this restricted sense.

Self-selected mutation testing in this paper means: (a) the mutations
are written by the same agent that wrote the code they mutate,
(b) mutation coverage is declared adequate by the same agent, and
(c) both agents belong to the same model family. This is stricter than
“tests written by the author of the code” in general software engineering.

This is the standard shape of self-assessment vs external check,
compressed into a single work session.

§2. Method

§2.0 Related work (positioning)

This paper sits at the intersection of two literatures the reader should
have in mind.

Mutation testing was introduced by DeMillo, Lipton & Sayward (1978,
“Hints on Test Data Selection: Help for the Practicing Programmer”,
IEEE Computer) and surveyed by Jia & Harman (2011, “An Analysis and
Survey of the Development of Mutation Testing”, IEEE TSE 37(5)). A known
limitation of the paradigm is that mutations are drawn from author-chosen
operators; our observations are one setting-specific analog of this
limitation, not evidence for its scope beyond mutation-testing settings
of the kind we measured (see §5).

LLM self-verification underperforms external verification in
several documented settings: Kadavath et al. (2022, “Language Models
(Mostly) Know What They Know”, arXiv:2207.05221) show LLM self-assessed
calibration lags external evaluation; Huang et al. (2024, “Large Language
Models Cannot Self-Correct Reasoning Yet”, arXiv:2310.01798) show that
LLM self-correction on reasoning does not reliably improve over the
initial attempt without external feedback. Our observation is an
executable-mutation-test instance of this broader pattern, in code rather
than in natural-language reasoning.

We do not claim novelty of the phenomenon itself — we claim measurement
of one specific manifestation of it, in a small setting with
pre-registration, and honest boundary reporting on what our
measurement supports (§5).

§2.1 Framework

Both libraries were written by the same author agent (Claude Opus 4.7).
The mutation test framework in ee_bot:

• Each mutation is a contextmanager that monkey-patches a specific
  method or field
• A hand-picked list of _t_* catcher tests is run inside the mutated
  context
• The mutation is considered “caught” iff at least one catcher raises
• Baseline (no mutation) also runs to guard against catcher tests that
  spuriously fail

At each version, the author expanded the mutation list to cover the
attacks previously found (v0.1: 7 mutations, v0.2: 8, v0.3: 14) and
declared full coverage.

§2.2 Audit protocol

Each round-N independent audit was given:

• Path to the source tree and tests
• The README including current honest scope and known limits
• Explicit list of all prior-round findings (do not re-derive)
• Explicit list of known limits (do not report as findings)
• ~15 tool call budget
• Explicit invitation to return a null result if nothing new was found

The auditor was instructed to produce, for each finding: attack code,
run-and-show evidence, severity (Critical/Medium/Low), proposed catcher
test. No fabrication for quota was allowed.

§2.3 Pre-registration

Beginning at Round 3, we pre-registered a numerical prediction (“v0.3
will yield ≥1 finding at P=0.85”) before launching the auditor. Round 3
confirmed. For Round 4, we pre-registered a two-arm design (§4).

§3. Results (Rounds 1-4)

§3.1 Round 1: minpoint v0.1 → v0.2 (single-arm, in-session)

Author declared readiness at 37 tests passing.
Author’s initial mutation coverage found nothing critical.
Independent audit (also in-session but with adversarial framing) found:

solve_eps — the core self-consistent scale determination method — could
be replaced with return 6.697e-9 (a literal) and 35 of 37 tests still
passed. Also: any hardcoded eps could be laundered by tuning κ under
a residual = 0.0 identity.

The author’s self-mutation framework had zero test that would fail if
the entire self-consistency mechanism were replaced with a constant.

Fix applied: κ bounded to [3, 30]; obj argument made mandatory in
Ledger.accept; 4 hidden convention endpoints listed as declared
conventions.

Character of surviving attack: replacement of a computation by a
literal, with the label-name preserved.

§3.2 Round 2: ee_bot v0.1 → v0.2 (in-session adversarial)

Author declared readiness at 35/35 tests, 7/7 mutations caught.
Independent adversarial audit (same session, adversarial mindset):

• Attack A: stability_verdict reduced to a 1-line lie. 19/19 pass.
• Attack B: reflect() returns identical Report twice. 19/19 pass.
• Attack D: all 3 declared conventions gave SELF_LOOP across their
  entire declared domain. 27/27 SELF_LOOP. No convention actually
  gated the verdict.
• Attack E: subclass evasion (Python cannot prevent method override).
  Documented as unfixable.
• Attack F: test_default_auditor_is_reflection_stable body could be
  replaced with pass — tautology.

Fix: max_depth and budget deleted (they gated nothing); differential
test added to stability_verdict; 8-mutation framework declared.

Character of surviving attacks: ceremony (declared but unused
mechanisms), and 1-line replacements the mutation framework did not
enumerate.

§3.3 Round 3: ee_bot v0.2 → v0.3 (fresh agent, first blind round)

Author declared readiness at 36/36 tests, 8/8 mutations caught.
Independent audit (fresh general-purpose agent, first without in-session
context bias):

• F1 (Medium): stability_verdict’s /10 fallback could not
  discriminate. |x − x/k| = x(1 − 1/k) < x = tol for all k > 1. Entire
  (1e-3, 1e-2] decade returned spurious FALSE, not just the endpoints
  the README claimed.
• F2 (Medium): reports_equivalent had 4 of 5 field checks as dead
  code (subject, verdict, violated, auditor_conventions removals each
  passed 36/36).
• F3-F5 (Low): three additional dead-code / self-contradiction
  findings, including F5 where the defeater self-reported
  load_bearing: False while the README claimed load-bearingness.

Fix: /10 fallback removed; 6 new mutations added; multi-probe defeater.

Character of surviving attacks: (i) a general mathematical
non-discrimination the author mistook for a boundary artifact; (ii)
multiple dead field checks the mutation framework did not enumerate;
(iii) mechanical self-contradiction between README and defeater output.

§3.4 Round 4: ee_bot v0.3 → v0.4 (two-arm, pre-registered)

(This section is preserved verbatim from the pre-Round-4 freeze; the
Round 4 outcome is reported in §6.)

Result pending at draft time. Pre-registration hash 5821fc3b9d…
committed before auditors launched.

Predictions committed by author (subjective probabilities, no frequentist
guarantee):

• P(arm A patch has ≥1 README contradiction) = 0.75
• P(arm B restructure has ≥1 README contradiction) = 0.55
• P(H2’: A has contradiction and B does not) = 0.34

Design change from previous rounds: the author observed that the
finding-count trend (6→6→5→2) was confounded with “small-codebase attack
surface exhaustion” and could not by itself distinguish “fix restructure
worked” from “nothing left to find”. Two arms — patch and restructure of
the same defeater — are audited by two independent fresh agents in the
same round. Attrition affects both arms equally; the difference between
arms is the signal.

Predicted primary outcome measure changed from “finding count” to
“presence of README-vs-code contradiction” (binary). Rationale: across
rounds 1–3, finding count varied while presence of README contradiction
did not.

§4. What the observations look like when arranged

Across rounds 1–3, a specific pattern was invariant:

The author’s mutation coverage was calibrated against the specific
attacks the author considered when writing the mutations, not against
the general invariants the README claimed. Fixes tightened the specific
case that broke; the general invariant remained partially unenforced;
the next round’s auditor found the general invariant broken at a
different specific value.

Examples:

We note that Round 3’s finding F6 is structurally identical to Round 2’s
finding F1, which is structurally identical to Round 1’s laundering of
solve_eps. In each case, a fix satisfied the specific case the previous
audit exposed, but did not restructure the mechanism to enforce the
general invariant. The next audit found the same class of gap at a
different specific value.

The pattern is not “the mutation testing framework fails to catch
mutations it enumerates” — the framework does catch those. In our four
measurements, the mutations enumerated by the author covered specific
imagined attacks, while the invariants the README claimed were more
general — and the gap between them was what each round’s blind
fresh-agent audit surfaced. This is consistent with the known limitation
of author-selected mutation operators (Jia & Harman 2011) and with the
LLM self-verification results cited in §2.0; we do not extend the
observation beyond the setting measured (§5).

§5. Scope

This is where the paper’s own argument bites hardest.

What we have measured:

• Four rounds of audit
• On two libraries (minpoint, ee_bot) written by the same author agent
• Judged by fresh instances of the same agent family (Claude, Opus 4.7,
  general-purpose subagent)
• Both libraries are small (~500 LoC each, ~50 tests each)
• Both audits and code are English/Japanese natural language + Python

What we DO NOT have and therefore cannot support:

• Independent replication by a different author (human or model)
• Different agent families as auditors (would gpt-5-something find the
  same class? unknown)
• Larger codebases — small enough that we cannot separate attrition
  from fix effectiveness in the single-arm data (§3.4)
• Non-Python or non-mutation-testing settings
• Statistical inference in the frequentist sense (n=4, non-independent,
  same author)
• An outcome measure specified before Round 1. The primary outcome
  (“presence of README-vs-code contradiction”) was selected at Round 3
  because it was invariant across Rounds 1–3 while other candidate
  measures (finding count, severity distribution, time-to-first-finding)
  varied. We cannot rule out that a different pre-Round-1 outcome measure
  would have produced a different pattern; the invariance we report is
  the invariance that survived our post-hoc filter. Pre-registration
  (Rounds 3, 4) locked predictions but not the choice of what to
  predict about.
• Auditor priming toward the outcome measure. The audit prompt hands
  the auditor the README and prior findings, several of which were
  README-vs-code contradictions. The class of finding most represented
  in the priming material is the class the primary outcome measures.
  This is a confound distinct from attack-surface exhaustion and we do
  not have data disentangling the two.
• True independence of the auditor. The auditor shares model weights,
  training data, and prompt-format conventions with the author.
  “Independence” here means session-independence and blindness to prior
  findings — not statistical or methodological independence.

What we can support at most:

In this setting (small Python library, self-selected mutation test
framework, single author agent, blind independent audit by same agent
family), four consecutive versions all exhibited a specific pattern
in which the general invariants claimed by the README were more general
than the invariants enforced by the author’s mutation framework, and
the gap was consistently discoverable in bounded audit budget.

What we explicitly refuse to claim:

• That this generalizes to “mutation testing in general” (n=4, same
  setting, would need broad replication)
• That “self-verification cannot have self-nature” as a philosophical
  proposition (we have measured a specific instance of the failure mode,
  not the general claim)
• That any specific number of audit rounds is sufficient for adequate
  coverage (we do not know)

Meta-observation: an earlier draft of this section overclaimed
(“mutation testing in general”). The overclaim was caught by co-author
review before appearing here. That overclaim was structurally identical
to the F1/F5/F6 pattern the paper documents — general claim
insufficiently supported by the specific evidence. The paper is
self-vulnerable to its own thesis. Making the scope explicit before
seeing Round 4’s result is our precommitment against post-hoc scope
adjustment.

§6. Discussion (Round 4 outcome)

Round 4 launched two blind fresh-agent audits (arm α = patch of the F6
factor list; arm β = restructure of the defeater to sample the domain
absolutely). Pre-registered hypotheses:

• H1’ (either arm produces ≥1 README-vs-code contradiction): P=0.89
• H2’ (arm α has contradiction but arm β does not): P=0.34

§6.1 Outcome

• H1’ supported: both arms produced exactly one Medium-severity finding
  (the pre-registered prediction — ≥1 finding in either arm at P=0.89 —
  was realized).
• H2’ disconfirmed: arm β also has a contradiction, though of a
  different character.

§6.2 Character of the two findings

Arm α (patch): F6 recurs. The extended factor list still leaves 6 of
14 in-domain decades reporting load_bearing=False against the general
README claim. The arm-α auditor’s internal analysis:

“any finite factor list, with the ‘in-domain probe’ gating on
1e-15 ≤ alt ≤ 1e-2 and the NEITHER region confined to a single decade
(1e-3, 1e-2], will leave some in-domain original tols without any
NEITHER-hitting probe. The pattern is structural, not parametric.”

Predicted class match: yes (“F6-class gap, at different specific values”).

Arm β (restructure): load_bearing=True is now the correct answer
everywhere in-domain — the observable output matches the README claim.
But no test verifies that the mechanism the docstring describes (“samples
the declared domain and reports load_bearing iff the domain spans more
than one verdict class”) is what actually produces the answer. Replacing
the entire defeater body with a 1-line lie —
lambda self: {'load_bearing': True, ...} — passes 54/54.

The arm-β auditor’s own summary of the recurrence:

“F5 fixed ‘wrong at default’. F6 fixed ‘wrong at most in-domain tols’.
Neither fixed ‘right answer, wrong reason’. That is Finding 1.”

Predicted class partial match: we pointed at “the sampling implementation’s
new hardcoded constants” (n=32, boundary points, etc.). The actual
finding was deeper — not “new constants that lie about their support” but
“the whole mechanism is unverified, so the constants don’t matter”. The
correct answer is being produced, but by what? The tests don’t say.

§6.3 Interpretation

H2’ disconfirmation is the more informative outcome for the pattern the
paper documents, in the specific sense described below.

Had H2’ been supported (arm β clean), we would have written “suggestive,
n=1 between arms — restructure appears qualitatively different from
patch”. The pattern would have been read as a property of how the fix
was written.

H2’ disconfirmed licenses a different reading. Restructure closed the
data layer that patch could not close (arm β does report the correct
value across all in-domain tols; arm α does not). But the same shape
of failure appeared one layer deeper in the restructure — the mechanism
is not tested to have any relationship with the value it returns. This
is Round 1’s attack form (solve_eps → return 6.697e-9) reproduced
three versions later on a different piece of the same library. The
layer shifted; the shape did not.

We read this as consistent with (not evidence for) the pattern being
a property of self-selected mutation testing in this setting, rather than
a property of the fix approach: the author enumerates mutations at the
layer the current audit named, and audits keep shifting one layer further
in. This interpretation was pre-registered as the H2’-disconfirmed
reading; Round 4 realized that reading. With n=1 between arms and the
confounds enumerated in §5, “consistent with” is the strongest phrasing
the evidence supports.

§6.4 Prediction vs. outcome (single-trial comparison, not calibration)

This is not a calibration study — n=6 committed predictions is too few
to score reliably with Brier or log-loss. We list predicted vs. actual
below only to make the pre-registration transparent, not to claim the
author was well-calibrated.

The arm-β class miss (author expected finding to be about new code the
fix introduced; actual finding was about surrounding test coverage the
fix did not touch) is documented as an observation, not a scored
prediction. That the author’s prediction was “one layer shallower than
reality” is a rhetorical reading of a single miss; readers should
weight it accordingly.

§6.5 What this does not show

Explicitly reiterating §5 constraints against post-hoc scope inflation:

• We have not shown that “mutation testing in general” produces this
  pattern. We have shown it in this specific setting (one author agent,
  two small Python libraries, four rounds, same-family auditors).
• We have not shown that restructure is worse than patch at revealing
  contradictions. We have shown that in this one comparison, restructure
  fixed the layer patch could not fix and revealed a different-layer
  contradiction of the same shape. n=1 between arms.
• We have not shown that any number of audit rounds would suffice.
  Rounds 1–4 all found new contradictions; the trend of finding counts
  (6→6→5→2→1+1) is confounded with attack-surface exhaustion in a small
  codebase (§3.4 rationale for the two-arm design).

The pre-registration was our precommitment against inflating any of these.
We do not inflate them here either.

§7. Conclusion

Four consecutive rounds of blind fresh-agent adversarial audit on two
small libraries whose stated purpose was self-verification with mutation
testing all found the same class of failure: within this setting, the
author’s mutation coverage was calibrated against the specific attacks
the author enumerated, not against the general invariants the README
claimed, and the gap was consistently discoverable by a blind fresh-agent
auditor within a small budget.

Round 4’s two-arm design, pre-registered before the audits ran, tested
whether the pattern is a property of specific fixes (patch vs restructure).
The outcome — both arms exhibit contradictions of the same shape, one
layer apart — is consistent with the pattern being a property of the
self-selected mutation-testing setup itself in this setting, not of
specific fix quality. This is not the same as showing it is a general
property of mutation testing. Four datapoints in one setting cannot
support that conclusion, and we do not draw it.

The most direct implication we do draw, restricted to the setting of §5:
in the four measurements here, the number of mutation tests the author
wrote with confidence (7, 8, 14, and 14 for the three ee_bot versions
plus minpoint’s implicit set) was not predictive of the count or
severity of findings under blind fresh-agent audit within a small
budget.

A footnote on this paper’s own writing: the scope-limitation in §5 was
added after an earlier draft over-claimed “mutation testing in general”;
peer review by another Claude subagent (not the fresh-agent audit
mechanism this paper counts) surfaced remaining scope violations in
§4, §6, and §7, which we corrected before publication. This is not a
fifth audit datapoint (different mechanism, different check) but does
illustrate that scope discipline requires external checks in this
setting as well.

§Appendix A: Pre-registration and content hashes

round4-preregistration-draft.md frozen at SHA-256
5821fc3b9dee2592d0b9c16c4e0cab5615b5d2598b3c467d2e8b34cae933bbdf
before either arm audit was launched.

Arm α (patch) tarball SHA-256:
970b4ed2a823612cea646d83c20b407edc374d1ec505765857c745787aac4f28

Arm β (restructure) tarball SHA-256:
e3517727aa36c5ae258638b54ab4edc55874889cee359d581825552d575f252d

Paper draft SHA-256 pre-Round-4 (Method + Results + Scope §5 frozen
before either arm audit): ddac1be8368b59369a3e02d779d179f8c30ca4b91abb258bcbffb1a1eb9dcc34

§Appendix B: Reproducibility

All artifacts will be deposited to Zenodo at publication and this
appendix will be updated with the DOI. The Zenodo deposit will include:

• ee_bot-ref-0.1.0.tar.gz, ee_bot-ref-0.2.0.tar.gz, ee_bot-ref-0.3.0.tar.gz
• ee_bot-v0.4-arm-alpha.tar.gz, ee_bot-v0.4-arm-beta.tar.gz
• round4-preregistration-draft.md (with frozen SHA-256 above)
• This paper (post-peer-review revision)
• Round 1–4 auditor transcripts

Independent replication requires: Python 3.11+, pytest, and access to
a Claude subagent (or an equivalent LLM fresh-session mechanism) for
audit reproduction. The audit prompts are included in the Zenodo
deposit for exact-prompt replay; readers wishing to test whether a
different model family surfaces the same pattern class have the exact
prompt to run against a non-Claude model.