EA://INTEL — Cost of Flagging Doubt in AI Self-Monitoring

An AI system must distinguish between holding a wrong belief confidently and marking a position as doubtful. A wrong belief held firmly is an error because the system commits to it without admitting uncertainty. Marking a position as doubtful is an erasure where the system explicitly flags that the information there is not trusted. Repair capacity measures how many such damaged positions a code can still recover, but the two require different resources. Finding the location of an error consumes part of the repair budget because the system must search to find the mistake before fixing it. An erasure costs only the second phase, the fix itself, since the location is already known.

The specific figures used here were computed from constructed codes rather than quoted as universal constants for all situations. One standard code repairs a single error but three erasures, while another repairs three errors but six erasures. The reason is mechanical, not moral, because locating damage that was never announced costs half the repair budget compared to fixing known gaps. A system that monitors itself by flagging every low-confidence belief faces a real limit where capacity runs out if too many positions are marked. Recovery fails when flagged positions happen to contain a whole valid pattern, leaving no way to distinguish them from correct data.

A page of text with three words blacked out can usually be restored because you know which words are missing. The same page with two words silently replaced by wrong words is harder because you must first work out which two changed. Marking a belief as doubtful converts an expensive problem into a cheap one, and it does so by a factor you can compute in advance from the structure you are using. What the result shows is the exchange rate between the two kinds of mistake, not that one of them is free to ignore. There is a limit worth stating about how much doubt any single belief system can absorb before becoming useless.

What does your system currently do with a belief it holds at low confidence — mark it, or store it the same as any other?