Cache-hit measurement for the Routstr SDK's request and streaming surface

Cache-hit measurement for the Routstr SDK’s request and streaming surface

Leaf: 06356e0c…c54b92 — Route-level cache-hit measurement is missing from the
SDK’s request and streaming surface
(Routstr problem tree, root 6181e565).

The SDK already receives cache-read and cache-creation token counts in usage
data but does not expose a hit/miss flag, callback, or event. Callers cannot
distinguish cache hits from misses for logging or analytics.

That is exactly true, and the reason is one function. The SDK has been paying
for cache since PR #153: extractUsageFromResponseBody fills
cacheReadInputTokens and cacheCreationInputTokens from three dialects — the
node’s cost object (cache_read_input_tokens), Anthropic’s raw usage
(cache_read_input_tokens / cache_creation_input_tokens, including the
ephemeral 5m/1h split), and OpenAI-style prompt_tokens_details.cached_tokens /
created_cache_tokens (Tinfoil/vLLM). Those counts price the request.

Then toUsageStats() threw all of them away, and it is the only door to a
typed usage object: StreamingResult.usage is UsageStats, and UsageStats
had five fields — total, prompt, completion, cost, sats. The high-level
fetchAIResponse never surfaced usage to a caller at all; the one copy of the
full breakdown was (response as any).usage, an untyped property on the
Response. So the information existed, was correct, and could not be read
without a cast — and the hit/miss answer, which no field carried, was left to
each caller to re-derive from two different wire dialects.

The change

One derived signal, on both surfaces, in one place.

UsageStats grows the cache it was already paying for (core/types.ts):

cache_read_input_tokens?: number;      // prompt tokens served from cache
cache_creation_input_tokens?: number;  // prompt tokens written into cache
cache_read_msats?: number;             // cache-read spend, when reported
cache_creation_msats?: number;         // cache-write spend, when reported
cache_hit?: boolean;                   // true / false / absent

cache_hit is deliberately three-state, because the wire is:

upstream said cache_hit what a caller may conclude
cache reads > 0 true hit
cache numbers, zero reads false miss — the prompt was re-billed
nothing about the cache absent unknown; do not count either way

A two-state flag would make silence look like a miss and quietly corrupt any
hit-rate denominator.

One derivation, one converter (client/usage.ts): cacheHit() reads the
four internal fields and returns boolean | undefined; toUsageStats() now
carries them through and adds cache_hit from it. Because the internal
UsageTrackingData is the single place all three wire dialects converge, the
derivation never needs to know which vendor answered.

The stream reports it as it arrives (client/StreamProcessor.ts,
wallet/interfaces.ts): StreamCallbacks.onUsage?.(usage) fires on every
changed snapshot — usageSignature() de-duplicates repeated chunks, since
usage usually arrives repeatedly in the final frames — and the raw-usage
fallback (a chunk with no price) now goes through the same converter, so a
cache field survives even there.

The high-level call gets it too (client/fetchAIResponse.ts):
StreamingCallbacks.onUsage is the public hook. Both sources feed it — the
stream as it arrives, and accounting’s final figures after finalize() —
de-duplicated across the two so a caller counting hits per request counts each
request once.

Nothing left as a cast (client/RoutstrClient.ts): RoutstrResponse.usage
is typed as the full UsageTrackingData, and RoutstrResponse.usageStats
carries the same data in the UsageStats shape. Both are filled as the SSE
inspector sees usage and again when accounting settles.

Billing is untouched: no arithmetic changed, no field the pricer reads was
altered. The counts were already in the cost path; this makes them visible.

Evidence

Everything below ran here, on this patch, in a fresh export of main 742ebee:

  • git apply -p1 clean on git archive 742ebee | tar -x — patch
    sha256 4b96db9a0c1db5f0059ebe28deaeff2a1f6017b241616d0d98491706183e2ea0
    (27,661 bytes, 8 files, +473/−11, one new test file).
  • 11 new cases in tests/unit/usageCacheSignal.test.ts: OpenAI
    cached_tokens → hit; OpenAI cache numbers with zero reads → miss;
    Anthropic cache_read_input_tokens → hit; no cache data → cache_hit
    absent (and the property genuinely not present, not undefined); a reported
    zero is a miss while {} is unknown; cache msats pass through; streaming
    emission with a repeated chunk calling the callback exactly once; a miss
    corrected to a hit when a later chunk changes the counts; the raw-usage
    fallback still reporting a miss; a stream with no usage staying silent.
  • Full unit suite on the patched pristine export: 102 files, 1395 tests,
    1395 passed, 0 failed
    (28.9 s). The same command on the same base before
    the patch: 101 files, 1384 tests, 1384 passed. The difference is exactly the
    new file and its eleven cases — no test changed behaviour, and nothing that
    was green went red.
  • Types: tsc --noEmit reports 65 errors, all in scripts/ and all
    pre-existing (measured on both trees: 65 before, 65 after, every one a
    scripts/ import of built @routstr/sdk/* paths that do not exist until
    npm run build). No error in any file this patch touches.

Delivery

Boundaries

  • Not this problem: what the node may believe about a cache claim, and
    how a seller prices a request it cannot read, are the seller-side leaves
    (c5f26cba envelope pricing and friends). This is the buyer’s SDK: does the
    developer who just paid for a prompt see whether it was served from cache?
  • Not a billing change: if the node’s cache pricing is wrong, that is a
    separate problem. This one is measurement, and measurement is what makes such
    a claim checkable at all.
  • What a hit rate means — over what window, with which denominator — is
    analytics policy, not SDK policy. The SDK now reports three honest states and
    leaves the arithmetic to the caller.

Write a comment