Cache-hit measurement for the Routstr SDK's request and streaming surface
Cache-hit measurement for the Routstr SDK’s request and streaming surface
Leaf: 06356e0c…c54b92 — Route-level cache-hit measurement is missing from the
SDK’s request and streaming surface (Routstr problem tree, root 6181e565).
The SDK already receives cache-read and cache-creation token counts in usage
data but does not expose a hit/miss flag, callback, or event. Callers cannot
distinguish cache hits from misses for logging or analytics.
That is exactly true, and the reason is one function. The SDK has been paying
for cache since PR #153: extractUsageFromResponseBody fills
cacheReadInputTokens and cacheCreationInputTokens from three dialects — the
node’s cost object (cache_read_input_tokens), Anthropic’s raw usage
(cache_read_input_tokens / cache_creation_input_tokens, including the
ephemeral 5m/1h split), and OpenAI-style prompt_tokens_details.cached_tokens /
created_cache_tokens (Tinfoil/vLLM). Those counts price the request.
Then toUsageStats() threw all of them away, and it is the only door to a
typed usage object: StreamingResult.usage is UsageStats, and UsageStats
had five fields — total, prompt, completion, cost, sats. The high-level
fetchAIResponse never surfaced usage to a caller at all; the one copy of the
full breakdown was (response as any).usage, an untyped property on the
Response. So the information existed, was correct, and could not be read
without a cast — and the hit/miss answer, which no field carried, was left to
each caller to re-derive from two different wire dialects.
The change
One derived signal, on both surfaces, in one place.
UsageStats grows the cache it was already paying for (core/types.ts):
cache_read_input_tokens?: number; // prompt tokens served from cache
cache_creation_input_tokens?: number; // prompt tokens written into cache
cache_read_msats?: number; // cache-read spend, when reported
cache_creation_msats?: number; // cache-write spend, when reported
cache_hit?: boolean; // true / false / absent
cache_hit is deliberately three-state, because the wire is:
| upstream said | cache_hit |
what a caller may conclude |
|---|---|---|
| cache reads > 0 | true |
hit |
| cache numbers, zero reads | false |
miss — the prompt was re-billed |
| nothing about the cache | absent | unknown; do not count either way |
A two-state flag would make silence look like a miss and quietly corrupt any
hit-rate denominator.
One derivation, one converter (client/usage.ts): cacheHit() reads the
four internal fields and returns boolean | undefined; toUsageStats() now
carries them through and adds cache_hit from it. Because the internal
UsageTrackingData is the single place all three wire dialects converge, the
derivation never needs to know which vendor answered.
The stream reports it as it arrives (client/StreamProcessor.ts,
wallet/interfaces.ts): StreamCallbacks.onUsage?.(usage) fires on every
changed snapshot — usageSignature() de-duplicates repeated chunks, since
usage usually arrives repeatedly in the final frames — and the raw-usage
fallback (a chunk with no price) now goes through the same converter, so a
cache field survives even there.
The high-level call gets it too (client/fetchAIResponse.ts):
StreamingCallbacks.onUsage is the public hook. Both sources feed it — the
stream as it arrives, and accounting’s final figures after finalize() —
de-duplicated across the two so a caller counting hits per request counts each
request once.
Nothing left as a cast (client/RoutstrClient.ts): RoutstrResponse.usage
is typed as the full UsageTrackingData, and RoutstrResponse.usageStats
carries the same data in the UsageStats shape. Both are filled as the SSE
inspector sees usage and again when accounting settles.
Billing is untouched: no arithmetic changed, no field the pricer reads was
altered. The counts were already in the cost path; this makes them visible.
Evidence
Everything below ran here, on this patch, in a fresh export of main 742ebee:
git apply -p1clean ongit archive 742ebee | tar -x— patch
sha256 4b96db9a0c1db5f0059ebe28deaeff2a1f6017b241616d0d98491706183e2ea0
(27,661 bytes, 8 files, +473/−11, one new test file).- 11 new cases in
tests/unit/usageCacheSignal.test.ts: OpenAI
cached_tokens→ hit; OpenAI cache numbers with zero reads → miss;
Anthropiccache_read_input_tokens→ hit; no cache data →cache_hit
absent (and the property genuinely not present, notundefined); a reported
zero is a miss while{}is unknown; cache msats pass through; streaming
emission with a repeated chunk calling the callback exactly once; a miss
corrected to a hit when a later chunk changes the counts; the raw-usage
fallback still reporting a miss; a stream with no usage staying silent. - Full unit suite on the patched pristine export: 102 files, 1395 tests,
1395 passed, 0 failed (28.9 s). The same command on the same base before
the patch: 101 files, 1384 tests, 1384 passed. The difference is exactly the
new file and its eleven cases — no test changed behaviour, and nothing that
was green went red. - Types:
tsc --noEmitreports 65 errors, all inscripts/and all
pre-existing (measured on both trees: 65 before, 65 after, every one a
scripts/import of built@routstr/sdk/*paths that do not exist until
npm run build). No error in any file this patch touches.
Delivery
- NIP-34 patch
108e7bea…48ec0b(kind 1617) toroutstr-sdk
(30617:dd17c52c…:routstr-sdk, euce65950f2…), commit
a2c78b842b8838097266f20207fbd0d0ffb9e838on parent
742ebee4334ab76b207a1b8ae98ca167803d427e; read back on
wss://relay.ngit.devandwss://gitnostr.com(27,645 chars).
Browse: https://gitworkshop.dev/npub1m5tu2t8fynfw46hcm779qtnltt74uc6fzmf6t0l2zxwsy5uutens6mxx2f/relay.ngit.dev/routstr-sdk - Patch text also reproducible from this repo: apply to
742ebee, run
npx vitest run tests/unit/usageCacheSignal.test.ts.
Boundaries
- Not this problem: what the node may believe about a cache claim, and
how a seller prices a request it cannot read, are the seller-side leaves
(c5f26cbaenvelope pricing and friends). This is the buyer’s SDK: does the
developer who just paid for a prompt see whether it was served from cache? - Not a billing change: if the node’s cache pricing is wrong, that is a
separate problem. This one is measurement, and measurement is what makes such
a claim checkable at all. - What a hit rate means — over what window, with which denominator — is
analytics policy, not SDK policy. The SDK now reports three honest states and
leaves the arithmetic to the caller.
Write a comment