Attested #5: Phala says an attacker could see in-flight requests on one inference VM

OpenRouter users of DeepSeek V4 Flash 0731 are in a precautionary window through Nov 8. Plus Notion's Fireworks transcripts and a $159 interposer that forges TDX attestation
Attested #5: Phala says an attacker could see in-flight requests on one inference VM

Phala: an unhardened inference VM and a wide exposure window

Phala’s preliminary incident report came out Nov 10.

If OpenRouter routed your DeepSeek V4 Flash 0731 requests to Phala between Oct 21, 00:00 UTC and Nov 8, 05:35 UTC, Phala says to treat them as exposed and rotate any keys, tokens, passwords or other secrets in them. Phala served the model only there; OpenRouter is notifying users. As of Nov 17, OpenRouter lists Phala on no model. Its Phala page shows daily traffic falling from about 33.5 billion tokens on Nov 10 to about 650,000 on Nov 11.

Some confidential VM owners are also affected. The attacker took application credentials, and secrets derived from Phala’s key management system may have been exposed. Phala contacted the affected tenant, and confirmed misuse of some stolen credentials belonging to one tenant. No notice means no action needed, Phala says.

On Oct 21 the attacker reached internal management interfaces on one server and planted malware. Phala detected it on Nov 5. The attacker chained a legacy internet-facing service with management interfaces that “relied on loopback access for trust and did not independently authenticate callers,” created unauthorized CVMs and took application credentials. Phala closed those paths and rotated the KMS root key on Nov 6.

On Nov 8 a third-party researcher showed DeepSeek requests were exposed. The cause was a new GPU server “serving production traffic without our full production security hardening.” The attacker took over that inference VM the same way, “making in-flight request data visible during processing.” Phala pulled the model, treats both intrusions as one incident by one actor, and says it was contained at 05:35 UTC on Nov 8. Service is back with added protections. A platform audit found no compromise of other models or products.

A researcher had planted unique canary credentials in requests from July 1 to Nov 7. Only canaries from Oct 21 and 22 were misused, first at 05:41 UTC on Oct 22, last on Nov 9. Phala rightly kept the wider window because the tests only sampled requests.

The report calls the second group “confidential virtual machines” but the DeepSeek box an “inference VM” on a “new GPU server,” and never says whether it ran in a TEE.

dstack, the open-source stack Phala Cloud runs on, has shipped two releases since the report. 0.6.4 (Nov 12) stops serving KMS key handover on the public port by default, and the KMS won’t start without an authenticated admin listener. 0.6.5 (Nov 15) adds launch tickets: the KMS releases an app’s keys only if the host presents a secret set at launch.


Fireworks, and what Notion says was in the stored data

Fireworks’ bulletin, last updated Nov 9, says the evidence “indicates links” to a group selling unauthorized LLM access through illicit routing gateways, “known as LLMJacking,” citing Google Threat Intelligence Group and Mandiant’s Nov 2 post. CEO Lin Qiao posted a longer account Nov 10.

Notion’s advisory is dated Nov 12, with updates Nov 11 and 13. Notion used Fireworks for Insights classification and open-weight inference. Per Notion, the dataset Fireworks confirmed on Nov 8 held Notion AI chat transcripts used for Insights, from under 1% of workspaces, mostly “unattributable”: no user or workspace label, no tool-call results. Notion had cut off Fireworks on Nov 3 and deleted its data there Nov 4. Insights is off and open-weight models no longer go through Fireworks. Customers should rotate credentials pasted into Notion AI. It revoked exposed Notion API keys on Nov 11 and reported exposed third-party keys to their vendors on Nov 12.

The bulletin reports no impact to Inference, the Training SDK or Nexus, which are ZDR by default. Qiao narrows that to “Inference (excluding batch inference).” With Palo Alto Networks Unit 42, Fireworks replaced credentials, cut elevated privileges and isolated affected systems. On Nov 15 it retired its dataset APIs: it no longer stores new datasets, and training and batch jobs read from local files or the customer’s own storage. Tinfoil’s Nov 16 post compares “zero data retention” across a dozen providers, with carve-outs for batch, caches and tools.


NEAR, a week later

NEAR’s Nov 9 incident report says a remote code execution bug in SGLang gave the attacker code execution on one inference server, where requests, including confidential ones, “could” have been observed “on November 8.” NEAR patched its servers, reported the bug upstream, and says it “will make any regulatory notifications that are required.”

NEAR’s public deployment repo shows the work. On Nov 11 NEAR added an image of SGLang v0.5.23 with three patches: aborts that match only the caller’s request ID (CVE-2026-112266 let clients abort other users’ requests), credential redaction, and customer data scrubbed from logs. By Nov 13 NEAR had moved models to v0.5.23, put some engines on internal-only networks without Hugging Face tokens, and made its proxy fetch images and audio itself and refuse video URLs. Upstream, SGLang merged a breaking change on Nov 14 (#49896) that puts weight loading, LoRA loading, profiling and other admin endpoints behind a separate admin key.


A $159 interposer that forges attestation, and vendor bulletins that don’t need one

The DDRop paper is out. The talk is Nov 18, in the morning “Attacks and defenses” session at ACM CCS in The Hague.

DDRop is a DDR5 RDIMM interposer, $159 in parts at a build of 10, labor excluded. At native DDR5 speed it injects a parity error and suppresses the alert, so a write is dropped and stale data reads as current. Intel TDX, Scalable SGX and AMD SEV-SNP skip cryptographic freshness for performance and capacity, so stale ciphertext still decrypts.

It needs no guest bug, only the hypervisor’s existing interfaces for moving protected memory. On TDX, under the default logical-integrity mode, it forces a victim trust domain into debug mode (50% per attempt) and dumps its memory. Even with cryptographic integrity it forges an attacker TD’s launch measurement so a backdoored VM gets a valid attestation report. On SEV-SNP on EPYC Turin, it copies victim pages through the page relocation interface. Schematics and firmware are on GitHub. Intel and AMD call physical DRAM attacks out of scope. AMD’s AMD-SB-3048 (Sep 14) assigns no CVE and plans no mitigation.

On Nov 10 AMD published SEV-SNP bulletins for two attacks needing only a malicious hypervisor, which SEV-SNP is meant to stop. Mior also needs a modified BIOS.

Mior (CVE-2026-0462, AMD-SB-3059), from ETH Zurich, is in the same CCS session as “Pluto: Breaking SEV-SNP with Software Defined Memory.” AMD says coarse access control in SEV firmware lets a hypervisor redirect MMIO and break guest memory integrity (CVSS 5.9). With the Data Fabric routing registers left unlocked by the BIOS, they hide MMIO routing from the PSP’s memory map and back it with GPU memory. PSP checks pass, the reverse-map table is never set up, and the hypervisor can enable debug on a production confidential VM and forge attestation reports. Shown on a Zen 5 EPYC 9135; source review suggests Zen 3 and 4 may be too. The fix is SEV firmware dated Aug 12, with BIOS targeted for December 2026. A patched host reports TCB[SNP] 0x1F on EPYC 7003, 0x1E on 8004 and 9004, and 0x8 on 8005 and 9005, so guests can refuse older hosts in attestation. Supermicro lists fixed BIOS versions for its H12, H13 and H14 boards.

Skyfall (CVE-2026-40671, AMD-SB-3061), from ETH Zurich and Google, is a transient-execution side channel. AMD says incomplete cleanup in VMRUN error handling may leak SEV-SNP guest registers via a Transient Scheduler Attack (CVSS 4.9). The researchers abort a VMRUN after guest state is decrypted into cache and sample it. Microcode is dated Aug 12, Sep 21 for MI300A; platform firmware is targeted for November 2026.

In AMD-SB-3064, also Nov 10, Chinese Academy of Sciences and Tsinghua researchers used Rowhammer to alter the SEV-SNP reverse-map table with ECC off. No CVE. AMD says keep ECC on.

Intel’s INTEL-SA-01580, also Nov 10, covers 30 CVEs in the TDX Migration Trust Domain (MigTD), used to live-migrate confidential VMs. Rated medium overall, three are critical: CVE-2026-61887 (9.4), a missing authentication step in SPDM rebind code that could let an unauthenticated attacker on the adjacent network run code; CVE-2026-44685 (9.3), spoofing-based authentication bypass in rebinding; CVE-2026-61473 (9.2), replay-based authentication bypass. Intel found 29 of the 30 itself. MigTD 0.7.2, released Sep 4, fixes them, with SPDM, TLS and key-wiping hardening from a security audit. Other TDX guests aren’t affected.

Per INTEL-SA-01520, Intel’s TDX Workload Attestation library before 0.2.0 accepted certificates outside their validity period (CVE-2026-42067, 5.9). The validity check landed in May and shipped in 0.2.0 on Sep 4.

At CCS, Ruhr University Bochum examined 115 TEE deployments on SGX, TDX and SEV-SNP: 91% weren’t reproducible, and 80% published neither source nor a reference build, so nobody outside can tie the quoted measurement to code. The talk was Nov 17. Also at CCS, From Prompt to Pwn, from KU Leuven’s DistriNet, has remote unprivileged users trigger GPU memory errors in PyTorch and cuDF with crafted inputs, then turn them into data leaks and GPU code injection.


What we measured

From the public attestation page at 07:04 UTC Nov 17, and the providers’ APIs.

66 of 93 checks pass hardware and TCB, 8 of 92 pass request route, none pass every gate, and 12 were unavailable. Two fail outright, both NanoGPT AMD SEV-SNP routes with TSME off: Kimi K3.1 Pro, failing since #4, and new Inkling 2 Small. NanoGPT’s DeepSeek V4.1 Flash, #4’s third TSME failure, now answers from Intel TDX and reads incomplete.

Tinfoil added Inkling 2 Small on Nov 14, experimental, at $0.30/$1.00 per million tokens ($0.03 cached). Kimi K3 is gone, as the changelog announced for Nov 12.

Enclave42 joined the directory with 22 models, priced per token (Gemma 4 31B TEE at $0.156/$0.481) on top of a plan that unlocks each model (Build is $99/month). Its evidence bundles need an account, so we can’t monitor it; these are its claims.


New names I’m looking at

NEAR AI Gateway (announcement, Nov 11), a macOS app, points Claude Code, Codex and OpenCode at NEAR AI Cloud. Private models run in enclaves; incognito models go to external providers on NEAR’s credentials, and those providers see the requests. On Nov 12 NEAR added Claude Fable 5.2 at $8/$40 per million, listed in its model API as an external Anthropic route, verifiable: false.

Maple Work (announcement, Nov 12), a desktop agent for Mac, Windows and Linux on paid plans. Maple says models run in enclaves and it can’t see chats; agent hand-offs, web search and connected apps are outside that, each behind a switch.

Fount (fount.sh, Show HN, Nov 4) says it runs spreadsheet and document agents on models in Tinfoil enclaves. Not monitored.

Solana Confidential Compute (announcement, Nov 13), built with Nito, verifies TDX and SEV-SNP attestations on-chain. It’s an unaudited devnet beta; computation stays off-chain.

OVHcloud OCCAM (Labs page), attestation for OVHcloud Bare Metal, free EU alpha from Nov 18. Verifies SEV-SNP on Turin and Genoa and TDX on Xeon 6, single-socket, and releases disk keys only to images whose measurements you signed. GPU servers, which OVHcloud says it already verifies with eight NVIDIA B300s per TDX VM, aren’t in the alpha.

Rafay and Fortanix (announcement, Nov 12) offer attested GPU pods on CNCF Confidential Containers, with Fortanix releasing keys after CPU and GPU evidence checks.

Phala’s reference values page (Nov 17) explains how each TDX measurement is computed, what changes it, and which values Phala publishes. On GPU machines RTMR3 changes every boot.


Also this week

  • Open Enclave SDK 0.19.22 (Nov 13) fixes three bugs letting a malicious host attack SGX enclaves. The critical one, GHSA-2rx6-6737-vpgh, is a use-after-free in the attester registry that gives the host code execution in the enclave.

  • NvRoot (CVE-2026-47591), detailed by Stealthium Nov 12: an NVIDIA driver bug lets an unprivileged process in a GPU container make the GPU write to read-only files and get host root. NVIDIA fixed it Aug 3 in drivers 610.57.04, 595.91.07 and 580.178.04, making the GPU mapping read-only whenever the CPU mapping is, and listed it in its Sep 30 bulletin. Stealthium says the R590, R575, R570 and older branches remain unpatched.


Every issue, plus the live provider data behind it, is archived at confidentialinference.net. You can follow attestation changes as they happen via the alerts RSS.

If you run one of these services and I got something wrong, reply to this email and I’ll fix it in public.

Aljaz


Write a comment