Session-Cleanup Heartbeat Pattern
Session-Cleanup Heartbeat Pattern
Long-running OpenClaw agents face a persistent operational hazard: orphaned sessions. When an agent crashes mid-task, loses connectivity, or gets force-killed by the OS, the sessions it owned don’t automatically evaporate. They linger in resource pools, hold file locks, occupy database connections, and — in multi-agent environments — block downstream agents that are waiting for those resources to free up. The Session-Cleanup Heartbeat Pattern is a disciplined approach to detecting and reclaiming these orphaned sessions before they accumulate into a systemic failure.
What the Pattern Solves
Without a heartbeat mechanism, session state is managed purely through explicit open/close calls. This works when everything goes right. Under failure conditions — OOM kills, SIGKILL signals, network partitions, kernel panics — the close call never happens. You’re left with sessions in a RUNNING or ACQUIRED state that have no living owner.
The heartbeat pattern inverts this assumption. Instead of trusting that sessions will be explicitly closed, it treats every session as expired-by-default unless the owning agent actively proves otherwise on a fixed interval. Sessions that miss their heartbeat window are marked for cleanup automatically.
Core Implementation
The pattern requires three components: a heartbeat emitter in the agent process, a heartbeat receiver (usually a sidecar or centralized session registry), and a reaper that acts on missed heartbeats.
Heartbeat emitter (agent side):
import threading
import time
import requests
HEARTBEAT_INTERVAL = 15 # seconds
SESSION_TTL = 45 # seconds — must be > 3x interval to tolerate transient failures
def heartbeat_loop(session_id: str, agent_token: str, stop_event: threading.Event):
while not stop_event.is_set():
try:
resp = requests.post(
"http://session-registry/heartbeat",
json={"session_id": session_id, "timestamp": time.time()},
headers={"Authorization": f"Bearer {agent_token}"},
timeout=5,
)
if resp.status_code == 410:
# Registry has already expired this session — stop the agent
raise SessionExpiredError(f"Session {session_id} was reaped by registry")
except requests.Timeout:
pass # transient — next beat will catch up
stop_event.wait(HEARTBEAT_INTERVAL)
Run this loop in a daemon thread so it doesn’t block agent task execution, but ensure it shares the same process lifecycle. If the process dies, the thread dies with it, and the heartbeats stop.
Registry receiver updates a last_seen timestamp in a fast key-value store (Redis is common):
# Inspect current session state
redis-cli HGETALL session:abc-1234
# Expected output:
# 1) "agent_id"
# 2) "worker-7"
# 3) "last_seen"
# 4) "1718823042.331"
# 5) "ttl"
# 6) "45"
# 7) "status"
# 8) "alive"
The reaper runs on a separate tick (typically every 10–20 seconds) and scans for sessions where now - last_seen > ttl:
def reap_expired_sessions(registry_client, current_time: float):
expired = registry_client.scan_expired(before=current_time - SESSION_TTL)
for session in expired:
registry_client.mark_status(session.id, "reaped")
release_resources(session) # unlock files, close DB connections, free pool slots
emit_audit_event("session.reaped", session_id=session.id, agent_id=session.agent_id)
Choosing the Right TTL
The TTL is the most operationally sensitive parameter. Set it too tight and transient network hiccups will cause false-positive reaps, which creates instability in healthy agents. Set it too loose and truly dead sessions persist long enough to cause resource exhaustion.
A reliable rule: TTL >= 3 * HEARTBEAT_INTERVAL. This tolerates two consecutive missed beats before triggering cleanup. If your network SLA is poor or agents run across availability zones with higher latency variance, extend to 5 * HEARTBEAT_INTERVAL.
During incidents, you can inspect reap rates to detect instability:
# Count reap events in the last 60 minutes
grep "session.reaped" /var/log/openclaw/audit.log \
| awk -F'"timestamp":' '{print $2}' \
| awk '{print $1}' \
| awk -v cutoff=$(date -d '60 minutes ago' +%s) '$1 > cutoff' \
| wc -l
A reap rate above ~2–3% of total session opens in any window is a signal worth investigating — it usually indicates either an agent crash pattern or a TTL that’s too aggressive for current network conditions.
Handling the 410 Gone Response
When the registry reaps a session, it should return HTTP 410 to any subsequent heartbeat from an agent that hasn’t realized it’s been reaped. This is the signal to the agent that its session is no longer valid. Ignoring this response is a common mistake — agents that continue operating after their session has been reaped can produce split-brain behavior where two agents believe they own the same resource.
The correct response to a 410 is immediate, graceful shutdown of the agent’s current task, followed by a re-registration attempt if the agent is capable of recovery. Log the event explicitly:
ERROR session=abc-1234 agent=worker-7 event=session_reaped_during_operation
task=file_sync resource=/data/output.parquet action=abort_and_release
Diagnostic Checklist
When sessions are piling up in reaped or zombie states and not being cleaned, work through this sequence:
# 1. Verify the reaper is running
systemctl status openclaw-session-reaper
# 2. Check reaper last execution time
redis-cli GET reaper:last_run
# 3. Manually inspect a stuck session
redis-cli HGETALL session:<id>
# 4. Force-expire a specific session (break-glass)
redis-cli HSET session:<id> status expired
redis-cli HSET session:<id> last_seen 0
# 5. Confirm resource release after forced expiry
lsof | grep <file_lock_pattern>
Write a comment