<!-- task=m-node-provider-execution-liveness-recovery/02+01_stall_watchdog plan=2 tag=API milestone-task=activity-contract,stall-watchdog -->

# PLAN — Node Response Stall Watchdog

## For the Implementing Agent

> **MANDATORY:** Do not begin until the dependency below has a PASS `complete.log`. Implement only this checklist, preserve unrelated user changes, and keep every edit inside the `stall-watchdog` slice. Do not update roadmap state, create follow-up plans, commit, push, or run an official code review. After implementation, fill every implementation-owned section of `CODE_REVIEW-cloud-G08.md` and leave active files in place for the review agent.

## Background

Node currently invokes normalized providers and raw tunnel adapters synchronously inside handler-owned cleanup. The normalized path defers terminals until admission release, but the tunnel path sends frames directly. Neither path observes provider-originated progress, derives request contexts from transport lifetime, or has an attempt-generation fence. A no-progress timeout therefore cannot safely race provider output, caller deadline, disconnect, cancel, or a provider that ignores cancellation.

This slice consumes the activity/config contract from `01_activity_contract`, installs a single Node-owned watchdog for both execution surfaces, and emits one stable `response_stalled` terminal with a confirmed or unconfirmed local fence. It deliberately emits `provider_health=unknown`; the dependent health-classification slice replaces that bounded fallback with target-aware probe evidence.

The first refinement corrected caller-metadata spoofing of Node-owned attempt identity. This second fresh-context replan closes the remaining material ambiguities before implementation: close grace is exactly `5s` on the injected clock, normalized terminal metadata must be attached to both `Failure.Metadata` and `RuntimeEvent.Metadata` so the existing protobuf mapper cannot drop it, tunnel and normalized terminals must use clones of the same safe map, and verification includes a credential-free real Edge/Node process cycle. No implementation or official review has started.

## Archive Evidence Snapshot

- Prior plan: `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G08_1.log`.
- Prior review stub: `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G08_1.log`.
- Prior verdict: none; implementation and implementation-owned evidence had not started.
- Required carryover: use Node-owned `run_id` as `attempt_id`; fix `defaultAttemptCloseGrace=5s`; clone one allowlisted metadata map onto normalized `Failure.Metadata`, normalized `RuntimeEvent.Metadata`, and the tunnel ERROR frame; prove wire preservation and spoof resistance.

## Dependencies

- `agent-task/m-node-provider-execution-liveness-recovery/01_activity_contract/complete.log`

At plan creation the predecessor is active and not complete. The implementing runtime must wait for its PASS completion, then use the resulting activity helpers, effective timeout lookup, generated wire field, and updated contracts rather than duplicating them.

## Analysis

### Files Read

- `AGENTS.md`
- `agent-ops/rules/project/rules.md`
- `agent-ops/rules/common/rules-roadmap.md`
- `agent-ops/rules/common/rules-agent-spec.md`
- `agent-ops/rules/project/domain/node/rules.md`
- `agent-ops/rules/project/domain/platform-common/rules.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-test/local/rules.md`
- `agent-test/local/node-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-spec/index.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `apps/node/internal/transport/session.go`
- `apps/node/internal/transport/session_test.go`
- `apps/node/internal/node/node.go`
- `apps/node/internal/node/run_handler.go`
- `apps/node/internal/node/run_manager.go`
- `apps/node/internal/node/runtime_sink.go`
- `apps/node/internal/node/runtime_bridge.go`
- `apps/node/internal/node/tunnel_handler.go`
- `apps/node/internal/node/run_cancel_test.go`
- `apps/node/internal/node/provider_tunnel_test.go`
- `apps/node/internal/node/node_test_support_test.go`
- `packages/go/execution/types.go`
- `packages/go/execution/failure.go`
- `packages/go/execution/emitter.go`
- `packages/go/execution/failure_test.go`
- `packages/go/execution/emitter_test.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `scripts/dev/edge-node-reconnect-diagnostic.sh`

### SDD Criteria

- SDD status: approved, D01 resolved, implementation lock released.
- Scenarios: the integrated run/tunnel lifecycle completes S01 / `activity-contract` and implements S02 / `stall-watchdog` (`SDD.md:92-93`).
- Evidence rows: S01 fake-clock activity/deadline/transport lifecycle evidence and S02 threshold, timer/event/cancel/close race, exactly-once terminal, confirmed/unconfirmed fence, and late-event fencing (`SDD.md:103-104`).
- Precedence: request hard deadline or current connection heartbeat/disconnect that wins first keeps its existing boundary (`SDD.md:68,92`).
- Output: normalized `RunEvent{type=error}` and tunnel `ProviderTunnelFrame{kind=ERROR}` exactly once, stable `response_stalled`, safe metadata, and `Retryable` true only when fence is confirmed (`SDD.md:75-77`).
- Prohibitions: no provider-specific watchdogs, no Node retry, no recovery eligibility, no late attempt revival or double resource release (`SDD.md:81-86`).

### Verification Context

- Baseline passed before plan creation:
  - `go test -count=1 ./apps/node/internal/node ./apps/node/internal/transport`
  - `go test -race -count=1 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
- Local deterministic fixtures are sufficient; no external provider or credentials are required.
- Timer tests must use an injected fake clock and synchronization channels, not wall-clock sleeps.
- Runtime execution changes require targeted unit/race tests, vet/full Go suite, auxiliary E2E, and `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh`, which starts the real Edge and Node dev entrypoints against temporary mock config without credentials.

### Test Coverage Gaps

- `terminalDeferringSink` suppresses post-terminal normalized events but has no liveness generation or activity notification.
- Raw `tunnelSink` does not fence late frames or claim one terminal.
- Run/tunnel handlers call providers on the cleanup-owning goroutine, so an adapter that ignores cancellation can retain or prematurely release admission/run ownership incorrectly.
- Session listeners pass `context.Background()` (`apps/node/internal/transport/session.go:50-52,75-87`), so disconnect does not cancel current request handlers.
- No tests cover timer/activity boundary ordering, hard deadline/disconnect precedence, close grace, or confirmed/unconfirmed resource ownership.
- `runEventToProto` serializes `RuntimeEvent.Metadata` and does not automatically forward `Failure.Metadata`; a plan that populates only the failure would silently lose safe liveness evidence on the normalized wire.

### Symbol References

- `apps/node/internal/node/run_handler.go:28-90` — config lock, resolve/admit, timeout context, run registration.
- `apps/node/internal/node/run_handler.go:97-142` — terminal-deferring sink, provider call, ticket/run cleanup, foreground/background return.
- `apps/node/internal/node/run_handler.go:246-265` — synthetic terminal construction.
- `apps/node/internal/node/tunnel_handler.go:41-78,107-154` — lookup/admission, sink, timeout context, run registration, direct tunnel call.
- `apps/node/internal/node/tunnel_handler.go:157-226` — plain tunnel error and direct frame conversion.
- `apps/node/internal/node/runtime_sink.go:25-82` — normalized terminal claim/defer behavior.
- `apps/node/internal/node/run_manager.go:13-51,71-92` — cancellation handle and drain lifecycle.
- `apps/node/internal/transport/session.go:50-87,211-225` — background request contexts and connection done/close boundary.
- `packages/go/execution/failure.go:12-32,83-129` — stable failure vocabulary and normalization.

### Split Judgment

- Classification: large. Correctness depends on temporal state, concurrent exactly-once claims, cancellation and transport precedence, and resource ownership across two execution variants.
- Cohesion: normalized and tunnel paths must share one watchdog/fence primitive so they cannot diverge on activity or terminal semantics.
- Predecessor: `01_activity_contract` is required and encoded in the directory name and Dependencies section.
- Successors: refined `03+02_health_probe_contract` defines the fail-closed probe result and `04+03_health_evidence` enriches the same terminal evidence without changing timer/fence ownership.
- Refinement retention: this already-refined pair remains atomic because observer state, normalized/tunnel terminal authority, cancellation/fence cleanup, and wire evidence form one S02 exactly-once invariant; no child would have an independently reviewable PASS boundary.
- Collision check: no active plan claimed these paths when prepared; dependency sequencing prevents overlap with predecessor contract files.

### Scope Rationale

- In scope: Node request lifetime context, fake-clock watchdog, provider-call isolation, progress reset, exactly-once terminal, cancellation/close grace, late emission fence, confirmed/unconfirmed evidence, and safe cleanup.
- Out of scope: actual target probe, Edge health overlay/binding, lease projection, ingress retry/recovery, metric surface, config schema (owned by predecessor), and provider-adapter-specific timers.
- A provider that ignores cancellation remains locally fenced but holds its Node admission/run ownership until its goroutine actually exits. This preserves capacity and refresh-drain integrity while reporting `attempt_fence=unconfirmed`.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
- Build closure: scope/context/verification/evidence/ownership/decision all closed and trusted; capability gap none.
- Build score: `scope=2`, `state=2`, `blast=1`, `evidence=1`, `verification=2` -> G08; `base_route_basis=local-fit`, `route_basis=risk-boundary`, lane `cloud`, file `PLAN-cloud-G08.md`.
- Build signals: `large_indivisible_context=false`, positive loop risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (`count=4`), `review_rework_count=0`, `evidence_integrity_failure=false`; risk boundary matched, recovery boundary not matched.
- Review closure: scope/context/verification/evidence/ownership/decision all closed and trusted; capability gap none.
- Review score: `scope=2`, `state=2`, `blast=1`, `evidence=1`, `verification=2` -> G08; `route_basis=official-review`, lane `cloud`, adapter/model `codex/gpt-5.6-sol`, reasoning `xhigh`, file `CODE_REVIEW-cloud-G08.md`.

## Implementation Checklist

- [ ] [API-1] Add a shared fake-clock-capable attempt observer and typed response-stalled evidence contract.
- [ ] [API-2] Integrate the observer into normalized execution with safe admission/run cleanup and late-event fencing.
- [ ] [API-3] Integrate the same observer into raw tunnels and bind both request paths to session disconnect.
- [ ] [TEST-1] Prove activity, precedence, threshold races, exactly-once terminal, confirmed/unconfirmed fence, and resource ownership deterministically.
- [ ] [DOC-1] Update the matching execution spec and execution/Edge-Node wire contracts for implemented Node watchdog behavior only.
- [ ] Run every command in Final Verification and record exact results in `CODE_REVIEW-cloud-G08.md`.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [API-1] Build one attempt observer and typed stall terminal

**Problem**

There is no owner for the no-progress clock or for racing provider terminal, watchdog, cancel, and late emission. Adding independent timers to `run_handler.go` and `tunnel_handler.go` would create variant drift and non-deterministic cleanup.

**Solution**

Add `apps/node/internal/node/liveness_watchdog.go` with package-private abstractions:

- an injectable clock/timer interface with a real default on `Node` and a deterministic manual test implementation;
- one attempt observer state machine that accepts the predecessor's `ProviderActivityDisposition`, resets on progress, stops on terminal, and exposes a single atomic/mutex-protected terminal/fence claim;
- an emission authority wrapper for each sink that drops every provider event/frame after the terminal claim and reports activity to the observer before forwarding valid non-terminal output;
- `const defaultAttemptCloseGrace = 5 * time.Second` and a bounded cancel/close grace timer owned by the same injected clock. On threshold, cancel provider execution, revoke provider emission authority immediately, and classify `confirmed` only when the provider call has returned within that exact grace; otherwise classify `unconfirmed`;
- after a timer signal, re-check request context and session lifetime before claiming stall so a simultaneous/earlier hard deadline, caller cancel, or disconnect is never reclassified;
- a stable `FailureCodeResponseStalled` in `packages/go/execution/failure.go`, known-code encoding/decoding support, and one allowlisted metadata builder for `failure_code`, `provider_health=unknown`, `liveness_classification=health_unknown`, `idle_duration_ms`, `run_id`, `attempt_id`, `attempt_fence`, `adapter`, and `target`. Set both identities from the concrete request's Node-owned run identity (`ExecutionSpec.RunID`/wire `run_id`). Clone this map into both normalized `Failure.Metadata` and `RuntimeEvent.Metadata`, and into the tunnel ERROR metadata, so normalized protobuf conversion and raw tunnel transport preserve identical keys without shared mutable aliases. Never derive values from caller-defined metadata or include raw output, prompt, reasoning, credentials, or `recovery_eligible`.

The normalized terminal is an error `RuntimeEvent` carrying `Failure{Code: response_stalled, Retryable: fence == confirmed}`. The tunnel terminal is an ERROR frame with the same safe metadata and stable error text. The health successor will replace only the unknown classification fields.

**Modified files**

- [ ] `packages/go/execution/failure.go`
- [ ] `packages/go/execution/failure_test.go`
- [ ] `apps/node/internal/node/node.go`
- [ ] `apps/node/internal/node/liveness_watchdog.go`

**Test decision**

Required. Unit-test the state machine and failure round-trip independently before handler integration.

**Verification**

- `go test -count=1 ./packages/go/execution ./apps/node/internal/node`
- `go test -race -count=1 ./packages/go/execution ./apps/node/internal/node`

### [API-2] Integrate normalized execution without releasing an unclosed attempt

**Problem**

The run closure invokes `adapter.Execute` directly and owns all defers (`apps/node/internal/node/run_handler.go:106-136`). It cannot emit a terminal while retaining admission/run ownership for an adapter that ignores cancel, and its existing terminal sink has only a boolean terminal observation.

**Solution**

Refactor normalized execution around an explicit provider-call result channel and exactly-once cleanup owner:

- create the observer after resolve/admission using the predecessor's effective timeout lookup;
- execute the provider in one goroutine with the fenced activity sink;
- have the coordinator select among provider return, observer expiry, request deadline/cancel, and session lifetime;
- preserve existing complete/error/cancel synthesis when provider return or context termination wins; on stall, claim and queue the typed stall terminal exactly once;
- release the admission ticket before flushing the terminal only after provider ownership is confirmed closed, preserving the existing Edge wake-up ordering;
- for an unconfirmed provider, emit/flush the terminal but move ticket release, run-manager deregistration, `done` close, credential/cancel cleanup if applicable, and final provider-return drain to one detached cleanup closure. It must execute exactly once when the provider eventually returns; until then refresh drain and capacity continue to see the old attempt;
- retain background request behavior and current store completion semantics, recording the stalled run as error without converting it to caller cancellation;
- replace or extend `terminalDeferringSink` so provider terminal, watchdog terminal, and late provider output share one terminal authority.

Do not start a replacement attempt and do not infer response commit/recovery eligibility.

**Modified files**

- [ ] `apps/node/internal/node/run_handler.go`
- [ ] `apps/node/internal/node/run_manager.go`
- [ ] `apps/node/internal/node/runtime_sink.go`
- [ ] `apps/node/internal/node/liveness_watchdog.go`

**Test decision**

Required. Include foreground and background execution, provider-emitted versus Node-synthesized terminal, cancel race, and admission/run ownership assertions.

**Verification**

- `go test -count=1 ./apps/node/internal/node`
- `go test -race -count=1 ./apps/node/internal/node`

### [API-3] Integrate raw tunnels and transport lifetime

**Problem**

Raw tunnel frames are sent directly and cleanup is deferred around the provider call (`apps/node/internal/node/tunnel_handler.go:107-154`). Session listeners pass background contexts, so an Edge/Node heartbeat disconnect closes the session but not the active request handler.

**Solution**

- Give each `Session` a connection-lifetime context canceled exactly once when `Done()` closes. Derive run and tunnel listener contexts from it and cancel per-request children on handler return; leave command/cancel listener semantics unchanged unless required for leak-free shared context plumbing.
- Add the same observer/fencing sink to tunnels. Response-start/header, non-empty body, and usage reset via the predecessor classifier; END/ERROR terminates; empty frames do nothing.
- Run `TunnelProvider` through the same result-channel coordination and cleanup invariants as normalized execution. On stall emit exactly one ERROR frame with typed metadata, then drop all late frames. On unconfirmed close, retain admission ticket/run handle until actual adapter return.
- When request hard deadline/caller cancellation/session disconnect wins first, cancel and finish through the existing error/transport boundary; do not synthesize `response_stalled`. A dead session must not be treated as confirmed provider progress or be revived for terminal delivery.
- Keep tunnel credential material zeroization tied to the real provider ownership lifetime; never return while a still-running adapter retains plaintext and then zero the buffer underneath it.

**Modified files**

- [ ] `apps/node/internal/transport/session.go`
- [ ] `apps/node/internal/node/tunnel_handler.go`
- [ ] `apps/node/internal/node/liveness_watchdog.go`

**Test decision**

Required. Session tests prove disconnect cancellation and no leaked request context. Tunnel tests prove hard deadline/disconnect precedence, terminal once, late frame drop, and confirmed/unconfirmed retention.

**Verification**

- `go test -count=1 ./apps/node/internal/transport ./apps/node/internal/node`
- `go test -race -count=1 ./apps/node/internal/transport ./apps/node/internal/node`

### [TEST-1] Exercise timer and cleanup boundaries without sleeps

**Problem**

The acceptance boundary is defined by race outcomes. Ordinary happy-path tests and real `time.Sleep` cannot prove deterministic ordering or absence of double cleanup.

**Solution**

Add a package-internal fake clock plus channel-controlled providers and table tests covering:

- start/no-reset, non-empty text/reasoning reset, response-start/header/body/usage reset, empty frame no-reset, and terminal stop;
- exact threshold minus one tick versus threshold, event-at-threshold, provider terminal-at-threshold, caller cancel, hard deadline, and session disconnect;
- provider returns within close grace (`confirmed`) and ignores cancel beyond grace (`unconfirmed`), including eventual return;
- one normalized terminal and one tunnel ERROR only, late delta/frame dropped, retryable only when confirmed;
- caller metadata containing spoofed `run_id` or `attempt_id` cannot override the Node-produced terminal identity on either surface; normalized domain failure, normalized protobuf event, and tunnel frame all preserve the same allowlisted values;
- admission count, run-manager presence, drain wait, credential lifetime, and release exactly once for both fence outcomes;
- `go test -race` with repeated boundary cases; assertions use channels/manual clock, never scheduler sleeps.

Extend the closest existing integration tests rather than duplicating all transport fixtures. Use the new focused test file for the shared state machine and cross-surface tables.

**Modified files**

- [ ] `apps/node/internal/node/liveness_watchdog_test.go`
- [ ] `apps/node/internal/node/run_cancel_test.go`
- [ ] `apps/node/internal/node/provider_tunnel_test.go`
- [ ] `apps/node/internal/transport/session_test.go`

**Test decision**

Required; this is the primary acceptance evidence for S01 lifecycle portions and S02.

**Verification**

- `go test -count=10 ./apps/node/internal/node ./apps/node/internal/transport`
- `go test -race -count=3 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`

### [DOC-1] Document Node watchdog and wire terminal semantics

**Problem**

The execution and Edge-Node wire contracts do not describe `response_stalled`, local fence meaning, precedence, or the safe terminal metadata implemented here.

**Solution**

Update the matching execution spec and both contracts to match code:

- Node owns detection, cancel, local emission fence, and local execution/transport close classification;
- exact activity reset and hard-deadline/disconnect precedence inherited from the predecessor;
- normalized/tunnel terminal shapes and metadata, including `provider_health=unknown` until bounded classification completes in the next slice;
- confirmed versus unconfirmed ownership and retryable-as-capability-only semantics;
- no Node retry, no `recovery_eligible`, no Edge overlay or stale-binding application in this slice.

Do not update roadmap or spec state; implementation contracts change alongside code.

**Modified files**

- [ ] `agent-contract/inner/execution-runtime.md`
- [ ] `agent-contract/inner/edge-node-runtime-wire.md`
- [ ] `agent-spec/runtime/edge-node-execution.md`

**Test decision**

No separate doc test; review maps contract statements to deterministic tests.

**Verification**

- `git diff --check`

## Modified Files Summary

| Path | Action | Checklist |
|------|--------|-----------|
| `packages/go/execution/failure.go` | modify | API-1 |
| `packages/go/execution/failure_test.go` | modify | API-1 |
| `apps/node/internal/node/node.go` | modify | API-1 |
| `apps/node/internal/node/liveness_watchdog.go` | add | API-1, API-2, API-3 |
| `apps/node/internal/node/run_handler.go` | modify | API-2 |
| `apps/node/internal/node/run_manager.go` | modify | API-2 |
| `apps/node/internal/node/runtime_sink.go` | modify | API-2 |
| `apps/node/internal/transport/session.go` | modify | API-3 |
| `apps/node/internal/node/tunnel_handler.go` | modify | API-3 |
| `apps/node/internal/node/liveness_watchdog_test.go` | add | TEST-1 |
| `apps/node/internal/node/run_cancel_test.go` | modify | TEST-1 |
| `apps/node/internal/node/provider_tunnel_test.go` | modify | TEST-1 |
| `apps/node/internal/transport/session_test.go` | modify | TEST-1 |
| `agent-contract/inner/execution-runtime.md` | modify | DOC-1 |
| `agent-contract/inner/edge-node-runtime-wire.md` | modify | DOC-1 |
| `agent-spec/runtime/edge-node-execution.md` | modify | DOC-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md` | update evidence | all |

## Final Verification

1. `go version && go env GOMOD`
2. `go test -count=1 ./packages/go/execution ./packages/go/streamgate ./packages/go/config`
3. `go test -count=1 ./packages/go/execution ./apps/node/...`
4. `go test -count=10 ./apps/node/internal/node ./apps/node/internal/transport`
5. `go test -race -count=3 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
6. `go vet ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
7. `go test -count=1 ./...`
8. `./scripts/e2e-smoke.sh`
9. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
10. `make readability-audit`
11. `git diff --check`

Record exact results and any environment-only E2E limitation in the review stub. A failed required deterministic or race test is a blocker; do not substitute a live provider smoke for these fixtures.

**After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.**
