<!-- task=m-node-provider-execution-liveness-recovery/06+05_health_overlay plan=1 tag=REFACTOR milestone-task=failure-handoff -->

# Reception-Fenced Provider Health Overlay

## For the Implementing Agent

Implement only the items below after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G09.md` with actual notes and raw command output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

Typed failures are not authoritative merely because they name a node or provider: Edge must bind them to the actual receiving connection and the immutable lease that dispatched the attempt. S04 also requires a generation/sequence-fenced runtime overlay that changes admission and snapshots without mutating config health, while every valid terminal still releases its old lease exactly once.

## Archive Evidence Snapshot

- Prior pair: `plan_local_G08_0.log` and `code_review_cloud_G08_0.log` in this task directory. It was unimplemented and has no official verdict, Required/Suggested/Nit finding, code change, or verification evidence.
- Material self-review finding: the prior overlay could be lowered by terminal evidence, but its only recovery input was a test helper; production had no bounded status-probe response carrying the same connection-scoped sequence, so S04 recovery could not occur outside fixtures.
- Replan carryover: retain reception/lease fencing and release-once scope, add a real exact-target CAPABILITIES status-probe path, and add queue/full-cycle verification. Predecessor `05+04_failure_wire` remains active and must produce `complete.log` before implementation.

## Analysis

### Files Read

- `apps/edge/internal/node/registry.go`, `apps/edge/internal/node/registry_test.go`
- `apps/edge/internal/transport/server.go`, `apps/edge/internal/transport/connection_handlers.go`, `apps/edge/internal/transport/server_test.go`
- `apps/edge/internal/bootstrap/runtime.go`, `apps/edge/internal/bootstrap/runtime_refresh_test.go`
- `apps/edge/internal/service/service.go`, `apps/edge/internal/service/provider_tunnel.go`, `apps/edge/internal/service/provider_pool.go`, `apps/edge/internal/service/provider_resolution.go`
- `apps/edge/internal/service/model_queue_types.go`, `apps/edge/internal/service/model_queue_admission.go`, `apps/edge/internal/service/model_queue_release.go`, `apps/edge/internal/service/model_queue_snapshot.go`, `apps/edge/internal/service/model_queue_test_support_test.go`, `apps/edge/internal/service/model_queue_admission_test.go`, `apps/edge/internal/service/queue_dispatch_test.go`
- `apps/edge/internal/service/node_command.go`, `apps/node/internal/node/command_handler.go`, `apps/node/internal/node/command_test.go`, `apps/node/internal/node/health_probe.go`, `apps/node/internal/node/health_probe_test.go`, `apps/node/internal/transport/session.go`
- `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-node-runtime-wire.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/edge-node-execution.md`, `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-test/local/edge-smoke.md`, `agent-test/local/node-smoke.md`, `scripts/e2e-smoke.sh`, `scripts/e2e-provider-capacity-smoke.sh`, `scripts/e2e-long-context-admission-smoke.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`, and this pair's first-line id is `milestone-task=failure-handoff`.
- Acceptance Scenario S04 and Evidence Map S04 require absent provider identity, stale connection/sequence, and identity mismatch to leave projection unchanged; only current bound fresh evidence may mark/recover overlay health, and terminal lease release is exactly once. A validated `unknown` probe does not change provider-wide health but still preserves a confirmed request-local stall handoff so the ingress owner may try a different provider.
- The S04 transition table fixes semantics: `unavailable` lowers; higher-sequence same-generation `available` from a later bounded exact-target status probe recovers prior unavailable; request-stalled/available and unknown do not lower. REFACTOR-1 covers reception/binding, REFACTOR-2 covers transition/admission/snapshot/release, REFACTOR-3 provides that production probe input, and the final commands include race/ordering/full-cycle fixtures.

### Verification Context

- Handoff supplied starting HEAD `56e7d78af3cda4a8d6a85af091ad26bce935f8b6`; baseline package tests passed fresh. This plan assumes `05+04_failure_wire/complete.log` exists and its optional failure fields compile.
- Current transport callbacks pass only a message although listener closures retain `*toki.TcpClient`; the registry already owns monotonic connection generations and compare-by-client fencing primitives.
- Existing queue leases hold node/provider/generation but omit adapter/target; provider resources hold immutable config capacity/enable plus connection generation but no observed health sequence.
- The existing CAPABILITIES command already reaches `ProviderProber`, but it bypasses the fail-closed `ProbeHealth` normalizer, does not allocate `Session.NextHealthObservationSeq`, and Edge returns the result without applying it. That path is the bounded on-demand S04 recovery input after this replan; ambiguous adapter/target -> provider binding or a stale response must be a no-op.
- External verification preflight was run from `/config/workspace/iop-s1` at HEAD `56e7d78af3cda4a8d6a85af091ad26bce935f8b6`: `bash scripts/e2e-long-context-admission-smoke.sh --preflight` passed `configs/edge.yaml` validation but returned rc=3 because `http://toki-labs.com:18083/v1/models` and the runner-local status URL were unreachable. No binary/artifact override or token was present; the script assumes the configured dev provider pool and its documented host/ports. The implementer must rerun preflight on a source-synchronized authorized dev runner, then run an applicable `--scenario`; inability is a verification blocker, not permission to claim PASS. Deterministic local coverage remains `e2e-provider-capacity-smoke.sh` plus focused/race tests.
- Confidence is medium because registry, queue, transport, command response, Node sequence, and live provider-pool behavior now form one temporal boundary.

### Test Coverage Gaps

- Transport proves RunEvent/tunnel separation but not authoritative client/generation context or stale-owner drops.
- Queue tests prove generation-fenced capacity and release-once, but not adapter/target identity or health sequence transitions.
- Snapshot tests project config/disconnect health only; runtime unhealthy/recovery overlay is absent.
- CAPABILITIES tests prove probing but not fail-closed health classification, shared observation sequence, stale-response rejection, or Edge overlay recovery.

### Symbol References

- Preserve existing one-argument `Service.HandleRunLifecycleEvent` and `Service.RouteProviderTunnelFrame` for direct callers/tests. Add reception-aware siblings for bootstrap wiring; no symbol is renamed or removed.
- Transport setter callback types change internally; call sites are `apps/edge/internal/bootstrap/runtime.go` and `apps/edge/internal/transport/server_test.go`.

### Split Judgment

- Stable predecessor contract: `05+04_failure_wire` supplies optional typed failure fields. Its `complete.log` is currently missing in the active sibling, so implementation must wait for PASS.
- This packet supplies immutable reception/binding validation and overlay projection. `07+06_retry_candidate_policy` depends on its candidate eligibility; `08+07_stall_recovery` depends transitively on both.
- The packet is cohesive because the same queue lock must order evidence validation, overlay transition, lease release, and the next admission pump; splitting that invariant would create an unsafe intermediate state.

### Scope Rationale

Do not create retry intents, pick alternate providers, consume StreamGate budget, add metrics, or mutate Node/config health. Retry selection belongs to 07, OpenAI recovery to 08, and `ops-evidence` is outside this Epic's allowed task ids.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true, scores `(2,2,2,1,2)`, grade G09, route `grade-boundary` -> `PLAN-cloud-G09.md`.
- Review closure true, scores `(2,2,2,1,2)`, grade G09, route `official-review` -> `CODE_REVIEW-cloud-G09.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.

## Implementation Checklist

- [ ] REFACTOR-1 propagates authoritative receiving node/generation for RunEvent and tunnel callbacks and binds it atomically to the current registry owner without trusting wire identity.
- [ ] REFACTOR-2 validates immutable lease identity, applies sequence-fenced runtime unhealthy/recovery transitions, gates admission/snapshots, annotates every confirmed bound stall for Edge-local recovery (including unknown health), and releases valid terminal leases exactly once.
- [ ] REFACTOR-3 turns the existing exact-target CAPABILITIES probe into fail-closed sequenced evidence and applies only an unambiguous current-generation higher-sequence available response to overlay recovery.
- [ ] Add focused stale-owner, missing/ambiguous identity, mismatch, sequence, production-probe recovery, normalized/tunnel, and release-race tests; synchronize contracts/specs without mutating config health semantics.
- [ ] Run focused, package, race, vet, provider-only/local-capacity/full-cycle, live preflight/scenario, and diff verification commands with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REFACTOR-1] Carry authoritative reception identity

**Problem:** `apps/edge/internal/transport/connection_handlers.go:15-54` captures the receiving client but invokes callbacks with only the wire message. A spoofed/stale event can therefore be processed without proving which registered connection delivered it.

**Solution:** Add an atomic registry lookup that returns a cloned current owner only when the supplied client still owns it. Change transport's internal lifecycle/tunnel callback contracts to include that authoritative node id and generation, drop callbacks from unregistered/stale clients, and wire the new service entry points from bootstrap. Keep observability fanout message-only.

Before (`apps/edge/internal/transport/connection_handlers.go:29`):

```go
if lifecycle != nil {
    lifecycle(e)
}
```

After:

```go
owner, ok := s.registry.CurrentOwnerForClient(client)
if ok && lifecycle != nil {
    lifecycle(owner.NodeID, owner.ConnectionGeneration, e)
}
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/node/registry.go`: add lock-safe current-owner-by-client lookup returning a clone.
- [ ] `apps/edge/internal/node/registry_test.go`: prove current owner success and stale/unregistered client rejection across reconnect.
- [ ] `apps/edge/internal/transport/server.go`: type reception-aware lifecycle/tunnel callbacks.
- [ ] `apps/edge/internal/transport/connection_handlers.go`: resolve current owner at receipt and fail closed for stale clients before correctness callbacks.
- [ ] `apps/edge/internal/transport/server_test.go`: assert authoritative node/generation and no callback from stale connection while observability separation remains intact.
- [ ] `apps/edge/internal/bootstrap/runtime.go`: wire reception-aware service methods.

**Test Strategy:** Extend transport/registry fixtures with two clients for one node generation; assert only the live receiver reaches lifecycle/tunnel callbacks and wire metadata cannot substitute another owner.

**Verification:** `go test -count=1 ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap -run 'CurrentOwner|Reception|Lifecycle|Tunnel'` must PASS (Go treats unmatched package patterns as no tests, but every named new test must run in its owning package).

### [REFACTOR-2] Apply a lease-bound runtime health overlay

**Problem:** `apps/edge/internal/service/model_queue_types.go:173-186` cannot verify adapter/target, `providerResourceState` at lines 463-477 has no runtime observation state, and `model_queue_admission.go:75-113` plus `model_queue_snapshot.go:47-71` consult only config/connectivity. `Service.HandleRunLifecycleEvent` at `service.go:106-115` releases before classifying typed evidence.

**Solution:** Extend the immutable lease with dispatch adapter/target and maintain a separate `(node_id, connection_generation, provider_id)` overlay state under the queue lock. For a terminal owned by the receiving generation, compare provider/adapter/target and strictly increasing observation sequence; only `unavailable` sets unhealthy and a later same-generation, higher-sequence `available` clears it. Unknown/request-stalled leaves provider-wide projection unchanged. Attach Edge-local `provider_id`, normalized `provider_health`, and `recovery_eligible=true` to every confirmed, current, identity-bound `response_stalled` terminal, including `unknown`; this marker authorizes only ingress evaluation and never same-provider fallback. Missing stable provider identity, unconfirmed fence, stale owner/sequence, or binding mismatch remains terminal-only. Then release through the existing idempotent lease transition and pump. Apply the same path before routing tunnel ERROR frames.

Before (`apps/edge/internal/service/service.go:110`):

```go
func (s *Service) HandleRunLifecycleEvent(event *iop.RunEvent) {
    if event == nil || s.queue == nil || !isTerminalRunEvent(event) { return }
    s.queue.releaseRun(event.GetRunId(), event.GetType())
}
```

After:

```go
func (s *Service) HandleReceivedRunLifecycle(nodeID string, generation uint64, event *iop.RunEvent) {
    s.queue.applyTerminalEvidenceAndRelease(nodeID, generation, event)
}
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/model_queue_types.go`: add immutable adapter/target binding and generation-scoped overlay sequence/health state separate from config.
- [ ] `apps/edge/internal/service/model_queue_admission.go`: mint full bindings and reject runtime-unhealthy candidates under the existing queue lock.
- [ ] `apps/edge/internal/service/model_queue_release.go`: atomically validate terminal evidence, transition overlay, annotate eligible failure, release once, and pump.
- [ ] `apps/edge/internal/service/model_queue_snapshot.go`: project effective runtime unhealthy/recovery without changing catalog config.
- [ ] `apps/edge/internal/service/service.go`: expose reception-aware normalized lifecycle handling while retaining the compatibility wrapper.
- [ ] `apps/edge/internal/service/provider_tunnel.go`: validate/annotate terminal ERROR before request routing; duplicate stream cleanup remains a no-op release.
- [ ] `apps/edge/internal/service/provider_health_overlay_test.go`: add the S04 table and normalized/tunnel release-race fixtures.
- [ ] `agent-contract/inner/execution-runtime.md`: document lease binding, Edge-local eligibility annotation, and release ordering.
- [ ] `agent-contract/inner/edge-node-runtime-wire.md`: document reception identity as out-of-band authority and stale evidence rejection.
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: distinguish immutable config health from runtime overlay.
- [ ] `agent-spec/runtime/edge-node-execution.md`: reflect reception fencing and terminal handoff.
- [ ] `agent-spec/runtime/provider-pool-config-refresh.md`: reflect effective admission/snapshot overlay behavior.

**Test Strategy:** Create table tests for missing provider id, wrong node/provider/adapter/target, stale generation, equal/lower sequence, unavailable transition, unknown/request-stalled overlay no-op, higher-sequence available recovery, and a new connection generation. Prove available, unavailable, and unknown confirmed bound stalls all receive the raw-free request-local handoff marker, while only unavailable/available mutate overlay state. Run normalized and tunnel terminal duplicates concurrently and assert one decrement, no negative count, and no newer lease release.

**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^(TestProviderHealthOverlay|TestReceived.*Failure|Test.*ReleaseOnce)'` must PASS.

### [REFACTOR-3] Feed recovery from the bounded status probe

**Problem:** `apps/node/internal/node/command_handler.go:49-102` directly calls `ProbeProvider`, maps errors to unavailable, and emits neither normalized `provider_health` nor the Session-owned observation sequence. `apps/edge/internal/service/node_command.go:53-132` returns CAPABILITIES results without binding the response to its sending connection or applying it to the runtime overlay. The prior plan could therefore recover only through tests.

**Solution:** Pass the transport Session into CAPABILITIES handling and reuse `ProbeHealth(caps.AdapterName, caps.InstanceKey, exactTarget, ResolveProbeFunc(adapter))`. Allocate `health_observation_seq` from that same Session and return only stable adapter/instance/target, normalized health/status, and sequence fields. On Edge, retain the resolved entry's node id and connection generation through the synchronous response, map adapter/target to exactly one current provider resource under the queue lock, and apply only `available` with a strictly greater sequence to clear an unavailable overlay. Unknown/unavailable status probes never clear it; stale generation, mismatched response identity, empty target, or zero/malformed/ambiguous provider mapping is a no-op. This does not auto-retry or mutate catalog/config health.

**Modified Files and Checklist:**

- [ ] `apps/node/internal/node/command_handler.go`: use `ProbeHealth`, Session sequence, and stable result keys for CAPABILITIES without returning raw probe detail as correctness evidence.
- [ ] `apps/node/internal/node/command_test.go`: cover exact available, timeout/error/unsupported/mismatch -> unknown, and monotonic CAPABILITIES evidence on one Session.
- [ ] `apps/edge/internal/service/node_command.go`: carry authoritative node/generation from request dispatch and offer successful CAPABILITIES evidence to the queue only after response validation.
- [ ] `apps/edge/internal/service/model_queue_release.go`: add the shared locked probe-evidence transition used by production and tests, with unambiguous provider binding and sequence fencing.
- [ ] `apps/edge/internal/service/provider_health_overlay_test.go`: drive the real CAPABILITIES response path for recover, stale generation, lower/equal sequence, ambiguous adapter/target, and unknown/unavailable no-clear cases.
- [ ] `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-node-runtime-wire.md`, `agent-spec/runtime/edge-node-execution.md`: document status-probe evidence ownership and fail-closed recovery conditions.

**Test Strategy:** Use a real Node command handler/session fixture and the Edge command service seam rather than calling an overlay test helper directly. Assert one unavailable terminal lowers admission/snapshot, a later current-generation exact available CAPABILITIES response with greater sequence restores it, and every stale/ambiguous/inconclusive response leaves state unchanged.

**Verification:** `go test -count=10 ./apps/node/internal/node ./apps/edge/internal/service -run '^(TestCapabilitiesHealthEvidence|TestProviderHealthOverlayCapabilitiesRecovery)'` must PASS every iteration and every named test must execute in its owning package.

## Dependencies and Execution Order

1. `05+04_failure_wire` must first produce `agent-task/m-node-provider-execution-liveness-recovery/05+04_failure_wire/complete.log`; it is active/missing at plan creation.
2. Implement REFACTOR-1, then REFACTOR-2, then REFACTOR-3. This subtask must PASS before `07+06_retry_candidate_policy` starts.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/node/registry.go` | REFACTOR-1 |
| `apps/edge/internal/node/registry_test.go` | REFACTOR-1 |
| `apps/edge/internal/transport/server.go` | REFACTOR-1 |
| `apps/edge/internal/transport/connection_handlers.go` | REFACTOR-1 |
| `apps/edge/internal/transport/server_test.go` | REFACTOR-1 |
| `apps/edge/internal/bootstrap/runtime.go` | REFACTOR-1 |
| `apps/edge/internal/service/model_queue_types.go` | REFACTOR-2 |
| `apps/edge/internal/service/model_queue_admission.go` | REFACTOR-2 |
| `apps/edge/internal/service/model_queue_release.go` | REFACTOR-2, REFACTOR-3 |
| `apps/edge/internal/service/model_queue_snapshot.go` | REFACTOR-2 |
| `apps/edge/internal/service/service.go` | REFACTOR-2 |
| `apps/edge/internal/service/provider_tunnel.go` | REFACTOR-2 |
| `apps/edge/internal/service/provider_health_overlay_test.go` | REFACTOR-2 |
| `apps/node/internal/node/command_handler.go` | REFACTOR-3 |
| `apps/node/internal/node/command_test.go` | REFACTOR-3 |
| `apps/edge/internal/service/node_command.go` | REFACTOR-3 |
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2, REFACTOR-3 |
| `agent-contract/inner/edge-node-runtime-wire.md` | REFACTOR-2, REFACTOR-3 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | REFACTOR-2 |
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2, REFACTOR-3 |
| `agent-spec/runtime/provider-pool-config-refresh.md` | REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/06+05_health_overlay/CODE_REVIEW-cloud-G09.md` | REFACTOR-1, REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS, including Node and Edge local profiles.
2. `go test -count=20 ./apps/edge/internal/service -run '^(TestProviderHealthOverlay|TestReceived.*Failure|Test.*ReleaseOnce)' && go test -count=10 ./apps/node/internal/node ./apps/edge/internal/service -run '^(TestCapabilitiesHealthEvidence|TestProviderHealthOverlayCapabilitiesRecovery)'` — PASS on every iteration and all named tests execute.
3. `go test -race -count=3 ./apps/node/internal/node ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/service` — PASS with no race report.
4. `go vet ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
5. `./scripts/e2e-smoke.sh` — PASS for provider-only Edge/Node dispatch, tunnel, queue, and reconnect fencing.
6. `./scripts/e2e-provider-capacity-smoke.sh` — PASS for a deterministic local capacity-1 provider pool and zeroed final counters.
7. `bash scripts/e2e-long-context-admission-smoke.sh --preflight && bash scripts/e2e-long-context-admission-smoke.sh --scenario normal-10` — PASS on the authorized synchronized dev runner; if reachability/identity remains blocked, record rc/output and do not claim completion.
8. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
