<!-- task=m-node-provider-execution-liveness-recovery/08+07_health_overlay plan=1 tag=REFACTOR milestone-task=failure-handoff -->

# Lease-Bound Provider Health Overlay and Recovery Probe

## For the Implementing Agent

Implement only this overlay/probe consumer after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G09.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

Authoritative reception identity must be compared with the immutable dispatch lease before typed stall evidence may affect provider-wide health. S04 also requires a separate generation/sequence-fenced runtime overlay, exactly-once terminal release, and a real later bounded exact-target probe that can recover an unavailable provider without mutating config health.

## Archive Evidence Snapshot

- Union preparation review archived the unimplemented plan=0 pair as `plan_cloud_G09_0.log` and `code_review_cloud_G09_0.log`; it had no verdict, implementation evidence, or verification output.
- Material ownership finding: reception/binding/fence confirmation must be an Edge handoff fact, not `recovery_eligible`. This replan uses `recovery_handoff=confirmed` only to prove the current binding and local fence; the OpenAI ingress recovery owner still decides commit, cancel, side effects, budget, candidates, and replay eligibility.
- Verification finding: this packet changes provider-pool eligibility and ProviderSnapshot projection, so the testing domain requires live long-context preflight plus the needed scenario as an auxiliary regression in addition to focused S04 evidence.

## Analysis

### Files Read

- `apps/edge/internal/bootstrap/runtime.go`
- `apps/edge/internal/service/service.go`, `apps/edge/internal/service/provider_tunnel.go`, `apps/edge/internal/service/provider_pool.go`, `apps/edge/internal/service/provider_resolution.go`
- `apps/edge/internal/service/model_queue_types.go`, `apps/edge/internal/service/model_queue_admission.go`, `apps/edge/internal/service/model_queue_release.go`, `apps/edge/internal/service/model_queue_snapshot.go`
- `apps/edge/internal/service/model_queue_test_support_test.go`, `apps/edge/internal/service/model_queue_admission_test.go`, `apps/edge/internal/service/queue_dispatch_test.go`, `apps/edge/internal/service/node_command.go`
- `apps/node/internal/node/command_handler.go`, `apps/node/internal/node/command_test.go`, `apps/node/internal/node/health_probe.go`, `apps/node/internal/node/health_probe_test.go`, `apps/node/internal/transport/session.go`
- `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-node-runtime-wire.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/edge-node-execution.md`, `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `agent-test/local/edge-smoke.md`, `agent-test/local/node-smoke.md`, `scripts/e2e-smoke.sh`, `scripts/e2e-provider-capacity-smoke.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; `milestone-task=failure-handoff`.
- S04/Evidence Map S04 requires missing identity, stale generation/sequence, or binding mismatch to be projection no-ops; only current bound `unavailable` lowers, and a later higher-sequence same-generation exact `available` probe recovers. Every valid terminal still releases once. These rules define both implementation items and verification tables.

### Verification Context

- `07+06_reception_fence` supplies authoritative callback identity. Queue leases currently omit adapter/target and provider resources have no runtime sequence/health overlay.
- Node CAPABILITIES currently bypasses fail-closed `ProbeHealth` and Session observation sequence; Edge returns the response without applying it. Focused/race tests and repository-native provider/queue smokes are the S04 semantic oracle. Because this packet changes provider-pool eligibility and ProviderSnapshot projection, the testing domain also requires live long-context preflight plus `normal-10` as an auxiliary regression. If an authorized live runner or credential is unavailable, record an `external-execution` verification blocker; it is not a product decision and does not weaken the focused oracle.
- Related SDDs keep retry/commit ownership in StreamGate Core and do not change this service-layer overlay boundary.

### Test Coverage Gaps

- Existing queue tests do not cover adapter/target binding or health sequence transitions; snapshots use config/connectivity only. CAPABILITIES tests do not prove normalized sequenced evidence or Edge recovery.

### Symbol References

- Add reception-aware service siblings while retaining one-argument compatibility methods. The predecessor callback contract is consumed in bootstrap; no public symbol is removed.

### Split Judgment

- Overlay transition/release and the production recovery probe share the queue lock, provider binding, observation sequence, contracts, and integration oracle, so they remain one child. Candidate selection waits for the completed effective eligibility projection.

### Scope Rationale

Do not create recovery intents, choose alternate providers, consume StreamGate budget, add metrics, or mutate Node/config health.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true, scores `(2,2,2,1,2)`, grade G09, route `grade-boundary` -> `PLAN-cloud-G09.md`.
- Review closure true, scores `(2,2,2,1,2)`, grade G09, route `official-review` -> `CODE_REVIEW-cloud-G09.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`.

## Implementation Checklist

- [ ] REFACTOR-1 validates reception plus immutable provider/adapter/target lease identity, sequence-fences runtime unhealthy/recovery transitions, gates admission/snapshots, annotates confirmed bound stalls with the non-approval token `recovery_handoff=confirmed`, and releases valid terminals exactly once.
- [ ] REFACTOR-2 turns exact-target CAPABILITIES into fail-closed Session-sequenced health evidence and applies only unambiguous current-generation higher-sequence `available` to overlay recovery.
- [ ] Add missing/ambiguous identity, stale/mismatch/sequence, normalized/tunnel release-race, and production-probe recovery fixtures; synchronize contracts/specs without mutating config health.
- [ ] Run focused, package, race, vet, provider-only/local-capacity full-cycles, required live long-context preflight/`normal-10` auxiliary regression, and diff verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REFACTOR-1] Apply lease-bound runtime health and terminal handoff

**Problem:** leases cannot verify adapter/target, provider state lacks runtime observation health, admission/snapshot consult config/connectivity only, and terminal release happens before typed evidence classification.

**Solution:** Extend immutable leases with adapter/target and keep `(node_id,generation,provider_id)` overlay state under the queue lock. Validate receiving generation plus full binding and increasing sequence. Only `unavailable` lowers; request-stalled/available and unknown do not. Annotate every confirmed current bound stall with Edge-local provider/health and `recovery_handoff=confirmed`, which proves only reception/binding/local-fence authority and never approves replay. Ingress remains the sole owner of full recovery eligibility. Then release/pump through the idempotent lease transition. Apply the same ordering to tunnel ERROR.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/bootstrap/runtime.go`: consume predecessor node/generation callbacks through reception-aware service methods.
- [ ] `apps/edge/internal/service/model_queue_types.go`: add immutable binding and separate overlay state.
- [ ] `apps/edge/internal/service/model_queue_admission.go`: mint full bindings and reject runtime-unhealthy candidates.
- [ ] `apps/edge/internal/service/model_queue_release.go`: validate, transition, annotate, release once, and pump atomically.
- [ ] `apps/edge/internal/service/model_queue_snapshot.go`: project effective runtime health without changing config.
- [ ] `apps/edge/internal/service/service.go`: expose reception-aware normalized lifecycle handling with compatibility wrapper.
- [ ] `apps/edge/internal/service/provider_tunnel.go`: validate/annotate tunnel terminal before routing and keep duplicate cleanup idempotent.
- [ ] `apps/edge/internal/service/provider_health_overlay_test.go`: add the S04 table and normalized/tunnel release races.
- [ ] `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-node-runtime-wire.md`, `agent-contract/inner/edge-config-runtime-refresh.md`: document binding, the non-approval handoff token, ingress-owned eligibility, release ordering, reception authority, and config/overlay separation.
- [ ] `agent-spec/runtime/edge-node-execution.md`, `agent-spec/runtime/provider-pool-config-refresh.md`: reflect terminal handoff and effective admission/snapshot behavior.

**Test Strategy:** Cover missing provider id, wrong node/provider/adapter/target, stale generation, equal/lower sequence, unavailable, unknown/request-stalled no-op, new generation, and duplicate normalized/tunnel terminals. Assert one decrement and no newer lease release.

**Verification:** overlay/release fixtures must PASS repeatedly.

### [REFACTOR-2] Feed recovery from the bounded status probe

**Problem:** Node CAPABILITIES calls the raw prober without normalized health/Session sequence, and Edge does not bind/apply the response.

**Solution:** Reuse `ProbeHealth` for an exact target, allocate `health_observation_seq` from the same Session, and return only stable identity/normalized status. Edge retains sending node/generation and clears unavailable only for an unambiguous current mapping with strictly higher `available` sequence. Empty/malformed/ambiguous/stale/unknown/unavailable results are no-ops.

**Modified Files and Checklist:**

- [ ] `apps/node/internal/node/command_handler.go`: use `ProbeHealth`, Session sequence, and stable result keys.
- [ ] `apps/node/internal/node/command_test.go`: cover exact available, fail-closed unknown rows, and monotonic sequence.
- [ ] `apps/edge/internal/service/node_command.go`: retain authoritative dispatch identity and offer validated evidence to the queue.
- [ ] `apps/edge/internal/service/model_queue_release.go`: share the locked probe-evidence transition.
- [ ] `apps/edge/internal/service/provider_health_overlay_test.go`: drive real CAPABILITIES recovery and all rejection rows.
- [ ] `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-node-runtime-wire.md`, `agent-spec/runtime/edge-node-execution.md`: document probe evidence ownership and recovery fences.

**Test Strategy:** Lower through a real unavailable terminal, recover only with a later exact current available CAPABILITIES response, and prove every stale/ambiguous/inconclusive response is a no-op.

**Verification:** Node/Edge capability recovery fixtures must PASS repeatedly.

## Dependencies and Execution Order

1. `07+06_reception_fence` must produce `agent-task/m-node-provider-execution-liveness-recovery/07+06_reception_fence/complete.log`.
2. Implement REFACTOR-1 before REFACTOR-2. This child must PASS before `09+08_retry_candidate_policy` starts.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/bootstrap/runtime.go` | REFACTOR-1 |
| `apps/edge/internal/service/model_queue_types.go` | REFACTOR-1 |
| `apps/edge/internal/service/model_queue_admission.go` | REFACTOR-1 |
| `apps/edge/internal/service/model_queue_release.go` | REFACTOR-1, REFACTOR-2 |
| `apps/edge/internal/service/model_queue_snapshot.go` | REFACTOR-1 |
| `apps/edge/internal/service/service.go` | REFACTOR-1 |
| `apps/edge/internal/service/provider_tunnel.go` | REFACTOR-1 |
| `apps/edge/internal/service/provider_health_overlay_test.go` | REFACTOR-1, REFACTOR-2 |
| `apps/node/internal/node/command_handler.go` | REFACTOR-2 |
| `apps/node/internal/node/command_test.go` | REFACTOR-2 |
| `apps/edge/internal/service/node_command.go` | REFACTOR-2 |
| `agent-contract/inner/execution-runtime.md` | REFACTOR-1, REFACTOR-2 |
| `agent-contract/inner/edge-node-runtime-wire.md` | REFACTOR-1, REFACTOR-2 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | REFACTOR-1 |
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-1, REFACTOR-2 |
| `agent-spec/runtime/provider-pool-config-refresh.md` | REFACTOR-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/CODE_REVIEW-cloud-G09.md` | REFACTOR-1, REFACTOR-2 |

## Final Verification

Fresh Go output is required.

1. `go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS.
2. `go test -count=20 ./apps/edge/internal/service -run '^(TestProviderHealthOverlay|TestReceived.*Failure|Test.*ReleaseOnce)' && go test -count=10 ./apps/node/internal/node ./apps/edge/internal/service -run '^(TestCapabilitiesHealthEvidence|TestProviderHealthOverlayCapabilitiesRecovery)'` — PASS and all named tests execute.
3. `go test -race -count=3 ./apps/node/internal/node ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/service` — PASS with no race report.
4. `go vet ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
5. `./scripts/e2e-smoke.sh` — PASS for the repository package smoke.
6. `./scripts/e2e-provider-capacity-smoke.sh` — PASS with zeroed final counters.
7. `./scripts/e2e-long-context-admission-smoke.sh --preflight` — PASS on the authorized live dev provider pool; otherwise capture the exact external-execution blocker.
8. `./scripts/e2e-long-context-admission-smoke.sh --scenario normal-10` — PASS as an auxiliary live admission/snapshot regression; it is not the S04 semantic oracle.
9. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
