<!-- task=m-node-provider-execution-liveness-recovery/08+07_health_overlay plan=2 tag=REVIEW_REFACTOR milestone-task=failure-handoff -->

# Exact Provider Probe Recovery Ordering

## For the Implementing Agent

Implement only the direct fix mapped below, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep the active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record the exact blocker, attempted commands/output, and resume condition only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The first health-overlay implementation correctly fences terminal evidence but resolves CAPABILITIES recovery only among unavailable overlay entries. That permits an ambiguous current provider catalog mapping to recover one provider and loses a newer available observation when it arrives before a delayed lower-sequence unavailable terminal. SDD S04 requires exact current mapping and monotonic same-generation observation ordering in both cases.

## Archive Evidence Snapshot

- The plan=1 pair is archived as `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/plan_cloud_G09_1.log` and `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/code_review_cloud_G09_1.log` with verdict `FAIL`, Required=1, Suggested=0, Nit=0.
- Required R1 affects `apps/edge/internal/service/model_queue_release.go` and `apps/edge/internal/service/provider_health_overlay_test.go`: probe recovery must resolve exactly one current catalog provider and retain a per-provider high-water mark even while effective health is available.
- Reviewer reproduction proved both failures: one unavailable overlay was recovered despite a second healthy catalog provider with the same adapter/target, and an available sequence 2 was discarded before a delayed unavailable sequence 1 made the provider unavailable.
- Fresh focused/package/vet/provider smokes passed. The exact race suite contradicted the recorded PASS by timing out once in `TestEdgeServerRegistrationFailureReasons`; its immediate targeted race rerun passed, so fresh whole-command evidence is required. The authorized live long-context provider and Edge status endpoints remain unreachable; do not repeat those unchanged external commands in this repository-fix packet.
- Roadmap carryover remains `milestone-task=failure-handoff`, SDD S04. Existing contracts/specs already state exact unambiguous higher-sequence recovery and require no semantic rewrite for R1.

## Finding Resolution Map

| Finding | Mode | Exact Fix Evidence | Changed/Satisfied Precondition |
|---------|------|--------------------|--------------------------------|
| R1 | direct-fix | Resolve adapter/target against the current Node provider catalog in `apps/edge/internal/service/model_queue_release.go`; retain the uniquely resolved provider's available observation sequence; add both regressions in `apps/edge/internal/service/provider_health_overlay_test.go`. | The catalog ambiguity and available-before-terminal paths change before verification, so this is not an unchanged-precondition rerun. |

## Analysis

### Files Read

- `apps/edge/internal/node/store.go`
- `apps/edge/internal/service/model_queue_types.go`
- `apps/edge/internal/service/model_queue_admission.go`
- `apps/edge/internal/service/model_queue_release.go`
- `apps/edge/internal/service/model_queue_snapshot.go`
- `apps/edge/internal/service/provider_resolution.go`
- `apps/edge/internal/service/node_command.go`
- `apps/edge/internal/service/provider_health_overlay_test.go`
- `apps/edge/internal/service/model_queue_admission_test.go`
- `apps/edge/internal/service/queue_dispatch_test.go`
- `apps/node/internal/node/command_handler.go`
- `apps/node/internal/node/command_test.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/edge-smoke.md`
- `scripts/e2e-smoke.sh`
- `scripts/e2e-provider-capacity-smoke.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; `milestone-task=failure-handoff`.
- Target: Acceptance Scenario S04 and its Evidence Map row. Missing/ambiguous identity and stale sequence are projection no-ops; only one exact current-generation provider mapping may consume a strictly newer available observation.
- These criteria produce the two mandatory regressions and require the focused repeated/race verification below. Existing contract/spec text already encodes the same invariant.

### Verification Context

- No separate `verification_context` handoff was supplied. The archived plan=1 review, reviewer reproduction, repository tests, local profiles, contracts, and SDD are the evidence sources.
- Preconditions: `07+06_reception_fence` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/07+06_reception_fence/complete.log`; the active code already has the lease-bound overlay and CAPABILITIES evidence path.
- Applied criteria: current catalog identity comes from `NodeStore.FindByID`, adapter normalization from `providerAdapterKey`, target membership from `providerCanServe`, and ordering from `providerRuntimeHealthOverlay.observationSeq` under the queue mutex.
- Constraints: config-owned provider health remains immutable; Node CAPABILITIES wire fields and ingress recovery ownership remain unchanged. Fresh Go output is required; cached results are not acceptable.
- External verification carryover: the prior preflight ran from `/config/workspace/iop-s1` at HEAD `170e8d88519260412f412d5f323b7052f4b9ee8e` with a dirty implementation worktree, valid `configs/edge.yaml`, Linux/arm64 assumptions, provider base `http://toki-labs.com:18083/v1`, and Edge status `http://127.0.0.1:18001/edges/edge-toki-labs-dev/status`. Both `/v1/models` and the status endpoint were unreachable, so `normal-10` produced 0/10 responses and no samples. Source sync, runtime identity, binaries, ports, and remote process state could not be proven beyond that output. The resume condition is an authorized live provider pool plus reachable matching Edge status runtime; unchanged external retries are excluded from this direct repository fix.
- Confidence: high for R1 because both failure modes were reproduced with the production transition function; medium for whole-suite race stability until the exact race command passes freshly.

### Test Coverage Gaps

- Current ambiguity coverage creates two unavailable overlays, not one unavailable and one healthy provider in the current catalog. Add the missing catalog-level regression.
- Current recovery coverage lowers before it recovers. Add available sequence 2 before delayed unavailable terminal sequence 1 and assert the provider remains effectively available at sequence 2.
- Existing tests already cover malformed, inconclusive, stale-generation, equal-sequence, exact recovery, duplicate terminal, admission, and snapshot behavior; retain them unchanged.

### Symbol References

None. No symbol is renamed or removed.

### Split Judgment

Keep one packet. Catalog uniqueness resolution and the available observation high-water mark are one atomic recovery invariant under the queue lock, and the two regression cases share the same transition function and deterministic oracle. Predecessor 07 is satisfied by the archived `complete.log` cited above.

### Scope Rationale

Exclude Node probe generation, wire schemas, ingress retry policy, metrics, config health, admission/snapshot implementations, and contract/spec edits. They already supply or consume the intended invariant; R1 is confined to Edge probe-evidence identity/ordering and its regression tests. Preserve the unresolved live-runner evidence for official review rather than changing unrelated scripts or endpoints.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `status=routed`; finalizer=`finalize-task-policy.sh`; mode=`pair`.
- Build closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`; closure basis is the reproduced R1 direct fix with deterministic local regressions; capability gap: none.
- Build scores: `scope_coupling=1`, `state_concurrency=2`, `blast_irreversibility=1`, `evidence_diagnosis=2`, `verification_complexity=1`; base=`local-fit`, route=`recovery-boundary`, lane=`cloud`, grade=`G07`, filename=`PLAN-cloud-G07.md`.
- Review closures are all true; capability gap: none. Review scores: `scope_coupling=1`, `state_concurrency=2`, `blast_irreversibility=1`, `evidence_diagnosis=2`, `verification_complexity=2`; route=`official-review`, lane=`cloud`, grade=`G08`, filename=`CODE_REVIEW-cloud-G08.md`, adapter=`codex`, model=`gpt-5.6-sol`, reasoning=`xhigh`.
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract` (3); `risk_boundary_matched=false`; `review_rework_count=1`; `evidence_integrity_failure=true`; `recovery_boundary_matched=true`.

## Implementation Checklist

- [ ] REVIEW_REFACTOR-1 resolves an available probe to exactly one current catalog provider, records its same-generation high-water mark even when already available, and prevents ambiguous or lower-sequence state changes.
- [ ] Add deterministic regressions for unavailable-plus-healthy catalog ambiguity and available-sequence-2-before-unavailable-sequence-1 ordering while retaining existing recovery cases.
- [ ] Run focused, package, race, vet, provider smoke, local-capacity smoke, and diff verification with fresh output; do not retry the unchanged blocked live endpoints.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_REFACTOR-1] Enforce exact catalog identity and monotonic probe ordering

**Problem:** `apps/edge/internal/service/model_queue_release.go:208-223` resolves only among unavailable overlay entries. It therefore treats one unavailable overlay as unambiguous even when another current catalog provider has the same adapter/target, and it discards available observations when no unavailable overlay exists.

**Before (`apps/edge/internal/service/model_queue_release.go:208`):**

```go
var matched *providerRuntimeHealthOverlay
for key, overlay := range m.runtimeHealth {
	if key.nodeID != nodeID || key.generation != generation || overlay == nil || !overlay.unavailable ||
		overlay.adapter != adapter || overlay.target != target {
		continue
	}
	if matched != nil {
		return false
	}
	matched = overlay
}
if matched == nil || sequence <= matched.observationSeq {
	return false
}
matched.observationSeq = sequence
matched.unavailable = false
```

**Solution:** Under `m.mu`, use the current `NodeStore` record to find providers whose normalized adapter key equals `adapter` and whose configured model list contains `target`. Fail closed unless exactly one non-empty provider id matches. Address `runtimeHealth` by `(nodeID, generation, providerID)`, reject `sequence <= observationSeq`, and create/update the overlay for a fresh exact available observation even when it does not change effective availability. Keep the return value tied to an actual unavailable-to-available recovery and pump only for that transition.

```go
providerID, ok := m.resolveCurrentProbeProviderLocked(nodeID, adapter, target)
if !ok {
	return false
}
key := providerRuntimeHealthKey{nodeID: nodeID, generation: generation, providerID: providerID}
overlay := m.runtimeHealth[key]
if overlay != nil && sequence <= overlay.observationSeq {
	return false
}
if overlay == nil {
	overlay = &providerRuntimeHealthOverlay{}
	m.runtimeHealth[key] = overlay
}
recovered := overlay.unavailable
overlay.adapter, overlay.target = adapter, target
overlay.observationSeq, overlay.unavailable = sequence, false
if recovered {
	m.pumpAllLocked()
}
return recovered
```

No new package import is required; reuse `providerAdapterKey` and `providerCanServe` from the same package.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/model_queue_release.go`: add fail-closed current-catalog uniqueness resolution and persist fresh exact available high-water observations.
- [ ] `apps/edge/internal/service/provider_health_overlay_test.go`: replace the insufficient overlay-only ambiguity oracle with catalog ambiguity coverage and add available-before-terminal ordering coverage.

**Test Strategy:** Add `TestProviderHealthOverlayCapabilitiesRecoveryRejectsCatalogAmbiguity` with one unavailable provider plus one healthy current catalog provider sharing adapter/target; assert no recovery and unchanged sequence. Add `TestProviderHealthOverlayCapabilitiesRecoveryPreservesAvailableHighWater` with one exact available observation at sequence 2 before a bound unavailable terminal at sequence 1; assert sequence 2 remains and effective health stays available. Keep existing exact recovery and rejection rows as regression coverage.

**Verification:** Run `go test -count=50 ./apps/edge/internal/service -run '^(TestProviderHealthOverlayCapabilitiesRecoveryRejectsCatalogAmbiguity|TestProviderHealthOverlayCapabilitiesRecoveryPreservesAvailableHighWater|TestProviderHealthOverlayCapabilitiesRecoveryRejectsInconclusiveEvidence|TestProviderHealthOverlayCapabilitiesRecovery)$'`; all named tests must execute and pass. Then run `go test -count=20 ./apps/edge/internal/service -run '^(TestProviderHealthOverlay|TestReceived.*Failure|Test.*ReleaseOnce)'`; all matching overlay/release tests must pass.

## Dependencies and Execution Order

1. Predecessor `07+06_reception_fence` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/07+06_reception_fence/complete.log`.
2. Complete REVIEW_REFACTOR-1 before rerunning the full verification suite. This child must not report PASS while R1 remains.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/service/model_queue_release.go` | REVIEW_REFACTOR-1 |
| `apps/edge/internal/service/provider_health_overlay_test.go` | REVIEW_REFACTOR-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/CODE_REVIEW-cloud-G08.md` | REVIEW_REFACTOR-1 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -count=50 ./apps/edge/internal/service -run '^(TestProviderHealthOverlayCapabilitiesRecoveryRejectsCatalogAmbiguity|TestProviderHealthOverlayCapabilitiesRecoveryPreservesAvailableHighWater|TestProviderHealthOverlayCapabilitiesRecoveryRejectsInconclusiveEvidence|TestProviderHealthOverlayCapabilitiesRecovery)$'` — PASS and every named test executes.
2. `go test -count=20 ./apps/edge/internal/service -run '^(TestProviderHealthOverlay|TestReceived.*Failure|Test.*ReleaseOnce)' && go test -count=10 ./apps/node/internal/node ./apps/edge/internal/service -run '^(TestCapabilitiesHealthEvidence|TestProviderHealthOverlayCapabilitiesRecovery)'` — PASS.
3. `go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS.
4. `go test -race -count=3 ./apps/node/internal/node ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/service` — PASS with no timeout or race report; record exact raw output because the prior whole-command evidence was contradicted.
5. `go vet ./packages/go/execution ./apps/node/... ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/bootstrap ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
6. `./scripts/e2e-smoke.sh` — PASS.
7. `(cd scripts && sed -e 's|^SCRIPT_DIR=.*|SCRIPT_DIR="$(pwd)"|' -e 's|TMP_DIR="$(mktemp -d /tmp/iop-provider-capacity-smoke.XXXXXX)"|TMP_DIR="$(mktemp -d "$REPO_ROOT/.tmp/iop-provider-capacity-smoke.XXXXXX")"|' e2e-provider-capacity-smoke.sh | bash)` — PASS with final provider available and counters zero; this is the already-proven `/tmp` noexec-safe form.
8. `git diff --check` — no whitespace errors.

Do not rerun the unchanged blocked long-context external endpoints in this packet. Preserve their archived evidence for official review.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
