<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->

# Edge Provider-Health Overlay Operational Evidence

## For the Implementing Agent

Implement only this provider-health observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The predecessor creates the generation/sequence-fenced runtime health overlay and exact-target probe recovery, but intentionally excludes metrics. Operators need bounded evidence that distinguishes an applied unhealthy transition, rejected stale evidence, and an applied recovery while the existing provider snapshot remains the identity-bearing source of truth.

## Analysis

### Files Read

- `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/PLAN-cloud-G09.md`
- `apps/edge/internal/bootstrap/runtime.go`
- `apps/edge/internal/service/service.go`
- `apps/edge/internal/service/model_queue_types.go`
- `apps/edge/internal/service/model_queue_release.go`
- `apps/edge/internal/service/model_queue_snapshot.go`
- `apps/edge/internal/service/provider_tunnel.go`
- `apps/edge/internal/service/model_queue_test_support_test.go`
- `apps/edge/internal/service/status_provider_test.go`
- `apps/edge/internal/openai/provider_observation.go`
- `apps/edge/internal/openai/provider_observability_test.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/edge-smoke.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
- Acceptance Scenario S06 and Evidence Map S06 require Edge metric/log evidence and provider snapshot projection to distinguish provider-unhealthy, stale evidence rejection, and a later recovered state without high-cardinality or raw request/response data.
- Those rows require one transition result object shared by metrics and logs, plus a deterministic normalized/tunnel table that queries the production snapshot after each accepted or rejected observation.

### Verification Context

- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
- Planning baseline `go test -count=1 ./apps/edge/internal/service -run 'ProviderSnapshot|ListNodeSnapshots|Reconnect'` passed. The Edge profile supplies package tests; the predecessor already requires provider-capacity full-cycle evidence.
- `08+07_health_overlay` is active and its `complete.log` is missing. Its plan promises one queue-locked overlay transition result for normalized/tunnel terminal evidence and CAPABILITIES probe recovery; this child must consume that result rather than reimplement validation.
- No external host is required. Service fixtures, fake transport clients, and `scripts/e2e-provider-capacity-smoke.sh` are repository-native evidence. Confidence is medium-high because exact observer placement depends on the predecessor's final transition helper but its ownership and state matrix are closed.

### Test Coverage Gaps

- Current snapshots read config/connectivity only; the predecessor will add overlay assertions but explicitly excludes metrics.
- No existing test captures applied/rejected transition logs or gathers a bounded health-evidence metric.
- No existing test proves a stale observation increments only a rejection series while leaving the unavailable snapshot unchanged, or that a later probe recovery changes both transition evidence and the snapshot.

### Symbol References

- None. Do not rename or remove the predecessor's overlay symbols. Add one internal observer interface/field and a startup logger setter; update only bootstrap construction and same-package fixtures.

### Split Judgment

- Stable child output: `08+07_health_overlay` owns validation, sequence/generation fencing, atomic state transition, admission, and snapshot projection. Its PASS is required and is currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` missing).
- This child owns only the immutable post-decision observation projection. It stages the result under the queue lock but performs metric/log I/O after unlocking, so it cannot alter overlay correctness or queue progress.
- Node stall evidence and OpenAI recovery-owner evidence remain in siblings 11 and 13.

### Scope Rationale

Do not change wire fields, evidence validation, provider binding, observation sequence ordering, overlay state, candidate eligibility, probe scheduling, queue release, recovery policy, or config health. Provider/node/run/request/session/lease/adapter/target identifiers and raw payload/credential values are excluded from metric labels and the dedicated log; exact provider identity remains available only through the existing snapshot surface.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.

## Implementation Checklist

- [ ] REFACTOR-1 projects every predecessor health-evidence decision into bounded Edge counters and a safe structured log after releasing the queue lock, without changing validation or overlay state.
- [ ] REFACTOR-2 proves normalized/tunnel provider-unhealthy, stale rejection, and later probe recovery through metrics/logs plus the production provider snapshot, and proves request/session/raw prompt/response and all high-cardinality identifiers are absent; synchronize matching contracts/specs.
- [ ] Run every focused, package, race, vet, provider-capacity smoke, and diff command in Final Verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REFACTOR-1] Observe the authoritative overlay decision

**Problem:** `apps/edge/internal/service/model_queue_types.go:465-481` has only capacity/connectivity resource state today, while `apps/edge/internal/service/model_queue_snapshot.go:50-71` directly projects effective provider values. The predecessor will add the authoritative overlay transition under the queue lock but explicitly excludes metrics, so observing wire metadata independently would duplicate and potentially disagree with its stale/binding decision.

**Solution:** Consume the predecessor's immutable transition result at the exact helper that classifies `applied`, `rejected_stale`, `rejected_binding`, `rejected_ambiguous`, or `inconclusive`. Add `iop_edge_provider_health_evidence_total{source,evidence_health,decision}` and `iop_edge_provider_health_transitions_total{from_health,to_health}` with closed mappings: source `stall|probe|unknown`; health `available|unavailable|unknown`; transition values `available|unavailable|unknown`; no identity labels. Emit `edge_provider_health_observation` with only those enums and a `state_changed` boolean. Stage the result while holding `modelQueueManager.mu`, then call the observer only after unlock; metrics/log failures are best-effort and must not block release/pump.

Before (`apps/edge/internal/service/model_queue_snapshot.go:50`):

```go
snaps = append(snaps, &iop.ProviderSnapshot{
    Status: effectiveStatus(connected),
    Health: effectiveHealth(connected, prov.Health),
```

After predecessor plus this slice (observation remains outside snapshot construction):

```go
result := m.applyProviderHealthEvidenceLocked(evidence)
// unlock before any observer call
m.healthObserver.Observe(result)
```

The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`. Production uses the default registry; tests inject private collectors and a zap observer.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/model_queue_types.go`: attach the observer to the queue manager without widening provider resource identity or overlay state.
- [ ] `apps/edge/internal/service/model_queue_release.go`: capture the predecessor transition/rejection result and emit after the critical section for normalized terminal, tunnel terminal, and probe evidence paths.
- [ ] `apps/edge/internal/service/service.go`: initialize the default observer and expose a startup-only logger/test injection seam without changing `New` callers.
- [ ] `apps/edge/internal/bootstrap/runtime.go`: bind the Edge runtime logger to the service observer before transport handlers start.
- [ ] `apps/edge/internal/service/provider_health_observability.go`: define metric collectors, closed label normalization, safe log projection, and best-effort observer behavior.

**Test Strategy:** Write tests in REFACTOR-2. Do not create a second overlay state or validate evidence in the observer.

**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` must pass and execute every decision row.

### [REFACTOR-2] Prove stale rejection, unhealthy projection, and recovery

**Problem:** `apps/edge/internal/service/status_provider_test.go:635-690` proves snapshots read resource state, but there is no liveness overlay metric/log oracle. A metric-only test could pass while stale evidence mutates the snapshot or while recovery never becomes operator-visible.

**Solution:** Drive the predecessor's production normalized and tunnel reception handlers with current bound unavailable evidence, a duplicate/lower-sequence stale available observation, and a later higher-sequence exact-target probe available result. At each step assert the metric decision/transition delta, one safe structured event, and the public `ListNodeSnapshots` health/status. Use high-card/raw sentinels in node/provider/run/session/adapter/target and message/body fields and assert none are present in gathered labels or dedicated log fields/messages.

Before (`apps/edge/internal/service/model_queue_snapshot.go:201`):

```go
func effectiveHealth(connected bool, health string) string {
    if connected {
        return health
    }
```

After predecessor behavior, verified by this child:

```go
// current unavailable -> snapshot unavailable
// stale available -> rejection metric, snapshot still unavailable
// later current available probe -> recovery metric, snapshot available
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/provider_health_observability_test.go`: add normalized/tunnel applied-unhealthy, stale-rejection, recovered-snapshot, exact-once, lock-safety, label allowlist, and log leakage tables.
- [ ] `agent-contract/inner/execution-runtime.md`: specify Edge health evidence/transition metric and safe-log semantics.
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: document that runtime overlay observations are separate from immutable config health and carry no provider identity labels.
- [ ] `agent-spec/runtime/edge-node-execution.md`: record reception-to-overlay observability and stale/recovery behavior.
- [ ] `agent-spec/runtime/provider-pool-config-refresh.md`: record the effective snapshot projection and operational evidence boundary.

**Test Strategy:** Create `TestProviderHealthObservability` with normalized and provider-tunnel subtests. Each uses the predecessor's real binding/generation/sequence path, queries the actual snapshot, gathers private Prometheus collectors, and captures zap entries. Include a blocking observer fixture to prove it is invoked after `modelQueueManager.mu` is released, plus duplicate terminal/probe rows to prove exactly-once transitions.

**Verification:** the focused test, service race suite, and provider-capacity smoke below must pass with no zero-match command.

## Dependencies and Execution Order

1. `08+07_health_overlay` must produce `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`; it is active and missing at plan creation.
2. Implement REFACTOR-1 before REFACTOR-2. Do not instrument raw wire reception independently of the predecessor's final transition decision.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/service/model_queue_types.go` | REFACTOR-1 |
| `apps/edge/internal/service/model_queue_release.go` | REFACTOR-1 |
| `apps/edge/internal/service/service.go` | REFACTOR-1 |
| `apps/edge/internal/bootstrap/runtime.go` | REFACTOR-1 |
| `apps/edge/internal/service/provider_health_observability.go` | REFACTOR-1 |
| `apps/edge/internal/service/provider_health_observability_test.go` | REFACTOR-2 |
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | REFACTOR-2 |
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2 |
| `agent-spec/runtime/provider-pool-config-refresh.md` | REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `test -f agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` — predecessor PASS evidence exists before implementation.
2. `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS every iteration and normalized/tunnel applied, stale, and recovered rows execute.
3. `go test -count=1 ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — PASS under the Edge local profile.
4. `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/bootstrap -run 'ProviderHealthObservability|ProviderHealthOverlay|Snapshot'` — PASS with no race report.
5. `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — no diagnostics.
6. `./scripts/e2e-provider-capacity-smoke.sh` — PASS with the final provider counters drained and no overlay regression.
7. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
