<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=3 tag=REVIEW_REFACTOR milestone-task=ops-evidence -->

# Restore Trustworthy Provider-Health Observability Evidence

## For the Implementing Agent

Implement only the test-evidence repair selected below, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G04.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record the exact blocker, attempted commands/output, and resume condition only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The production observer is a bounded post-decision projection, but the current S06 test does not exercise the matrix or leakage inputs that its review evidence claims. This follow-up repairs only the deterministic test oracle so normalized and tunnel decisions, public snapshots, production recovery, and raw-free output are actually proven.

## Archive Evidence Snapshot

- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/plan_cloud_G08_2.log` and `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/code_review_cloud_G08_2.log`; verdict `FAIL` with Required R1.
- R1 evidence: the tunnel row sent stale evidence through the normalized handler, only the recovered state queried a private queue snapshot helper, and the sentinel assertion searched for a value that was never placed in any input.
- Reviewer verification: the focused observability command passed before unrelated concurrent work appeared. A later package/race/vet rerun was blocked by the independently owned active `09+08_retry_candidate_policy` test file `apps/edge/internal/service/provider_recovery_selection_test.go`; this follow-up must not modify that file. `git diff --check` passed.
- Carryover: keep all production observer, overlay, contract, and spec changes unchanged. The archived `08+07_health_overlay/complete.log` remains the satisfied predecessor evidence, and SDD S06 remains the acceptance source.

## Finding Resolution Map

| Finding | Mode | Exact resolution | Changed precondition |
|---------|------|------------------|----------------------|
| Required R1 | `direct-fix` | Repair `apps/edge/internal/service/provider_health_observability_test.go` so each normalized/tunnel row drives unavailable and stale terminals through its selected production handler, performs exact-target recovery through `Service.Capabilities`, checks `Service.ListNodeSnapshots` after every decision, and injects/searches actual identity/raw sentinels. | The focused command will exercise the previously absent tunnel-stale, public-snapshot, production-recovery, and leakage assertions instead of repeating the false-positive test. |

## Analysis

### Files Read

- `apps/edge/internal/service/provider_health_observability.go`
- `apps/edge/internal/service/provider_health_observability_test.go`
- `apps/edge/internal/service/provider_health_overlay_test.go`
- `apps/edge/internal/service/model_queue_release.go`
- `apps/edge/internal/service/model_queue_snapshot.go`
- `apps/edge/internal/service/provider_tunnel.go`
- `apps/edge/internal/service/node_command.go`
- `apps/edge/internal/service/service.go`
- `apps/edge/internal/bootstrap/runtime.go`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`
- `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/PLAN-local-G06.md`
- `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/plan_cloud_G08_2.log`
- `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/code_review_cloud_G08_2.log`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; `milestone-task=ops-evidence`.
- Acceptance Scenario S06 requires provider-unhealthy, stale rejection, later recovery, provider snapshot evidence, and no high-cardinality/raw exposure.
- Evidence Map S06 requires metric label guards, structured-log capture, and provider snapshot overlay recovery. R1 maps these requirements directly into the decision table, public snapshot assertions, and injected forbidden-input checks below.

### Verification Context

- No external handoff was supplied. Repository-native review used Go `1.26.2`, module `/config/workspace/iop-s1/go.mod`, the approved SDD, current source, and private Prometheus/zap fixtures.
- `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` passed, but source inspection proved its tunnel-stale and sentinel assertions were absent.
- Fresh package/race/vet reruns later failed only because active sibling `09+08_retry_candidate_policy` had an unused `context` import in its separately owned new test. That file is outside this packet. If the sibling remains incomplete during verification, record the exact compiler output and stop; do not absorb its work.
- No external host, credential, or user authorization is required. The prior pair already preserves the unchanged provider-capacity smoke and two-process diagnostic evidence.
- Confidence is high: one test file owns the missing oracle and production behavior is unchanged.

### Test Coverage Gaps

- The tunnel variant does not route stale evidence through `HandleReceivedProviderTunnelFrame`.
- Unavailable and stale states do not query the public `Service.ListNodeSnapshots` surface.
- Recovery calls the queue helper directly instead of the exact-target `Service.Capabilities` path.
- The leakage test never injects the searched sentinel and omits actual node/provider/run/session/adapter/target plus message/body values from its forbidden set.

### Symbol References

- None. No production symbol is renamed or removed.

### Split Judgment

- Keep one compact test-only packet: the normalized/tunnel state matrix, public snapshot oracle, production recovery path, and leakage guard form one S06 evidence unit.
- Subtask predecessor `08+07_health_overlay` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`.
- Active sibling `09+08_retry_candidate_policy` is not a runtime dependency and owns a different test file.

### Scope Rationale

- Do not modify production Go files, contracts, specs, roadmap files, generated protobufs, smoke scripts, or any sibling task artifact.
- In particular, do not modify `apps/edge/internal/service/provider_recovery_selection_test.go`; its transient compile state belongs to `09+08_retry_candidate_policy`.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closures are true; scores `(0,1,0,2,1)`, grade G04, base `local-fit`, escalated by `recovery-boundary` because `review_rework_count=1` and `evidence_integrity_failure=true` -> `PLAN-cloud-G04.md`.
- Review closures are true; scores `(0,1,0,2,1)`, grade G04, route `official-review` -> `CODE_REVIEW-cloud-G04.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `variant_product` (3). `risk_boundary=false`; `recovery_boundary=true`; no capability gap.

## Implementation Checklist

- [ ] REVIEW_REFACTOR-1 replaces the false-positive observability proof with a deterministic normalized/tunnel unavailable-stale-recovery matrix, public snapshot assertions after every decision, production `Capabilities` recovery, and actual identity/raw sentinel injection plus absence checks.
- [ ] Run the focused, package, race, vet, and diff commands in Final Verification with fresh output; if an independently owned active sibling still makes the package uncompilable, record its exact path and compiler output without modifying it.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_REFACTOR-1] Replace the false-positive S06 oracle

**Problem:** `apps/edge/internal/service/provider_health_observability_test.go:59-120` selects normalized or tunnel only for the unavailable terminal; line 79 always sends stale evidence through the normalized handler, line 82 calls the queue recovery helper directly, and line 85 queries a private snapshot only after recovery. At lines 192-220, the searched sentinel is created after the event and never appears in any input.

**Solution:** Make the table's terminal sender own both unavailable and stale deliveries, using `RunEvent` for normalized and `ProviderTunnelFrame` for tunnel. Back each row with the existing fake TCP command pattern so `Service.Capabilities` returns the higher-sequence exact-target available probe. After unavailable, stale, and recovery, call `Service.ListNodeSnapshots`, locate the exact node/provider, and assert expected status, health, and effective capacity. Put distinct forbidden values into the event/frame node, provider metadata, run, session, adapter, target, message/error, body/header, and credential-like fields; scan every gathered label and dedicated log message/key/value for every actual forbidden value.

Before (`apps/edge/internal/service/provider_health_observability_test.go:77`):

```go
stale := stallFailure("run-stale", overlayAdapter, overlayTarget, "available", "request_stalled", 3)
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, &iop.RunEvent{RunId: "run-stale", Type: "error", Failure: stale})
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 3)

if !svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusAvailable, 4, func() bool { return true }) {
    t.Fatal("current exact-target probe did not report recovery")
}
snapshot := svc.queue.getSnapshotForNode(entry.NodeID, record, true)[0]
```

After:

```go
sendTerminal(executionPath, staleEvidence)
assertPublicProviderSnapshot(t, svc.ListNodeSnapshots(), "unavailable", "unavailable", 0)

if _, err := svc.Capabilities(ctx, exactTargetRequest); err != nil {
    t.Fatalf("CAPABILITIES recovery: %v", err)
}
assertPublicProviderSnapshot(t, svc.ListNodeSnapshots(), "available", "available", 1)
assertForbiddenInputsAbsent(t, registry, logs, forbiddenInputs)
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/provider_health_observability_test.go`: repair the production-path decision table, public snapshot assertions, fake CAPABILITIES recovery, and actual forbidden-input guard.
- [ ] `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/CODE_REVIEW-cloud-G04.md`: record implementation decisions and fresh output.

**Test Strategy:** Update `TestProviderHealthObservability` so both variants prove applied unavailable, rejected stale, and recovered states through metrics, logs, and `ListNodeSnapshots`. Replace `TestProviderHealthObservabilityDoesNotExposeSentinels` with an assertion over actual forbidden input values; retain private registries, repeated-service construction, duplicate terminal, and lock-safety coverage.

**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` must pass with both variant rows and all dedicated tests selected.

## Dependencies and Execution Order

1. `08+07_health_overlay` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`.
2. Modify the test oracle, then run the fresh verification commands. Do not take ownership of concurrent sibling files.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/service/provider_health_observability_test.go` | REVIEW_REFACTOR-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/CODE_REVIEW-cloud-G04.md` | REVIEW_REFACTOR-1 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -v -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS and display normalized, tunnel, duplicate, default-collector, lock-safety, and forbidden-input rows.
2. `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS every iteration.
3. `go test -count=1 ./apps/edge/internal/service` — PASS for the complete service package after all independently owned active sibling files compile.
4. `go test -race -count=3 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS with no race report.
5. `go vet ./apps/edge/internal/service` — no diagnostics.
6. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
