<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=5 tag=REVIEW_REFACTOR milestone-task=ops-evidence -->

# Plan - Anchor Exact Liveness Metric and Safe-Log Oracles

## For the Implementing Agent

Filling every implementation-owned section of `CODE_REVIEW-cloud-G04.md` is mandatory. Execute this plan without changing its owner or write boundary, run every verification command, paste actual notes and stdout/stderr into the review stub, leave both active files in place, and report ready for review. Final verdict, archive renames, `complete.log`, and task-directory archival belong only to the code-review skill. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The confirmed-only production boundary and handler behavior are now correct, but the matrix still proves only partial observability predicates. This follow-up makes the test independent from production metric-name constants and compares every safe replacement log against an exact lifecycle sequence, without changing production code, contracts, or specs.

## Archive Evidence Snapshot

- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/plan_cloud_G05_4.log` and `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/code_review_cloud_G05_4.log`; official verdict `FAIL` with Required R2, no Suggested or Nit findings.
- R1 is closed: `newOpenAIProviderErrorEventFromFailure` again admits only `recovery_handoff=confirmed`, and the explicit unconfirmed handler row stays a single 502 terminal with one submit and zero liveness metric families.
- R2 remains: the gathered-family helper follows production name constants and ignores other families, while the safe-log helper checks only common fields, one eligibility anchor, and the final row instead of the exact row sequence.
- Fresh review verification passed focused count loops, selected package tests, race tests, vet, fake-provider smoke, the two-process reconnect diagnostic, predecessor evidence, formatting, and `git diff --check`; command success does not close the source-level oracle gap.
- Roadmap carryover: preserve approved SDD S06 and `milestone-task=ops-evidence`; this packet contributes only exact bounded Edge eligibility/result metric and raw-free safe-log evidence.

## Finding Resolution Map

| Finding | Mode | Exact Fix / Dependency Evidence | Changed or Satisfied Precondition |
|---------|------|---------------------------------|-----------------------------------|
| R2 | `direct-fix` | In `apps/edge/internal/openai/liveness_recovery_observability_test.go`, anchor the two documented metric family names as literal test expectations, reject extra/missing gathered families, and compare the complete ordered six-field safe-log sequence for every handler outcome. | The matrix will fail on a production/test family rename, any extra or missing liveness family/series, and any missing, duplicate, reordered, or incorrectly valued safe-log row. |

## Analysis

### Files Read

- `apps/edge/internal/openai/liveness_recovery_observability_test.go`
- `apps/edge/internal/openai/liveness_recovery_observability.go`
- `apps/edge/internal/openai/stream_gate_runtime.go`
- `apps/edge/internal/openai/stream_gate_filters.go`
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
- `apps/edge/internal/openai/provider_test_support_test.go`
- `apps/edge/internal/openai/responses_stream_gate.go`
- `apps/edge/internal/openai/server.go`
- `packages/go/streamgate/recovery_coordinator.go`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-test/local/testing-smoke.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; lock released; no `USER_REVIEW.md`.
- First-line milestone task id: `ops-evidence`.
- Target Acceptance Scenario: S06.
- Evidence Map driver: S06 requires Node/Edge metric label guards and structured-log capture across liveness/fence/health/commit/recovery axes without high-cardinality or raw content.
- Effect on this packet: the implementation checklist anchors the two exact Edge metric contracts and the exact bounded safe-log lifecycle for positive and negative handler rows; final verification reruns both focused matrices plus integrated Edge and local execution paths.

### Verification Context

- No separate handoff was supplied. The archived verdict, current test/source, approved SDD, matching contract/specs, and repository-native local profiles supplied the context.
- Fresh reviewer commands passed: predecessor existence, `gofmt -d`, `go test -count=20` for `TestOpenAILivenessObservationSink`, `go test -count=10` for `TestOpenAILivenessRecoveryObservability`, selected package tests, race tests, vet, fake-vLLM smoke, the reconnect diagnostic, and `git diff --check`.
- Preconditions: `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` exists; SDD S06 is approved; the dependent subtask retains `ops-evidence` scope.
- Constraints: test-only change; preserve confirmed-only admission, runtime projection behavior, public HTTP/SSE envelopes, production metric/log code, contracts, specs, shared StreamGate code, and unrelated dirty-worktree files.
- Gap and confidence: `assertGatheredLivenessMetrics` at lines 902-957 follows production family-name constants, and `assertSafeLogSchemaAndValues` at lines 979-1053 does not compare each row to an exact expected map. The direct test fix is local and high-confidence.
- External Verification Preflight: not applicable. Both runtime commands use repository-native local entrypoints; the vLLM profile is fake and credential-free.

### Test Coverage Gaps

- Metric-family identity: current gathering proves current constant-driven families and labels but cannot detect a simultaneous contract-breaking rename or an unexpected extra family.
- Safe-log lifecycle: current assertions allow intermediate phase/eligibility/result drift and do not reject extra, missing, duplicated, or reordered rows outside the eligibility and final anchors.
- Existing handler matrix already covers Chat/Responses, normalized/provider-tunnel, redispatch, plan rejection, unconfirmed terminal, and dispatch failure; no new production fixture is needed.

### Symbol References

None. No production symbol is renamed or removed.

### Split Judgment

One compact test-only packet is required because exact family and log assertions share the same handler matrix and isolated registry/logger fixtures. The `13+10_recovery_observability` directory depends on predecessor index 10, satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`.

### Scope Rationale

Only `apps/edge/internal/openai/liveness_recovery_observability_test.go` may change, plus the implementation-owned active review evidence file. Production source, shared StreamGate code, contracts, specs, smoke scripts, and unrelated worktree files are already behaviorally correct and remain excluded.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`.
- Build closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`; capability gap not observed.
- Build grade scores: scope coupling 1, state/concurrency 1, blast/irreversibility 0, evidence/diagnosis 1, verification complexity 1; grade G04.
- Build route: base `local-fit`, promoted by `recovery-boundary` because `review_rework_count=3` and `evidence_integrity_failure=true`; lane `cloud`; canonical filename `PLAN-cloud-G04.md`.
- Review closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`; capability gap not observed.
- Review grade scores: scope coupling 1, state/concurrency 1, blast/irreversibility 0, evidence/diagnosis 1, verification complexity 1; grade G04.
- Review route: `official-review`, lane `cloud`, adapter `codex`, model `gpt-5.6-sol`, reasoning effort `xhigh`; canonical filename `CODE_REVIEW-cloud-G04.md`.
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `variant_product` (count 3); `risk_boundary_matched=false`; `recovery_boundary_matched=true`.

## Dependencies and Execution Order

1. Predecessor subtask `10+09_stall_recovery` is complete at `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`.
2. Implement REVIEW_REFACTOR-1 before REVIEW_REFACTOR-2 so the safe-log rows are compared alongside a contract-anchored family set.
3. Run the complete verification only after both exact oracles pass repeatedly.

## Implementation Checklist

- [ ] REVIEW_REFACTOR-1 anchors the two documented liveness metric family names as literal test expectations and rejects every missing, renamed, or extra gathered family, label key, label value, series, and count across positive and negative handler rows.
- [ ] REVIEW_REFACTOR-2 compares the exact ordered six-field safe-log context sequence for each outcome, rejecting extra, missing, duplicated, reordered, or incorrectly valued intermediate/final rows while retaining unsafe-key/sentinel and generic-log suppression checks.
- [ ] Run every focused, package, race, vet, fake-provider auxiliary smoke, two-process Edge/Node diagnostic, predecessor, formatting, and diff command in Final Verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_REFACTOR-1] Anchor the complete gathered metric contract

**Problem**

`apps/edge/internal/openai/liveness_recovery_observability_test.go:909-917` selects gathered families with `livenessMetricEligibilityName` and `livenessMetricResultsName`, the same production constants used to register them. A simultaneous rename therefore changes both production and the test oracle, while unrecognized gathered families are ignored.

**Solution**

Define test-owned literal expectations and compare the complete gathered family-name set before inspecting metrics:

```go
const (
    wantLivenessEligibilityFamily = "iop_edge_liveness_recovery_eligibility_total"
    wantLivenessResultsFamily     = "iop_edge_liveness_recovery_results_total"
)
```

For positive rows, require the sorted gathered family names to equal those two literals exactly, then retain exact metric count, counter value, label-key order, and label-value assertions. For the unconfirmed negative row, require no gathered liveness family. Do not derive expected names from production constants or accept unknown names.

**Modified Files and Checklist**

- [ ] Modify `apps/edge/internal/openai/liveness_recovery_observability_test.go` only.
- [ ] Add literal test-owned family names and compare the complete gathered family set.
- [ ] Retain exact one-series/one-count and static label-schema/value assertions for positive rows.
- [ ] Retain zero-family assertions for the unconfirmed negative row.

**Test Strategy**

Strengthen `TestOpenAILivenessRecoveryObservability`; do not add a second fixture. The isolated registry already contains only the liveness collectors, so its gathered family set is a deterministic contract oracle.

**Verification**

Run `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'`; every endpoint/path/outcome row must pass with literal family-name guards.

### [REVIEW_REFACTOR-2] Compare every safe-log row exactly

**Problem**

`apps/edge/internal/openai/liveness_recovery_observability_test.go:988-1051` validates field presence and common path/health/commit values, finds any matching eligibility row, and checks only the final row. It does not prove the exact row count/order or the `phase`, `eligibility`, and `recovery_result` values of each intermediate row.

**Solution**

Represent every expected log as one six-field map and compare it by index with `entry.ContextMap()`. Encode the current lifecycle sequences explicitly rather than deriving them from actual logs:

- ignored unconfirmed terminal: filter-start `idle`, then filter-evaluated `idle`, both with empty eligibility/result;
- plan rejection: filter-start `idle`, eligible evaluation `eligible_pending/eligible`, then final `idle/plan_rejected`;
- redispatch and dispatch failure: filter-start `idle`, eligible evaluation, selected/aborted/rebuilt `eligible_pending` intermediate rows, then final `idle/redispatched|dispatch_failed`.

Every row must carry the exact expected execution path, `provider_health=unknown`, `commit_state=transport_uncommitted`, eligibility, and result. Require exact slice length before row comparison. Keep unsafe-key/type/sentinel rejection and the constructor-default generic-log absence assertion.

**Modified Files and Checklist**

- [ ] Modify `apps/edge/internal/openai/liveness_recovery_observability_test.go` only.
- [ ] Add explicit expected row sequences for every outcome and execution path.
- [ ] Compare all six fields for every row by index after checking exact row count.
- [ ] Preserve unsafe-key, string-type, sentinel, negative-row, and generic-log suppression assertions.

**Test Strategy**

Strengthen the existing 16-row production handler matrix. The handler fixture and isolated zap observer already produce deterministic lifecycle order, so no synthetic source or production change is needed.

**Verification**

Run `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink$'` and `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'`; all exact row sequences must pass repeatedly.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/liveness_recovery_observability_test.go` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/CODE_REVIEW-cloud-G04.md` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `test -f agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` — archived predecessor PASS evidence exists.
2. `gofmt -d apps/edge/internal/openai/liveness_recovery_observability_test.go` — exits 0 with no output.
3. `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink$'` — explicit-sink ownership and synthetic lifecycle regressions pass repeatedly.
4. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'` — every endpoint/path/outcome row passes with literal metric-family and exact safe-log sequence oracles.
5. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — selected Edge/platform-common packages pass.
6. `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'` — exits 0 with no race report.
7. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — exits 0 with no diagnostics.
8. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — credential-free Chat streaming/non-streaming Edge -> Node -> fake-provider smoke passes.
9. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — separate Edge/Node registration, two same-session messages, reconnect, post-reconnect message, payload equality, status commands, and exactly-once terminal ordering pass.
10. `git diff --check` — exits 0 with no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
