<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=4 tag=REVIEW_REFACTOR milestone-task=ops-evidence -->

# Plan - Restore Confirmed-Only Liveness Admission and Exact Handler Oracles

## For the Implementing Agent

Filling every implementation-owned section of `CODE_REVIEW-cloud-G05.md` is mandatory. Execute this plan without changing its owner or write boundary, run every verification command, paste actual notes and stdout/stderr into the review stub, leave both active files in place, and report ready for review. Final verdict, archive renames, `complete.log`, and task-directory archival belong only to the code-review skill. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The production handler matrix now passes, but it does so by widening the typed-stall mapper to accept an unconfirmed handoff that the current contract excludes. The same matrix also leaves its HTTP status oracle unused and does not inspect the gathered Prometheus schema or exact safe-log values. This follow-up restores the confirmed-only runtime boundary and turns the matrix into exact positive and negative evidence without changing public behavior or documentation.

## Archive Evidence Snapshot

- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/plan_cloud_G04_3.log` and `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/code_review_cloud_G04_3.log`; official verdict `FAIL` with Required R1 and R2, no Suggested or Nit findings.
- R1: `newOpenAIProviderErrorEventFromFailure` admits `recovery_handoff=unconfirmed`, contradicting the confirmed-only OpenAI/StreamGate contract and the prior test-only scope. R2: the matrix never reads `wantCode`, does not gather exact metric family/label descriptors, and does not compare exact safe-log values.
- Affected files: `apps/edge/internal/openai/stream_gate_runtime.go` and `apps/edge/internal/openai/liveness_recovery_observability_test.go`. Contracts and living specs are source-of-truth inputs and must not change.
- Fresh review verification passed the two focused count loops, selected package tests, race tests, vet, fake-provider smoke, two-process reconnect diagnostic, predecessor evidence check, and `git diff --check`; passing commands did not close the source-level contract and oracle gaps.
- Roadmap carryover: preserve approved SDD S06 and `milestone-task=ops-evidence`; the packet contributes bounded Edge commit/eligibility/result evidence only for contract-valid private liveness cycles.

## Finding Resolution Map

| Finding | Mode | Exact Fix / Dependency Evidence | Changed or Satisfied Precondition |
|---------|------|---------------------------------|-----------------------------------|
| R1 | `direct-fix` | Restore confirmed-only admission in `apps/edge/internal/openai/stream_gate_runtime.go` and make the unconfirmed handler outcome a negative private-cycle assertion in `apps/edge/internal/openai/liveness_recovery_observability_test.go`. | The production mapper again matches the contract while the real handlers prove unconfirmed metadata cannot create eligibility/result metrics or redispatch. |
| R2 | `direct-fix` | Strengthen `TestOpenAILivenessRecoveryObservability` in `apps/edge/internal/openai/liveness_recovery_observability_test.go` to consume every response oracle and compare gathered metric and safe-log schemas/values exactly. | Repeated verification now fails on wrong HTTP status, metric family/label drift, missing/extra series, or unsafe/non-closed structured-log values. |

## Analysis

### Files Read

- `apps/edge/internal/openai/stream_gate_runtime.go`
- `apps/edge/internal/openai/stream_gate_filters.go`
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
- `apps/edge/internal/openai/liveness_recovery_observability.go`
- `apps/edge/internal/openai/liveness_recovery_observability_test.go`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/filter_observation_sink.go`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-test/local/testing-smoke.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; lock released; no `USER_REVIEW.md`.
- First-line milestone task id: `ops-evidence`.
- Target Acceptance Scenario: S06.
- Evidence Map driver: S06 requires Node/Edge metric label guards and structured-log capture across liveness/fence/health/commit/recovery axes without high-cardinality or raw content.
- Effect on this packet: contract-invalid unconfirmed handoffs remain terminal outside a private liveness cycle, while confirmed handler outcomes must prove exact bounded metric and safe-log evidence. The checklist therefore restores admission first and then verifies positive and negative handler rows with exact schemas.

### Verification Context

- No separate handoff was supplied. The archived current-pair verdict, current source, approved SDD, matching contract/specs, and repository-native test profiles supplied the context.
- The review reran `go test -count=20` for `TestOpenAILivenessObservationSink`, `go test -count=10` for `TestOpenAILivenessRecoveryObservability`, selected package tests, race tests, vet, fake-vLLM smoke, the reconnect diagnostic, predecessor check, and `git diff --check`; all passed.
- Preconditions: `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` exists; SDD S06 is approved; the current dependent subtask retains `ops-evidence` scope.
- Constraints: preserve Edge-confirmed typed-stall admission, StreamGate arbitration/budget/terminal behavior, public HTTP/SSE envelopes, process-global collectors, explicit-sink ownership, and the shared dirty worktree. Do not change contracts, specs, other production behavior, or unrelated files.
- Gap and confidence: line 106 visibly broadens admission; `wantCode` has no read; the matrix has no registry `Gather`/descriptor inspection. The fixes and deterministic oracles are local and high-confidence.
- External Verification Preflight: not applicable. Both smoke commands use repository-native local entrypoints and the fake-provider profile requires no external host or credential.

### Test Coverage Gaps

- Confirmed-only admission: existing stall recovery tests cover confirmed and generic failures, but the new explicit `recovery_handoff=unconfirmed` fixture currently passes only because production admission was widened. The matrix must become the regression proving zero private-cycle metrics and no redispatch for that value.
- HTTP outcome: `wantCode` is populated for all four outcome rows but unused; non-redispatch rows can return an unexpected status without failing.
- Metric schema: collector values and series counts are asserted through handles, but gathered family names and label-key sets are not inspected.
- Safe-log schema: keys and sentinel absence are checked, but exact closed values for each expected eligibility/final-result row are not compared.

### Symbol References

None. No symbol is renamed or removed.

### Split Judgment

One compact packet is required because the negative handler oracle is correct only after the mapper's confirmed-only condition is restored. The `13+10_recovery_observability` directory depends on predecessor index 10, satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`.

### Scope Rationale

Only the typed-stall admission condition and the liveness observability test may change. The contract/spec already describe the intended boundary, and liveness projection/source behavior outside the accidental admission widening is not reopened. No shared StreamGate package, service fixture, contract, spec, or smoke script change is allowed.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`.
- Build closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`; capability gap not observed.
- Build grade scores: scope coupling 1, state/concurrency 1, blast/irreversibility 1, evidence/diagnosis 1, verification complexity 1; grade G05.
- Build route: base `local-fit`, promoted by `recovery-boundary` because `review_rework_count=2` and `evidence_integrity_failure=true`; lane `cloud`; canonical filename `PLAN-cloud-G05.md`.
- Review closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`; capability gap not observed.
- Review grade scores: scope coupling 1, state/concurrency 1, blast/irreversibility 1, evidence/diagnosis 1, verification complexity 1; grade G05.
- Review route: `official-review`, lane `cloud`, adapter `codex`, model `gpt-5.6-sol`, reasoning effort `xhigh`; canonical filename `CODE_REVIEW-cloud-G05.md`.
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `variant_product` (count 3); `risk_boundary_matched=false`; `recovery_boundary_matched=true`.

## Dependencies and Execution Order

1. Predecessor subtask `10+09_stall_recovery` is complete at `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`.
2. Implement REVIEW_REFACTOR-1 before REVIEW_REFACTOR-2 so the matrix is built against the restored production contract.
3. Run the complete verification only after both focused tests pass.

## Implementation Checklist

- [ ] REVIEW_REFACTOR-1 restores confirmed-only typed-stall admission and proves an explicit unconfirmed handoff stays a single sanitized handler terminal with one submit, no redispatch, and no private eligibility/result metric cycle across Chat/Responses and normalized/provider-tunnel paths.
- [ ] REVIEW_REFACTOR-2 consumes every matrix HTTP status oracle and asserts exact gathered liveness metric family names, label-key sets, expected series/counts, exact six-field safe-log contexts, and absence of generic high-cardinality liveness rows for every contract-valid outcome.
- [ ] Run every focused, package, race, vet, fake-provider auxiliary smoke, two-process Edge/Node diagnostic, predecessor, formatting, and diff command in Final Verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_REFACTOR-1] Restore confirmed-only handoff admission

**Problem**

`apps/edge/internal/openai/stream_gate_runtime.go:104-108` accepts both confirmed and unconfirmed recovery handoff metadata:

```go
if metadata["failure_code"] != openAIStallFailureCode ||
    metadata[openAIStallAttemptFenceKey] != openAIStallHandoffCause ||
    (metadata[openAIStallHandoffKey] != "confirmed" && metadata[openAIStallHandoffKey] != "unconfirmed") ||
```

The mapper comment, outer OpenAI contract, and Stream Evidence Gate spec allow only the Edge-confirmed token. Accepting `unconfirmed` lets unvalidated metadata enter the private liveness evaluator and changed production behavior outside the prior test-only scope.

**Solution**

Restore the single confirmed comparison using the existing constant:

```go
if metadata["failure_code"] != openAIStallFailureCode ||
    metadata[openAIStallAttemptFenceKey] != openAIStallHandoffCause ||
    metadata[openAIStallHandoffKey] != openAIStallHandoffCause ||
```

Keep the handler matrix's explicit unconfirmed failure, but classify it as a negative admission row: HTTP 502, one submit, no recovery marker, zero eligibility/result metric families, and only the bounded `provider_error_ignored` safe observation row if emitted. Do not change projection or Core behavior to manufacture a private terminal cycle.

**Modified Files and Checklist**

- [ ] Modify `apps/edge/internal/openai/stream_gate_runtime.go` only at the confirmed-only condition.
- [ ] Modify `apps/edge/internal/openai/liveness_recovery_observability_test.go` to make the unconfirmed terminal row a negative private-cycle case.
- [ ] Assert the row returns its exact HTTP status, submits once, does not render recovered output, produces no liveness metric family, and emits no high-cardinality generic liveness log.

**Test Strategy**

Use the existing `terminal` row in `TestOpenAILivenessRecoveryObservability` across all four endpoint/path surfaces. Keep its fully populated failure metadata with only `recovery_handoff` changed to `unconfirmed`; this proves the mapper rejects that exact near-valid boundary through real handlers.

**Verification**

Run `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'`; every negative terminal row must pass without a private metric cycle.

### [REVIEW_REFACTOR-2] Enforce exact handler response and observability schemas

**Problem**

`apps/edge/internal/openai/liveness_recovery_observability_test.go:770` defines `wantCode`, but lines 867-876 use a name-specific check and never compare the field. Lines 882-892 read collector handles without inspecting gathered family names or label descriptors, and lines 895-923 check safe-log keys without exact expected values.

**Solution**

Compare `response.Code` with `outcome.wantCode` for every row before body-specific assertions. Add local test helpers that call `reg.Gather()`, select only `iop_edge_liveness_recovery_eligibility_total` and `iop_edge_liveness_recovery_results_total`, and compare exact sorted label-key sets plus the single expected label/value/count row. Positive rows must have both exact families; the unconfirmed negative row must have neither. Build exact expected context maps for the safe eligibility/intermediate/final rows and compare all six fields (`phase`, `execution_path`, `provider_health`, `commit_state`, `eligibility`, `recovery_result`) and their closed values, while continuing to reject identifiers, sentinel values, and constructor-default generic private rows.

**Modified Files and Checklist**

- [ ] Modify `apps/edge/internal/openai/liveness_recovery_observability_test.go` only.
- [ ] Consume `wantCode` for every outcome and retain recovered/non-recovered body plus submit-count assertions.
- [ ] Gather and compare exact metric family names, sorted label-key sets, label values, and counter values for positive rows; assert no liveness family for the negative unconfirmed row.
- [ ] Compare the exact ordered or explicitly normalized safe-log context maps for each row, with no extra/missing keys or non-closed values.
- [ ] Keep the explicit same-concrete-type sink regression and all existing synthetic lifecycle coverage unchanged.

**Test Strategy**

Extend only `TestOpenAILivenessRecoveryObservability` and local helpers in the same file. Use the isolated Prometheus registry and zap observer already created per subtest, so schema and log assertions remain deterministic and do not inspect the process-global registry.

**Verification**

Run `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink$'` and `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'`; all iterations must pass with the exact oracles active.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/stream_gate_runtime.go` | REVIEW_REFACTOR-1 |
| `apps/edge/internal/openai/liveness_recovery_observability_test.go` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/CODE_REVIEW-cloud-G05.md` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `test -f agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` — archived predecessor PASS evidence exists.
2. `gofmt -d apps/edge/internal/openai/stream_gate_runtime.go apps/edge/internal/openai/liveness_recovery_observability_test.go` — exits 0 with no output.
3. `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink$'` — the explicit same-type sink and synthetic lifecycle regressions pass repeatedly.
4. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability$'` — all endpoint/path/outcome rows pass with exact HTTP, metric, and log oracles.
5. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — selected Edge/platform-common packages pass.
6. `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'` — exits 0 with no race report.
7. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — exits 0 with no diagnostics.
8. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — credential-free Chat streaming/non-streaming Edge -> Node -> fake-provider smoke passes.
9. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — separate Edge/Node registration, two same-session messages, reconnect, post-reconnect message, payload equality, status commands, and exactly-once terminal ordering pass.
10. `git diff --check` — exits 0 with no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
