<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->

# OpenAI Liveness-Recovery Operational Evidence

## For the Implementing Agent

Implement only this recovery-observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The predecessor makes the OpenAI host the sole liveness recovery owner and emits a private filter decision into the existing StreamGate observation timeline. The generic zap sink logs request/attempt/target correlation, so it cannot directly serve S06's bounded operational evidence. This slice adds a request-local projection that records commit state, eligibility, and the final recovery result while preserving Core decisions and suppressing high-cardinality fields only from the default liveness-specific log path.

## Archive Evidence Snapshot

- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/plan_cloud_G08_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/code_review_cloud_G08_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
- Replan finding: collector registration lifetime was not closed even though `observationSink()` constructs request-local wrappers, default-versus-custom sink suppression lacked an exact detection contract, and the verification list treated fake-provider smoke as sufficient without the testing rule's direct Edge/Node entrypoint diagnostic.
- Carryover: preserve the `10+09_stall_recovery` dependency, request-local deduplication state, S06 commit/eligibility/result axes, and custom-sink forwarding; add one process-global collector set injected into wrappers, explicit default-sink type detection, repeated-server/request coverage, and the repository-native two-process diagnostic.

## Analysis

### Files Read

- `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/PLAN-cloud-G08.md`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/stream_gate_ingress.go`
- `apps/edge/internal/openai/stream_gate_filters.go`
- `apps/edge/internal/openai/stream_gate_policy.go`
- `apps/edge/internal/openai/stream_gate_dispatcher.go`
- `apps/edge/internal/openai/filter_observation_sink.go`
- `apps/edge/internal/openai/provider_observation.go`
- `apps/edge/internal/openai/stream_gate_filters_test.go`
- `apps/edge/internal/openai/stream_gate_dispatcher_test.go`
- `apps/edge/internal/openai/filter_observation_sink_test.go`
- `apps/edge/internal/openai/provider_observability_test.go`
- `packages/go/streamgate/filter_observation.go`
- `packages/go/streamgate/runtime.go`
- `packages/go/streamgate/recovery_coordinator.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
- `scripts/dev/edge.sh`
- `scripts/dev/node.sh`
- `scripts/dev/edge-node-reconnect-diagnostic.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
- Acceptance Scenario S06 and Evidence Map S06 require the Edge recovery owner to expose commit state, eligibility, and recovery result for deterministic run/tunnel recovery while request/session/raw prompt/response and high-cardinality values are absent from metric labels and the dedicated structured log.
- Those rows require a per-request state machine over the predecessor's private filter observation and existing Core recovery lifecycle, not a second retry counter or a reconstruction from HTTP results.

### Verification Context

- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
- Planning baseline `go test -count=1 ./apps/edge/internal/openai -run 'OpenAIProviderErrorFoundation|OpenAIAttemptDispatcher|OpenAIAttemptController'` passed. Read-only preflight returned `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s1/go.mod`, and executable Edge/Node dev entrypoints, fake-vLLM smoke, and reconnect diagnostic. Edge/platform profiles supply OpenAI, service, StreamGate, race, and vet commands.
- `10+09_stall_recovery` is active and its `complete.log` is missing. Its plan promises exactly one private liveness filter for StreamGate-enabled requests, sanitized health/fence/handoff evidence, ExactReplay eligibility, provider avoidance, and deterministic Chat/Responses normalized/tunnel matrices.
- Current Core observations already expose filter attribution/evidence/commit state and recovery selected/dispatched/failure kinds. `recovery_plan_rejected` intentionally omits recovery info, so a request-local sink must remember only whether the immediately active cycle came from the private liveness filter. Confidence is medium-high because no Core or API behavior changes, but sequencing and deduplication span parallel filter evaluation and recovery lifecycle variants.
- No external host is required. Synthetic `ObservationSequencer` fixtures and the predecessor's scripted provider pool give deterministic semantic evidence; `IOP_VLLM_MODE=fake` is an auxiliary OpenAI process smoke, while `scripts/dev/edge-node-reconnect-diagnostic.sh` separately supplies the required real Edge/Node entrypoint cycle with temporary mock configs, ordered message relay, commands, and reconnect.

### Test Coverage Gaps

- The generic `streamgate_filter_observation` log includes correlation, attempt, model, and provider fields and has no Prometheus liveness-recovery projection.
- Existing Core observations emit several intermediate recovery kinds; no test selects exactly one final result per liveness cycle or associates plan rejection with the preceding private liveness decision.
- No test covers the Cartesian Chat/Responses × normalized/tunnel matrix for safe eligibility/result labels and structured-log leakage.
- No test proves repeated `Server` and request-wrapper construction reuses one process-global collector set, or that only the concrete default `*zapFilterObservationSink` is suppressed while custom and `NoopObservationSink` behavior stays explicit.

### Symbol References

- None. No symbol is renamed or removed. `Server.observationSink()` keeps its call sites and returns a fresh wrapper around the configured sink; custom observation sinks continue to receive the original immutable observations.

### Split Judgment

- Stable child output: `10+09_stall_recovery` owns typed stall mapping, the private liveness filter, eligibility, old-attempt teardown, provider handoff, and terminal behavior. Its PASS is required and currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` missing).
- This child owns only the request-local observation projection. The private filter id and sanitized descriptor/cause vocabulary are its input contract; Core observation kinds are the final-result oracle.
- Provider-health overlay transitions are independently observed by `12+08_health_overlay_observability`; no dependency on sibling 12 is required.

### Scope Rationale

Do not change filter decisions, arbitration, recovery budgets, commit boundary, dispatcher/provider selection, HTTP/SSE terminals, public error bodies, generic non-liveness observations, or Core observation types. Metric labels and dedicated logs must omit correlation/request/attempt/run/session/model/provider/node/lease/slot/credential identifiers and raw prompt/response/tool/provider content.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.

## Implementation Checklist

- [ ] REFACTOR-1 wraps each request's StreamGate observation sink with a failure-isolated liveness projector that reuses one process-global production collector set and emits one bounded eligibility observation and one final result per private liveness cycle without changing filter/recovery behavior.
- [ ] REFACTOR-2 proves Chat/Responses normalized/tunnel eligible, rejected, redispatched, terminal/failure, and repeated server/request construction outcomes through exact metric labels and safe structured logs, suppresses liveness high-cardinality fields only from the concrete default generic zap path, and synchronizes matching contracts/specs.
- [ ] Run every focused, package, race, vet, fake-provider auxiliary smoke, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REFACTOR-1] Project the request-local liveness lifecycle

**Problem:** `apps/edge/internal/openai/server.go:193-200` returns one shared generic sink, while `packages/go/streamgate/runtime.go:1055-1147` and `packages/go/streamgate/recovery_coordinator.go:395-417,527-664` emit multiple observations for one recovery. Counting those events directly would duplicate a cycle, and `recovery_plan_rejected` has no strategy field.

**Solution:** Register one package-level production collector set exactly once with the default Prometheus registerer. `Server` holds the reusable collector/logger dependencies, and every fresh `openAILivenessObservationSink` receives those handles; never call `promauto.New*` or `MustRegister` from `NewServer`, `observationSink()`, or the request wrapper. Tests construct isolated collectors with an explicit `prometheus.Registerer`. The wrapper keeps only a mutex-protected bounded phase (`idle|eligible_pending`) plus the current closed health/path values—never raw identifiers. On the predecessor-defined private liveness filter's evaluated observation, normalize `execution_path`, `provider_health`, `commit_state`, and its sanitized eligibility descriptor through closed maps and increment `iop_edge_liveness_recovery_eligibility_total{execution_path,provider_health,commit_state,eligibility}` exactly once. Ineligible decisions finish immediately with result `terminal`; eligible decisions wait for Core lifecycle. Record exactly one `iop_edge_liveness_recovery_results_total{execution_path,provider_health,recovery_result}` on `recovery_dispatched`, plan rejection, abort/rebuild/dispatch failure, non-liveness plan selection, or terminal fallback, then reset for a later bounded cycle. Intermediate lifecycle observations never increment results. Sink/metric/log failures remain observation-only.

Before (`apps/edge/internal/openai/server.go:193`):

```go
func (s *Server) observationSink() streamgate.ObservationSink {
    s.mu.RLock()
    defer s.mu.RUnlock()
    if s.obsSink == nil {
        return streamgate.NoopObservationSink{}
    }
    return s.obsSink
}
```

After:

```go
func (s *Server) observationSink() streamgate.ObservationSink {
    s.mu.RLock()
    downstream, logger := s.obsSink, s.logger
    s.mu.RUnlock()
    return newOpenAILivenessObservationSink(downstream, logger)
}
```

Use the predecessor's private liveness filter constant rather than duplicating its string. Closed eligibility values must cover `eligible`, `no_owner`, `post_commit`, `unconfirmed_fence`, `caller_cancelled`, `tool_side_effect`, `budget_exhausted`, `no_candidate`, `same_provider_forbidden`, and `other`; closed result values are `redispatched`, `plan_rejected`, `abort_failed`, `rebuild_failed`, `dispatch_failed`, `not_selected`, `terminal`, and `other`. Path is `normalized|provider_tunnel|unknown`; health is `available|unavailable|unknown`; commit state uses Core's closed values with `unknown` fallback.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/server.go`: create one liveness wrapper per request around the configured observation sink and logger.
- [ ] `apps/edge/internal/openai/liveness_recovery_observability.go`: implement request-local phase/deduplication, closed classification, default collectors, test injection, and safe log projection.

**Test Strategy:** Write tests in REFACTOR-2. Do not modify `packages/go/streamgate`; the existing immutable observations are sufficient.

**Verification:** `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` must pass every iteration.

### [REFACTOR-2] Prove bounded labels, safe logs, and variant outcomes

**Problem:** `apps/edge/internal/openai/filter_observation_sink.go:44-58` writes `correlation_id`, `attempt_id`, model, and provider on the generic path. Forwarding the predecessor's private liveness filter and ExactReplay lifecycle through that default sink would violate S06's liveness-log boundary even if the new metric labels were safe.

**Solution:** Snapshot the downstream under `Server.mu` and use an exact type assertion to distinguish the concrete default `*zapFilterObservationSink`; do not infer default ownership from logger/core equality. For that default only, the request-local wrapper consumes private-liveness and its pending ExactReplay lifecycle without forwarding those observations to the high-cardinality generic writer; it writes `edge_liveness_recovery_observation` with only `phase`, the closed labels above, and no identifiers. Non-liveness observations remain unchanged. A sink explicitly installed through `SetObservationSink` remains an application-owned observation backend and receives the original immutable observations while the safe operational projection still emits; `NoopObservationSink` stays no-op downstream but does not disable the safe projection. Unit tests create observations through `streamgate.ObservationSequencer`; end-to-end tests reuse the predecessor's scripted pool to cover Chat/Responses and normalized/tunnel decisions.

Before (`apps/edge/internal/openai/filter_observation_sink.go:44`):

```go
fields = append(fields,
    zap.String("correlation_id", obs.StableCorrelation()),
    zap.String("attempt_id", obs.AttemptID()),
    zap.String("actual_provider", obs.AttemptTarget().Provider()),
)
```

After (default liveness route):

```go
logger.Info("edge_liveness_recovery_observation",
    zap.String("phase", phase),
    zap.String("commit_state", commitState),
    zap.String("eligibility", eligibility),
    zap.String("recovery_result", result),
)
```

The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, `go.uber.org/zap`, and `iop/packages/go/streamgate`. Never log or label `StableCorrelation`, `AttemptID`, `AttemptTarget` identity fields other than normalized execution path, plan/shared ids, cause detail, evidence fingerprint, or raw terminal content.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/liveness_recovery_observability_test.go`: add synthetic sequencing/deduplication/default-sink suppression/custom-sink forwarding and Chat/Responses normalized/tunnel safety matrices.
- [ ] `agent-contract/inner/execution-runtime.md`: specify the Edge recovery-owner eligibility/result metric and dedicated log contract.
- [ ] `agent-contract/outer/openai-compatible-api.md`: record transparent pre-commit recovery operational evidence without changing the public response.
- [ ] `agent-spec/runtime/stream-evidence-gate.md`: record request-local liveness observation projection and generic-sink suppression boundary.
- [ ] `agent-spec/input/openai-compatible-surface.md`: record the metric/log safety vocabulary for Chat and Responses variants.

**Test Strategy:** `TestOpenAILivenessObservationSink` feeds private filter evaluated, plan-selected, rejection/failure, dispatched, terminal, duplicate, and unrelated continuation observations through `ObservationSequencer`; it asserts one eligibility/result per cycle, repeated default server/request construction without duplicate registration, default-only suppression, explicit Noop handling, and custom downstream preservation. `TestOpenAILivenessRecoveryObservability` drives Chat/Responses × normalized/tunnel with available/unavailable/unknown and rejection/result rows, seeds request/session/prompt/response/provider/credential sentinels, and asserts exact gathered label names plus absence from dedicated/default liveness logs.

**Verification:** both focused commands below must pass repeatedly, and log capture must observe no default `streamgate_filter_observation` entry for the consumed private-liveness/ExactReplay rows.

## Dependencies and Execution Order

1. `10+09_stall_recovery` must produce `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`; it is active and missing at plan creation.
2. Implement REFACTOR-1 before REFACTOR-2. If the predecessor's stable filter descriptor names differ, map those exact stable values in the closed classifier and record the mapping in implementation evidence; do not parse raw causes or invent new retry semantics.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/server.go` | REFACTOR-1 |
| `apps/edge/internal/openai/liveness_recovery_observability.go` | REFACTOR-1 |
| `apps/edge/internal/openai/liveness_recovery_observability_test.go` | REFACTOR-2 |
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
| `agent-contract/outer/openai-compatible-api.md` | REFACTOR-2 |
| `agent-spec/runtime/stream-evidence-gate.md` | REFACTOR-2 |
| `agent-spec/input/openai-compatible-surface.md` | REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `test -f agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` — predecessor PASS evidence exists before implementation.
2. `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` — PASS every iteration for eligible/rejected/final/deduplicated lifecycle rows.
3. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability'` — PASS every iteration and Chat/Responses normalized/tunnel subtests execute.
4. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS under the Edge/platform-common profiles.
5. `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'` — PASS with no race report.
6. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
7. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — auxiliary smoke PASS for credential-free Chat streaming/non-streaming Edge -> Node -> fake provider behavior.
8. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — PASS using separate `scripts/dev/edge.sh` and `scripts/dev/node.sh` processes; registration, the first two same-session messages, post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering are all verified. This is the required repository-native full-cycle diagnostic.
9. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
