<!-- task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery plan=2 tag=API milestone-task=bounded-retry -->

# OpenAI Typed Stall Recovery Handoff

## For the Implementing Agent

Implement only the items below after all predecessors PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The OpenAI host currently maps every normalized/tunnel terminal error to generic StreamGate provider errors, and the provider-error filter intentionally never constructs recovery. S05 requires a typed `response_stalled` handoff that remains terminal unless Edge confirmed the attempt fence and the request is uncommitted, uncanceled, side-effect-safe, and within the existing shared Core budget; eligible recovery must use a new run identity and prefer another provider.

## Archive Evidence Snapshot

- Prior pair: `plan_cloud_G08_1.log` and `code_review_cloud_G08_1.log` in this task directory. It was unimplemented and has no official verdict, implementation evidence, code change, or verification output.
- Material fresh-review finding: the internal liveness-owner design correctly preserves the latest output-filter contract, but the mandatory `normal-10` live scenario exercises long-context admission and does not prove typed stall mapping, recovery gating, provider avoidance, or bounded dispatch.
- Replan carryover: keep the raw-free mapper, private enabled-gate registration, Core budget/commit/cancel ownership, and S05 matrix; use focused/race plus fake-vLLM/provider-capacity repository-native full cycles as the completion oracle. Gate-disabled/unsupported surfaces remain the explicit no-owner terminal boundary.

## Analysis

### Files Read

- `apps/edge/internal/openai/stream_gate_ingress.go`, `apps/edge/internal/openai/stream_gate_policy.go`, `apps/edge/internal/openai/stream_gate_filters.go`, `apps/edge/internal/openai/stream_gate_runtime.go`
- `apps/edge/internal/openai/stream_gate_dispatcher.go`, `apps/edge/internal/openai/run_result.go`, `apps/edge/internal/openai/responses_stream_gate.go`, `apps/edge/internal/openai/chat_completion.go`
- `apps/edge/internal/openai/stream_gate_filters_test.go`, `apps/edge/internal/openai/stream_gate_dispatcher_test.go`, `apps/edge/internal/openai/stream_gate_pipeline_test.go`, `apps/edge/internal/openai/stream_gate_vertical_slice_test.go`
- `apps/edge/internal/openai/provider_test_support_test.go`, `apps/edge/internal/openai/server_test_support_test.go`
- `packages/go/streamgate/runtime.go`, `packages/go/streamgate/recovery_coordinator.go`, `packages/go/streamgate/recovery_plan.go`, `packages/go/streamgate/commit_boundary.go`, `packages/go/streamgate/event.go`, `packages/go/streamgate/filter_registry.go`, `packages/go/streamgate/terminal.go`
- `packages/go/config/config.go`, `agent-contract/inner/edge-config-runtime-refresh.md`, `agent-contract/inner/execution-runtime.md`, `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`, `agent-spec/input/openai-compatible-surface.md`, `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `agent-test/local/edge-smoke.md`, `agent-test/local/platform-common-smoke.md`, `scripts/e2e-smoke.sh`, `scripts/e2e-openai-vllm.sh`, `scripts/e2e-provider-capacity-smoke.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`, and this pair's first-line id is `milestone-task=bounded-retry`.
- Acceptance Scenario S05 and Evidence Map S05 require healthy request stall, unhealthy failover, unknown probe, same-provider-only, no owner, post-commit, unconfirmed fence, caller cancel/tool-side-effect, and shared-budget fixtures with a new run identity and bounded dispatch/terminal count. Unknown health may recover through an alternate candidate but never grants same-provider fallback.
- API-1 derives a raw-free typed event, API-2 implements exact-replay eligibility/fence teardown/provider handoff, and API-3 proves both Chat/Responses plus normalized/tunnel variants against those rows.

### Verification Context

- Handoff supplied starting HEAD `56e7d78af3cda4a8d6a85af091ad26bce935f8b6`; fresh baseline tests passed for execution, Node, Edge transport/service/OpenAI.
- This plan waits for `09+08_retry_candidate_policy/complete.log`; transitively, typed wire and Edge eligibility/overlay contracts are also PASS before implementation.
- Core already owns `transport_uncommitted`, caller-cancel propagation, abort-before-dispatch, new `AttemptBinding`, and one request-local total/per-strategy fault budget. No liveness counter or Core recovery coordinator is needed.
- Existing OpenAI event sources discard typed failures (`stream_gate_runtime.go:128-188`, `402-478`, `responses_stream_gate.go:110-135`) and `collectRunResult` flattens them (`run_result.go:87-95`). Provider-error filtering is explicitly foundation-only (`stream_gate_filters.go:29-32`, `198-237`).
- `openai.stream_evidence_gate.enabled` defaults false and `filters[]` is optional; configured filters alone therefore cannot own the S05 handoff. The host must add a private liveness registration only to enabled request runtimes, without adding a configured filter/capability requirement or changing generic `provider_error` behavior.
- No required verification leaves this checkout. Focused/race fixtures directly exercise S05, while fake-vLLM and provider-capacity scripts cover repository-native OpenAI and queue full cycles; the latest output-filter and Hot Path SDDs preserve Core retry/terminal ownership and do not add a second liveness loop. Confidence is medium-high due to multi-variant runtime and terminal ordering.

### Test Coverage Gaps

- Typed normalized and tunnel stalls are not distinguished from generic provider errors.
- Provider-error filter has only unmatched PASS coverage; no test proves that configured-filter absence still installs exactly one internal liveness owner or that gate-disabled requests remain no-owner terminal.
- Recovery dispatcher does not hand the failed provider to pool admission or distinguish a confirmed Node terminal from a still-running attempt during abort.
- Existing vertical slices prove shared-budget recovery generally, but not Chat/Responses stall variants, unknown/no-owner/post-commit/unconfirmed gates, or duplicate-terminal absence.

### Symbol References

- No symbol is renamed or removed. Constructor signatures for request-local dispatcher/filter wiring may gain internal state; update all call sites in `stream_gate_runtime.go`, `responses_stream_gate.go`, and their direct tests.

### Split Judgment

- Predecessor `09+08_retry_candidate_policy` is active with missing `complete.log`; implementation waits for it. That predecessor transitively requires `05+04_failure_wire_contract`, `06+05_failure_wire_mapping`, `07+06_reception_fence`, and `08+07_health_overlay` PASS.
- This final packet is indivisible at the OpenAI host boundary: the same raw-free eligibility token must drive filter intent, confirmed-terminal teardown, failed-provider handoff, and endpoint terminal rendering. Partial wiring could either duplicate dispatch or authorize an unfenced replay.

### Scope Rationale

Do not add a Core/Node/Edge retry loop, new counter, schema, metric, non-OpenAI recovery owner, or retry to legacy surfaces. Do not expose raw provider messages/metadata. StreamGate Core behavior remains unchanged; this packet consumes its existing budget/commit/cancel/side-effect contracts.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true, scores `(2,2,1,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
- Review closure true, scores `(2,2,1,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.

## Implementation Checklist

- [ ] API-1 preserves typed normalized/buffered/tunnel stalls as one raw-free StreamGate `response_stalled` provider error, retaining only sanitized fence/health/handoff tokens while generic failures keep existing terminal behavior.
- [ ] API-2 installs one internal liveness recovery filter for every StreamGate-enabled request independent of configured semantic filters/capabilities; only confirmed Edge-eligible, uncommitted, side-effect-safe stalls produce ExactReplay, close the fenced old transport, and hand the failed provider/fallback evidence to admission.
- [ ] API-3 adds Chat/Responses normalized/tunnel fixtures for available, unavailable, and unknown alternate recovery; available-only same-provider fallback; unavailable/unknown same-only terminal; no-owner, post-commit, unconfirmed, cancel/tool-side-effect, and shared-budget exhaustion; synchronize contracts/specs.
- [ ] Run focused, package, race, vet, provider-only/OpenAI/local-capacity full-cycles, and diff verification with fresh output; assert new identities plus exactly one terminal/dispatch per allowed cycle.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [API-1] Convert typed execution stalls into raw-free StreamGate events

**Problem:** `apps/edge/internal/openai/stream_gate_runtime.go:182-184` and `473-478` emit generic `run_failed`/`provider_tunnel_error`, while `apps/edge/internal/openai/run_result.go:87-95` converts a terminal event into an untyped formatted error. Buffered Chat/Responses therefore cannot preserve the same failure semantics as live/tunnel paths.

**Solution:** Add an internal terminal error that defensively retains the protobuf failure while its `Error()` exposes only a stable code. Centralize conversion of typed execution failure to `ExternalDescriptor(code=response_stalled)` and bounded allowlisted fence, provider-health, provider-id, and Edge handoff tokens; never copy raw message or arbitrary metadata into StreamGate. Use it in live run, buffered collector, Responses, and tunnel ERROR sources. Nil/other typed failures retain existing generic terminal behavior. Mapping preserves `unknown` as a health token; it does not decide retry eligibility.

Before (`apps/edge/internal/openai/run_result.go:87`):

```go
case "error", "cancelled":
    return "", "", "", nil, nil, false, fmt.Errorf("%s", msg)
```

After:

```go
case "error", "cancelled":
    return "", "", "", nil, nil, false, newOpenAIRunTerminalError(event)
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/run_result.go`: retain cloned typed terminal failure behind a safe internal error.
- [ ] `apps/edge/internal/openai/stream_gate_runtime.go`: centralize failure-to-event conversion and apply it to live run, buffered Chat, and tunnel sources.
- [ ] `apps/edge/internal/openai/responses_stream_gate.go`: apply the same conversion to buffered normalized Responses attempts.

**Test Strategy:** API-3 covers present/absent typed failures across every source. Unit-level assertions inspect descriptor/cause tokens and prove raw messages, provider bodies, prompts, credentials, and arbitrary metadata are absent.

**Verification:** `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallEventMapping)'` must PASS.

### [API-2] Gate exact replay and hand off the failed provider

**Problem:** `apps/edge/internal/openai/stream_gate_filters.go:198-237` always passes provider errors, and `stream_gate_policy.go:288-318` registers that filter only when explicitly listed in `filters[]`; simply making it recover would silently leave gate-enabled requests without that config entry ownerless. `stream_gate_dispatcher.go:353-376` also treats CancelRun success as the only abort result, while recovery admission builders copy no failed-provider hint.

**Solution:** Store the current eligible stalled provider plus its allowlisted probe classification in request-local ingress state. Whenever StreamGate is enabled for Chat or Responses, register exactly one private `response_stalled` liveness filter through the existing extra-registration seam; do not require or mutate `filters[]`, the configurable `provider_error` foundation filter, selector policy, or provider capability admission. The private filter emits ExactReplay only when descriptor/cause proves a confirmed Edge handoff, `EvidenceBatch` is `transport_uncommitted`, there is no tool fragment/side-effect in current/pending/look-behind evidence, and a request snapshot ref exists. Generic provider errors PASS to their existing terminal behavior; gate-disabled/unsupported ingress has no recovery owner and stays terminal. `available`, `unavailable`, and `unknown` confirmed handoffs may request recovery because the pool can find an alternate. For an eligible terminal, close request-local transport/lease without sending another CancelRun or inferring a fence; other recovery reasons keep current cancel behavior. Consume the recorded provider once into `AvoidProviderID`, set fallback only for exact `available`, and clear/replace state per serialized cycle.

Before (`apps/edge/internal/openai/stream_gate_filters.go:222`):

```go
case openAIOutputFilterProviderError:
    if batchHasProviderError(batch) {
        descriptor = "provider_error_observed_unmatched"
    }
```

After:

```go
case openAIOutputFilterProviderError:
    return f.evaluateProviderError(fctx, batch)
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_ingress.go`: own concurrency-safe request-local eligible-stall state and construct one internal liveness registration only for enabled StreamGate requests.
- [ ] `apps/edge/internal/openai/stream_gate_policy.go`: keep configured semantic filter/capability resolution unchanged and prove the internal registration is outside that admission policy.
- [ ] `apps/edge/internal/openai/stream_gate_filters.go`: add the private liveness evaluator for descriptor/cause, commit state, request ref, and side effects; keep configurable generic `provider_error` foundation behavior unchanged.
- [ ] `apps/edge/internal/openai/stream_gate_dispatcher.go`: use confirmed-terminal close semantics and pass request-local state through recovery controllers.
- [ ] `apps/edge/internal/openai/stream_gate_runtime.go`: share state across Chat/tunnel builders/controllers and set `AvoidProviderID` plus the available-derived fallback flag on pool recovery only.
- [ ] `apps/edge/internal/openai/responses_stream_gate.go`: share the identical state through Responses builders/controllers.
- [ ] `apps/edge/internal/openai/stream_gate_filters_test.go`: add available/unavailable/unknown eligible, unconfirmed, post-commit, and tool-side-effect private-filter tests; preserve configured generic-provider-error unmatched PASS and prove no configured-filter capability coupling.
- [ ] `apps/edge/internal/openai/stream_gate_dispatcher_test.go`: assert confirmed terminal closes without cancel, ordinary recovery still cancels, provider/available-fallback hints are consumed once, and controllers remain idempotent.

**Test Strategy:** Build batches with stable descriptor/cause tokens and each commit/side-effect boundary. Assert a gate-enabled request with empty configured `filters[]` owns exactly one liveness registration and can emit `RecoveryStrategyExactReplay`; explicitly configured `provider_error` neither duplicates that intent nor changes candidate capabilities. Gate-disabled, unconfirmed/unbound/post-commit/unsafe rows have no intent. Dispatcher spies assert zero extra CancelRun for an already confirmed terminal, one close, one avoided-provider handoff, and fallback only for available.

**Verification:** `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)'` must PASS every iteration.

### [API-3] Prove bounded recovery across OpenAI variants

**Problem:** Existing StreamGate vertical slices prove generic recovery and path switching, but none establish S05's typed liveness gates or same failure semantics across Chat/Responses and normalized/tunnel transports.

**Solution:** Add a focused scripted provider-pool matrix. Each recoverable fixture starts uncommitted with a confirmed eligible stall and returns a successful new attempt with a different run id. Available, unavailable, and unknown evidence all select an alternate when one exists; only available evidence permits a runtime-eligible same-provider fallback when no alternate exists. Unavailable/unknown same-only, no-owner, post-commit, unconfirmed, canceled, tool-bearing/side-effect, and exhausted shared budget remain one typed terminal with no duplicate provider dispatch. Exercise both streaming and buffered response release paths without widening public error data.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: add normalized/tunnel and Chat/Responses S05 matrix with dispatch/cancel/close/terminal identity assertions.
- [ ] `agent-contract/inner/execution-runtime.md`: document Edge eligibility -> OpenAI recovery handoff, confirmed-terminal close, and provider avoidance ownership.
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: document that the internal liveness owner follows `stream_evidence_gate.enabled`, not configured `filters[]` or provider capability admission, and that disabled ingress remains no-owner terminal.
- [ ] `agent-contract/outer/openai-compatible-api.md`: document terminal versus transparent pre-commit recovery behavior without exposing internals/raw data.
- [ ] `agent-spec/runtime/stream-evidence-gate.md`: record typed provider-error matcher, ExactReplay gates, and shared budget reuse.
- [ ] `agent-spec/input/openai-compatible-surface.md`: record Chat/Responses variant behavior and no-owner boundary.
- [ ] `agent-spec/runtime/edge-node-execution.md`: reflect final failure-handoff-to-retry integration and new attempt identity.

**Test Strategy:** Use existing scripted pool service and response sinks. Assert request count is initial+at-most-shared-budget, every recovery run id differs, `AvoidProviderID` equals the actual stalled provider, and `AllowAvoidedProviderFallback` is true only for the available same-only row. Cover unknown-with-alternate success and unknown-same-only terminal explicitly. Assert no leaked raw failure data, one old close, and one caller terminal. Include a two-fault fixture where another recovery strategy already consumes budget, proving no liveness-specific counter.

**Verification:** `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecovery'` must PASS every iteration.

## Dependencies and Execution Order

1. `09+08_retry_candidate_policy` must produce `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`; it is active/missing at refinement.
2. Implement API-1, then API-2, then API-3. Do not enable recovery before the typed mapper and controller/provider handoff are both present.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/run_result.go` | API-1 |
| `apps/edge/internal/openai/stream_gate_runtime.go` | API-1, API-2 |
| `apps/edge/internal/openai/responses_stream_gate.go` | API-1, API-2 |
| `apps/edge/internal/openai/stream_gate_ingress.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_policy.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_filters.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_dispatcher.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_filters_test.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_dispatcher_test.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | API-3 |
| `agent-contract/inner/execution-runtime.md` | API-3 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | API-3 |
| `agent-contract/outer/openai-compatible-api.md` | API-3 |
| `agent-spec/runtime/stream-evidence-gate.md` | API-3 |
| `agent-spec/input/openai-compatible-surface.md` | API-3 |
| `agent-spec/runtime/edge-node-execution.md` | API-3 |
| `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G08.md` | API-1, API-2, API-3 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)'` — PASS every iteration.
2. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecovery'` — PASS every iteration.
3. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS under the Edge local profile.
4. `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/openai` — PASS with no race report.
5. `go vet ./packages/go/streamgate ./apps/edge/internal/service ./apps/edge/internal/openai ./apps/edge/internal/controlplane` — no diagnostics.
6. `./scripts/e2e-smoke.sh` — PASS for provider-only dispatch/tunnel/queue/reconnect fencing.
7. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — PASS for credential-free OpenAI Chat streaming/non-streaming Edge -> Node -> provider full-cycle.
8. `./scripts/e2e-provider-capacity-smoke.sh` — PASS for deterministic local provider-pool queue and release behavior.
9. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
