<!-- task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery plan=3 tag=API milestone-task=bounded-retry -->

# OpenAI Typed Stall Recovery Handoff

## For the Implementing Agent

Implement only the items below after all predecessors PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The OpenAI host currently maps every normalized/tunnel terminal error to generic StreamGate provider errors, and the provider-error filter intentionally never constructs recovery. S05 requires a typed `response_stalled` handoff that remains terminal unless Edge confirmed the attempt fence and the request is uncommitted, uncanceled, side-effect-safe, and within the existing shared Core budget; eligible recovery must use a new run identity and prefer another provider.

## Archive Evidence Snapshot

- Prior pair: `plan_cloud_G08_1.log` and `code_review_cloud_G08_1.log` in this task directory. It was unimplemented and has no official verdict, implementation evidence, code change, or verification output.
- Prior fresh-review finding: `normal-10` is a long-context admission smoke and does not prove typed stall mapping, recovery gating, provider avoidance, or bounded dispatch.
- Union preparation review archived the unimplemented plan=2 pair as `plan_cloud_G08_2.log` and `code_review_cloud_G08_2.log`; it had no verdict or implementation evidence. The material scope defect was treating default `stream_evidence_gate.enabled=false` OpenAI requests as ownerless even though the approved SDD assigns typed-stall recovery to the supported OpenAI-compatible host.
- Replan carryover: preserve the raw-free mapper, shared Core commit/cancel/side-effect/budget ownership, and S05 matrix. Install one internal liveness owner for every supported OpenAI Chat/Responses normalized or tunnel request regardless of semantic-gate enablement/configuration; only unsupported or non-OpenAI surfaces remain the no-owner typed-terminal boundary.

## Analysis

### Files Read

- `apps/edge/internal/openai/stream_gate_ingress.go`, `apps/edge/internal/openai/stream_gate_policy.go`, `apps/edge/internal/openai/stream_gate_filters.go`, `apps/edge/internal/openai/stream_gate_runtime.go`
- `apps/edge/internal/openai/stream_gate_dispatcher.go`, `apps/edge/internal/openai/run_result.go`, `apps/edge/internal/openai/responses_stream_gate.go`, `apps/edge/internal/openai/chat_completion.go`
- `apps/edge/internal/openai/stream_gate_filters_test.go`, `apps/edge/internal/openai/stream_gate_dispatcher_test.go`, `apps/edge/internal/openai/stream_gate_pipeline_test.go`, `apps/edge/internal/openai/stream_gate_vertical_slice_test.go`
- `apps/edge/internal/openai/provider_test_support_test.go`, `apps/edge/internal/openai/server_test_support_test.go`
- `packages/go/streamgate/runtime.go`, `packages/go/streamgate/recovery_coordinator.go`, `packages/go/streamgate/recovery_plan.go`, `packages/go/streamgate/commit_boundary.go`, `packages/go/streamgate/event.go`, `packages/go/streamgate/filter_registry.go`, `packages/go/streamgate/terminal.go`
- `packages/go/config/config.go`, `agent-contract/inner/edge-config-runtime-refresh.md`, `agent-contract/inner/execution-runtime.md`, `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`, `agent-spec/input/openai-compatible-surface.md`, `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `agent-test/local/edge-smoke.md`, `agent-test/local/platform-common-smoke.md`, `scripts/e2e-smoke.sh`, `scripts/e2e-openai-vllm.sh`, `scripts/e2e-provider-capacity-smoke.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`, and this pair's first-line id is `milestone-task=bounded-retry`.
- Acceptance Scenario S05 and Evidence Map S05 require healthy request stall, unhealthy failover, unknown probe, same-provider-only, no owner, post-commit, unconfirmed fence, caller cancel/tool-side-effect, and shared-budget fixtures with a new run identity and bounded dispatch/terminal count. Unknown health may recover through an alternate candidate but never grants same-provider fallback.
- API-1 derives a raw-free typed event, API-2 implements exact-replay eligibility/fence teardown/provider handoff, and API-3 proves both Chat/Responses plus normalized/tunnel variants against those rows.

### Verification Context

- Handoff supplied starting HEAD `56e7d78af3cda4a8d6a85af091ad26bce935f8b6`; fresh baseline tests passed for execution, Node, Edge transport/service/OpenAI.
- This plan waits for `09+08_retry_candidate_policy/complete.log`; transitively, typed wire and Edge eligibility/overlay contracts are also PASS before implementation.
- Core already owns `transport_uncommitted`, caller-cancel propagation, abort-before-dispatch, new `AttemptBinding`, and one request-local total/per-strategy fault budget. No liveness counter or Core recovery coordinator is needed.
- Existing OpenAI event sources discard typed failures (`stream_gate_runtime.go:128-188`, `402-478`, `responses_stream_gate.go:110-135`) and `collectRunResult` flattens them (`run_result.go:87-95`). Provider-error filtering is explicitly foundation-only (`stream_gate_filters.go:29-32`, `198-237`).
- `openai.stream_evidence_gate.enabled` defaults false, while the approved liveness SDD assigns typed-stall recovery to the supported OpenAI-compatible host without conditioning ownership on that flag. Separate semantic evidence-gate activation from liveness ownership: every supported Chat/Responses normalized or tunnel request gets exactly one internal liveness owner and the existing request-local commit/recovery coordinator; the flag and `filters[]` continue to control only configured semantic filters, evidence holding, and capability admission. Normal non-stall behavior with the semantic gate disabled must remain byte/ordering compatible.
- No required verification leaves this checkout. Focused/race fixtures directly exercise S05, while fake-vLLM and provider-capacity scripts cover repository-native OpenAI and queue full cycles; the latest output-filter and Hot Path SDDs preserve Core retry/terminal ownership and do not add a second liveness loop. Confidence is medium-high due to multi-variant runtime and terminal ordering.

### Test Coverage Gaps

- Typed normalized and tunnel stalls are not distinguished from generic provider errors.
- Provider-error filter has only unmatched PASS coverage; no test proves that both semantic-gate-enabled and semantic-gate-disabled supported OpenAI requests install exactly one internal liveness owner, while unsupported/non-OpenAI surfaces remain ownerless terminal.
- Recovery dispatcher does not hand the failed provider to pool admission or distinguish a confirmed Node terminal from a still-running attempt during abort.
- Existing vertical slices prove shared-budget recovery generally, but not Chat/Responses stall variants, unknown/no-owner/post-commit/unconfirmed gates, or duplicate-terminal absence.

### Symbol References

- No symbol is renamed or removed. Constructor signatures for request-local dispatcher/filter wiring may gain internal state; update all call sites in `stream_gate_runtime.go`, `responses_stream_gate.go`, and their direct tests.

### Split Judgment

- Predecessor `09+08_retry_candidate_policy` is active with missing `complete.log`; implementation waits for it. That predecessor transitively requires `05+04_failure_wire_contract`, `06+05_failure_wire_mapping`, `07+06_reception_fence`, and `08+07_health_overlay` PASS.
- This final packet is indivisible at the OpenAI host boundary: the same raw-free eligibility token must drive filter intent, confirmed-terminal teardown, failed-provider handoff, and endpoint terminal rendering. Partial wiring could either duplicate dispatch or authorize an unfenced replay.

### Scope Rationale

Do not add a Core/Node/Edge retry loop, new counter, schema, metric, or non-OpenAI/unsupported recovery owner. Do not expose raw provider messages/metadata. StreamGate Core behavior remains unchanged; this packet consumes its existing budget/commit/cancel/side-effect contracts.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true, scores `(2,2,1,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
- Review closure true, scores `(2,2,1,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.

## Implementation Checklist

- [ ] API-1 preserves typed normalized/buffered/tunnel stalls as one raw-free StreamGate `response_stalled` provider error, retaining only sanitized fence/health and `recovery_handoff=confirmed` authority tokens while generic failures keep existing terminal behavior.
- [ ] API-2 installs exactly one internal liveness recovery owner for every supported OpenAI Chat/Responses normalized or tunnel request independent of `stream_evidence_gate.enabled` and configured semantic filters/capabilities; only confirmed handoff, uncommitted, uncanceled, side-effect-safe, budget-available stalls produce ExactReplay, close the fenced old transport, and hand failed-provider/fallback evidence to admission.
- [ ] API-3 adds semantic-gate-enabled/disabled Chat/Responses normalized/tunnel fixtures for available, unavailable, and unknown alternate recovery; available-only same-provider fallback; unavailable/unknown same-only terminal; unsupported/no-owner, post-commit, unconfirmed, cancel/tool-side-effect, and shared-budget exhaustion; synchronize contracts/specs.
- [ ] Run focused, package, race, vet, provider-only/OpenAI/local-capacity full-cycles, and diff verification with fresh output; assert new identities plus exactly one terminal/dispatch per allowed cycle.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [API-1] Convert typed execution stalls into raw-free StreamGate events

**Problem:** `apps/edge/internal/openai/stream_gate_runtime.go:182-184` and `473-478` emit generic `run_failed`/`provider_tunnel_error`, while `apps/edge/internal/openai/run_result.go:87-95` converts a terminal event into an untyped formatted error. Buffered Chat/Responses therefore cannot preserve the same failure semantics as live/tunnel paths.

**Solution:** Add an internal terminal error that defensively retains the protobuf failure while its `Error()` exposes only a stable code. Centralize conversion of typed execution failure to `ExternalDescriptor(code=response_stalled)` and bounded allowlisted fence, provider-health, provider-id, and Edge `recovery_handoff=confirmed` authority token; never copy raw message or arbitrary metadata into StreamGate. Use it in live run, buffered collector, Responses, and tunnel ERROR sources. Nil/other typed failures retain existing generic terminal behavior. Mapping preserves `unknown` as a health token; it does not decide retry eligibility.

Before (`apps/edge/internal/openai/run_result.go:87`):

```go
case "error", "cancelled":
    return "", "", "", nil, nil, false, fmt.Errorf("%s", msg)
```

After:

```go
case "error", "cancelled":
    return "", "", "", nil, nil, false, newOpenAIRunTerminalError(event)
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/run_result.go`: retain cloned typed terminal failure behind a safe internal error.
- [ ] `apps/edge/internal/openai/stream_gate_runtime.go`: centralize failure-to-event conversion and apply it to live run, buffered Chat, and tunnel sources.
- [ ] `apps/edge/internal/openai/responses_stream_gate.go`: apply the same conversion to buffered normalized Responses attempts.

**Test Strategy:** API-3 covers present/absent typed failures across every source. Unit-level assertions inspect descriptor/cause tokens and prove raw messages, provider bodies, prompts, credentials, and arbitrary metadata are absent.

**Verification:** `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallEventMapping)'` must PASS.

### [API-2] Gate exact replay and hand off the failed provider

**Problem:** `apps/edge/internal/openai/stream_gate_filters.go:198-237` always passes provider errors, while runtime construction is skipped when `stream_evidence_gate.enabled=false`. Merely changing the configured `provider_error` filter would leave the default supported OpenAI path ownerless and would incorrectly couple liveness to semantic filter configuration. `stream_gate_dispatcher.go:353-376` also treats CancelRun success as the only abort result, while recovery admission builders copy no failed-provider hint.

**Solution:** Store the current eligible stalled provider plus its allowlisted probe classification in request-local ingress state. For every supported OpenAI Chat/Responses normalized or tunnel request, construct the request-local StreamGate commit/recovery host and register exactly one private `response_stalled` liveness filter through the existing extra-registration seam, even when semantic evidence gating is disabled. Do not require or mutate `filters[]`, the configurable `provider_error` foundation filter, selector policy, or provider capability admission. With semantic gating disabled, normal non-stall events release without semantic holding and preserve legacy wire ordering; only typed stalls enter the liveness evaluator. The private filter emits ExactReplay only when descriptor/cause proves a confirmed Edge handoff, `EvidenceBatch` is `transport_uncommitted`, there is no tool fragment/side-effect in current/pending/look-behind evidence, and a request snapshot ref exists. Generic provider errors PASS to their existing terminal behavior; only unsupported/non-OpenAI ingress has no recovery owner and stays terminal. `available`, `unavailable`, and `unknown` confirmed handoffs may request recovery because the pool can find an alternate. For an eligible terminal, close request-local transport/lease without sending another CancelRun or inferring a fence; other recovery reasons keep current cancel behavior. Consume the recorded provider once into `AvoidProviderID`, set fallback only for exact `available`, and clear/replace state per serialized cycle.

Before (`apps/edge/internal/openai/stream_gate_filters.go:222`):

```go
case openAIOutputFilterProviderError:
    if batchHasProviderError(batch) {
        descriptor = "provider_error_observed_unmatched"
    }
```

After:

```go
case openAIOutputFilterProviderError:
    return f.evaluateProviderError(fctx, batch)
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_ingress.go`: own concurrency-safe request-local eligible-stall state and construct exactly one internal liveness registration for every supported OpenAI request, independent of semantic-gate enablement.
- [ ] `apps/edge/internal/openai/stream_gate_policy.go`: separate always-on supported-path liveness ownership from `stream_evidence_gate.enabled`; keep configured semantic filter/capability resolution unchanged and prove the internal registration is outside that admission policy.
- [ ] `apps/edge/internal/openai/stream_gate_filters.go`: add the private liveness evaluator for descriptor/cause, commit state, request ref, and side effects; keep configurable generic `provider_error` foundation behavior unchanged.
- [ ] `apps/edge/internal/openai/stream_gate_dispatcher.go`: use confirmed-terminal close semantics and pass request-local state through recovery controllers.
- [ ] `apps/edge/internal/openai/stream_gate_runtime.go`: share state across Chat/tunnel builders/controllers and set `AvoidProviderID` plus the available-derived fallback flag on pool recovery only.
- [ ] `apps/edge/internal/openai/responses_stream_gate.go`: share the identical state through Responses builders/controllers.
- [ ] `apps/edge/internal/openai/stream_gate_filters_test.go`: add available/unavailable/unknown eligible, unconfirmed, post-commit, and tool-side-effect private-filter tests; preserve configured generic-provider-error unmatched PASS and prove no configured-filter capability coupling.
- [ ] `apps/edge/internal/openai/stream_gate_dispatcher_test.go`: assert confirmed terminal closes without cancel, ordinary recovery still cancels, provider/available-fallback hints are consumed once, and controllers remain idempotent.

**Test Strategy:** Build batches with stable descriptor/cause tokens and each commit/side-effect boundary. Assert both semantic-gate-enabled and disabled supported requests own exactly one liveness registration and can emit `RecoveryStrategyExactReplay`; explicitly configured `provider_error` neither duplicates that intent nor changes candidate capabilities. Prove semantic-gate-disabled non-stall output remains legacy-compatible. Unsupported/non-OpenAI, unconfirmed/unbound/post-commit/unsafe rows have no intent. Dispatcher spies assert zero extra CancelRun for an already confirmed terminal, one close, one avoided-provider handoff, and fallback only for available.

**Verification:** `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)'` must PASS every iteration.

### [API-3] Prove bounded recovery across OpenAI variants

**Problem:** Existing StreamGate vertical slices prove generic recovery and path switching, but none establish S05's typed liveness gates or same failure semantics across Chat/Responses and normalized/tunnel transports.

**Solution:** Add a focused scripted provider-pool matrix. Each recoverable fixture starts uncommitted with a confirmed eligible stall and returns a successful new attempt with a different run id. Available, unavailable, and unknown evidence all select an alternate when one exists; only available evidence permits a runtime-eligible same-provider fallback when no alternate exists. Unavailable/unknown same-only, unsupported/no-owner, post-commit, unconfirmed, canceled, tool-bearing/side-effect, and exhausted shared budget remain one typed terminal with no duplicate provider dispatch. Run recoverable rows with semantic gating both enabled and disabled. Exercise both streaming and buffered response release paths without widening public error data.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: add normalized/tunnel and Chat/Responses S05 matrix with dispatch/cancel/close/terminal identity assertions.
- [ ] `agent-contract/inner/execution-runtime.md`: document Edge eligibility -> OpenAI recovery handoff, confirmed-terminal close, and provider avoidance ownership.
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: document that supported-path internal liveness ownership is independent of `stream_evidence_gate.enabled`, configured `filters[]`, and provider capability admission; the flag continues to control semantic evidence-gate behavior.
- [ ] `agent-contract/outer/openai-compatible-api.md`: document terminal versus transparent pre-commit recovery behavior without exposing internals/raw data.
- [ ] `agent-spec/runtime/stream-evidence-gate.md`: record typed provider-error matcher, ExactReplay gates, and shared budget reuse.
- [ ] `agent-spec/input/openai-compatible-surface.md`: record Chat/Responses variant behavior and no-owner boundary.
- [ ] `agent-spec/runtime/edge-node-execution.md`: reflect final failure-handoff-to-retry integration and new attempt identity.

**Test Strategy:** Use existing scripted pool service and response sinks. Assert request count is initial+at-most-shared-budget, every recovery run id differs, `AvoidProviderID` equals the actual stalled provider, and `AllowAvoidedProviderFallback` is true only for the available same-only row. Cover unknown-with-alternate success and unknown-same-only terminal explicitly. Run supported-path recovery with semantic gating enabled and disabled, and reserve no-owner terminal coverage for unsupported/non-OpenAI surfaces. Assert no leaked raw failure data, one old close, and one caller terminal. Include a two-fault fixture where another recovery strategy already consumes budget, proving no liveness-specific counter.

**Verification:** `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecovery'` must PASS every iteration.

## Dependencies and Execution Order

1. `09+08_retry_candidate_policy` must produce `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`; it is active/missing at refinement.
2. Implement API-1, then API-2, then API-3. Do not enable recovery before the typed mapper and controller/provider handoff are both present.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/run_result.go` | API-1 |
| `apps/edge/internal/openai/stream_gate_runtime.go` | API-1, API-2 |
| `apps/edge/internal/openai/responses_stream_gate.go` | API-1, API-2 |
| `apps/edge/internal/openai/stream_gate_ingress.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_policy.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_filters.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_dispatcher.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_filters_test.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_dispatcher_test.go` | API-2 |
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | API-3 |
| `agent-contract/inner/execution-runtime.md` | API-3 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | API-3 |
| `agent-contract/outer/openai-compatible-api.md` | API-3 |
| `agent-spec/runtime/stream-evidence-gate.md` | API-3 |
| `agent-spec/input/openai-compatible-surface.md` | API-3 |
| `agent-spec/runtime/edge-node-execution.md` | API-3 |
| `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G08.md` | API-1, API-2, API-3 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)'` — PASS every iteration.
2. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecovery'` — PASS every iteration.
3. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS under the Edge local profile.
4. `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/openai` — PASS with no race report.
5. `go vet ./packages/go/streamgate ./apps/edge/internal/service ./apps/edge/internal/openai ./apps/edge/internal/controlplane` — no diagnostics.
6. `./scripts/e2e-smoke.sh` — PASS for provider-only dispatch/tunnel/queue/reconnect fencing.
7. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — PASS for credential-free OpenAI Chat streaming/non-streaming Edge -> Node -> provider full-cycle.
8. `./scripts/e2e-provider-capacity-smoke.sh` — PASS for deterministic local provider-pool queue and release behavior.
9. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
