<!-- task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery plan=5 tag=REVIEW_API milestone-task=bounded-retry -->

# Compatibility-Capable Always-On OpenAI Stall Recovery

## For the Implementing Agent

Implement only the two direct fixes below and run every verification command exactly as written. Filling the implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and complete raw output is mandatory. Keep the active pair in place and report ready for review; finalization belongs to the code-review skill. If blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The typed `response_stalled` mapper, private recovery filter, confirmed-transport close, provider avoidance, and shared StreamGate budget are present and independently verifiable. The always-on response-runtime conversion was reverted because the disabled-semantic path regressed endpoint-native behavior, so the default false configuration still bypasses the only liveness owner. The named matrix tests also stop at registry/filter evaluation and do not prove handler/runtime recovery or exactly-once public outcomes.

## Archive Evidence Snapshot

- The reviewed plan=4 pair is archived in this task directory as `plan_cloud_G08_4.log` and `code_review_cloud_G08_4.log` with verdict `FAIL`.
- Required R1: supported Chat/Responses normalized and tunnel entry points still select the liveness runtime through `streamGateEnabled()`, so `openai.stream_evidence_gate.enabled=false` remains ownerless; introduce an always-on supported-path owner while keeping semantic filters and capability admission flag-controlled and preserving disabled-semantic wire behavior.
- Required R2: `TestOpenAIStallRecoveryMatrix` invokes the private filter directly and `TestOpenAISemanticGateDisabledCompatibility` only counts registrations; neither proves handler dispatch, provider selection, new identity, shared budget, old-transport close, unsafe/no-owner terminals, or exactly-once rendering.
- Fresh reviewer reruns passed all eleven exact commands, including separate service and OpenAI race runs. Evidence integrity is trusted; the blocking deficiency is implementation and coverage completeness.

## Finding Resolution Map

| Finding | Mode | Exact Fix / Evidence | Changed or Satisfied Precondition |
|---------|------|----------------------|-----------------------------------|
| Required R1 | `direct-fix` | Split supported-path runtime ownership from semantic activation in the OpenAI host, suppress configured semantic policy when disabled, adapt runtime release behavior to the legacy public contract, and synchronize active contracts/specs. | Every supported Chat/Responses normalized/tunnel response enters one runtime even when the semantic flag is false; ordinary disabled-semantic output no longer requires an ownerless legacy branch. |
| Required R2 | `direct-fix` | Replace registry/filter-only matrix assertions with deterministic handler/runtime fixtures covering dispatch, provider choice, identity, budget, close, terminal, cancellation, and public rendering. | Re-running the named matrix commands will exercise the production lifecycle and can close SDD S05 instead of repeating unchanged filter evidence. |

## Analysis

### Files Read

- `apps/edge/internal/openai/stream_gate_runtime.go`
- `apps/edge/internal/openai/stream_gate_policy.go`
- `apps/edge/internal/openai/stream_gate_filters.go`
- `apps/edge/internal/openai/stream_gate_dispatcher.go`
- `apps/edge/internal/openai/stream_gate_release_sink.go`
- `apps/edge/internal/openai/chat_handler.go`
- `apps/edge/internal/openai/chat_completion.go`
- `apps/edge/internal/openai/buffered_sse.go`
- `apps/edge/internal/openai/normalized_sse.go`
- `apps/edge/internal/openai/provider_tunnel.go`
- `apps/edge/internal/openai/responses_handler.go`
- `apps/edge/internal/openai/responses_stream_gate.go`
- `apps/edge/internal/openai/run_result.go`
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
- `apps/edge/internal/openai/stream_gate_pipeline_test.go`
- `apps/edge/internal/openai/stream_gate_vertical_slice_test.go`
- `packages/go/streamgate/runtime.go`
- `packages/go/streamgate/recovery_coordinator.go`
- `packages/go/streamgate/commit_boundary.go`
- `packages/go/streamgate/filter_registry.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[Approved]`, lock released, no user review.
- Header contribution id: `milestone-task=bounded-retry`.
- Acceptance Scenario S05 requires the ingress host to replay only confirmed, uncommitted, side-effect-safe requests through the shared StreamGate fault budget with a new run identity; post-commit, unconfirmed, and ownerless requests terminate.
- Evidence Map S05 requires commit-boundary/shared-budget, provider-pool failover, no-owner terminal, recovery-owner gating, new run identity, and bounded dispatch-count assertions.
- These rows require REVIEW_API-1 to install the supported-host owner and REVIEW_API-2 to drive the production endpoint/path/config lifecycle. Unit-only mapper/filter evidence cannot satisfy the map.

### Verification Context

- No external handoff was supplied. Inputs are the active implementation artifact, the approved SDD, active contracts/specs, repository source/tests, the satisfied predecessor completion log, and fresh reviewer reruns.
- Local preconditions: repository root `/config/workspace/iop-s1`, Go module at `go.mod`, current dirty checkout preserved, fake-mode vLLM smoke, and repository-native shell smokes. No command leaves the checkout or requires a remote runner.
- Applied criteria: focused tests repeat 10-20 times, package integration uses `-count=1`, race packages run separately with `-count=3`, vet must emit no diagnostics, all three smokes must report PASS, and `git diff --check` must be clean. Cached output is not acceptable where `-count` is specified.
- Existing reviewer output proves the commands are executable and the retained typed subset is stable. It does not prove that the named matrix tests traverse handlers or runtime recovery, so confidence is high in the root cause and low in current S05 completeness.

### Test Coverage Gaps

- Always-on ownership: no test invokes each supported handler with semantic enablement false and proves that the private liveness runtime owns the response.
- Recovery lifecycle: no current matrix proves alternate/same-provider re-admission, new run identity, shared-budget consumption, old-transport close, or one replacement dispatch.
- Terminal guards: no current matrix proves zero re-admission for unconfirmed, committed, cancelled, side-effect/tool, missing-snapshot, exhausted-budget, unsupported, and no-owner rows.
- Compatibility: no current matrix compares disabled-semantic JSON/SSE/tunnel status, headers, bytes/order, cancellation, strict/tool validation, reasoning fallback, finish reason, usage, and exactly-one terminal against endpoint-native expectations.

### Symbol References

- `streamGateEnabled()` currently appears in `chat_handler.go:336`, `responses_handler.go:151,508,532`, `chat_completion.go:40`, `buffered_sse.go:15`, `normalized_sse.go:41`, and `provider_tunnel.go:33,579`; every supported response-ownership call site must move to the explicit always-on decision or directly to the runtime.
- `streamGateSemanticEnabled()` appears at provider candidate-admission call sites in `chat_handler.go` and `responses_handler.go`; it must remain semantic-only and must not delegate to the response-ownership decision.
- No public symbol rename is planned. If the internal ownership helper is renamed, update every call site listed above and keep semantic admission references separate.

### Split Judgment

Keep one plan because response commit, caller cancellation, side-effect state, shared recovery budget, attempt transport close, failed-provider avoidance, re-admission, and final rendering form one request-local correctness boundary. The encoded predecessor `09+08_retry_candidate_policy` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log` with final PASS.

### Scope Rationale

Do not add another retry loop, liveness counter, config field, metric, wire field, Core/Node recovery owner, or non-OpenAI owner. Do not change the typed failure mapper, service candidate policy, raw provider error policy, or unsupported-surface behavior unless a listed OpenAI call-site adaptation is strictly required. Preserve unrelated dirty-worktree changes.

### Final Routing

- `status=routed`; `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`.
- Build closures: scope/context/verification/evidence/ownership/decision all true; capability gap absent. Scores `(2,2,1,1,2)` produce G08 with base `local-fit`; `large_indivisible_context=false`.
- Positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product`; count 4 and risk boundary matched.
- Recovery signals: `review_rework_count=2`, `evidence_integrity_failure=false`; recovery boundary matched and build route is `recovery-boundary`, cloud, `PLAN-cloud-G08.md`.
- Review closures are all true; scores `(2,2,1,1,2)` produce G08. Official review is cloud Codex `gpt-5.6-sol` xhigh at `CODE_REVIEW-cloud-G08.md`.

## Implementation Checklist

- [ ] REVIEW_API-1 gives every supported OpenAI Chat/Responses normalized and tunnel request exactly one private typed-stall recovery owner independent of semantic gate enablement, while the flag alone controls configured semantic filters and candidate capability admission and disabled-semantic public behavior remains compatible.
- [ ] REVIEW_API-2 replaces registry/filter-only coverage with deterministic production handler/runtime lifecycle tests for the S05 endpoint/path/config, provider-selection, safety-terminal, identity, budget, close, cancellation, and exactly-once matrices.
- [ ] Synchronize the active execution/config/OpenAI contracts and matching specs with always-on supported-path liveness ownership and semantic-only flag behavior, then run every exact verification command.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_API-1] Install the compatibility-capable supported-path owner

**Problem:** `stream_gate_runtime.go:796-807` makes response ownership and semantic activation the same boolean. Supported handlers consequently keep legacy ownerless branches under the default false flag. Changing only the predicate is insufficient because prior always-on wiring changed cancellation, strict/tool and reasoning rendering, tunnel error ordering, and write-failure behavior.

Before (`apps/edge/internal/openai/stream_gate_runtime.go:796-807`):

```go
func (s *Server) streamGateEnabled() bool {
	s.mu.RLock()
	defer s.mu.RUnlock()
	return s.cfg.StreamEvidenceGate.Enabled
}

func (s *Server) streamGateSemanticEnabled() bool { return s.streamGateEnabled() }
```

After:

```go
func (s *Server) openAIResponseRuntimeOwned() bool { return true }

func (s *Server) streamGateSemanticEnabled() bool {
	s.mu.RLock()
	defer s.mu.RUnlock()
	return s.cfg.StreamEvidenceGate.Enabled
}
```

**Solution:** Route each listed supported Chat/Responses normalized and tunnel result through the existing request-local runtime exactly once. Make `openAIOutputFilterRegistrations` return no configured semantic registrations/policies when `gateCfg.Enabled` is false, while the no-op mechanics, private typed-stall registration, and request-local tool validation remain active. Propagate an explicit semantic-disabled compatibility mode into the existing event-source/release adapters and repair mismatches there: preserve native JSON/SSE/tunnel status/header/body order, strict/tool retry and validation, reasoning fallback, finish reason, caller cancellation/write-failure handling, usage finalization, and one terminal. Do not retain an ownerless fallback and do not introduce a parallel retry loop.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_runtime.go`: separate predicates, always build one supported response runtime, and carry compatibility mode through runtime construction.
- [ ] `apps/edge/internal/openai/stream_gate_policy.go`: suppress configured semantic registrations and capability requirements when disabled.
- [ ] `apps/edge/internal/openai/chat_handler.go`: retain semantic-only candidate admission and route provider-pool results through the runtime owner.
- [ ] `apps/edge/internal/openai/responses_handler.go`: retain semantic-only candidate admission and route normalized/tunnel results through the runtime owner.
- [ ] `apps/edge/internal/openai/chat_completion.go`, `apps/edge/internal/openai/buffered_sse.go`, `apps/edge/internal/openai/normalized_sse.go`, `apps/edge/internal/openai/provider_tunnel.go`: remove ownerless supported response selection.
- [ ] `apps/edge/internal/openai/stream_gate_release_sink.go`, `apps/edge/internal/openai/responses_stream_gate.go`: make disabled-semantic release, terminal, cancellation, write-failure, strict/tool, reasoning, finish, tunnel, and usage behavior endpoint-compatible.

**Test Strategy:** Write regression coverage in `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`. `TestOpenAISemanticGateDisabledCompatibility` must invoke Chat/Responses handlers for normalized and tunnel success/cancel/error fixtures and compare public status, headers, JSON/SSE bytes/order, finish/reasoning/tool output, usage finalization, and terminal count. Reuse existing package fakes; do not test only helper predicates or registration counts.

**Verification:** Run verification commands 2-7 and 11. All supported disabled-semantic subtests must enter the runtime, configured semantic filters/capability admission must remain absent, outputs must match native expectations, and race/vet/diff checks must pass.

### [REVIEW_API-2] Prove the S05 handler/runtime lifecycle matrix

**Problem:** `stream_gate_stall_recovery_test.go:141-190` labels endpoint/path/config combinations but constructs a registry and calls `stall.Filter().Evaluate` directly; lines 195-213 only count registrations. Those tests cannot detect the ownerless handler branches or prove re-admission and public terminal invariants.

Before (`apps/edge/internal/openai/stream_gate_stall_recovery_test.go:168-184`):

```go
filter := stall.Filter()
decision, err := filter.Evaluate(t.Context(), stallFilterContext(...), stallBatch(...))
if err != nil || decision.RecoveryIntent() == nil { /* fail */ }
provider, sameProviderFallback, ok := state.consumeAdmission()
```

After fixture shape:

```go
result := driveOpenAIStallHandler(t, endpoint, path, semantic, fixture)
assertRecoveryDispatch(t, result, fixture.wantDispatches, fixture.wantProvider)
assertAttemptIdentityAndBudget(t, result)
assertTransportCloseAndPublicTerminal(t, result)
```

**Solution:** Replace the label-only matrix with deterministic production handler/runtime fixtures across Chat and Responses, normalized and tunnel, semantic false and true. For confirmed uncommitted safe stalls, assert alternate recovery for `available|unavailable|unknown`; permit same-provider-only recovery only for `available`; require terminal for unavailable/unknown same-only. Assert exactly one replacement dispatch, a new run/attempt identity, one shared-budget debit, failed-provider avoidance, confirmed old transport close without duplicate cancel, and one public terminal. Add zero-recovery terminal rows for generic/unconfirmed, post-commit, caller-cancelled, tool/side-effect, missing snapshot, exhausted budget, unsupported, and no-owner cases. Keep typed failures and public errors sanitized.

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: implement the production lifecycle, guard-terminal, and compatibility fixtures and assertions.
- [ ] `agent-contract/inner/execution-runtime.md`, `agent-contract/inner/edge-config-runtime-refresh.md`, `agent-contract/outer/openai-compatible-api.md`: state always-on supported-path liveness ownership and semantic-only configuration behavior.
- [ ] `agent-spec/runtime/stream-evidence-gate.md`, `agent-spec/input/openai-compatible-surface.md`, `agent-spec/runtime/edge-node-execution.md`: synchronize current implementation and test evidence pointers.
- [ ] `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G08.md`: record implementation decisions and complete raw output.

**Test Strategy:** Rewrite `TestOpenAIStallRecoveryMatrix` and `TestOpenAISemanticGateDisabledCompatibility` as handler/runtime integration tests using deterministic fake run/tunnel transports, candidate catalogs, identities, budgets, usage recorders, response writers, close/cancel counters, and dispatch counters. Retain focused mapper/filter units as lower-level regressions.

**Verification:** Run all eleven final commands. The two named tests must execute every product row repeatedly, package/race/vet suites must pass, smokes must remain green, and the review artifact must contain the complete output of each separate invocation.

## Dependencies and Execution Order

1. The predecessor `09+08_retry_candidate_policy` is complete at `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`.
2. Implement REVIEW_API-1 and its compatibility assertions before relying on REVIEW_API-2 recovery rows.
3. Complete REVIEW_API-2 and synchronize contracts/specs, then run final verification from separate fresh command invocations.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/stream_gate_runtime.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/stream_gate_policy.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/chat_handler.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/responses_handler.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/chat_completion.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/buffered_sse.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/normalized_sse.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/provider_tunnel.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/stream_gate_release_sink.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/responses_stream_gate.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | REVIEW_API-1, REVIEW_API-2 |
| `agent-contract/inner/execution-runtime.md` | REVIEW_API-2 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | REVIEW_API-2 |
| `agent-contract/outer/openai-compatible-api.md` | REVIEW_API-2 |
| `agent-spec/runtime/stream-evidence-gate.md` | REVIEW_API-2 |
| `agent-spec/input/openai-compatible-surface.md` | REVIEW_API-2 |
| `agent-spec/runtime/edge-node-execution.md` | REVIEW_API-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G08.md` | REVIEW_API-1, REVIEW_API-2 |

## Final Verification

Fresh output is required. Record each command and its complete stdout/stderr separately; Go cached output is not acceptable where `-count` is present.

1. `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)$'` — PASS every iteration.
2. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecoveryMatrix$'` — PASS every endpoint/path/config/recovery/terminal row.
3. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAISemanticGateDisabledCompatibility$'` — PASS every endpoint-native compatibility row.
4. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS.
5. `go test -race -count=3 ./apps/edge/internal/service` — PASS with no race report.
6. `go test -race -count=3 ./apps/edge/internal/openai` — PASS with no race report.
7. `go vet ./packages/go/streamgate ./apps/edge/internal/service ./apps/edge/internal/openai ./apps/edge/internal/controlplane` — no diagnostics.
8. `./scripts/e2e-smoke.sh` — PASS.
9. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — PASS.
10. `./scripts/e2e-provider-capacity-smoke.sh` — PASS.
11. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
