<!-- task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery plan=8 tag=REVIEW_API milestone-task=bounded-retry -->

# Restore Responses Streaming-Tunnel Stall Re-admission

## For the Implementing Agent

Implement only the two direct fixes below. Run every verification command exactly as written, fill the implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with actual notes and complete raw output, keep the active pair in place, and report ready for review. Finalization belongs to the code-review skill. If blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The always-on runtime now owns supported Chat and Responses paths, but exact replay for a streaming Responses tunnel is rejected before replacement provider-path selection. The production recovery matrix passes because every endpoint/path recovery row sends `stream=false`, so it does not execute the supported product that exposes this ordering defect.

## Archive Evidence Snapshot

- The reviewed plan=7 pair is archived in this task directory as `plan_cloud_G10_7.log` and `code_review_cloud_G10_7.log` with verdict `FAIL`.
- Required R1: `newOpenAIResponsesRecoveryAdmissionBuilder` constructs a normalized dispatch context before replacement path selection, and `newResponsesDispatchContext` rejects the exact `stream=true` replay body even when the next candidate is another tunnel.
- Required R2: `TestOpenAIStallRecoveryMatrix` hard-codes `stream=false` for every recovery product and therefore does not cover the supported Responses streaming-tunnel recovery path.
- All eleven declared verification commands passed on fresh reviewer reruns. A focused production-handler probe failed for semantic false and true with one dispatch and `recovery_failed`. Routing signals are `review_rework_count=5` and `evidence_integrity_failure=true`.

## Finding Resolution Map

| Finding | Mode | Exact Fix / Evidence | Changed or Satisfied Precondition |
|---------|------|----------------------|-----------------------------------|
| Required R1 | `direct-fix` | In `apps/edge/internal/openai/responses_stream_gate.go`, preserve a decoded public replay as a tunnel-capable attempt context and move normalized-only construction/validation into `PrepareRun`, after `SubmitProviderPool` selects a normalized candidate. | The replacement path now decides which request contract applies; a tunnel candidate can retain `stream=true`, while a normalized candidate still rejects unsupported streaming through the existing constructor. |
| Required R2 | `direct-fix` | In `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`, add explicit semantic-false and semantic-true Responses `stream=true` tunnel recovery rows using the production handler/runtime. | The named matrix will execute the previously absent product and fail if recovery stops after the first dispatch or renders a recovery error terminal. |

## Analysis

### Files Read

- `apps/edge/internal/openai/responses_stream_gate.go`
- `apps/edge/internal/openai/responses_handler.go`
- `apps/edge/internal/openai/dispatch_context.go`
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
- `apps/edge/internal/openai/provider_test_support_test.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/plan_cloud_G10_7.log`
- `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/code_review_cloud_G10_7.log`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[Approved]`, lock released, no user-review gate.
- Contribution: `milestone-task=bounded-retry`; targeted scenario S05.
- S05 permits replay only for a confirmed, transport-uncommitted, uncancelled, side-effect-safe request with remaining shared fault budget and a recovery owner. Its Evidence Map requires commit-boundary/shared-budget, provider-pool failover, no-owner terminal, new run identity, and bounded dispatch evidence.
- The implementation checklist therefore keeps candidate-path admission and its production streaming regression atomic. Final verification retains the shared package, race, vet, and local smoke evidence.

### Verification Context

- No separate verification handoff was supplied. Repository-native evidence comes from the active implementation, the archived plan/review pair, local test profiles, and the exact commands below.
- All verification runs in `/config/workspace/iop-s1` against the current checkout. Fake vLLM and provider-capacity smoke require no external account, credential, remote runner, device, or live provider.
- Fresh reviewer reruns passed all eleven prior commands. The focused temporary probe `go test -count=1 ./apps/edge/internal/openai -run '^TestReviewProbeResponsesStreamingStallRecovery$'` failed for both semantic modes: semantic false returned HTTP 502 JSON `recovery_failed`, semantic true returned an HTTP 200 SSE error terminal, and each executed only one dispatch. The temporary diagnostic file was removed after reproduction.
- Constraint: normalized `/v1/responses` still rejects public `stream=true`; only a selected provider tunnel may accept it. Recovery must preserve that candidate-dependent validation order.
- Confidence: high; the failing constructor call is on the only exact-replay admission path and the focused production handler evidence matches it.

### Test Coverage Gaps

- The existing matrix covers Chat/Responses, normalized/tunnel, and semantic false/true only with `stream=false`.
- It has no successful `stream=true` Responses tunnel replacement and therefore cannot detect candidate-independent normalized validation.
- Existing lower-level filter/controller/dispatcher, compatibility, package, race, and smoke tests remain regression evidence but do not close this product gap.

### Symbol References

- No symbol is renamed or removed. `newOpenAIResponsesRecoveryAdmissionBuilder`, `newResponsesDispatchContext`, and `newOpenAIResponsesPoolTunnelDispatchContext` retain their current call sites.

### Split Judgment

- Keep one plan. Candidate-path selection, request-context binding, and production terminal rendering are one recovery-attempt invariant; splitting the regression from the admission fix would leave no independently valid intermediate state.
- The dependent subtask `10+09_stall_recovery` requires predecessor index 09. It is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`.

### Scope Rationale

- Exclude Chat admission, Node watchdog, typed failure/wire mapping, Edge health overlay, provider candidate policy, and shared budget logic; fresh review found no defect in those owners.
- Exclude contract/spec edits because the active documents already require supported tunnel replay through S05; this follow-up restores implementation conformance without changing the public contract.
- Exclude new configuration, protobuf, and provider-profile changes. The defect is local ordering inside the Responses recovery admission adapter.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; finalizer mode `pair`.
- Build closures: scope/context/verification/evidence/ownership/decision are closed. Grade scores: scope=1, state=2, blast=2, evidence=2, verification=2; base and final route basis `grade-boundary`; lane `cloud`; grade `G09`; filename `PLAN-cloud-G09.md`.
- Build signals: `large_indivisible_context=false`; matched loop risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product`; count=4; `review_rework_count=5`; `evidence_integrity_failure=true`; risk and recovery boundaries matched without replacing the grade basis.
- Review closures are closed. Grade scores: scope=2, state=2, blast=2, evidence=2, verification=2; route basis `official-review`; lane `cloud`; grade `G10`; filename `CODE_REVIEW-cloud-G10.md`; adapter `codex`, model `gpt-5.6-sol`, reasoning `xhigh`.

## Implementation Checklist

- [ ] REVIEW_API-1 makes Responses exact replay candidate-dependent: tunnel replacements retain `stream=true`, normalized replacements perform the existing strict validation only in `PrepareRun`, and every admitted attempt binds the matching request context.
- [ ] REVIEW_API-2 extends the production stall matrix with semantic-false and semantic-true Responses streaming-tunnel recovery rows that prove replacement identity, provider avoidance, shared budget, close/cancel behavior, sanitized output, and exactly one successful SSE terminal.
- [ ] Run and record every exact final verification command separately after both implementation items are complete.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_API-1] Defer Responses normalized validation until candidate selection

**Problem:** `apps/edge/internal/openai/responses_stream_gate.go:1017-1029` decodes an exact replay body and immediately calls `newResponsesDispatchContext`. That constructor rejects `req.Stream` at `apps/edge/internal/openai/responses_handler.go:193-195`, before `SubmitProviderPool` can select a tunnel replacement that supports streaming.

**Solution:** Decode and retain the exact public `responsesRequest` without applying normalized-only validation. For a provider-pool replay, bind a tunnel-capable context containing the decoded request before dispatch; construct and bind the strict normalized context inside `PrepareRun` only when a normalized candidate is selected. Keep private continuation handling unchanged, preserve direct normalized replay validation, update `pool.Tunnel.Stream`/body/metadata from the admitted context, and ensure `state.set` always identifies the context for the selected replacement attempt.

Before:

```go
// responses_stream_gate.go:1023
var req responsesRequest
if err = decodeResponsesRequest(json.NewDecoder(bytes.NewReader(body)), &req); err == nil {
    dc, err = server.newResponsesDispatchContext(initial.responsesRequestContext, req)
}
```

After:

```go
// Preserve the decoded public replay for provider-path-specific admission.
// A tunnel attempt binds its request context directly; PrepareRun alone calls
// newResponsesDispatchContext and therefore owns normalized-only validation.
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/responses_stream_gate.go`: separate exact public replay decoding from normalized construction, bind per-path attempt state, and preserve existing continuation/direct behavior.

**Test Strategy:** A regression test is mandatory and belongs to REVIEW_API-2 in `stream_gate_stall_recovery_test.go`. Existing non-streaming normalized/tunnel and continuation tests remain unchanged regression coverage.

**Verification:** The focused `TestOpenAIStallRecoveryMatrix/recover/responses/provider_tunnel/stream=true` command must execute two replacement products and pass in both semantic modes.

### [REVIEW_API-2] Add the missing Responses streaming-tunnel recovery product

**Problem:** `apps/edge/internal/openai/stream_gate_stall_recovery_test.go:241-275` labels its loop as every endpoint/path/semantic recovery product but calls `runStallMatrixHandler(..., false, ...)` at line 251. The named green test never sends a streaming Responses request.

**Solution:** Replace the implicit endpoint/path loop input with explicit supported recovery cases that include Responses provider-tunnel `stream=true` for semantic false and true. Emit a valid Responses SSE success sequence for the replacement tunnel. Name the rows `recover/responses/provider_tunnel/stream=true/semantic=<bool>` and assert exactly two dispatches, failed-provider avoidance without unsafe fallback, distinct run identities, the single shared-budget replacement, exactly one close per transport, no duplicate cancel, no raw stalled payload, exactly one `response.completed`, and one `[DONE]`.

Before:

```go
// stream_gate_stall_recovery_test.go:251
w := runStallMatrixHandler(t, stallMatrixServer(service, semantic, 1), endpoint, false, nil)
```

After:

```go
// Each explicit case carries endpoint, provider path, stream mode, and semantic
// mode; supported streaming tunnel cases emit and assert Responses SSE.
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: add explicit stream-mode cases, a deterministic streaming Responses tunnel success fixture, and lifecycle/terminal assertions.
- [ ] `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G10.md`: record implementation decisions and complete raw output for every command.

**Test Strategy:** Extend `TestOpenAIStallRecoveryMatrix` rather than adding a shallow helper-only test. Use the existing scripted provider-pool service and typed confirmed-stall frames so the regression traverses `handleResponses`, the Responses runtime, recovery admission, replacement dispatch, and the public release sink.

**Verification:** Run the focused matrix subtest ten times, then the entire matrix ten times. Both commands must pass without a recovery error terminal.

## Dependencies and Execution Order

1. Predecessor `09+08_retry_candidate_policy` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/complete.log`.
2. Complete REVIEW_API-1 candidate-dependent admission.
3. Complete REVIEW_API-2 against the corrected admission path.
4. Run every final verification command and fill the active review evidence.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/openai/responses_stream_gate.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | REVIEW_API-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G10.md` | REVIEW_API-2 |

## Final Verification

1. `go test -count=10 ./apps/edge/internal/openai -run 'TestOpenAIStallRecoveryMatrix/recover/responses/provider_tunnel/stream=true'` — PASS both semantic modes and execute exactly one safe replacement per row.
2. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecoveryMatrix$'` — PASS every production endpoint/path/config/recovery/terminal row.
3. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAISemanticGateDisabledCompatibility$'` — PASS every runtime-owned endpoint compatibility row.
4. `go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)$'` — PASS every iteration.
5. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS.
6. `go test -race -count=3 ./apps/edge/internal/service` — PASS with no race report.
7. `go test -race -count=3 ./apps/edge/internal/openai` — PASS with no race report.
8. `go vet ./packages/go/streamgate ./apps/edge/internal/service ./apps/edge/internal/openai ./apps/edge/internal/controlplane` — no diagnostics.
9. `./scripts/e2e-smoke.sh` — PASS.
10. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — PASS.
11. `./scripts/e2e-provider-capacity-smoke.sh` — PASS.
12. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
