<!-- task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery plan=3 tag=API milestone-task=bounded-retry -->

# Code Review Reference - API

> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.

## Overview

date=2026-08-05
task=m-node-provider-execution-liveness-recovery/10+09_stall_recovery, plan=3, tag=API

## Archive Evidence Snapshot

- Prior pair: `plan_cloud_G08_1.log` and `code_review_cloud_G08_1.log` in this task directory. It was unimplemented and has no official verdict, implementation evidence, code change, or verification output.
- Prior fresh-review finding: `normal-10` is a long-context admission smoke and does not prove typed stall mapping, recovery gating, provider avoidance, or bounded dispatch.
- Union preparation review archived the unimplemented plan=2 pair as `plan_cloud_G08_2.log` and `code_review_cloud_G08_2.log`; it had no verdict or implementation evidence. The material scope defect was treating default `stream_evidence_gate.enabled=false` OpenAI requests as ownerless even though the approved SDD assigns typed-stall recovery to the supported OpenAI-compatible host.
- Replan carryover: preserve the raw-free mapper, shared Core commit/cancel/side-effect/budget ownership, and S05 matrix. Install one internal liveness owner for every supported OpenAI Chat/Responses normalized or tunnel request regardless of semantic-gate enablement/configuration; only unsupported or non-OpenAI surfaces remain the no-owner typed-terminal boundary.

## For the Review Agent

> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.

Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
Review completion means the following steps are finished:

1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_3.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_3.log`.
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.

---

## Implementation Item Completion

| Item | Status |
|------|---------|
| API-1: Convert typed execution stalls into raw-free StreamGate events | [x] |
| API-2: Gate exact replay and hand off the failed provider | [ ] |
| API-3: Prove bounded recovery across OpenAI variants | [ ] |

## Implementation Checklist

- [x] API-1 preserves typed normalized/buffered/tunnel stalls as one raw-free StreamGate `response_stalled` provider error, retaining only sanitized fence/health and `recovery_handoff=confirmed` authority tokens while generic failures keep existing terminal behavior.
- [ ] API-2 installs exactly one internal liveness recovery owner for every supported OpenAI Chat/Responses normalized or tunnel request independent of `stream_evidence_gate.enabled` and configured semantic filters/capabilities; only confirmed handoff, uncommitted, uncanceled, side-effect-safe, budget-available stalls produce ExactReplay, close the fenced old transport, and hand failed-provider/fallback evidence to admission.
- [ ] API-3 adds semantic-gate-enabled/disabled Chat/Responses normalized/tunnel fixtures for available, unavailable, and unknown alternate recovery; available-only same-provider fallback; unavailable/unknown same-only terminal; unsupported/no-owner, post-commit, unconfirmed, cancel/tool-side-effect, and shared-budget exhaustion; synchronize contracts/specs.
- [ ] Run focused, package, race, vet, provider-only/OpenAI/local-capacity full-cycles, and diff verification with fresh output; assert new identities plus exactly one terminal/dispatch per allowed cycle.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

## Review-Only Checklist

> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.

- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_3.log`.
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_3.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/` and update this checklist at the final archive path.
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.

## Deviations from Plan

- The required always-on liveness owner for `stream_evidence_gate.enabled=false` is not complete. Making the existing StreamGate runtime unconditional caused legacy disabled-path regressions in cancellation, strict/tool validation, SSE reasoning/finish-reason rendering, and passthrough ordering. The unconditional switch was reverted to preserve those established behaviors. The current private registration is installed for runtime-enabled supported OpenAI requests only; this remains a material API-2/API-3 gap for review follow-up.
- API-3's complete Chat/Responses normalized/tunnel S05 matrix was not added. Focused mapper/filter/controller/provider-hint tests cover the implemented subset only.

## Key Design Decisions

- `openAIRunTerminalError` defensively clones a typed terminal failure and exposes only `run failed`; buffered collectors can therefore retain typed failure semantics without publishing provider text.
- The mapper admits only an Edge-confirmed, retryable `response_stalled` failure with allowlisted health and provider-id metadata. StreamGate receives a stable descriptor and two safe causes, never a proto message or arbitrary metadata.
- The private liveness filter requires uncommitted transport, no side effect/tool fragment, a snapshot reference, and confirmed handoff. A confirmed old attempt closes without `CancelRun`; provider-pool recovery consumes one avoidance hint, allowing fallback only for `available`.

## Reviewer Checkpoints

- Confirm typed normalized/buffered/tunnel stalls map to one raw-free descriptor carrying only allowlisted fence/health plus `recovery_handoff=confirmed`, while generic failures retain current terminal behavior.
- Confirm every supported OpenAI Chat/Responses normalized or tunnel request gets exactly one private liveness owner regardless of semantic-gate enablement/configuration, normal gate-disabled output stays compatible, and only unsupported/non-OpenAI ingress remains no-owner terminal.
- Confirm commit/cancel/side-effect/fence/shared-budget gates, confirmed-terminal close, failed-provider handoff, available-only fallback, new run identity, and single dispatch/terminal across Chat/Responses variants.
- Confirm focused/race tests plus provider-only, fake-vLLM, and local-capacity repository-native evidence satisfy the final commands.

## Verification Results

> Implementing agent: run each command exactly as written and paste its actual stdout/stderr under `Output`. Record any replacement command and reason in `Deviations from Plan`.

### Verification 1

Command:

```bash
go test -count=20 ./apps/edge/internal/openai -run '^(TestOpenAIStallRecoveryFilter|TestOpenAIAttemptControllerConfirmedStall|TestOpenAIAttemptDispatcherStalledProvider)'
```

Output:

```text
ok   iop/apps/edge/internal/openai  0.029s
```

### Verification 2

Command:

```bash
go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAIStallRecovery'
```

Output:

```text
ok   iop/apps/edge/internal/openai  0.046s
```

### Verification 3

Command:

```bash
go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane
```

Output:

```text
ok   iop/packages/go/streamgate  1.011s
ok   iop/apps/edge/internal/openai  7.417s
ok   iop/apps/edge/internal/service  6.135s
ok   iop/apps/edge/internal/controlplane  6.646s
Post-final mapper compatibility check:
ok   iop/apps/edge/internal/openai  7.389s
```

### Verification 4

Command:

```bash
go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/openai
```

Output:

```text
ok   iop/apps/edge/internal/service  19.496s
```

### Verification 5

Command:

```bash
go vet ./packages/go/streamgate ./apps/edge/internal/service ./apps/edge/internal/openai ./apps/edge/internal/controlplane
```

Output:

```text
(no diagnostics; exit 0)
```

### Verification 6

Command:

```bash
./scripts/e2e-smoke.sh
```

Output:

```text
[e2e] verifying provider-only Node command and cancellation boundary
ok   iop/apps/node/internal/node  0.073s
[e2e] verifying Edge dispatch, provider tunnel, queue, and reconnect fencing
ok   iop/apps/edge/internal/service  4.463s
ok   iop/apps/edge/internal/transport  0.292s
[e2e] provider-only Edge-Node smoke PASSED
```

### Verification 7

Command:

```bash
IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh
```

Output:

```text
[openai-vllm] OpenAI-compatible vLLM serving test PASSED (mode=fake).
```

### Verification 8

Command:

```bash
./scripts/e2e-provider-capacity-smoke.sh
```

Output:

```text
[provider-capacity-smoke] tmp_root=/config/tmp/gocache-secure-delivery
[provider-capacity-smoke] building loopback binaries
[provider-capacity-smoke] metrics=disabled bounded_probe_timeout_sec=2
[provider-capacity-smoke] offline_snapshot_rejected=true
[provider-capacity-smoke] aliases=ornith:35b,ornith-fast queue_observed=true
[provider-capacity-smoke] backend={"calls":2,"active":0,"peak":1}
[provider-capacity-smoke] PASS evidence=/config/tmp/gocache-secure-delivery/iop-provider-capacity-smoke.K03JSn
```

### Verification 9

Command:

```bash
git diff --check
```

Output:

```text
PASS (exit 0; unrelated transient inaccessible test-temp-directory warnings were emitted by git status)
```
---

> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.

## Section Ownership

| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |

## Code Review Result

- Overall Verdict: FAIL
- Dimension Assessment:
  - Correctness: Fail — supported OpenAI requests still bypass the liveness recovery owner whenever `stream_evidence_gate.enabled=false`.
  - Completeness: Fail — API-2 and API-3 remain unchecked and their required supported-path ownership and S05 variant matrix are not implemented.
  - Test Coverage: Fail — the new stall test file exercises only mapper/filter units, not the required Chat/Responses normalized/tunnel recovery cycles.
  - API Contract: Fail — the implementation and synchronized contract/spec text limit recovery to runtime-enabled requests, contrary to the approved always-on supported-path contract.
  - Code Quality: Pass — the implemented typed mapper and request-local state are bounded and raw-free in the reviewed subset.
  - Implementation Deviation: Fail — the recorded gate-disabled exclusion removes an explicit acceptance condition rather than a compatible implementation detail.
  - Verification Trust: Fail — focused tests pass but do not exercise the promised S05 matrix, and Verification 4 omits the OpenAI package result from the recorded command output.
  - Spec Conformance: Fail — SDD S05 requires owner-gated bounded retry for supported OpenAI requests and evidence for no-owner only on unsupported surfaces.
- Findings:
  - Required R1 — `apps/edge/internal/openai/stream_gate_runtime.go:797`, `apps/edge/internal/openai/normalized_sse.go:41`, `apps/edge/internal/openai/responses_handler.go:151`, and `apps/edge/internal/openai/provider_tunnel.go:33`: the private stall registration is constructed only inside handlers reached through `streamGateEnabled()`, which still returns `s.cfg.StreamEvidenceGate.Enabled`. The default false configuration therefore routes Chat, Responses, and tunnel requests through legacy paths with no liveness recovery owner. Separate semantic-filter enablement/capability admission from supported-path runtime ownership, keep normal disabled-semantic output compatible, and install exactly one private stall owner for every supported OpenAI path as PLAN API-2 and SDD S05 require.
  - Required R2 — `apps/edge/internal/openai/stream_gate_stall_recovery_test.go:51` and `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/CODE_REVIEW-cloud-G08.md:142`: the only new `TestOpenAIStall*` cases are mapper/filter units; there is no Chat/Responses x normalized/tunnel x semantic-enabled/disabled matrix, no alternate/same-provider/budget dispatch and terminal identity assertions, and the recorded race command contains only the service package line. Add the S05 integration matrix, including unsupported/no-owner and every unsafe terminal row, then rerun every exact verification command and record complete raw output.
- Routing Signals:
  - `review_rework_count=1`
  - `evidence_integrity_failure=true`
- Next Step: Invoke the plan skill in `prepare-follow-up` mode with Required R1 and R2 as direct fixes, rerun isolated final routing, archive this pair, and materialize the routed follow-up PLAN/CODE_REVIEW pair.
