<!-- task=m-iop-hot-path-one-shot-execution/19+17,18_observation_lifecycle plan=2 tag=REVIEW_API milestone-task=route-observability -->

# Complete and prove the raw-free Hot Path observation lifecycle

## For the Implementing Agent

Implement only this plan. Run every verification command exactly as written, paste actual stdout/stderr into `CODE_REVIEW-cloud-G09.md`, complete its implementation-owned sections, and leave both active files in place for official review. If blocked, record the exact blocker, attempted command/output, and resume condition only in those implementation-owned evidence fields. Do not ask the user, create control-plane stop files, classify the next state, archive task files, or write `complete.log`.

## Background

Plan 1 added the closed observation schema and several lifecycle call sites, but production still discards the log projection, failed provider attempts are absent, direct continuations duplicate admission, and light terminal metrics can precede the endpoint write that determines the winning disposition. The follow-up keeps the predecessor schema and makes the production path and its evidence satisfy SDD S15.

## Archive Evidence Snapshot

- `agent-task/m-iop-hot-path-one-shot-execution/19+17,18_observation_lifecycle/plan_local_G08_1.log` and `agent-task/m-iop-hot-path-one-shot-execution/19+17,18_observation_lifecycle/code_review_cloud_G09_1.log` contain plan 1, its implementation evidence, and the FAIL verdict requiring a production zap sink, failed-attempt coverage, one logical admission, post-write terminal ownership, and non-vacuous actual-path evidence.
- `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/17+14,15,16_endpoint_error_matrix/complete.log` and `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/18+17_observation_schema/complete.log` satisfy directory predecessors 17 and 18.

## Analysis

### Files Read

- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-hot-path-one-shot-execution.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-contract/outer/openai-compatible-api.md`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/hot_path_observation.go`
- `apps/edge/internal/openai/hot_path_metrics.go`
- `apps/edge/internal/openai/hot_path_dispatch.go`
- `apps/edge/internal/openai/hot_path_direct.go`
- `apps/edge/internal/openai/hot_path_light.go`
- `apps/edge/internal/openai/hot_path_cleanup.go`
- `apps/edge/internal/openai/request_identity_ingress.go`
- `apps/edge/internal/openai/request_coordinator_ttl.go`
- `apps/edge/internal/openai/hot_path_terminal_control.go`
- `apps/edge/internal/openai/hot_path_observation_test.go`
- `apps/edge/internal/openai/hot_path_light_test.go`
- `apps/edge/internal/openai/cancellation_routes_test.go`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`

### SDD Criteria

- Approved SDD S15 requires raw-free, bounded Hot Path metric/log projections joined by request, preset, mode, stage, attempt, transition, cleanup/orphan, and final outcome.
- The SDD Evidence Map requires an allowlist test on actual direct/light and failure paths. S16 owns external provider smoke, so no external runner or credential is required here.

### Verification Context

- No verification handoff was supplied. Repository-native fallback evidence came from the active SDD/spec/contract, the affected Go call graph, the two predecessor `complete.log` files, and fresh local commands.
- Current host: `go version go1.26.2 linux/arm64`; repository commands run from `/config/workspace/iop-s0`.
- Fresh baseline passed: targeted `go test -race -count=1` (`ok ... 6.668s`), common regression (`streamgate 2.882s`, `config 4.631s`, `openai 34.590s`, `service 8.768s`), `git diff --check`, and empty `gofmt -d` output.
- Constraint: the worktree contains unrelated sibling-task changes. Modify only the exact files in this plan and do not clean or rewrite unrelated state.
- Gap: no external/full-cycle run is required for S15; provider-backed smoke remains in S16. Confidence is high because all pass criteria use deterministic in-process handlers, fake providers, zap capture, Prometheus collector deltas, and `-race`.

### Test Coverage Gaps

- Existing lifecycle tests install a recording observer explicitly, so they cannot detect that `NewServer` installs a production noop.
- The current sentinel list is not inserted into actual prompt, output, tool arguments/results, headers, credentials, or provider errors.
- `at least one` assertions cannot prove exact admission, stage attempt, transition, cleanup, or terminal cardinality; direct terminal tests bypass the handler and dispatch path.
- No actual-path row observes provider error, timeout, caller cancellation, endpoint write failure, or the observer-failure metric delta.

### Symbol References

- No public API rename or removal is planned.
- Internal call sites to update are `NewServer`, `emitHotPathObservation`, `dispatchPresetTurn`, both initial/continuation ingress branches, `runHotPathLightStage`, review/repair/cleanup transitions, `writeHotPathTerminal`, and TTL orphan logging.

### Split Judgment

- Do not split. Production sink installation, lifecycle ownership, and exact captured evidence are one correctness invariant: a separate test-only or sink-only child could pass while the real request path still drops or duplicates events.

### Scope Rationale

- Include only production observation wiring, exact logical-request/stage/transition/terminal ownership, raw-free TTL logging, and deterministic regressions.
- Exclude dashboard/storage backends, raw payload hashes, schema redesign unrelated to a required outcome field, external provider smoke, and other milestone tasks.

### Final Routing

- `evaluation_mode=isolated-reassessment`; all build/review closure dimensions are true and no capability gap exists.
- Finalizer: `finalize-task-policy.sh pair`.
- Build scores `2/2/1/2/2` = G09; matched risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4); `large_indivisible_context=false`; `review_rework_count=1`; `evidence_integrity_failure=true`.
- Finalizer output: build `grade-boundary` -> `PLAN-cloud-G09.md`; review `official-review` -> `CODE_REVIEW-cloud-G09.md` with Codex `gpt-5.6-sol` xhigh.

## Implementation Checklist

- [ ] [REVIEW_API-1] Install the bounded zap observer in production and account for isolated observer failures without exposing non-allowlisted fields.
- [ ] [REVIEW_API-2] Make admission, failed stage attempts, repair/cleanup transitions, and the post-write terminal disposition exact across direct and light lifecycles.
- [ ] [REVIEW_API-3] Replace vacuous evidence with exact actual-handler traces, truly seeded raw/secret fixtures, and exact metric/logger deltas for success and failure rows.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_API-1] Production bounded logger and failure accounting

**Problem:** `apps/edge/internal/openai/server.go:112-118` installs `hotPathNoopObserver`, while `apps/edge/internal/openai/hot_path_observation.go:484-494` says the zap projection is deferred to this lifecycle child. All production lifecycle logs are therefore discarded. `hotPathMetrics.recordObserverFailure` exists, but `emitHotPathObservation` never calls it.

**Solution:** Add a zap-backed `hotPathObserver` that emits one fixed message with only `logProjectionKeys()` fields after validation. Install it in `NewServer`. At the server seam, compose the built-in `observerFailures` increment with the optional diagnostic hook so observer errors and panics remain best effort and never affect request, cancellation, or cleanup behavior.

Before (`server.go:112-118`, `hot_path_observation.go:484-494`):

```go
s := &Server{
    cfg: cfg, service: svc, logger: logger, obsSink: newZapFilterObservationSink(logger),
    // ...
    hotPathObserver: hotPathNoopObserver{},
}

// hotPathBoundedObserver is a placeholder for the production bounded logger
// that will be wired in a later child.
```

After:

```go
s := &Server{
    cfg: cfg, service: svc, logger: logger, obsSink: newZapFilterObservationSink(logger),
    // ...
    hotPathObserver: newZapHotPathObserver(logger),
}

failureHook := func(projection hotPathLogProjection, err error) {
    initHotPathMetrics().recordObserverFailure(s.edgeIDValue())
    invokeHotPathObserverFailureHookSafely(hook, projection, err)
}
```

The zap observer must not emit raw errors, prompt/output/tool values, provider/credential/header data, workspace paths, or dynamic keys. A nil logger remains safe through `zap.NewNop()`.

**Modified Files and Checklist:**

- [ ] Modify `apps/edge/internal/openai/server.go` to install the production observer and compose built-in failure accounting with the optional hook.
- [ ] Modify `apps/edge/internal/openai/hot_path_observation.go` to implement the fixed-message, fixed-key zap observer and retain validation/failure isolation.

**Test Strategy:** Write tests in `apps/edge/internal/openai/hot_path_observation_test.go`: `TestHotPathObservationLifecycle_ProductionZapObserver` drives a real handler from `NewServer` and asserts one allowlisted zap entry; `TestHotPathObservationLifecycle_ObserverFailureMetric` uses erroring and panicking observers and asserts unchanged responses plus exact `observerFailures` deltas.

**Verification:** run the targeted command in Final Verification; both tests pass under `-race` and captured zap fields equal the allowlist.

### [REVIEW_API-2] Exact lifecycle ownership and final disposition

**Problem:** `apps/edge/internal/openai/hot_path_dispatch.go:1110-1113` observes admission every time `dispatchPresetTurn` runs, including a direct tool continuation for the same logical request. `apps/edge/internal/openai/hot_path_light.go:868-880` returns on dispatch error before emitting the attempt. `apps/edge/internal/openai/hot_path_cleanup.go:351-404` records cleanup/terminal before the endpoint write, so a caller-write failure can leave a false success terminal. Only the local-to-review transition is emitted; review repair and cleanup handoffs are absent. The legacy TTL log at `request_coordinator_ttl.go:98-120` also emits non-allowlisted workspace/state fields beside the closed observer.

**Solution:** Mark only newly created Chat/Messages admissions in ingress metadata and let `dispatchPresetTurn` emit the route decision only for that marker; continuations retain correlation IDs but do not re-admit. Move the light stage observation into a single post-attempt path that always records duration, stage kind, attempt bucket, and a normalized success/error disposition without raw causes. Emit bounded transition projections when review enters repair and when cleanup becomes responsible. For logical terminals, write the endpoint response first, resolve the winning outer disposition (including caller-write cancellation), then emit exactly one terminal metric/log; cleanup remains ordered before terminal. Remove the duplicate legacy TTL logger payload and retain the closed orphan observer as the sole orphan log/metric owner.

Before (`hot_path_dispatch.go:1110-1113`, `hot_path_light.go:868-880`, `hot_path_cleanup.go:356-404`):

```go
s.observeHotPathDispatch(r.Context(), hotPathNormalizeMode(string(decision.Mode)), "", requestID, stageID, preset.ID)

output, correlation, err := s.dispatchHotPathStage(r.Context(), r, snapshot, outer)
if err != nil {
    return s.writeHotPathPrimaryError(/* ... */)
}
s.observeHotPathStage(/* success-only */)

s.observeHotPathCleanup(/* ... */)
s.observeHotPathTerminal(/* pre-write disposition */)
return s.writeHotPathStageResponse(/* ... */)
```

After:

```go
if isInitialHotPathAdmission(runMeta) {
    s.observeHotPathDispatch(/* first logical route only */)
}

output, correlation, dispatchErr := s.dispatchHotPathStage(/* ... */)
s.observeHotPathStage(/* normalized result for success or dispatchErr */)
if dispatchErr != nil {
    return s.writeHotPathPrimaryError(/* ... */)
}

writeErr := s.writeHotPathStageResponse(/* ... */)
winning := resolveHotPathObservedDisposition(outer, intent.Disposition, writeErr)
s.observeHotPathTerminal(/* winning post-write disposition */)
return writeErr
```

Keep admission and terminal guards request-scoped and concurrency-safe; do not infer ownership from response contents. All new outcome values must pass existing closed normalizers before logging or labeling.

**Modified Files and Checklist:**

- [ ] Modify `apps/edge/internal/openai/request_identity_ingress.go` to distinguish initial admissions from direct/light continuations for observation ownership in both protocols.
- [ ] Modify `apps/edge/internal/openai/hot_path_dispatch.go` to gate dispatch observation to the initial logical route decision.
- [ ] Modify `apps/edge/internal/openai/hot_path_light.go` to observe every provider attempt and explicit local/review/repair transitions with closed outcomes.
- [ ] Modify `apps/edge/internal/openai/hot_path_cleanup.go` to emit cleanup/cleanup-transition evidence and the winning terminal only after the endpoint write result is known.
- [ ] Modify `apps/edge/internal/openai/request_coordinator_ttl.go` to remove the parallel non-allowlisted orphan payload and keep the closed orphan observer as the single log/metric owner.

**Test Strategy:** Write regressions in `apps/edge/internal/openai/hot_path_observation_test.go` for both OpenAI and Anthropic handlers: direct tool continuation has one admission and one final terminal across two HTTP turns; light pass/repair have exact stage/transition sequences; provider error, timeout, and caller cancel each have one failed attempt with normalized outcome; cleanup and a failing writer preserve cleanup-before-terminal and select the winning terminal exactly once; TTL produces one closed orphan and no terminal.

**Verification:** run the targeted command in Final Verification; every scenario's ordered projection slice and exact collector delta match its table.

### [REVIEW_API-3] Non-vacuous actual-path evidence

**Problem:** `apps/edge/internal/openai/hot_path_observation_test.go:1455-1462` declares sentinels but does not inject them, `:1580-1595` accepts non-exact event counts, `:1811-1840` bypasses the HTTP handler for direct terminals, and the dispatch-rejection/metric tests do not prove the claimed exact metric deltas.

**Solution:** Extend the existing scripted handler fixtures with unique raw values in actual prompt, model output/reasoning, tool arguments/result, authorization/API-key headers, route credential/target, and provider error. Capture both the production zap core and observer projections. Compare ordered typed projection slices, not lower bounds; assert exact before/after values for dispatch, stage, terminal, cleanup, orphan, and observer-failure collectors using unique bounded labels. Assert the serialized zap entries contain none of the seeded values and no key outside `logProjectionKeys()`.

Before (`hot_path_observation_test.go:1455-1462`, `:1580-1595`):

```go
var hotPathRawSentinels = []string{"prompt", "output", "tool_args", "tool_result", /* ... */}

if counts[hotPathEventClassDispatch] == 0 { /* ... */ }
if counts[hotPathEventClassStage] == 0 { /* ... */ }
if counts[hotPathEventClassLight] == 0 { /* ... */ }
```

After:

```go
seed := newHotPathRawSeed(t) // inserted into request, provider, tool, header, credential, and error fixtures
got := captureActualHotPathLifecycle(t, seed, scenario)
assertHotPathTraceEqual(t, scenario.wantOrderedProjections, projectTrace(got.projections))
assertExactHotPathMetricDeltas(t, scenario.wantMetricDeltas, got.before, got.after)
assertSeedAbsentFromZapAndProjections(t, seed, got.logs, got.projections)
```

Use repository test helpers rather than a new framework. Do not weaken exact assertions to `>=`, `<=`, or `at least one` except where Prometheus process-global pre-existing series are isolated by a before/after value for one exact label set.

**Modified Files and Checklist:**

- [ ] Modify `apps/edge/internal/openai/hot_path_observation_test.go` with seeded actual-handler fixtures, exact scenario tables, zap allowlist checks, response-failure writer, and exact metric deltas.
- [ ] Fill actual implementation notes, deviations, decisions, and raw command output in `agent-task/m-iop-hot-path-one-shot-execution/19+17,18_observation_lifecycle/CODE_REVIEW-cloud-G09.md`.

**Test Strategy:** Add/replace `TestHotPathObservationLifecycle_ProductionZapObserver`, `_LightPass`, `_LightRepair`, `_ProviderError`, `_Timeout`, `_CallerCancel`, `_CleanupFailure`, `_CallerWriteFailure`, `_OrphanTTL`, `_DirectToolContinuation`, and `_ObserverFailureMetric`; every row runs both protocols where the endpoint contract applies.

**Verification:** run all Final Verification commands; fresh race tests pass, formatting output is empty, and the diff contains no whitespace errors.

## Dependencies and Execution Order

1. Directory predecessor 17 is satisfied by `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/17+14,15,16_endpoint_error_matrix/complete.log`.
2. Directory predecessor 18 is satisfied by `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/18+17_observation_schema/complete.log`.
3. Implement REVIEW_API-1, then REVIEW_API-2, then REVIEW_API-3 so the tests exercise the final production seam and ownership model.

## Modified Files Summary

| File | Item |
|---|---|
| `apps/edge/internal/openai/server.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/hot_path_observation.go` | REVIEW_API-1, REVIEW_API-2 |
| `apps/edge/internal/openai/request_identity_ingress.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_dispatch.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_light.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_cleanup.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/request_coordinator_ttl.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_observation_test.go` | REVIEW_API-1, REVIEW_API-2, REVIEW_API-3 |
| `agent-task/m-iop-hot-path-one-shot-execution/19+17,18_observation_lifecycle/CODE_REVIEW-cloud-G09.md` | REVIEW_API-3 |

## Final Verification

```bash
go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObservationLifecycle|ObservationRejectsRawValues|MetricLabels|ObserverFailureIsolation|EndpointTerminalMatrix|Cleanup)'
go test -race -count=1 ./packages/go/streamgate ./packages/go/config ./apps/edge/internal/openai ./apps/edge/internal/service
gofmt -d apps/edge/internal/openai/server.go apps/edge/internal/openai/hot_path_observation.go apps/edge/internal/openai/request_identity_ingress.go apps/edge/internal/openai/hot_path_dispatch.go apps/edge/internal/openai/hot_path_light.go apps/edge/internal/openai/hot_path_cleanup.go apps/edge/internal/openai/request_coordinator_ttl.go apps/edge/internal/openai/hot_path_observation_test.go
git diff --check
```

Expected: every command exits 0; both `go test` commands are fresh and race-clean; `gofmt -d` and `git diff --check` print no output; actual handler logs use only the fixed allowlist and contain no seeded raw/secret values; each scenario has exactly one logical admission, exact ordered attempts/transitions, and one winning terminal or orphan responsibility as applicable.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
