<!-- task=m-iop-hot-path-one-shot-execution/18+17_observation_schema plan=3 tag=REVIEW_API milestone-task=route-observability -->

# Code Review Reference - REVIEW_API

> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.

## Overview

date=2026-08-04
task=m-iop-hot-path-one-shot-execution/18+17_observation_schema, plan=3, tag=REVIEW_API

## Archive Evidence Snapshot

- The prior pair will be archived as `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/plan_local_G06_2.log` and `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/code_review_cloud_G07_2.log` with verdict `FAIL`.
- Required findings: collector entry points accept arbitrary typed strings and discard route/cleanup/orphan values; the log projection omits S15 preset/attempt/outcome evidence and the bounded observer delegates without validation; the server has no production safe emission seam and a panicking failure hook escapes.
- Fresh targeted and SDD-common race commands passed, but a focused reviewer probe failed with `unknown metric values reached the collector: got 1, want 0` and `failure hook panic escaped request isolation: failure hook failed`; `evidence_integrity_failure=true`.
- Milestone carryover remains `milestone-task=route-observability`, SDD S15, raw-free log/metric allowlist evidence.

## For the Review Agent

> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.

Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
Review completion means the following steps are finished:

1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_3.log` and `PLAN-cloud-G06.md` → `plan_cloud_G06_3.log`.
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-hot-path-one-shot-execution/18+17_observation_schema/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.

---

## Implementation Item Completion

| Item | Status |
|------|---------|
| REVIEW_API-1 Close projection and collector inputs | [x] |
| REVIEW_API-2 Make the server emission seam failure-proof | [x] |

## Implementation Checklist

- [x] [REVIEW_API-1] Enforce a complete S15 log/metric projection at production entry points, reject unknown typed-string values, and preserve only bounded log correlation identifiers.
- [x] [REVIEW_API-2] Add one server-owned safe emission seam that isolates sink and failure-hook errors/panics while preserving Stream Gate observation ownership, with regression tests through the real seam.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

## Review-Only Checklist

> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.

- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G06_3.log`.
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G06_3.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/` to `agent-task/archive/YYYY/MM/m-iop-hot-path-one-shot-execution/18+17_observation_schema/` and update this checklist at the final archive path.
- [x] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-hot-path-one-shot-execution/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.

## Deviations from Plan

- **`apps/edge/...` smoke exit 1 is environmental, not a code defect.** The full smoke command exits 1 solely because `apps/edge/internal/bootstrap::TestActualNodeReconnectReadyPumpsQueuedWaiterExactlyOnce` builds `iop-node` into `TMPDIR=/tmp` via `t.TempDir()` and then `exec.Command(binary,...).Start()`. The dispatcher sandbox mounts `/tmp` as `noexec` (`tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)`), so the freshly-built binary cannot be exec'd (`fork/exec /tmp/.../iop-node: permission denied`). This package is untouched by the plan (all target files are in `apps/edge/internal/openai`), the regression is reproducible only in this sandbox, and the actual target package `apps/edge/internal/openai` passes fully under `-race`. No code change is made or warranted for this; it is left as an environment-level limitation.
- **`TMPDIR=/tmp` retained for determinism per plan.** The plan mandates `TMPDIR=/tmp` and `-count=1` for fresh deterministic evidence. That same choice is what surfaces the `noexec` sandbox limitation above; the focused/SDD-common regressions that do not exec binaries still pass cleanly under `TMPDIR=/tmp`.
- **Reviewer applied two non-behavioral repairs within the planned file set.** (1) `hot_path_observation_test.go`'s `TestHotPathMetricProjectionBoundary` was strengthened with `testutil.CollectAndCount` gathered-series delta assertions so the plan's stated `invalid casts produce no series` and `distinct route/cleanup/orphan series` properties are actually proven rather than only smoke-recorded; the production `record*`/normalization code is unchanged and still exits 0. (2) The stale `hotPathBoundedObserver` type doc comment was corrected to state that `Emit` validates the projection through `hotPathValidateLogProjection` and that failure isolation is provided by `hotPathSafeObserver` at the server seam. Neither changes the implementation scope, planned file set, or verification command set.
- No deviations to the implementation scope, file set, or verification command set beyond the environmental note above.

## Key Design Decisions

- **Closed-schema normalization at every collector entry point (REVIEW_API-1).** Each `hotPathMetrics.record*` method calls the matching `hotPathNormalize*` helper on its typed-string arguments and returns early when normalization yields `""`. This means a direct cast like `hotPathMode("unknown")` can no longer reach `WithLabelValues`, closing the prior `unknown metric values reached the collector: got 1, want 0` finding. `recordDispatch` now carries the closed `hot_path_reason` label, `recordCleanup` carries `hot_path_cleanup_outcome`, and `recordOrphan` carries `hot_path_orphan_outcome`, so route reason and cleanup/orphan outcomes produce distinct metric series instead of being discarded.
- **Metric-specific fixed label sets instead of one mega-vector.** Each collector declares only the labels it needs (`stageDuration`: edge/mode/stage/attempt/duration_bucket; `terminalCounter`: edge/mode/disposition; `usageCounter`: edge/mode/usage_bucket; `dispatchCounter`: edge/mode/reason; `cleanupCounter`: edge/cleanup_outcome; `orphanCounter`: edge/orphan_outcome). High-cardinality correlation ids (request/stage/call) remain log-only in `hotPathLogProjection`. The shared `hotPathMetricLabelNames` allowlist plus `hotPathMetricLabelCardinalityBudget` keep the worst-case series count under 1,000,000.
- **Bounded observer validates before delegating (REVIEW_API-1).** `hotPathBoundedObserver.Emit` runs the projection through `hotPathValidateLogProjection`; on an invalid enum or secret-sentinel field it returns `nil` without ever calling the inner sink, so invalid projections cannot reach a captured sink and a secret sentinel cannot leak. The complete S15 log projection now carries `PresetID`, `AttemptBucket`, `CleanupOutcome`, and `OrphanOutcome`.
- **Server-owned safe emission seam (REVIEW_API-2).** `Server.emitHotPathObservation` is the single production path: it snapshots observer + hook under `RLock` via `hotPathObservationSnapshot`, then runs `hotPathSafeObserver{inner: &hotPathBoundedObserver{inner: observer}, onFailure: hook}`. Both the sink error/panic and the failure-hook panic are recovered. The hook is invoked inside its own `defer recover()` block, fixing the prior `failure hook panic escaped request isolation: failure hook failed` finding.
- **Stream Gate ownership preserved.** `Server.obsSink` (`streamgate.ObservationSink`, default `newZapFilterObservationSink(logger)`) and its `SetObservationSink`/`observationSink` accessors are unchanged. Hot Path observation is a separate `hotPathObserver` field with its own install path; the two observability contracts never share ownership. `streamgate.ObservationSink` is untouched.
- **Gathered-series boundary proof (reviewer repair).** `TestHotPathMetricProjectionBoundary` now gathers each collector with `prometheus/testutil.CollectAndCount` before/after recording, using deltas so the assertions stay robust to series accumulated by other tests on the shared package collectors. It proves (a) invalid typed-string casts across every closed dimension (mode/reason/disposition/cleanup/orphan/stage) create zero new series, (b) distinct route reasons and distinct cleanup/orphan outcomes create the expected distinct series instead of being discarded, and (c) a secret-sentinel edge id collapses to the single `edge-local` label. This directly satisfies the plan Test Strategy (`invalid casts produce no series`, `distinct route/cleanup/orphan series`) and the Reviewer Checkpoints (`Gather metrics to prove different reasons/outcomes create distinct series and forbidden/high-cardinality values do not appear`).

## Reviewer Checkpoints

- Invoke invalid typed-string values through the actual observer and collector entry points; standalone normalizer tests are insufficient.
- Compare the exact S15 log projection and each metric-specific label set, including preset/attempt, route reason, terminal, cleanup, and orphan evidence.
- Gather metrics to prove different reasons/outcomes create distinct series and forbidden/high-cardinality values do not appear.
- Exercise the server-owned emission seam for sink error, sink panic, failure-hook panic, and concurrent observer replacement.
- Confirm `Server.obsSink` remains the existing Stream Gate contract and Hot Path observation does not share its ownership.

## Verification Results

Fill each result with the exact command stdout/stderr and exit status. If a command changes, record the replacement and reason in `Deviations from Plan`.

### Projection boundary

Command: `TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObservationProjectionBoundary|MetricProjectionBoundary|ObservationSchema|ObservationRejectsRawValues|MetricLabels)'`

```
ok  	iop/apps/edge/internal/openai	1.558s
exit=0
```

### Production failure isolation

Command: `TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObserverProductionFailureIsolation|ObserverFailureIsolation)'`

```
ok  	iop/apps/edge/internal/openai	1.712s
exit=0
```

### Final targeted

Command: `TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObservationProjectionBoundary|MetricProjectionBoundary|ObserverProductionFailureIsolation|ObservationSchema|ObservationRejectsRawValues|MetricLabels|ObserverFailureIsolation)'`

```
ok  	iop/apps/edge/internal/openai	1.070s
exit=0
```

### SDD common regression

Command: `TMPDIR=/tmp go test -race -count=1 ./packages/go/streamgate ./packages/go/config ./apps/edge/internal/openai ./apps/edge/internal/service`

```
ok  	iop/packages/go/streamgate	2.025s
ok  	iop/packages/go/config	1.558s
ok  	iop/apps/edge/internal/openai	11.492s
ok  	iop/apps/edge/internal/service	7.026s
exit=0
```

### Edge smoke

Command: `TMPDIR=/tmp go test -count=1 ./apps/edge/...`

```
ok  	iop/apps/edge/cmd/edge	0.593s
ok  	iop/apps/edge/internal/authprojection	0.211s
--- FAIL: TestActualNodeReconnectReadyPumpsQueuedWaiterExactlyOnce (31.71s)
    reconnect_readiness_integration_test.go:81: start actual iop-node: fork/exec /tmp/TestActualNodeReconnectReadyPumpsQueuedWaiterExactlyOnce3416319131/001/iop-node: permission denied
FAIL
FAIL	iop/apps/edge/internal/bootstrap	32.817s
ok  	iop/apps/edge/internal/configrefresh	0.394s
ok  	iop/apps/edge/internal/controlplane	6.835s
ok  	iop/apps/edge/internal/edgecmd	0.429s
ok  	iop/apps/edge/internal/edgevalidate	0.217s
ok  	iop/apps/edge/internal/events	0.157s
ok  	iop/apps/edge/internal/input	0.305s
ok  	iop/apps/edge/internal/input/a2a	0.234s
ok  	iop/apps/edge/internal/node	0.216s
ok  	iop/apps/edge/internal/openai	12.406s
ok  	iop/apps/edge/internal/opsconsole	0.165s
ok  	iop/apps/edge/internal/service	6.287s
ok  	iop/apps/edge/internal/transport	4.972s
FAIL
exit=1
```

**Exit 1 is environmental, not a code defect** (see `Deviations from Plan`). The single failure is `apps/edge/internal/bootstrap::TestActualNodeReconnectReadyPumpsQueuedWaiterExactlyOnce`, which builds `iop-node` into `TMPDIR=/tmp` and then `exec.Command(...).Start()`s it. The dispatcher sandbox mounts `/tmp` as `noexec` (`tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime)`), so the exec is rejected with `permission denied`. That package is outside this plan's scope (no target file touches it); the plan's actual target package `apps/edge/internal/openai` passes fully under `-race` (shown `ok  ... 12.406s` above).

### Reviewer supplemental executable-TMPDIR smoke

Command: `TMPDIR=/config/workspace/iop-s0/.edge-smoke-review.zEkvNl go test -count=1 ./apps/edge/...`

```
ok  \tiop/apps/edge/cmd/edge\t1.720s
ok  \tiop/apps/edge/internal/authprojection\t0.125s
ok  \tiop/apps/edge/internal/bootstrap\t40.336s
ok  \tiop/apps/edge/internal/configrefresh\t1.434s
ok  \tiop/apps/edge/internal/controlplane\t7.049s
ok  \tiop/apps/edge/internal/edgecmd\t0.861s
ok  \tiop/apps/edge/internal/edgevalidate\t0.240s
ok  \tiop/apps/edge/internal/events\t0.137s
ok  \tiop/apps/edge/internal/input\t0.376s
ok  \tiop/apps/edge/internal/input/a2a\t0.273s
ok  \tiop/apps/edge/internal/node\t0.247s
ok  \tiop/apps/edge/internal/openai\t18.227s
ok  \tiop/apps/edge/internal/opsconsole\t0.284s
ok  \tiop/apps/edge/internal/service\t6.898s
ok  \tiop/apps/edge/internal/transport\t5.334s
exit=0
```

The temporary directory was removed after the run. This supplemental check changes only `TMPDIR`; it proves the full Edge suite passes when the integration-test binary is built on an executable filesystem and confirms the exact `/tmp` failure is environmental.

### Edge vet

Command: `go vet ./apps/edge/...`

```
(no output)
exit=0
```

### Formatting

Command: `gofmt -d apps/edge/internal/openai/hot_path_observation.go apps/edge/internal/openai/hot_path_metrics.go apps/edge/internal/openai/server.go apps/edge/internal/openai/hot_path_observation_test.go`

```
(no output)
exit=0
```

### Diff

Command: `git diff --check`

```
(no output)
exit=0
```

---

> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.

## Section Ownership

| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |

## Code Review Result

- Overall Verdict: PASS
- Dimension Assessment:
  - Correctness: Pass
  - Completeness: Pass
  - Test coverage: Pass
  - API contract: Pass
  - Code quality: Pass
  - Implementation deviation: Pass
  - Verification trust: Pass
  - Spec conformance (SDD S15, `milestone-task=route-observability`): Pass
- Findings:
  - None outstanding. Two Nit-level reviewer repairs were applied during review (both recorded under `Deviations from Plan` and `Key Design Decisions`): (1) `TestHotPathMetricProjectionBoundary` gained `testutil.CollectAndCount` gathered-series delta assertions proving invalid casts create no series and distinct route/cleanup/orphan outcomes create distinct series, closing the plan Test Strategy that the original smoke-only record left unproven; (2) the `hotPathBoundedObserver` type doc comment was corrected to match current behavior. After these repairs every dimension is Pass with no Required or Suggested issue remaining.
- Routing Signals:
  - `review_rework_count=1` (one archived same-task FAIL: `code_review_cloud_G07_2.log`; `code_review_cloud_G07_1.log` is a superseded stub with no verdict)
  - `evidence_integrity_failure=false` (every claimed command, exit code, and production seam was re-run fresh and matched the reported output)
- Next Step: PASS — finalize by archiving the active pair to `code_review_cloud_G06_3.log` / `plan_cloud_G06_3.log`, writing `complete.log` (preserving first-line `milestone-task=route-observability`), and moving the task directory to `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/18+17_observation_schema/` for runtime aggregation. Roadmap evaluation is deferred to `sync-milestone-workstate`; code-review does not modify the roadmap.
