<!-- task=m-iop-hot-path-one-shot-execution/18+17_observation_schema plan=3 tag=REVIEW_API milestone-task=route-observability -->

# Enforce the Hot Path observation boundary

## For the Implementing Agent

Implement the checklist, run every verification command, and fill the implementation-owned sections of `CODE_REVIEW-cloud-G06.md` with actual notes and output. Keep the active files in place and report ready for review; finalization is code-review-only. If blocked, record only the exact blocker, attempted commands/output, and resume condition. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The previous schema implementation defined normalizer helpers but did not enforce them at the observer or metric boundaries. Reviewer probes proved that arbitrary enum text reaches Prometheus labels and that a failure-hook panic escapes observation isolation. This follow-up closes the raw-free projection and server-owned failure-isolation contract before lifecycle wiring begins.

## Archive Evidence Snapshot

- The prior pair will be archived as `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/plan_local_G06_2.log` and `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/code_review_cloud_G07_2.log` with verdict `FAIL`.
- Required findings: collector entry points accept arbitrary typed strings and discard route/cleanup/orphan values; the log projection omits S15 preset/attempt/outcome evidence and the bounded observer delegates without validation; the server has no production safe emission seam and a panicking failure hook escapes.
- Fresh targeted and SDD-common race commands passed, but a focused reviewer probe failed with `unknown metric values reached the collector: got 1, want 0` and `failure hook panic escaped request isolation: failure hook failed`; `evidence_integrity_failure=true`.
- Milestone carryover remains `milestone-task=route-observability`, SDD S15, raw-free log/metric allowlist evidence.

## Analysis

### Files Read

- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-hot-path-one-shot-execution.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-contract/index.md`
- `agent-ops/rules/project/domain/edge/rules.md`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`
- `apps/edge/internal/openai/hot_path_observation.go`
- `apps/edge/internal/openai/hot_path_metrics.go`
- `apps/edge/internal/openai/hot_path_observation_test.go`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/filter_observation_sink.go`
- `apps/edge/internal/openai/usage_metrics.go`
- `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/PLAN-local-G06.md`
- `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/CODE_REVIEW-cloud-G07.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`, status `[승인됨]`, lock released.
- Metadata: `milestone-task=route-observability`.
- Acceptance Scenario S15 requires direct/light and failure metrics/logs to correlate request, preset, mode, stage, attempt, and outcome without raw prompt/output/credential data.
- Evidence Map S15 requires raw-free log/metric field allowlist tests. This drives production-boundary rejection, complete bounded dimensions, sink/collector descriptor assertions, and the final race verification.

### Verification Context

- No external handoff was supplied. Repository-native sources are `agent-test/local/rules.md`, `agent-test/local/edge-smoke.md`, the approved SDD, and current package tests.
- Local preflight: repository `/config/workspace/iop-s0`, current shared dirty checkout, `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s0/go.mod`; no credential or external backend is required.
- Fresh reviewer evidence: targeted Hot Path schema tests passed; the SDD-common race suite passed for `streamgate`, `config`, `openai`, and `service`; `git diff --check` passed. The focused boundary probe failed in both arbitrary-label rejection and hook-panic isolation.
- Dependency `17` is satisfied by `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/17+14,15,16_endpoint_error_matrix/complete.log`.
- Use `TMPDIR=/tmp` and `-count=1` for fresh deterministic evidence. Confidence is high because the probe invoked the actual collector and safe-observer entry points.

### Test Coverage Gaps

- Existing normalization tests call helpers directly but do not pass invalid values through `record*` methods or inspect gathered labels.
- Existing allowlist tests compare declared key slices but do not prove that a sink receives a validated, complete S15 projection.
- Existing failure tests cover sink error/panic, not failure-hook panic or the server-owned production emission seam.
- Existing tests do not prove that route reason and cleanup/orphan outcomes produce distinct metric series or that preset/attempt evidence is present.

### Symbol References

- No rename or removal is planned. `rg` found the new observer/metric symbols only in their definitions, `server.go`, and `hot_path_observation_test.go`; lifecycle call sites remain intentionally absent until the wiring child.

### Split Judgment

- Keep one compact plan: projection validation and safe emission are one boundary invariant, and neither half independently proves raw-free, behavior-neutral observation.
- The `18+17` directory dependency is satisfied by the archived child-17 `complete.log` cited above.

### Scope Rationale

- Exclude dispatch/light/cleanup lifecycle call-site wiring, dashboard/backend/retention, payload hashing, external telemetry, and Stream Gate `Server.obsSink` changes. This child only makes the schema and emission seam safe for the later wiring child.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, mode `pair`.
- Build closures are all true; scores `1/1/1/2/1` produce `G06` with base `local-fit`. `large_indivisible_context=false`; matched risks are `boundary_contract`, `concurrent_consistency`, and `variant_product` (3). `review_rework_count=1` and `evidence_integrity_failure=true` select `recovery-boundary`, yielding `PLAN-cloud-G06.md`.
- Review closures are all true; scores `1/1/1/2/1` produce official cloud review `G06`, yielding `CODE_REVIEW-cloud-G06.md`. No capability gap exists.

## Implementation Checklist

- [ ] [REVIEW_API-1] Enforce a complete S15 log/metric projection at production entry points, reject unknown typed-string values, and preserve only bounded log correlation identifiers.
- [ ] [REVIEW_API-2] Add one server-owned safe emission seam that isolates sink and failure-hook errors/panics while preserving Stream Gate observation ownership, with regression tests through the real seam.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_API-1] Close projection and collector inputs

**Problem:** `hot_path_metrics.go:193-313` writes enum arguments directly, while route reason and cleanup/orphan outcomes are accepted but discarded. `hot_path_observation.go:298-325` omits preset, attempt, cleanup, and orphan evidence, and `hotPathBoundedObserver.Emit` forwards unchecked projections.

**Solution:** Define the complete S15 schema once. Use metric-specific fixed label sets for mode/stage/disposition, route reason, attempt bucket, and cleanup/orphan outcomes; keep request/stage/call correlation identifiers log-only. Validate or normalize at every `record*` and observer entry point so direct casts cannot create arbitrary series. Add the bounded preset identity and closed endpoint/attempt dimensions needed to join S15 events, and reject invalid projections before an inner sink sees them.

Before (`hot_path_metrics.go:212`):

```go
func (m *hotPathMetrics) recordTerminal(edgeID string, mode hotPathMode, disposition hotPathTerminalDispositionKind) {
    m.terminalCounter.WithLabelValues(edgeID, string(hotPathEventClassTerminal), string(mode), "", string(disposition), "", "").Inc()
}
```

After:

```go
func (m *hotPathMetrics) recordTerminal(edgeID string, mode hotPathMode, disposition hotPathTerminalDispositionKind) {
    mode = hotPathNormalizeMode(string(mode))
    disposition = hotPathNormalizeDisposition(string(disposition))
    if m == nil || mode == "" || disposition == "" {
        return
    }
    m.terminalCounter.WithLabelValues(edgeID, string(mode), string(disposition)).Inc()
}
```

**Modified Files and Checklist:**

- [ ] Modify `apps/edge/internal/openai/hot_path_observation.go` with complete projection fields and one validating/bounding path.
- [ ] Modify `apps/edge/internal/openai/hot_path_metrics.go` with metric-specific label sets, enforced normalization, and observable reason/outcome dimensions.
- [ ] Modify `apps/edge/internal/openai/hot_path_observation_test.go` with gathered-label and sink-projection tables covering valid, unknown, and secret-sentinel inputs.

**Test Strategy:** Add `TestHotPathObservationProjectionBoundary` and `TestHotPathMetricProjectionBoundary`. Assert exact accepted keys/labels, distinct route/cleanup/orphan series, absent high-cardinality metric ids, invalid casts produce no series, and a sentinel cannot reach the captured sink or gathered descriptor.

**Verification:** `TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObservationProjectionBoundary|MetricProjectionBoundary|ObservationSchema|ObservationRejectsRawValues|MetricLabels)'` exits 0.

### [REVIEW_API-2] Make the server emission seam failure-proof

**Problem:** `hot_path_observation.go:444-469` recovers sink panics but calls the failure hook without its own recovery. `server.go:238-266` stores raw observer/hook values, documents a nonexistent `emitHotPathObservation`, and exposes no production path that guarantees the safe/bounded chain.

**Solution:** Add one unexported `Server.emitHotPathObservation` that snapshots observer and hook under `RLock`, then invokes a bounded observer inside a safe wrapper. Isolate hook panic separately so neither sink nor reporting failures escape. Keep `Server.obsSink` and `streamgate.ObservationSink` untouched, and retain a noop default.

Before (`server.go:248`):

```go
func (s *Server) HotPathObserver() hotPathObserver {
    // raw observer accessor only; no production emit path exists
}
```

After:

```go
func (s *Server) emitHotPathObservation(ctx context.Context, projection hotPathLogProjection) {
    observer, hook := s.hotPathObservationSnapshot()
    safe := hotPathSafeObserver{inner: &hotPathBoundedObserver{inner: observer}, onFailure: hook}
    _ = safe.Emit(ctx, projection)
}
```

**Modified Files and Checklist:**

- [ ] Modify `apps/edge/internal/openai/hot_path_observation.go` so sink errors, sink panics, and hook panics are all contained.
- [ ] Modify `apps/edge/internal/openai/server.go` with the single safe emission seam and race-safe snapshot while preserving `obsSink` unchanged.
- [ ] Modify `apps/edge/internal/openai/hot_path_observation_test.go` with production-seam error/panic/hook-panic and concurrent set/emit tests.

**Test Strategy:** Add `TestHotPathObserverProductionFailureIsolation` as a table for success, sink error, sink panic, and hook panic; assert the request-side call never panics/returns failure, valid projections reach the sink once, invalid projections do not, and `go test -race` reports no observer swap race.

**Verification:** `TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObserverProductionFailureIsolation|ObserverFailureIsolation)'` exits 0.

## Dependencies and Execution Order

1. Child `17+14,15,16_endpoint_error_matrix` is complete at `agent-task/archive/2026/08/m-iop-hot-path-one-shot-execution/17+14,15,16_endpoint_error_matrix/complete.log`.
2. Implement REVIEW_API-1 before REVIEW_API-2 so the server seam can rely on one validated projection contract.

## Modified Files Summary

| File | Item |
|---|---|
| `apps/edge/internal/openai/hot_path_observation.go` | REVIEW_API-1, REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_metrics.go` | REVIEW_API-1 |
| `apps/edge/internal/openai/server.go` | REVIEW_API-2 |
| `apps/edge/internal/openai/hot_path_observation_test.go` | REVIEW_API-1, REVIEW_API-2 |
| `agent-task/m-iop-hot-path-one-shot-execution/18+17_observation_schema/CODE_REVIEW-cloud-G06.md` | implementation evidence |

## Final Verification

```bash
TMPDIR=/tmp go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPath(ObservationProjectionBoundary|MetricProjectionBoundary|ObserverProductionFailureIsolation|ObservationSchema|ObservationRejectsRawValues|MetricLabels|ObserverFailureIsolation)'
TMPDIR=/tmp go test -race -count=1 ./packages/go/streamgate ./packages/go/config ./apps/edge/internal/openai ./apps/edge/internal/service
TMPDIR=/tmp go test -count=1 ./apps/edge/...
go vet ./apps/edge/...
gofmt -d apps/edge/internal/openai/hot_path_observation.go apps/edge/internal/openai/hot_path_metrics.go apps/edge/internal/openai/server.go apps/edge/internal/openai/hot_path_observation_test.go
git diff --check
```

Expected: all commands exit 0 with fresh output; invalid typed strings create no log/metric observation, S15 route/outcome dimensions remain distinguishable, all observer/hook failures are isolated, Stream Gate ownership is unchanged, formatting and diff checks are empty.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
