<!-- task=m-iop-hot-path-one-shot-execution/16+15_route_observability plan=0 tag=API milestone-task=route-observability -->

# Raw-free Hot Path route observability

## For the Implementing Agent

구현 후 `CODE_REVIEW-cloud-G10.md`의 구현 담당 섹션에 실제 notes/output을 채우고 active 파일을 유지한다. 차단 시 정확한 blocker/시도/출력/재개 조건만 기록하며 사용자 질문, archive, `complete.log` 작성은 하지 않는다.

## Background

현재 dispatch logging은 run/provider 필드를 기록하지만 Hot Path logical request의 preset/mode/stage/attempt/terminal을 한 lifecycle로 연결하지 않는다. 관측 payload가 prompt/output/tool argument/credential을 포함하지 않도록 단일 allowlist event를 추가한다.

## Analysis

### Files Read

- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-hot-path-one-shot-execution/SDD.md`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/chat_handler.go`
- `apps/edge/internal/openai/hot_path_dispatch.go`
- `apps/edge/internal/openai/hot_path_light.go`
- `apps/edge/internal/openai/hot_path_cleanup.go`
- `apps/edge/internal/openai/hot_path_cleanup_test.go`
- `apps/edge/internal/openai/filter_observation_sink.go`
- `apps/edge/internal/openai/usage_metrics.go`
- `packages/go/streamgate/filter_observation.go`
- `agent-spec/runtime/stream-evidence-gate.md`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`

### SDD Criteria

- 승인 SDD, `milestone-task=route-observability`, S15.
- Evidence Map S15의 raw-free log/metric field allowlist를 직접 구현·검증한다. request/preset/mode/stage/attempt, route reason, timing, terminal outcome 연결과 secret/raw absence가 pass 조건이다.

### Verification Context

- handoff 없음. local edge profile과 in-memory observation sink/log observer를 사용한다. repo/branch/HEAD=`/config/workspace/iop-s0`, `feature/iop-hot-path-one-shot-execution`, `6650e9f70d0104220d8077dd1d469b6a1facb9da`.
- 외부 metrics backend 없이 deterministic unit/integration evidence로 닫는다.

### Test Coverage Gaps

- 기존 tests는 typed `streamgate.FilterObservation`과 usage 일부를 확인하지만 Hot Path lifecycle field allowlist, cardinality, raw/secret 금지, orphan responsibility outcome을 확인하지 않는다. 기존 `streamgate.ObservationSink`는 `FilterObservation` 전용이므로 Hot Path 임의 event sink로 재사용할 수 없다.

### Symbol References

- rename/remove 없음. 기존 `Server.logger`와 Prometheus registry 관례를 사용하고 `Server.obsSink`/`streamgate.FilterObservation` 계약은 변경하지 않는다.

### Split Judgment

- stable contract: Hot Path lifecycle → low-cardinality raw-free observation allowlist.
- predecessor 15 (`15+13,14_error_cancel`) active `complete.log`는 현재 missing이며 terminal outcomes 확정 후 구현한다.

### Scope Rationale

- backend/dashboard/alert와 new durable store는 제외한다. raw payload hashing도 유출/고카디널리티 위험 때문에 추가하지 않는다. request/stage/attempt id는 log correlation에만 두고 metric label에서는 제외한다.

### Final Routing

- evaluation_mode=write, finalizer=`finalize-task-policy.sh pair`.
- build closures 모두 true, scores=2/2/2/1/2, G09, grade-boundary → `PLAN-cloud-G09.md`.
- review closures 모두 true, scores=2/2/2/2/2, G10, official-review → `CODE_REVIEW-cloud-G10.md`.
- `large_indivisible_context=false`; risks=`temporal_state,boundary_contract,variant_product`(3); recovery=0/false; capability gap 없음.

## Implementation Checklist

- [ ] [API-1] Emit one allowlisted Hot Path lifecycle as request-correlated logs plus low-cardinality metrics across admission, stage attempts/transitions, terminal, cleanup, and orphan responsibility without raw or secret fields.
- [ ] [API-2] Add exact log/metric allowlist, cardinality, redaction, and lifecycle tests and run targeted plus SDD common verification.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [API-1] Lifecycle observation schema

**Problem:** `chat_handler.go:244` logs generic dispatch fields, but no Hot Path event links logical request and stage transitions; ad-hoc logging risks raw payload leakage.

**Solution:** Add `hotPathObservation` with a closed log field set: edge_id, request_id, preset_id/generation, mode, endpoint, stage role/id, attempt ordinal, provider/run ids, route reason, elapsed bucket, outcome, cleanup/orphan responsibility. Emit through `Server.logger` at admission, dispatch start/end, transition, terminal, cleanup result, and TTL/orphan handoff. Add dedicated counters/histograms in `hot_path_metrics.go` whose bounded labels are only edge_id, preset_id, mode, endpoint, stage_role, route_reason, outcome, and cleanup responsibility; request/stage/attempt/provider/run ids remain log-only. Never accept arbitrary maps or payload strings; omit prompt/content/reasoning/tool arguments, headers, credential refs and raw errors. Normalize outcome/reason to enums and isolate metric/log failures from request behavior.

New source imports are explicit:

```go
// hot_path_observation.go
import (
    "context"
    "time"

    "go.uber.org/zap"
)

// hot_path_metrics.go
import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)
```

Before (`server.go:61`):

```go
obsSink streamgate.ObservationSink // FilterObservation only
```

After:

```go
func (s *Server) observeHotPath(event hotPathObservation)
func recordHotPathMetric(event hotPathObservation)
```

**Modified Files and Checklist:**

- [ ] Add `apps/edge/internal/openai/hot_path_observation.go` with closed event/enum types and sink/log projection.
- [ ] Add `apps/edge/internal/openai/hot_path_metrics.go` with bounded-label counters/histograms; do not include request/stage/attempt/provider/run ids in labels.
- [ ] Modify `apps/edge/internal/openai/hot_path_dispatch.go`, `hot_path_light.go`, and `hot_path_cleanup.go` at lifecycle boundaries.

**Test Strategy:** API-2 provides exact allowlist and lifecycle tests.

**Verification:** `go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPathObservation'` exits 0.

### [API-2] Raw-free evidence

**Problem:** S15 requires positive field coverage and negative raw-data proof.

**Solution:** Capture zap observer fields and Prometheus families for direct success, light pass, repair, provider error, cancel, cleanup failure/orphan. Seed sentinel prompt/output/tool args/auth header/credential/error text and assert no log/metric key or value contains them. Assert exact log key set, exact metric label names, absence of high-cardinality ids from metrics, enum values, attempt ordering, request-stage joins, bounded timing bucket, and exactly-once counter increments.

New test-only imports:

```go
import (
    "testing"

    "github.com/prometheus/client_golang/prometheus/testutil"
    "go.uber.org/zap/zaptest/observer"
)
```

**Modified Files and Checklist:**

- [ ] Add `apps/edge/internal/openai/hot_path_observation_test.go` with `TestHotPathObservationFieldAllowlist`, `TestHotPathMetricLabelAllowlist`, `TestHotPathObservationRejectsRawValues`, and `TestHotPathObservationLifecycle`.
- [ ] Fill `agent-task/m-iop-hot-path-one-shot-execution/16+15_route_observability/CODE_REVIEW-cloud-G10.md`.

**Test Strategy:** New tests mandatory; compare sorted exact keys and sentinel absence in serialized observation/log fields.

**Verification:** run Final Verification; allowlist and all lifecycle rows pass.

## Dependencies and Execution Order

1. `15+13,14_error_cancel` must produce `agent-task/m-iop-hot-path-one-shot-execution/15+13,14_error_cancel/complete.log` before implementation.
2. Implement API-1, then API-2.

## Modified Files Summary

| File | Item |
|---|---|
| `apps/edge/internal/openai/hot_path_observation.go` | API-1 |
| `apps/edge/internal/openai/hot_path_metrics.go` | API-1 |
| `apps/edge/internal/openai/hot_path_dispatch.go` | API-1 |
| `apps/edge/internal/openai/hot_path_light.go` | API-1 |
| `apps/edge/internal/openai/hot_path_cleanup.go` | API-1 |
| `apps/edge/internal/openai/hot_path_observation_test.go` | API-2 |
| `agent-task/m-iop-hot-path-one-shot-execution/16+15_route_observability/CODE_REVIEW-cloud-G10.md` | API-2 |

## Final Verification

```bash
go test -race -count=1 ./apps/edge/internal/openai -run 'TestHotPathObservation|TestHotPathMetric|TestHotPath(EndpointTerminalMatrix|CancelCompleteRace|Cleanup)'
go test -race -count=1 ./packages/go/streamgate ./packages/go/config ./apps/edge/internal/openai ./apps/edge/internal/service
git diff --check
```

Expected: exit 0, exact log/metric allowlists, no high-cardinality metric ids or sentinel/raw/credential data, joined lifecycle and terminal outcome, empty diff check. Cached output is not acceptable.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
