<!-- task=m-iop-owned-single-request-agent-execution/14+05,12,13_workspace_observation plan=0 tag=API milestone-task=cleanup-observation -->

# Raw-free Single-request Timing Observation

## For the Implementing Agent

Do not start until packets 05, 12, and 13 each have `complete.log`. Instrument the stable lifecycle without changing request outcomes, use only closed fields/labels, run every command, and fill `CODE_REVIEW-cloud-G08.md`. Review finalization and any external smoke remain outside this packet.

## Background

Ingress count, internal tools, and cleanup will exist, but S07 still needs linked request/stage/tool/cleanup/total timing and outcomes without raw prompt, path, command, output, credential, provider, or unbounded metric labels. This packet adds failure-isolated Edge metrics/log projection and Node tool logs over the completed lifecycle.

## Analysis

### Files Read

- `AGENTS.md`
- `agent-ops/rules/project/rules.md`
- `agent-ops/rules/common/rules-roadmap.md`
- `agent-ops/rules/common/rules-agent-spec.md`
- `agent-ops/rules/project/domain/edge/rules.md`
- `agent-ops/rules/project/domain/node/rules.md`
- `agent-ops/rules/project/domain/platform-common/rules.md`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/node-smoke.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-owned-single-request-agent-execution/SDD.md`
- `apps/edge/internal/service/service.go`
- `apps/edge/internal/bootstrap/runtime.go`
- `apps/edge/internal/input/manager.go`
- `apps/edge/internal/openai/server.go`
- `apps/edge/internal/openai/hot_path_observation.go`
- `apps/edge/internal/openai/hot_path_metrics.go`
- `packages/go/observability/observability.go`
- `agent-spec/input/openai-compatible-surface.md`
- `agent-spec/runtime/edge-node-execution.md`

### SDD Criteria

- S07 requires raw-free request/stage/tool/total timing and outcome linked across success/error/cancel, alongside cleanup preservation evidence.
- S12 later requires actual Claude/Mac evidence; this packet supplies runtime instrumentation and deterministic synthetic evidence only.
- D10 forbids internal reasoning/tool protocol in outer output; observation must also exclude raw command/path/output and credentials.

### Verification Context

- Existing Hot Path observation demonstrates closed enums, explicit log-key allowlists, bounded Prometheus labels, injected sinks, and failure isolation; single-request metrics remain a separate owner.
- Packet 05 supplies a no-label ingress counter. Packets 12/13 supply stable lifecycle seams and Node response durations.
- Baseline packages passed fresh; deterministic tests use an injected clock and Prometheus gatherer, no external runner.

### State and Concurrency Findings

- Total wall time begins at accepted marked admission and ends after cleanup plus terminal acknowledgement/outcome.
- Stage pure time accumulates only provider-active intervals; it pauses during `internal_tool` Node execution and resumes on result continuation. Tool duration comes from typed Node execution and cleanup has its own interval.
- Observer panic/error must not change execution or terminal. One terminal event wins the same coordinator race as the terminal itself.

### Test Coverage Gaps

- Packet 05 observes ingress only; no stage/tool/cleanup/total collectors or linked log schema exists.
- Node workspace results have duration fields but no closed raw-free local observation seam.

### Symbol References

- Do not merge with route-01 `hotPathObserver` or Stream Gate observation; use distinct metric names/types.
- Extend packet 05's `single_request_metrics.go` only for the existing ingress test accessor/correlation assertions; service owns lifecycle metrics.
- No API/wire/config schema changes are required.

### Split Judgment

- Observation is independently PASS-capable after lifecycle completion and cannot alter cleanup or terminal semantics.
- Actual Claude/Mac smoke remains the separate `claude-smoke` Task because credentials/device evidence is external.

### Scope Rationale

- Include closed event DTO, injected clock/observer, service metrics/zap adapter, Node safe logs, bootstrap wiring, allowlist/cardinality/failure tests, living specs.
- Exclude raw payload logging, request-derived metric labels, dashboards/ledger, external smoke, and semantic outcome changes.

### Final Routing

- `evaluation_mode=first-pass`; build closures true, scores 2/1/1/1/2 = G07.
- Finalizer route `risk-boundary`, lane `cloud`, filename `PLAN-cloud-G07.md`; risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4).
- Review scores 2/1/1/2/2 = G08; official filename `CODE_REVIEW-cloud-G08.md`; no capability/recovery gap.

## Dependencies and Execution Order

1. Require packet 05 ingress metric/HTTP evidence.
2. Require packet 12 tool loop and packet 13 cleanup lifecycle.
3. Define closed events and tests, instrument service/Node, wire production observer, then update specs.

## Implementation Checklist

- [ ] Define a closed, copy-safe single-request observation schema and explicit log/metric allowlists that exclude all raw or unbounded values.
- [ ] Measure request total, provider-active stage, Node tool, and cleanup durations/outcomes exactly once without counting tool time as stage pure time.
- [ ] Add failure-isolated bounded Prometheus/zap observers, wire them at Edge startup, and emit raw-free Node tool/cleanup logs.
- [ ] Prove cardinality, correlation, timing math, terminal races, observer panic/error isolation, secret sentinels, and ingress-to-total count consistency.
- [ ] Synchronize input/runtime specs with deterministic evidence and explicit external Claude/Mac smoke deferral.
- [ ] Run all dependency, focused race, package, vet, deterministic search, and whitespace verification.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

## Implementation Plan

### [API-1] Define closed observation and timing semantics

**Problem**

- `apps/edge/internal/service/service.go:28` has no observer or clock for packet 03/12/13 lifecycle.
- `apps/edge/internal/openai/hot_path_observation.go:1` is route-01-specific and includes different states/identities.

**Solution**

Add service-owned closed enums for event class (`request`, `stage`, `tool`, `cleanup`, `terminal`), stage (`plan`, `work`, `review`), operation, and outcome/error class. The DTO may include a bounded generated execution correlation id in logs, but metric labels are only fixed Edge id plus closed enums. It contains durations/counts/truncated booleans, never request text, public model, provider id, Node/root/path, command/template/env, tool input/output, error string, header, credential, or raw terminal.

Add an injectable clock and failure-isolated observer snapshot on `Service`. Accumulate provider-active stage intervals around `internal_tool`, record Node tool duration once, cleanup once, terminal once, and request total after acknowledgement/cancel resolution. Unknown enum/value normalizes to empty and is dropped.

**Modified Files and Checklist**

- [ ] `apps/edge/internal/service/service.go` — configure/snapshot observer and clock safely.
- [ ] `apps/edge/internal/service/single_request_observation.go` — closed DTO/enums, allowlists, sanitizer, timer accumulation, safe emit.
- [ ] `apps/edge/internal/service/single_request.go` — request/stage/terminal/total timing hooks.
- [ ] `apps/edge/internal/service/single_request_tool_loop.go` — tool/pause/resume timing hooks.
- [ ] `apps/edge/internal/service/single_request_observation_test.go` — deterministic clock, success/error/cancel/race, pure-time math, schema/sentinel, observer failure.

**Test Strategy**

- Use a manual clock and capturing/panicking observer. Assert exact event count/order and `stage_active + tool + cleanup <= total` with tool duration excluded from stage active time.

**Verification**

- `go test -race ./apps/edge/internal/service -run 'TestSingleRequestObservation' -count=1`
- Expected: timing/event ownership is deterministic and observer failure never changes terminal behavior.

### [API-2] Emit bounded Edge metrics/logs and safe Node events

**Problem**

- Packet 05's planned `single_request_metrics.go` contains only `iop_anthropic_single_request_ingress_total`.
- Node workspace runtime has no allowlisted outcome log projection.

**Solution**

Create service lifecycle Prometheus histograms/counters with closed labels and a zap observer with an exact key allowlist. Register once, normalize Edge id, and expose test-only gather helpers without request-derived labels. In Edge bootstrap, install this observer on the service before input servers are created. Extend the packet 05 test metric helper only as needed to compare ingress and request-total deltas.

Add a workspace observer at the single `Runtime.Execute`/cleanup completion seams. Its exact zap fields are execution correlation, operation, closed outcome/error code, duration, truncation, process/artifact counts; never path/content/command/env/stdout/stderr. Observer failure is swallowed.

**Modified Files and Checklist**

- [ ] `apps/edge/internal/service/single_request_metrics.go` — bounded collectors and production zap/metric observer.
- [ ] `apps/edge/internal/service/single_request_metrics_test.go` — collector labels/cardinality, exact deltas, duplicate-terminal protection.
- [ ] `apps/edge/internal/bootstrap/runtime.go` — install the production observer before request handling.
- [ ] `apps/edge/internal/bootstrap/single_request_observation_test.go` — assert wiring and failure isolation.
- [ ] `apps/edge/internal/openai/single_request_metrics.go` — preserve ingress owner and expose bounded test correlation only.
- [ ] `apps/node/internal/workspace/observation.go` — raw-free Node event projection and safe zap observer.
- [ ] `apps/node/internal/workspace/runtime.go` — emit one tool event at the common completion seam.
- [ ] `apps/node/internal/workspace/cleanup.go` — emit one cleanup event.
- [ ] `apps/node/internal/workspace/observation_test.go` — exact key allowlist, sentinel rejection, outcome count, observer panic/error.

**Test Strategy**

- Gather metrics before/after deterministic flows and inspect captured zap cores. Assert no forbidden keys/values and no request id in metric labels.

**Verification**

- `go test -race ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/node/internal/workspace -run 'Test(SingleRequestMetrics|SingleRequestObservationWiring|WorkspaceObservation)' -count=1`
- Expected: bounded metrics/logs emit once and cannot influence runtime outcomes.

### [API-3] Link ingress, lifecycle, and documented evidence

**Problem**

- S07 evidence needs one linked synthetic flow and specs describing pure-time/cardinality/privacy semantics; current living specs have only generic/route-01 observation.

**Solution**

Extend packet 12's real-POST test to snapshot ingress and lifecycle metrics, run multiple stages/tools plus cleanup, and assert deltas: ingress=1, request-total=1, terminal=1, expected stage/tool/cleanup counts. Capture logs using the safe generated correlation id and verify forbidden sentinels are absent. Keep actual Claude/Mac timing evidence explicitly deferred to `claude-smoke`.

**Modified Files and Checklist**

- [ ] `apps/edge/internal/openai/single_request_handler_test.go` — ingress/lifecycle delta and raw-free correlation integration assertions.
- [ ] `agent-spec/input/openai-compatible-surface.md` — document ingress/terminal correlation and privacy.
- [ ] `agent-spec/runtime/edge-node-execution.md` — document stage-pure/tool/cleanup/total timing, labels, Node logs, and external-smoke deferral.

**Test Strategy**

- Add/extend `TestAnthropicSingleRequestObservation` using the real marked POST fixture and deterministic internal tools.

**Verification**

- `go test ./apps/edge/internal/openai -run 'TestAnthropicSingleRequestObservation' -count=1`
- `rg --sort path -n 'stage.*pure|tool.*duration|cleanup|total|raw|cardinality|Claude.*defer' agent-spec/input/openai-compatible-surface.md agent-spec/runtime/edge-node-execution.md`
- Expected: one synthetic request links all closed timing/outcome evidence and docs do not claim external smoke.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/service/service.go` | API-1 |
| `apps/edge/internal/service/single_request_observation.go` | API-1 |
| `apps/edge/internal/service/single_request.go` | API-1 |
| `apps/edge/internal/service/single_request_tool_loop.go` | API-1 |
| `apps/edge/internal/service/single_request_observation_test.go` | API-1 |
| `apps/edge/internal/service/single_request_metrics.go` | API-2 |
| `apps/edge/internal/service/single_request_metrics_test.go` | API-2 |
| `apps/edge/internal/bootstrap/runtime.go` | API-2 |
| `apps/edge/internal/bootstrap/single_request_observation_test.go` | API-2 |
| `apps/edge/internal/openai/single_request_metrics.go` | API-2 |
| `apps/node/internal/workspace/observation.go` | API-2 |
| `apps/node/internal/workspace/runtime.go` | API-2 |
| `apps/node/internal/workspace/cleanup.go` | API-2 |
| `apps/node/internal/workspace/observation_test.go` | API-2 |
| `apps/edge/internal/openai/single_request_handler_test.go` | API-3 |
| `agent-spec/input/openai-compatible-surface.md` | API-3 |
| `agent-spec/runtime/edge-node-execution.md` | API-3 |
| `agent-task/m-iop-owned-single-request-agent-execution/14+05,12,13_workspace_observation/CODE_REVIEW-cloud-G08.md` | API-1, API-2, API-3 |

## Final Verification

1. `test -f agent-task/m-iop-owned-single-request-agent-execution/05+03_single_ingress/complete.log || test "$(compgen -G 'agent-task/archive/*/*/m-iop-owned-single-request-agent-execution/05+03_single_ingress/complete.log' | wc -l)" -eq 1`
2. `test -f agent-task/m-iop-owned-single-request-agent-execution/12+05,08,11_internal_tool_loop/complete.log || test "$(compgen -G 'agent-task/archive/*/*/m-iop-owned-single-request-agent-execution/12+05,08,11_internal_tool_loop/complete.log' | wc -l)" -eq 1`
3. `test -f agent-task/m-iop-owned-single-request-agent-execution/13+12_workspace_cleanup/complete.log || test "$(compgen -G 'agent-task/archive/*/*/m-iop-owned-single-request-agent-execution/13+12_workspace_cleanup/complete.log' | wc -l)" -eq 1`
4. `go test -race ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/node/internal/workspace -run 'Test(SingleRequestObservation|SingleRequestMetrics|SingleRequestObservationWiring|WorkspaceObservation)' -count=1`
5. `go test ./apps/edge/internal/openai -run 'TestAnthropicSingleRequestObservation' -count=1`
6. `go test ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/openai ./apps/node/internal/workspace -count=1`
7. `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/openai ./apps/node/internal/workspace`
8. `rg --sort path -n 'stage.*pure|tool.*duration|cleanup|total|raw|cardinality|Claude.*defer' agent-spec/input/openai-compatible-surface.md agent-spec/runtime/edge-node-execution.md`
9. `git diff --check`

Expected: all predecessors are uniquely complete; timing math, cardinality, privacy, failure isolation, and ingress-to-terminal deltas pass; docs defer external smoke. Cached tests are not acceptable.

**After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.**
