<!-- task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability plan=2 tag=REFACTOR milestone-task=ops-evidence -->

# Node Response-Stall Operational Evidence

## For the Implementing Agent

Implement only this Node liveness-observability slice, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G05.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The Node already produces one fenced `response_stalled` terminal with joined health evidence for normalized and tunnel attempts, but operators cannot count or time those stalls without inspecting request-scoped events. This slice adds a bounded metric and structured-log contract at the existing exactly-once stall finalization seam without changing execution, wire, or retry behavior.

## Archive Evidence Snapshot

- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/plan_local_G05_1.log` and `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/code_review_cloud_G05_1.log`; it was an unimplemented plan=1 pair with no official verdict, implementation evidence, code change, or verification output.
- Replan finding: the plan's contract/spec write set overlapped independently runnable predecessor and sibling work (`08+07_health_overlay`, `09+08_retry_candidate_policy`, and observability siblings 12/13), so the child could create avoidable merge conflicts despite owning only Node-local evidence.
- Carryover: preserve the two existing claimed-stall seams, S06 label/log boundary, process-global production collectors, isolated test registries, duplicate-construction coverage, and the repository-native two-process diagnostic; keep this child source/test-only and leave living-contract consolidation to ordered dependent work or Milestone closure.

## Analysis

### Files Read

- `apps/node/internal/node/node.go`
- `apps/node/internal/node/liveness_watchdog.go`
- `apps/node/internal/node/liveness_health_evidence.go`
- `apps/node/internal/node/liveness_watchdog_test.go`
- `apps/node/internal/node/liveness_watchdog_lifecycle_test.go`
- `apps/node/internal/node/liveness_health_evidence_test.go`
- `apps/node/internal/node/provider_tunnel_liveness_test.go`
- `apps/edge/internal/openai/usage_metrics.go`
- `apps/edge/internal/openai/provider_observation.go`
- `apps/edge/internal/openai/provider_observability_test.go`
- `packages/go/observability/observability.go`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/node-smoke.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
- `scripts/dev/edge.sh`
- `scripts/dev/node.sh`
- `scripts/dev/edge-node-reconnect-diagnostic.sh`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
- Acceptance Scenario S06 and Evidence Map S06 require Node stall count/duration plus fence/probe result for deterministic normalized-run and tunnel stalls, with request/session/raw prompt/response and high-cardinality values absent from metric labels and the dedicated structured log.
- Those rows define REFACTOR-1's closed label vocabulary and REFACTOR-2's two-path health matrix and negative leakage assertions.

### Verification Context

- No handoff artifact was supplied; the user supplied starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be`, which matched the checkout during planning.
- The local Node profile supplied `go version && go env GOMOD`, `go test -count=1 ./packages/go/execution ./apps/node/...`, and `git diff --check`. Read-only preflight returned `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s1/go.mod`, and executable `scripts/dev/edge.sh`, `scripts/dev/node.sh`, and `scripts/dev/edge-node-reconnect-diagnostic.sh`. Planning baseline `go test -count=1 ./apps/node/internal/node -run 'Liveness|Watchdog|HealthEvidence|ProviderTunnelLiveness'` passed.
- The current stall seams are `liveness_watchdog.go:213-224` and `liveness_watchdog.go:304-311`; both already follow a successful fence claim and produce exactly one terminal. Confidence is high because the change can observe the immutable `stallObservation` without adding lifecycle state.
- No external verification is required. The repository's manual clocks and fake normalized/tunnel providers provide deterministic local evidence. The testing rule additionally requires the real Edge/Node entrypoints; `scripts/dev/edge-node-reconnect-diagnostic.sh` creates temporary mock configs, starts `scripts/dev/edge.sh` and `scripts/dev/node.sh` separately, proves registration, three ordered runs including two in one session, `/nodes`, `/capabilities`, `/transport`, reconnect, Node-to-Edge payload equality, and exactly-once terminal ordering.

### Test Coverage Gaps

- Existing watchdog tests verify terminal metadata and races but do not gather Prometheus series or capture a dedicated safe structured log.
- No test proves normalized and tunnel attempts use the same bounded labels for both `request_stalled`/available and `provider_unhealthy`/unavailable evidence.
- No test rejects run, attempt, request, session, adapter, target, prompt, response, or credential values from the new label/log surface.
- No test proves constructing multiple `Node` instances reuses one process-global production collector set instead of registering the same metric names repeatedly.

### Symbol References

- None. No existing symbol is renamed or removed; `Node` gains one internal observer field initialized by `New` and replaceable only by same-package tests.

### Split Judgment

- This child is the stable Node producer: one immutable `stallObservation` is mapped to one counter, one duration histogram, and one dedicated log for both execution paths. It has no active predecessor because the watchdog/health-evidence producers it consumes are already present at the supplied HEAD.
- `12+08_health_overlay_observability` and `13+10_recovery_observability` own Edge overlay and recovery evidence and do not share Node source/test files. This child also relinquishes shared contract/spec writes so independently runnable siblings cannot collide there.

### Scope Rationale

Do not change stall detection, timer reset, fence/probe ordering, wire metadata, retryability, Edge ingestion, provider overlay, recovery selection, dashboards, config, contracts, or specs. Do not add node/run/attempt/provider/session/adapter/target identifiers as metric labels or dedicated log fields. Contract/spec consolidation is intentionally outside this independently runnable child to keep sibling write boundaries disjoint.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closure true; scores `(1,1,2,0,1)`, grade G05, route `local-fit` -> `PLAN-local-G05.md`.
- Review closure true; scores `(1,1,2,0,1)`, grade G05, route `official-review` -> `CODE_REVIEW-cloud-G05.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `concurrent_consistency`, `variant_product` (2). No recovery signal, capability gap, review rework, or evidence-integrity failure.

## Implementation Checklist

- [ ] REFACTOR-1 emits exactly one Node response-stall counter observation, duration sample, and safe structured log from the normalized and tunnel stall-finalization seams using one process-global production collector set and only bounded execution-path, health, classification, and fence values.
- [ ] REFACTOR-2 proves request-stalled-but-provider-available and provider-unhealthy outcomes on deterministic normalized/tunnel fixtures, verifies exact metric families/labels and repeated Node construction, and proves request/session/raw prompt/response plus other high-cardinality values are absent from the dedicated log and metric labels.
- [ ] Run every focused, package, race, vet, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REFACTOR-1] Emit bounded Node stall metrics and logs

**Problem:** `apps/node/internal/node/liveness_watchdog.go:213-224` and `apps/node/internal/node/liveness_watchdog.go:304-311` finalize typed stall evidence but expose it only through request-scoped terminals. Operators cannot count or time stalls by safe fence/probe axes.

**Solution:** Add a test-injectable `nodeLivenessObserver`. Register one package-level production collector set exactly once with the default Prometheus registerer and reuse it from every `Node`; a constructor that accepts an explicit `prometheus.Registerer` creates isolated collectors only for tests. Never call `promauto.New*` or `MustRegister` from `Node.New` or per attempt. Emit `iop_node_response_stalls_total{execution_path,provider_health,liveness_classification,attempt_fence}` and `iop_node_response_stall_duration_seconds` with the identical four-label set. Normalize every label through closed allowlists (`normalized|provider_tunnel|unknown`, the three health/classification pairs, and `confirmed|unconfirmed|unknown`). Write `node_response_stall_observation` with only those labels and numeric `idle_duration_ms`. Install the reusable observer on `Node` and invoke it immediately after `stallObservationFrom` in each already-claimed stall branch; observer failure or disabled logging must never change terminal delivery.

Before (`apps/node/internal/node/liveness_watchdog.go:213`):

```go
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
```

After:

```go
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
n.liveness.Observe("normalized", obs)
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
```

Apply the same call with `provider_tunnel` before `emitClaimedTerminal` at line 309. The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`; do not add an alternate metrics server.

**Modified Files and Checklist:**

- [ ] `apps/node/internal/node/node.go`: hold the internal observer and initialize its production collectors/logger without changing the public constructor signature.
- [ ] `apps/node/internal/node/liveness_watchdog.go`: invoke the observer once in each claimed normalized/tunnel stall path.
- [ ] `apps/node/internal/node/liveness_observability.go`: define collectors, closed normalization, safe log fields, and the test-injection constructor.

**Test Strategy:** Write tests in REFACTOR-2; do not alter existing lifecycle fixtures except to reuse their manual clocks/providers.

**Verification:** `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` must pass every iteration and report both paths.

### [REFACTOR-2] Prove the bounded evidence matrix

**Problem:** `apps/node/internal/node/liveness_health_evidence.go:43-70` intentionally includes run/attempt/adapter/target in terminal metadata, so copying that map into metrics or the dedicated log would violate S06 even though the wire terminal itself is valid. Existing tests do not guard this new boundary.

**Solution:** Add a two-path table using the production watchdog seams and private Prometheus registry/zap observer. Cover available/request-stalled and unavailable/provider-unhealthy with confirmed and unconfirmed fences where deterministic. Assert counter delta one, histogram count/duration, exact label names and allowlisted values, one dedicated log per claimed stall, and absence of sentinel high-card/raw values from labels and encoded log fields. Construct multiple default `Node` values in one process and assert no duplicate-registration panic while a private registry remains isolated. Preserve the existing richer internal terminal metadata contract without editing shared contracts/specs from this independent child.

Before (`apps/node/internal/node/liveness_health_evidence.go:56`):

```go
metadata := map[string]string{
    "failure_code": string(runtime.FailureCodeResponseStalled),
    "run_id": runID,
    "attempt_id": runID,
```

After (observability projection, not terminal metadata replacement):

```go
labels := normalizeNodeLivenessLabels(path, obs)
observer.stalls.WithLabelValues(labels...).Inc()
observer.duration.WithLabelValues(labels...).Observe(obs.idle.Seconds())
```

**Modified Files and Checklist:**

- [ ] `apps/node/internal/node/liveness_observability_test.go`: add deterministic normalized/tunnel metric, duration, exact-once, allowlist, and log-leakage cases.

**Test Strategy:** Create `TestNodeLivenessObservability` subtests for `normalized/request-stalled`, `normalized/provider-unhealthy`, `provider_tunnel/request-stalled`, and `provider_tunnel/provider-unhealthy`, plus a repeated-default-construction row. Seed run/session/adapter/target/prompt/response/credential sentinels and inspect gathered DTO labels plus zap fields/message text for absence.

**Verification:** the focused test above plus the Node package/race commands below must pass with no zero-match test run.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/node/internal/node/node.go` | REFACTOR-1 |
| `apps/node/internal/node/liveness_watchdog.go` | REFACTOR-1 |
| `apps/node/internal/node/liveness_observability.go` | REFACTOR-1 |
| `apps/node/internal/node/liveness_observability_test.go` | REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/CODE_REVIEW-cloud-G05.md` | REFACTOR-1, REFACTOR-2 |

## Final Verification

Fresh Go output is required; cached output is not acceptable.

1. `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — PASS every iteration and all four named path/health subtests execute.
2. `go test -count=1 ./packages/go/execution ./apps/node/...` — PASS under the Node local profile.
3. `go test -race -count=3 ./apps/node/internal/node -run 'LivenessObservability|Watchdog|HealthEvidence'` — PASS with no race report.
4. `go vet ./packages/go/execution ./apps/node/...` — no diagnostics.
5. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — PASS using separate `scripts/dev/edge.sh` and `scripts/dev/node.sh` processes; registration, the first two same-session messages, post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, and exactly-once terminal ordering are all verified. This is the required repository-native full-cycle diagnostic, not an auxiliary smoke substitute.
6. `git diff --check` — no whitespace errors.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
