feat(epic): liveness-operations 작업을 준비한다
This commit is contained in:
parent
9dee77b4f9
commit
80b675d4f5
12 changed files with 2145 additions and 0 deletions
|
|
@ -0,0 +1,157 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability, plan=1, tag=REFACTOR
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/plan_local_G05_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/code_review_cloud_G05_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed and the verification list substituted package checks for the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the two existing claimed-stall seams, S06 label/log boundary, and Node-only scope; add one process-global production collector set, isolated test registries, duplicate-construction coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G05.md` → `code_review_cloud_G05_1.log` and `PLAN-local-G05.md` → `plan_local_G05_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 emits exactly one Node response-stall counter observation, duration sample, and safe structured log from the normalized and tunnel stall-finalization seams using one process-global production collector set and only bounded execution-path, health, classification, and fence values.
|
||||
- [ ] REFACTOR-2 proves request-stalled-but-provider-available and provider-unhealthy outcomes on deterministic normalized/tunnel fixtures, verifies exact metric families/labels and repeated Node construction, and proves request/session/raw prompt/response plus other high-cardinality values are absent from the dedicated log and metric labels; synchronize the matching contracts/spec.
|
||||
- [ ] Run every focused, package, race, vet, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G05_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_local_G05_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify both claimed-stall branches call one observer only after immutable fence/probe evidence exists and that terminal behavior is unchanged.
|
||||
- Verify default collectors are registered once at package lifetime, every `Node` reuses them, and private-registerer tests cannot mutate or duplicate the default registry.
|
||||
- Verify metric family names and label names/values are closed and contain no identifier fallback.
|
||||
- Verify the dedicated log carries only bounded classifications plus numeric duration and that the test seeds and rejects high-cardinality/raw sentinels.
|
||||
- Verify normalized and provider-tunnel fixtures cover available/request-stalled and unavailable/provider-unhealthy outcomes without sleeps.
|
||||
- Verify contract/spec edits describe only implemented observability and do not mark Edge overlay/recovery complete.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'`
|
||||
|
||||
Expected: PASS every iteration and all four named path/health subtests execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=1 ./packages/go/execution ./apps/node/...`
|
||||
|
||||
Expected: PASS under the Node local profile.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -race -count=3 ./apps/node/internal/node -run 'LivenessObservability|Watchdog|HealthEvidence'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go vet ./packages/go/execution ./apps/node/...`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
Expected: PASS using separate Edge/Node entrypoints for registration, two same-session messages, one post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 6
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Node Response-Stall Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this Node liveness-observability slice, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G05.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The Node already produces one fenced `response_stalled` terminal with joined health evidence for normalized and tunnel attempts, but operators cannot count or time those stalls without inspecting request-scoped events. This slice adds a bounded metric and structured-log contract at the existing exactly-once stall finalization seam without changing execution, wire, or retry behavior.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/plan_local_G05_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/code_review_cloud_G05_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed and the verification list substituted package checks for the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the two existing claimed-stall seams, S06 label/log boundary, and Node-only scope; add one process-global production collector set, isolated test registries, duplicate-construction coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `apps/node/internal/node/node.go`
|
||||
- `apps/node/internal/node/liveness_watchdog.go`
|
||||
- `apps/node/internal/node/liveness_health_evidence.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_test.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_lifecycle_test.go`
|
||||
- `apps/node/internal/node/liveness_health_evidence_test.go`
|
||||
- `apps/node/internal/node/provider_tunnel_liveness_test.go`
|
||||
- `apps/edge/internal/openai/usage_metrics.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `packages/go/observability/observability.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/node-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
|
||||
- `scripts/dev/edge.sh`
|
||||
- `scripts/dev/node.sh`
|
||||
- `scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require Node stall count/duration plus fence/probe result for deterministic normalized-run and tunnel stalls, with request/session/raw prompt/response and high-cardinality values absent from metric labels and the dedicated structured log.
|
||||
- Those rows define REFACTOR-1's closed label vocabulary and REFACTOR-2's two-path health matrix and negative leakage assertions.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; the user supplied starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be`, which matched the checkout during planning.
|
||||
- The local Node profile supplied `go version && go env GOMOD`, `go test -count=1 ./packages/go/execution ./apps/node/...`, and `git diff --check`. Read-only preflight returned `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s1/go.mod`, and executable `scripts/dev/edge.sh`, `scripts/dev/node.sh`, and `scripts/dev/edge-node-reconnect-diagnostic.sh`. Planning baseline `go test -count=1 ./apps/node/internal/node -run 'Liveness|Watchdog|HealthEvidence|ProviderTunnelLiveness'` passed.
|
||||
- The current stall seams are `liveness_watchdog.go:213-224` and `liveness_watchdog.go:304-311`; both already follow a successful fence claim and produce exactly one terminal. Confidence is high because the change can observe the immutable `stallObservation` without adding lifecycle state.
|
||||
- No external verification is required. The repository's manual clocks and fake normalized/tunnel providers provide deterministic local evidence. The testing rule additionally requires the real Edge/Node entrypoints; `scripts/dev/edge-node-reconnect-diagnostic.sh` creates temporary mock configs, starts `scripts/dev/edge.sh` and `scripts/dev/node.sh` separately, proves registration, three ordered runs including two in one session, `/nodes`, `/capabilities`, `/transport`, reconnect, Node-to-Edge payload equality, and exactly-once terminal ordering.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing watchdog tests verify terminal metadata and races but do not gather Prometheus series or capture a dedicated safe structured log.
|
||||
- No test proves normalized and tunnel attempts use the same bounded labels for both `request_stalled`/available and `provider_unhealthy`/unavailable evidence.
|
||||
- No test rejects run, attempt, request, session, adapter, target, prompt, response, or credential values from the new label/log surface.
|
||||
- No test proves constructing multiple `Node` instances reuses one process-global production collector set instead of registering the same metric names repeatedly.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. No existing symbol is renamed or removed; `Node` gains one internal observer field initialized by `New` and replaceable only by same-package tests.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child is the stable Node producer: one immutable `stallObservation` is mapped to one counter, one duration histogram, and one dedicated log for both execution paths. It has no active predecessor because the watchdog/health-evidence producers it consumes are already present at the supplied HEAD.
|
||||
- `12+08_health_overlay_observability` and `13+10_recovery_observability` own Edge overlay and recovery evidence and do not share Node files.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change stall detection, timer reset, fence/probe ordering, wire metadata, retryability, Edge ingestion, provider overlay, recovery selection, dashboards, or config. Do not add node/run/attempt/provider/session/adapter/target identifiers as metric labels or dedicated log fields.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,1,2,0,1)`, grade G05, route `local-fit` -> `PLAN-local-G05.md`.
|
||||
- Review closure true; scores `(1,1,2,0,1)`, grade G05, route `official-review` -> `CODE_REVIEW-cloud-G05.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `concurrent_consistency`, `variant_product` (2). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 emits exactly one Node response-stall counter observation, duration sample, and safe structured log from the normalized and tunnel stall-finalization seams using one process-global production collector set and only bounded execution-path, health, classification, and fence values.
|
||||
- [ ] REFACTOR-2 proves request-stalled-but-provider-available and provider-unhealthy outcomes on deterministic normalized/tunnel fixtures, verifies exact metric families/labels and repeated Node construction, and proves request/session/raw prompt/response plus other high-cardinality values are absent from the dedicated log and metric labels; synchronize the matching contracts/spec.
|
||||
- [ ] Run every focused, package, race, vet, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Emit bounded Node stall metrics and logs
|
||||
|
||||
**Problem:** `apps/node/internal/node/liveness_watchdog.go:213-224` and `apps/node/internal/node/liveness_watchdog.go:304-311` finalize typed stall evidence but expose it only through request-scoped terminals. Operators cannot count or time stalls by safe fence/probe axes.
|
||||
|
||||
**Solution:** Add a test-injectable `nodeLivenessObserver`. Register one package-level production collector set exactly once with the default Prometheus registerer and reuse it from every `Node`; a constructor that accepts an explicit `prometheus.Registerer` creates isolated collectors only for tests. Never call `promauto.New*` or `MustRegister` from `Node.New` or per attempt. Emit `iop_node_response_stalls_total{execution_path,provider_health,liveness_classification,attempt_fence}` and `iop_node_response_stall_duration_seconds` with the identical four-label set. Normalize every label through closed allowlists (`normalized|provider_tunnel|unknown`, the three health/classification pairs, and `confirmed|unconfirmed|unknown`). Write `node_response_stall_observation` with only those labels and numeric `idle_duration_ms`. Install the reusable observer on `Node` and invoke it immediately after `stallObservationFrom` in each already-claimed stall branch; observer failure or disabled logging must never change terminal delivery.
|
||||
|
||||
Before (`apps/node/internal/node/liveness_watchdog.go:213`):
|
||||
|
||||
```go
|
||||
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
n.liveness.Observe("normalized", obs)
|
||||
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
|
||||
```
|
||||
|
||||
Apply the same call with `provider_tunnel` before `emitClaimedTerminal` at line 309. The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`; do not add an alternate metrics server.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/node/internal/node/node.go`: hold the internal observer and initialize its production collectors/logger without changing the public constructor signature.
|
||||
- [ ] `apps/node/internal/node/liveness_watchdog.go`: invoke the observer once in each claimed normalized/tunnel stall path.
|
||||
- [ ] `apps/node/internal/node/liveness_observability.go`: define collectors, closed normalization, safe log fields, and the test-injection constructor.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2; do not alter existing lifecycle fixtures except to reuse their manual clocks/providers.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` must pass every iteration and report both paths.
|
||||
|
||||
### [REFACTOR-2] Prove the evidence matrix and synchronize contracts
|
||||
|
||||
**Problem:** `apps/node/internal/node/liveness_health_evidence.go:43-70` intentionally includes run/attempt/adapter/target in terminal metadata, so copying that map into metrics or the dedicated log would violate S06 even though the wire terminal itself is valid. Existing tests do not guard this new boundary.
|
||||
|
||||
**Solution:** Add a two-path table using the production watchdog seams and private Prometheus registry/zap observer. Cover available/request-stalled and unavailable/provider-unhealthy with confirmed and unconfirmed fences where deterministic. Assert counter delta one, histogram count/duration, exact label names and allowlisted values, one dedicated log per claimed stall, and absence of sentinel high-card/raw values from labels and encoded log fields. Construct multiple default `Node` values in one process and assert no duplicate-registration panic while a private registry remains isolated. Document the new names, label vocabulary, process-global collector lifetime, exact-once point, and prohibition boundary while preserving the existing richer internal terminal metadata contract.
|
||||
|
||||
Before (`apps/node/internal/node/liveness_health_evidence.go:56`):
|
||||
|
||||
```go
|
||||
metadata := map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled),
|
||||
"run_id": runID,
|
||||
"attempt_id": runID,
|
||||
```
|
||||
|
||||
After (observability projection, not terminal metadata replacement):
|
||||
|
||||
```go
|
||||
labels := normalizeNodeLivenessLabels(path, obs)
|
||||
observer.stalls.WithLabelValues(labels...).Inc()
|
||||
observer.duration.WithLabelValues(labels...).Observe(obs.idle.Seconds())
|
||||
```
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/node/internal/node/liveness_observability_test.go`: add deterministic normalized/tunnel metric, duration, exact-once, allowlist, and log-leakage cases.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: add the Node operational evidence schema and explicitly separate it from terminal metadata.
|
||||
- [ ] `agent-contract/inner/edge-node-runtime-wire.md`: record that the new observation is Node-local and does not widen wire metadata.
|
||||
- [ ] `agent-spec/runtime/edge-node-execution.md`: mark the current Node stall metric/log behavior and verification evidence.
|
||||
|
||||
**Test Strategy:** Create `TestNodeLivenessObservability` subtests for `normalized/request-stalled`, `normalized/provider-unhealthy`, `provider_tunnel/request-stalled`, and `provider_tunnel/provider-unhealthy`, plus a repeated-default-construction row. Seed run/session/adapter/target/prompt/response/credential sentinels and inspect gathered DTO labels plus zap fields/message text for absence.
|
||||
|
||||
**Verification:** the focused test above plus the Node package/race commands below must pass with no zero-match test run.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/node/internal/node/node.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_watchdog.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_observability.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/inner/edge-node-runtime-wire.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/CODE_REVIEW-cloud-G05.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — PASS every iteration and all four named path/health subtests execute.
|
||||
2. `go test -count=1 ./packages/go/execution ./apps/node/...` — PASS under the Node local profile.
|
||||
3. `go test -race -count=3 ./apps/node/internal/node -run 'LivenessObservability|Watchdog|HealthEvidence'` — PASS with no race report.
|
||||
4. `go vet ./packages/go/execution ./apps/node/...` — no diagnostics.
|
||||
5. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — PASS using separate `scripts/dev/edge.sh` and `scripts/dev/node.sh` processes; registration, the first two same-session messages, post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, and exactly-once terminal ordering are all verified. This is the required repository-native full-cycle diagnostic, not an auxiliary smoke substitute.
|
||||
6. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability, plan=0, tag=REFACTOR
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G05.md` → `code_review_cloud_G05_0.log` and `PLAN-local-G05.md` → `plan_local_G05_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 emits exactly one Node response-stall counter observation, duration sample, and safe structured log from the normalized and tunnel stall-finalization seams using only bounded execution-path, health, classification, and fence values.
|
||||
- [ ] REFACTOR-2 proves request-stalled-but-provider-available and provider-unhealthy outcomes on deterministic normalized/tunnel fixtures, verifies exact metric families/labels, and proves request/session/raw prompt/response plus other high-cardinality values are absent from the dedicated log and metric labels; synchronize the matching contracts/spec.
|
||||
- [ ] Run every focused, package, race, vet, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G05_0.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_local_G05_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify both claimed-stall branches call one observer only after immutable fence/probe evidence exists and that terminal behavior is unchanged.
|
||||
- Verify metric family names and label names/values are closed and contain no identifier fallback.
|
||||
- Verify the dedicated log carries only bounded classifications plus numeric duration and that the test seeds and rejects high-cardinality/raw sentinels.
|
||||
- Verify normalized and provider-tunnel fixtures cover available/request-stalled and unavailable/provider-unhealthy outcomes without sleeps.
|
||||
- Verify contract/spec edits describe only implemented observability and do not mark Edge overlay/recovery complete.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'`
|
||||
|
||||
Expected: PASS every iteration and all four named path/health subtests execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=1 ./packages/go/execution ./apps/node/...`
|
||||
|
||||
Expected: PASS under the Node local profile.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -race -count=3 ./apps/node/internal/node -run 'LivenessObservability|Watchdog|HealthEvidence'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go vet ./packages/go/execution ./apps/node/...`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,171 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/11_node_liveness_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Node Response-Stall Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this Node liveness-observability slice, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G05.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The Node already produces one fenced `response_stalled` terminal with joined health evidence for normalized and tunnel attempts, but operators cannot count or time those stalls without inspecting request-scoped events. This slice adds a bounded metric and structured-log contract at the existing exactly-once stall finalization seam without changing execution, wire, or retry behavior.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `apps/node/internal/node/node.go`
|
||||
- `apps/node/internal/node/liveness_watchdog.go`
|
||||
- `apps/node/internal/node/liveness_health_evidence.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_test.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_lifecycle_test.go`
|
||||
- `apps/node/internal/node/liveness_health_evidence_test.go`
|
||||
- `apps/node/internal/node/provider_tunnel_liveness_test.go`
|
||||
- `apps/edge/internal/openai/usage_metrics.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `packages/go/observability/observability.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/node-smoke.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require Node stall count/duration plus fence/probe result for deterministic normalized-run and tunnel stalls, with request/session/raw prompt/response and high-cardinality values absent from metric labels and the dedicated structured log.
|
||||
- Those rows define REFACTOR-1's closed label vocabulary and REFACTOR-2's two-path health matrix and negative leakage assertions.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; the user supplied starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be`, which matched the checkout during planning.
|
||||
- The local Node profile supplied `go version && go env GOMOD`, `go test -count=1 ./packages/go/execution ./apps/node/...`, and `git diff --check`. Planning baseline `go test -count=1 ./apps/node/internal/node -run 'Liveness|Watchdog|HealthEvidence|ProviderTunnelLiveness'` passed.
|
||||
- The current stall seams are `liveness_watchdog.go:213-224` and `liveness_watchdog.go:304-311`; both already follow a successful fence claim and produce exactly one terminal. Confidence is high because the change can observe the immutable `stallObservation` without adding lifecycle state.
|
||||
- No external verification is required. The repository's manual clocks and fake normalized/tunnel providers provide deterministic local evidence.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing watchdog tests verify terminal metadata and races but do not gather Prometheus series or capture a dedicated safe structured log.
|
||||
- No test proves normalized and tunnel attempts use the same bounded labels for both `request_stalled`/available and `provider_unhealthy`/unavailable evidence.
|
||||
- No test rejects run, attempt, request, session, adapter, target, prompt, response, or credential values from the new label/log surface.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. No existing symbol is renamed or removed; `Node` gains one internal observer field initialized by `New` and replaceable only by same-package tests.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child is the stable Node producer: one immutable `stallObservation` is mapped to one counter, one duration histogram, and one dedicated log for both execution paths. It has no active predecessor because the watchdog/health-evidence producers it consumes are already present at the supplied HEAD.
|
||||
- `12+08_health_overlay_observability` and `13+10_recovery_observability` own Edge overlay and recovery evidence and do not share Node files.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change stall detection, timer reset, fence/probe ordering, wire metadata, retryability, Edge ingestion, provider overlay, recovery selection, dashboards, or config. Do not add node/run/attempt/provider/session/adapter/target identifiers as metric labels or dedicated log fields.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,1,2,0,1)`, grade G05, route `local-fit` -> `PLAN-local-G05.md`.
|
||||
- Review closure true; scores `(1,1,2,0,1)`, grade G05, route `official-review` -> `CODE_REVIEW-cloud-G05.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `concurrent_consistency`, `variant_product` (2). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 emits exactly one Node response-stall counter observation, duration sample, and safe structured log from the normalized and tunnel stall-finalization seams using only bounded execution-path, health, classification, and fence values.
|
||||
- [ ] REFACTOR-2 proves request-stalled-but-provider-available and provider-unhealthy outcomes on deterministic normalized/tunnel fixtures, verifies exact metric families/labels, and proves request/session/raw prompt/response plus other high-cardinality values are absent from the dedicated log and metric labels; synchronize the matching contracts/spec.
|
||||
- [ ] Run every focused, package, race, vet, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Emit bounded Node stall metrics and logs
|
||||
|
||||
**Problem:** `apps/node/internal/node/liveness_watchdog.go:213-224` and `apps/node/internal/node/liveness_watchdog.go:304-311` finalize typed stall evidence but expose it only through request-scoped terminals. Operators cannot count or time stalls by safe fence/probe axes.
|
||||
|
||||
**Solution:** Add a test-injectable `nodeLivenessObserver` backed by the default Prometheus registerer in production and private collectors in tests. Emit `iop_node_response_stalls_total{execution_path,provider_health,liveness_classification,attempt_fence}` and `iop_node_response_stall_duration_seconds` with the identical four-label set. Normalize every label through closed allowlists (`normalized|provider_tunnel|unknown`, the three health/classification pairs, and `confirmed|unconfirmed|unknown`). Write `node_response_stall_observation` with only those labels and numeric `idle_duration_ms`. Install the observer on `Node` and invoke it immediately after `stallObservationFrom` in each already-claimed stall branch; observer failure or disabled logging must never change terminal delivery.
|
||||
|
||||
Before (`apps/node/internal/node/liveness_watchdog.go:213`):
|
||||
|
||||
```go
|
||||
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
n.liveness.Observe("normalized", obs)
|
||||
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
|
||||
```
|
||||
|
||||
Apply the same call with `provider_tunnel` before `emitClaimedTerminal` at line 309. The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`; do not add an alternate metrics server.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/node/internal/node/node.go`: hold the internal observer and initialize its production collectors/logger without changing the public constructor signature.
|
||||
- [ ] `apps/node/internal/node/liveness_watchdog.go`: invoke the observer once in each claimed normalized/tunnel stall path.
|
||||
- [ ] `apps/node/internal/node/liveness_observability.go`: define collectors, closed normalization, safe log fields, and the test-injection constructor.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2; do not alter existing lifecycle fixtures except to reuse their manual clocks/providers.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` must pass every iteration and report both paths.
|
||||
|
||||
### [REFACTOR-2] Prove the evidence matrix and synchronize contracts
|
||||
|
||||
**Problem:** `apps/node/internal/node/liveness_health_evidence.go:43-70` intentionally includes run/attempt/adapter/target in terminal metadata, so copying that map into metrics or the dedicated log would violate S06 even though the wire terminal itself is valid. Existing tests do not guard this new boundary.
|
||||
|
||||
**Solution:** Add a two-path table using the production watchdog seams and private Prometheus registry/zap observer. Cover available/request-stalled and unavailable/provider-unhealthy with confirmed and unconfirmed fences where deterministic. Assert counter delta one, histogram count/duration, exact label names and allowlisted values, one dedicated log per claimed stall, and absence of sentinel high-card/raw values from labels and encoded log fields. Document the new names, label vocabulary, exact-once point, and prohibition boundary while preserving the existing richer internal terminal metadata contract.
|
||||
|
||||
Before (`apps/node/internal/node/liveness_health_evidence.go:56`):
|
||||
|
||||
```go
|
||||
metadata := map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled),
|
||||
"run_id": runID,
|
||||
"attempt_id": runID,
|
||||
```
|
||||
|
||||
After (observability projection, not terminal metadata replacement):
|
||||
|
||||
```go
|
||||
labels := normalizeNodeLivenessLabels(path, obs)
|
||||
observer.stalls.WithLabelValues(labels...).Inc()
|
||||
observer.duration.WithLabelValues(labels...).Observe(obs.idle.Seconds())
|
||||
```
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/node/internal/node/liveness_observability_test.go`: add deterministic normalized/tunnel metric, duration, exact-once, allowlist, and log-leakage cases.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: add the Node operational evidence schema and explicitly separate it from terminal metadata.
|
||||
- [ ] `agent-contract/inner/edge-node-runtime-wire.md`: record that the new observation is Node-local and does not widen wire metadata.
|
||||
- [ ] `agent-spec/runtime/edge-node-execution.md`: mark the current Node stall metric/log behavior and verification evidence.
|
||||
|
||||
**Test Strategy:** Create `TestNodeLivenessObservability` subtests for `normalized/request-stalled`, `normalized/provider-unhealthy`, `provider_tunnel/request-stalled`, and `provider_tunnel/provider-unhealthy`. Seed run/session/adapter/target/prompt/response/credential sentinels and inspect gathered DTO labels plus zap fields/message text for absence.
|
||||
|
||||
**Verification:** the focused test above plus the Node package/race commands below must pass with no zero-match test run.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/node/internal/node/node.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_watchdog.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_observability.go` | REFACTOR-1 |
|
||||
| `apps/node/internal/node/liveness_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/inner/edge-node-runtime-wire.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/11_node_liveness_observability/CODE_REVIEW-cloud-G05.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `go test -count=20 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — PASS every iteration and all four named path/health subtests execute.
|
||||
2. `go test -count=1 ./packages/go/execution ./apps/node/...` — PASS under the Node local profile.
|
||||
3. `go test -race -count=3 ./apps/node/internal/node -run 'LivenessObservability|Watchdog|HealthEvidence'` — PASS with no race report.
|
||||
4. `go vet ./packages/go/execution ./apps/node/...` — no diagnostics.
|
||||
5. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability, plan=1, tag=REFACTOR
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/plan_cloud_G08_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/code_review_cloud_G08_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed despite frequent `Service.New` use in one process, and the verification list treated provider-capacity smoke as sufficient without the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the `08+07_health_overlay` dependency, post-lock immutable transition projection, S06 stale/recovery matrix, and snapshot oracle; add process-global production collectors, isolated test registries, repeated-service coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 projects every predecessor health-evidence decision through one process-global production collector set into bounded Edge counters and a safe structured log after releasing the queue lock, without changing validation or overlay state.
|
||||
- [ ] REFACTOR-2 proves normalized/tunnel provider-unhealthy, stale rejection, later probe recovery, and repeated Service construction through metrics/logs plus the production provider snapshot, and proves request/session/raw prompt/response and all high-cardinality identifiers are absent; synchronize matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, provider-capacity auxiliary smoke, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify the observer consumes only the predecessor's authoritative immutable transition result and does not repeat binding, generation, sequence, source, or health validation.
|
||||
- Verify default collectors are registered once at package lifetime, every `Service`/queue manager reuses them, and private-registerer tests remain isolated from the default registry.
|
||||
- Verify every metric/log call occurs after `modelQueueManager.mu` is released and observer failure cannot block queue release, pump, or snapshot progress.
|
||||
- Verify metric family names and every label value are closed, and the dedicated event omits provider/node/run/request/session/lease/adapter/target identity and raw payload or credentials.
|
||||
- Verify normalized and tunnel fixtures cover applied unavailable, stale available rejection, and later exact-target probe recovery against the public production snapshot.
|
||||
- Verify contract/spec changes describe only implemented post-decision observability and retain the predecessor as owner of overlay state and admission behavior.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `test -f agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`
|
||||
|
||||
Expected: predecessor PASS evidence exists before implementation.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'`
|
||||
|
||||
Expected: PASS every iteration and normalized/tunnel applied, stale, and recovered rows execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -count=1 ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: PASS under the Edge local profile.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/bootstrap -run 'ProviderHealthObservability|ProviderHealthOverlay|Snapshot'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 6
|
||||
|
||||
Command: `./scripts/e2e-provider-capacity-smoke.sh`
|
||||
|
||||
Expected: auxiliary smoke PASS with the final provider counters drained and no overlay regression.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 7
|
||||
|
||||
Command: `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
Expected: PASS using separate Edge/Node entrypoints for registration, two same-session messages, one post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 8
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,201 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Edge Provider-Health Overlay Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this provider-health observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The predecessor creates the generation/sequence-fenced runtime health overlay and exact-target probe recovery, but intentionally excludes metrics. Operators need bounded evidence that distinguishes an applied unhealthy transition, rejected stale evidence, and an applied recovery while the existing provider snapshot remains the identity-bearing source of truth.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/plan_cloud_G08_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/code_review_cloud_G08_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed despite frequent `Service.New` use in one process, and the verification list treated provider-capacity smoke as sufficient without the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the `08+07_health_overlay` dependency, post-lock immutable transition projection, S06 stale/recovery matrix, and snapshot oracle; add process-global production collectors, isolated test registries, repeated-service coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/PLAN-cloud-G09.md`
|
||||
- `apps/edge/internal/bootstrap/runtime.go`
|
||||
- `apps/edge/internal/service/service.go`
|
||||
- `apps/edge/internal/service/model_queue_types.go`
|
||||
- `apps/edge/internal/service/model_queue_release.go`
|
||||
- `apps/edge/internal/service/model_queue_snapshot.go`
|
||||
- `apps/edge/internal/service/provider_tunnel.go`
|
||||
- `apps/edge/internal/service/model_queue_test_support_test.go`
|
||||
- `apps/edge/internal/service/status_provider_test.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-spec/runtime/provider-pool-config-refresh.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/edge-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
|
||||
- `scripts/dev/edge.sh`
|
||||
- `scripts/dev/node.sh`
|
||||
- `scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require Edge metric/log evidence and provider snapshot projection to distinguish provider-unhealthy, stale evidence rejection, and a later recovered state without high-cardinality or raw request/response data.
|
||||
- Those rows require one transition result object shared by metrics and logs, plus a deterministic normalized/tunnel table that queries the production snapshot after each accepted or rejected observation.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
|
||||
- Planning baseline `go test -count=1 ./apps/edge/internal/service -run 'ProviderSnapshot|ListNodeSnapshots|Reconnect'` passed. Read-only preflight returned `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s1/go.mod`, and executable Edge/Node dev entrypoints plus the reconnect diagnostic. The Edge profile supplies package tests; the predecessor already requires provider-capacity auxiliary smoke evidence.
|
||||
- `08+07_health_overlay` is active and its `complete.log` is missing. Its plan promises one queue-locked overlay transition result for normalized/tunnel terminal evidence and CAPABILITIES probe recovery; this child must consume that result rather than reimplement validation.
|
||||
- No external host is required. Service fixtures and fake transport clients are the semantic oracle, `scripts/e2e-provider-capacity-smoke.sh` is auxiliary provider-pool evidence, and `scripts/dev/edge-node-reconnect-diagnostic.sh` separately supplies the required real Edge/Node entrypoint cycle with temporary mock configs, ordered message relay, commands, and reconnect. Confidence is medium-high because exact observer placement depends on the predecessor's final transition helper but its ownership and state matrix are closed.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Current snapshots read config/connectivity only; the predecessor will add overlay assertions but explicitly excludes metrics.
|
||||
- No existing test captures applied/rejected transition logs or gathers a bounded health-evidence metric.
|
||||
- No existing test proves a stale observation increments only a rejection series while leaving the unavailable snapshot unchanged, or that a later probe recovery changes both transition evidence and the snapshot.
|
||||
- No test proves repeated `Service.New` construction reuses one process-global production collector set instead of duplicate-registering the same metric names.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. Do not rename or remove the predecessor's overlay symbols. Add one internal observer interface/field and a startup logger setter; update only bootstrap construction and same-package fixtures.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Stable child output: `08+07_health_overlay` owns validation, sequence/generation fencing, atomic state transition, admission, and snapshot projection. Its PASS is required and is currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` missing).
|
||||
- This child owns only the immutable post-decision observation projection. It stages the result under the queue lock but performs metric/log I/O after unlocking, so it cannot alter overlay correctness or queue progress.
|
||||
- Node stall evidence and OpenAI recovery-owner evidence remain in siblings 11 and 13.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change wire fields, evidence validation, provider binding, observation sequence ordering, overlay state, candidate eligibility, probe scheduling, queue release, recovery policy, or config health. Provider/node/run/request/session/lease/adapter/target identifiers and raw payload/credential values are excluded from metric labels and the dedicated log; exact provider identity remains available only through the existing snapshot surface.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
|
||||
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 projects every predecessor health-evidence decision through one process-global production collector set into bounded Edge counters and a safe structured log after releasing the queue lock, without changing validation or overlay state.
|
||||
- [ ] REFACTOR-2 proves normalized/tunnel provider-unhealthy, stale rejection, later probe recovery, and repeated Service construction through metrics/logs plus the production provider snapshot, and proves request/session/raw prompt/response and all high-cardinality identifiers are absent; synchronize matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, provider-capacity auxiliary smoke, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Observe the authoritative overlay decision
|
||||
|
||||
**Problem:** `apps/edge/internal/service/model_queue_types.go:465-481` has only capacity/connectivity resource state today, while `apps/edge/internal/service/model_queue_snapshot.go:50-71` directly projects effective provider values. The predecessor will add the authoritative overlay transition under the queue lock but explicitly excludes metrics, so observing wire metadata independently would duplicate and potentially disagree with its stale/binding decision.
|
||||
|
||||
**Solution:** Consume the predecessor's immutable transition result at the exact helper that classifies `applied`, `rejected_stale`, `rejected_binding`, `rejected_ambiguous`, or `inconclusive`. Register one package-level production collector set exactly once with the default Prometheus registerer and reuse it from every `Service`/queue manager; an explicit-registerer constructor creates isolated collectors only for tests. Never call `promauto.New*` or `MustRegister` from `Service.New`, observer setters, or evidence handling. Add `iop_edge_provider_health_evidence_total{source,evidence_health,decision}` and `iop_edge_provider_health_transitions_total{from_health,to_health}` with closed mappings: source `stall|probe|unknown`; health `available|unavailable|unknown`; transition values `available|unavailable|unknown`; no identity labels. Emit `edge_provider_health_observation` with only those enums and a `state_changed` boolean. Stage the result while holding `modelQueueManager.mu`, then call the observer only after unlock; metrics/log failures are best-effort and must not block release/pump.
|
||||
|
||||
Before (`apps/edge/internal/service/model_queue_snapshot.go:50`):
|
||||
|
||||
```go
|
||||
snaps = append(snaps, &iop.ProviderSnapshot{
|
||||
Status: effectiveStatus(connected),
|
||||
Health: effectiveHealth(connected, prov.Health),
|
||||
```
|
||||
|
||||
After predecessor plus this slice (observation remains outside snapshot construction):
|
||||
|
||||
```go
|
||||
result := m.applyProviderHealthEvidenceLocked(evidence)
|
||||
// unlock before any observer call
|
||||
m.healthObserver.Observe(result)
|
||||
```
|
||||
|
||||
The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`. Production uses the default registry; tests inject private collectors and a zap observer.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/service/model_queue_types.go`: attach the observer to the queue manager without widening provider resource identity or overlay state.
|
||||
- [ ] `apps/edge/internal/service/model_queue_release.go`: capture the predecessor transition/rejection result and emit after the critical section for normalized terminal, tunnel terminal, and probe evidence paths.
|
||||
- [ ] `apps/edge/internal/service/service.go`: initialize the default observer and expose a startup-only logger/test injection seam without changing `New` callers.
|
||||
- [ ] `apps/edge/internal/bootstrap/runtime.go`: bind the Edge runtime logger to the service observer before transport handlers start.
|
||||
- [ ] `apps/edge/internal/service/provider_health_observability.go`: define metric collectors, closed label normalization, safe log projection, and best-effort observer behavior.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2. Do not create a second overlay state or validate evidence in the observer.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` must pass and execute every decision row.
|
||||
|
||||
### [REFACTOR-2] Prove stale rejection, unhealthy projection, and recovery
|
||||
|
||||
**Problem:** `apps/edge/internal/service/status_provider_test.go:635-690` proves snapshots read resource state, but there is no liveness overlay metric/log oracle. A metric-only test could pass while stale evidence mutates the snapshot or while recovery never becomes operator-visible.
|
||||
|
||||
**Solution:** Drive the predecessor's production normalized and tunnel reception handlers with current bound unavailable evidence, a duplicate/lower-sequence stale available observation, and a later higher-sequence exact-target probe available result. At each step assert the metric decision/transition delta, one safe structured event, and the public `ListNodeSnapshots` health/status. Construct multiple default services in one process and prove no duplicate-registration panic while private registries remain isolated. Use high-card/raw sentinels in node/provider/run/session/adapter/target and message/body fields and assert none are present in gathered labels or dedicated log fields/messages.
|
||||
|
||||
Before (`apps/edge/internal/service/model_queue_snapshot.go:201`):
|
||||
|
||||
```go
|
||||
func effectiveHealth(connected bool, health string) string {
|
||||
if connected {
|
||||
return health
|
||||
}
|
||||
```
|
||||
|
||||
After predecessor behavior, verified by this child:
|
||||
|
||||
```go
|
||||
// current unavailable -> snapshot unavailable
|
||||
// stale available -> rejection metric, snapshot still unavailable
|
||||
// later current available probe -> recovery metric, snapshot available
|
||||
```
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/service/provider_health_observability_test.go`: add normalized/tunnel applied-unhealthy, stale-rejection, recovered-snapshot, exact-once, lock-safety, label allowlist, and log leakage tables.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: specify Edge health evidence/transition metric and safe-log semantics.
|
||||
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: document that runtime overlay observations are separate from immutable config health and carry no provider identity labels.
|
||||
- [ ] `agent-spec/runtime/edge-node-execution.md`: record reception-to-overlay observability and stale/recovery behavior.
|
||||
- [ ] `agent-spec/runtime/provider-pool-config-refresh.md`: record the effective snapshot projection and operational evidence boundary.
|
||||
|
||||
**Test Strategy:** Create `TestProviderHealthObservability` with normalized and provider-tunnel subtests. Each uses the predecessor's real binding/generation/sequence path, queries the actual snapshot, gathers private Prometheus collectors, and captures zap entries. Include a repeated-default-service row, a blocking observer fixture to prove it is invoked after `modelQueueManager.mu` is released, plus duplicate terminal/probe rows to prove exactly-once transitions.
|
||||
|
||||
**Verification:** the focused test, service race suite, and provider-capacity smoke below must pass with no zero-match command.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `08+07_health_overlay` must produce `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`; it is active and missing at plan creation.
|
||||
2. Implement REFACTOR-1 before REFACTOR-2. Do not instrument raw wire reception independently of the predecessor's final transition decision.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/edge/internal/service/model_queue_types.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/model_queue_release.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/service.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/bootstrap/runtime.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/provider_health_observability.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/provider_health_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/inner/edge-config-runtime-refresh.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/provider-pool-config-refresh.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `test -f agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` — predecessor PASS evidence exists before implementation.
|
||||
2. `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS every iteration and normalized/tunnel applied, stale, and recovered rows execute.
|
||||
3. `go test -count=1 ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — PASS under the Edge local profile.
|
||||
4. `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/bootstrap -run 'ProviderHealthObservability|ProviderHealthOverlay|Snapshot'` — PASS with no race report.
|
||||
5. `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — no diagnostics.
|
||||
6. `./scripts/e2e-provider-capacity-smoke.sh` — auxiliary smoke PASS with the final provider counters drained and no overlay regression.
|
||||
7. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — PASS using separate `scripts/dev/edge.sh` and `scripts/dev/node.sh` processes; registration, the first two same-session messages, post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering are all verified. This is the required repository-native full-cycle diagnostic.
|
||||
8. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,158 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability, plan=0, tag=REFACTOR
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 projects every predecessor health-evidence decision into bounded Edge counters and a safe structured log after releasing the queue lock, without changing validation or overlay state.
|
||||
- [ ] REFACTOR-2 proves normalized/tunnel provider-unhealthy, stale rejection, and later probe recovery through metrics/logs plus the production provider snapshot, and proves request/session/raw prompt/response and all high-cardinality identifiers are absent; synchronize matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, provider-capacity smoke, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_0.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify the observer consumes only the predecessor's authoritative immutable transition result and does not repeat binding, generation, sequence, source, or health validation.
|
||||
- Verify every metric/log call occurs after `modelQueueManager.mu` is released and observer failure cannot block queue release, pump, or snapshot progress.
|
||||
- Verify metric family names and every label value are closed, and the dedicated event omits provider/node/run/request/session/lease/adapter/target identity and raw payload or credentials.
|
||||
- Verify normalized and tunnel fixtures cover applied unavailable, stale available rejection, and later exact-target probe recovery against the public production snapshot.
|
||||
- Verify contract/spec changes describe only implemented post-decision observability and retain the predecessor as owner of overlay state and admission behavior.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `test -f agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`
|
||||
|
||||
Expected: predecessor PASS evidence exists before implementation.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'`
|
||||
|
||||
Expected: PASS every iteration and normalized/tunnel applied, stale, and recovered rows execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -count=1 ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: PASS under the Edge local profile.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/bootstrap -run 'ProviderHealthObservability|ProviderHealthOverlay|Snapshot'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 6
|
||||
|
||||
Command: `./scripts/e2e-provider-capacity-smoke.sh`
|
||||
|
||||
Expected: PASS with the final provider counters drained and no overlay regression.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 7
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Edge Provider-Health Overlay Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this provider-health observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The predecessor creates the generation/sequence-fenced runtime health overlay and exact-target probe recovery, but intentionally excludes metrics. Operators need bounded evidence that distinguishes an applied unhealthy transition, rejected stale evidence, and an applied recovery while the existing provider snapshot remains the identity-bearing source of truth.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/PLAN-cloud-G09.md`
|
||||
- `apps/edge/internal/bootstrap/runtime.go`
|
||||
- `apps/edge/internal/service/service.go`
|
||||
- `apps/edge/internal/service/model_queue_types.go`
|
||||
- `apps/edge/internal/service/model_queue_release.go`
|
||||
- `apps/edge/internal/service/model_queue_snapshot.go`
|
||||
- `apps/edge/internal/service/provider_tunnel.go`
|
||||
- `apps/edge/internal/service/model_queue_test_support_test.go`
|
||||
- `apps/edge/internal/service/status_provider_test.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-spec/runtime/provider-pool-config-refresh.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/edge-smoke.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require Edge metric/log evidence and provider snapshot projection to distinguish provider-unhealthy, stale evidence rejection, and a later recovered state without high-cardinality or raw request/response data.
|
||||
- Those rows require one transition result object shared by metrics and logs, plus a deterministic normalized/tunnel table that queries the production snapshot after each accepted or rejected observation.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
|
||||
- Planning baseline `go test -count=1 ./apps/edge/internal/service -run 'ProviderSnapshot|ListNodeSnapshots|Reconnect'` passed. The Edge profile supplies package tests; the predecessor already requires provider-capacity full-cycle evidence.
|
||||
- `08+07_health_overlay` is active and its `complete.log` is missing. Its plan promises one queue-locked overlay transition result for normalized/tunnel terminal evidence and CAPABILITIES probe recovery; this child must consume that result rather than reimplement validation.
|
||||
- No external host is required. Service fixtures, fake transport clients, and `scripts/e2e-provider-capacity-smoke.sh` are repository-native evidence. Confidence is medium-high because exact observer placement depends on the predecessor's final transition helper but its ownership and state matrix are closed.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Current snapshots read config/connectivity only; the predecessor will add overlay assertions but explicitly excludes metrics.
|
||||
- No existing test captures applied/rejected transition logs or gathers a bounded health-evidence metric.
|
||||
- No existing test proves a stale observation increments only a rejection series while leaving the unavailable snapshot unchanged, or that a later probe recovery changes both transition evidence and the snapshot.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. Do not rename or remove the predecessor's overlay symbols. Add one internal observer interface/field and a startup logger setter; update only bootstrap construction and same-package fixtures.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Stable child output: `08+07_health_overlay` owns validation, sequence/generation fencing, atomic state transition, admission, and snapshot projection. Its PASS is required and is currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` missing).
|
||||
- This child owns only the immutable post-decision observation projection. It stages the result under the queue lock but performs metric/log I/O after unlocking, so it cannot alter overlay correctness or queue progress.
|
||||
- Node stall evidence and OpenAI recovery-owner evidence remain in siblings 11 and 13.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change wire fields, evidence validation, provider binding, observation sequence ordering, overlay state, candidate eligibility, probe scheduling, queue release, recovery policy, or config health. Provider/node/run/request/session/lease/adapter/target identifiers and raw payload/credential values are excluded from metric labels and the dedicated log; exact provider identity remains available only through the existing snapshot surface.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
|
||||
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 projects every predecessor health-evidence decision into bounded Edge counters and a safe structured log after releasing the queue lock, without changing validation or overlay state.
|
||||
- [ ] REFACTOR-2 proves normalized/tunnel provider-unhealthy, stale rejection, and later probe recovery through metrics/logs plus the production provider snapshot, and proves request/session/raw prompt/response and all high-cardinality identifiers are absent; synchronize matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, provider-capacity smoke, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Observe the authoritative overlay decision
|
||||
|
||||
**Problem:** `apps/edge/internal/service/model_queue_types.go:465-481` has only capacity/connectivity resource state today, while `apps/edge/internal/service/model_queue_snapshot.go:50-71` directly projects effective provider values. The predecessor will add the authoritative overlay transition under the queue lock but explicitly excludes metrics, so observing wire metadata independently would duplicate and potentially disagree with its stale/binding decision.
|
||||
|
||||
**Solution:** Consume the predecessor's immutable transition result at the exact helper that classifies `applied`, `rejected_stale`, `rejected_binding`, `rejected_ambiguous`, or `inconclusive`. Add `iop_edge_provider_health_evidence_total{source,evidence_health,decision}` and `iop_edge_provider_health_transitions_total{from_health,to_health}` with closed mappings: source `stall|probe|unknown`; health `available|unavailable|unknown`; transition values `available|unavailable|unknown`; no identity labels. Emit `edge_provider_health_observation` with only those enums and a `state_changed` boolean. Stage the result while holding `modelQueueManager.mu`, then call the observer only after unlock; metrics/log failures are best-effort and must not block release/pump.
|
||||
|
||||
Before (`apps/edge/internal/service/model_queue_snapshot.go:50`):
|
||||
|
||||
```go
|
||||
snaps = append(snaps, &iop.ProviderSnapshot{
|
||||
Status: effectiveStatus(connected),
|
||||
Health: effectiveHealth(connected, prov.Health),
|
||||
```
|
||||
|
||||
After predecessor plus this slice (observation remains outside snapshot construction):
|
||||
|
||||
```go
|
||||
result := m.applyProviderHealthEvidenceLocked(evidence)
|
||||
// unlock before any observer call
|
||||
m.healthObserver.Observe(result)
|
||||
```
|
||||
|
||||
The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, and `go.uber.org/zap`. Production uses the default registry; tests inject private collectors and a zap observer.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/service/model_queue_types.go`: attach the observer to the queue manager without widening provider resource identity or overlay state.
|
||||
- [ ] `apps/edge/internal/service/model_queue_release.go`: capture the predecessor transition/rejection result and emit after the critical section for normalized terminal, tunnel terminal, and probe evidence paths.
|
||||
- [ ] `apps/edge/internal/service/service.go`: initialize the default observer and expose a startup-only logger/test injection seam without changing `New` callers.
|
||||
- [ ] `apps/edge/internal/bootstrap/runtime.go`: bind the Edge runtime logger to the service observer before transport handlers start.
|
||||
- [ ] `apps/edge/internal/service/provider_health_observability.go`: define metric collectors, closed label normalization, safe log projection, and best-effort observer behavior.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2. Do not create a second overlay state or validate evidence in the observer.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` must pass and execute every decision row.
|
||||
|
||||
### [REFACTOR-2] Prove stale rejection, unhealthy projection, and recovery
|
||||
|
||||
**Problem:** `apps/edge/internal/service/status_provider_test.go:635-690` proves snapshots read resource state, but there is no liveness overlay metric/log oracle. A metric-only test could pass while stale evidence mutates the snapshot or while recovery never becomes operator-visible.
|
||||
|
||||
**Solution:** Drive the predecessor's production normalized and tunnel reception handlers with current bound unavailable evidence, a duplicate/lower-sequence stale available observation, and a later higher-sequence exact-target probe available result. At each step assert the metric decision/transition delta, one safe structured event, and the public `ListNodeSnapshots` health/status. Use high-card/raw sentinels in node/provider/run/session/adapter/target and message/body fields and assert none are present in gathered labels or dedicated log fields/messages.
|
||||
|
||||
Before (`apps/edge/internal/service/model_queue_snapshot.go:201`):
|
||||
|
||||
```go
|
||||
func effectiveHealth(connected bool, health string) string {
|
||||
if connected {
|
||||
return health
|
||||
}
|
||||
```
|
||||
|
||||
After predecessor behavior, verified by this child:
|
||||
|
||||
```go
|
||||
// current unavailable -> snapshot unavailable
|
||||
// stale available -> rejection metric, snapshot still unavailable
|
||||
// later current available probe -> recovery metric, snapshot available
|
||||
```
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/service/provider_health_observability_test.go`: add normalized/tunnel applied-unhealthy, stale-rejection, recovered-snapshot, exact-once, lock-safety, label allowlist, and log leakage tables.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: specify Edge health evidence/transition metric and safe-log semantics.
|
||||
- [ ] `agent-contract/inner/edge-config-runtime-refresh.md`: document that runtime overlay observations are separate from immutable config health and carry no provider identity labels.
|
||||
- [ ] `agent-spec/runtime/edge-node-execution.md`: record reception-to-overlay observability and stale/recovery behavior.
|
||||
- [ ] `agent-spec/runtime/provider-pool-config-refresh.md`: record the effective snapshot projection and operational evidence boundary.
|
||||
|
||||
**Test Strategy:** Create `TestProviderHealthObservability` with normalized and provider-tunnel subtests. Each uses the predecessor's real binding/generation/sequence path, queries the actual snapshot, gathers private Prometheus collectors, and captures zap entries. Include a blocking observer fixture to prove it is invoked after `modelQueueManager.mu` is released, plus duplicate terminal/probe rows to prove exactly-once transitions.
|
||||
|
||||
**Verification:** the focused test, service race suite, and provider-capacity smoke below must pass with no zero-match command.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `08+07_health_overlay` must produce `agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log`; it is active and missing at plan creation.
|
||||
2. Implement REFACTOR-1 before REFACTOR-2. Do not instrument raw wire reception independently of the predecessor's final transition decision.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/edge/internal/service/model_queue_types.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/model_queue_release.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/service.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/bootstrap/runtime.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/provider_health_observability.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/service/provider_health_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/inner/edge-config-runtime-refresh.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/edge-node-execution.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/provider-pool-config-refresh.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/12+08_health_overlay_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `test -f agent-task/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` — predecessor PASS evidence exists before implementation.
|
||||
2. `go test -count=20 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — PASS every iteration and normalized/tunnel applied, stale, and recovered rows execute.
|
||||
3. `go test -count=1 ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — PASS under the Edge local profile.
|
||||
4. `go test -race -count=3 ./apps/edge/internal/service ./apps/edge/internal/bootstrap -run 'ProviderHealthObservability|ProviderHealthOverlay|Snapshot'` — PASS with no race report.
|
||||
5. `go vet ./apps/edge/internal/service ./apps/edge/internal/bootstrap ./apps/edge/internal/controlplane` — no diagnostics.
|
||||
6. `./scripts/e2e-provider-capacity-smoke.sh` — PASS with the final provider counters drained and no overlay regression.
|
||||
7. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,183 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability, plan=1, tag=REFACTOR
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/plan_cloud_G08_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/code_review_cloud_G08_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed even though `observationSink()` constructs request-local wrappers, default-versus-custom sink suppression lacked an exact detection contract, and the verification list treated fake-provider smoke as sufficient without the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the `10+09_stall_recovery` dependency, request-local deduplication state, S06 commit/eligibility/result axes, and custom-sink forwarding; add one process-global collector set injected into wrappers, explicit default-sink type detection, repeated-server/request coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 wraps each request's StreamGate observation sink with a failure-isolated liveness projector that reuses one process-global production collector set and emits one bounded eligibility observation and one final result per private liveness cycle without changing filter/recovery behavior.
|
||||
- [ ] REFACTOR-2 proves Chat/Responses normalized/tunnel eligible, rejected, redispatched, terminal/failure, and repeated server/request construction outcomes through exact metric labels and safe structured logs, suppresses liveness high-cardinality fields only from the concrete default generic zap path, and synchronizes matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, fake-provider auxiliary smoke, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify `Server.observationSink()` returns a fresh request-local wrapper, preserves custom downstream observations, and never changes StreamGate filter or recovery decisions.
|
||||
- Verify production collectors are registered once at package lifetime and injected into every wrapper; repeated `Server`/request construction and private registries cannot duplicate or contaminate the default registry.
|
||||
- Verify exact `*zapFilterObservationSink` type detection suppresses only the concrete default path, while custom sinks receive originals and Noop remains no-op downstream without disabling the safe projection.
|
||||
- Verify one private-liveness evaluation produces exactly one eligibility observation and at most one terminal result across intermediate, duplicate, rejection, dispatch, and failure observations.
|
||||
- Verify closed metric labels and the dedicated event omit correlation/request/attempt/run/session/model/provider/node/lease/slot/credential identifiers and all raw content.
|
||||
- Verify the default high-cardinality zap path suppresses only the private-liveness cycle while unrelated observations remain unchanged and custom sinks receive the originals.
|
||||
- Verify Chat/Responses × normalized/tunnel fixtures cover eligible, rejected, redispatched, and terminal/failure outcomes without changing public HTTP/SSE behavior.
|
||||
- Verify contract/spec edits describe only the implemented request-local projection and preserve the predecessor/Core ownership boundary.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `test -f agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`
|
||||
|
||||
Expected: predecessor PASS evidence exists before implementation.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'`
|
||||
|
||||
Expected: PASS every iteration for eligible/rejected/final/deduplicated lifecycle rows.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability'`
|
||||
|
||||
Expected: PASS every iteration and Chat/Responses normalized/tunnel subtests execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: PASS under the Edge/platform-common profiles.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 6
|
||||
|
||||
Command: `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 7
|
||||
|
||||
Command: `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh`
|
||||
|
||||
Expected: auxiliary smoke PASS for credential-free Chat streaming/non-streaming Edge -> Node -> fake provider behavior.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 8
|
||||
|
||||
Command: `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
Expected: PASS using separate Edge/Node entrypoints for registration, two same-session messages, one post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 9
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,217 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=1 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# OpenAI Liveness-Recovery Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this recovery-observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The predecessor makes the OpenAI host the sole liveness recovery owner and emits a private filter decision into the existing StreamGate observation timeline. The generic zap sink logs request/attempt/target correlation, so it cannot directly serve S06's bounded operational evidence. This slice adds a request-local projection that records commit state, eligibility, and the final recovery result while preserving Core decisions and suppressing high-cardinality fields only from the default liveness-specific log path.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior pair: `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/plan_cloud_G08_0.log` and `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/code_review_cloud_G08_0.log`; it was an unimplemented preparation pair with no official verdict, implementation evidence, code change, or verification output.
|
||||
- Replan finding: collector registration lifetime was not closed even though `observationSink()` constructs request-local wrappers, default-versus-custom sink suppression lacked an exact detection contract, and the verification list treated fake-provider smoke as sufficient without the testing rule's direct Edge/Node entrypoint diagnostic.
|
||||
- Carryover: preserve the `10+09_stall_recovery` dependency, request-local deduplication state, S06 commit/eligibility/result axes, and custom-sink forwarding; add one process-global collector set injected into wrappers, explicit default-sink type detection, repeated-server/request coverage, and the repository-native two-process diagnostic.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/PLAN-cloud-G08.md`
|
||||
- `apps/edge/internal/openai/server.go`
|
||||
- `apps/edge/internal/openai/stream_gate_ingress.go`
|
||||
- `apps/edge/internal/openai/stream_gate_filters.go`
|
||||
- `apps/edge/internal/openai/stream_gate_policy.go`
|
||||
- `apps/edge/internal/openai/stream_gate_dispatcher.go`
|
||||
- `apps/edge/internal/openai/filter_observation_sink.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/stream_gate_filters_test.go`
|
||||
- `apps/edge/internal/openai/stream_gate_dispatcher_test.go`
|
||||
- `apps/edge/internal/openai/filter_observation_sink_test.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `packages/go/streamgate/filter_observation.go`
|
||||
- `packages/go/streamgate/runtime.go`
|
||||
- `packages/go/streamgate/recovery_coordinator.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-spec/runtime/stream-evidence-gate.md`
|
||||
- `agent-spec/input/openai-compatible-surface.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/edge-smoke.md`
|
||||
- `agent-test/local/platform-common-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/project/e2e-smoke/SKILL.md`
|
||||
- `scripts/dev/edge.sh`
|
||||
- `scripts/dev/node.sh`
|
||||
- `scripts/dev/edge-node-reconnect-diagnostic.sh`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require the Edge recovery owner to expose commit state, eligibility, and recovery result for deterministic run/tunnel recovery while request/session/raw prompt/response and high-cardinality values are absent from metric labels and the dedicated structured log.
|
||||
- Those rows require a per-request state machine over the predecessor's private filter observation and existing Core recovery lifecycle, not a second retry counter or a reconstruction from HTTP results.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
|
||||
- Planning baseline `go test -count=1 ./apps/edge/internal/openai -run 'OpenAIProviderErrorFoundation|OpenAIAttemptDispatcher|OpenAIAttemptController'` passed. Read-only preflight returned `go version go1.26.2 linux/arm64`, module `/config/workspace/iop-s1/go.mod`, and executable Edge/Node dev entrypoints, fake-vLLM smoke, and reconnect diagnostic. Edge/platform profiles supply OpenAI, service, StreamGate, race, and vet commands.
|
||||
- `10+09_stall_recovery` is active and its `complete.log` is missing. Its plan promises exactly one private liveness filter for StreamGate-enabled requests, sanitized health/fence/handoff evidence, ExactReplay eligibility, provider avoidance, and deterministic Chat/Responses normalized/tunnel matrices.
|
||||
- Current Core observations already expose filter attribution/evidence/commit state and recovery selected/dispatched/failure kinds. `recovery_plan_rejected` intentionally omits recovery info, so a request-local sink must remember only whether the immediately active cycle came from the private liveness filter. Confidence is medium-high because no Core or API behavior changes, but sequencing and deduplication span parallel filter evaluation and recovery lifecycle variants.
|
||||
- No external host is required. Synthetic `ObservationSequencer` fixtures and the predecessor's scripted provider pool give deterministic semantic evidence; `IOP_VLLM_MODE=fake` is an auxiliary OpenAI process smoke, while `scripts/dev/edge-node-reconnect-diagnostic.sh` separately supplies the required real Edge/Node entrypoint cycle with temporary mock configs, ordered message relay, commands, and reconnect.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- The generic `streamgate_filter_observation` log includes correlation, attempt, model, and provider fields and has no Prometheus liveness-recovery projection.
|
||||
- Existing Core observations emit several intermediate recovery kinds; no test selects exactly one final result per liveness cycle or associates plan rejection with the preceding private liveness decision.
|
||||
- No test covers the Cartesian Chat/Responses × normalized/tunnel matrix for safe eligibility/result labels and structured-log leakage.
|
||||
- No test proves repeated `Server` and request-wrapper construction reuses one process-global collector set, or that only the concrete default `*zapFilterObservationSink` is suppressed while custom and `NoopObservationSink` behavior stays explicit.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. No symbol is renamed or removed. `Server.observationSink()` keeps its call sites and returns a fresh wrapper around the configured sink; custom observation sinks continue to receive the original immutable observations.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Stable child output: `10+09_stall_recovery` owns typed stall mapping, the private liveness filter, eligibility, old-attempt teardown, provider handoff, and terminal behavior. Its PASS is required and currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` missing).
|
||||
- This child owns only the request-local observation projection. The private filter id and sanitized descriptor/cause vocabulary are its input contract; Core observation kinds are the final-result oracle.
|
||||
- Provider-health overlay transitions are independently observed by `12+08_health_overlay_observability`; no dependency on sibling 12 is required.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change filter decisions, arbitration, recovery budgets, commit boundary, dispatcher/provider selection, HTTP/SSE terminals, public error bodies, generic non-liveness observations, or Core observation types. Metric labels and dedicated logs must omit correlation/request/attempt/run/session/model/provider/node/lease/slot/credential identifiers and raw prompt/response/tool/provider content.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
|
||||
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 wraps each request's StreamGate observation sink with a failure-isolated liveness projector that reuses one process-global production collector set and emits one bounded eligibility observation and one final result per private liveness cycle without changing filter/recovery behavior.
|
||||
- [ ] REFACTOR-2 proves Chat/Responses normalized/tunnel eligible, rejected, redispatched, terminal/failure, and repeated server/request construction outcomes through exact metric labels and safe structured logs, suppresses liveness high-cardinality fields only from the concrete default generic zap path, and synchronizes matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, fake-provider auxiliary smoke, two-process Edge/Node diagnostic, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Project the request-local liveness lifecycle
|
||||
|
||||
**Problem:** `apps/edge/internal/openai/server.go:193-200` returns one shared generic sink, while `packages/go/streamgate/runtime.go:1055-1147` and `packages/go/streamgate/recovery_coordinator.go:395-417,527-664` emit multiple observations for one recovery. Counting those events directly would duplicate a cycle, and `recovery_plan_rejected` has no strategy field.
|
||||
|
||||
**Solution:** Register one package-level production collector set exactly once with the default Prometheus registerer. `Server` holds the reusable collector/logger dependencies, and every fresh `openAILivenessObservationSink` receives those handles; never call `promauto.New*` or `MustRegister` from `NewServer`, `observationSink()`, or the request wrapper. Tests construct isolated collectors with an explicit `prometheus.Registerer`. The wrapper keeps only a mutex-protected bounded phase (`idle|eligible_pending`) plus the current closed health/path values—never raw identifiers. On the predecessor-defined private liveness filter's evaluated observation, normalize `execution_path`, `provider_health`, `commit_state`, and its sanitized eligibility descriptor through closed maps and increment `iop_edge_liveness_recovery_eligibility_total{execution_path,provider_health,commit_state,eligibility}` exactly once. Ineligible decisions finish immediately with result `terminal`; eligible decisions wait for Core lifecycle. Record exactly one `iop_edge_liveness_recovery_results_total{execution_path,provider_health,recovery_result}` on `recovery_dispatched`, plan rejection, abort/rebuild/dispatch failure, non-liveness plan selection, or terminal fallback, then reset for a later bounded cycle. Intermediate lifecycle observations never increment results. Sink/metric/log failures remain observation-only.
|
||||
|
||||
Before (`apps/edge/internal/openai/server.go:193`):
|
||||
|
||||
```go
|
||||
func (s *Server) observationSink() streamgate.ObservationSink {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
if s.obsSink == nil {
|
||||
return streamgate.NoopObservationSink{}
|
||||
}
|
||||
return s.obsSink
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
func (s *Server) observationSink() streamgate.ObservationSink {
|
||||
s.mu.RLock()
|
||||
downstream, logger := s.obsSink, s.logger
|
||||
s.mu.RUnlock()
|
||||
return newOpenAILivenessObservationSink(downstream, logger)
|
||||
}
|
||||
```
|
||||
|
||||
Use the predecessor's private liveness filter constant rather than duplicating its string. Closed eligibility values must cover `eligible`, `no_owner`, `post_commit`, `unconfirmed_fence`, `caller_cancelled`, `tool_side_effect`, `budget_exhausted`, `no_candidate`, `same_provider_forbidden`, and `other`; closed result values are `redispatched`, `plan_rejected`, `abort_failed`, `rebuild_failed`, `dispatch_failed`, `not_selected`, `terminal`, and `other`. Path is `normalized|provider_tunnel|unknown`; health is `available|unavailable|unknown`; commit state uses Core's closed values with `unknown` fallback.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/openai/server.go`: create one liveness wrapper per request around the configured observation sink and logger.
|
||||
- [ ] `apps/edge/internal/openai/liveness_recovery_observability.go`: implement request-local phase/deduplication, closed classification, default collectors, test injection, and safe log projection.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2. Do not modify `packages/go/streamgate`; the existing immutable observations are sufficient.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` must pass every iteration.
|
||||
|
||||
### [REFACTOR-2] Prove bounded labels, safe logs, and variant outcomes
|
||||
|
||||
**Problem:** `apps/edge/internal/openai/filter_observation_sink.go:44-58` writes `correlation_id`, `attempt_id`, model, and provider on the generic path. Forwarding the predecessor's private liveness filter and ExactReplay lifecycle through that default sink would violate S06's liveness-log boundary even if the new metric labels were safe.
|
||||
|
||||
**Solution:** Snapshot the downstream under `Server.mu` and use an exact type assertion to distinguish the concrete default `*zapFilterObservationSink`; do not infer default ownership from logger/core equality. For that default only, the request-local wrapper consumes private-liveness and its pending ExactReplay lifecycle without forwarding those observations to the high-cardinality generic writer; it writes `edge_liveness_recovery_observation` with only `phase`, the closed labels above, and no identifiers. Non-liveness observations remain unchanged. A sink explicitly installed through `SetObservationSink` remains an application-owned observation backend and receives the original immutable observations while the safe operational projection still emits; `NoopObservationSink` stays no-op downstream but does not disable the safe projection. Unit tests create observations through `streamgate.ObservationSequencer`; end-to-end tests reuse the predecessor's scripted pool to cover Chat/Responses and normalized/tunnel decisions.
|
||||
|
||||
Before (`apps/edge/internal/openai/filter_observation_sink.go:44`):
|
||||
|
||||
```go
|
||||
fields = append(fields,
|
||||
zap.String("correlation_id", obs.StableCorrelation()),
|
||||
zap.String("attempt_id", obs.AttemptID()),
|
||||
zap.String("actual_provider", obs.AttemptTarget().Provider()),
|
||||
)
|
||||
```
|
||||
|
||||
After (default liveness route):
|
||||
|
||||
```go
|
||||
logger.Info("edge_liveness_recovery_observation",
|
||||
zap.String("phase", phase),
|
||||
zap.String("commit_state", commitState),
|
||||
zap.String("eligibility", eligibility),
|
||||
zap.String("recovery_result", result),
|
||||
)
|
||||
```
|
||||
|
||||
The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, `go.uber.org/zap`, and `iop/packages/go/streamgate`. Never log or label `StableCorrelation`, `AttemptID`, `AttemptTarget` identity fields other than normalized execution path, plan/shared ids, cause detail, evidence fingerprint, or raw terminal content.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/openai/liveness_recovery_observability_test.go`: add synthetic sequencing/deduplication/default-sink suppression/custom-sink forwarding and Chat/Responses normalized/tunnel safety matrices.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: specify the Edge recovery-owner eligibility/result metric and dedicated log contract.
|
||||
- [ ] `agent-contract/outer/openai-compatible-api.md`: record transparent pre-commit recovery operational evidence without changing the public response.
|
||||
- [ ] `agent-spec/runtime/stream-evidence-gate.md`: record request-local liveness observation projection and generic-sink suppression boundary.
|
||||
- [ ] `agent-spec/input/openai-compatible-surface.md`: record the metric/log safety vocabulary for Chat and Responses variants.
|
||||
|
||||
**Test Strategy:** `TestOpenAILivenessObservationSink` feeds private filter evaluated, plan-selected, rejection/failure, dispatched, terminal, duplicate, and unrelated continuation observations through `ObservationSequencer`; it asserts one eligibility/result per cycle, repeated default server/request construction without duplicate registration, default-only suppression, explicit Noop handling, and custom downstream preservation. `TestOpenAILivenessRecoveryObservability` drives Chat/Responses × normalized/tunnel with available/unavailable/unknown and rejection/result rows, seeds request/session/prompt/response/provider/credential sentinels, and asserts exact gathered label names plus absence from dedicated/default liveness logs.
|
||||
|
||||
**Verification:** both focused commands below must pass repeatedly, and log capture must observe no default `streamgate_filter_observation` entry for the consumed private-liveness/ExactReplay rows.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `10+09_stall_recovery` must produce `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`; it is active and missing at plan creation.
|
||||
2. Implement REFACTOR-1 before REFACTOR-2. If the predecessor's stable filter descriptor names differ, map those exact stable values in the closed classifier and record the mapping in implementation evidence; do not parse raw causes or invent new retry semantics.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/edge/internal/openai/server.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/openai/liveness_recovery_observability.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/openai/liveness_recovery_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/outer/openai-compatible-api.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/stream-evidence-gate.md` | REFACTOR-2 |
|
||||
| `agent-spec/input/openai-compatible-surface.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `test -f agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` — predecessor PASS evidence exists before implementation.
|
||||
2. `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` — PASS every iteration for eligible/rejected/final/deduplicated lifecycle rows.
|
||||
3. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability'` — PASS every iteration and Chat/Responses normalized/tunnel subtests execute.
|
||||
4. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS under the Edge/platform-common profiles.
|
||||
5. `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'` — PASS with no race report.
|
||||
6. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
|
||||
7. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — auxiliary smoke PASS for credential-free Chat streaming/non-streaming Edge -> Node -> fake provider behavior.
|
||||
8. `IOP_DEV_RECONNECT_BIND_TIMEOUT=45 ./scripts/dev/edge-node-reconnect-diagnostic.sh` — PASS using separate `scripts/dev/edge.sh` and `scripts/dev/node.sh` processes; registration, the first two same-session messages, post-reconnect message, Node-to-Edge payload equality, `/nodes`, `/capabilities`, `/transport`, reconnect, and exactly-once terminal ordering are all verified. This is the required repository-native full-cycle diagnostic.
|
||||
9. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,167 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# Code Review Reference - REFACTOR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-05
|
||||
task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability, plan=0, tag=REFACTOR
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REFACTOR-1 | [ ] |
|
||||
| REFACTOR-2 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 wraps each request's StreamGate observation sink with a failure-isolated liveness projector that emits one bounded eligibility observation and one final result per private liveness cycle without changing filter/recovery behavior.
|
||||
- [ ] REFACTOR-2 proves Chat/Responses normalized/tunnel eligible, rejected, redispatched, and terminal/failure outcomes through exact metric labels and safe structured logs, suppresses liveness high-cardinality fields from the default generic zap path, and synchronizes matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, fake-provider full-cycle, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_0.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-node-provider-execution-liveness-recovery`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify `Server.observationSink()` returns a fresh request-local wrapper, preserves custom downstream observations, and never changes StreamGate filter or recovery decisions.
|
||||
- Verify one private-liveness evaluation produces exactly one eligibility observation and at most one terminal result across intermediate, duplicate, rejection, dispatch, and failure observations.
|
||||
- Verify closed metric labels and the dedicated event omit correlation/request/attempt/run/session/model/provider/node/lease/slot/credential identifiers and all raw content.
|
||||
- Verify the default high-cardinality zap path suppresses only the private-liveness cycle while unrelated observations remain unchanged and custom sinks receive the originals.
|
||||
- Verify Chat/Responses × normalized/tunnel fixtures cover eligible, rejected, redispatched, and terminal/failure outcomes without changing public HTTP/SSE behavior.
|
||||
- Verify contract/spec edits describe only the implemented request-local projection and preserve the predecessor/Core ownership boundary.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Fill each output block with actual stdout/stderr. If a command changes, record the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Verification 1
|
||||
|
||||
Command: `test -f agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`
|
||||
|
||||
Expected: predecessor PASS evidence exists before implementation.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 2
|
||||
|
||||
Command: `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'`
|
||||
|
||||
Expected: PASS every iteration for eligible/rejected/final/deduplicated lifecycle rows.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 3
|
||||
|
||||
Command: `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability'`
|
||||
|
||||
Expected: PASS every iteration and Chat/Responses normalized/tunnel subtests execute.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 4
|
||||
|
||||
Command: `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: PASS under the Edge/platform-common profiles.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 5
|
||||
|
||||
Command: `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'`
|
||||
|
||||
Expected: PASS with no race report.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 6
|
||||
|
||||
Command: `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane`
|
||||
|
||||
Expected: no diagnostics.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 7
|
||||
|
||||
Command: `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh`
|
||||
|
||||
Expected: PASS for credential-free Chat streaming/non-streaming Edge -> Node -> fake provider full-cycle.
|
||||
|
||||
Output:
|
||||
|
||||
### Verification 8
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: no whitespace errors.
|
||||
|
||||
Output:
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,204 @@
|
|||
<!-- task=m-node-provider-execution-liveness-recovery/13+10_recovery_observability plan=0 tag=REFACTOR milestone-task=ops-evidence -->
|
||||
|
||||
# OpenAI Liveness-Recovery Operational Evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only this recovery-observability slice after the predecessor PASS, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record exact blocker evidence, attempted commands/output, and resume conditions only. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The predecessor makes the OpenAI host the sole liveness recovery owner and emits a private filter decision into the existing StreamGate observation timeline. The generic zap sink logs request/attempt/target correlation, so it cannot directly serve S06's bounded operational evidence. This slice adds a request-local projection that records commit state, eligibility, and the final recovery result while preserving Core decisions and suppressing high-cardinality fields only from the default liveness-specific log path.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/PLAN-cloud-G08.md`
|
||||
- `apps/edge/internal/openai/server.go`
|
||||
- `apps/edge/internal/openai/stream_gate_ingress.go`
|
||||
- `apps/edge/internal/openai/stream_gate_filters.go`
|
||||
- `apps/edge/internal/openai/stream_gate_policy.go`
|
||||
- `apps/edge/internal/openai/stream_gate_dispatcher.go`
|
||||
- `apps/edge/internal/openai/filter_observation_sink.go`
|
||||
- `apps/edge/internal/openai/provider_observation.go`
|
||||
- `apps/edge/internal/openai/stream_gate_filters_test.go`
|
||||
- `apps/edge/internal/openai/stream_gate_dispatcher_test.go`
|
||||
- `apps/edge/internal/openai/filter_observation_sink_test.go`
|
||||
- `apps/edge/internal/openai/provider_observability_test.go`
|
||||
- `packages/go/streamgate/filter_observation.go`
|
||||
- `packages/go/streamgate/runtime.go`
|
||||
- `packages/go/streamgate/recovery_coordinator.go`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-spec/runtime/stream-evidence-gate.md`
|
||||
- `agent-spec/input/openai-compatible-surface.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
|
||||
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
|
||||
- `agent-test/local/edge-smoke.md`
|
||||
- `agent-test/local/platform-common-smoke.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`; first-line `milestone-task=ops-evidence`.
|
||||
- Acceptance Scenario S06 and Evidence Map S06 require the Edge recovery owner to expose commit state, eligibility, and recovery result for deterministic run/tunnel recovery while request/session/raw prompt/response and high-cardinality values are absent from metric labels and the dedicated structured log.
|
||||
- Those rows require a per-request state machine over the predecessor's private filter observation and existing Core recovery lifecycle, not a second retry counter or a reconstruction from HTTP results.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff artifact was supplied; starting HEAD `0e594dfa3723431d2f8d83863a677d0c3d9b60be` matched during planning.
|
||||
- Planning baseline `go test -count=1 ./apps/edge/internal/openai -run 'OpenAIProviderErrorFoundation|OpenAIAttemptDispatcher|OpenAIAttemptController'` passed. Edge/platform profiles supply OpenAI, service, StreamGate, race, vet, and fake-provider full-cycle commands.
|
||||
- `10+09_stall_recovery` is active and its `complete.log` is missing. Its plan promises exactly one private liveness filter for StreamGate-enabled requests, sanitized health/fence/handoff evidence, ExactReplay eligibility, provider avoidance, and deterministic Chat/Responses normalized/tunnel matrices.
|
||||
- Current Core observations already expose filter attribution/evidence/commit state and recovery selected/dispatched/failure kinds. `recovery_plan_rejected` intentionally omits recovery info, so a request-local sink must remember only whether the immediately active cycle came from the private liveness filter. Confidence is medium-high because no Core or API behavior changes, but sequencing and deduplication span parallel filter evaluation and recovery lifecycle variants.
|
||||
- No external host is required. Synthetic `ObservationSequencer` fixtures and the predecessor's scripted provider pool give deterministic evidence; `IOP_VLLM_MODE=fake` is the repository-native full-cycle fallback.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- The generic `streamgate_filter_observation` log includes correlation, attempt, model, and provider fields and has no Prometheus liveness-recovery projection.
|
||||
- Existing Core observations emit several intermediate recovery kinds; no test selects exactly one final result per liveness cycle or associates plan rejection with the preceding private liveness decision.
|
||||
- No test covers the Cartesian Chat/Responses × normalized/tunnel matrix for safe eligibility/result labels and structured-log leakage.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- None. No symbol is renamed or removed. `Server.observationSink()` keeps its call sites and returns a fresh wrapper around the configured sink; custom observation sinks continue to receive the original immutable observations.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Stable child output: `10+09_stall_recovery` owns typed stall mapping, the private liveness filter, eligibility, old-attempt teardown, provider handoff, and terminal behavior. Its PASS is required and currently unsatisfied (`agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` missing).
|
||||
- This child owns only the request-local observation projection. The private filter id and sanitized descriptor/cause vocabulary are its input contract; Core observation kinds are the final-result oracle.
|
||||
- Provider-health overlay transitions are independently observed by `12+08_health_overlay_observability`; no dependency on sibling 12 is required.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change filter decisions, arbitration, recovery budgets, commit boundary, dispatcher/provider selection, HTTP/SSE terminals, public error bodies, generic non-liveness observations, or Core observation types. Metric labels and dedicated logs must omit correlation/request/attempt/run/session/model/provider/node/lease/slot/credential identifiers and raw prompt/response/tool/provider content.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
|
||||
- Build closure true; scores `(1,2,2,1,2)`, grade G08, base `local-fit`, escalated by `risk-boundary` -> `PLAN-cloud-G08.md`.
|
||||
- Review closure true; scores `(1,2,2,1,2)`, grade G08, route `official-review` -> `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4). No recovery signal, capability gap, review rework, or evidence-integrity failure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] REFACTOR-1 wraps each request's StreamGate observation sink with a failure-isolated liveness projector that emits one bounded eligibility observation and one final result per private liveness cycle without changing filter/recovery behavior.
|
||||
- [ ] REFACTOR-2 proves Chat/Responses normalized/tunnel eligible, rejected, redispatched, and terminal/failure outcomes through exact metric labels and safe structured logs, suppresses liveness high-cardinality fields from the default generic zap path, and synchronizes matching contracts/specs.
|
||||
- [ ] Run every focused, package, race, vet, fake-provider full-cycle, and diff command in Final Verification with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REFACTOR-1] Project the request-local liveness lifecycle
|
||||
|
||||
**Problem:** `apps/edge/internal/openai/server.go:193-200` returns one shared generic sink, while `packages/go/streamgate/runtime.go:1055-1147` and `packages/go/streamgate/recovery_coordinator.go:395-417,527-664` emit multiple observations for one recovery. Counting those events directly would duplicate a cycle, and `recovery_plan_rejected` has no strategy field.
|
||||
|
||||
**Solution:** Return a fresh `openAILivenessObservationSink` from each `Server.observationSink()` call. It wraps the configured downstream sink and keeps only a mutex-protected bounded phase (`idle|eligible_pending`) plus the current closed health/path values—never raw identifiers. On the predecessor-defined private liveness filter's evaluated observation, normalize `execution_path`, `provider_health`, `commit_state`, and its sanitized eligibility descriptor through closed maps and increment `iop_edge_liveness_recovery_eligibility_total{execution_path,provider_health,commit_state,eligibility}` exactly once. Ineligible decisions finish immediately with result `terminal`; eligible decisions wait for Core lifecycle. Record exactly one `iop_edge_liveness_recovery_results_total{execution_path,provider_health,recovery_result}` on `recovery_dispatched`, plan rejection, abort/rebuild/dispatch failure, non-liveness plan selection, or terminal fallback, then reset for a later bounded cycle. Intermediate lifecycle observations never increment results. Sink/metric/log failures remain observation-only.
|
||||
|
||||
Before (`apps/edge/internal/openai/server.go:193`):
|
||||
|
||||
```go
|
||||
func (s *Server) observationSink() streamgate.ObservationSink {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
if s.obsSink == nil {
|
||||
return streamgate.NoopObservationSink{}
|
||||
}
|
||||
return s.obsSink
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
func (s *Server) observationSink() streamgate.ObservationSink {
|
||||
s.mu.RLock()
|
||||
downstream, logger := s.obsSink, s.logger
|
||||
s.mu.RUnlock()
|
||||
return newOpenAILivenessObservationSink(downstream, logger)
|
||||
}
|
||||
```
|
||||
|
||||
Use the predecessor's private liveness filter constant rather than duplicating its string. Closed eligibility values must cover `eligible`, `no_owner`, `post_commit`, `unconfirmed_fence`, `caller_cancelled`, `tool_side_effect`, `budget_exhausted`, `no_candidate`, `same_provider_forbidden`, and `other`; closed result values are `redispatched`, `plan_rejected`, `abort_failed`, `rebuild_failed`, `dispatch_failed`, `not_selected`, `terminal`, and `other`. Path is `normalized|provider_tunnel|unknown`; health is `available|unavailable|unknown`; commit state uses Core's closed values with `unknown` fallback.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/openai/server.go`: create one liveness wrapper per request around the configured observation sink and logger.
|
||||
- [ ] `apps/edge/internal/openai/liveness_recovery_observability.go`: implement request-local phase/deduplication, closed classification, default collectors, test injection, and safe log projection.
|
||||
|
||||
**Test Strategy:** Write tests in REFACTOR-2. Do not modify `packages/go/streamgate`; the existing immutable observations are sufficient.
|
||||
|
||||
**Verification:** `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` must pass every iteration.
|
||||
|
||||
### [REFACTOR-2] Prove bounded labels, safe logs, and variant outcomes
|
||||
|
||||
**Problem:** `apps/edge/internal/openai/filter_observation_sink.go:44-58` writes `correlation_id`, `attempt_id`, model, and provider on the generic path. Forwarding the predecessor's private liveness filter and ExactReplay lifecycle through that default sink would violate S06's liveness-log boundary even if the new metric labels were safe.
|
||||
|
||||
**Solution:** For the default `zapFilterObservationSink`, the request-local wrapper consumes private-liveness and its pending ExactReplay lifecycle without forwarding those observations to the high-cardinality generic writer; it writes `edge_liveness_recovery_observation` with only `phase`, the closed labels above, and no identifiers. Non-liveness observations remain unchanged. A sink explicitly installed through `SetObservationSink` remains an application-owned observation backend and receives the original immutable observations while the safe operational projection still emits. Unit tests create observations through `streamgate.ObservationSequencer`; end-to-end tests reuse the predecessor's scripted pool to cover Chat/Responses and normalized/tunnel decisions.
|
||||
|
||||
Before (`apps/edge/internal/openai/filter_observation_sink.go:44`):
|
||||
|
||||
```go
|
||||
fields = append(fields,
|
||||
zap.String("correlation_id", obs.StableCorrelation()),
|
||||
zap.String("attempt_id", obs.AttemptID()),
|
||||
zap.String("actual_provider", obs.AttemptTarget().Provider()),
|
||||
)
|
||||
```
|
||||
|
||||
After (default liveness route):
|
||||
|
||||
```go
|
||||
logger.Info("edge_liveness_recovery_observation",
|
||||
zap.String("phase", phase),
|
||||
zap.String("commit_state", commitState),
|
||||
zap.String("eligibility", eligibility),
|
||||
zap.String("recovery_result", result),
|
||||
)
|
||||
```
|
||||
|
||||
The new file imports `github.com/prometheus/client_golang/prometheus`, `github.com/prometheus/client_golang/prometheus/promauto`, `go.uber.org/zap`, and `iop/packages/go/streamgate`. Never log or label `StableCorrelation`, `AttemptID`, `AttemptTarget` identity fields other than normalized execution path, plan/shared ids, cause detail, evidence fingerprint, or raw terminal content.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/edge/internal/openai/liveness_recovery_observability_test.go`: add synthetic sequencing/deduplication/default-sink suppression/custom-sink forwarding and Chat/Responses normalized/tunnel safety matrices.
|
||||
- [ ] `agent-contract/inner/execution-runtime.md`: specify the Edge recovery-owner eligibility/result metric and dedicated log contract.
|
||||
- [ ] `agent-contract/outer/openai-compatible-api.md`: record transparent pre-commit recovery operational evidence without changing the public response.
|
||||
- [ ] `agent-spec/runtime/stream-evidence-gate.md`: record request-local liveness observation projection and generic-sink suppression boundary.
|
||||
- [ ] `agent-spec/input/openai-compatible-surface.md`: record the metric/log safety vocabulary for Chat and Responses variants.
|
||||
|
||||
**Test Strategy:** `TestOpenAILivenessObservationSink` feeds private filter evaluated, plan-selected, rejection/failure, dispatched, terminal, duplicate, and unrelated continuation observations through `ObservationSequencer`; it asserts one eligibility/result per cycle and custom downstream preservation. `TestOpenAILivenessRecoveryObservability` drives Chat/Responses × normalized/tunnel with available/unavailable/unknown and rejection/result rows, seeds request/session/prompt/response/provider/credential sentinels, and asserts exact gathered label names plus absence from dedicated/default liveness logs.
|
||||
|
||||
**Verification:** both focused commands below must pass repeatedly, and log capture must observe no default `streamgate_filter_observation` entry for the consumed private-liveness/ExactReplay rows.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `10+09_stall_recovery` must produce `agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log`; it is active and missing at plan creation.
|
||||
2. Implement REFACTOR-1 before REFACTOR-2. If the predecessor's stable filter descriptor names differ, map those exact stable values in the closed classifier and record the mapping in implementation evidence; do not parse raw causes or invent new retry semantics.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `apps/edge/internal/openai/server.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/openai/liveness_recovery_observability.go` | REFACTOR-1 |
|
||||
| `apps/edge/internal/openai/liveness_recovery_observability_test.go` | REFACTOR-2 |
|
||||
| `agent-contract/inner/execution-runtime.md` | REFACTOR-2 |
|
||||
| `agent-contract/outer/openai-compatible-api.md` | REFACTOR-2 |
|
||||
| `agent-spec/runtime/stream-evidence-gate.md` | REFACTOR-2 |
|
||||
| `agent-spec/input/openai-compatible-surface.md` | REFACTOR-2 |
|
||||
| `agent-task/m-node-provider-execution-liveness-recovery/13+10_recovery_observability/CODE_REVIEW-cloud-G08.md` | REFACTOR-1, REFACTOR-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Fresh Go output is required; cached output is not acceptable.
|
||||
|
||||
1. `test -f agent-task/m-node-provider-execution-liveness-recovery/10+09_stall_recovery/complete.log` — predecessor PASS evidence exists before implementation.
|
||||
2. `go test -count=20 ./apps/edge/internal/openai -run '^TestOpenAILivenessObservationSink'` — PASS every iteration for eligible/rejected/final/deduplicated lifecycle rows.
|
||||
3. `go test -count=10 ./apps/edge/internal/openai -run '^TestOpenAILivenessRecoveryObservability'` — PASS every iteration and Chat/Responses normalized/tunnel subtests execute.
|
||||
4. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — PASS under the Edge/platform-common profiles.
|
||||
5. `go test -race -count=3 ./packages/go/streamgate ./apps/edge/internal/openai -run 'LivenessObservation|LivenessRecovery|Observation'` — PASS with no race report.
|
||||
6. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
|
||||
7. `IOP_VLLM_MODE=fake ./scripts/e2e-openai-vllm.sh` — PASS for credential-free Chat streaming/non-streaming Edge -> Node -> fake provider full-cycle.
|
||||
8. `git diff --check` — no whitespace errors.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
Loading…
Reference in a new issue