<!-- task=m-node-provider-execution-liveness-recovery/02+01_stall_watchdog plan=5 tag=REVIEW_REVIEW_REVIEW_API milestone-task=activity-contract,stall-watchdog -->

# Code Review Reference - REVIEW_REVIEW_REVIEW_API

> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.

## Overview

date=2026-08-04
task=m-node-provider-execution-liveness-recovery/02+01_stall_watchdog, plan=5, tag=REVIEW_REVIEW_REVIEW_API

## Archive Evidence Snapshot

- The current pair will archive as `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G08_5.log` and `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G08_5.log`.
- Prior verdict: FAIL. Required=2, Suggested=0, Nit=0.
- Required fixes: bind a consumed expiry to the timer arm that produced it across the receive-before-capture race; use a cold-build-tolerant local reconnect verification setup without weakening transcript assertions.
- Fresh reviewer evidence: the existing focused count-10 tests, session tests, race count 3, vet, Node packages, full Go suite, auxiliary E2E, formatting, touched readability comparison, and diff check passed. A temporary deterministic reviewer test failed when progress reset the observer after consuming `expired()` but before `captureExpiry()`. The 45-second reconnect command repeatedly expired during cold Go builds; the same checkout passed all registration, three-run payload ordering, command, terminal, and reconnect checks with a 300-second registration ceiling.
- Roadmap carryover: preserve `milestone-task=activity-contract,stall-watchdog`; satisfy approved SDD S01/S02 evidence only and do not update roadmap state.

## For the Review Agent

> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.

Compare implementation of each item against source files and verify that output in `Verification Results` matches code.
Review completion means the following steps are finished:

1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_5.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_5.log`.
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.

---

## Implementation Item Completion

| Item | Status |
|------|---------|
| FIX-1 — timer-arm expiry validity | [x] |
| VERIFY-1 — cold-build-tolerant reconnect evidence | [x] |

## Implementation Checklist

- [x] [FIX-1] Bind each consumed watchdog expiry to the timer arm that produced it, reject progress-reset stale signals before or after validity capture, and add deterministic normalized/tunnel regressions while preserving exactly-once terminal/fence behavior.
- [x] [VERIFY-1] Run the cold-build-tolerant local reconnect diagnostic and every final verification command, recording literal zero-exit output without weakening transcript assertions.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

## Review-Only Checklist

> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.

- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_5.log`.
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_5.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/` to `agent-task/archive/YYYY/MM/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/` and update this checklist at the final archive path.
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-node-provider-execution-liveness-recovery/` or verify it was kept due to remaining siblings/files.
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.

## Deviations from Plan

No product-code or test-scope deviation.

The implementation handoff left the implementation-owned checklist, notes, and verification fields pending. The official reviewer repaired this non-behavioral artifact drift only after independently reading the source and obtaining fresh command output.

The first `IOP_DEV_RECONNECT_BIND_TIMEOUT=300` diagnostic attempt exhausted the registration ceiling while `scripts/dev/node.sh` was still in its local `go build` phase. No Node runtime had started. A second identical command, with the cache warmed by that build, passed the complete transcript without changing the script, configuration semantics, or assertions. Both attempts are recorded below.

## Key Design Decisions

- `attemptTimer.C()` carries the timer's monotonic fire time. `attemptObserver.expiryForSignal` compares that fixed signal time with the observer-owned `armedAt`, so a progress reset between channel receive and validity capture rejects the old arm.
- The captured observer epoch remains part of `attemptExpiry`; `claimFence` compares it again so progress after validity capture but before sink claim also rejects the stale expiry.
- The package-private before/after expiry-capture seams exist only for deterministic normalized/tunnel race tests. The tests prove the stale arm does not cancel or fence, then prove the reset arm produces exactly one confirmed stall after its full threshold.
- The reconnect diagnostic retained its three-run payload, terminal-ordering, command, and reconnect assertions. Only the documented local build-inclusive registration ceiling was set to 300 seconds.

## Reviewer Checkpoints

- Force progress after the old timer signal is consumed but before expiry validity is captured; the old signal must not fence normalized or tunnel execution.
- Retain the existing progress-after-capture/before-claim tests and confirm both orderings reject the old signal.
- Fire the reset timer only after its full threshold and confirm it produces exactly one stall terminal and one fence result.
- Confirm provider terminal, caller cancel, hard deadline, session disconnect, exact `defaultAttemptCloseGrace=5s`, and confirmed/unconfirmed ownership remain unchanged.
- Confirm confirmed tunnel admission/run/credential cleanup precedes terminal visibility and unconfirmed ownership remains until provider return.
- Confirm normalized Failure, normalized protobuf, and tunnel metadata remain Node-owned, independently cloned, secret-free, and omit `recovery_eligible`.
- Confirm the 300-second local diagnostic still enforces three message runs, Node==Edge payload order, exactly-one terminal after payload, commands, and reconnect.
- Confirm touched readability values stay no greater than baseline without modifying baseline/read-set files.
- Confirm header ids remain `activity-contract,stall-watchdog` and evidence is limited to approved SDD S01/S02.

## Verification Results

> Replace every pending line below with the command's literal stdout/stderr and exit code. If output is saved outside the repository, record the exact output path and command.

### `go version && go env GOMOD`

```text
go version go1.26.2 linux/arm64
/config/workspace/iop-s1/go.mod
```

Exit code: 0.

### `go test -count=20 ./apps/node/internal/node -run 'Test((Run|Tunnel)WatchdogStaleExpiry(BeforeCapture)?YieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal|RunWatchdogLifecycle|TunnelWatchdogLifecycle|TunnelSinkStallClaimSerializesAcceptedFrame|TunnelCredentialFailureReleasesAdmission)$'`

```text
ok  	iop/apps/node/internal/node	2.676s
```

Exit code: 0.

### `go test -count=10 ./apps/node/internal/transport -run 'TestSessionLifetimeCancels(Run|Tunnel)Handler$'`

```text
ok  	iop/apps/node/internal/transport	0.644s
```

Exit code: 0.

### `go test -race -count=3 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`

```text
ok  	iop/packages/go/execution	1.230s
ok  	iop/apps/node/internal/node	10.875s
ok  	iop/apps/node/internal/transport	19.743s
```

Exit code: 0.

### `go vet ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`

```text
<no stdout/stderr>
```

Exit code: 0.

### `go test -count=1 ./packages/go/execution ./apps/node/...`

```text
ok  	iop/packages/go/execution	0.560s
ok  	iop/apps/node/cmd/node	2.319s
ok  	iop/apps/node/internal/adapters	2.096s
?   	iop/apps/node/internal/adapters/mock	[no test files]
ok  	iop/apps/node/internal/adapters/ollama	1.786s
ok  	iop/apps/node/internal/adapters/openai_compat	1.883s
ok  	iop/apps/node/internal/adapters/vllm	1.693s
ok  	iop/apps/node/internal/bootstrap	3.969s
ok  	iop/apps/node/internal/node	3.729s
ok  	iop/apps/node/internal/router	1.479s
ok  	iop/apps/node/internal/store	1.701s
ok  	iop/apps/node/internal/transport	7.917s
```

Exit code: 0.

### `go test -count=1 ./...`

```text
ok  	iop/apps/control-plane/cmd/control-plane	4.780s
ok  	iop/apps/control-plane/internal/credentiallease	0.658s
ok  	iop/apps/control-plane/internal/credentialops	0.811s
ok  	iop/apps/control-plane/internal/credentialseal	0.550s
ok  	iop/apps/control-plane/internal/credentialstore	1.286s
ok  	iop/apps/control-plane/internal/wire	3.005s
ok  	iop/apps/edge/cmd/edge	1.269s
ok  	iop/apps/edge/internal/authprojection	0.445s
ok  	iop/apps/edge/internal/bootstrap	2.042s
ok  	iop/apps/edge/internal/configrefresh	0.969s
ok  	iop/apps/edge/internal/controlplane	8.097s
ok  	iop/apps/edge/internal/edgecmd	0.980s
ok  	iop/apps/edge/internal/edgevalidate	0.861s
ok  	iop/apps/edge/internal/events	0.362s
ok  	iop/apps/edge/internal/input	0.682s
ok  	iop/apps/edge/internal/input/a2a	0.565s
ok  	iop/apps/edge/internal/node	0.382s
ok  	iop/apps/edge/internal/openai	11.856s
ok  	iop/apps/edge/internal/opsconsole	1.201s
ok  	iop/apps/edge/internal/service	8.022s
ok  	iop/apps/edge/internal/transport	6.829s
ok  	iop/apps/node/cmd/node	0.629s
ok  	iop/apps/node/internal/adapters	0.420s
?   	iop/apps/node/internal/adapters/mock	[no test files]
ok  	iop/apps/node/internal/adapters/ollama	0.253s
ok  	iop/apps/node/internal/adapters/openai_compat	0.349s
ok  	iop/apps/node/internal/adapters/vllm	0.325s
ok  	iop/apps/node/internal/bootstrap	2.380s
ok  	iop/apps/node/internal/node	1.733s
ok  	iop/apps/node/internal/router	0.740s
ok  	iop/apps/node/internal/store	0.339s
ok  	iop/apps/node/internal/transport	6.240s
?   	iop/apps/worker/cmd/worker	[no test files]
ok  	iop/packages/go/audit	0.045s
ok  	iop/packages/go/auth	10.563s
ok  	iop/packages/go/config	0.514s
ok  	iop/packages/go/credentiallease	0.375s
?   	iop/packages/go/events	[no test files]
ok  	iop/packages/go/execution	0.147s
ok  	iop/packages/go/hostsetup	0.118s
?   	iop/packages/go/jobs	[no test files]
?   	iop/packages/go/metadata	[no test files]
ok  	iop/packages/go/observability	0.367s
?   	iop/packages/go/policy	[no test files]
ok  	iop/packages/go/streamgate	1.624s
?   	iop/packages/go/version	[no test files]
?   	iop/proto/gen/iop	[no test files]
ok  	iop/scripts/inventory-query	0.174s
```

Exit code: 0.

### `./scripts/e2e-smoke.sh`

```text
[e2e] verifying provider-only Node command and cancellation boundary
ok  	iop/apps/node/internal/node	1.637s
[e2e] verifying Edge dispatch, provider tunnel, queue, and reconnect fencing
ok  	iop/apps/edge/internal/service	5.581s
ok  	iop/apps/edge/internal/transport	0.777s
[e2e] provider-only Edge-Node smoke PASSED
```

Exit code: 0.

### `IOP_DEV_RECONNECT_BIND_TIMEOUT=300 ./scripts/dev/edge-node-reconnect-diagnostic.sh`

```text
Attempt 1:
[diagnostic] Starting edge-node-reconnect-diagnostic (repo-internal)...
[diagnostic] Starting edge.sh...
[diagnostic] Starting node.sh...
[diagnostic] Awaiting node registration...
[diagnostic] Timeout waiting for node registration
[diagnostic] Cleaning up...
[edge] config=/tmp/iop-reconnect-diag-inFPRv/edge.yaml
IOP Edge console listening on 127.0.0.1:34625
Console target node= adapter=mock target=mock-stream session=diagnostic-correlation background=false
Start node.sh on another host, then type a message here.
Commands: /nodes, /node <id|alias>, /session <id>, /background on|off, /capabilities, /transport, /exit
edge>

Exit code: 1. Process inspection during the retry showed `scripts/dev/node.sh` still running `go build -o /config/workspace/iop-s1/build/dev/iop-node ./apps/node/cmd/node`; no Node runtime had started.

Attempt 2, identical command:
[diagnostic] Starting edge-node-reconnect-diagnostic (repo-internal)...
[diagnostic] Starting edge.sh...
[diagnostic] Starting node.sh...
[diagnostic] Awaiting node registration...
[diagnostic] Node registered
[diagnostic] Message 1 completed
[diagnostic] Message 2 completed
[diagnostic] Killing node for reconnect test...
[diagnostic] Restarting node...
[node0-evt] connected reason="registered"
[diagnostic] Node reconnected
[diagnostic] Message 3 completed
[diagnostic] Verifying payload sequence, terminal ordering, and command responses...
[diagnostic] Checking run 1 run_id=manual-1785796050554805881 token=IOP_E2E_HELLO_BASIC
[diagnostic] Checking run 2 run_id=manual-1785796051900060007 token=IOP_E2E_HELLO_FORMAL
[diagnostic] Checking run 3 run_id=manual-1785796114414681841 token=IOP_E2E_PING_BASIC
[diagnostic] PASS: 3 mock-provider runs verified — registration, ordered Node==Edge payloads, one terminal after the last payload, /nodes, /capabilities, /transport, reconnect, and absent session/status ownership commands.
[diagnostic] Cleaning up...

Exit code: 0. The full successful transcript also showed matching Node and Edge payload text for all three run ids, one complete event after each final payload, `/nodes`, `/capabilities`, `/transport`, one transport disconnect, and the second registered connection.
```

### `make readability-audit || test $? -eq 2`

```text
python3 scripts/readability_audit.py --check --input-mode worktree --output build/readability-audit.json
RATCHET FAIL: new or increased violations:
  <read-set:edge-transport-readability>: read_set_total=2155 level=- (task total increased from 2152 to 2155)
  agent-ops/skills/project/openai-usage-token-issue/scripts/issue_token.py: file_loc=1363 level=exception (new violation not in baseline)
  agent-ops/skills/project/openai-usage-token-issue/scripts/issue_token.py: function_loc func=execute=153 level=split_review (new violation not in baseline)
  agent-ops/skills/project/openai-usage-token-issue/scripts/issue_token.py: function_loc func=selftest=83 level=warning (new violation not in baseline)
  agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py: file_loc=7227 level=exception (value increased from 7215)
  agent-ops/skills/project/orchestrate-agent-task-loop/scripts/dispatch.py: function_loc func=run_review=134 level=split_review (value increased from 122)
  agent-ops/skills/project/orchestrate-agent-task-loop/tests/test_dispatch.py: file_loc=12872 level=split_review (value increased from 12738)
  agent-ops/skills/project/orchestrate-agent-task-loop/tests/test_dispatch.py: function_loc func=DispatcherConvergenceSimulationTest.test_review_finalization_mismatch_keeps_dispatcher_running=92 level=warning (new violation not in baseline)
  apps/node/internal/node/liveness_watchdog.go: file_loc=538 level=warning (new violation not in baseline)
  apps/node/internal/node/liveness_watchdog_test.go: file_loc=989 level=warning (new violation not in baseline)
readability-audit: 490 files, 224806 LOC, 6732 functions, 537 violations
make: *** [Makefile:79: readability-audit] Error 4
```

`make readability-audit` exit code: 2; the planned `make readability-audit || test $? -eq 2` command exit code: 0.

### Touched readability baseline comparison

```bash
python3 - <<'PY'
import json

audit = json.load(open('build/readability-audit.json', encoding='utf-8'))
baseline = json.load(open('scripts/readability_baseline.json', encoding='utf-8'))
targets = {
    ('apps/node/internal/node/run_handler.go', 'Node.OnRunRequest'),
    ('apps/node/internal/node/tunnel_handler.go', 'Node.OnProviderTunnelRequest'),
    ('apps/node/internal/transport/session.go', 'newSession'),
}
base_functions = {(x['path'], x.get('function')): x['value'] for x in baseline['function_thresholds']}
current_functions = {(x['path'], x.get('function')): x['value'] for x in audit['violations'] if x.get('metric') == 'function_loc'}
bad = {key: current_functions.get(key, 0) for key in targets if current_functions.get(key, 0) > base_functions[key]}
base_sets = {x['task_id']: x['value'] for x in baseline['task_read_set_totals']}
current_sets = {x['task_id']: x['total_loc'] for x in audit['task_read_sets']}
if current_sets['node-core-readability'] > base_sets['node-core-readability']:
    bad['node-core-readability'] = current_sets['node-core-readability']
if bad:
    raise SystemExit(f'touched readability regression: {bad}')
print('touched readability regression: none')
PY
```

```text
touched readability regression: none
```

Exit code: 0.

### `test -z "$(gofmt -l apps/node/internal/node/liveness_watchdog.go apps/node/internal/node/liveness_watchdog_test.go)"`

```text
<no stdout/stderr>
```

Exit code: 0.

### `git diff --check`

```text
<no stdout/stderr>
```

Exit code: 0.

---

> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.

## Section Ownership

| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |

## Code Review Result

- Overall Verdict: FAIL
- Dimension Assessment:
  - Correctness: Fail
  - Completeness: Fail
  - Test coverage: Fail
  - API contract: Fail
  - Code quality: Pass
  - Implementation deviation: Fail
  - Verification trust: Fail
  - Spec conformance: Fail
- Findings:
  - Required — `apps/node/internal/node/liveness_watchdog.go:63`: `newAttemptObserver` calls `clock.NewTimer(timeout)` before recording `armedAt=clock.Now()`, and the progress path at lines 89-90 similarly calls `Reset` before updating `armedAt`. A valid positive timeout can therefore fire during that gap; its current-arm signal time is then earlier than `armedAt`, so `expiryForSignal` rejects the only expiry as stale and the attempt can remain unfenced indefinitely. Fresh deterministic reviewer evidence used an immediate timer whose current signal fired at `t`, delayed the arm timestamp to `t+1ms`, and failed with `current timer signal was rejected because armedAt was recorded after the timer fired`. A simple statement reorder is not enough for the reset boundary because an old arm can fire while progress owns the observer: bind an explicit generation/deadline to every armed signal, or otherwise prove current-arm identity across creation, reset, receive-before-capture, and capture-before-claim. Add a deterministic initial-arm fire-before-bookkeeping regression plus normalized/tunnel reset-during-fire regressions, then retain the existing stale-expiry and exactly-once lifecycle evidence.
- Routing Signals: `review_rework_count=4`, `evidence_integrity_failure=true`
- Next Step: Archive this pair and create the routed follow-up PLAN/CODE_REVIEW pair through plan `prepare-follow-up` mode.
