<!-- task=m-node-provider-execution-liveness-recovery/02+01_stall_watchdog plan=5 tag=REVIEW_REVIEW_REVIEW_API milestone-task=activity-contract,stall-watchdog -->

# PLAN — Bind Watchdog Expiry to Its Timer Arm

## For the Implementing Agent

> **MANDATORY:** Implement only this follow-up checklist and preserve unrelated worktree changes. Run every verification command, fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with literal results, keep the active pair in place, and report ready for review. If blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence. Do not ask the user, call user-input tools, create stop files, classify next state, archive logs, or write `complete.log`; finalization belongs to the official code-review agent.

## Background

The confirmed tunnel terminal now closes Node-owned resources before publication, and the existing named temporal/race tests pass. Official review still reproduced one uncovered ordering: a timer tick can be consumed, provider progress can reset the observer before the tick captures validity, and the old tick can then inherit the new epoch and fence the attempt. The local reconnect diagnostic also needs a cold-build-tolerant registration ceiling so compilation time is not mistaken for a runtime registration failure.

## Archive Evidence Snapshot

- The current pair will archive as `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G08_5.log` and `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G08_5.log`.
- Prior verdict: FAIL. Required=2, Suggested=0, Nit=0.
- Required fixes: bind a consumed expiry to the timer arm that produced it across the receive-before-capture race; use a cold-build-tolerant local reconnect verification setup without weakening transcript assertions.
- Fresh reviewer evidence: the existing focused count-10 tests, session tests, race count 3, vet, Node packages, full Go suite, auxiliary E2E, formatting, touched readability comparison, and diff check passed. A temporary deterministic reviewer test failed when progress reset the observer after consuming `expired()` but before `captureExpiry()`. The 45-second reconnect command repeatedly expired during cold Go builds; the same checkout passed all registration, three-run payload ordering, command, terminal, and reconnect checks with a 300-second registration ceiling.
- Roadmap carryover: preserve `milestone-task=activity-contract,stall-watchdog`; satisfy approved SDD S01/S02 evidence only and do not update roadmap state.

## Dependencies and Execution Order

- Runtime predecessor `01_activity_contract` remains satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/01_activity_contract/complete.log`.
- Complete FIX-1 before VERIFY-1 so the full-cycle evidence exercises the final watchdog implementation.

## Analysis

### Files Read

- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/PLAN-cloud-G08.md`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G09_3.log`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G09_3.log`
- `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/01_activity_contract/complete.log`
- `agent-roadmap/priority-queue.md`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `apps/node/internal/node/liveness_watchdog.go`
- `apps/node/internal/node/liveness_watchdog_test.go`
- `apps/node/internal/node/run_handler.go`
- `apps/node/internal/node/runtime_sink.go`
- `apps/node/internal/node/tunnel_handler.go`
- `apps/node/internal/node/node.go`
- `apps/node/internal/node/run_manager.go`
- `packages/go/execution/liveness.go`
- `packages/go/execution/failure.go`
- `packages/go/execution/types.go`
- `scripts/dev/edge-node-reconnect-diagnostic.sh`
- `scripts/dev/edge.sh`
- `scripts/dev/node.sh`
- `agent-test/local/rules.md`
- `agent-test/local/node-smoke.md`
- `agent-test/local/testing-smoke.md`
- `agent-ops/rules/project/domain/node/rules.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-ops/skills/project/e2e-smoke/SKILL.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`, lock released, no user review.
- Header ids remain `activity-contract,stall-watchdog`; both ids exist in the selected active Milestone.
- S01 requires provider progress to reset the no-progress clock and retain earlier request/session boundaries.
- S02 requires threshold and timer/event/cancel/close races to produce exactly one terminal and a trustworthy confirmed/unconfirmed local fence.
- Evidence Map rows S01/S02 require fake-clock normalized/tunnel activity, threshold races, exactly-once terminal, confirmed/unconfirmed ownership, and late-output fencing. FIX-1 adds the missing receive-before-capture race while retaining the existing post-capture and ownership evidence; final verification repeats both paths under race detection.

### Verification Context

- `update-test mode=resolve-context`: environment `local`; rules state `usable`; sources `agent-test/local/rules.md`, `agent-test/local/node-smoke.md`, and `agent-test/local/testing-smoke.md`.
- Profile commands: `go version && go env GOMOD`, `go test -count=1 ./packages/go/execution ./apps/node/...`, and `git diff --check`; Node execution must exit zero and requires no external provider or credential. Fresh test execution is required.
- Repository-native additions: the approved SDD, contracts, focused temporal tests, `go test -race`, `./scripts/e2e-smoke.sh`, and the repo-internal reconnect diagnostic from the testing domain and `e2e-smoke` skill.
- Preflight result: Go `go1.26.2 linux/arm64`; module `/config/workspace/iop-s1/go.mod`. The current executor can run all required commands without external authorization or secret material.
- Diagnostic constraint: `IOP_DEV_RECONNECT_BIND_TIMEOUT=45` is too short for cold Edge/Node compilation on this host. Secret-safe live logs showed Edge and Node runtime registration succeeding after build completion, and `IOP_DEV_RECONNECT_BIND_TIMEOUT=300` passed the complete transcript. The follow-up uses 300 seconds as a build-tolerant ceiling; it does not change runtime heartbeat, reconnect, or transcript assertions.
- Gaps: none after adding the receive-before-capture oracle and the build-tolerant diagnostic ceiling. Confidence: high because the remaining defect has a deterministic direct reproducer and all runners are local.

### Test Coverage Gaps

- Progress after expiry capture but before sink claim is covered by `TestRunWatchdogStaleExpiryYieldsToProgress` and `TestTunnelWatchdogStaleExpiryYieldsToProgress`.
- Progress after timer-channel receive but before expiry capture is not covered; current code deterministically accepts the old tick after reset.
- Confirmed cleanup-before-terminal, unconfirmed retention, exact 5-second close grace, cancel/deadline/session precedence, metadata cloning, and ordinary run/tunnel regressions are covered and must remain green.
- The reconnect transcript is functionally covered but the former 45-second build-inclusive ceiling is not reliable on this host; the 300-second run proved the runtime path.

### Symbol References

- No public symbol is renamed or removed.
- Package-local `attemptClock`, `attemptTimer`, `attemptObserver.expired`, `attemptExpiry`, `captureExpiry`, `claimFence`, and `awaitAttempt` are referenced only by `apps/node/internal/node/liveness_watchdog.go`, `apps/node/internal/node/liveness_watchdog_test.go`, and `Node.watchdogClock`; update every listed call if the expiry representation changes.

### Split Judgment

- Keep one atomic follow-up. The expiry representation, both sink claims, and normalized/tunnel deterministic tests form one timer-arm validity invariant. VERIFY-1 is the same packet's required local execution evidence and has no independent code artifact.
- Predecessor index `01` is satisfied by the archived `complete.log` listed above.

### Scope Rationale

- In scope: shared Node watchdog expiry validity, deterministic normalized/tunnel receive-before-capture regressions, and cold-build-tolerant local verification evidence.
- Excluded: confirmed tunnel cleanup ordering already fixed, activity/config/protobuf propagation, provider health probing, Edge health overlay/recovery, metrics, contracts/spec wording, diagnostic script semantics, readability baselines/read sets, and roadmap state.
- Do not modify `scripts/dev/edge-node-reconnect-diagnostic.sh`, default configs, or transcript assertions solely to make the local runner pass.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
- Build closures: scope/context/verification/evidence/ownership/decision all true; capability gap none.
- Build scores: scope=1, state=2, blast=1, evidence=2, verification=2 => G08; base `local-fit`; `recovery-boundary` selects cloud and `PLAN-cloud-G08.md`.
- Build signals: `large_indivisible_context=false`; positive risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4); `review_rework_count=3`; `evidence_integrity_failure=false`; risk and recovery boundaries matched, with recovery precedence.
- Review closures: scope/context/verification/evidence/ownership/decision all true; scores 1/2/1/2/2 => G08; `official-review`, cloud, Codex `gpt-5.6-sol` xhigh, `CODE_REVIEW-cloud-G08.md`.

## Implementation Checklist

- [ ] [FIX-1] Bind each consumed watchdog expiry to the timer arm that produced it, reject progress-reset stale signals before or after validity capture, and add deterministic normalized/tunnel regressions while preserving exactly-once terminal/fence behavior.
- [ ] [VERIFY-1] Run the cold-build-tolerant local reconnect diagnostic and every final verification command, recording literal zero-exit output without weakening transcript assertions.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [FIX-1] Bind expiry validity before the receive race

**Problem**

At `apps/node/internal/node/liveness_watchdog.go:184-186`, `awaitAttempt` receives from the timer and only then calls `captureExpiry`. At lines 91-97, that method copies the current observer epoch. Progress between those operations increments the epoch and resets the timer, so the consumed old tick is mislabeled with the new epoch and `claimFence` accepts it.

**Solution**

Before (`liveness_watchdog.go:184-186`):

```go
case <-observer.expired():
    expiry, valid := observer.captureExpiry()
    if !valid || !contextStillActive(execCtx) || !claimStall(expiry) {
```

After:

```go
case firedAt := <-observer.expired():
    expiry, valid := observer.expiryForSignal(firedAt)
    if !valid || !contextStillActive(execCtx) || !claimStall(expiry) {
```

- Bind validity to information carried by the timer signal itself. Use its monotonic fire time against observer-owned last-progress time, or an equivalent generation-specific signal that is fixed when the timer arm is created; do not capture the current epoch after receive as the sole proof.
- Extend the injected clock/manual timer only as needed to provide deterministic monotonic times. A progress reset at or after the consumed signal must invalidate it, while the later reset timer signal must remain valid after a full threshold.
- Keep sink emission authority, provider/context precedence, exact `defaultAttemptCloseGrace=5s`, confirmed cleanup-before-terminal, unconfirmed retention, and exactly-once fencing unchanged.

**Modified Files and Checklist**

- [ ] `apps/node/internal/node/liveness_watchdog.go` — make expiry validity originate from the timer arm/signal rather than post-receive current state.
- [ ] `apps/node/internal/node/liveness_watchdog_test.go` — cover receive-before-capture and existing capture-before-claim races for normalized and tunnel handlers.

**Test Strategy**

Required. Add `TestRunWatchdogStaleExpiryBeforeCaptureYieldsToProgress` and `TestTunnelWatchdogStaleExpiryBeforeCaptureYieldsToProgress` with a package-private deterministic seam immediately after timer receive and before expiry validity capture. Consume the old timer, let provider progress reset and emit, release the watchdog, assert no stall, then fire the reset timer after its full threshold and assert exactly one terminal. Retain the existing `...StaleExpiryYieldsToProgress` tests for the post-capture/pre-claim ordering.

**Verification**

- `go test -count=20 ./apps/node/internal/node -run 'Test((Run|Tunnel)WatchdogStaleExpiry(BeforeCapture)?YieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal)$'`
- `go test -race -count=3 ./apps/node/internal/node -run 'Test((Run|Tunnel)WatchdogStaleExpiry(BeforeCapture)?YieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal)$'`

### [VERIFY-1] Separate local build latency from reconnect runtime evidence

**Problem**

At `CODE_REVIEW-cloud-G08.md:163-179`, the required 45-second reconnect diagnostic expired before Node registration. Review live logs showed `go run`/`go build` consuming that window, while `scripts/dev/edge-node-reconnect-diagnostic.sh:109-117` otherwise observed registration and the same checkout passed its complete transcript with a 300-second ceiling.

**Solution**

- Use `IOP_DEV_RECONNECT_BIND_TIMEOUT=300` for this local verification so cold compilation is included without changing runtime reconnect semantics.
- Preserve the script's fixed temporary config, mock provider, three messages, Node-versus-Edge payload equality, exactly-one terminal ordering, `/nodes`, `/capabilities`, `/transport`, reconnect, and removed-command assertions.
- Do not edit the diagnostic script or default configs. If the 300-second run fails, record its literal secret-safe output and exact resume condition instead of classifying it as a product success.

**Modified Files and Checklist**

- [ ] `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md` — record literal final verification, including the 300-second full diagnostic.

**Test Strategy**

No new diagnostic-script test. The existing fail-fast script is the oracle; only its build-inclusive registration ceiling changes for this local run.

**Verification**

- `IOP_DEV_RECONNECT_BIND_TIMEOUT=300 ./scripts/dev/edge-node-reconnect-diagnostic.sh`

## Modified Files Summary

| Path | Action | Checklist |
|------|--------|-----------|
| `apps/node/internal/node/liveness_watchdog.go` | modify | FIX-1 |
| `apps/node/internal/node/liveness_watchdog_test.go` | modify | FIX-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md` | update evidence | FIX-1, VERIFY-1 |

## Final Verification

1. `go version && go env GOMOD`
2. `go test -count=20 ./apps/node/internal/node -run 'Test((Run|Tunnel)WatchdogStaleExpiry(BeforeCapture)?YieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal|RunWatchdogLifecycle|TunnelWatchdogLifecycle|TunnelSinkStallClaimSerializesAcceptedFrame|TunnelCredentialFailureReleasesAdmission)$'`
3. `go test -count=10 ./apps/node/internal/transport -run 'TestSessionLifetimeCancels(Run|Tunnel)Handler$'`
4. `go test -race -count=3 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
5. `go vet ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
6. `go test -count=1 ./packages/go/execution ./apps/node/...`
7. `go test -count=1 ./...`
8. `./scripts/e2e-smoke.sh`
9. `IOP_DEV_RECONNECT_BIND_TIMEOUT=300 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
10. `make readability-audit || test $? -eq 2`
11. `python3 - <<'PY'
import json

audit = json.load(open('build/readability-audit.json', encoding='utf-8'))
baseline = json.load(open('scripts/readability_baseline.json', encoding='utf-8'))
targets = {
    ('apps/node/internal/node/run_handler.go', 'Node.OnRunRequest'),
    ('apps/node/internal/node/tunnel_handler.go', 'Node.OnProviderTunnelRequest'),
    ('apps/node/internal/transport/session.go', 'newSession'),
}
base_functions = {(x['path'], x.get('function')): x['value'] for x in baseline['function_thresholds']}
current_functions = {(x['path'], x.get('function')): x['value'] for x in audit['violations'] if x.get('metric') == 'function_loc'}
bad = {key: current_functions.get(key, 0) for key in targets if current_functions.get(key, 0) > base_functions[key]}
base_sets = {x['task_id']: x['value'] for x in baseline['task_read_set_totals']}
current_sets = {x['task_id']: x['total_loc'] for x in audit['task_read_sets']}
if current_sets['node-core-readability'] > base_sets['node-core-readability']:
    bad['node-core-readability'] = current_sets['node-core-readability']
if bad:
    raise SystemExit(f'touched readability regression: {bad}')
print('touched readability regression: none')
PY`
12. `test -z "$(gofmt -l apps/node/internal/node/liveness_watchdog.go apps/node/internal/node/liveness_watchdog_test.go)"`
13. `git diff --check`

Expected: commands 1-9 and 11-13 exit 0. Command 10 may exit 0 or the known Make exit 2 only; command 11 must prove no touched-function/read-set regression and remaining unrelated audit findings must be recorded literally. Fresh execution is required; Go test cache output is not acceptance evidence. Do not modify readability baselines.

**After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.**
