<!-- task=m-node-provider-execution-liveness-recovery/02+01_stall_watchdog plan=6 tag=REVIEW_REVIEW_REVIEW_REVIEW_API milestone-task=activity-contract,stall-watchdog -->

# PLAN — Bind Watchdog Validity to the Scheduled Expiry Deadline

## For the Implementing Agent

> **MANDATORY:** Implement only this follow-up checklist and preserve unrelated worktree changes. Run every verification command, fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with literal results, keep the active pair in place, and report ready for review. If blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence. Do not ask the user, call user-input tools, create stop files, classify next state, archive logs, or write `complete.log`; finalization belongs to the official code-review agent.

## Background

The watchdog now rejects expiry signals invalidated before or after validity capture and passes the existing repeated/race/full-cycle checks. Official review found one remaining arm-identity hole: the observer records `armedAt` after creating or resetting the timer, so a valid short timeout can fire before bookkeeping and be rejected as stale forever. The follow-up must make the scheduled expiry deadline authoritative across initial arm, reset, receive-before-capture, and capture-before-claim orderings.

## Archive Evidence Snapshot

- The current pair will archive as `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G08_6.log` and `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G08_6.log`.
- Prior verdict: FAIL. Required=1, Suggested=0, Nit=0.
- Required fix: bind current timer-arm identity before the timer can fire and reject old-arm signals that race a progress reset without losing the only current-arm expiry.
- Fresh reviewer evidence: every planned focused/repeated/session/race/vet/Node/full-suite/smoke/readability/format/diff check passed, and the final prebuilt reconnect diagnostic passed its complete three-run transcript. A temporary deterministic reviewer test still failed when the current timer fired before constructor bookkeeping: `current timer signal was rejected because armedAt was recorded after the timer fired`.
- Roadmap carryover: preserve `milestone-task=activity-contract,stall-watchdog`; satisfy approved SDD S01/S02 evidence only and do not update roadmap state.

## Dependencies and Execution Order

- Runtime predecessor `01_activity_contract` is satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/01_activity_contract/complete.log`.
- Complete FIX-1 before VERIFY-1 so final verification exercises the corrected timer-arm contract.

## Analysis

### Files Read

- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/PLAN-cloud-G08.md`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/plan_cloud_G08_4.log`
- `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/code_review_cloud_G08_4.log`
- `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/01_activity_contract/complete.log`
- `agent-roadmap/phase/operational-observability-provider-management/PHASE.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `apps/node/internal/node/liveness_watchdog.go`
- `apps/node/internal/node/liveness_watchdog_test.go`
- `apps/node/internal/node/run_handler.go`
- `apps/node/internal/node/tunnel_handler.go`
- `apps/node/internal/node/runtime_sink.go`
- `apps/node/internal/node/node.go`
- `apps/node/internal/node/run_manager.go`
- `apps/node/internal/transport/session.go`
- `apps/node/internal/transport/session_test.go`
- `packages/go/execution/liveness.go`
- `packages/go/execution/failure.go`
- `packages/go/execution/types.go`
- `scripts/dev/edge-node-reconnect-diagnostic.sh`
- `scripts/dev/edge.sh`
- `scripts/dev/node.sh`
- `agent-test/local/rules.md`
- `agent-test/local/node-smoke.md`
- `agent-test/local/testing-smoke.md`
- `agent-ops/rules/project/domain/node/rules.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-ops/skills/project/e2e-smoke/SKILL.md`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status `[승인됨]`, lock released, no user review.
- Header ids remain `activity-contract,stall-watchdog`; both ids exist in the selected active Milestone.
- S01 requires every accepted provider progress event to reset the no-progress clock. S02 requires timer/event/reset/cancel/close races to converge on exactly one terminal and a trustworthy local fence.
- Evidence Map rows S01/S02 require fake-clock normalized/tunnel activity, threshold races, exactly-once terminal, confirmed/unconfirmed ownership, and late-output fencing. FIX-1 adds the missing initial-arm and reset-during-fire variants; VERIFY-1 reruns the existing S01/S02 matrix and full-cycle evidence.

### Verification Context

- No external handoff was supplied. Repository-native local rules, the approved SDD, contracts, current source/tests, and fresh reviewer commands are authoritative.
- Sources: `agent-test/local/rules.md`, `agent-test/local/node-smoke.md`, `agent-test/local/testing-smoke.md`, the testing domain rule, and `e2e-smoke` skill.
- Preconditions: Go `go1.26.2 linux/arm64`; module `/config/workspace/iop-s1/go.mod`; no external provider, credential, remote runner, or user authorization is required.
- Fresh reviewer reproduction: a package-private immediate timer fired at `t` before constructor bookkeeping returned `Now=t+1ms`; `expiryForSignal` rejected that current signal and the focused test exited 1. The temporary reviewer file was removed after capture.
- Existing evidence: focused count 20, session count 10, race count 3, vet, Node packages, full Go suite, auxiliary E2E, final reconnect transcript, touched readability comparison, formatting, and diff checks passed.
- Build-latency isolation: prebuild the Node binary to `/tmp/iop-review-node`, then pass it through `IOP_NODE_BIN` for the reconnect diagnostic. This preserves all transcript assertions while keeping local compilation outside the registration ceiling.
- Gaps: current tests cover old signals received before capture and before claim, but not a current signal firing before its deadline bookkeeping or an old arm firing during the reset operation. Confidence: high because the missing behavior has a deterministic direct reproducer and all runners are local.

### Test Coverage Gaps

- Initial current-arm fire before constructor bookkeeping: uncovered; the only current expiry is rejected and no later timer can stall the attempt.
- Old-arm fire during progress reset: uncovered; a fire time alone cannot identify the timer generation when it lands between progress bookkeeping and Stop/Reset.
- Receive-before-capture, capture-before-claim, normalized/tunnel terminal ordering, confirmed/unconfirmed ownership, deadline/cancel/session precedence, metadata cloning, and reconnect behavior are covered and must remain green.

### Symbol References

- No public symbol is renamed or removed.
- Package-local `attemptClock`, `attemptTimer`, `manualAttemptClock`, `manualAttemptTimer`, `attemptObserver.expired`, `attemptExpiry`, `expiryForSignal`, and `claimFence` are referenced only by `apps/node/internal/node/liveness_watchdog.go` and `apps/node/internal/node/liveness_watchdog_test.go`; update every listed call if the signal representation changes.

### Split Judgment

- Keep one atomic follow-up. Initial arm, reset, signal capture, and sink claim are one timer-generation invariant shared by normalized and tunnel handlers; splitting production logic from its deterministic temporal regressions would leave no independently safe intermediate state.
- Predecessor index `01` is satisfied by the archived `complete.log` listed above.

### Scope Rationale

- In scope: shared Node watchdog arm/deadline identity, deterministic observer/normalized/tunnel current-versus-old arm regressions, and fresh local verification evidence.
- Excluded: provider health probing, Edge runtime health overlay/recovery, activity/config/protobuf propagation, contract/spec wording, metrics, diagnostic-script semantics, readability baselines/read sets, and roadmap state. Existing contracts/spec already state the intended S01/S02 behavior.

### Final Routing

- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`; status `routed`.
- Build closures: scope/context/verification/evidence/ownership/decision all true; capability gap none.
- Build scores: scope=1, state=2, blast=1, evidence=2, verification=2 => G08; base `local-fit`; `recovery-boundary` selects cloud and `PLAN-cloud-G08.md`.
- Build signals: `large_indivisible_context=false`; positive risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product` (4); `review_rework_count=4`; `evidence_integrity_failure=true`; risk and recovery boundaries matched, with recovery precedence.
- Review closures: scope/context/verification/evidence/ownership/decision all true; scores 1/2/1/2/2 => G08; `official-review`, cloud, Codex `gpt-5.6-sol` xhigh, `CODE_REVIEW-cloud-G08.md`.

## Implementation Checklist

- [ ] [FIX-1] Make the scheduled expiry deadline or explicit arm generation authoritative before a timer can fire; reject old-arm signals across reset interleavings while accepting the sole current-arm signal, and add deterministic observer/normalized/tunnel regressions without weakening exactly-once terminal/fence behavior.
- [ ] [VERIFY-1] Run the focused temporal matrix and every final verification command, using a prebuilt `/tmp` Node binary for the unchanged reconnect transcript and recording literal zero-exit output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [FIX-1] Make timer-arm identity atomic with scheduling

**Problem**

At `apps/node/internal/node/liveness_watchdog.go:63-67`, `newAttemptObserver` calls `clock.NewTimer(timeout)` before `clock.Now()` records `armedAt`. At lines 88-90, progress similarly calls `timer.Reset` before updating `armedAt`. If a valid positive timeout fires during either gap, its signal can be older than the post-arm timestamp and be rejected as stale. Simply moving `Now` before `Reset` does not fully identify an old arm that fires while progress is stopping and rearming the timer.

**Solution**

Before (`liveness_watchdog.go:63-67,88-90,105-110`):

```go
return &attemptObserver{clock: clock, timer: clock.NewTimer(timeout), deadline: timeout, armedAt: clock.Now()}

o.epoch++
o.timer.Reset(o.deadline)
o.armedAt = o.clock.Now()

valid := !o.terminal && !o.fenced && !firedAt.Before(o.armedAt)
```

After:

```go
scheduledExpiry := clock.Now().Add(timeout)
observer := &attemptObserver{clock: clock, deadline: timeout, scheduledExpiry: scheduledExpiry}
observer.timer = clock.NewTimer(timeout)

nextExpiry := o.clock.Now().Add(o.deadline)
// Stop/drain the old arm, advance the generation, publish nextExpiry, then reset.

valid := !o.terminal && !o.fenced && !firedAt.Before(o.scheduledExpiry)
```

- Use `scheduledExpiry` or an explicit generation-bearing timer signal fixed before scheduling. A current arm must never be rejected because bookkeeping ran after fire; an old arm must never inherit the new epoch when it fires during reset.
- Keep the captured epoch through `claimFence` so progress after validity capture still invalidates the claim.
- Update the manual timer to emit its scheduled fire time, not an unrelated later `Now`, and add deterministic Stop/Reset interleaving control without scheduler sleeps.
- Preserve provider terminal/caller cancel/deadline/session precedence, exact `defaultAttemptCloseGrace=5s`, confirmed cleanup-before-terminal, unconfirmed retention, safe metadata, and exactly-once fencing.

**Modified Files and Checklist**

- [ ] `apps/node/internal/node/liveness_watchdog.go` — bind validity to the scheduled current arm before timer creation/reset and retain post-capture generation fencing.
- [ ] `apps/node/internal/node/liveness_watchdog_test.go` — model scheduled fire times and cover initial current-arm fire plus old-arm fire during normalized/tunnel resets.

**Test Strategy**

Required. Add `TestAttemptObserverCurrentArmSignalSurvivesImmediateFire`, `TestRunWatchdogOldArmFireDuringResetYieldsToProgress`, and `TestTunnelWatchdogOldArmFireDuringResetYieldsToProgress`. The first forces the current timer to fire before constructor bookkeeping can finish and must accept/fence it. The handler tests force the old arm to fire while accepted progress owns reset, assert no cancellation or terminal from that arm, then fire the new arm after its full threshold and assert exactly one confirmed terminal. Retain all existing receive-before-capture and capture-before-claim tests.

**Verification**

- `go test -count=20 ./apps/node/internal/node -run 'Test(AttemptObserverCurrentArmSignalSurvivesImmediateFire|RunWatchdogOldArmFireDuringResetYieldsToProgress|TunnelWatchdogOldArmFireDuringResetYieldsToProgress|RunWatchdogStaleExpiryYieldsToProgress|TunnelWatchdogStaleExpiryYieldsToProgress|RunWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal)$'`
- `go test -race -count=3 ./apps/node/internal/node -run 'Test(AttemptObserverCurrentArmSignalSurvivesImmediateFire|RunWatchdogOldArmFireDuringResetYieldsToProgress|TunnelWatchdogOldArmFireDuringResetYieldsToProgress|RunWatchdogStaleExpiryYieldsToProgress|TunnelWatchdogStaleExpiryYieldsToProgress|RunWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal)$'`

### [VERIFY-1] Re-run the complete S01/S02 evidence

**Problem**

The existing verification matrix passed while the new reviewer reproducer failed, so prior green output is insufficient evidence for the corrected arm-identity invariant. The reconnect transcript also showed that build latency can consume the registration ceiling when `node.sh` builds internally.

**Solution**

- Run the focused temporal tests before the complete repeated/race/package suite.
- Build the Node binary to `/tmp/iop-review-node` and set `IOP_NODE_BIN=/tmp/iop-review-node` for the unchanged reconnect diagnostic. Do not edit its configuration or transcript assertions.
- Record literal stdout/stderr and exit codes for every command; fresh execution is required and Go test cache output is not acceptance evidence.

**Modified Files and Checklist**

- [ ] `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md` — record literal final verification and implementation decisions.

**Test Strategy**

No new product test beyond FIX-1. The existing fail-fast diagnostic remains the full-cycle oracle; the prebuilt binary isolates compilation from runtime registration without weakening any assertion.

**Verification**

- `go build -o /tmp/iop-review-node ./apps/node/cmd/node`
- `IOP_NODE_BIN=/tmp/iop-review-node IOP_DEV_RECONNECT_BIND_TIMEOUT=300 ./scripts/dev/edge-node-reconnect-diagnostic.sh`

## Modified Files Summary

| Path | Action | Checklist |
|------|--------|-----------|
| `apps/node/internal/node/liveness_watchdog.go` | modify | FIX-1 |
| `apps/node/internal/node/liveness_watchdog_test.go` | modify | FIX-1 |
| `agent-task/m-node-provider-execution-liveness-recovery/02+01_stall_watchdog/CODE_REVIEW-cloud-G08.md` | update evidence | FIX-1, VERIFY-1 |

## Final Verification

1. `go version && go env GOMOD`
2. `go test -count=20 ./apps/node/internal/node -run 'Test(AttemptObserverCurrentArmSignalSurvivesImmediateFire|RunWatchdogOldArmFireDuringResetYieldsToProgress|TunnelWatchdogOldArmFireDuringResetYieldsToProgress|RunWatchdogStaleExpiryYieldsToProgress|TunnelWatchdogStaleExpiryYieldsToProgress|RunWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelWatchdogStaleExpiryBeforeCaptureYieldsToProgress|TunnelConfirmedFenceClosesOwnershipBeforeTerminal|RunWatchdogLifecycle|TunnelWatchdogLifecycle|TunnelSinkStallClaimSerializesAcceptedFrame|TunnelCredentialFailureReleasesAdmission)$'`
3. `go test -count=10 ./apps/node/internal/transport -run 'TestSessionLifetimeCancels(Run|Tunnel)Handler$'`
4. `go test -race -count=3 ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
5. `go vet ./packages/go/execution ./apps/node/internal/node ./apps/node/internal/transport`
6. `go test -count=1 ./packages/go/execution ./apps/node/...`
7. `go test -count=1 ./...`
8. `./scripts/e2e-smoke.sh`
9. `go build -o /tmp/iop-review-node ./apps/node/cmd/node`
10. `IOP_NODE_BIN=/tmp/iop-review-node IOP_DEV_RECONNECT_BIND_TIMEOUT=300 ./scripts/dev/edge-node-reconnect-diagnostic.sh`
11. `make readability-audit || test $? -eq 2`
12. `python3 - <<'PY'
import json

audit = json.load(open('build/readability-audit.json', encoding='utf-8'))
baseline = json.load(open('scripts/readability_baseline.json', encoding='utf-8'))
targets = {
    ('apps/node/internal/node/run_handler.go', 'Node.OnRunRequest'),
    ('apps/node/internal/node/tunnel_handler.go', 'Node.OnProviderTunnelRequest'),
    ('apps/node/internal/transport/session.go', 'newSession'),
}
base_functions = {(x['path'], x.get('function')): x['value'] for x in baseline['function_thresholds']}
current_functions = {(x['path'], x.get('function')): x['value'] for x in audit['violations'] if x.get('metric') == 'function_loc'}
bad = {key: current_functions.get(key, 0) for key in targets if current_functions.get(key, 0) > base_functions[key]}
base_sets = {x['task_id']: x['value'] for x in baseline['task_read_set_totals']}
current_sets = {x['task_id']: x['total_loc'] for x in audit['task_read_sets']}
if current_sets['node-core-readability'] > base_sets['node-core-readability']:
    bad['node-core-readability'] = current_sets['node-core-readability']
if bad:
    raise SystemExit(f'touched readability regression: {bad}')
print('touched readability regression: none')
PY`
13. `test -z "$(gofmt -l apps/node/internal/node/liveness_watchdog.go apps/node/internal/node/liveness_watchdog_test.go)"`
14. `git diff --check`

Expected: commands 1-10 and 12-14 exit 0. Command 11 may exit 0 or the known Make exit 2 only; command 12 must prove no touched-function/read-set regression and remaining unrelated audit findings must be recorded literally. The initial current-arm and reset-during-fire tests must fail on the reviewed implementation and pass after FIX-1. Do not modify readability baselines.

**After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.**
