<!-- task=m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy plan=5 tag=REVIEW_REFACTOR milestone-task=bounded-retry -->

# Recovery Integration Evidence and Safe Temporary Root Follow-up

## For the Implementing Agent

Implement only the direct fixes below, run every verification command, and fill all implementation-owned sections of `CODE_REVIEW-cloud-G08.md` with actual notes and raw output. Keep active files in place and report ready for review; finalization belongs to the code-review skill. If blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.

## Background

The recovery policy now applies under the queue lock, but its required integration evidence still skips the `SubmitProviderPool` queued resolver seam and several unavailable/unknown terminal branches. The capacity smoke also accepts relative or symlinked temporary roots that resolve inside the checkout, contradicting its no-repository-artifact invariant. This follow-up closes those two review-trust gaps without changing the recovery contract or adding retry ownership.

## Archive Evidence Snapshot

- The failed pair is `plan_cloud_G08_4.log` and `code_review_cloud_G08_4.log` in this task directory. Verdict: FAIL with Required R1-R2, `review_rework_count=2`, and `evidence_integrity_failure=true`.
- Fresh focused, race, selected package, vet, `git diff --check`, provider-only E2E, normal capacity smoke, and forced-noexec `/tmp` capacity smoke all passed.
- Fresh `IOP_PROVIDER_CAPACITY_SMOKE_TMP_ROOT=. ./scripts/e2e-provider-capacity-smoke.sh` selected `tmp_root=.` and built temporary binaries below the repository before cleanup, proving the raw lexical exclusion is insufficient. `KEEP_TMP=1` would preserve those artifacts.
- The previous remote long-context preflight remains source-unsynchronized and blocked on the Control Plane status endpoint. It is auxiliary evidence and is not repeated against an unchanged precondition in this packet.
- Roadmap carryover remains `milestone-task=bounded-retry`, SDD S05. This packet supplies only candidate-selection and deterministic smoke evidence; StreamGate recovery ownership remains outside this task.

## Finding Resolution Map

| Finding | Mode | Exact fix evidence | Changed precondition |
|---------|------|--------------------|----------------------|
| Required R1 | direct-fix | `apps/edge/internal/service/provider_recovery_selection_test.go` | A real `SubmitProviderPool` waiter crosses default live resolver/catalog re-resolution with no operation/custom predicate, while same-only unavailable/unknown branches have explicit terminal assertions and dispatch/lease counts. |
| Required R2 | direct-fix | `scripts/e2e-provider-capacity-smoke.sh` | Every candidate is absolute and physically canonicalized before its execution probe; roots resolving at or below the checkout are skipped, including relative inputs and symlink aliases. |

## Analysis

### Files Read

- `apps/edge/internal/service/provider_pool.go`
- `apps/edge/internal/service/model_queue_admission.go`
- `apps/edge/internal/service/model_queue_types.go`
- `apps/edge/internal/service/provider_recovery_selection_test.go`
- `scripts/e2e-provider-capacity-smoke.sh`
- `agent-contract/inner/execution-runtime.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-roadmap/phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md`
- `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`
- `agent-test/local/rules.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-test/local/testing-smoke.md`
- `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/code_review_cloud_G06_3.log`

### SDD Criteria

- SDD: `agent-roadmap/sdd/operational-observability-provider-management/node-provider-execution-liveness-recovery/SDD.md`; status approved, lock released.
- First-line scope: `milestone-task=bounded-retry`; targeted Acceptance Scenario and Evidence Map row: S05.
- S05 requires provider-pool failover and a bounded dispatch count under ingress-owned recovery. The follow-up therefore proves request-local policy retention through the public service queue seam, live candidate-universe changes, unavailable/unknown terminals, and exactly one dispatch/lease. It does not add StreamGate recovery, retry counters, or replay ownership.

### Verification Context

- No neutral handoff was supplied. Repository-native fallback came from the Edge/testing local profiles, current source/tests, the failed review, and fresh reviewer commands.
- Fresh reviewer PASS: `go test -count=20` focused recovery tests, `go test -race -count=3` focused recovery tests, selected package tests, selected `go vet`, `git diff --check`, `scripts/e2e-smoke.sh`, normal capacity smoke, and forced-noexec `/tmp` capacity smoke.
- Fresh reviewer FAIL: relative override selected `tmp_root=.` and built under the checkout. Cleanup removed the dynamic directory in the default mode, but the selector violated its physical-root invariant and `KEEP_TMP=1` would retain it.
- External Verification Preflight: the declared runner remains `ssh toki@toki-labs.com`, workdir `/Users/toki/agent-work/iop-dev`, previously observed HEAD `61016d5bd0940033d68e1862bc20e1b7108b8875`, with one unrelated untracked backup. That checkout is not synchronized to this worktree, and the long-context preflight remains blocked by its Control Plane status endpoint. The packet changes only local tests and the deterministic local smoke root selector, so repeating the unchanged remote preflight would add no evidence. Confidence: high.

### Test Coverage Gaps

- Immediate queue-core tests cover healthy alternate preference and explicit same-provider fallback, but do not cover a same-only runtime-unavailable or configured-unknown provider terminal.
- The only full `SubmitProviderPool` recovery test dispatches immediately; it cannot detect lost recovery hints in the default queued resolver path.
- The queued test calls `admitWithRecovery` directly and changes only the overlay. It does not change the service's live catalog candidate universe before a pump.
- Normal and `/tmp` capacity runs cover executable fallback, but no check rejects relative roots or absolute symlinks that physically resolve into the repository.

### Symbol References

- No production symbol is renamed or removed.

### Split Judgment

- Keep one compact packet. Both fixes close the same failed review's evidence-integrity boundary for S05, and neither creates useful standalone Milestone completion evidence. The production recovery policy remains unchanged.
- Split predecessor `08+07_health_overlay` remains satisfied by `agent-task/archive/2026/08/m-node-provider-execution-liveness-recovery/08+07_health_overlay/complete.log` as recorded in the prior plan.

### Scope Rationale

- Do not modify `provider_pool.go`, queue production logic, contracts, specs, SDD, or roadmap unless a newly added deterministic regression fails and proves the existing production behavior is wrong; this packet is scoped to missing evidence and temporary-root safety.
- Do not implement StreamGate recovery, retry budgets, new run identity, replay eligibility, runtime-health transitions, or provider priority changes.
- Do not rerun the source-unsynchronized remote long-context scenario against its unchanged status-endpoint blocker.

### Final Routing

- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh pair`.
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `(1,2,1,2,2)`, G08, base `local-fit`, final `recovery-boundary` because `review_rework_count=2` and `evidence_integrity_failure=true`; canonical file `PLAN-cloud-G08.md`.
- Review closures: all closed. Scores `(1,2,1,2,2)`, G08, route `official-review`; canonical file `CODE_REVIEW-cloud-G08.md` (`codex`, `gpt-5.6-sol`, `xhigh`).
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract` (3). No capability gap.

## Implementation Checklist

- [ ] REVIEW_REFACTOR-1 adds service-level queued catalog re-resolution and same-only unavailable/unknown terminal regressions, preserving no-operation/no-custom-predicate recovery hints and proving exactly one lease/wire dispatch.
- [ ] REVIEW_REFACTOR-2 physically canonicalizes temporary-root candidates before probing, rejects relative and repo-resolving roots, and proves relative/symlink overrides cannot create or retain repository-local binaries.
- [ ] Run focused, race, selected package/vet, provider-only/capacity, root-safety, and deterministic diff verification with raw current output.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.

### [REVIEW_REFACTOR-1] Close the public queued resolver evidence gap

**Problem:** `apps/edge/internal/service/provider_recovery_selection_test.go:213` queues through `admitWithRecovery` directly and always returns the same two-candidate slice. It cannot detect a regression in `SubmitProviderPool`'s default resolver composition, does not mutate the catalog universe before pumping, and the immediate table at line 88 lacks same-only unavailable/unknown terminal cases required by the prior plan.

**Solution:** Add a service/net.Pipe queued integration that initially fills the alternate provider, submits a recovery request with empty operation and nil custom predicate, changes the live model/provider candidate universe before the pump, and asserts fallback-true or fallback-false behavior through `SubmitProviderPool`. Retain distinct provider adapter identities on the wire and assert one recovery dispatch, one recovery lease, no forbidden avoided-provider send without permission, and settled counters. Extend the terminal table for a same-only runtime-unavailable avoided provider and a configured-unknown provider.

Before (`apps/edge/internal/service/provider_recovery_selection_test.go:245`):

```go
candidate, _, admitErr := m.admitWithRecovery(ctx, recoveryGroupKey, "", recoveryServed,
    candidates, groupPolicy{}, resolver, false, true, recovery)
```

After:

```go
result, err := svc.SubmitProviderPool(ctx, ProviderPoolDispatchRequest{
    Run: SubmitRunRequest{ModelGroupKey: recoveryGroupKey, ProviderPool: true, Background: true},
    AvoidProviderID: recoveryAvoidID,
    AllowAvoidedProviderFallback: allowFallback,
})
```

**Modified Files and Checklist:**

- [ ] `apps/edge/internal/service/provider_recovery_selection_test.go`: add the public queued resolver/catalog regression, unavailable/unknown terminal cases, exact dispatch/lease assertions, and cleanup.

**Test Strategy:** Add `TestProviderRecoverySelectionServiceQueuedReresolution` with fallback true/false subtests and distinct tunnel adapters. Extend or add `TestProviderRecoverySelectionServiceRejectsUnavailableOrUnknownAvoidedProvider`. Use bounded contexts, wait for exactly one pending provider-pool item, change the live catalog/provider availability before pumping, and assert the resulting error or `DispatchInfo.ProviderID`, wire count, lease count, and final counters.

**Verification:** `go test -count=20 ./apps/edge/internal/service -run '^TestProviderRecoverySelection'` and the matching race command must pass every iteration.

### [REVIEW_REFACTOR-2] Exclude physical repository roots from smoke temporaries

**Problem:** `scripts/e2e-provider-capacity-smoke.sh:82` compares raw candidate strings with an absolute `REPO_ROOT`. Relative `.` bypasses the comparison, and an absolute symlink into the checkout has the same defect. Fresh relative-root execution logged `tmp_root=.` and built every temporary binary under the repository.

**Solution:** Resolve the physical repository root once. Require candidate roots to be absolute, create only absolute candidates, resolve each accepted directory with `pwd -P`, and compare the physical result against the physical repository root before calling `probe_exec_root`. Print and use only the validated physical path. A rejected caller override falls through to the next safe candidate.

Before (`scripts/e2e-provider-capacity-smoke.sh:80`):

```bash
for root in "${candidates[@]}"; do
    case "$root" in
    "$REPO_ROOT" | "$REPO_ROOT"/*) continue ;;
    esac
    if probe_exec_root "$root"; then
```

After:

```bash
for root in "${candidates[@]}"; do
    case "$root" in /*) ;; *) continue ;; esac
    mkdir -p "$root" 2>/dev/null || continue
    physical_root="$(cd "$root" && pwd -P)"
    case "$physical_root" in
    "$REPO_ROOT_PHYSICAL" | "$REPO_ROOT_PHYSICAL"/*) continue ;;
    esac
    if probe_exec_root "$physical_root"; then
```

**Modified Files and Checklist:**

- [ ] `scripts/e2e-provider-capacity-smoke.sh`: canonicalize candidate roots, exclude physical repository paths before execution probing, and keep exact cleanup/`KEEP_TMP` behavior.
- [ ] `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/CODE_REVIEW-cloud-G08.md`: record raw current root-safety and smoke output only.

**Test Strategy:** Run the normal and forced-noexec smokes. Then run the smoke with `.` and with an absolute symlink to the checkout as caller overrides, capture output outside the repository, assert the selected physical root is outside the checkout, and compare before/after repository temp-directory snapshots. No repository-local probe or binary may remain even with a rejected override.

**Verification:** The root-safety command in Final Verification must pass and every capacity run must emit its deterministic PASS line from a non-repository physical root.

## Dependencies and Execution Order

1. Add REVIEW_REFACTOR-1 evidence without changing production recovery ownership.
2. Fix REVIEW_REFACTOR-2 root selection before running the capacity and root-safety commands.
3. Run all final verification from the current checkout; do not repeat the unchanged remote blocker.

## Modified Files Summary

| File | Item |
|------|------|
| `apps/edge/internal/service/provider_recovery_selection_test.go` | REVIEW_REFACTOR-1 |
| `scripts/e2e-provider-capacity-smoke.sh` | REVIEW_REFACTOR-2 |
| `agent-task/m-node-provider-execution-liveness-recovery/09+08_retry_candidate_policy/CODE_REVIEW-cloud-G08.md` | REVIEW_REFACTOR-2 evidence |

## Final Verification

Fresh output is required; cached or reconstructed output is not acceptable.

1. `go version && go env GOMOD` — Go and the current module root resolve.
2. `go test -count=20 ./apps/edge/internal/service -run '^TestProviderRecoverySelection'` — immediate, public queued/catalog, unavailable/unknown, and dispatch-count cases pass repeatedly.
3. `go test -race -count=3 ./apps/edge/internal/service -run '^TestProviderRecoverySelection'` — no race is reported.
4. `go test -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — selected packages pass.
5. `go vet ./packages/go/streamgate ./apps/edge/internal/openai ./apps/edge/internal/service ./apps/edge/internal/controlplane` — no diagnostics.
6. `./scripts/e2e-smoke.sh` — provider-only Edge/Node smoke passes.
7. `./scripts/e2e-provider-capacity-smoke.sh` — deterministic capacity smoke passes from an executable non-repository root.
8. `IOP_PROVIDER_CAPACITY_SMOKE_TMP_ROOT=/tmp ./scripts/e2e-provider-capacity-smoke.sh` — noexec `/tmp` is rejected and the smoke safely falls back.
9. Run the following root-safety regression; both overrides must fall back outside the physical checkout and the repository temp-directory snapshot must remain unchanged:

```bash
set -euo pipefail
repo_root="$(pwd -P)"
evidence_root="$(mktemp -d "$(go env GOCACHE)/iop-capacity-root-check.XXXXXX")"
trap 'rm -rf "$evidence_root"' EXIT
ln -s "$repo_root" "$evidence_root/repo-link"
before="$(find "$repo_root" -maxdepth 1 -type d -name 'iop-provider-capacity-smoke.*' -print | sort)"
for candidate in . "$evidence_root/repo-link"; do
    log_file="$evidence_root/$(basename "$candidate").log"
    IOP_PROVIDER_CAPACITY_SMOKE_TMP_ROOT="$candidate" ./scripts/e2e-provider-capacity-smoke.sh | tee "$log_file"
    selected="$(sed -n 's/^\[provider-capacity-smoke\] tmp_root=//p' "$log_file" | head -n 1)"
    test -n "$selected"
    selected_physical="$(cd "$selected" && pwd -P)"
    case "$selected_physical" in "$repo_root" | "$repo_root"/*) exit 1 ;; esac
done
after="$(find "$repo_root" -maxdepth 1 -type d -name 'iop-provider-capacity-smoke.*' -print | sort)"
test "$before" = "$after"
```

10. `git status --short -- scripts/e2e-provider-capacity-smoke.sh apps/edge/internal/service/provider_recovery_selection_test.go && git diff --check` — only intended files are present and no whitespace error exists.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
