<!-- task=m-provider-resource-admission-ownership-alignment/11+08,13,15_reconnect_candidate_recovery plan=4 tag=REVIEW_REFACTOR -->

# Reconnect ownership의 queue 선형화와 재현 가능한 직접 증거

## 이 파일을 읽는 구현 에이전트에게

Implementation 모델은 Plan/Review 모델과 달라야 한다. 코드·테스트·검증을 수행하고 review stub의 구현 소유 섹션을 채운 뒤 `review-ready`를 보고하고 종료한다. code-review를 실행하거나 다음 에이전트를 시작하지 않는다.

## 역할 모델

- Plan: `gpt-5.6-terra`
- Implementation/Review identity 원장은 `CODE_REVIEW-local-G10.md`에 있으며 Runtime이 각 역할 시작 시 기록한다.

## 배경

직전 FAIL은 ready callback의 provider activation/queue pump가 authoritative unregister와 같은 ownership gate에 있지 않음을 확인했다. provider-pool의 normalized/tunnel send race, direct wire payload 검증, 그리고 `scripts/dev` transcript도 성공처럼 보였지만 필수 관찰을 실제로 단언하지 않았다. 이 작업은 S02/S12/S15가 요구하는 current generation의 정확히 한 번 handoff·activation·waiter recovery를 실제 queue와 직접 사용자 흐름에서 재현 가능하게 만든다.

## 사용자 리뷰 요청 흐름

선택 Milestone의 `구현 잠금 > 결정 필요`만 active review stub의 `사용자 리뷰 요청`에 기록한다. 구현 중 직접 user prompt는 금지되며 code-review가 요청 검증과 실제 `USER_REVIEW.md` 작성을 소유한다. 환경·endpoint·credential 제한은 검증 evidence와 재개 조건으로 기록하며 사용자 리뷰 사유가 아니다.

## Archive Evidence Snapshot

- 직전 pair evidence: `plan_local_G09_3.log`, `code_review_local_G09_3.log`; verdict=`FAIL`. `NEXT_STATE.md`의 task/plan/verdict가 이 follow-up과 일치하며 Review 소유 파일이므로 변경하지 않는다.
- Required: (1) ready activation/event/unregister를 실제 queue activation까지 하나의 ownership linearization으로 묶고 stale/pending activation·pump·event가 0임을 증명, (2) normalized/tunnel에서 disconnect-winning과 send-winning을 각각 결정적으로 고정해 wire/handle/lease/terminal/counter를 증명, (3) every direct wire의 exact protobuf payload를 type별 exactly-once로 단언, (4) `scripts/dev` 직접 transcript에서 Node/Edge payload, terminal ordering, command 응답을 fail-fast 비교, (5) gofmt와 load-sensitive transport suite 신뢰 회복.
- 유지 evidence: live provider preflight `rc=3`은 PASS가 아닌 외부 revalidation blocker다. 추가 문맥이 필요한 경우에만 위 두 log와 `NEXT_STATE.md`를 읽는다.

## Roadmap Targets

- Milestone: `agent-roadmap/phase/operational-observability-provider-management/milestones/provider-resource-admission-ownership-alignment.md`
- Milestone link: [Milestone 문서](../../../agent-roadmap/phase/operational-observability-provider-management/milestones/provider-resource-admission-ownership-alignment.md)
- Task ids:
  - `reconnect-candidate-recovery`: current accepted generation만 candidate 복구·waiter pump·connected event를 열고 stale/rejected transition은 no-op으로 유지
- Completion mode: check-on-pass

## 분석 결과

### 읽은 파일

- `apps/edge/internal/node/registry.go:166-310`, `registry_test.go:58-424`
- `apps/edge/internal/transport/connection_handlers.go:145-285`, `integration_test.go:1345-1490`
- `apps/edge/internal/service/service.go:45-110`, `provider_pool.go:150-255`, `provider_tunnel.go:220-270`, `run_dispatch_internal_test.go:1095-1530`
- `apps/edge/internal/bootstrap/runtime.go:113-114`, `reconnect_readiness_integration_test.go` (targeted existing test inventory)
- `scripts/dev/edge-node-reconnect-diagnostic.sh:1-290`, `scripts/dev/edge.sh`, `scripts/dev/node.sh`, `scripts/e2e-smoke.sh`
- `agent-contract/inner/edge-node-runtime-wire.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-spec/runtime/edge-node-execution.md`, `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-test/local/rules.md`, `agent-test/local/edge-smoke.md`, `agent-test/local/node-smoke.md`
- `agent-task/m-provider-resource-admission-ownership-alignment/11+08,13,15_reconnect_candidate_recovery/plan_local_G09_3.log`, `code_review_local_G09_3.log`, `NEXT_STATE.md`

### SDD 기준

- `agent-roadmap/sdd/operational-observability-provider-management/provider-resource-admission-ownership-alignment/SDD.md`: `[승인됨]`, SDD 잠금 `해제`.
- S02 (`lease-lifecycle`)는 disconnect 경쟁에서도 lease/counter를 정확히 한 번 종료하도록, S12 (`disconnect-exclusion`)는 current owner만 fenced generation을 막도록, S15 (`reconnect-candidate-recovery`)는 current reconnect만 queued waiter를 즉시 pump하도록 요구한다.
- Evidence Map의 deterministic ownership/queue/race evidence가 체크리스트 1–3과 final race commands를, direct transcript는 user-flow 검증을 형성한다. 이 pair가 PASS해도 `Roadmap Completion`은 `reconnect-candidate-recovery`만 주장한다.

### 테스트 환경 규칙

- `test_env=local`; `agent-test/local/rules.md`, `edge-smoke.md`, `node-smoke.md`를 적용한다. Go 검증은 cache를 허용하지 않고 모두 `-count=1` 또는 반복 `-count=20`을 사용한다.
- direct diagnostic은 `/tmp` 임시 config/FIFO/log와 `IOP_EDGE_CONFIG`/`IOP_NODE_CONFIG` dev-only override만 사용한다. tracked config, bootstrap UX, proto 계약은 수정하지 않는다.
- live preflight는 `--preflight`의 실제 stdout/stderr와 rc를 기록한다. rc=0일 때만 `--scenario normal-10`을 추가 실행하고, rc=3은 `/models`와 Control Plane status 재개 조건을 적는 blocker다.

### 테스트 커버리지 공백

- ready 이후 `notifyNodeConnected`는 registry gate 밖에서 service queue를 activate할 수 있다. 기존 race는 raw unregister와 dummy handler여서 S12/S15 actual queue interleaving을 닫지 못한다.
- 현재 normalized/tunnel race는 `Sleep`과 `Unregister`에 의존하고 success만 기대한다. disconnect-winning/send-winning 양쪽의 wire, handle, lease, terminal, counter 증거가 없다.
- ready-after direct test는 type count만 보존한다. request id, run id, background, adapter/target/input, tunnel method/path/body, cancel action/run id, command id/type/payload의 exact proto identity가 없다.
- direct diagnostic은 run id를 Edge message line에서 찾으므로 payload comparison과 command assertions를 skip한 채 PASS할 수 있다.

### 심볼 참조

- 변경 후보: `Registry.UnregisterIfClient`, `Registry.MarkDispatchReadyOwner`, `Registry.WithCurrentOwner`, `Registry.WithCurrentDispatchOwner`, `Server.handleReadyRequest`, `Server.bindDisconnectListener`, `Service.HandleNodeConnect`, `Service.HandleNodeDisconnect`, `modelQueueManager.activateNode`, `dispatchProviderPoolRun`, `dispatchProviderPoolTunnel`.
- direct send call sites: `SubmitRun`, `SubmitProviderTunnel`, `CancelRun`, `TerminateSession`, `UsageStatus`, `SessionList`, `TransportStatus`, `OllamaAPI`, `ExecuteCommand(agent.command)`; renamed/removed symbol은 없다.

### 분할 판단

- 선택 candidate는 FAIL follow-up `11+08,13,15_reconnect_candidate_recovery` 하나이며 immediate sibling set도 이 하나다.
- write set: Edge registry/transport/service 및 focused tests, `scripts/dev/edge-node-reconnect-diagnostic.sh`. shared mutable state는 owner generation, queue lease, disconnect ordering, one console session이므로 병렬 분리는 안전하지 않다.
- directory predecessor `08,13,15`는 이미 명시된 runtime dependency다. 추가 split은 production gate와 interleaving proof를 분리해 Required의 atomicity를 잃으므로 하지 않는다.

### 범위 결정 근거

- proto field, Node handler protocol, provider selection/admission policy, default config, contract/spec 문서는 Required 해결 수단이 아니므로 수정하지 않는다.
- `ResolveNode`의 관찰용 lookup과 accepted-pending visibility는 유지한다. 변경은 current ready owner의 activation/send/handoff와 그 proof에 한정한다.
- `scripts/e2e-smoke.sh`는 auxiliary smoke로 남기며 direct `scripts/dev` transcript의 대체 evidence가 아니다.

### 최종 라우팅

- `evaluation_mode=isolated-reassessment`; 이전 lane/grade/score/rationale/filename은 입력과 하한에서 제외했다.
- build closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`. 현재 source와 FAIL evidence가 scope를 닫고 deterministic barriers·payload capture·direct transcript·blocker 조건으로 판정 가능하다. scores=`2,2,2,2,2`; lane=`local`, grade=`G10`, filename=`PLAN-local-G10.md`.
- review closures: `scope_closed=true`, `context_closed=true`, `verification_closed=true`, `evidence_trusted=true`, `ownership_closed=true`, `decision_closed=true`. review는 actual queue linearization, 두 send 순서, cloned proto identity, transcript fail-fast와 repeated test output을 source와 대조한다. scores=`2,2,2,2,2`; lane=`local`, grade=`G10`, filename=`CODE_REVIEW-local-G10.md`.

## 구현 체크리스트

- [ ] ready activation/event/unregister를 actual provider queue lifecycle과 동일 ownership linearization으로 만들고 stale/pending activation·pump·event 0 및 current generation 1을 deterministic하게 검증한다.
- [ ] normalized와 tunnel 각각에서 disconnect-winning/send-winning schedule을 channel barrier로 고정하고 no-send/no-handle 또는 wire 1/terminal-release 1/final counter 0을 검증한다.
- [ ] pending/ready direct fence를 per-wire cloned protobuf exact payload assertions로 바꾼다.
- [ ] `scripts/dev` direct reconnect diagnostic이 Node/Edge payload sequence, terminal ordering, required command response를 실제로 fail-fast 검증하게 만든다.
- [ ] gofmt diff를 0으로 만들고 load-sensitive transport race suite를 반복해 실제 stdout/stderr를 기록한다.
- [ ] CODE_REVIEW-*-G??.md의 구현 에이전트 소유 섹션을 채우고 review-ready를 보고한 뒤 현재 실행을 종료한다. code-review를 실행하거나 다음 에이전트를 시작하지 않는다.

### [REVIEW_REFACTOR-1] Ready/disconnect와 queue activation을 한 ownership gate로 선형화

문제: `apps/edge/internal/transport/connection_handlers.go:157-175`는 ready snapshot 뒤 `notifyNodeConnected`을 registry 밖에서 실행하고, `apps/edge/internal/service/service.go:81-92`가 별도 generation check 뒤 queue를 activate한다. `UnregisterIfClient`(`registry.go:171-182`)가 그 사이 완료되면 old continuation이 stale resource activation/pump를 실행할 수 있다.

해결 방법:

```go
// Before: connection_handlers.go:168-175 — lifecycle은 owner gate 밖이다.
s.notifyNodeConnected(nodeID, entry.ConnectionGeneration)
s.registry.WithCurrentOwner(entry, func() { s.emitNodeReady(entry) })

// After: one current-owner callback contains lifecycle activation and event.
s.registry.WithCurrentOwner(entry, func() {
    s.notifyNodeConnected(entry.NodeID, entry.ConnectionGeneration)
    s.emitNodeReady(entry)
})
```

`WithCurrentOwner` callback이 queue activation을 수행하는 동안 `UnregisterIfClient`와 직렬화되도록, handler/registry/service 경계를 좁게 조정한다. lock 안에서 socket I/O나 unrelated queue lock 대기는 하지 않으며, disconnect는 same owner gate 뒤 authoritative release와 pump를 유지한다.

수정 파일 및 체크리스트:

- [ ] `apps/edge/internal/node/registry.go`, `registry_test.go`: current-ready owner callback의 lock/ordering contract를 최소 API로 명확히 하고 stale/duplicate/replaced owner no-op을 검증한다.
- [ ] `apps/edge/internal/transport/connection_handlers.go`, `integration_test.go`: ready activation, connected event, unregister ordering을 하나의 owner gate로 묶는다. real `Service`/queue hook + channel barrier로 old ready → current disconnect → accepted pending reconnect → release를 강제해 stale/pending activation·pump·event=0, new ready activation·pump·event=1을 검증한다.
- [ ] `apps/edge/internal/service/service.go`, focused service/bootstrap reconnect tests: stale callback이 activate하지 않고 current reconnect가 stranded waiter를 정확히 한 번 terminal까지 수렴시키는지 보강한다.

테스트 작성: `time.Sleep`와 manual `Registry.Unregister`로 순서를 추정하지 않는다. authoritative disconnect listener가 완료된 signal과 actual queue activation/pump hook을 사용하며 S12/S15의 stale no-op 및 reconnect recovery를 함께 단언한다.

### [REVIEW_REFACTOR-2] Normalized/tunnel send 경합의 두 승패 순서를 결정적으로 증명

문제: `apps/edge/internal/service/run_dispatch_internal_test.go:1310-1510`은 write barrier 전후에 `Sleep`과 raw `Unregister`를 사용하며 두 path 모두 success handle을 기대한다. `provider_pool.go:169-239`와 `provider_tunnel.go:242-253`의 send/handoff가 authoritative disconnect와 경쟁할 때 실제 winner와 lease lifecycle이 증명되지 않는다.

해결 방법:

```go
// Before: test schedule is inferred from sleeps and expects success.
time.Sleep(50 * time.Millisecond)
reg.Unregister(nodeID)

// After: a fake client reports fence/write entry; test chooses winner explicitly.
<-writeEntered
disconnectCurrentOwner()
releaseWrite(false) // disconnect-winning: no wire, no handle
```

Each path has two subtests. Disconnect-winning prevents enqueue/send and returns no success handle; send-winning records exactly one wire then authoritative disconnect produces exactly one terminal/release. Both assert reservation identity and all general/long/queue counters finish at zero; temporary `DIAGNOSTIC` logging is removed.

수정 파일 및 체크리스트:

- [ ] `apps/edge/internal/service/provider_pool.go`, `provider_tunnel.go`, `run_submit.go` only if a narrow testable ownership/send boundary is necessary; preserve admission policy and avoid holding unrelated queue/service locks across network I/O.
- [ ] `apps/edge/internal/service/run_dispatch_internal_test.go`: replace sleep/raw-unregister races with barrier-driven current-owner disconnect for normalized and tunnel, each in disconnect-winning and send-winning variants.
- [ ] focused queue reservation/service tests: assert one lease id has one terminal/release and final capacity/long/queue counters are zero.

테스트 작성: fake client must separately signal ownership-gate/write entry and optionally allow one write. The receiver captures a cloned `RunRequest` or `ProviderTunnelRequest`; no `result`/handle may escape in disconnect-winning, while send-winning has exactly one wire, one disconnect terminal/release, and no duplicate lifecycle.

### [REVIEW_REFACTOR-3] Direct ready-after wire를 exact protobuf payload로 검증

문제: `apps/edge/internal/service/run_dispatch_internal_test.go:1095-1307`는 string type count만 기록해 fields가 비었거나 잘못된 요청도 통과할 수 있다.

해결 방법:

```go
// Before: aggregate type count loses request identity.
received[messageType]++

// After: listener clones each protobuf and asserts its contract fields.
runs = append(runs, proto.Clone(req).(*iop.RunRequest))
requireRun(t, runs[0], wantRunID, "cli", "fake-cli", input, true)
```

Pending fixture의 zero-send guard는 보존하고 ready 뒤 every direct call을 one-by-one capture한다. `RunRequest`, `ProviderTunnelRequest`, both `CancelRequest` actions, `USAGE_STATUS`, `CAPABILITIES`, `SESSION_LIST`, `TRANSPORT_STATUS`, `OLLAMA_API`, and `ExecuteCommand(agent.command)` must each have exact one payload and identity.

수정 파일 및 체크리스트:

- [ ] `apps/edge/internal/service/run_dispatch_internal_test.go`: synchronized typed clone capture and exact assertions for run/request IDs, background, adapter, target, input, tunnel method/path/body, cancel action/run ID, command ID/type/payload.
- [ ] narrow existing service helper test only when a shared assertion helper needs an independently focused regression; do not alter public contracts.

테스트 작성: response-capable listeners avoid command timeout. Assert no unexpected message and exactly one captured message per expected wire; assertion failure names the wire and differing proto field.

### [REVIEW_REFACTOR-4] Two-process dev transcript를 실제 payload·terminal·command evidence로 fail-fast화

문제: `scripts/dev/edge-node-reconnect-diagnostic.sh:237-263` assumes Edge `[node0-msg]` contains `run_id`, skips all actual run comparisons when it does not, and never checks Node payload or required command response before printing PASS.

해결 방법:

```bash
# Before: empty selection silently skips validation.
EDGE_MSGS=$(grep "\\[node0-msg\\]" "$EDGE_OUT" | grep "run_id=$run_id" || true)
[ -z "$EDGE_MSGS" ] && continue

# After: start/complete-bounded traces must contain exact expected payloads.
extract_run_trace "$EDGE_OUT" "$run_id" || fail "missing edge trace"
assert_same_payload_sequence "$node_trace" "$edge_trace" || fail "payload mismatch"
assert_complete_once_after_last_message "$edge_trace" || fail "terminal ordering"
```

Use the existing two dev entrypoints and one `/tmp` mock CLI instance. The script must fail on missing/extra token, Node-vs-Edge sequence mismatch, complete not exactly once/last, missing `/nodes`, `/capabilities`, `/transport`, `/sessions`, or `/terminate-session` response, and must preserve cleanup and secret-safe output.

수정 파일 및 체크리스트:

- [ ] `scripts/dev/edge-node-reconnect-diagnostic.sh`: implement run-bound trace extraction from real log format, expected three token payload sequences, one terminal after last payload, command response checks, reconnect post-run validation, and `exit 1` on every missing assertion.
- [ ] preserve `scripts/dev/edge.sh`, `scripts/dev/node.sh`, and `scripts/e2e-smoke.sh` behavior; diagnostic stays repo-internal and never becomes bootstrap documentation.

테스트 작성: `bash -n` is mandatory. Runtime transcript command must emit the compared evidence and exit non-zero for an injected missing payload/command assertion fixture if such injection can be made without changing production defaults; otherwise the script's fail-fast branches are exercised through shell helpers and reviewer inspection.

### [REVIEW_REFACTOR-5] Formatting과 flaky race evidence를 회복

문제: review re-run found a `gofmt -d` diff at `integration_test.go:1381`, and the combined `-count=20 -race` suite timed out `TestRegistrationDoesNotDispatchBeforeReady` once. Claimed PASS output is therefore not trustworthy.

해결 방법: format every touched Go file with `gofmt -w`, remove debug logging, and replace fixed 2-second/request or sleep assumptions that make the transport test load-sensitive with deterministic readiness/barrier synchronization while keeping its semantic timeout. Run the exact repeated race command from a cold test cache policy (`-count=20`) and record full stdout/stderr; a recurrence is FAIL, not a retry-only success.

수정 파일 및 체크리스트:

- [ ] all touched `apps/edge/**/*.go`: `gofmt -w` then `gofmt -d` has no output; remove `DIAGNOSTIC` test logs.
- [ ] `apps/edge/internal/transport/integration_test.go` and narrow helper tests: eliminate the identified load-sensitive scheduling assumption without broadening production timeouts.
- [ ] review stub verification output: paste exact command output, including any blocker; do not retain prior reconstructed success summaries.

테스트 작성: no new standalone behavior test beyond deterministic barrier regressions above. Repeated race command is the regression test for the former flake.

## 의존 관계 및 구현 순서

- Directory dependency `11+08,13,15_reconnect_candidate_recovery` is the runtime source of truth; predecessor `08`, `13`, `15` completion is required before this task starts.
- REVIEW_REFACTOR-1 establishes the queue-inclusive owner gate; REVIEW_REFACTOR-2 and -3 consume the boundary for dispatch proof; REVIEW_REFACTOR-4 verifies the direct user flow; REVIEW_REFACTOR-5 validates all changed Go evidence.

## 수정 파일 요약

| 파일 | 항목 |
|---|---|
| `apps/edge/internal/node/registry.go`, `registry_test.go` | REVIEW_REFACTOR-1 |
| `apps/edge/internal/transport/connection_handlers.go`, `integration_test.go` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-5 |
| `apps/edge/internal/service/service.go`, `provider_pool.go`, `provider_tunnel.go`, `run_submit.go` | REVIEW_REFACTOR-1, REVIEW_REFACTOR-2 |
| `apps/edge/internal/service/run_dispatch_internal_test.go` and narrow focused queue tests | REVIEW_REFACTOR-2, REVIEW_REFACTOR-3, REVIEW_REFACTOR-5 |
| `apps/edge/internal/bootstrap/reconnect_readiness_integration_test.go` | REVIEW_REFACTOR-1 |
| `scripts/dev/edge-node-reconnect-diagnostic.sh` | REVIEW_REFACTOR-4 |

## 최종 검증

- workdir `/config/workspace/iop`: `gofmt -d apps/edge/internal/node/registry.go apps/edge/internal/node/registry_test.go apps/edge/internal/transport/connection_handlers.go apps/edge/internal/transport/integration_test.go apps/edge/internal/service/service.go apps/edge/internal/service/provider_pool.go apps/edge/internal/service/provider_tunnel.go apps/edge/internal/service/run_submit.go apps/edge/internal/service/run_dispatch_internal_test.go apps/edge/internal/bootstrap/reconnect_readiness_integration_test.go` — output 없음.
- workdir `/config/workspace/iop`: `git diff --check -- apps/edge/internal/node apps/edge/internal/transport apps/edge/internal/service apps/edge/internal/bootstrap scripts/dev` and `bash -n scripts/dev/edge.sh scripts/dev/node.sh scripts/dev/edge-node-reconnect-diagnostic.sh scripts/e2e-smoke.sh` — both exit 0.
- workdir `/config/workspace/iop`: `go test -count=20 -race -timeout 300s ./apps/edge/internal/node ./apps/edge/internal/transport ./apps/edge/internal/service -run 'Ready|Disconnect|Ownership|Generation|PendingClientDirectSendFence|ProviderPool|RegistrationDoesNotDispatchBeforeReady'` — all 20 repetitions pass; no timeout or race.
- workdir `/config/workspace/iop`: `go test -count=10 -race -timeout 300s ./apps/edge/internal/bootstrap -run '^TestActualNodeReconnectReadyPumpsQueuedWaiterExactlyOnce$'` and `go test -count=1 -timeout 600s ./apps/edge/... ./apps/node/...` — PASS.
- workdir `/config/workspace/iop`: `IOP_DEV_RECONNECT_BIND_TIMEOUT=60 IOP_DEV_RECONNECT_PROFILE=mock ./scripts/dev/edge-node-reconnect-diagnostic.sh` — two existing dev processes, message x2 plus reconnect run, exact Node/Edge payload sequence, terminal ordering, and five command responses all PASS.
- workdir `/config/workspace/iop`: `IOP_E2E_BIND_TIMEOUT=60 IOP_E2E_PROFILE=mock ./scripts/e2e-smoke.sh` — auxiliary smoke PASS and is recorded separately from direct transcript.
- workdir `/config/workspace/iop`: `scripts/e2e-long-context-admission-smoke.sh --preflight`; only on rc=0 run `scripts/e2e-long-context-admission-smoke.sh --scenario normal-10`. rc=3 is recorded as a live revalidation blocker, never PASS.

검증 결과를 review stub에 기록하고 review-ready를 보고한 뒤 현재 실행을 종료한다. code-review를 실행하거나 다음 에이전트를 시작하지 않는다.
