<!-- task=m-principal-provider-credential-slot-routing/10+08,09_verification_contracts plan=2 tag=REVIEW_TEST milestone-task=slot-smoke,contract-ops -->

# Repair credential-slot qualification and canonical verification evidence

## For the Implementing Agent

Implement only the listed review repairs. Preserve unrelated worktree changes and do not archive task artifacts, write `complete.log`, update the roadmap, ask the user for authorization, or execute the external provider command without authorization already supplied through the execution context. Fill every implementation-owned section in the paired `CODE_REVIEW-*-G??.md` with exact fresh output. A deterministic failure or unavailable external capability is evidence, never a substituted pass.

## Background

The prior implementation added a smoke script and partial documentation, but the script cannot complete on the current runner, its live mode is unreachable and does not contact the selected real provider, the deterministic assertions do not prove the credential-slot security boundary, and canonical contracts/specs remain inconsistent with the implemented 08/09 projection and lease path. This follow-up repairs those review findings without redesigning production Go behavior.

## Archive Evidence Snapshot

- Prior plan: `agent-task/m-principal-provider-credential-slot-routing/10+08,09_verification_contracts/plan_local_G07_1.log`.
- Prior review: `agent-task/m-principal-provider-credential-slot-routing/10+08,09_verification_contracts/code_review_cloud_G07_1.log`.
- Verdict: FAIL. Findings: 4 Required, 0 Suggested, 0 Nit. Routing signals: `review_rework_count=1`, `evidence_integrity_failure=true`.
- Required scope: repair early live-mode parsing/gating and the actual one-shot path; make the deterministic smoke executable and acceptance-complete; synchronize every planned canonical contract/spec/index/runbook; replace reconstructed evidence and the false predecessor blocker with exact fresh results.
- Affected implementation files are the credential-slot smoke and fixture plus the canonical contract/spec/runbook files listed in `Modified Files Summary`; no production Go change was authorized by this follow-up.
- Fresh reviewer evidence: `bash -n` and fixture JSON validation passed; `./scripts/e2e-credential-slot-smoke.sh --self-test-live-gate` exited 1 and `make test-credential-slot-smoke` exited 2 because the script executed binaries from a noexec `/tmp`; the exact broad package test exited 1 in the same noexec environment, while the affected package set passed with an executable external `TMPDIR`; `go vet`, pointer/leak checks, and `git diff --check` passed.
- Both predecessor gates are already satisfied by exact PASS evidence at `agent-task/archive/2026/08/m-principal-provider-credential-slot-routing/08+03,04,07_secure_delivery/complete.log` and `agent-task/archive/2026/08/m-principal-provider-credential-slot-routing/09+08_migration_attribution/complete.log`. Do not search archive siblings or require restoration into active task paths.
- Roadmap carryover: milestone `principal-provider-credential-slot-routing`, task ids `slot-smoke,contract-ops`, SDD criteria S16-S17. Do not write `complete.log` or update roadmap state from implementation.

## Roadmap Targets

- Milestone: `agent-roadmap/phase/operational-observability-provider-management/milestones/principal-provider-credential-slot-routing.md`
- SDD: `agent-roadmap/sdd/operational-observability-provider-management/principal-provider-credential-slot-routing/SDD.md`
- Task ids: `slot-smoke`, `contract-ops`
- S16 acceptance: deterministic vendor/profile/two-slot qualification plus one explicitly authorized representative live slot request with only safe evidence.
- S17 acceptance: canonical contracts/specs/indexes/runbooks match the implemented secure transport, projection, lease, migration, attribution, redaction, rotation, and revocation behavior.

## Analysis

### Files Read

- `scripts/e2e-credential-slot-smoke.sh`
- `scripts/fixtures/credential-slot-vendors.json`
- `Makefile`
- `apps/control-plane/internal/wire/edge_server.go`
- `apps/control-plane/cmd/control-plane/credential_http_handlers.go`
- `apps/edge/internal/controlplane/connector.go`
- `apps/edge/internal/bootstrap/runtime.go`
- `apps/node/internal/node/tunnel_handler.go`
- `agent-contract/index.md`
- `agent-contract/inner/client-control-plane-wire.md`
- `agent-contract/inner/control-plane-edge-wire.md`
- `agent-contract/inner/edge-node-runtime-wire.md`
- `agent-contract/inner/edge-config-runtime-refresh.md`
- `agent-contract/outer/openai-compatible-api.md`
- `agent-contract/outer/anthropic-compatible-api.md`
- `agent-spec/index.md`
- `agent-spec/control/control-plane-operations.md`
- `agent-spec/runtime/edge-node-execution.md`
- `agent-spec/runtime/provider-pool-config-refresh.md`
- `agent-spec/input/openai-compatible-surface.md`
- `docs/edge-local-dev-guide.md`
- `docs/openai-usage-grafana.md`
- `agent-test/local/rules.md`
- `agent-test/local/control-plane-smoke.md`
- `agent-test/local/edge-smoke.md`
- `agent-test/local/node-smoke.md`
- `agent-test/local/platform-common-smoke.md`
- `agent-test/local/testing-smoke.md`

### Current Behavior and Failure Mechanism

- The smoke script creates executable artifacts under hard-coded `/tmp`, which is noexec on the current runner. This prevents the fake provider and subsequent stack from starting.
- Mode parsing occurs after deterministic setup and the only live dispatch check. The documented multi-flag command is not parsed, and the live function is both unavailable at its call site and implemented with another loopback fake rather than the selected provider.
- Generated peer certificates are independent self-signed certificates rather than identities signed by the generated CA with the required role/name/SAN claims. The fixture is validated but not consumed as runtime input.
- The fake provider accepts any nonempty authorization value, does not faithfully expose both Chat and Messages operations, and the revoke check creates a route instead of proving that a later request obtains no lease and reaches no upstream.
- The current Go source actively supplies a principal projection during Edge hello and refresh, acquires leases over the authenticated CP-Edge connection, fences bindings at Edge, and consumes sealed material immediately before Node adapter execution. Several canonical documents still describe projection as inactive, the connector as plaintext, or caller provider-auth forwarding as the current path.
- Prior verification prose reported successful commands that fail when rerun and claimed predecessor logs were missing even though exact archived PASS logs exist.

### Required Security and Evidence Invariants

- Deterministic mode must be loopback-only, consume the tracked fixture, use CA-signed role/name identities, and verify the credential HTTPS, CP-Edge mTLS, and Edge-Node mTLS boundaries without writing secret material into the repository.
- The fake must compare exact expected bearer and API-key header values, count each operation and lease/upstream attempt, and support the concrete Chat and Messages operation paths selected by the fixture.
- At-rest assertions must prove provider plaintext is absent from SQLite and captured files while encrypted slot state remains usable. Attribution may expose only stable safe slot reference/revision dimensions, never raw secret, principal token, lease id, private key, prompt, or response body.
- Rotation must move to the new revision; revocation must make a subsequent full request fail closed with unchanged lease-decrypt/upstream counters and no fallback to the other same-model slot.
- Live mode must parse all documented flags before any build, stdin read, or network access; without both opt-in and `--secret-stdin` it exits immediately. With authorization, it runs exactly one no-retry request through the temporary managed IOP slot route to the selected real endpoint/model, validates structural success, discards content, and emits only the approved sanitized JSON fields.
- All verification records must be copied from fresh command stdout/stderr and exit status. Do not use prose such as “no output = pass” unless that is the literal captured output plus a separately recorded exit status.

### Dependencies and Split Judgment

- The two predecessor dependencies are satisfied by the two exact archived PASS logs cited in `Archive Evidence Snapshot`.
- The smoke fixes and documentation sync remain one packet because S16 execution evidence is the source for S17 operational/redaction claims and the same leak guard covers both.
- No production credential-plane redesign is in scope. If the corrected smoke exposes a production defect that cannot be fixed within the exact files claimed below, record the failing command and blocker in the review artifact; do not silently broaden the write set.

### Final Routing

- Evaluation mode: `isolated-reassessment`.
- Build route: `grade-boundary`, `large_indivisible_context=false`, positive risks `temporal_state`, `boundary_contract`, `structured_interpretation`, and `variant_product`; scores `2,2,2,2,2`; `PLAN-cloud-G10.md`.
- Review route: `official-review`; scores `2,2,2,2,2`; `CODE_REVIEW-cloud-G10.md`; Codex `gpt-5.6-sol`, reasoning effort `xhigh`.
- Recovery inputs: `review_rework_count=1`, `evidence_integrity_failure=true`; recovery boundary matched.

## Implementation Checklist

- [ ] Repair deterministic temp/TLS/fixture setup and prove the complete two-slot credential lifecycle with exact security assertions.
- [ ] Repair early live argument parsing, fail-before-I/O gating, one-attempt real-provider routing, and the deterministic live-gate self-test.
- [ ] Synchronize every listed canonical contract, spec, index, and operator guide with implemented 08/09 behavior.
- [ ] Run fresh deterministic, package, vet, pointer, drift, leak, and diff verification with literal output and exit codes.
- [ ] Record the single live attempt only when already authorized; otherwise record the exact external blocker and resume command without claiming S16 PASS.
- [ ] Fill all implementation-owned sections in `CODE_REVIEW-*-G??.md` and leave review-only finalization untouched.

### [REVIEW_TEST-1] Repair deterministic full-cycle qualification

**Problem:** The current smoke cannot execute from this runner's noexec `/tmp` and does not prove the security properties required by S16.

**Solution:** Make the script honor an explicit executable `TMPDIR`, preflight that a task-owned child directory can execute a freshly built probe, and fail with a precise message before stack startup if it cannot. Generate one CA and CA-sign every required server/client identity with the role, peer name, SAN, EKU, and server-name values expected by the existing TLS loaders. Drive runtime profiles and routes from the tracked fixture. Start a strict loopback fake that supports both concrete operations, compares the exact bearer/API-key headers, and exposes sanitized request/lease counters. Exercise principal bootstrap, authenticated credential HTTPS slot creation, projection, signed/sealed lease issuance, Node injection, rotation, safe attribution, and revocation.

**Modified Files and Checklist:**

- [ ] Update `scripts/e2e-credential-slot-smoke.sh` to select/preflight executable temporary storage, build fresh binaries, create CA-signed identities, configure all credential-bearing hops, and clean all task-owned artifacts on success, failure, and signal.
- [ ] Update `scripts/fixtures/credential-slot-vendors.json` so the script consumes stable secret-free Chat and Messages profiles, operation paths, auth schemes/header names, same-model slot routes, and expected safe identifiers.
- [ ] Require the fake to reject missing or incorrect auth, support both fixture operations, and expose only counters/safe metadata.
- [ ] Assert ciphertext-only persistence, correct initial and rotated revision attribution, negative plaintext/no-cert/wrong-peer cases, and a post-revoke full request that leaves lease-decrypt and upstream counts unchanged and never falls back.

**Test Strategy:** Use random loopback ports and fresh task-owned temporary material. Capture every process stdout/stderr into the temporary tree, then scan the entire tree and command streams with a sentinel and credential/private-key patterns before cleanup. Check exact status codes, operation paths, auth headers, slot refs/revisions, and before/after counters rather than log-message approximations.

**Verification:** Run syntax and fixture validation, the live-gate self-test, and `make test-credential-slot-smoke` with the executable temporary root shown in Final Verification.

### [REVIEW_TEST-2] Repair the gated representative live one-shot

**Problem:** The documented live command is not parsed before setup and its implementation calls a loopback fake instead of the selected external provider.

**Solution:** Parse `--live`, `--secret-stdin`, `--profile`, `--model`, `--reasoning-effort`, and `--max-completion-tokens` before defining or invoking any mode-specific work. Validate exactly one mode and every value. Missing `IOP_ALLOW_LIVE_PROVIDER=1` or `--secret-stdin` must exit before reading stdin, building, or opening any socket. Authorized live mode must create a temporary managed slot/route in the same local TLS stack, route one short non-stream request to the selected real profile endpoint/model, disable application and transport retries, accept only the required structural completion, discard body/prompt/secret, and print one sanitized JSON record containing only provider, model, safe slot ref, credential revision, UTC date, and result.

**Modified Files and Checklist:**

- [ ] Move argument parsing and gate validation to the script entry boundary and invoke mode functions only after all function definitions.
- [ ] Make the live path use the selected fixture/profile endpoint and managed slot route, not the deterministic fake endpoint.
- [ ] Add a deterministic self-test hook using an instrumented loopback endpoint to prove no-opt-in zero I/O, one authorized call, zero retry, exact forwarded options, structural validation, and sentinel absence from stdout/stderr/files.
- [ ] Preserve the repository qualification command for `openai`, `gpt-5.6-luna`, `reasoning_effort=high`, and `max_completion_tokens=128`; do not substitute another provider/model or retry.

**Test Strategy:** The gate test supplies unreadable/sentinel stdin and a call counter; the no-opt-in branch must leave the counter at zero and the sentinel unread. The authorized self-test must record exactly one call. The real command is executed once only when authorization is already present after every deterministic check passes.

**Verification:** Run `--self-test-live-gate` deterministically. Record sanitized literal stdout, stderr, and exit status from the exact real pipeline, or the exact current authorization/credential/provider blocker and unmodified resume command.

### [REVIEW_TEST-3] Synchronize canonical contracts, specs, indexes, and runbooks

**Problem:** Canonical documentation contradicts active source and omits several planned 08/09 behavior updates.

**Solution:** Update the existing canonical documents in place. Describe server-authenticated credential HTTPS with principal bearer auth and host-local bootstrap; CP-Edge mTLS identity plus hello/refresh projection and authenticated lease acquisition; Edge binding/revision/generation fences; Edge-Node mTLS and Node-targeted sealed lease consumption; managed-versus-legacy provider auth selection; public OpenAI/Anthropic schema compatibility; safe slot/revision attribution; and rotate/revoke/expiry/redaction operator procedures. Preserve existing canonical IDs and link rows, and document only behavior supported by current source/tests.

**Modified Files and Checklist:**

- [ ] Update `agent-contract/inner/client-control-plane-wire.md`, `agent-contract/inner/control-plane-edge-wire.md`, `agent-contract/inner/edge-node-runtime-wire.md`, and `agent-contract/inner/edge-config-runtime-refresh.md`.
- [ ] Update `agent-contract/outer/openai-compatible-api.md` and `agent-contract/outer/anthropic-compatible-api.md`.
- [ ] Update `agent-contract/index.md` in place without duplicate canonical rows.
- [ ] Update `agent-spec/control/control-plane-operations.md`, `agent-spec/runtime/edge-node-execution.md`, `agent-spec/runtime/provider-pool-config-refresh.md`, `agent-spec/input/openai-compatible-surface.md`, and `agent-spec/index.md`.
- [ ] Update `docs/edge-local-dev-guide.md` and `docs/openai-usage-grafana.md` with external key/cert mounts, TLS startup, safe slot lifecycle, rotate/revoke/lease-expiry confirmation, attribution interpretation, and incident redaction checks.

**Test Strategy:** Check every index pointer, compare contract statements with the active projection/lease/TLS symbols cited in Analysis, reject stale “inactive/plain/caller-auth forwarding” claims for managed mode, and run a scoped leak scan excluding encrypted SOPS input and archive history.

**Verification:** Run the pointer, source-symbol, stale-claim, leak, and diff commands in Final Verification.

### [REVIEW_TEST-4] Replace untrustworthy verification evidence

**Problem:** The prior review artifact contains reconstructed successful results that conflict with fresh execution and cites a false dependency blocker.

**Solution:** Run all deterministic checks from the current worktree using a verified executable external temporary root. Copy literal stdout/stderr and numeric exit status into the new review artifact. Cite the two exact archived predecessor PASS paths from this plan. Do not infer success from expected silence, reuse prior output, mark an unchecked item complete, or report the live one-shot as run when it was not.

**Modified Files and Checklist:**

- [ ] Fill `agent-task/m-principal-provider-credential-slot-routing/10+08,09_verification_contracts/CODE_REVIEW-cloud-G10.md` with actual implementation notes, deviations, exact commands, literal sanitized stdout/stderr, and exit statuses.
- [ ] Record separate deterministic smoke, package/vet, contract/spec pointer-drift-leak, and live-attempt sections.
- [ ] Keep every Implementation Item Completion and Implementation Checklist box aligned with actual evidence; leave Review-Only Checklist unchanged.

**Test Strategy:** A reviewer must be able to rerun every deterministic command verbatim and reconcile the recorded exit code. If the external one-shot is not authorized or unavailable, only that item remains incomplete with a precise blocker; deterministic results remain independently reviewable.

**Verification:** Compare recorded evidence to the exact Final Verification commands and current files before handoff.

## Dependencies and Execution Order

1. Treat the two archived predecessor PASS logs cited above as satisfied dependencies; do not restore or relocate them.
2. Complete REVIEW_TEST-1 and the deterministic portion of REVIEW_TEST-2 before documentation claims or evidence capture.
3. Complete REVIEW_TEST-3 from the corrected executable behavior and current Go sources.
4. Run all deterministic commands in REVIEW_TEST-4. Run the real provider pipeline at most once and only when authorization is already present.
5. Fill the paired review artifact and stop for independent code review.

## Modified Files Summary

| File | Items |
|------|-------|
| `scripts/e2e-credential-slot-smoke.sh` | REVIEW_TEST-1, REVIEW_TEST-2 |
| `scripts/fixtures/credential-slot-vendors.json` | REVIEW_TEST-1 |
| `agent-contract/index.md` | REVIEW_TEST-3 |
| `agent-contract/inner/client-control-plane-wire.md` | REVIEW_TEST-3 |
| `agent-contract/inner/control-plane-edge-wire.md` | REVIEW_TEST-3 |
| `agent-contract/inner/edge-node-runtime-wire.md` | REVIEW_TEST-3 |
| `agent-contract/inner/edge-config-runtime-refresh.md` | REVIEW_TEST-3 |
| `agent-contract/outer/openai-compatible-api.md` | REVIEW_TEST-3 |
| `agent-contract/outer/anthropic-compatible-api.md` | REVIEW_TEST-3 |
| `agent-spec/index.md` | REVIEW_TEST-3 |
| `agent-spec/control/control-plane-operations.md` | REVIEW_TEST-3 |
| `agent-spec/runtime/edge-node-execution.md` | REVIEW_TEST-3 |
| `agent-spec/runtime/provider-pool-config-refresh.md` | REVIEW_TEST-3 |
| `agent-spec/input/openai-compatible-surface.md` | REVIEW_TEST-3 |
| `docs/edge-local-dev-guide.md` | REVIEW_TEST-3 |
| `docs/openai-usage-grafana.md` | REVIEW_TEST-3 |
| `agent-task/m-principal-provider-credential-slot-routing/10+08,09_verification_contracts/CODE_REVIEW-cloud-G10.md` | REVIEW_TEST-4 |

## Final Verification

Run deterministic commands from `/config/workspace/iop-s0`. `/config/workspace` is the current runner's verified executable temporary parent; on a different runner, set `credential_smoke_parent` to a task-owned, writable, executable external temporary parent before running the same commands.

```bash
credential_smoke_parent="$(mktemp -d /config/workspace/iop-credential-slot-review.XXXXXX)"
trap 'rmdir "$credential_smoke_parent" 2>/dev/null || true' EXIT
command -v go && go version && go env GOROOT
command -v bash && command -v jq && command -v openssl && openssl version
command -v sops && command -v curl && command -v netstat
git status --short --branch
bash -n scripts/e2e-credential-slot-smoke.sh
jq -e '.profiles | length >= 4 and any(.[]; .id == "seulgi_chat") and any(.[]; .id == "seulgi_messages")' scripts/fixtures/credential-slot-vendors.json >/dev/null
TMPDIR="$credential_smoke_parent" ./scripts/e2e-credential-slot-smoke.sh --self-test-live-gate
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
TMPDIR="$credential_smoke_parent" go test -count=1 ./apps/control-plane/... ./apps/edge/... ./apps/node/... ./packages/go/...
TMPDIR="$credential_smoke_parent" go vet ./apps/control-plane/... ./apps/edge/... ./apps/node/... ./packages/go/...
git diff --check
python3 - <<'PY'
import re
from pathlib import Path
for index in (Path('agent-contract/index.md'), Path('agent-spec/index.md')):
    data = index.read_text()
    targets = set(re.findall(r'`((?:agent-contract|agent-spec)/[^`\s]+\.md)`', data))
    targets.update(re.findall(r'\(((?:agent-contract|agent-spec)/[^)\s]+\.md)\)', data))
    for target in sorted(targets):
        if not Path(target.split('#', 1)[0]).is_file():
            raise SystemExit(f'broken pointer: {index}: {target}')
PY
rg -n 'SetCredentialPlane|PrincipalProjectionApplyRequest|AcquireCredentialLease|SetCredentialLeaseProvider|CredentialLease' apps/control-plane apps/edge apps/node
if rg -n 'projection.*inactive|plain connector|caller.*provider.*auth.*current' agent-contract agent-spec docs; then exit 1; fi
if rg --sort path -n --hidden --glob '!agent-task/archive/**' --glob '!*.sops.yaml' --glob '!go.sum' --glob '!PLAN-*.md' --glob '!*.log' 'BEGIN (AGE|OPENSSH|RSA|EC) PRIVATE KEY|sk-[A-Za-z0-9_-]{20,}' agent-contract agent-spec docs scripts agent-task/m-principal-provider-credential-slot-routing/10+08,09_verification_contracts; then exit 1; fi
rmdir "$credential_smoke_parent"
trap - EXIT
```

Expected: every deterministic command exits 0; both guarded `rg` commands produce no matching secret/stale managed-mode claim; the source-symbol search shows the active projection/lease path; the script records exact Chat and Messages auth, ciphertext, revision, attribution, TLS-negative, and post-revoke counter assertions without leaking sensitive values.

Run the following pipeline once only after explicit authorization and only after all deterministic commands pass:

```bash
credential_smoke_parent="$(mktemp -d /config/workspace/iop-credential-slot-live.XXXXXX)"
trap 'rmdir "$credential_smoke_parent" 2>/dev/null || true' EXIT
SOPS_AGE_KEY_FILE=/config/.config/sops/age/keys.txt \
  sops --decrypt --input-type yaml --output-type binary /config/.config/sops/iop/openai.sops.yaml \
  | IOP_ALLOW_LIVE_PROVIDER=1 TMPDIR="$credential_smoke_parent" ./scripts/e2e-credential-slot-smoke.sh \
      --live --secret-stdin --profile openai --model gpt-5.6-luna \
      --reasoning-effort high --max-completion-tokens 128
rmdir "$credential_smoke_parent"
trap - EXIT
```

Expected live evidence: exactly one sanitized JSON object with only provider, model, safe credential slot reference, credential revision, UTC date, and result; sanitized stderr; exit 0; exactly one upstream request and no retry. If authorization, credential readability, or provider/model availability is absent, record the exact blocker and this unchanged resume command; do not substitute another request or claim `slot-smoke` PASS.

After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
