diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/CODE_REVIEW-cloud-G09.md b/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/CODE_REVIEW-cloud-G09.md new file mode 100644 index 00000000..5f7318df --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/CODE_REVIEW-cloud-G09.md @@ -0,0 +1,166 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-10 +task=m-agent-comparison-benchmark-pipeline/06_connectivity_contract, plan=0, tag=API + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files and verify that output in `Verification Results` matches code. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Freeze route and effort preflight contracts | [ ] | +| API-2 Classify and persist secret-safe connection evidence | [ ] | +| API-3 Preserve Claude Code high-tier effort through IOP | [ ] | + +## Implementation Checklist + +- [ ] Add frozen caller preflight, effective binding, issue, and result contracts with exact no-substitution validation. +- [ ] Add the closed registration-versus-implementation gap classifier and secret-safe, canonical, no-overwrite preflight evidence writer. +- [ ] Accept and preserve only the closed `low|medium|high|xhigh|max` Anthropic effort set across native and Chat-bridge routes without substitution. +- [ ] Update the Anthropic outer contract and current implementation spec, add deterministic Python/Go coverage, and run focused, aggregate, scoped Edge, complete-Go-or-blocker, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G09_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm every status/issue/binding field is a closed immutable contract and exact mismatches fail closed. +- Confirm persisted bytes and returned errors exclude raw secrets, private endpoints, prompt/tool content and unbounded caller output. +- Confirm registration-required never becomes ready and implementation gaps are not routed as registration. +- Confirm `xhigh|max` are preserved exactly across their selected native/bridge route, never aliased or downgraded, while unknown effort values fail before provider wire. +- Confirm contract/spec match the implementation and only files claimed by the plan changed. +- Confirm deterministic local verification passes; do not treat this predecessor packet as downstream live preflight or Milestone Task completion evidence. + +## Verification Results + +Paste actual stdout/stderr and exit code for every command. If blocked, include the exact resume condition; do not summarize output. + +### V1 Focused connectivity tests + +Command: `python3 -m unittest scripts.agent_benchmark.connectivity_test -v` + +```text + +``` + +Exit code: `` + +### V2 Aggregate benchmark tests + +Command: `make test-agent-comparison-benchmark` + +```text + +``` + +Exit code: `` + +### V3 Focused Anthropic compatibility tests + +Command: `go test ./apps/edge/internal/openai -run 'TestAnthropic(ChatBridge|Native)' -count=1` + +```text + +``` + +Exit code: `` + +### V4 Scoped Edge regression + +Command: `go test ./apps/edge/... -count=1` + +```text + +``` + +Exit code: `` + +### V5 Complete Go regression or exact local blocker + +Command: `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + +```text + +``` + +Exit code: `` + +### V6 Patch integrity + +Command: `git diff --check` + +```text + +``` + +Exit code: `` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/PLAN-cloud-G09.md b/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/PLAN-cloud-G09.md new file mode 100644 index 00000000..9cff1d76 --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/PLAN-cloud-G09.md @@ -0,0 +1,176 @@ + + +# Plan - API: IOP connectivity contract and high-tier effort compatibility + +## For the Implementing Agent + +이 계획의 공통 preflight 계약과 IOP Anthropic 호환 입력의 high-tier effort 보존만 구현한다. 모든 검증을 실행하고 실제 출력과 구현 메모를 active `CODE_REVIEW-cloud-G09.md`의 구현 소유 섹션에 채운 뒤 파일을 그대로 두고 review-ready로 보고한다. 막히면 정확한 blocker, 실행한 명령/출력, 재개 조건만 evidence 필드에 기록한다. 사용자 질문, user-input 도구, control-plane stop 파일, 다음 상태 분류, 로그 archive, `complete.log` 작성은 하지 않는다. + +## Background + +manifest는 requested route/model/effort와 expected binding을 보존하지만 caller 공통 preflight 결과와 failure taxonomy가 없다. 또한 Claude Code가 요구하는 `xhigh`와 `max`를 IOP Anthropic Chat bridge의 strict decoder가 현재 거부하므로, downstream live preflight만으로는 목표 matrix를 검증할 수 없다. 세 caller를 연결하기 전에 no-substitution binding 검증, 등록 필요와 구현 gap 구분, secret-safe durable evidence, high-tier effort의 무손실 전달을 하나의 선행 계약으로 고정해야 한다. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/plan/templates/review-stub-template.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-ops/skills/common/update-test/SKILL.md` +- `agent-ops/skills/common/code-review/SKILL.md` +- `agent-ops/skills/common/sync-milestone-workstate/SKILL.md` +- `agent-roadmap/current.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` +- `agent-spec/index.md` +- `agent-spec/input/openai-compatible-surface.md` +- `agent-contract/index.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `scripts/agent_benchmark/manifest.py` +- `apps/edge/internal/openai/anthropic_types.go` +- `apps/edge/internal/openai/anthropic_bridge.go` +- `apps/edge/internal/openai/anthropic_bridge_test.go` +- `apps/edge/internal/openai/anthropic_native_test.go` +- `apps/edge/internal/openai/chat_policy.go` +- `apps/edge/internal/openai/chat_types.go` +- `apps/edge/internal/openai/chat_handler.go` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-test/local/edge-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `agent-ops/rules/project/domain/edge/rules.md` + +### SDD Criteria + +- SDD 상태는 `[승인됨]`, 잠금은 `해제`다. +- milestone-task는 `effort-route,connection-gap`, Acceptance는 `S09`, `S10`이다. +- S09는 requested/effective route/model/effort matrix와 no-substitution evidence를 요구하고, S10은 blocker classifier와 follow-up routing test를 요구한다. +- SDD D07은 endpoint/auth/protocol/stream gap을 registration blocker와 분리된 구현 후보로 남기라고 명시한다. 이미 확인 가능한 effort protocol gap을 외부 blocker로 미루지 않고 이 packet에서 닫는다. + +### Verification Context + +- 별도 `verification_context` handoff는 없었다. local/edge test rules, Make target, 현재 source/test와 안전한 read-only probe로 보완했다. +- 현재 runner는 `/config/workspace/iop-s0`, branch `feature/agent-comparison-benchmark-pipeline`, Linux aarch64, Python 3.12.3 환경이다. +- 설치 CLI help 기준 Claude Code 2.1.223은 `--effort low|medium|high|xhigh|max`, agy 1.1.11은 `--effort low|medium|high`, codex-cli 0.147.0은 Responses provider용 `exec --json --ephemeral --ignore-user-config --strict-config` 경로를 제공한다. +- read-only inventory에서는 dev `gemini-3.6-flash`만 exact active match이고 `claude-sonnet-5`, `gpt-5.6-luna`는 미등록이다. 이는 classifier 입력 근거이며 ready 성공으로 간주하지 않는다. +- 현재 `go test ./... -count=1`은 ignored `build/r14-remote-anthropic_handler.go`가 `iop/build`로 수집되어 internal import 오류로 차단된다. 이 사용자 산출물을 수정하거나 삭제하지 않고 존재 여부와 실제 실패를 evidence에 남긴다. +- 이 packet은 provider 호출을 하지 않는다. 실제 CLI/IOP direct preflight와 full-cycle evidence는 downstream `10+07,08,09_connectivity_runner`가 소유하며, 이 packet의 PASS만으로 Milestone Task 완료를 주장하지 않는다. + +### Test Coverage Gaps + +- 기존 benchmark 테스트는 missing adapter만 `CapabilityUnavailable`로 묶고 requested/effective binding 불일치, registration 필요, endpoint/auth/protocol/stream 구현 gap을 구분하지 않는다. +- `preflight/preflight-NNNNNN.json`의 closed schema, canonical ordering, sequence/no-overwrite, secret/private endpoint 비노출 검증이 없다. +- 기존 Anthropic bridge 테스트는 `output_config.effort=high`만 검증한다. `xhigh|max`의 strict 수용, exact bridge mapping, native raw preservation, unknown 값의 fail-closed 경계를 증명하지 않는다. + +### Symbol References + +rename/remove하는 symbol은 없다. 새 계약은 기존 `MatrixCell`/`IopCell`을 import하고 후속 adapter/runner가 소비한다. `output_config.effort`는 `decodeAnthropicMessageRequest`에서 검증되어 `prepareAnthropicChatBridge`가 동일 문자열을 `reasoning_effort`로 매핑하며, native Messages 경로는 envelope/model rewrite 외 request bytes를 보존한다. + +### Split Judgment + +공통 계약과 이미 알려진 effort ingress gap은 같은 preflight invariant다. contract가 `xhigh|max`를 valid 요청으로 분류하면서 IOP가 이를 wire 전에 거부하면 독립 PASS가 성립하지 않으므로 하나의 선행 packet으로 유지한다. caller별 process adapter와 live provider 실행은 별도 안정 경계를 가지므로 기존 `06 → 07/08/09 → 10` dependency graph는 변경하지 않는다. + +### Scope Rationale + +새 caller 명령/JSONL parser, CLI wiring, skill 문서, live provider 호출은 제외한다. `xhigh|max`를 임의 값 전체로 일반화하거나 alias/downshift하지 않으며, OpenAI public Chat/Responses 입력 계약과 provider capability 정책도 변경하지 않는다. 수정 범위는 benchmark 공통 preflight 모듈과 Anthropic 호환 surface의 strict bridge 경계, 직접 관련 test/contract/spec로 제한한다. + +### Final Routing + +- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh`, mode `pair`. +- build closures: scope/context/verification/evidence/ownership/decision 모두 true. scores `2/1/2/2/2`; base/route `grade-boundary`; lane `cloud`; grade `G09`; filename `PLAN-cloud-G09.md`; catalog `worker/cloud/G09`. +- review closures: 모두 true. scores `2/1/2/2/2`; route `official-review`; lane `cloud`; grade `G09`; filename `CODE_REVIEW-cloud-G09.md`; catalog `review/cloud/G09`. +- large_indivisible_context: `false`; positive risks: `boundary_contract`, `structured_interpretation`, `temporal_state`, `variant_product`; count `4`; review_rework_count `0`; evidence_integrity_failure `false`; capability gap 없음. + +## Implementation Checklist + +- [ ] Add frozen caller preflight, effective binding, issue, and result contracts with exact no-substitution validation. +- [ ] Add the closed registration-versus-implementation gap classifier and secret-safe, canonical, no-overwrite preflight evidence writer. +- [ ] Accept and preserve only the closed `low|medium|high|xhigh|max` Anthropic effort set across native and Chat-bridge routes without substitution. +- [ ] Update the Anthropic outer contract and current implementation spec, add deterministic Python/Go coverage, and run focused, aggregate, scoped Edge, complete-Go-or-blocker, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Freeze route and effort preflight contracts + +**Problem:** `scripts/agent_benchmark/manifest.py` stores requested values and expected bindings, but there is no typed oracle proving that a caller observed the exact route/model/effort without alias substitution. + +**Solution:** Add a standard-library-only `connectivity.py` with frozen dataclasses/protocols for caller capability, requested/effective binding, issue and per-cell result. Validate caller/cell identity, route kind/id, requested/effective model and effort, complete stage bindings, canonical order, and closed status values. Any unsupported effort, alias substitution, missing effective field, or extra field fails closed. + +**Modified Files and Checklist:** + +- [ ] Create `scripts/agent_benchmark/connectivity.py` with frozen closed contracts and exact validators. +- [ ] Create `scripts/agent_benchmark/connectivity_test.py` with direct/preset, effort, alias, missing/extra field, order and immutability cases. + +**Test Strategy:** Use synthetic `MatrixCell` values only. Cover valid direct/preset results and every field-level no-substitution failure without process or network calls. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.connectivity_test -v` exits 0. + +### [API-2] Classify and persist secret-safe connection evidence + +**Problem:** SDD S10 requires credential/model absence to route to registration and endpoint/auth/protocol/stream incompatibility to route to implementation work, but no closed classifier or durable evidence format exists. + +**Solution:** Map only closed issue codes to `ready`, `registration_required`, or `implementation_gap`; never downgrade a blocker or synthesize effective values. Serialize canonical sorted JSON containing public caller/cell/route/model/effort and hashed endpoint/config identity only. Reject raw credentials, authorization values, private endpoint strings, prompt/tool content, symlinks, existing targets, and schema drift. + +**Modified Files and Checklist:** + +- [ ] Implement the closed classifier and canonical evidence serializer in `scripts/agent_benchmark/connectivity.py`. +- [ ] Test precedence, redaction sentinels, canonical bytes, containment, symlink and no-overwrite behavior in `scripts/agent_benchmark/connectivity_test.py`. + +**Test Strategy:** Use temporary directories and secret/endpoint sentinels. Assert sentinels never occur in returned errors or persisted bytes and corrupted/duplicate evidence fails closed. + +**Verification:** focused Python tests and the aggregate benchmark target exit 0. + +### [API-3] Preserve Claude Code high-tier effort through IOP + +**Problem:** `decodeAnthropicMessageRequest` rejects `output_config.effort=xhigh|max` before a Chat-profile candidate reaches the provider wire, even though the target benchmark fixture requires Claude→GPT `xhigh` and Claude→Sonnet `max`. Treating this as only an external registration blocker would hide a repository-fixable protocol gap. + +**Solution:** Extend only the closed Anthropic compatibility effort enum to `low|medium|high|xhigh|max`. Preserve the exact token: native Messages keeps request bytes except the model rewrite, and the Chat bridge maps `output_config.effort` unchanged to outbound `reasoning_effort`. Do not lowercase, alias, cap, or silently downgrade it. Unknown values remain a pre-wire `invalid_request_error`; an upstream capability rejection remains an `implementation_gap` evidence outcome. + +**Modified Files and Checklist:** + +- [ ] Update `apps/edge/internal/openai/anthropic_types.go` validation and error text for the closed high-tier set. +- [ ] Add exact `xhigh|max` bridge mapping and invalid-value tests in `apps/edge/internal/openai/anthropic_bridge_test.go`. +- [ ] Add `max` request-byte preservation evidence to `apps/edge/internal/openai/anthropic_native_test.go`. +- [ ] Update `agent-contract/outer/anthropic-compatible-api.md` with native/bridge effort semantics and no-substitution behavior. +- [ ] Update `agent-spec/input/openai-compatible-surface.md` so the current implementation spec matches the expanded compatibility surface. + +**Test Strategy:** Drive the HTTP surface with fake provider tunnels. Assert `xhigh` and `max` reach the selected bridge body exactly, native bytes retain `max`, and an unknown token returns 400 with zero provider requests. + +**Verification:** focused and package-level Edge tests exit 0; downstream live preflight remains the separate full-cycle gate. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_benchmark/connectivity.py` | API-1, API-2 | +| `scripts/agent_benchmark/connectivity_test.py` | API-1, API-2 | +| `apps/edge/internal/openai/anthropic_types.go` | API-3 | +| `apps/edge/internal/openai/anthropic_bridge_test.go` | API-3 | +| `apps/edge/internal/openai/anthropic_native_test.go` | API-3 | +| `agent-contract/outer/anthropic-compatible-api.md` | API-3 | +| `agent-spec/input/openai-compatible-surface.md` | API-3 | +| `agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/CODE_REVIEW-cloud-G09.md` | API-1, API-2, API-3 evidence | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.connectivity_test -v` + - Expected: all contract, classifier, canonical evidence and secret-sentinel cases pass without network access. +2. `make test-agent-comparison-benchmark` + - Expected: aggregate benchmark tests and tracked example manifest validation pass. +3. `go test ./apps/edge/internal/openai -run 'TestAnthropic(ChatBridge|Native)' -count=1` + - Expected: high-tier effort, native preservation, bridge, stream and error-path coverage pass fresh. +4. `go test ./apps/edge/... -count=1` + - Expected: scoped Edge regression passes fresh. +5. `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + - Expected: on a clean verification checkout the complete Go suite passes. If the ignored artifact remains, record this exact blocker and do not delete or modify it. +6. `git diff --check` + - Expected: no whitespace errors. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/CODE_REVIEW-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/CODE_REVIEW-cloud-G07.md new file mode 100644 index 00000000..8a3a758b --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/CODE_REVIEW-cloud-G07.md @@ -0,0 +1,149 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-10 +task=m-agent-comparison-benchmark-pipeline/07+06_claude_iop, plan=0, tag=API + + + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files and verify that output in `Verification Results` matches code. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Build a bounded IOP-only Claude invocation | [ ] | +| API-2 Parse terminal evidence and redact Claude output | [ ] | + +## Implementation Checklist + +- [ ] Implement a Claude Code invocation builder that uses a fresh non-persistent session, one stdin task, IOP-only runtime inputs, and exact model/effort without fallback. +- [ ] Implement fail-closed Claude stream-json finish/idle and preflight binding parsing with structural redaction before durable capture. +- [ ] Add a secret-safe Claude JSONL fixture and normal/boundary adapter tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G07_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G07_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm Claude cannot inherit or select a non-IOP provider/fallback model and never resumes prior session state. +- Confirm exact model/effort/preset inputs are preserved and absent effective values never become success. +- Confirm finish and idle derive from distinct ordered protocol evidence and malformed terminal events fail closed. +- Confirm every durable capture/error excludes secret, private endpoint, prompt and tool sentinels. + +## Verification Results + +Paste actual stdout/stderr and exit code for every command; blockers require an exact resume condition. + +### V1 Predecessor + +Command: `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` + +```text + +``` +Exit code: `` + +### V2 Focused Claude adapter tests + +Command: `python3 -m unittest scripts.agent_benchmark.claude_iop_test -v` + +```text + +``` +Exit code: `` + +### V3 Aggregate benchmark tests + +Command: `make test-agent-comparison-benchmark` + +```text + +``` +Exit code: `` + +### V4 Complete Go regression or blocker + +Command: `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + +```text + +``` +Exit code: `` + +### V5 Patch integrity + +Command: `git diff --check` + +```text + +``` +Exit code: `` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/PLAN-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/PLAN-cloud-G07.md new file mode 100644 index 00000000..7089813e --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/PLAN-cloud-G07.md @@ -0,0 +1,188 @@ + + +# Plan - API: Claude Code IOP adapter + +## For the Implementing Agent + +`06_connectivity_contract`의 완료 계약 위에서 Claude Code adapter와 전용 fixture/test만 구현한다. 검증 출력과 구현 메모를 active `CODE_REVIEW-cloud-G07.md`에 채우고 review-ready로 보고한다. 막힘은 exact command/output/resume condition만 evidence 필드에 기록한다. 사용자 질문, control-plane stop 파일, next-state 분류, archive, `complete.log`는 구현자가 만들지 않는다. + +## Background + +generic lifecycle은 caller별 argv, environment, JSONL parser와 redactor를 의도적으로 모른다. Claude Code가 Anthropic-compatible IOP endpoint를 사용하면서 direct/preset alias와 effort를 임의 치환하지 않고 finish→idle을 증명하도록 독립 adapter가 필요하다. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/common/rules.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/update-test/SKILL.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-ops/skills/common/plan/templates/review-stub-template.md` +- `agent-roadmap/current.md` +- `agent-roadmap/priority-queue.md` +- `agent-spec/index.md` +- `agent-contract/index.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `agent-spec/input/openai-compatible-surface.md` +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/workspace.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/workspace_test.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `agent-test/inventory.yaml` +- `agent-test/inventory-agent.yaml` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` + +### SDD Criteria + +- SDD는 `[승인됨]`, 잠금 `해제`; milestone-task `claude-iop`; Acceptance `S06`; Evidence Map S06. +- S06은 Sonnet/Gemini/GPT direct auth/model/stream/terminal과 arbitrary preset adapter 계약을 요구하고, evidence row는 redacted Claude Code→IOP preflight를 요구한다. +- 이 packet은 adapter 계약과 credential-free stream fixture를 완성한다. 실제 dev preflight와 route readiness evidence는 세 adapter를 연결하는 ordered consumer `10+07,08,09_connectivity_runner`가 수행한다. + +### Verification Context + +- 별도 verification handoff는 없고 local rules, current CLI help/version, source/tests/contracts로 보완했다. +- runner: `/config/workspace/iop-s0`; branch `feature/agent-comparison-benchmark-pipeline`; HEAD `df0ecdbf1d6cbf8878c84a9b154bc4d022ab035d`; Linux aarch64; Python 3.12.3. 요청 시작 HEAD 이후 두 커밋은 중앙 관리 Agent-Ops 파일만 바꿨고 대상 benchmark source/test는 동일하다. tracked worktree는 materialization 전에 clean이었으며 현재 활성 변경은 이 Epic의 plan/review pair뿐이다. +- Claude binary: `/config/.local/bin/claude`, version `2.1.223`. `claude --help`에서 `--bare`, `--print`, `--input-format`, `--output-format=stream-json`, `--model`, `--effort`, `--no-session-persistence`, `--tools`를 확인했다. +- caller config/credential은 repo 밖 사용자 관리 상태이며 원문을 읽지 않았다. adapter는 base URL 값과 secret env-var 이름을 런타임 입력으로만 받고 evidence에는 digest/존재 여부만 남겨야 한다. 로컬 포트는 필요 없고 external host는 redacted dev IOP Edge다. +- dev inventory는 Gemini direct exact match를 보였지만 Sonnet 5와 GPT-5.6 luna는 미등록이다. network/provider call은 NOT_RUN; 현재 성공으로 간주하지 않는다. +- 기준선 benchmark 215 tests는 통과한다. 전체 Go 검증은 ignored `build/r14-remote-anthropic_handler.go` 때문에 차단되며 파일은 보존한다. Confidence: high for adapter contract, medium for live readiness. + +### Test Coverage Gaps + +- Claude stream-json의 finish/idle/error/usage parsing과 malformed/duplicate/out-of-order fail-closed coverage가 없다. +- argv가 fresh non-persistent session, one stdin submission, exact model/effort, disabled tools를 강제하는 테스트가 없다. +- raw prompt/tool text, base URL, bearer/API key sentinel이 capture/evidence에서 제거되는 테스트가 없다. + +### Symbol References + +rename/remove symbol은 없다. 새 adapter는 `InvocationSpec`, `run_invocation`, `MatrixCell`, `PreparedWorkspace`, 06의 connectivity contract를 소비한다. + +### Split Judgment + +Claude adapter는 agy/Codex와 파일을 공유하지 않고 fixture-driven PASS가 가능하다. `07+06`은 predecessor `06_connectivity_contract`만 요구한다. 현재 `agent-task/m-agent-comparison-benchmark-pipeline/06_connectivity_contract/complete.log`는 아직 없으며 runtime은 해당 active 또는 archived exact `complete.log`가 생기기 전 구현을 시작하면 안 된다. + +### Scope Rationale + +CLI registry, public preflight/run/resume, agy/Codex, report/timing/web/scoring, IOP/Edge 제품 코드, live hybrid preset은 제외한다. raw secret/private endpoint/prompt/tool 내용은 tracked fixture나 evidence에 넣지 않는다. + +### Final Routing + +- evaluation_mode `first-pass`; finalizer `finalize-task-policy.sh pair`. +- build closures 모두 true; scores `2/1/1/1/2`; base `local-fit`, route `risk-boundary`; lane `cloud`; grade `G07`; filename `PLAN-cloud-G07.md`; catalog `worker/cloud/G07`. +- review closures 모두 true; scores `2/1/1/1/2`; route `official-review`; lane `cloud`; grade `G07`; filename `CODE_REVIEW-cloud-G07.md`; catalog `review/cloud/G07`. +- large_indivisible_context `false`; risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product`; count `4`; review_rework_count `0`; evidence_integrity_failure `false`; capability gap 없음. + +## Implementation Checklist + +- [ ] Implement a Claude Code invocation builder that uses a fresh non-persistent session, one stdin task, IOP-only runtime inputs, and exact model/effort without fallback. +- [ ] Implement fail-closed Claude stream-json finish/idle and preflight binding parsing with structural redaction before durable capture. +- [ ] Add a secret-safe Claude JSONL fixture and normal/boundary adapter tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Build a bounded IOP-only Claude invocation + +**Problem:** `scripts/agent_benchmark/lifecycle.py:1766-1798` accepts an injected spec/parser but no production code converts a Claude cell and prepared workspace into a safe invocation. Reusing ambient Claude settings could bypass IOP or reuse session state. + +**Solution:** Add `claude_iop.py` implementing the 06 adapter protocol. Resolve the exact binary, require runtime base/secret inputs without serializing them, translate them only to the Anthropic-compatible child environment, and build a tuple equivalent to the form below. Use the prepared clean workspace as cwd, stdin exactly once, no session persistence, no tools, no fallback model and exact `request_model`/`requested_effort`. + +Before (`scripts/agent_benchmark/lifecycle.py:1781-1786`): + +```python +spec: Frozen invocation specification. +parse_event: Adapter parser mapping (stream, line) to terminal/metric events. +redact: Optional adapter redactor for exact secret values. +``` + +After (adapter-owned shape): + +```python +("claude", "--bare", "--print", "--verbose", + "--input-format", "text", "--output-format", "stream-json", + "--model", cell.iop.request_model, "--effort", cell.iop.requested_effort, + "--no-session-persistence", "--permission-mode", "dontAsk", "--tools=") +``` + +**Modified Files and Checklist:** + +- [ ] Create `scripts/agent_benchmark/claude_iop.py` with config validation, argv/env construction, preflight and invocation methods. +- [ ] Create `scripts/agent_benchmark/claude_iop_test.py` covering missing binary/config, exact argv/env allowlist, fresh session, direct/preset data and no fallback. + +**Test Strategy:** Use a temporary fake `claude` executable; never invoke network/provider. Assert stdin bytes are submitted once, ambient provider variables are not inherited, and model/effort are exact. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.claude_iop_test -v` passes. + +### [API-2] Parse terminal evidence and redact Claude output + +**Problem:** lifecycle accepts only `finish`, `idle`, or bounded metric events (`lifecycle.py:1282-1312`) and persists redacted lines. Claude stream-json can contain message/tool/prompt content, so substring redaction or optimistic result parsing would leak content or synthesize success. + +**Solution:** Parse each line as an exact JSON object with closed recognized event shapes. Emit finish only from an assistant terminal stop and idle only from the matching final result; validate session/model/effort/binding evidence without filling absent fields. Structurally replace content/tool arguments plus exact task/base/secret sentinels before lifecycle publication. Unknown informational events may be ignored; malformed claimed terminal/binding events fail closed. + +Before (`scripts/agent_benchmark/lifecycle.py:1283-1291`): + +```python +redacted = self._redact(line) +capture.record(redacted) +parsed = self.parse_event(stream, line) +``` + +After (adapter contract): + +```python +parse_claude_event(stream, raw_line) -> None | "finish" | "idle" | "metric:..." +redact_claude_event(raw_line, sensitive_values) -> canonical_redacted_json +``` + +**Modified Files and Checklist:** + +- [ ] Add strict event/binding parsing and structural redaction in `scripts/agent_benchmark/claude_iop.py`. +- [ ] Add `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl` with synthetic public metadata and redacted content only. +- [ ] Test ordered success, missing/duplicate/out-of-order terminal, malformed JSON, mismatched model/effort and sentinel non-disclosure in `scripts/agent_benchmark/claude_iop_test.py`. + +**Test Strategy:** Replay the tracked fixture and independent mutations through `run_invocation` with a fake process. Assert success only on submitted→finish→idle→quiet, and assert secret/prompt/tool/base sentinels are absent from every durable file/error. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.claude_iop_test -v` passes without external access. + +## Dependencies and Execution Order + +- `06_connectivity_contract` must have exactly one active or archived `complete.log` before implementation. The dependency is encoded by `07+06_claude_iop`; it is currently missing during planning. +- This packet does not wait on 08/09 and does not perform the downstream live preflight. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_benchmark/claude_iop.py` | API-1, API-2 | +| `scripts/agent_benchmark/claude_iop_test.py` | API-1, API-2 | +| `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl` | API-2 | +| `agent-task/m-agent-comparison-benchmark-pipeline/07+06_claude_iop/CODE_REVIEW-cloud-G07.md` | API-1, API-2 evidence | + +## Final Verification + +1. `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` + - Expected: exactly one predecessor completion path. +2. `python3 -m unittest scripts.agent_benchmark.claude_iop_test -v` + - Expected: fake CLI, JSONL, binding and redaction cases pass; no network. +3. `make test-agent-comparison-benchmark` + - Expected: all benchmark tests and example validation pass freshly. +4. `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + - Expected: clean-checkout Go suite passes; if blocked, preserve the ignored artifact and record exact evidence. +5. `git diff --check` + - Expected: no whitespace errors. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/CODE_REVIEW-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/CODE_REVIEW-cloud-G07.md new file mode 100644 index 00000000..6c3529d4 --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/CODE_REVIEW-cloud-G07.md @@ -0,0 +1,149 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-10 +task=m-agent-comparison-benchmark-pipeline/08+06_agy_iop, plan=0, tag=API + + + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files and verify that output in `Verification Results` matches code. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Gate agy execution on a proven IOP transport | [ ] | +| API-2 Normalize agy terminal evidence without substitution | [ ] | + +## Implementation Checklist + +- [ ] Implement agy capability/config preflight that permits only a proven IOP endpoint/auth path and otherwise returns an exact implementation gap before launch. +- [ ] Implement fresh one-shot agy invocation plus fail-closed stream-json finish/idle, exact model/effort/binding validation, and structural redaction. +- [ ] Add a secret-safe agy JSONL fixture and supported/gap/boundary tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G07_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G07_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm absent/unknown agy IOP transport fails before launch and cannot use ambient Gemini. +- Confirm credential/model gaps and implementation gaps are distinct closed outcomes with no fallback PASS. +- Confirm exact Gemini model/high effort and ordered finish/idle are required. +- Confirm all persisted/error output is secret-, endpoint-, prompt- and tool-safe. + +## Verification Results + +Paste exact stdout/stderr and exit codes; blockers include exact resume conditions. + +### V1 Predecessor + +Command: `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` + +```text + +``` +Exit code: `` + +### V2 Focused agy tests + +Command: `python3 -m unittest scripts.agent_benchmark.agy_iop_test -v` + +```text + +``` +Exit code: `` + +### V3 Aggregate benchmark tests + +Command: `make test-agent-comparison-benchmark` + +```text + +``` +Exit code: `` + +### V4 Complete Go regression or blocker + +Command: `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + +```text + +``` +Exit code: `` + +### V5 Patch integrity + +Command: `git diff --check` + +```text + +``` +Exit code: `` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/PLAN-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/PLAN-cloud-G07.md new file mode 100644 index 00000000..a784700a --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/PLAN-cloud-G07.md @@ -0,0 +1,184 @@ + + +# Plan - API: agy IOP adapter + +## For the Implementing Agent + +`06_connectivity_contract` 완료 뒤 agy adapter와 전용 fixture/test만 구현한다. 실제 notes/output은 active `CODE_REVIEW-cloud-G07.md`에 기록하고 review-ready로 보고한다. blocker는 exact attempts/output/resume condition만 구현 evidence에 남긴다. 사용자에게 결정 요청, stop 파일, review verdict, archive, `complete.log`는 만들지 않는다. + +## Background + +agy는 현재 CLI에 model/effort/stream-json은 노출하지만 IOP endpoint/auth override가 실제로 지원되는지는 별도 검증이 필요하다. adapter는 지원되는 경로만 실행하고, 지원되지 않으면 정확한 endpoint/auth/protocol/stream implementation gap으로 분류해 ambient Gemini 호출이나 우회 성공을 금지해야 한다. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/common/rules.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/update-test/SKILL.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-ops/skills/common/plan/templates/review-stub-template.md` +- `agent-roadmap/current.md` +- `agent-roadmap/priority-queue.md` +- `agent-spec/index.md` +- `agent-contract/index.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `agent-spec/input/openai-compatible-surface.md` +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/workspace.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/workspace_test.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `agent-test/inventory.yaml` +- `agent-test/inventory-agent.yaml` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` + +### SDD Criteria + +- SDD `[승인됨]`, 잠금 `해제`; milestone-task `agy-iop`; Acceptance `S07`; Evidence Map S07. +- S07은 Gemini direct preflight와 generic preset fixture에서 IOP 경유를 입증하거나 exact compatibility gap을 기록하도록 한다. D07/D08은 gap을 별도 Plan 후보로 분류하고 dispatcher/ambient provider fallback을 금지한다. +- adapter와 fixture/parser unit evidence는 이 packet, 실제 redacted dev preflight 또는 exact blocker는 downstream `10+07,08,09_connectivity_runner`가 소유한다. + +### Verification Context + +- 별도 verification handoff 없음; local rules, repo source/tests/contracts, CLI read-only probes로 보완했다. +- runner `/config/workspace/iop-s0`; branch `feature/agent-comparison-benchmark-pipeline`; HEAD `df0ecdbf1d6cbf8878c84a9b154bc4d022ab035d`; Linux aarch64; Python 3.12.3. 요청 시작 HEAD 이후 두 커밋은 중앙 관리 Agent-Ops 파일만 바꿨고 대상 benchmark source/test는 동일하다. tracked worktree는 materialization 전에 clean이었으며 현재 활성 변경은 이 Epic의 plan/review pair뿐이다. +- agy wrapper `/config/.local/bin/agy`, version `1.1.11`; underlying `/config/.local/bin/agy-real`. Help에서 `--print`, `--output-format stream-json`, `--model`, `--effort low|medium|high`, `--mode`, `--sandbox`를 확인했으나 public base-url/provider override option은 확인되지 않았다. +- agent inventory는 agy를 configured지만 quota-blocked로 기록한다. dev `gemini-3.6-flash` route는 exact active match다. provider/network probe는 NOT_RUN이며 이 조합을 ready로 단정하지 않는다. +- caller config/auth path와 private endpoint/secret 원문은 읽지 않았다. 로컬 포트 요구 없음; external host는 redacted dev IOP Edge. +- benchmark 215-test 기준선은 통과한다. 전체 Go 회귀는 ignored build artifact 때문에 현재 차단된다. Confidence: high for fail-closed adapter behavior, medium for live agy compatibility. + +### Test Coverage Gaps + +- agy endpoint/auth capability가 없을 때 실행 전 implementation gap으로 끝나는 테스트가 없다. +- agy stream-json finish/idle/malformed/duplicate parsing과 exact model/effort 검증이 없다. +- ambient Gemini auth/provider 사용 금지, secret/private endpoint/prompt/tool non-disclosure fixture가 없다. + +### Symbol References + +rename/remove symbol은 없다. 새 adapter는 06 contract와 기존 `InvocationSpec`, `run_invocation`, `PreparedWorkspace`, `MatrixCell`을 소비한다. + +### Split Judgment + +agy adapter는 독립 file/fixture로 credential-free PASS가 가능하다. `08+06`의 유일한 predecessor는 `06_connectivity_contract`; 현재 exact `complete.log`는 missing이며 runtime은 active/archive completion 전 시작하면 안 된다. Claude/Codex packet과 직접 dependency는 없다. + +### Scope Rationale + +agy 제품 수정, undocumented setting 주입, ambient Google/Gemini 호출, CLI registry, live route 등록, report/evidence normalization, hybrid preset live success는 제외한다. unsupported endpoint/auth를 adapter 내부 proxy나 임시 provider call로 보완하지 않는다. + +### Final Routing + +- evaluation_mode `first-pass`; finalizer `finalize-task-policy.sh pair`. +- build closures 모두 true; scores `2/1/1/1/2`; base `local-fit`, route `risk-boundary`; cloud G07; `PLAN-cloud-G07.md`; catalog `worker/cloud/G07`. +- review closures 모두 true; 같은 scores; `official-review`; cloud G07; `CODE_REVIEW-cloud-G07.md`; catalog `review/cloud/G07`. +- large_indivisible_context `false`; risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product`; count `4`; rework `0`; evidence integrity `false`; capability gap 없음. + +## Implementation Checklist + +- [ ] Implement agy capability/config preflight that permits only a proven IOP endpoint/auth path and otherwise returns an exact implementation gap before launch. +- [ ] Implement fresh one-shot agy invocation plus fail-closed stream-json finish/idle, exact model/effort/binding validation, and structural redaction. +- [ ] Add a secret-safe agy JSONL fixture and supported/gap/boundary tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Gate agy execution on a proven IOP transport + +**Problem:** agy 1.1.11 exposes model/effort/stream output but its help has no base-url/provider override. Launching with ambient configuration could call Gemini directly and falsely label the result as IOP. + +**Solution:** Add `agy_iop.py` implementing the 06 adapter protocol. Probe only documented/versioned caller capabilities and validated runtime inputs. Build an IOP-only child environment/config when a supported override is provable; otherwise return `endpoint_compatibility`, `auth_compatibility`, or `protocol_compatibility` as `implementation_gap` before any caller process starts. Missing credential/model maps to registration, not implementation, and neither path falls back. + +Before (agy 1.1.11 help observed in preflight): + +```text +--model, --effort, --output-format, --print, --sandbox +(no documented base-url/provider option) +``` + +After (adapter decision): + +```python +capability = inspect_agy_iop_capability(binary, runtime_inputs) +if not capability.iop_transport_supported: + return PreflightResult(..., status="implementation_gap", issues=(...,)) +return build_agy_invocation(...) +``` + +**Modified Files and Checklist:** + +- [ ] Create `scripts/agent_benchmark/agy_iop.py` with capability inspection, closed failure mapping and guarded invocation construction. +- [ ] Create `scripts/agent_benchmark/agy_iop_test.py` proving no launch on absent/unknown endpoint/auth capability, registration classification and no ambient provider fallback. + +**Test Strategy:** Fake help/version and executable behavior in temp paths. Cover supported transport, missing credential/model, endpoint/auth/protocol gaps, unknown versions and quota/provider errors without network. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.agy_iop_test -v` passes. + +### [API-2] Normalize agy terminal evidence without substitution + +**Problem:** generic lifecycle requires ordered finish then idle, but agy-specific stream JSON and content redaction are not implemented. A generic success exit cannot prove IOP route or exact Gemini high effort. + +**Solution:** Build a fresh sandboxed print-mode invocation with one stdin task, exact request model/effort and no conversation reuse. Parse closed JSON shapes for response finish and fully-idle state, validate any reported model/route/effort exactly, and redact structured content/tool fields plus sensitive runtime values before capture. Unavailable effective evidence remains unavailable and blocks ready. + +Before (`scripts/agent_benchmark/lifecycle.py:1301-1312`): + +```python +if parsed.startswith(METRIC_PREFIX): ... +if parsed not in PARSER_TERMINAL_KINDS: + self.reason = REASON_MALFORMED_EVENT +``` + +After (agy parser boundary): + +```python +parse_agy_event(stream, raw_line) -> None | "finish" | "idle" | "metric:..." +redact_agy_event(raw_line, sensitive_values) -> canonical_redacted_json +``` + +**Modified Files and Checklist:** + +- [ ] Implement invocation/parser/redactor and exact binding checks in `scripts/agent_benchmark/agy_iop.py`. +- [ ] Add `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` containing synthetic public metadata and redacted content. +- [ ] Add ordered terminal, malformed/duplicate, mismatch, quota error and sentinel tests in `scripts/agent_benchmark/agy_iop_test.py`. + +**Test Strategy:** Replay fixture and independent mutations through fake agy/lifecycle. Success requires submitted→finish→idle→quiet; quota remains a blocker; every durable byte excludes raw content/secrets/endpoints. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.agy_iop_test -v` passes credential-free. + +## Dependencies and Execution Order + +- `06_connectivity_contract` must produce exactly one active/archive `complete.log`; encoded by `08+06_agy_iop` and currently missing. +- Live readiness is a downstream consumer check, not a hidden predecessor of this adapter unit packet. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_benchmark/agy_iop.py` | API-1, API-2 | +| `scripts/agent_benchmark/agy_iop_test.py` | API-1, API-2 | +| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-2 | +| `agent-task/m-agent-comparison-benchmark-pipeline/08+06_agy_iop/CODE_REVIEW-cloud-G07.md` | API-1, API-2 evidence | + +## Final Verification + +1. `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` +2. `python3 -m unittest scripts.agent_benchmark.agy_iop_test -v` + - Expected: supported and exact-gap fixtures pass without provider/network access. +3. `make test-agent-comparison-benchmark` +4. `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + - Expected: clean checkout passes; otherwise record the ignored-artifact blocker without modifying it. +5. `git diff --check` + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/CODE_REVIEW-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/CODE_REVIEW-cloud-G07.md new file mode 100644 index 00000000..5349c881 --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/CODE_REVIEW-cloud-G07.md @@ -0,0 +1,149 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-10 +task=m-agent-comparison-benchmark-pipeline/09+06_codex_iop, plan=0, tag=API + + + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files and verify that output in `Verification Results` matches code. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Build an isolated IOP Responses invocation | [ ] | +| API-2 Bridge Codex JSONL finish to verified idle | [ ] | + +## Implementation Checklist + +- [ ] Implement a fresh Codex exec invocation with isolated IOP Responses provider config, one stdin task, exact GPT model/xhigh, and no ambient config or fallback. +- [ ] Implement a bounded JSONL bridge/parser that proves turn finish then process idle, validates effective binding when observable, and structurally redacts content and runtime secrets. +- [ ] Add a secret-safe Codex JSONL fixture and provider/config/terminal/gap tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G07_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G07_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm ambient Codex config/provider/session cannot influence the invocation. +- Confirm exact IOP Responses provider, GPT model and xhigh request with no fallback/substitution. +- Confirm separate turn finish and verified child idle evidence plus owned cleanup. +- Confirm raw endpoint/secret/prompt/tool data never reaches tracked config, capture, error or fixture. + +## Verification Results + +Paste exact stdout/stderr and exit code for every command; blockers need exact resume condition. + +### V1 Predecessor + +Command: `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` + +```text + +``` +Exit code: `` + +### V2 Focused Codex tests + +Command: `python3 -m unittest scripts.agent_benchmark.codex_iop_test -v` + +```text + +``` +Exit code: `` + +### V3 Aggregate benchmark tests + +Command: `make test-agent-comparison-benchmark` + +```text + +``` +Exit code: `` + +### V4 Complete Go regression or blocker + +Command: `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + +```text + +``` +Exit code: `` + +### V5 Patch integrity + +Command: `git diff --check` + +```text + +``` +Exit code: `` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/PLAN-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/PLAN-cloud-G07.md new file mode 100644 index 00000000..19f24c6d --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/PLAN-cloud-G07.md @@ -0,0 +1,185 @@ + + +# Plan - API: Codex IOP adapter + +## For the Implementing Agent + +`06_connectivity_contract` 완료 계약 위에서 Codex adapter와 전용 fixture/test만 구현한다. 실제 구현/evidence를 active `CODE_REVIEW-cloud-G07.md`에 채우고 review-ready로 보고한다. blocker는 exact commands/output/resume condition만 기록한다. 사용자 결정 요청, stop 파일, verdict, archive, `complete.log`는 구현자 범위가 아니다. + +## Background + +Codex exec는 custom model provider와 JSONL 실행을 제공하지만 현재 benchmark에는 IOP-only provider config, fresh session, finish/idle bridge, exact GPT model/xhigh evidence가 없다. ambient OpenAI 설정이나 alias fallback 없이 IOP를 소비하고 관측 불가능한 effective binding을 gap으로 남기는 adapter가 필요하다. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/common/rules.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/update-test/SKILL.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-ops/skills/common/plan/templates/review-stub-template.md` +- `agent-roadmap/current.md` +- `agent-roadmap/priority-queue.md` +- `agent-spec/index.md` +- `agent-contract/index.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `agent-spec/input/openai-compatible-surface.md` +- `agent-spec/runtime/provider-pool-config-refresh.md` +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/workspace.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/workspace_test.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `agent-test/inventory.yaml` +- `agent-test/inventory-agent.yaml` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` + +### SDD Criteria + +- SDD `[승인됨]`, 잠금 `해제`; milestone-task `codex-iop`; Acceptance `S08`; Evidence Map S08. +- S08은 GPT direct preflight/generic preset fixture에서 IOP 경유를 입증하거나 endpoint/auth/protocol/stream exact blocker를 남기도록 한다. S09는 xhigh no-substitution을 요구한다. +- adapter unit evidence는 이 packet, 실제 redacted dev preflight 또는 exact blocker는 downstream `10+07,08,09_connectivity_runner`가 수행한다. + +### Verification Context + +- verification handoff 없음; local rules, repository source/tests/contracts와 installed CLI read-only probes로 보완했다. +- runner `/config/workspace/iop-s0`; branch `feature/agent-comparison-benchmark-pipeline`; HEAD `df0ecdbf1d6cbf8878c84a9b154bc4d022ab035d`; Linux aarch64; Python 3.12.3. 요청 시작 HEAD 이후 두 커밋은 중앙 관리 Agent-Ops 파일만 바꿨고 대상 benchmark source/test는 동일하다. tracked worktree는 materialization 전에 clean이었으며 현재 활성 변경은 이 Epic의 plan/review pair뿐이다. +- Codex binary `/config/.npm-global/bin/codex`, version `codex-cli 0.147.0`. `codex exec --help`에서 `--json`, `--ephemeral`, `--ignore-user-config`, `--strict-config`, `-m/--model`, `-c/--config`, `-C/--cd`, sandbox options와 stdin prompt form을 확인했다. +- current agent inventory의 기본 GPT profile은 luna가 아니며 dev `gpt-5.6-luna` exact route도 미등록이다. external network/provider is NOT_RUN; readiness로 간주하지 않는다. +- auth/config/private endpoint 원문은 읽지 않았다. adapter는 caller-supplied runtime env를 child에만 전달하고 config/argv/capture에는 raw secret을 남기지 않는다. local port 없음; external host는 redacted dev IOP Edge. +- benchmark 215-test baseline과 patch check는 pass. full Go는 ignored build artifact로 blocked. Confidence: high for adapter contract, medium for live route/effective evidence. + +### Test Coverage Gaps + +- Codex custom IOP provider config가 ambient config를 완전히 차단하고 exact Responses wire/model/xhigh를 쓰는 테스트가 없다. +- `thread/turn/item` JSONL과 process exit를 finish→idle로 안전하게 연결하는 parser/shim 테스트가 없다. +- endpoint/secret/prompt/tool output non-disclosure 및 unavailable effective model fail-closed coverage가 없다. + +### Symbol References + +rename/remove는 없다. 새 adapter는 06 contract와 `InvocationSpec`, `run_invocation`, `PreparedWorkspace`, `MatrixCell`을 소비한다. + +### Split Judgment + +Codex adapter는 독립 fixture와 fake binary로 PASS할 수 있다. `09+06` predecessor는 `06_connectivity_contract` 하나이며 현재 exact completion은 missing이다. runtime은 active/archive `complete.log` 전 구현을 시작하지 않는다; Claude/agy와 직접 dependency는 없다. + +### Scope Rationale + +Codex 제품/SDK 수정, user config 변경, OpenAI direct fallback, shared session/resume, CLI registry, live model registration, reporting/normalization, hybrid preset live readiness는 제외한다. raw endpoint/secret을 tracked TOML/JSONL/evidence에 기록하지 않는다. + +### Final Routing + +- evaluation_mode `first-pass`; finalizer `finalize-task-policy.sh pair`. +- build closures 모두 true; scores `2/1/1/1/2`; base `local-fit`; route `risk-boundary`; cloud G07; `PLAN-cloud-G07.md`; `worker/cloud/G07`. +- review closures 모두 true; same scores; `official-review`; cloud G07; `CODE_REVIEW-cloud-G07.md`; `review/cloud/G07`. +- large context false; risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product`; count 4; rework 0; integrity false; capability gap 없음. + +## Implementation Checklist + +- [ ] Implement a fresh Codex exec invocation with isolated IOP Responses provider config, one stdin task, exact GPT model/xhigh, and no ambient config or fallback. +- [ ] Implement a bounded JSONL bridge/parser that proves turn finish then process idle, validates effective binding when observable, and structurally redacts content and runtime secrets. +- [ ] Add a secret-safe Codex JSONL fixture and provider/config/terminal/gap tests; run predecessor, focused, aggregate, Go baseline, and patch-integrity verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Build an isolated IOP Responses invocation + +**Problem:** lifecycle has no Codex-specific command builder, while invoking `codex exec` with ambient config could select another provider/model or persistent session. + +**Solution:** Add `codex_iop.py` implementing the 06 adapter. Use `codex exec --json --ephemeral --ignore-user-config --strict-config -C -m -` with closed `-c` overrides for a dedicated IOP provider, Responses wire and exact reasoning effort. Obtain base/secret from caller-supplied env, use a named secret env key, serialize neither raw value, and refuse unknown config keys/unsupported effort. + +Before (`codex exec --help`, observed): + +```text +-c/--config, --strict-config, -m/--model, --ephemeral, +--ignore-user-config, --json, -C/--cd +``` + +After (conceptual exact builder): + +```python +build_codex_spec(cell, prepared, runtime) -> InvocationSpec +# model_provider=iop_benchmark, wire_api=responses, +# model=, model_reasoning_effort= +``` + +**Modified Files and Checklist:** + +- [ ] Create `scripts/agent_benchmark/codex_iop.py` with runtime validation and isolated invocation construction. +- [ ] Create `scripts/agent_benchmark/codex_iop_test.py` covering exact provider/wire/model/effort, stdin once, absent/unknown config, no ambient provider and no persistence. + +**Test Strategy:** Use fake codex and temp environment. Assert command/config equality without writing raw endpoint/credential and fail before launch on unsupported keys/effort. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.codex_iop_test -v` passes. + +### [API-2] Bridge Codex JSONL finish to verified idle + +**Problem:** Codex JSONL exposes turn/item events, while generic lifecycle requires two ordered terminal facts and cannot infer idle from a zero exit (`lifecycle.py:1369-1379`). Mapping a single `turn.completed` to both would violate the one-event parser and duplicate evidence rules. + +**Solution:** Use a bounded adapter shim inside the owned process group to stream child JSONL unchanged through structural redaction, emit finish only for the successful terminal turn, and emit one synthetic closed `adapter.idle` record only after child exit/EOF is verified. Parse that separate record as idle. Validate reported route/model/effort exactly when present; classify missing effective evidence rather than filling it from request config. + +Before (`scripts/agent_benchmark/lifecycle.py:1376-1379`): + +```python +if self.exit_code != 0: + self.reason = REASON_NONZERO_EXIT +elif self.idle_at is None: + self.reason = REASON_MISSING_IDLE +``` + +After (shim sequence): + +```text +turn.completed -> finish +verified child exit + stdout EOF -> adapter.idle -> idle +``` + +**Modified Files and Checklist:** + +- [ ] Implement bounded child bridge, JSONL parser and structural redactor in `scripts/agent_benchmark/codex_iop.py`. +- [ ] Add `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` with synthetic public events/redacted content. +- [ ] Test finish/idle order, child failure, missing/duplicate terminal, malformed JSON, binding mismatch/unavailable and sentinel removal in `scripts/agent_benchmark/codex_iop_test.py`. + +**Test Strategy:** Replay fixture/mutations through fake child and real generic lifecycle. Assert owned process cleanup and durable evidence coherence in addition to parser outcomes. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.codex_iop_test -v` passes without network. + +## Dependencies and Execution Order + +- `06_connectivity_contract` must have one exact active/archive `complete.log`; dependency encoded by `09+06_codex_iop`, currently missing. +- Downstream integration owns live dev preflight and is not a predecessor of this fixture-driven adapter packet. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_benchmark/codex_iop.py` | API-1, API-2 | +| `scripts/agent_benchmark/codex_iop_test.py` | API-1, API-2 | +| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | API-2 | +| `agent-task/m-agent-comparison-benchmark-pipeline/09+06_codex_iop/CODE_REVIEW-cloud-G07.md` | API-1, API-2 evidence | + +## Final Verification + +1. `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; i="06"; a=Path("agent-task")/g; r=Path("agent-task/archive"); p=sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str); assert len(p)==1,[str(x) for x in p]; print(p[0])'` +2. `python3 -m unittest scripts.agent_benchmark.codex_iop_test -v` + - Expected: fake provider/config/JSONL/bridge/redaction tests pass without network. +3. `make test-agent-comparison-benchmark` +4. `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + - Expected: clean checkout passes; otherwise record ignored-artifact blocker and preserve it. +5. `git diff --check` + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/CODE_REVIEW-cloud-G10.md b/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/CODE_REVIEW-cloud-G10.md new file mode 100644 index 00000000..f54ad479 --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/CODE_REVIEW-cloud-G10.md @@ -0,0 +1,230 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-10 +task=m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner, plan=0, tag=API + + + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files and verify that output in `Verification Results` matches code. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_0.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Add append-only public route preflight | [ ] | +| API-2 Wire ready runs and append-only resume attempts | [ ] | +| API-3 Publish the exact skill and fixture contract | [ ] | +| API-4 Capture direct dev evidence or an exact external blocker | [ ] | + +## Implementation Checklist + +- [ ] Add the three-adapter registry and public `preflight` command; persist canonical append-only preflight attempts and expose only redacted ready/registration/implementation summaries. +- [ ] Enable run/resume to preflight under the run writer, stop before attempt allocation on any blocker, and otherwise prepare fresh workspaces and invoke each exact caller once through `run_slots`. +- [ ] Update public exports, direct-live/generic-preset fixtures, project skill, and contract/integration tests as one CLI compatibility change while keeping report-output unavailable and public prepare absent. +- [ ] Run dependency, focused, aggregate, CLI/fake integration, direct dev preflight, full Go, and patch-integrity verification; record exact external blocker evidence without fallback. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G10_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm preflight records are append-only, writer-owned, schema-validated and never counted as scored attempts. +- Confirm any blocker prevents attempt allocation and no model/effort/provider fallback can produce ready/PASS. +- Confirm run/resume preserve old attempts, fresh session/workspace and one task submission with finish→idle. +- Confirm CLI help, project skill and contract tests agree exactly, report-output remains unavailable and public prepare stays absent. +- Confirm direct live evidence covers all five cells or official review stops on exact external blocker; generic preset remains fixture-only. +- Confirm tracked/source/evidence diff contains no secret, private endpoint, prompt/tool raw content or user config mutation. + +## Verification Results + +Paste exact stdout/stderr and exit code for every command. For long output cite an exact durable workspace path and the command that wrote it. External blockers must include authorization state and exact resume condition. + +### V1 Predecessors + +Command: `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; ids=("07","08","09"); a=Path("agent-task")/g; r=Path("agent-task/archive"); f={i:sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str) for i in ids}; bad={i:[str(x) for x in p] for i,p in f.items() if len(p)!=1}; assert not bad,bad; print("\n".join(str(f[i][0]) for i in ids))'` + +```text + +``` +Exit code: `` + +### V2 Focused integration tests + +Command: `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test scripts.agent_benchmark.manifest_test -v` + +```text + +``` +Exit code: `` + +### V3 CLI surface + +Command: `python3 scripts/agent_comparison_benchmark.py --help && python3 scripts/agent_comparison_benchmark.py validate --help && python3 scripts/agent_comparison_benchmark.py preflight --help && python3 scripts/agent_comparison_benchmark.py run --help && python3 scripts/agent_comparison_benchmark.py resume --help && python3 scripts/agent_comparison_benchmark.py status --help` + +```text + +``` +Exit code: `` + +### V4 Fixture validation + +Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json && python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` + +```text + +``` +Exit code: `` + +### V5 Aggregate benchmark tests + +Command: `make test-agent-comparison-benchmark` + +```text + +``` +Exit code: `` + +### V6 Caller binaries + +Command: `command -v claude && claude --version && command -v agy && agy --version && command -v codex && codex --version` + +```text + +``` +Exit code: `` + +### V7A Sonnet dev route inventory + +Command: `go run ./scripts/inventory-query --env dev --model claude-sonnet-5` + +```text + +``` +Exit code: `` + +### V7B Gemini dev route inventory + +Command: `go run ./scripts/inventory-query --env dev --model gemini-3.6-flash` + +```text + +``` +Exit code: `` + +### V7C GPT dev route inventory + +Command: `go run ./scripts/inventory-query --env dev --model gpt-5.6-luna` + +```text + +``` +Exit code: `` + +### V8 Runtime input presence + +Command: `python3 -c 'import os; pairs=(("IOP_BENCH_CLAUDE_BASE_URL","IOP_BENCH_CLAUDE_SECRET_ENV"),("IOP_BENCH_AGY_BASE_URL","IOP_BENCH_AGY_SECRET_ENV"),("IOP_BENCH_CODEX_BASE_URL","IOP_BENCH_CODEX_SECRET_ENV")); missing=[]; [(missing.append(base) if not os.environ.get(base) else None, missing.append(ref) if not os.environ.get(ref) else (missing.append(ref+"->secret") if not os.environ.get(os.environ[ref]) else None)) for base,ref in pairs]; assert not missing,missing; print("ok: caller runtime inputs present")'` + +```text + +``` +Exit code: `` + +### V9 Authorized direct dev preflight + +Command: `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` + +```text + +``` +Exit code: `` + +Durable redacted evidence path: `` +Authorization/runtime state: `` +Resume condition: `` + +### V10 Complete Go regression or blocker + +Command: `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` + +```text + +``` +Exit code: `` + +### V11 Patch integrity + +Command: `git diff --check` + +```text + +``` +Exit code: `` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Fixed at stub creation | Implementing agent fills in command output only; command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/PLAN-cloud-G10.md b/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/PLAN-cloud-G10.md new file mode 100644 index 00000000..38794fd9 --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/PLAN-cloud-G10.md @@ -0,0 +1,274 @@ + + +# Plan - API: connectivity preflight runner integration + +## For the Implementing Agent + +완료된 07/08/09 adapter를 deterministic CLI, append-only run state, project skill에 원자적으로 연결한다. 로컬 fixture 검증과 redacted dev preflight를 실행하고 actual stdout/stderr, run/evidence path, exit code를 active `CODE_REVIEW-cloud-G10.md`에 채운 뒤 review-ready로 보고한다. 외부 등록/credential/quota 때문에 막히면 exact preflight, authorization 상태와 재개 조건만 기록하고 사용자 질문·stop 파일·next-state 분류는 하지 않는다. review verdict, archive, `complete.log`는 reviewer 전용이다. + +## Background + +현재 CLI의 valid run/resume는 caller adapter capability-unavailable로 종료하며 실제 adapter registry, preflight evidence, workspace/attempt execution을 연결하지 않는다. 세 adapter가 준비된 뒤 public preflight와 run/resume를 함께 갱신해야 skill 문서와 CLI가 같은 IOP-only, no-substitution, append-only 상태 계약을 유지한다. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/common/rules.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/update-test/SKILL.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-ops/skills/common/plan/templates/review-stub-template.md` +- `agent-roadmap/current.md` +- `agent-roadmap/priority-queue.md` +- `agent-spec/index.md` +- `agent-contract/index.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` +- `agent-spec/input/openai-compatible-surface.md` +- `agent-spec/runtime/provider-pool-config-refresh.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `scripts/agent_comparison_benchmark.py` +- `scripts/agent_benchmark/__init__.py` +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/workspace.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/workspace_test.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark/prompt.md` +- `scripts/fixtures/agent-comparison-benchmark/reference.txt` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-test/README.md` +- `agent-test/inventory.yaml` +- `agent-test/inventory-agent.yaml` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `Makefile` + +### SDD Criteria + +- SDD `[승인됨]`, 잠금 `해제`; milestone-task `claude-iop,agy-iop,codex-iop,effort-route,connection-gap`; Acceptance S06-S10; Evidence Map S06-S10. +- S06 requires redacted Claude→IOP auth/model/stream evidence; S07/S08 accept supported evidence or exact caller compatibility blocker; S09 requires requested/effective matrix and no substitution; S10 requires registration-vs-implementation classification with no bypass PASS. +- D01 limits this Epic to direct live connectivity plus generic preset fixture; hybrid preset live readiness remains bench-02. D02/D08 require all scored calls through IOP and direct CLI entrypoints, not dispatchers. D04/D10 require one user task, fresh session/workspace and finish→idle. +- Hence the checklist joins adapters, preflight state, exact binding gate, attempt execution, public skill contract, direct-live/generic fixtures and external evidence without adding report/timing/web/scoring. + +### Verification Context + +- No formal verification handoff was supplied. Sources are local test rules/profiles, Makefile, current source/tests/contracts, caller help/version and bounded inventory queries. +- Current checkout: `/config/workspace/iop-s0`; branch `feature/agent-comparison-benchmark-pipeline`; HEAD `df0ecdbf1d6cbf8878c84a9b154bc4d022ab035d`. Since user start HEAD `01abc9d17b9d8c88add8a54fb79d5911b486a004`, two commits changed only centrally managed `agent-ops/.version` and `prepare-epic-work-items` script/tests; target benchmark files are unchanged. The tracked worktree was clean before materialization, and the only active changes are this Epic's plan/review pairs. +- OS/arch Linux 6.10 aarch64; Python 3.12.3; Go 1.26.2. Caller binaries: Claude `/config/.local/bin/claude` 2.1.223, agy `/config/.local/bin/agy` 1.1.11, Codex `/config/.npm-global/bin/codex` 0.147.0. Required help was read without invoking providers. +- Repo testbed is read-only `../iop-s2`; run artifacts stay below manifest `agent-test/runs///`. No local listening port is required; external host is the caller-supplied redacted dev IOP Edge. +- Config paths/credentials are user-managed outside the repo and were not read. The runner must accept only public env-var names for caller base/config/secret references, check presence without printing values, and keep raw values out of argv evidence, run JSON, output and errors. +- Inventory: dev `gemini-3.6-flash` exact active; `claude-sonnet-5` and `gpt-5.6-luna` absent; agy is configured but quota-blocked. Thus live S06-S09 completion is presently expected to block until registration/quota/runtime readiness changes; this is external evidence, not a fabricated PASS or product decision. +- Baseline `make test-agent-comparison-benchmark` passed 215 tests and manifest validation; `git diff --check` passed. `go test ./... -count=1` is currently blocked by ignored user artifact `build/r14-remote-anthropic_handler.go`; preserve it and record exact evidence. +- External Verification Preflight: runner=current host; repo/head/dirty as above; source sync=current; binaries/versions as above; config=user-owned env references; runtime identity=dev IOP Edge; ports=not applicable/local none; external host=redacted; OS/arch=Linux/aarch64. Network calls were NOT_RUN during planning. Setup/resume condition is exact route registration, caller auth config, quota and endpoint reachability supplied by the authorized operator. Confidence: high for local integration oracle; medium for current live readiness. + +### Test Coverage Gaps + +- `attempts_test.py:652-669` asserts run/resume stay unavailable and must be replaced with preflight-before-attempt, ready/blocker persistence, retry and status tests. +- `skill_contract_test.py:31-35`, `379-405`, `530-599` fixes caller-adapter unavailable and exactly four commands; it must validate the new preflight and available run/resume contract while retaining report-output unavailable and no public prepare. +- No integration test proves all three adapters are registered, blockers allocate no attempt, successful preflight is not reused as scored output, or resume appends rather than overwrites preflight/attempt evidence. +- Example fixture lacks Claude generic preset and target direct preflight matrix. Separate generic contract and direct-live fixtures are needed so hybrid preset live readiness is not required. + +### Symbol References + +- No symbol is renamed/removed. `run_slots` signature/contract changes; all call sites found are `scripts/agent_benchmark/attempts_test.py:274,289,296,299,301,310`, the definition at `attempts.py:719`, export at `__init__.py:63,114`, and the new CLI integration. +- `CapabilityUnavailable("capability-unavailable: caller-adapter")` behavior references are in `agent_comparison_benchmark.py:93-100`, `skill_contract_test.py:31,379-405,589-599,731-740`, and project skill lines 4,54-61,98-105,118,127; update them cohesively rather than leaving contradictory docs/tests. + +### Split Judgment + +CLI/state/skill integration is one indivisible compatibility boundary: exposing preflight or enabling run without registry, append-only state and updated contract tests would create a false public capability. Its stable predecessors are `07+06_claude_iop`, `08+06_agy_iop`, `09+06_codex_iop`; all three exact `complete.log` files are missing during planning. `10+07,08,09` encodes exactly these dependencies; 06 is transitive and not duplicated. + +### Scope Rationale + +Excluded: Edge/provider product implementation, credential/model registration, route-02 hybrid preset live success, 9-cell scored comparison, timing/usage normalization, web validation, blind scoring, report output, roadmap/SDD/spec/contract changes, common Agent-Ops files. The implementation may report external blockers but must not invent config, mutate user caller profiles, or write raw secrets/private endpoints/prompt/tool content. + +### Final Routing + +- evaluation_mode `first-pass`; finalizer `finalize-task-policy.sh pair`. +- build closures all true; scores `2/2/2/2/2`; base/route `grade-boundary`; cloud G10; `PLAN-cloud-G10.md`; catalog `worker/cloud/G10`. +- review closures all true; same scores; `official-review`; cloud G10; `CODE_REVIEW-cloud-G10.md`; catalog `review/cloud/G10`. +- large_indivisible_context false after safe split; risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product`; count 5; review_rework_count 0; evidence_integrity_failure false; capability gap none because exact external blocker handling is part of the closed plan. + +## Implementation Checklist + +- [ ] Add the three-adapter registry and public `preflight` command; persist canonical append-only preflight attempts and expose only redacted ready/registration/implementation summaries. +- [ ] Enable run/resume to preflight under the run writer, stop before attempt allocation on any blocker, and otherwise prepare fresh workspaces and invoke each exact caller once through `run_slots`. +- [ ] Update public exports, direct-live/generic-preset fixtures, project skill, and contract/integration tests as one CLI compatibility change while keeping report-output unavailable and public prepare absent. +- [ ] Run dependency, focused, aggregate, CLI/fake integration, direct dev preflight, full Go, and patch-integrity verification; record exact external blocker evidence without fallback. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Add append-only public route preflight + +**Problem:** `agent_comparison_benchmark.py:57-64` exposes only validate/run/resume/status, while `:92-100` hard-stops valid run/resume. There is no operator command to collect S06-S10 setup evidence separately from scored attempts. + +**Solution:** Add `preflight --manifest`. Build the exact registry from the three adapter modules, verify all manifest callers before state mutation, create a run identity for durable setup evidence, and append `preflight/preflight-NNNNNN.json` under the run writer. Return 0 only when every direct cell is ready and bindings are exact; return 69 with run id and closed blocker summary otherwise. Never print raw runtime inputs. Generic preset cells are contract-validated locally, not live-gated in bench-01. + +Before (`scripts/agent_comparison_benchmark.py:57-64`): + +```python +for command in ("run", "resume", "status"): + entry = sub.add_parser(command, help=f"Safely {command} benchmark state.") +``` + +After: + +```python +for command in ("preflight", "run", "resume", "status"): ... +adapters = build_adapter_registry(runtime_environment()) +results = preflight_manifest(manifest, adapters, direct_live=True) +path = store.record_preflight(run, results) +``` + +**Modified Files and Checklist:** + +- [ ] Update `scripts/agent_comparison_benchmark.py` parser, sanitized outputs, adapter registry and preflight handler. +- [ ] Update `scripts/agent_benchmark/attempts.py` with writer-owned append-only preflight sequencing/read validation and status projection. +- [ ] Export connectivity/adapters through `scripts/agent_benchmark/__init__.py` only after all modules exist. +- [ ] Add preflight state/order/corruption/secret tests to `scripts/agent_benchmark/attempts_test.py` and `scripts/agent_benchmark/connectivity_integration_test.py`. + +**Test Strategy:** Fake all caller binaries/runtime inputs. Cover all-ready, each blocker class, mixed blockers, no adapter, duplicate/concurrent preflight, corrupted/symlink evidence, exact status and no attempt directories on blocked preflight. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test scripts.agent_benchmark.manifest_test -v` passes and fake preflight output contains only closed redacted fields. + +### [API-2] Wire ready runs and append-only resume attempts + +**Problem:** `attempts.py:719-750` accepts plain callables and prepares/invokes slots, but caller/cell/prepared workspace binding and preflight results are not coupled. CLI run never creates a run or calls this path. + +**Solution:** Evolve `run_slots` to consume typed adapters, find the exact `MatrixCell`, prepare one clean workspace, and pass cell+prepared identity to one invocation. Under the existing fail-fast writer, append a fresh preflight record before any new attempt on run/resume; any blocker stops with no allocation. Ready run calls `RunStore.create`, resume calls `open`, both preserve prior failures and `--retry-failed` allocates a new attempt only. Preflight usage/evidence never becomes scored attempt output. + +Before (`scripts/agent_benchmark/attempts.py:724-750`): + +```python +adapters: dict[str, Callable[..., InvocationResult]] +... +attempt = store.allocate(bound_run, slot) +completed.append(store.execute_attempt(attempt, prepare=prepare, + invoke=adapters[adapter])) +``` + +After: + +```python +adapters: Mapping[str, CallerAdapter] +cell = cells_by_id[slot.cell_id] +prepared = prepare(attempt) +adapter.invoke(cell, prepared, attempt, on_started) +``` + +**Modified Files and Checklist:** + +- [ ] Update orchestration and durable validation in `scripts/agent_benchmark/attempts.py`. +- [ ] Wire run/resume/status in `scripts/agent_comparison_benchmark.py` without nested writers or mutable prior evidence. +- [ ] Update every listed `run_slots` call and add ready/blocker/retry/crash tests in `scripts/agent_benchmark/attempts_test.py`. +- [ ] Add fake three-caller end-to-end run/resume/status coverage in `scripts/agent_benchmark/connectivity_integration_test.py`. + +**Test Strategy:** Use temp repo/testbed, fake adapters and deterministic clock/token. Assert preflight precedes allocation, preparation precedes one task submission, blocked cells never run, ready cells all run, crash/retry preserves old bytes and every status is read-only. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test scripts.agent_benchmark.manifest_test -v` passes. + +### [API-3] Publish the exact skill and fixture contract + +**Problem:** project skill and its validator currently state adapters/providers must never run and exact CLI commands exclude preflight. The example has generic agy/Codex preset cells but no Claude preset and uses stale direct model names; there is no direct-only live readiness fixture. + +**Solution:** Update the project skill to delegate preflight/run/resume only to the deterministic CLI, prohibit ad-hoc provider calls/dispatchers and retain report-output unavailable/public prepare absent. Update validator semantics and mutations to require preflight, IOP-only delegation, exact output and blocker handling. Keep the generic preset example non-live and add a separate direct-preflight manifest with Claude→Sonnet 5 `max`, Claude→Gemini 3.6 Flash `high`, Claude→GPT-5.6 luna `xhigh`, agy→Gemini `high`, and Codex→GPT `xhigh`. Unsupported target effort is a blocker, never an alias substitution. + +Before (`agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md:118-128`): + +```markdown +- Stop immediately and report `capability-unavailable: caller-adapter` for run/resume ... +- Do not invoke caller adapters, provider APIs, or any external service. +``` + +After: + +```markdown +- Delegate preflight/run/resume only to the deterministic benchmark CLI. +- Do not invoke a caller/provider outside that CLI or bypass a classified blocker. +``` + +**Modified Files and Checklist:** + +- [ ] Update `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` inputs/procedure/output/safety/stop/prohibitions for preflight and available adapters. +- [ ] Update `scripts/agent_benchmark/skill_contract_test.py` command parity, semantic/mutation and stateful fake delegation tests while retaining report-output unavailable/no prepare. +- [ ] Update `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` with all three generic preset adapter contracts and no live readiness claim. +- [ ] Add `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` with five direct target cells and only public aliases. +- [ ] Update `scripts/agent_benchmark/manifest_test.py` to validate both fixtures, exact cell scopes and D01 separation. + +**Test Strategy:** Credential-free contract tests invoke only help/fake adapters and tracked fixtures. Independent mutations add ad-hoc provider fallback, preset live claim, missing preflight, model/effort substitution, secret output and blocker bypass; each must fail. + +**Verification:** Final Verification commands 2-5 pass exactly: focused contract/manifest integration, every CLI help form, both tracked fixture validations, and `make test-agent-comparison-benchmark`. + +### [API-4] Capture direct dev evidence or an exact external blocker + +**Problem:** current planning probes show missing Sonnet/GPT routes and agy quota block, so local fake tests cannot satisfy live S06-S09. Treating them as ready would violate S10. + +**Solution:** After local PASS, verify caller binaries and secret-safe runtime input presence, run the direct-only preflight manifest, and retain the exact run/preflight evidence path. Completion requires all five direct cells ready with requested/effective route/model/effort and ordered stream terminal evidence. Exit 69 remains valid S10 blocker evidence but does not close S06-S09; record registration/quota/compatibility class and resume condition for official external-execution review. + +**Modified Files and Checklist:** + +- [ ] Record exact external command/output and durable evidence path in `agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/CODE_REVIEW-cloud-G10.md`. +- [ ] Verify tracked diff contains no raw credential, private endpoint, prompt/tool output or private config path. + +**Test Strategy:** One authorized direct preflight only; no hybrid preset or scored run. Do not retry blindly or substitute models/efforts. If environment is unavailable, stop with exact blocker rather than changing code to bypass it. + +**Verification:** direct preflight exits 0 with five ready cells for completion, or exact exit-69 evidence and resume condition are recorded without a PASS claim. + +## Dependencies and Execution Order + +- `07+06_claude_iop`, `08+06_agy_iop`, and `09+06_codex_iop` must each have exactly one active/archive `complete.log` before implementation. All are missing during planning. +- The directory name encodes exactly predecessors 07,08,09. Completion of 06 is transitively required by those predecessors and is not an extra direct dependency. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_comparison_benchmark.py` | API-1, API-2 | +| `scripts/agent_benchmark/__init__.py` | API-1 | +| `scripts/agent_benchmark/attempts.py` | API-1, API-2 | +| `scripts/agent_benchmark/attempts_test.py` | API-1, API-2 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-2 | +| `scripts/agent_benchmark/skill_contract_test.py` | API-3 | +| `scripts/agent_benchmark/manifest_test.py` | API-3 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 | +| `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` | API-3 | +| `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` | API-3, API-4 | +| `agent-task/m-agent-comparison-benchmark-pipeline/10+07,08,09_connectivity_runner/CODE_REVIEW-cloud-G10.md` | API-1, API-2, API-3, API-4 evidence | + +## Final Verification + +1. `python3 -c 'from pathlib import Path; g="m-agent-comparison-benchmark-pipeline"; ids=("07","08","09"); a=Path("agent-task")/g; r=Path("agent-task/archive"); f={i:sorted([*a.glob(f"{i}_*/complete.log"),*a.glob(f"{i}+*/complete.log"),*r.glob(f"*/*/{g}/{i}_*/complete.log"),*r.glob(f"*/*/{g}/{i}+*/complete.log")],key=str) for i in ids}; bad={i:[str(x) for x in p] for i,p in f.items() if len(p)!=1}; assert not bad,bad; print("\n".join(str(f[i][0]) for i in ids))'` +2. `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test scripts.agent_benchmark.manifest_test -v` +3. `python3 scripts/agent_comparison_benchmark.py --help && python3 scripts/agent_comparison_benchmark.py validate --help && python3 scripts/agent_comparison_benchmark.py preflight --help && python3 scripts/agent_comparison_benchmark.py run --help && python3 scripts/agent_comparison_benchmark.py resume --help && python3 scripts/agent_comparison_benchmark.py status --help` + - Expected: exactly validate/preflight/run/resume/status with documented required flags; no public prepare/report command. +4. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json && python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +5. `make test-agent-comparison-benchmark` + - Expected: fresh full benchmark suite and example validation pass with no network. +6. `command -v claude && claude --version && command -v agy && agy --version && command -v codex && codex --version` +7. `go run ./scripts/inventory-query --env dev --model claude-sonnet-5` +8. `go run ./scripts/inventory-query --env dev --model gemini-3.6-flash` +9. `go run ./scripts/inventory-query --env dev --model gpt-5.6-luna` + - Expected for completion: every command returns one active exact route. Current planning result is missing/match/missing; preserve each nonzero command as a registration blocker. +10. `python3 -c 'import os; pairs=(("IOP_BENCH_CLAUDE_BASE_URL","IOP_BENCH_CLAUDE_SECRET_ENV"),("IOP_BENCH_AGY_BASE_URL","IOP_BENCH_AGY_SECRET_ENV"),("IOP_BENCH_CODEX_BASE_URL","IOP_BENCH_CODEX_SECRET_ENV")); missing=[]; [(missing.append(base) if not os.environ.get(base) else None, missing.append(ref) if not os.environ.get(ref) else (missing.append(ref+"->secret") if not os.environ.get(os.environ[ref]) else None)) for base,ref in pairs]; assert not missing,missing; print("ok: caller runtime inputs present")'` + - Expected: prints only the fixed success string, never secret/env values. +11. `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` + - Completion expected: exit 0, five ready direct cells, exact requested/effective binding evidence and redacted path. Exit 69 must be recorded as S10 blocker and cannot close S06-S09. +12. `if [ -e build/r14-remote-anthropic_handler.go ]; then echo 'BLOCKED: ignored build artifact shadows iop/build' >&2; exit 69; fi; go test ./... -count=1` +13. `git diff --check` + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.