diff --git a/agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md b/agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md new file mode 100644 index 00000000..43f8f4b6 --- /dev/null +++ b/agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md @@ -0,0 +1,99 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run the plan verification, paste actual output, and leave this active pair in place. Do not archive files, write `complete.log`, or classify the next state. + +## Overview + +date=2026-08-14 +task=m-thin-agent-model-comparison-benchmark, plan=1, tag=TEST + +## Archive Evidence Snapshot + +- Replaced unstarted pair: `plan_local_G08_0.log`, `code_review_cloud_G08_0.log`; no prior verdict. +- Replan closes the missing authenticated catalog call, exact caller/render procedure, and evidence write boundary while preserving the benchmark scope. + +## For the Review Agent + +Rerun applicable deterministic checks and inspect immutable external evidence. Append the official verdict only after implementation is submitted. On PASS, archive this pair with suffix `1`, write `complete.log` preserving the first-line metadata, and move the task directory under the dated archive. Roadmap aggregation is a later `sync-milestone-workstate` action. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| TEST-1 Consume the Immutable Nine-Row Matrix | [ ] | +| TEST-2 Render, Score Once, and Conclude | [ ] | + +## Implementation Checklist + +- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace. +- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery. +- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero. +- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport. +- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic. +- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality. +- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** Implementing agents must not modify this checklist. + +- [ ] Append one verdict with verified routing signals. +- [ ] Verify verdict, dimensions, and finding severities agree. +- [ ] Rerun required deterministic verification and inspect the nine immutable attempt ledgers/streams. +- [ ] Record evidence, root cause, selected fix, files/tests, and acceptance commands for each Required/Suggested finding. +- [ ] Archive this file to `code_review_cloud_G08_1.log` and the plan to `plan_local_G08_1.log`. +- [ ] Verify the managed `.gitignore` block and artifact visibility. +- [ ] On PASS, write `complete.log`, preserve milestone metadata, move the task directory to the dated archive, and update this checklist there. +- [ ] On WARN/FAIL, create only the next state required by the code-review skill and do not write `complete.log`. + +## Deviations from Plan + +_Replace with actual deviations or `None`._ + +## Key Design Decisions + +_Replace with actual implementation decisions._ + +## Reviewer Checkpoints + +- Confirm the authenticated catalog body check passed before any producer workspace existed. +- Confirm the exact expanded command for each of nine rows, one ledger/stream per row, and no hidden caller retry or session continuation. +- Confirm no product/config/script/manifest/state-store change entered the worktree. +- Confirm route facts were absent from every opaque scoring directory until scores were frozen. +- Confirm usage was caller-provided or `미제공`; failures/unscorable artifacts were not converted to zero. +- Confirm each scorable source has two one-shot renders, direct anchor evidence, and correct arithmetic. + +## Verification Results + +### External gate and producer attempts + +Paste the redacted authenticated gate output, exact expanded commands, sole exit status, and each `attempt.txt`. Do not paste credentials or raw sensitive provider payloads. + +_Replace with actual output._ + +### Local deterministic checks + +Run the exact final checks from `PLAN-local-G08.md` and paste stdout/stderr plus exit statuses. + +_Replace with actual output._ + +### Manual scorecard review + +Record reviewer arithmetic, anchor/evidence, opaque-blinding, render-count, usage, and bounded-conclusion findings. + +_Replace with actual findings._ + +--- + +## Section Ownership + +| Section | Owner | Note | +|---|---|---| +| Header, overview, archive snapshot, reviewer instructions | Fixed | Implementer must not modify | +| Implementation item/checklist status | Implementer | Check only after actual completion | +| Review-Only Checklist | Review agent | Implementer must not modify | +| Deviations, decisions, verification results | Implementer, then reviewer | Replace placeholders with actual evidence | +| Code Review Result | Review agent | Appended only during official review | diff --git a/agent-task/m-thin-agent-model-comparison-benchmark/PLAN-local-G08.md b/agent-task/m-thin-agent-model-comparison-benchmark/PLAN-local-G08.md new file mode 100644 index 00000000..25092816 --- /dev/null +++ b/agent-task/m-thin-agent-model-comparison-benchmark/PLAN-local-G08.md @@ -0,0 +1,365 @@ + + +# Plan - Executable Thin Agent Single-Attempt Comparison + +## For the Implementing Agent + +Fill the implementation-owned sections in `CODE_REVIEW-cloud-G08.md`. Run the commands exactly once per matrix row, paste actual output, and leave the active pair in place for official review. If a pre-attempt gate fails, stop before creating producer workspaces. If a producer command starts, its exit, timeout, missing artifact, or malformed terminal is that row's final result; never rerun, resume, or replace it. + +## Background + +The first plan correctly bounded the benchmark but did not provide an executable authenticated catalog check, exact caller invocations, or exact render commands, and omitted required evidence paths from its write boundary. This replan closes those gaps before any producer attempt is consumed. It keeps the same nine qualified routes, fixed prompt, one-attempt rule, blind scorecard, and documentation-only result. + +## Archive Evidence Snapshot + +- Replaced unstarted pair: `agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log`, `agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log`. +- Prior verdict: none; the review file was an unfilled implementation stub. +- Preserved decisions: one atomic packet, no product/config/runner changes, one producer attempt per row, ignored raw evidence, opaque single-pass scoring. +- Corrected defects: catalog gate had no catalog request, caller/render steps were prose-only, and evidence files were outside `Modified Files Summary`. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `agent-ops/skills/common/router.md` +- `agent-ops/skills/common/plan/SKILL.md` +- `agent-ops/skills/common/code-review/SKILL.md` +- `agent-ops/skills/common/finalize-task-routing/SKILL.md` +- `agent-test/local/rules.md` +- `agent-test/dev/rules.md` +- `agent-test/dev/testing-smoke.md` +- `agent-test/inventory-agent.yaml` +- `agent-test/inventory-dev.yaml` +- `agent-test/dev/iop-thin-agent-model-comparison.md` +- `agent-test/dev/iop-benchmark-route-minimal-html-smoke.md` +- `agent-roadmap/current.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/thin-agent-model-comparison-benchmark.md` +- `agent-client/claude/README.md` +- `docs/dev-opencode-settings-guide.md` +- `opencode.json` +- `agent-task/responses_provider_bridge/PLAN-local-G08.md` +- archived pair listed above + +### SDD Criteria + +SDD is not required. The Milestone records this as a test-only observation of existing caller/product paths with no API, state-machine, retry, or schema change. + +### Verification Context + +No handoff was supplied. Repository-native dev rules select `toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, port `18083`, the existing SOPS principal token, and the managed CA at `build/dev-runtime/.secrets/credential-plane/ca.pem`. + +Fresh read-only preflight on 2026-08-14 confirmed clean branch `dev` at `16b7aba95a282b6c5d1e88d3b1849eaa1208b28a`, Claude Code `2.1.177`, OpenCode `1.18.3`, Codex `0.146.0`, open ports `18083`/`19093`, config/secret presence, caller flags, `/opt/homebrew/bin/gtimeout`, `jq`, `sops`, and managed CA presence. The exact catalog body check below remains a hard gate. The live checkout is intentionally the smoke-qualified runtime identity; do not deploy or change it in this benchmark. + +Local `/config/.local/bin/chromium` is the declared render executor. Each scorable source is rendered once at each fixed viewport. Credentials and raw provider payloads stay only on the remote runner or ignored evidence paths and never enter tracked output. + +### Test Coverage Gaps + +- Live availability has no deterministic unit substitute; the authenticated catalog gate and nine immutable attempts are the evidence. +- Caller-native usage shapes may differ. Record only an explicit usage field in the one producer stream; otherwise write `미제공`. +- Visual scoring is manual by design; deterministic selector, render-count, anchor, and arithmetic checks bound it. + +### Symbol References + +None; no product symbol changes. + +### Split Judgment + +Keep one atomic plan because execution records, opaque mapping, score rows, and conclusion must bind to the same immutable nine attempts. `large_indivisible_context=false`; explicit row commands and deterministic evidence reduce the packet. + +### Scope Rationale + +Writable tracked files are the comparison document and active review evidence. Writable ignored evidence is limited to `agent-test/runs/bench-lite-01/**`. Product source/config, caller installation/user config, roadmap, spec, contract, runner scripts, manifests, lifecycle stores, and route-smoke records are excluded. + +### Final Routing + +- evaluation_mode: `first-pass` for the complete replacement packet +- finalizer: `finalize-task-policy.sh pair local-fit false 1 0 false 1 2 1 2 2 official-review 1 2 1 2 2` +- closures: scope/context/verification/evidence/ownership/decision closed for build and review +- build: G08, `local-fit`, `PLAN-local-G08.md` +- review: G08, `official-review`, `CODE_REVIEW-cloud-G08.md` +- loop risk: `variant_product`; recovery signals false/0 + +## Implementation Checklist + +- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace. +- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery. +- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero. +- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport. +- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic. +- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality. +- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Consume the Immutable Nine-Row Matrix + +**Problem:** The archived plan required one attempt but its catalog gate never called `/v1/models`, and it described rather than specified the nine producer commands. + +**Solution:** In one remote `zsh` session, complete the gate below, create all `row-01` through `row-09` directories, save the fixed prompt once, and then run the following row mapping exactly once in document order: + +1. Claude / `claude-sonnet-5` +2. Claude / `gemini-3.6-flash` +3. OpenCode / `gemini-3.6-flash` +4. Claude / `gpt-5.6-luna` +5. Codex / `gpt-5.6-luna` +6. Claude / `gemini-hybrid` +7. OpenCode / `gemini-hybrid` +8. Claude / `gpt-hybrid` +9. Codex / `gpt-hybrid` + +For each row, write `attempt.txt` before invocation with row, caller/model, version, UTC start, `attempt_count=1`, `retry=0`, `resume=0`, and workspace-empty check. Redirect caller stdout/stderr to the row's sole `producer.jsonl`, append exit/end/elapsed to `attempt.txt`, and do not invoke that row again under any outcome. Use `/opt/homebrew/bin/gtimeout 900` as an outer bound; timeout exit 124 is a final failure. Claude uses `CLAUDE_CODE_MAX_RETRIES=0`, print stream-json, bare/no persistence, and the fixed model. OpenCode uses command-scoped `OPENCODE_CONFIG_CONTENT`, `run --pure --auto --format json --dir`, with no `--continue`/`--session`. Codex uses a fresh `mktemp -d` `CODEX_HOME` outside the repository, copied auth and the repository-proven `iop-direct.config.toml`, `exec --ephemeral --json --sandbox workspace-write --cd`, and no resume command; delete that temporary home immediately after the command. Direct rows use workspace `index.html`; preset rows extract the single terminal fenced HTML block only after the invocation ends. Missing/multiple blocks are `채점 불가`, not a second attempt. + +Use these exact caller forms, substituting only the fixed `row`, `model`, and caller from the numbered mapping above. Run each expanded block once, not a loop or repository script: + +```bash +# Claude rows 01, 02, 04, 06, 08 +row=row-01 model=claude-sonnet-5 +workspace="$run_root/$row/workspace" +started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)" +printf 'row=%s\ncaller=claude\nmodel=%s\ncaller_version=2.1.177\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt" +set +e +ANTHROPIC_BASE_URL="${base_url%/v1}" ANTHROPIC_AUTH_TOKEN="$iop_token" NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" CLAUDE_CODE_MAX_RETRIES=0 /opt/homebrew/bin/gtimeout 900 claude --print --output-format stream-json --bare --no-session-persistence --dangerously-skip-permissions --model "$model" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1 +status=$? +set -e +end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt" + +# OpenCode rows 03, 07 +row=row-03 model=gemini-3.6-flash +workspace="$run_root/$row/workspace" +export IOP_BENCH_TOKEN="$iop_token" +export OPENCODE_CONFIG_CONTENT="$(jq -cn --arg base "${base_url%/v1}/v1" --arg model "$model" '{permission:{read:"allow",write:"allow",edit:"allow",glob:"allow",bash:"allow"},provider:{iop:{npm:"@ai-sdk/openai-compatible",options:{baseURL:$base,apiKey:"{env:IOP_BENCH_TOKEN}"},models:{($model):{name:$model}}}}}')" +started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)" +printf 'row=%s\ncaller=opencode\nmodel=%s\ncaller_version=1.18.3\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt" +set +e +NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 opencode run --pure --auto --model "iop/$model" --agent build --format json --dir "$workspace" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1 +status=$? +set -e +end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt" +unset OPENCODE_CONFIG_CONTENT IOP_BENCH_TOKEN + +# Codex rows 05, 09 +row=row-05 model=gpt-5.6-luna +workspace="$run_root/$row/workspace"; codex_home="$(mktemp -d)" +mkdir -p "$codex_home"; cp /Users/toki/.codex/auth.json "$codex_home/auth.json"; cp /Users/toki/.codex/iop-direct.config.toml "$codex_home/iop-direct.config.toml" +started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)" +printf 'row=%s\ncaller=codex\nmodel=%s\ncaller_version=0.146.0\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt" +set +e +CODEX_HOME="$codex_home" IOP_CODEX_API_KEY="$iop_token" CODEX_CA_CERTIFICATE="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 codex exec --ephemeral --json --sandbox workspace-write --skip-git-repo-check --cd "$workspace" --profile iop-direct --model "$model" --output-last-message "$run_root/$row/terminal.txt" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1 +status=$? +set -e +end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt" +rm -rf "$codex_home" +``` + +Before row 01, run `for n in {01..09}; do mkdir -p "$run_root/row-$n/workspace"; test -z "$(find "$run_root/row-$n/workspace" -mindepth 1 -maxdepth 1 -print -quit)"; done`. Before each later row, expand a fresh caller block with its fixed tuple and verify that its `attempt.txt` and `producer.jsonl` do not exist. After row 09, unset `iop_token`. From the current checkout, transfer once with `rsync -a toki@toki-labs.com:/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01/ agent-test/runs/bench-lite-01/`, then perform source extraction, opaque assignment, rendering, and tracked result editing locally. + +For direct rows, accept `workspace/index.html` only when the row terminal contains `BENCH_LITE_01_DONE` exactly once. For preset rows, first extract the caller's single terminal text into `terminal.txt` (`jq -r 'select(.type == "result") | .result // empty'` for Claude, the final text-part event selected from the OpenCode JSONL shape observed in that sole stream, and Codex's `--output-last-message` file). Then run this one-shot strict extractor locally for each preset terminal; it succeeds only for exactly one final `html` fence and writes the bytes between fences without modifying them: + +```bash +ruby -e 's=File.binread(ARGV[0]); m=s.scan(/```html\r?\n(.*?)\r?\n```/m); abort("expected exactly one html fence") unless m.length==1; File.binwrite(ARGV[1],m[0][0])' terminal.txt index.html +``` + +If the OpenCode terminal event shape cannot be selected unambiguously from its one `producer.jsonl`, record the row as `채점 불가`; do not infer text from intermediate tool events and do not rerun it. + +After all rows, create the shuffled bijection exactly once and never regenerate it: + +```bash +test ! -e "$run_root/opaque-map.txt" +ruby -e 'rows=(1..9).map { |n| format("row-%02d",n) }; ids=(1..9).map { |n| format("E%02d",n) }.shuffle; File.write(ARGV[0],rows.zip(ids).map { |r,i| "#{r} #{i}\n" }.join)' "$run_root/opaque-map.txt" +``` + +For every mapped direct row with a valid marker/source, run `mkdir -p "$run_root/$opaque_id" && cp "$run_root/$row/workspace/index.html" "$run_root/$opaque_id/index.html"`. For every mapped preset row with an unambiguous terminal, run the strict extractor below with `"$run_root/$row/terminal.txt"` and `"$run_root/$opaque_id/index.html"`. Then record only `sha256=` in `"$run_root/$opaque_id/source.txt"` using `shasum -a 256`; never put row, caller, route, model, time, or usage in an `E*` directory. Keep the mapping closed until every score/evidence block is frozen. + +**Modified Files and Checklist:** + +- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: replace result placeholders with immutable row evidence. +- [ ] `agent-test/runs/bench-lite-01/prompt.txt`: exact fixed prompt copied from the tracked document before attempts. +- [ ] `agent-test/runs/bench-lite-01/catalog.json`: authenticated catalog body with credentials absent. +- [ ] `agent-test/runs/bench-lite-01/runtime.txt`: redacted gate identity/output. +- [ ] `agent-test/runs/bench-lite-01/row-01/attempt.txt` through `row-09/attempt.txt`: one immutable attempt ledger each. +- [ ] `agent-test/runs/bench-lite-01/row-01/producer.jsonl` through `row-09/producer.jsonl`: one caller stream each. +- [ ] `agent-test/runs/bench-lite-01/opaque-map.txt`: post-attempt row/opaque bijection. +- [ ] `agent-test/runs/bench-lite-01/E01/index.html` through `E09/index.html`: exact source only for scorable rows. + +**Test Strategy:** No test code or common runner. The nine explicit caller commands are the measured behavior. The per-row ledger and unique stream/source paths prove single invocation without creating lifecycle automation. + +**Verification:** Run the gate and matrix commands in `Final Verification`, then the local evidence checks. Expected: gate 200 with all five ids before workspace creation, nine attempt ledgers/streams with `attempt_count=1`, and no retry/resume/recovery marker. + +### [TEST-2] Render, Score Once, and Conclude + +**Problem:** The archived plan had no runnable fixed-viewport render command and did not enumerate render/evidence paths in its write boundary. + +**Solution:** For every `E*/index.html` that exists, run Chromium exactly once per viewport with a fresh temporary profile outside the repository, `--headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw`, `--window-size=1440,900` to `desktop.png`, then `--window-size=390,844` to `mobile.png`. Record the two exact commands and exit codes in `render.txt`; do not repeat a failed render. Score only from the opaque directory's source and images. Fill A from exact selectors, B-D using only 0/1/3/5 anchors, record direct evidence and one reason per deduction, and verify `A+B+C+D`. Join route mapping only after every score/evidence block is frozen. Write the bounded conclusion without zero-substituting failures, unscorable sources, or missing usage. + +**Modified Files and Checklist:** + +- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: score table, evidence blocks, correction notes if any, and bounded conclusion. +- [ ] `agent-test/runs/bench-lite-01/E01/desktop.png` through `E09/desktop.png`: one desktop render for each scorable source. +- [ ] `agent-test/runs/bench-lite-01/E01/mobile.png` through `E09/mobile.png`: one mobile render for each scorable source. +- [ ] `agent-test/runs/bench-lite-01/E01/render.txt` through `E09/render.txt`: exact two commands and outcomes for each scorable source. + +**Test Strategy:** No automated judge or browser pass/fail gate. Exact source, two immutable renders, locked anchors, and reviewer arithmetic provide the required one-pass evidence. + +**Verification:** Run the render loop once and final checks below. Expected: every scorable ID has one source, two images, one two-entry render ledger, evidence-backed anchors, correct total, and no route/model/time/usage in opaque evidence. + +## Modified Files Summary + +| File | Items | +|---|---| +| `agent-test/dev/iop-thin-agent-model-comparison.md` | TEST-1, TEST-2 | +| `agent-test/runs/bench-lite-01/prompt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/catalog.json` | TEST-1 | +| `agent-test/runs/bench-lite-01/runtime.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/opaque-map.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-01/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-01/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-01/workspace/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-02/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-02/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-02/workspace/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-03/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-03/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-03/workspace/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-04/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-04/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-04/workspace/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-05/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-05/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-05/terminal.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-05/workspace/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-06/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-06/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-06/terminal.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-07/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-07/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-07/terminal.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-08/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-08/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-08/terminal.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-09/attempt.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-09/producer.jsonl` | TEST-1 | +| `agent-test/runs/bench-lite-01/row-09/terminal.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E01/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E01/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E01/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E01/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E01/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E02/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E02/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E02/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E02/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E02/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E03/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E03/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E03/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E03/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E03/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E04/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E04/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E04/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E04/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E04/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E05/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E05/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E05/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E05/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E05/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E06/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E06/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E06/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E06/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E06/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E07/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E07/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E07/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E07/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E07/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E08/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E08/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E08/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E08/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E08/render.txt` | TEST-2 | +| `agent-test/runs/bench-lite-01/E09/index.html` | TEST-1 | +| `agent-test/runs/bench-lite-01/E09/source.txt` | TEST-1 | +| `agent-test/runs/bench-lite-01/E09/desktop.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E09/mobile.png` | TEST-2 | +| `agent-test/runs/bench-lite-01/E09/render.txt` | TEST-2 | +| `agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md` | TEST-1, TEST-2 | + +## Final Verification + +Before any producer workspace exists, run on the dev runner in one shell. This command makes the authenticated request, preserves only the credential-free body, and proves all fixed model IDs: + +```bash +set -euo pipefail +cd /Users/toki/agent-work/iop-dev +run_root=/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01 +test ! -e "$run_root" +export SOPS_AGE_KEY_FILE=/Users/toki/.config/sops/age/keys.txt +secret=/Users/toki/.config/iop/secrets/dev-openai-toki.sops.yaml +base_url="$(/opt/homebrew/bin/sops -d --extract '["base_url"]' "$secret")" +iop_token="$(/opt/homebrew/bin/sops -d --extract '["tokens"]["toki-dev-cline"]' "$secret")" +test -n "$iop_token" +tmp_catalog="$(mktemp)" +http_code="$(curl --cacert build/dev-runtime/.secrets/credential-plane/ca.pem -sS -o "$tmp_catalog" -w '%{http_code}' -H "Authorization: Bearer $iop_token" "$base_url/v1/models")" +test "$http_code" = 200 +for model in claude-sonnet-5 gemini-3.6-flash gpt-5.6-luna gemini-hybrid gpt-hybrid; do jq -e --arg model "$model" '.data[] | select(.id == $model)' "$tmp_catalog" >/dev/null; done +test -z "$(git status --short)" +test "$(git branch --show-current)" = dev +test "$(git rev-parse HEAD)" = 16b7aba95a282b6c5d1e88d3b1849eaa1208b28a +test "$(claude --version | head -1)" = '2.1.177 (Claude Code)' +test "$(opencode --version)" = '1.18.3' +codex --version | rg -x 'codex-cli 0\.146\.0' +nc -z 127.0.0.1 18083 +nc -z 127.0.0.1 19093 +mkdir -p "$run_root" +mv "$tmp_catalog" "$run_root/catalog.json" +cp /dev/null "$run_root/runtime.txt" +printf 'branch=dev\nhead=%s\nclaude=2.1.177\nopencode=1.18.3\ncodex=0.146.0\nports=18083,19093\ncatalog_http=200\n' "$(git rev-parse HEAD)" > "$run_root/runtime.txt" +unset iop_token +``` + +Copy the exact fixed prompt block to `prompt.txt`, create `row-01` through `row-09` before starting row 01, and execute the nine commands using the caller-specific forms fixed in TEST-1. The implementation evidence must paste each expanded command with secrets replaced by ``, its sole exit code, and the corresponding `attempt.txt`; this is required because no shared benchmark script may be added. + +For each scorable opaque ID, render locally with this block exactly once (replace `E01` with that ID; the block records both attempted commands and statuses and never retries): + +```bash +opaque_id=E01; opaque_dir="$PWD/agent-test/runs/bench-lite-01/$opaque_id"; render_log="$opaque_dir/render.txt" +test ! -e "$render_log"; : > "$render_log" +profile_desktop="$(mktemp -d)" +printf 'viewport=1440x900\n' >> "$render_log" +set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_desktop" --window-size=1440,900 --screenshot="$opaque_dir/desktop.png" "file://$opaque_dir/index.html"; desktop_status=$?; set -e +printf 'exit_status=%s\n' "$desktop_status" >> "$render_log" +profile_mobile="$(mktemp -d)" +printf 'viewport=390x844\n' >> "$render_log" +set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_mobile" --window-size=390,844 --screenshot="$opaque_dir/mobile.png" "file://$opaque_dir/index.html"; mobile_status=$?; set -e +printf 'exit_status=%s\n' "$mobile_status" >> "$render_log" +``` + +Run fresh final checks: + +```bash +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name attempt.txt -type f | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name producer.jsonl -type f | wc -l)" -eq 9 +test "$(rg -l '^attempt_count=1$' agent-test/runs/bench-lite-01/row-*/attempt.txt | wc -l)" -eq 9 +test "$(cut -d' ' -f1 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9 +test "$(cut -d' ' -f2 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name desktop.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name mobile.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name render.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name source.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" +! rg -n '미실행|미측정|미확인|미부여-[0-9]|미채점' agent-test/dev/iop-thin-agent-model-comparison.md +! rg -n '(retry|resume|recovery)[[:space:]]*[:=][[:space:]]*(true|yes|[1-9])' agent-test/runs/bench-lite-01 +! rg -n --hidden '(sk-|Bearer [A-Za-z0-9._-]{16,}|api[_-]?key[[:space:]]*[:=][[:space:]]*[A-Za-z0-9._-]{16,})' agent-test/dev/iop-thin-agent-model-comparison.md agent-test/runs/bench-lite-01 +! rg -n 'Claude|OpenCode|Codex|claude-sonnet|gemini|gpt|hybrid|경과|usage' agent-test/runs/bench-lite-01/E0* +git diff --check -- agent-test/dev/iop-thin-agent-model-comparison.md agent-task/m-thin-agent-model-comparison-benchmark +git diff --name-only -- . ':(exclude)agent-test/dev/iop-thin-agent-model-comparison.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/**' +``` + +The last command must print nothing. Reviewer inspection must also prove: gate preceded workspace creation; the nine expanded producer commands match TEST-1 and each ran once; caller usage is explicit or `미제공`; every scorable ID has one source/two renders/two-entry render ledger; route facts were unavailable during scoring; A-D evidence uses locked anchors and totals are correct; the conclusion excludes failures/unscorable rows and avoids statistical generalization. + +After completing all work, fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`. diff --git a/agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log b/agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log new file mode 100644 index 00000000..89c8f2e1 --- /dev/null +++ b/agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log @@ -0,0 +1,116 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> Complete the implementation-owned sections, run the plan verification, paste actual output, leave active files in place, and report ready for review. If blocked, record exact evidence and resume conditions only. Do not ask the user, create control-plane stop files, classify the next state, archive files, or write `complete.log`. + +## Overview + +date=2026-08-14 +task=m-thin-agent-model-comparison-benchmark, plan=0, tag=TEST + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** Compare the implementation with source and rerun applicable verification. Append the verdict and routing signals, archive the pair using suffix `0`, and on PASS preserve `milestone-task` metadata in `complete.log` before moving the task directory. Roadmap evaluation belongs to `sync-milestone-workstate`. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| TEST-1 Consume the Single-Attempt Matrix | [ ] | +| TEST-2 Score Once and Conclude Within Bounds | [ ] | + +## Implementation Checklist + +- [ ] Execute the nine-row matrix exactly once from empty workspaces and preserve one producer record per row with no retry/resume/recovery. +- [ ] Replace the result table placeholders with success/failure, elapsed time, caller-provided usage or `미제공`, exact source SHA/terminal evidence, artifact path, opaque ID, and a short observation. +- [ ] Render each scorable exact source once at 1440x900 and 390x844, score it once while route/model/time/usage are hidden, and record anchor-backed evidence and arithmetic totals. +- [ ] Write the bounded conclusion using only scorable successes, keeping failures, unscorable artifacts, and missing usage separate from zero scores. +- [ ] Run the final no-automation, attempt-count, rubric, arithmetic, and secret-safety verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** Implementing agents must not modify this checklist. + +- [ ] Append one `PASS`, `WARN`, or `FAIL` verdict plus `review_rework_count` and `evidence_integrity_failure`. +- [ ] Verify verdict, dimensions, and finding severities agree. +- [ ] Rerun applicable verification and record fresh output. +- [ ] For each Required/Suggested finding, record evidence, exact root cause, one selected fix, affected files/tests, and acceptance commands. +- [ ] Archive this file to `code_review_cloud_G08_0.log` and the plan to `plan_local_G08_0.log`. +- [ ] Verify the Agent-Ops managed `.gitignore` block. +- [ ] On PASS, write `complete.log`, preserve first-line milestone metadata, move the task directory to the dated archive, and update this checklist there. +- [ ] On WARN/FAIL, create only the next state required by the code-review skill and do not write `complete.log`. + +## Deviations from Plan + +_Replace with actual deviations or `None`._ + +## Key Design Decisions + +_Replace with actual implementation decisions; do not restate fixed product decisions._ + +## Reviewer Checkpoints + +- Confirm the catalog gate passed before any producer workspace existed. +- Confirm exactly one immutable caller invocation per route and no hidden retry/session continuation. +- Confirm no product/config/script/manifest/state-store change entered the diff. +- Confirm route facts were hidden during scoring and opaque mapping was joined only afterward. +- Confirm failures, unscorable sources, and missing usage were not converted to zero. +- Confirm every score has direct source/render evidence, a locked anchor, and correct arithmetic. + +## Verification Results + +### External preflight + +Command: use the exact pre-attempt SSH command from `PLAN-local-G08.md`. + +_Paste actual stdout/stderr and exit status._ + +### Attempt and artifact counts + +Commands: + +```bash +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name attempt.txt -type f | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name producer.jsonl -type f | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" -le 9 +test "$(cut -d' ' -f1 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9 +``` + +_Paste actual stdout/stderr and exit statuses._ + +### Document, retry, and secret checks + +Commands: + +```bash +! rg -n '미실행|미측정|미확인|미부여-[0-9]' agent-test/dev/iop-thin-agent-model-comparison.md +! rg -n '(retry|resume|recovery)[[:space:]]*[:=][[:space:]]*(true|yes|[1-9])' agent-test/runs/bench-lite-01 +! rg -n --hidden '(sk-|Bearer [A-Za-z0-9._-]{16,}|api[_-]?key[[:space:]]*[:=][[:space:]]*[A-Za-z0-9._-]{16,})' agent-test/dev/iop-thin-agent-model-comparison.md agent-test/runs/bench-lite-01 +git diff --check -- agent-test/dev/iop-thin-agent-model-comparison.md +git diff --name-only -- . ':(exclude)agent-test/dev/iop-thin-agent-model-comparison.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/PLAN-local-G08.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md' +``` + +_Paste actual stdout/stderr and exit statuses._ + +### Manual scorecard review + +_Record reviewer arithmetic, anchor/evidence, opaque-blinding, render-count, usage, and bounded-conclusion findings._ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned placeholder and checklist item, then leave this active file in place. + +## Section Ownership + +| Section | Owner | Note | +|---|---|---| +| Header, Overview, Review instructions | Fixed | Implementer must not modify | +| Implementation completion/checklist status | Implementer | Check only after actual completion | +| Review-Only Checklist | Review agent | Implementer must not modify | +| Deviations, Key Design Decisions | Implementer | Replace placeholders with actual evidence | +| Reviewer Checkpoints | Fixed | Reviewer applies them | +| Verification Results | Implementer, then reviewer | Implementer records initial output; reviewer reruns applicable commands | +| Code Review Result | Review agent appends | Not part of this stub | diff --git a/agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log b/agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log new file mode 100644 index 00000000..0e6a33fa --- /dev/null +++ b/agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log @@ -0,0 +1,173 @@ + + +# Plan - Thin Agent Single-Attempt Comparison + +## For the Implementing Agent + +Filling implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is mandatory. Run the verification exactly as written, paste actual output, keep both active files in place, and report ready for review. Finalization belongs to the code-review skill. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields; do not ask the user, call user-input tools, create stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The nine caller/model/route combinations have already passed the separate route smoke. This task consumes exactly one producer attempt per combination, records only caller-provided operational facts, scores each scorable artifact once with the locked rubric, and writes a bounded comparison without adding a benchmark runner or product change. + +## Analysis + +### Files Read + +- `AGENTS.md` +- `agent-ops/rules/project/rules.md` +- `agent-ops/rules/common/rules-roadmap.md` +- `agent-ops/rules/common/rules-agent-spec.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `agent-test/local/rules.md` +- `agent-test/dev/rules.md` +- `agent-test/dev/testing-smoke.md` +- `agent-test/inventory-dev.yaml` +- `agent-test/inventory-agent.yaml` +- `agent-roadmap/current.md` +- `agent-roadmap/priority-queue.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/thin-agent-model-comparison-benchmark.md` +- `agent-test/dev/iop-thin-agent-model-comparison.md` +- `agent-test/dev/iop-benchmark-route-minimal-html-smoke.md` +- `agent-spec/index.md` +- `agent-spec/input/openai-compatible-surface.md` +- `agent-spec/runtime/edge-node-execution.md` +- `agent-spec/runtime/provider-pool-config-refresh.md` +- `agent-contract/index.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/gemini-compatible-api.md` +- `agent-contract/inner/execution-runtime.md` +- `scripts/e2e-single-request-claude.sh` +- `agent-client/claude/README.md` +- `docs/dev-opencode-settings-guide.md` +- `opencode.json` + +### SDD Criteria + +SDD is not required. The Milestone records this as a test-only observation of existing callers and product routes with no API, state-machine, retry, or schema change. + +### Verification Context + +No verification handoff was supplied. Repository-native rules select the dev runner `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, Edge config `build/dev-runtime/edge.yaml`, public caller endpoint port `18083`, and exact prompt/rubric in `agent-test/dev/iop-thin-agent-model-comparison.md`. + +External Verification Preflight observed on 2026-08-14: + +- Runner: Darwin/arm64; checkout branch `dev`, HEAD `16b7aba95a282b6c5d1e88d3b1849eaa1208b28a`, clean. +- Source sync: stale relative to preparation HEAD `3d10de0652ee909101afc596f394b1f5e44178a2`; do not deploy or mutate tracked runtime config for this benchmark. The task consumes the already smoke-qualified live runtime and records its identity before attempts. +- Login-shell callers: Claude Code `2.1.177` at `/opt/homebrew/bin/claude`, OpenCode `1.18.3` at `/Users/toki/.local/bin/opencode`, Codex `0.146.0` at `/opt/homebrew/bin/codex`. +- Required caller flags exist: Claude print/stream-json/no-session-persistence/bare; OpenCode run model/agent/format/dir; Codex exec model/cd/json/output-last-message. +- Runtime: `build/dev-runtime/edge.yaml`, Edge binary, ports `18083` and `19093`, SOPS binary, age key, and existing secret file are present. Config exposes direct models `claude-sonnet-5`, `gemini-3.6-flash`, `gpt-5.6-luna` and preset models `gemini-hybrid`, `gpt-hybrid`; preset ids are `preset-gemini-hybrid`, `preset-gpt-hybrid`. +- Current authenticated `/v1/models` probe returned HTTP 400. This is a hard pre-attempt gate: fix only command-scoped credential/base-path selection and obtain HTTP 200 with all five model ids before creating any producer workspace. Do not consume an attempt while the gate fails. +- Rendering: the remote runner has no browser tool; this checkout has `/config/.local/bin/chromium`. Copy exact source into ignored `agent-test/runs/bench-lite-01//index.html` and render locally once per viewport. +- Constraints: no secret/raw provider payload in tracked files, no global CA override, no tracked config change, no retry/resume/recovery, no replacement attempt, no new script/manifest/state store, and no automatic pass/fail browser gate. +- Confidence: high for scope and evidence rules; medium for live readiness until the catalog gate returns 200. + +### Test Coverage Gaps + +- There is no deterministic unit test for live caller/provider availability; the nine immutable producer records are the required evidence. +- Attempt count is verified from one workspace and one raw caller event file per matrix row, with no second invocation record. +- Score correctness is covered by selector/render evidence and arithmetic checks, not by an automated judge. + +### Symbol References + +None; no product symbols change. + +### Split Judgment + +Keep one atomic plan. The result table, opaque mapping, scorecard, and conclusion all depend on the same immutable set of nine producer attempts; splitting would allow a retry or route identity to drift between execution and scoring. The packet remains reducible to explicit row-level rules and deterministic evidence checks, so `large_indivisible_context=false`. + +### Scope Rationale + +Only `agent-test/dev/iop-thin-agent-model-comparison.md` and ignored per-run evidence are writable. Product source, config, caller installation, roadmap, specs, contracts, benchmark scripts, runners, manifests, lifecycle stores, and prior route-smoke evidence are excluded. A preflight failure is recorded as a blocker and must not be repaired by changing product/runtime scope in this task. + +### Final Routing + +- evaluation_mode: `first-pass` +- finalizer: `finalize-task-policy.sh`, mode `pair` +- closures: build/review scope, context, verification, evidence, ownership, and decision are closed +- build scores: scope 1, state 2, blast 1, evidence 2, verification 2 = G08; base/final route `local-fit`, lane `local` +- review scores: scope 1, state 2, blast 1, evidence 2, verification 2 = G08; route `official-review`, lane `cloud` +- large_indivisible_context: `false` +- positive loop-risk: `variant_product` (1) +- recovery: `review_rework_count=0`, `evidence_integrity_failure=false` +- capability gap: none +- canonical files: `PLAN-local-G08.md`, `CODE_REVIEW-cloud-G08.md` + +## Implementation Checklist + +- [ ] Execute the nine-row matrix exactly once from empty workspaces and preserve one producer record per row with no retry/resume/recovery. +- [ ] Replace the result table placeholders with success/failure, elapsed time, caller-provided usage or `미제공`, exact source SHA/terminal evidence, artifact path, opaque ID, and a short observation. +- [ ] Render each scorable exact source once at 1440x900 and 390x844, score it once while route/model/time/usage are hidden, and record anchor-backed evidence and arithmetic totals. +- [ ] Write the bounded conclusion using only scorable successes, keeping failures, unscorable artifacts, and missing usage separate from zero scores. +- [ ] Run the final no-automation, attempt-count, rubric, arithmetic, and secret-safety verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Consume the Single-Attempt Matrix + +**Problem:** `agent-test/dev/iop-thin-agent-model-comparison.md:30-40` contains nine `미실행` rows, while the Milestone requires exactly one producer attempt per row and forbids recovery. + +**Solution:** First make the authenticated catalog gate return HTTP 200 and confirm the five fixed model ids. Then create all nine empty workspaces before the first invocation, record a UTC start/end and caller version, and invoke each fixed pairing once in document order: Claude Code with `claude-sonnet-5`, `gemini-3.6-flash`, `gpt-5.6-luna`, `gemini-hybrid`, and `gpt-hybrid`; OpenCode with `gemini-3.6-flash` and `gemini-hybrid`; Codex with `gpt-5.6-luna` and `gpt-hybrid`. Use the exact prompt at line 11, caller-native JSON/stream output, `CLAUDE_CODE_MAX_RETRIES=0`, no OpenCode `--continue`/`--session`, and a fresh temporary `CODEX_HOME` with Responses wire and no session reuse. A nonzero exit, timeout, missing source, or missing terminal is the final row result; never invoke that row again. + +For direct rows, preserve the workspace `index.html`. For preset rows, extract only the exact HTML fenced block from the single terminal response because the Edge-private workspace is cleaned. Copy exact sources to deterministic ignored paths `agent-test/runs/bench-lite-01/E01/index.html` through `E09/index.html`; preserve caller JSON as `producer.jsonl` and timing/exit/usage metadata as `attempt.txt` in the same folder. Assign `E01`-`E09` only after all attempts, using a shuffled mapping held in ignored `agent-test/runs/bench-lite-01/opaque-map.txt`; do not expose route facts to the scoring view. + +**Modified Files and Checklist:** + +- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: fill the nine result rows from immutable evidence. +- [ ] `agent-test/runs/bench-lite-01/opaque-map.txt`: record route-to-opaque mapping after attempts. +- [ ] `agent-test/runs/bench-lite-01/E01/attempt.txt` through `E09/attempt.txt`: preserve one attempt record per row. + +**Test Strategy:** No test code. The live producer attempt is the behavior under measurement; mocks or reruns would invalidate it. + +**Verification:** Run the final verification commands below. Expect nine non-placeholder result rows, nine unique opaque ids, and exactly one `attempt.txt`/`producer.jsonl` per row. + +### [TEST-2] Score Once and Conclude Within Bounds + +**Problem:** `agent-test/dev/iop-thin-agent-model-comparison.md:60-78` is unscored and has no conclusion, but scoring must remain blind to route facts and cannot turn failures into zeros. + +**Solution:** For each scorable opaque source, run local Chromium exactly once at `1440x900` and once at `390x844`, saving `desktop.png` and `mobile.png` beside the source. Review only the opaque id, exact source, and those two renders. Fill A from the ten locked selectors, B-D only with 0/1/3/5 anchors, write one evidence block per scorable id, and calculate `A+B+C+D`. Do not reopen an evaluation except to correct arithmetic or transcription, in which case append the correction reason. After all scores are frozen, join the opaque mapping and write a short conclusion comparing only successful scorable rows; report success/time/usage separately and make no statistical or absolute-quality claim. + +**Modified Files and Checklist:** + +- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: fill score rows, evidence blocks, and bounded conclusion. +- [ ] `agent-test/runs/bench-lite-01/E01/desktop.png` through `E09/desktop.png`: keep one desktop render for each scorable source. +- [ ] `agent-test/runs/bench-lite-01/E01/mobile.png` through `E09/mobile.png`: keep one mobile render for each scorable source. + +**Test Strategy:** No automated judge or browser gate. Use source selectors, fixed viewport renders, locked anchors, and arithmetic validation; unscorable rows remain `채점 불가`. + +**Verification:** Run the final verification commands below. Expect every scorable row to have evidence-backed A-D values and a correct total; failed/unscorable/missing-usage values must never be numeric zero by substitution. + +## Modified Files Summary + +| File | Items | +|---|---| +| `agent-test/dev/iop-thin-agent-model-comparison.md` | TEST-1, TEST-2 | +| `agent-test/runs/bench-lite-01/opaque-map.txt` | TEST-1 | +| `agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md` | TEST-1, TEST-2 implementation evidence | + +## Final Verification + +Before any producer attempt, run this read-only gate on the dev runner and paste redacted output. It must show a clean checkout, the three expected caller versions, open ports, HTTP 200, and all five models; otherwise stop without creating a producer workspace: + +```bash +ssh -o BatchMode=yes toki@toki-labs.com 'zsh -lic '\''set -eu; cd /Users/toki/agent-work/iop-dev; git status --short; git branch --show-current; git rev-parse HEAD; claude --version; opencode --version; codex --version; nc -z 127.0.0.1 18083; nc -z 127.0.0.1 19093; test -f build/dev-runtime/edge.yaml; test -f /Users/toki/.config/iop/secrets/dev-openai-toki.sops.yaml'\''' +``` + +After the nine attempts and scoring, run fresh local checks (cached output is not applicable): + +```bash +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name attempt.txt -type f | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name producer.jsonl -type f | wc -l)" -eq 9 +test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)" -le 9 +test "$(cut -d' ' -f1 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9 +! rg -n '미실행|미측정|미확인|미부여-[0-9]' agent-test/dev/iop-thin-agent-model-comparison.md +! rg -n '(retry|resume|recovery)[[:space:]]*[:=][[:space:]]*(true|yes|[1-9])' agent-test/runs/bench-lite-01 +! rg -n --hidden '(sk-|Bearer [A-Za-z0-9._-]{16,}|api[_-]?key[[:space:]]*[:=][[:space:]]*[A-Za-z0-9._-]{16,})' agent-test/dev/iop-thin-agent-model-comparison.md agent-test/runs/bench-lite-01 +git diff --check -- agent-test/dev/iop-thin-agent-model-comparison.md +git diff --name-only -- . ':(exclude)agent-test/dev/iop-thin-agent-model-comparison.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/PLAN-local-G08.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md' +``` + +The last command must print nothing. Reviewer inspection must also confirm: all nine result rows have one attempt; usage is caller-provided or `미제공`; every scorable id has exactly one source and two renders; all A-D evidence uses locked anchors; totals are arithmetically correct; the conclusion excludes failed/unscorable rows from quality comparison and makes no statistical generalization. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`.