chore(epic): thin-run 준비 결과를 검증한다
This commit is contained in:
parent
e09aa66c3c
commit
02753c5225
4 changed files with 586 additions and 186 deletions
|
|
@ -1,22 +1,23 @@
|
|||
<!-- task=m-thin-agent-model-comparison-benchmark plan=1 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
<!-- task=m-thin-agent-model-comparison-benchmark plan=2 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run the plan verification, paste actual output, and leave this active pair in place. Do not archive files, write `complete.log`, or classify the next state.
|
||||
> **[IMPLEMENTING AGENT — READ FIRST]** Complete implementation-owned sections, paste actual output, and leave this pair active. Do not archive files, write `complete.log`, ask the user, or classify the next state.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-14
|
||||
task=m-thin-agent-model-comparison-benchmark, plan=1, tag=TEST
|
||||
task=m-thin-agent-model-comparison-benchmark, plan=2, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Replaced unstarted pair: `plan_local_G08_0.log`, `code_review_cloud_G08_0.log`; no prior verdict.
|
||||
- Replan closes the missing authenticated catalog call, exact caller/render procedure, and evidence write boundary while preserving the benchmark scope.
|
||||
- Pre-refine intent is checkpoint `e09aa66c3cdb829366463c10f8bc5f5801e3136e`.
|
||||
- Replaced unstarted refinement: `plan_local_G08_1.log`, `code_review_cloud_G08_1.log`; no verdict.
|
||||
- This replan fixes URL normalization, token lifetime, and Claude row-workspace binding without changing the benchmark scope.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
Rerun applicable deterministic checks and inspect immutable external evidence. Append the official verdict only after implementation is submitted. On PASS, archive this pair with suffix `1`, write `complete.log` preserving the first-line metadata, and move the task directory under the dated archive. Roadmap aggregation is a later `sync-milestone-workstate` action.
|
||||
Rerun applicable deterministic checks and inspect immutable evidence. Append an official verdict only after implementation is submitted. On PASS, archive this pair with suffix `2`, preserve first-line milestone metadata in `complete.log`, and move the task directory to the dated archive; roadmap aggregation remains a later runtime action.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
|
|
@ -27,25 +28,25 @@ Rerun applicable deterministic checks and inspect immutable external evidence. A
|
|||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace.
|
||||
- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero.
|
||||
- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport.
|
||||
- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality.
|
||||
- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Pass the authenticated catalog/runtime gate without creating a producer workspace.
|
||||
- [ ] Create nine empty row workspaces and execute each fixed caller/model tuple exactly once in its row workspace, with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence, using caller-provided usage or `미제공`.
|
||||
- [ ] After all attempts, create one shuffled opaque bijection and copy/extract each scorable exact source without route facts.
|
||||
- [ ] Render each scorable opaque source once at desktop and once at mobile, then score it once with locked anchors and direct evidence.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating operational facts from quality.
|
||||
- [ ] Run the final count, isolation, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Implementing agents must not modify this checklist.
|
||||
|
||||
- [ ] Append one verdict with verified routing signals.
|
||||
- [ ] Append one verdict with verified `review_rework_count` and `evidence_integrity_failure`.
|
||||
- [ ] Verify verdict, dimensions, and finding severities agree.
|
||||
- [ ] Rerun required deterministic verification and inspect the nine immutable attempt ledgers/streams.
|
||||
- [ ] Record evidence, root cause, selected fix, files/tests, and acceptance commands for each Required/Suggested finding.
|
||||
- [ ] Archive this file to `code_review_cloud_G08_1.log` and the plan to `plan_local_G08_1.log`.
|
||||
- [ ] Verify the managed `.gitignore` block and artifact visibility.
|
||||
- [ ] Rerun required checks and inspect the nine ledgers/streams plus score evidence.
|
||||
- [ ] For every Required/Suggested finding, record evidence, exact root cause, one selected fix, affected files/tests, and acceptance commands.
|
||||
- [ ] Archive this file to `code_review_cloud_G08_2.log` and the plan to `plan_local_G08_2.log`.
|
||||
- [ ] Verify the Agent-Ops `.gitignore` block.
|
||||
- [ ] On PASS, write `complete.log`, preserve milestone metadata, move the task directory to the dated archive, and update this checklist there.
|
||||
- [ ] On WARN/FAIL, create only the next state required by the code-review skill and do not write `complete.log`.
|
||||
|
||||
|
|
@ -59,18 +60,18 @@ _Replace with actual implementation decisions._
|
|||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the authenticated catalog body check passed before any producer workspace existed.
|
||||
- Confirm the exact expanded command for each of nine rows, one ledger/stream per row, and no hidden caller retry or session continuation.
|
||||
- Confirm URL normalization yields one `/v1/models`, the token remains available through row 09, and no producer workspace predates gate success.
|
||||
- Confirm all nine exact tuples ran once and each direct caller was bound to its declared empty row workspace.
|
||||
- Confirm no product/config/script/manifest/state-store change entered the worktree.
|
||||
- Confirm route facts were absent from every opaque scoring directory until scores were frozen.
|
||||
- Confirm usage was caller-provided or `미제공`; failures/unscorable artifacts were not converted to zero.
|
||||
- Confirm each scorable source has two one-shot renders, direct anchor evidence, and correct arithmetic.
|
||||
- Confirm route facts were absent from opaque scoring inputs until all scores froze.
|
||||
- Confirm usage is caller-provided or `미제공`, and failures/unscorable artifacts are not zero.
|
||||
- Confirm every scorable source has one SHA record, two one-shot renders, direct anchor evidence, and correct arithmetic.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### External gate and producer attempts
|
||||
|
||||
Paste the redacted authenticated gate output, exact expanded commands, sole exit status, and each `attempt.txt`. Do not paste credentials or raw sensitive provider payloads.
|
||||
Paste redacted gate output, each expanded command, sole exit status, and `attempt.txt`. Do not paste credentials or sensitive raw provider payloads.
|
||||
|
||||
_Replace with actual output._
|
||||
|
||||
|
|
@ -82,7 +83,7 @@ _Replace with actual output._
|
|||
|
||||
### Manual scorecard review
|
||||
|
||||
Record reviewer arithmetic, anchor/evidence, opaque-blinding, render-count, usage, and bounded-conclusion findings.
|
||||
Record reviewer arithmetic, anchor/evidence, opaque isolation, render count, usage handling, and bounded-conclusion findings.
|
||||
|
||||
_Replace with actual findings._
|
||||
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
<!-- task=m-thin-agent-model-comparison-benchmark plan=1 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
<!-- task=m-thin-agent-model-comparison-benchmark plan=2 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
|
||||
# Plan - Executable Thin Agent Single-Attempt Comparison
|
||||
# Plan - Run and Score the Thin-Agent Matrix Once
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Fill the implementation-owned sections in `CODE_REVIEW-cloud-G08.md`. Run the commands exactly once per matrix row, paste actual output, and leave the active pair in place for official review. If a pre-attempt gate fails, stop before creating producer workspaces. If a producer command starts, its exit, timeout, missing artifact, or malformed terminal is that row's final result; never rerun, resume, or replace it.
|
||||
Execute the pre-attempt gate and the nine rows exactly as specified. A started producer command consumes that row even on timeout, failure, or malformed output; never retry, resume, recover, or replace it. Fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`, then leave the active pair for official review.
|
||||
|
||||
## Background
|
||||
|
||||
The first plan correctly bounded the benchmark but did not provide an executable authenticated catalog check, exact caller invocations, or exact render commands, and omitted required evidence paths from its write boundary. This replan closes those gaps before any producer attempt is consumed. It keeps the same nine qualified routes, fixed prompt, one-attempt rule, blind scorecard, and documentation-only result.
|
||||
The checkpoint pair established the intended atomic benchmark and the first refinement added explicit evidence paths. That refinement remained non-executable: it could form `/v1/v1/models`, unset the token before producer work, and ran Claude outside the row workspace. This replacement preserves the original nine-route, single-attempt, blind-score intent while closing those command and evidence defects.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Replaced unstarted pair: `agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log`, `agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log`.
|
||||
- Prior verdict: none; the review file was an unfilled implementation stub.
|
||||
- Preserved decisions: one atomic packet, no product/config/runner changes, one producer attempt per row, ignored raw evidence, opaque single-pass scoring.
|
||||
- Corrected defects: catalog gate had no catalog request, caller/render steps were prose-only, and evidence files were outside `Modified Files Summary`.
|
||||
- Pre-refine intent: checkpoint `e09aa66c3cdb829366463c10f8bc5f5801e3136e`, with one atomic pair covering all four Milestone Task ids.
|
||||
- Replaced unstarted refinement: `plan_local_G08_1.log`, `code_review_cloud_G08_1.log`; no verdict or implementation evidence.
|
||||
- Earlier unstarted pair: `plan_local_G08_0.log`, `code_review_cloud_G08_0.log`; no verdict.
|
||||
- Preserved invariants: fixed prompt and nine tuples, empty row workspaces, one producer invocation per tuple, no benchmark runner/state machine, opaque single-pass scoring, failures and missing usage are not zero.
|
||||
|
||||
## Analysis
|
||||
|
||||
|
|
@ -24,44 +24,40 @@ The first plan correctly bounded the benchmark but did not provide an executable
|
|||
- `AGENTS.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `agent-ops/rules/common/rules-roadmap.md`
|
||||
- `agent-ops/rules/common/rules-agent-spec.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/common/router.md`
|
||||
- `agent-ops/skills/common/plan/SKILL.md`
|
||||
- `agent-ops/skills/common/code-review/SKILL.md`
|
||||
- `agent-ops/skills/common/refine-plans/SKILL.md`
|
||||
- `agent-ops/skills/common/finalize-task-routing/SKILL.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/dev/rules.md`
|
||||
- `agent-test/dev/testing-smoke.md`
|
||||
- `agent-test/inventory-agent.yaml`
|
||||
- `agent-test/inventory-dev.yaml`
|
||||
- `agent-test/dev/iop-thin-agent-model-comparison.md`
|
||||
- `agent-test/dev/iop-benchmark-route-minimal-html-smoke.md`
|
||||
- `agent-roadmap/current.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/thin-agent-model-comparison-benchmark.md`
|
||||
- `agent-client/claude/README.md`
|
||||
- `agent-spec/index.md`
|
||||
- `agent-contract/index.md`
|
||||
- `docs/dev-opencode-settings-guide.md`
|
||||
- `opencode.json`
|
||||
- `agent-task/responses_provider_bridge/PLAN-local-G08.md`
|
||||
- archived pair listed above
|
||||
- `scripts/e2e-hot-path-agents.sh`
|
||||
- `scripts/e2e-single-request-claude.sh`
|
||||
- the four task-local logs named above
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
SDD is not required. The Milestone records this as a test-only observation of existing caller/product paths with no API, state-machine, retry, or schema change.
|
||||
SDD is not required. The Milestone records a test-only observation of existing caller/product paths with no API, state-machine, retry, or schema change. This pair contributes exactly `single-attempt-matrix`, `minimal-result-table`, `single-pass-scorecard`, and `bounded-conclusion`.
|
||||
|
||||
### Verification Context
|
||||
|
||||
No handoff was supplied. Repository-native dev rules select `toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, port `18083`, the existing SOPS principal token, and the managed CA at `build/dev-runtime/.secrets/credential-plane/ca.pem`.
|
||||
No handoff was supplied. Repository-native dev rules select `toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, Edge port `18083`, the existing SOPS principal token, and the managed CA. Before creating `run_root`, normalize the decrypted URL once to `api_root` by removing one trailing slash and one trailing `/v1`; use `${api_root}/v1/models`, `ANTHROPIC_BASE_URL=$api_root`, and OpenCode `baseURL=${api_root}/v1`. Keep the shell-local token through row 09 and unset it immediately afterward.
|
||||
|
||||
Fresh read-only preflight on 2026-08-14 confirmed clean branch `dev` at `16b7aba95a282b6c5d1e88d3b1849eaa1208b28a`, Claude Code `2.1.177`, OpenCode `1.18.3`, Codex `0.146.0`, open ports `18083`/`19093`, config/secret presence, caller flags, `/opt/homebrew/bin/gtimeout`, `jq`, `sops`, and managed CA presence. The exact catalog body check below remains a hard gate. The live checkout is intentionally the smoke-qualified runtime identity; do not deploy or change it in this benchmark.
|
||||
|
||||
Local `/config/.local/bin/chromium` is the declared render executor. Each scorable source is rendered once at each fixed viewport. Credentials and raw provider payloads stay only on the remote runner or ignored evidence paths and never enter tracked output.
|
||||
External preflight must prove the smoke-qualified clean `dev` runtime identity, required caller versions/flags, ports, credential inputs, timeout tool, and authenticated five-model catalog. It does not deploy or mutate tracked runtime config. Local `/config/.local/bin/chromium` renders each scorable source once at each fixed viewport. Secrets and raw caller streams remain ignored evidence.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Live availability has no deterministic unit substitute; the authenticated catalog gate and nine immutable attempts are the evidence.
|
||||
- Caller-native usage shapes may differ. Record only an explicit usage field in the one producer stream; otherwise write `미제공`.
|
||||
- Visual scoring is manual by design; deterministic selector, render-count, anchor, and arithmetic checks bound it.
|
||||
- Live availability has no unit substitute; the authenticated catalog gate and immutable row ledgers are the evidence.
|
||||
- Caller usage schemas differ; use an explicit caller field from the sole stream or `미제공`.
|
||||
- Visual judgment is manual by design; opaque input, fixed renders, locked anchors, arithmetic, and direct evidence bound it.
|
||||
|
||||
### Symbol References
|
||||
|
||||
|
|
@ -69,142 +65,86 @@ None; no product symbol changes.
|
|||
|
||||
### Split Judgment
|
||||
|
||||
Keep one atomic plan because execution records, opaque mapping, score rows, and conclusion must bind to the same immutable nine attempts. `large_indivisible_context=false`; explicit row commands and deterministic evidence reduce the packet.
|
||||
Keep one atomic pair. The result table, post-attempt opaque bijection, scorecard, and conclusion must bind to the same immutable nine-attempt set; independent children could expose route identity early or weaken the no-retry boundary. The explicit row protocol keeps `large_indivisible_context=false`.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Writable tracked files are the comparison document and active review evidence. Writable ignored evidence is limited to `agent-test/runs/bench-lite-01/**`. Product source/config, caller installation/user config, roadmap, spec, contract, runner scripts, manifests, lifecycle stores, and route-smoke records are excluded.
|
||||
Writable tracked output is only the comparison document and active review evidence. Writable ignored evidence is the enumerated `agent-test/runs/bench-lite-01` files below. Product source/config, caller installation or user config, roadmap/spec/contract, scripts, runner/manifest/state-store, and prior smoke evidence are excluded.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- evaluation_mode: `first-pass` for the complete replacement packet
|
||||
- finalizer: `finalize-task-policy.sh pair local-fit false 1 0 false 1 2 1 2 2 official-review 1 2 1 2 2`
|
||||
- closures: scope/context/verification/evidence/ownership/decision closed for build and review
|
||||
- build: G08, `local-fit`, `PLAN-local-G08.md`
|
||||
- review: G08, `official-review`, `CODE_REVIEW-cloud-G08.md`
|
||||
- loop risk: `variant_product`; recovery signals false/0
|
||||
- evaluation_mode: `first-pass` for this semantic replacement
|
||||
- finalizer: `finalize-task-policy.sh`, mode `pair`
|
||||
- closures: scope/context/verification/evidence/ownership/decision are true for build and review
|
||||
- build scores: scope 1, state 2, blast 1, evidence 2, verification 2 = G08; `local-fit`; `PLAN-local-G08.md`
|
||||
- review scores: scope 1, state 2, blast 1, evidence 2, verification 2 = G08; `official-review`; `CODE_REVIEW-cloud-G08.md`
|
||||
- positive loop risk: `variant_product` (1); `large_indivisible_context=false`; rework 0; integrity failure false
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace.
|
||||
- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero.
|
||||
- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport.
|
||||
- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality.
|
||||
- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Pass the authenticated catalog/runtime gate without creating a producer workspace.
|
||||
- [ ] Create nine empty row workspaces and execute each fixed caller/model tuple exactly once in its row workspace, with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence, using caller-provided usage or `미제공`.
|
||||
- [ ] After all attempts, create one shuffled opaque bijection and copy/extract each scorable exact source without route facts.
|
||||
- [ ] Render each scorable opaque source once at desktop and once at mobile, then score it once with locked anchors and direct evidence.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating operational facts from quality.
|
||||
- [ ] Run the final count, isolation, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Consume the Immutable Nine-Row Matrix
|
||||
|
||||
**Problem:** The archived plan required one attempt but its catalog gate never called `/v1/models`, and it described rather than specified the nine producer commands.
|
||||
**Problem:** `agent-test/dev/iop-thin-agent-model-comparison.md:28-41` requires nine empty-workspace single attempts. The prior refinement appended `/v1/models` to an unnormalized URL, released the token before attempts, and invoked Claude from the repository root rather than the row workspace; those defects can fail the gate or invalidate that comparison.
|
||||
|
||||
**Solution:** In one remote `zsh` session, complete the gate below, create all `row-01` through `row-09` directories, save the fixed prompt once, and then run the following row mapping exactly once in document order:
|
||||
**Solution:** Run one remote `zsh` session. Complete the gate in Final Verification, keeping `api_root`, `iop_token`, `ca`, `run_root`, and the prompt alive. Create all row workspaces before row 01. Execute these tuples in order: row 01 Claude/`claude-sonnet-5`; 02 Claude/`gemini-3.6-flash`; 03 OpenCode/`gemini-3.6-flash`; 04 Claude/`gpt-5.6-luna`; 05 Codex/`gpt-5.6-luna`; 06 Claude/`gemini-hybrid`; 07 OpenCode/`gemini-hybrid`; 08 Claude/`gpt-hybrid`; 09 Codex/`gpt-hybrid`.
|
||||
|
||||
1. Claude / `claude-sonnet-5`
|
||||
2. Claude / `gemini-3.6-flash`
|
||||
3. OpenCode / `gemini-3.6-flash`
|
||||
4. Claude / `gpt-5.6-luna`
|
||||
5. Codex / `gpt-5.6-luna`
|
||||
6. Claude / `gemini-hybrid`
|
||||
7. OpenCode / `gemini-hybrid`
|
||||
8. Claude / `gpt-hybrid`
|
||||
9. Codex / `gpt-hybrid`
|
||||
|
||||
For each row, write `attempt.txt` before invocation with row, caller/model, version, UTC start, `attempt_count=1`, `retry=0`, `resume=0`, and workspace-empty check. Redirect caller stdout/stderr to the row's sole `producer.jsonl`, append exit/end/elapsed to `attempt.txt`, and do not invoke that row again under any outcome. Use `/opt/homebrew/bin/gtimeout 900` as an outer bound; timeout exit 124 is a final failure. Claude uses `CLAUDE_CODE_MAX_RETRIES=0`, print stream-json, bare/no persistence, and the fixed model. OpenCode uses command-scoped `OPENCODE_CONFIG_CONTENT`, `run --pure --auto --format json --dir`, with no `--continue`/`--session`. Codex uses a fresh `mktemp -d` `CODEX_HOME` outside the repository, copied auth and the repository-proven `iop-direct.config.toml`, `exec --ephemeral --json --sandbox workspace-write --cd`, and no resume command; delete that temporary home immediately after the command. Direct rows use workspace `index.html`; preset rows extract the single terminal fenced HTML block only after the invocation ends. Missing/multiple blocks are `채점 불가`, not a second attempt.
|
||||
|
||||
Use these exact caller forms, substituting only the fixed `row`, `model`, and caller from the numbered mapping above. Run each expanded block once, not a loop or repository script:
|
||||
Before each command, require missing `attempt.txt` and `producer.jsonl`, require an empty workspace, and write caller/model/version/start plus `attempt_count=1`, `retry=0`, `resume=0`. Wrap the sole invocation with `/opt/homebrew/bin/gtimeout 900`, redirect its only stream to `producer.jsonl`, and append end/elapsed/exit. Use the following exact caller forms, substituting only the tuple values:
|
||||
|
||||
```bash
|
||||
# Claude rows 01, 02, 04, 06, 08
|
||||
row=row-01 model=claude-sonnet-5
|
||||
workspace="$run_root/$row/workspace"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=claude\nmodel=%s\ncaller_version=2.1.177\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
ANTHROPIC_BASE_URL="${base_url%/v1}" ANTHROPIC_AUTH_TOKEN="$iop_token" NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" CLAUDE_CODE_MAX_RETRIES=0 /opt/homebrew/bin/gtimeout 900 claude --print --output-format stream-json --bare --no-session-persistence --dangerously-skip-permissions --model "$model" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
# Claude rows: invoke from the row workspace.
|
||||
( cd "$workspace" && ANTHROPIC_BASE_URL="$api_root" ANTHROPIC_AUTH_TOKEN="$iop_token" NODE_EXTRA_CA_CERTS="$ca" CLAUDE_CODE_MAX_RETRIES=0 /opt/homebrew/bin/gtimeout 900 claude --print --output-format stream-json --verbose --bare --no-session-persistence --dangerously-skip-permissions --model "$model" "$(cat "$run_root/prompt.txt")" ) > "$run_root/$row/producer.jsonl" 2>&1
|
||||
|
||||
# OpenCode rows 03, 07
|
||||
row=row-03 model=gemini-3.6-flash
|
||||
workspace="$run_root/$row/workspace"
|
||||
export IOP_BENCH_TOKEN="$iop_token"
|
||||
export OPENCODE_CONFIG_CONTENT="$(jq -cn --arg base "${base_url%/v1}/v1" --arg model "$model" '{permission:{read:"allow",write:"allow",edit:"allow",glob:"allow",bash:"allow"},provider:{iop:{npm:"@ai-sdk/openai-compatible",options:{baseURL:$base,apiKey:"{env:IOP_BENCH_TOKEN}"},models:{($model):{name:$model}}}}}')"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=opencode\nmodel=%s\ncaller_version=1.18.3\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 opencode run --pure --auto --model "iop/$model" --agent build --format json --dir "$workspace" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
unset OPENCODE_CONFIG_CONTENT IOP_BENCH_TOKEN
|
||||
# OpenCode rows: --dir and command-scoped config bind the row workspace.
|
||||
IOP_BENCH_TOKEN="$iop_token" OPENCODE_CONFIG_CONTENT="$(jq -cn --arg base "$api_root/v1" --arg model "$model" '{permission:{read:"allow",write:"allow",edit:"allow",glob:"allow",bash:"allow"},provider:{iop:{npm:"@ai-sdk/openai-compatible",options:{baseURL:$base,apiKey:"{env:IOP_BENCH_TOKEN}"},models:{($model):{name:$model}}}}}')" NODE_EXTRA_CA_CERTS="$ca" /opt/homebrew/bin/gtimeout 900 opencode run --pure --auto --model "iop/$model" --agent build --format json --dir "$workspace" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
|
||||
# Codex rows 05, 09
|
||||
row=row-05 model=gpt-5.6-luna
|
||||
workspace="$run_root/$row/workspace"; codex_home="$(mktemp -d)"
|
||||
mkdir -p "$codex_home"; cp /Users/toki/.codex/auth.json "$codex_home/auth.json"; cp /Users/toki/.codex/iop-direct.config.toml "$codex_home/iop-direct.config.toml"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=codex\nmodel=%s\ncaller_version=0.146.0\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
CODEX_HOME="$codex_home" IOP_CODEX_API_KEY="$iop_token" CODEX_CA_CERTIFICATE="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 codex exec --ephemeral --json --sandbox workspace-write --skip-git-repo-check --cd "$workspace" --profile iop-direct --model "$model" --output-last-message "$run_root/$row/terminal.txt" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
# Codex rows: isolated auth/config home and explicit row cwd.
|
||||
codex_home="$(mktemp -d)"
|
||||
cp /Users/toki/.codex/auth.json "$codex_home/auth.json"
|
||||
cp /Users/toki/.codex/iop-direct.config.toml "$codex_home/iop-direct.config.toml"
|
||||
CODEX_HOME="$codex_home" IOP_CODEX_API_KEY="$iop_token" CODEX_CA_CERTIFICATE="$ca" /opt/homebrew/bin/gtimeout 900 codex exec --ephemeral --json --sandbox workspace-write --skip-git-repo-check --cd "$workspace" --profile iop-direct --model "$model" --output-last-message "$run_root/$row/terminal.txt" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
rm -rf "$codex_home"
|
||||
```
|
||||
|
||||
Before row 01, run `for n in {01..09}; do mkdir -p "$run_root/row-$n/workspace"; test -z "$(find "$run_root/row-$n/workspace" -mindepth 1 -maxdepth 1 -print -quit)"; done`. Before each later row, expand a fresh caller block with its fixed tuple and verify that its `attempt.txt` and `producer.jsonl` do not exist. After row 09, unset `iop_token`. From the current checkout, transfer once with `rsync -a toki@toki-labs.com:/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01/ agent-test/runs/bench-lite-01/`, then perform source extraction, opaque assignment, rendering, and tracked result editing locally.
|
||||
Capture status with `set +e`/`status=$?`/`set -e` around each exact invocation; cleanup is not a retry. For direct rows 01-05, accept `workspace/index.html` only with exactly one terminal `BENCH_LITE_01_DONE`. For preset rows 06-09, extract terminal text once from that row's sole stream; if its caller-native terminal event is absent or ambiguous, mark `채점 불가`. A terminal is scorable only when this strict one-shot extractor finds exactly one `html` fence: `ruby -e 's=File.binread(ARGV[0]);m=s.scan(/```html\r?\n(.*?)\r?\n```/m);abort("expected exactly one html fence") unless m.length==1;File.binwrite(ARGV[1],m[0][0])' terminal.txt index.html`.
|
||||
|
||||
For direct rows, accept `workspace/index.html` only when the row terminal contains `BENCH_LITE_01_DONE` exactly once. For preset rows, first extract the caller's single terminal text into `terminal.txt` (`jq -r 'select(.type == "result") | .result // empty'` for Claude, the final text-part event selected from the OpenCode JSONL shape observed in that sole stream, and Codex's `--output-last-message` file). Then run this one-shot strict extractor locally for each preset terminal; it succeeds only for exactly one final `html` fence and writes the bytes between fences without modifying them:
|
||||
|
||||
```bash
|
||||
ruby -e 's=File.binread(ARGV[0]); m=s.scan(/```html\r?\n(.*?)\r?\n```/m); abort("expected exactly one html fence") unless m.length==1; File.binwrite(ARGV[1],m[0][0])' terminal.txt index.html
|
||||
```
|
||||
|
||||
If the OpenCode terminal event shape cannot be selected unambiguously from its one `producer.jsonl`, record the row as `채점 불가`; do not infer text from intermediate tool events and do not rerun it.
|
||||
|
||||
After all rows, create the shuffled bijection exactly once and never regenerate it:
|
||||
|
||||
```bash
|
||||
test ! -e "$run_root/opaque-map.txt"
|
||||
ruby -e 'rows=(1..9).map { |n| format("row-%02d",n) }; ids=(1..9).map { |n| format("E%02d",n) }.shuffle; File.write(ARGV[0],rows.zip(ids).map { |r,i| "#{r} #{i}\n" }.join)' "$run_root/opaque-map.txt"
|
||||
```
|
||||
|
||||
For every mapped direct row with a valid marker/source, run `mkdir -p "$run_root/$opaque_id" && cp "$run_root/$row/workspace/index.html" "$run_root/$opaque_id/index.html"`. For every mapped preset row with an unambiguous terminal, run the strict extractor below with `"$run_root/$row/terminal.txt"` and `"$run_root/$opaque_id/index.html"`. Then record only `sha256=<digest>` in `"$run_root/$opaque_id/source.txt"` using `shasum -a 256`; never put row, caller, route, model, time, or usage in an `E*` directory. Keep the mapping closed until every score/evidence block is frozen.
|
||||
After row 09, unset `iop_token`, transfer the ignored run directory once to this checkout, then create `opaque-map.txt` exactly once with a shuffled row/E01-E09 bijection. Copy only scorable exact HTML into the mapped `E*/index.html`; write only its SHA-256 to `source.txt`. Do not place route, caller, model, time, usage, or row id in any `E*` directory. Freeze all scores before joining the mapping.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: replace result placeholders with immutable row evidence.
|
||||
- [ ] `agent-test/runs/bench-lite-01/prompt.txt`: exact fixed prompt copied from the tracked document before attempts.
|
||||
- [ ] `agent-test/runs/bench-lite-01/catalog.json`: authenticated catalog body with credentials absent.
|
||||
- [ ] `agent-test/runs/bench-lite-01/runtime.txt`: redacted gate identity/output.
|
||||
- [ ] `agent-test/runs/bench-lite-01/row-01/attempt.txt` through `row-09/attempt.txt`: one immutable attempt ledger each.
|
||||
- [ ] `agent-test/runs/bench-lite-01/row-01/producer.jsonl` through `row-09/producer.jsonl`: one caller stream each.
|
||||
- [ ] `agent-test/runs/bench-lite-01/opaque-map.txt`: post-attempt row/opaque bijection.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/index.html` through `E09/index.html`: exact source only for scorable rows.
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: immutable result rows and observations.
|
||||
- [ ] `agent-test/runs/bench-lite-01/prompt.txt`: exact fixed prompt.
|
||||
- [ ] `agent-test/runs/bench-lite-01/catalog.json`: credential-free catalog body.
|
||||
- [ ] `agent-test/runs/bench-lite-01/runtime.txt`: redacted preflight identity.
|
||||
- [ ] `agent-test/runs/bench-lite-01/opaque-map.txt`: one post-attempt bijection.
|
||||
- [ ] Row and opaque evidence files enumerated in Modified Files Summary.
|
||||
|
||||
**Test Strategy:** No test code or common runner. The nine explicit caller commands are the measured behavior. The per-row ledger and unique stream/source paths prove single invocation without creating lifecycle automation.
|
||||
**Test Strategy:** No new test code or common runner. The measured behavior is the nine live caller invocations; immutable ledgers and sole streams are the regression evidence.
|
||||
|
||||
**Verification:** Run the gate and matrix commands in `Final Verification`, then the local evidence checks. Expected: gate 200 with all five ids before workspace creation, nine attempt ledgers/streams with `attempt_count=1`, and no retry/resume/recovery marker.
|
||||
**Verification:** Run Final Verification. Expect gate success before `run_root`, nine ledgers/streams, one attempt marker per row, exact empty-workspace binding, and no retry/resume/recovery.
|
||||
|
||||
### [TEST-2] Render, Score Once, and Conclude
|
||||
|
||||
**Problem:** The archived plan had no runnable fixed-viewport render command and did not enumerate render/evidence paths in its write boundary.
|
||||
**Problem:** `agent-test/dev/iop-thin-agent-model-comparison.md:43-91` requires source plus desktop/mobile evidence under a single blind evaluation pass, while its conclusion must not turn missing or failed data into zero.
|
||||
|
||||
**Solution:** For every `E*/index.html` that exists, run Chromium exactly once per viewport with a fresh temporary profile outside the repository, `--headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw`, `--window-size=1440,900` to `desktop.png`, then `--window-size=390,844` to `mobile.png`. Record the two exact commands and exit codes in `render.txt`; do not repeat a failed render. Score only from the opaque directory's source and images. Fill A from exact selectors, B-D using only 0/1/3/5 anchors, record direct evidence and one reason per deduction, and verify `A+B+C+D`. Join route mapping only after every score/evidence block is frozen. Write the bounded conclusion without zero-substituting failures, unscorable sources, or missing usage.
|
||||
**Solution:** For each scorable `E*/index.html`, create fresh temporary Chromium profiles outside the repository and invoke Chromium once with `--window-size=1440,900 --screenshot=desktop.png`, then once with `--window-size=390,844 --screenshot=mobile.png`. Record viewport and sole exit status in `render.txt`; a failed render is not repeated. Score only the opaque source/renders, use the fixed A selectors and B-D anchors 0/1/3/5, record direct evidence and each deduction, and verify A+B+C+D. Join route facts only after all score blocks are frozen. Compare only successful scorable results; keep failure, unscorable output, and `미제공` separate.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: score table, evidence blocks, correction notes if any, and bounded conclusion.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/desktop.png` through `E09/desktop.png`: one desktop render for each scorable source.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/mobile.png` through `E09/mobile.png`: one mobile render for each scorable source.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/render.txt` through `E09/render.txt`: exact two commands and outcomes for each scorable source.
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: score rows, evidence blocks, allowed correction notes, bounded conclusion.
|
||||
- [ ] Opaque render evidence enumerated in Modified Files Summary.
|
||||
|
||||
**Test Strategy:** No automated judge or browser pass/fail gate. Exact source, two immutable renders, locked anchors, and reviewer arithmetic provide the required one-pass evidence.
|
||||
**Test Strategy:** No automated judge/browser gate. Fixed source, two one-shot renders, locked anchors, reviewer inspection, and arithmetic are the required evidence.
|
||||
|
||||
**Verification:** Run the render loop once and final checks below. Expected: every scorable ID has one source, two images, one two-entry render ledger, evidence-backed anchors, correct total, and no route/model/time/usage in opaque evidence.
|
||||
**Verification:** Expect every scorable ID to have exactly one source, SHA record, two images, and one two-entry render ledger; every score has anchor/evidence and correct arithmetic.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
|
|
@ -243,6 +183,7 @@ For every mapped direct row with a valid marker/source, run `mkdir -p "$run_root
|
|||
| `agent-test/runs/bench-lite-01/row-09/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-09/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-09/terminal.txt` | TEST-1 |
|
||||
| `agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md` | TEST-1, TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E01/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E01/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E01/desktop.png` | TEST-2 |
|
||||
|
|
@ -288,78 +229,72 @@ For every mapped direct row with a valid marker/source, run `mkdir -p "$run_root
|
|||
| `agent-test/runs/bench-lite-01/E09/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E09/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E09/render.txt` | TEST-2 |
|
||||
| `agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md` | TEST-1, TEST-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Before any producer workspace exists, run on the dev runner in one shell. This command makes the authenticated request, preserves only the credential-free body, and proves all fixed model IDs:
|
||||
Before creating `run_root`, run in one remote `zsh` shell:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
cd /Users/toki/agent-work/iop-dev
|
||||
run_root=/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01
|
||||
run_root="$PWD/agent-test/runs/bench-lite-01"
|
||||
test ! -e "$run_root"
|
||||
export SOPS_AGE_KEY_FILE=/Users/toki/.config/sops/age/keys.txt
|
||||
ca="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem"
|
||||
secret=/Users/toki/.config/iop/secrets/dev-openai-toki.sops.yaml
|
||||
export SOPS_AGE_KEY_FILE=/Users/toki/.config/sops/age/keys.txt
|
||||
base_url="$(/opt/homebrew/bin/sops -d --extract '["base_url"]' "$secret")"
|
||||
api_root="${base_url%/}"; api_root="${api_root%/v1}"
|
||||
iop_token="$(/opt/homebrew/bin/sops -d --extract '["tokens"]["toki-dev-cline"]' "$secret")"
|
||||
test -n "$iop_token"
|
||||
test -n "$api_root"; test -n "$iop_token"; test -f "$ca"
|
||||
test -z "$(git status --short)"; test "$(git branch --show-current)" = dev
|
||||
test "$(git rev-parse HEAD)" = 16b7aba95a282b6c5d1e88d3b1849eaa1208b28a
|
||||
command -v /opt/homebrew/bin/gtimeout jq ruby claude opencode codex
|
||||
claude --help | rg -- '--print|--output-format|--verbose|--no-session-persistence|--bare'
|
||||
opencode run --help | rg -- '--pure|--model|--agent|--format|--dir'
|
||||
codex exec --help | rg -- '--ephemeral|--json|--sandbox|--cd|--profile|--model|--output-last-message'
|
||||
nc -z 127.0.0.1 18083; nc -z 127.0.0.1 19093
|
||||
tmp_catalog="$(mktemp)"
|
||||
http_code="$(curl --cacert build/dev-runtime/.secrets/credential-plane/ca.pem -sS -o "$tmp_catalog" -w '%{http_code}' -H "Authorization: Bearer $iop_token" "$base_url/v1/models")"
|
||||
http_code="$(curl --cacert "$ca" -sS -o "$tmp_catalog" -w '%{http_code}' -H "Authorization: Bearer $iop_token" "$api_root/v1/models")"
|
||||
test "$http_code" = 200
|
||||
for model in claude-sonnet-5 gemini-3.6-flash gpt-5.6-luna gemini-hybrid gpt-hybrid; do jq -e --arg model "$model" '.data[] | select(.id == $model)' "$tmp_catalog" >/dev/null; done
|
||||
test -z "$(git status --short)"
|
||||
test "$(git branch --show-current)" = dev
|
||||
test "$(git rev-parse HEAD)" = 16b7aba95a282b6c5d1e88d3b1849eaa1208b28a
|
||||
test "$(claude --version | head -1)" = '2.1.177 (Claude Code)'
|
||||
test "$(opencode --version)" = '1.18.3'
|
||||
codex --version | rg -x 'codex-cli 0\.146\.0'
|
||||
nc -z 127.0.0.1 18083
|
||||
nc -z 127.0.0.1 19093
|
||||
mkdir -p "$run_root"
|
||||
mv "$tmp_catalog" "$run_root/catalog.json"
|
||||
cp /dev/null "$run_root/runtime.txt"
|
||||
printf 'branch=dev\nhead=%s\nclaude=2.1.177\nopencode=1.18.3\ncodex=0.146.0\nports=18083,19093\ncatalog_http=200\n' "$(git rev-parse HEAD)" > "$run_root/runtime.txt"
|
||||
unset iop_token
|
||||
mkdir "$run_root"; mv "$tmp_catalog" "$run_root/catalog.json"
|
||||
printf 'branch=dev\nhead=%s\nports=18083,19093\ncatalog_http=200\n' "$(git rev-parse HEAD)" > "$run_root/runtime.txt"
|
||||
```
|
||||
|
||||
Copy the exact fixed prompt block to `prompt.txt`, create `row-01` through `row-09` before starting row 01, and execute the nine commands using the caller-specific forms fixed in TEST-1. The implementation evidence must paste each expanded command with secrets replaced by `<redacted>`, its sole exit code, and the corresponding `attempt.txt`; this is required because no shared benchmark script may be added.
|
||||
Copy the exact fixed prompt into `prompt.txt`, create all nine row workspaces, and execute the expanded caller forms in TEST-1. Preserve redacted expanded commands, exit statuses, and ledgers in the active review. Unset `iop_token` only after row 09.
|
||||
|
||||
For each scorable opaque ID, render locally with this block exactly once (replace `E01` with that ID; the block records both attempted commands and statuses and never retries):
|
||||
For each scorable opaque ID, run once with a fresh profile per viewport and record both statuses:
|
||||
|
||||
```bash
|
||||
opaque_id=E01; opaque_dir="$PWD/agent-test/runs/bench-lite-01/$opaque_id"; render_log="$opaque_dir/render.txt"
|
||||
test ! -e "$render_log"; : > "$render_log"
|
||||
profile_desktop="$(mktemp -d)"
|
||||
printf 'viewport=1440x900\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_desktop" --window-size=1440,900 --screenshot="$opaque_dir/desktop.png" "file://$opaque_dir/index.html"; desktop_status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$desktop_status" >> "$render_log"
|
||||
profile_mobile="$(mktemp -d)"
|
||||
printf 'viewport=390x844\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_mobile" --window-size=390,844 --screenshot="$opaque_dir/mobile.png" "file://$opaque_dir/index.html"; mobile_status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$mobile_status" >> "$render_log"
|
||||
profile="$(mktemp -d)"; printf 'viewport=1440x900\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile" --window-size=1440,900 --screenshot="$opaque_dir/desktop.png" "file://$opaque_dir/index.html"; status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$status" >> "$render_log"
|
||||
profile="$(mktemp -d)"; printf 'viewport=390x844\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile" --window-size=390,844 --screenshot="$opaque_dir/mobile.png" "file://$opaque_dir/index.html"; status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$status" >> "$render_log"
|
||||
```
|
||||
|
||||
Run fresh final checks:
|
||||
Run fresh local checks; cached output is not applicable:
|
||||
|
||||
```bash
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name attempt.txt -type f | wc -l)" -eq 9
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name producer.jsonl -type f | wc -l)" -eq 9
|
||||
test "$(rg -l '^attempt_count=1$' agent-test/runs/bench-lite-01/row-*/attempt.txt | wc -l)" -eq 9
|
||||
test "$(rg -l '^workspace_initial_entries=0$' agent-test/runs/bench-lite-01/row-*/attempt.txt | wc -l)" -eq 9
|
||||
test "$(cut -d' ' -f1 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9
|
||||
test "$(cut -d' ' -f2 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name desktop.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name mobile.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name render.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name source.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
scorable="$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -path '*/E*/index.html' -type f | wc -l)"
|
||||
for name in desktop.png mobile.png render.txt source.txt; do test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -path "*/E*/$name" -type f | wc -l)" -eq "$scorable"; done
|
||||
! rg -n '미실행|미측정|미확인|미부여-[0-9]|미채점' agent-test/dev/iop-thin-agent-model-comparison.md
|
||||
! rg -n '(retry|resume|recovery)[[:space:]]*[:=][[:space:]]*(true|yes|[1-9])' agent-test/runs/bench-lite-01
|
||||
! rg -n --hidden '(sk-|Bearer [A-Za-z0-9._-]{16,}|api[_-]?key[[:space:]]*[:=][[:space:]]*[A-Za-z0-9._-]{16,})' agent-test/dev/iop-thin-agent-model-comparison.md agent-test/runs/bench-lite-01
|
||||
! rg -n 'Claude|OpenCode|Codex|claude-sonnet|gemini|gpt|hybrid|경과|usage' agent-test/runs/bench-lite-01/E0*
|
||||
! rg -n 'Claude|OpenCode|Codex|claude-sonnet|gemini|gpt|hybrid|경과|usage|row-[0-9]' agent-test/runs/bench-lite-01/E*/source.txt agent-test/runs/bench-lite-01/E*/render.txt
|
||||
git diff --check -- agent-test/dev/iop-thin-agent-model-comparison.md agent-task/m-thin-agent-model-comparison-benchmark
|
||||
git diff --name-only -- . ':(exclude)agent-test/dev/iop-thin-agent-model-comparison.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/**'
|
||||
```
|
||||
|
||||
The last command must print nothing. Reviewer inspection must also prove: gate preceded workspace creation; the nine expanded producer commands match TEST-1 and each ran once; caller usage is explicit or `미제공`; every scorable ID has one source/two renders/two-entry render ledger; route facts were unavailable during scoring; A-D evidence uses locked anchors and totals are correct; the conclusion excludes failures/unscorable rows and avoids statistical generalization.
|
||||
The last command must print nothing. Reviewer inspection must confirm gate-before-workspace ordering, exact tuple/workspace binding, one producer invocation per row, explicit usage or `미제공`, opaque isolation through score freeze, fixed-anchor arithmetic, and a conclusion that excludes failed/unscorable rows from quality comparison.
|
||||
|
||||
After completing all work, fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`.
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,99 @@
|
|||
<!-- task=m-thin-agent-model-comparison-benchmark plan=1 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run the plan verification, paste actual output, and leave this active pair in place. Do not archive files, write `complete.log`, or classify the next state.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-14
|
||||
task=m-thin-agent-model-comparison-benchmark, plan=1, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Replaced unstarted pair: `plan_local_G08_0.log`, `code_review_cloud_G08_0.log`; no prior verdict.
|
||||
- Replan closes the missing authenticated catalog call, exact caller/render procedure, and evidence write boundary while preserving the benchmark scope.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
Rerun applicable deterministic checks and inspect immutable external evidence. Append the official verdict only after implementation is submitted. On PASS, archive this pair with suffix `1`, write `complete.log` preserving the first-line metadata, and move the task directory under the dated archive. Roadmap aggregation is a later `sync-milestone-workstate` action.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| TEST-1 Consume the Immutable Nine-Row Matrix | [ ] |
|
||||
| TEST-2 Render, Score Once, and Conclude | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace.
|
||||
- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero.
|
||||
- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport.
|
||||
- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality.
|
||||
- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Implementing agents must not modify this checklist.
|
||||
|
||||
- [ ] Append one verdict with verified routing signals.
|
||||
- [ ] Verify verdict, dimensions, and finding severities agree.
|
||||
- [ ] Rerun required deterministic verification and inspect the nine immutable attempt ledgers/streams.
|
||||
- [ ] Record evidence, root cause, selected fix, files/tests, and acceptance commands for each Required/Suggested finding.
|
||||
- [ ] Archive this file to `code_review_cloud_G08_1.log` and the plan to `plan_local_G08_1.log`.
|
||||
- [ ] Verify the managed `.gitignore` block and artifact visibility.
|
||||
- [ ] On PASS, write `complete.log`, preserve milestone metadata, move the task directory to the dated archive, and update this checklist there.
|
||||
- [ ] On WARN/FAIL, create only the next state required by the code-review skill and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Replace with actual deviations or `None`._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Replace with actual implementation decisions._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the authenticated catalog body check passed before any producer workspace existed.
|
||||
- Confirm the exact expanded command for each of nine rows, one ledger/stream per row, and no hidden caller retry or session continuation.
|
||||
- Confirm no product/config/script/manifest/state-store change entered the worktree.
|
||||
- Confirm route facts were absent from every opaque scoring directory until scores were frozen.
|
||||
- Confirm usage was caller-provided or `미제공`; failures/unscorable artifacts were not converted to zero.
|
||||
- Confirm each scorable source has two one-shot renders, direct anchor evidence, and correct arithmetic.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### External gate and producer attempts
|
||||
|
||||
Paste the redacted authenticated gate output, exact expanded commands, sole exit status, and each `attempt.txt`. Do not paste credentials or raw sensitive provider payloads.
|
||||
|
||||
_Replace with actual output._
|
||||
|
||||
### Local deterministic checks
|
||||
|
||||
Run the exact final checks from `PLAN-local-G08.md` and paste stdout/stderr plus exit statuses.
|
||||
|
||||
_Replace with actual output._
|
||||
|
||||
### Manual scorecard review
|
||||
|
||||
Record reviewer arithmetic, anchor/evidence, opaque-blinding, render-count, usage, and bounded-conclusion findings.
|
||||
|
||||
_Replace with actual findings._
|
||||
|
||||
---
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---|---|---|
|
||||
| Header, overview, archive snapshot, reviewer instructions | Fixed | Implementer must not modify |
|
||||
| Implementation item/checklist status | Implementer | Check only after actual completion |
|
||||
| Review-Only Checklist | Review agent | Implementer must not modify |
|
||||
| Deviations, decisions, verification results | Implementer, then reviewer | Replace placeholders with actual evidence |
|
||||
| Code Review Result | Review agent | Appended only during official review |
|
||||
|
|
@ -0,0 +1,365 @@
|
|||
<!-- task=m-thin-agent-model-comparison-benchmark plan=1 tag=TEST milestone-task=single-attempt-matrix,minimal-result-table,single-pass-scorecard,bounded-conclusion -->
|
||||
|
||||
# Plan - Executable Thin Agent Single-Attempt Comparison
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Fill the implementation-owned sections in `CODE_REVIEW-cloud-G08.md`. Run the commands exactly once per matrix row, paste actual output, and leave the active pair in place for official review. If a pre-attempt gate fails, stop before creating producer workspaces. If a producer command starts, its exit, timeout, missing artifact, or malformed terminal is that row's final result; never rerun, resume, or replace it.
|
||||
|
||||
## Background
|
||||
|
||||
The first plan correctly bounded the benchmark but did not provide an executable authenticated catalog check, exact caller invocations, or exact render commands, and omitted required evidence paths from its write boundary. This replan closes those gaps before any producer attempt is consumed. It keeps the same nine qualified routes, fixed prompt, one-attempt rule, blind scorecard, and documentation-only result.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Replaced unstarted pair: `agent-task/m-thin-agent-model-comparison-benchmark/plan_local_G08_0.log`, `agent-task/m-thin-agent-model-comparison-benchmark/code_review_cloud_G08_0.log`.
|
||||
- Prior verdict: none; the review file was an unfilled implementation stub.
|
||||
- Preserved decisions: one atomic packet, no product/config/runner changes, one producer attempt per row, ignored raw evidence, opaque single-pass scoring.
|
||||
- Corrected defects: catalog gate had no catalog request, caller/render steps were prose-only, and evidence files were outside `Modified Files Summary`.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `AGENTS.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `agent-ops/rules/common/rules-roadmap.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/common/router.md`
|
||||
- `agent-ops/skills/common/plan/SKILL.md`
|
||||
- `agent-ops/skills/common/code-review/SKILL.md`
|
||||
- `agent-ops/skills/common/finalize-task-routing/SKILL.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/dev/rules.md`
|
||||
- `agent-test/dev/testing-smoke.md`
|
||||
- `agent-test/inventory-agent.yaml`
|
||||
- `agent-test/inventory-dev.yaml`
|
||||
- `agent-test/dev/iop-thin-agent-model-comparison.md`
|
||||
- `agent-test/dev/iop-benchmark-route-minimal-html-smoke.md`
|
||||
- `agent-roadmap/current.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/thin-agent-model-comparison-benchmark.md`
|
||||
- `agent-client/claude/README.md`
|
||||
- `docs/dev-opencode-settings-guide.md`
|
||||
- `opencode.json`
|
||||
- `agent-task/responses_provider_bridge/PLAN-local-G08.md`
|
||||
- archived pair listed above
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
SDD is not required. The Milestone records this as a test-only observation of existing caller/product paths with no API, state-machine, retry, or schema change.
|
||||
|
||||
### Verification Context
|
||||
|
||||
No handoff was supplied. Repository-native dev rules select `toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, port `18083`, the existing SOPS principal token, and the managed CA at `build/dev-runtime/.secrets/credential-plane/ca.pem`.
|
||||
|
||||
Fresh read-only preflight on 2026-08-14 confirmed clean branch `dev` at `16b7aba95a282b6c5d1e88d3b1849eaa1208b28a`, Claude Code `2.1.177`, OpenCode `1.18.3`, Codex `0.146.0`, open ports `18083`/`19093`, config/secret presence, caller flags, `/opt/homebrew/bin/gtimeout`, `jq`, `sops`, and managed CA presence. The exact catalog body check below remains a hard gate. The live checkout is intentionally the smoke-qualified runtime identity; do not deploy or change it in this benchmark.
|
||||
|
||||
Local `/config/.local/bin/chromium` is the declared render executor. Each scorable source is rendered once at each fixed viewport. Credentials and raw provider payloads stay only on the remote runner or ignored evidence paths and never enter tracked output.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Live availability has no deterministic unit substitute; the authenticated catalog gate and nine immutable attempts are the evidence.
|
||||
- Caller-native usage shapes may differ. Record only an explicit usage field in the one producer stream; otherwise write `미제공`.
|
||||
- Visual scoring is manual by design; deterministic selector, render-count, anchor, and arithmetic checks bound it.
|
||||
|
||||
### Symbol References
|
||||
|
||||
None; no product symbol changes.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
Keep one atomic plan because execution records, opaque mapping, score rows, and conclusion must bind to the same immutable nine attempts. `large_indivisible_context=false`; explicit row commands and deterministic evidence reduce the packet.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Writable tracked files are the comparison document and active review evidence. Writable ignored evidence is limited to `agent-test/runs/bench-lite-01/**`. Product source/config, caller installation/user config, roadmap, spec, contract, runner scripts, manifests, lifecycle stores, and route-smoke records are excluded.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- evaluation_mode: `first-pass` for the complete replacement packet
|
||||
- finalizer: `finalize-task-policy.sh pair local-fit false 1 0 false 1 2 1 2 2 official-review 1 2 1 2 2`
|
||||
- closures: scope/context/verification/evidence/ownership/decision closed for build and review
|
||||
- build: G08, `local-fit`, `PLAN-local-G08.md`
|
||||
- review: G08, `official-review`, `CODE_REVIEW-cloud-G08.md`
|
||||
- loop risk: `variant_product`; recovery signals false/0
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] Pass the authenticated catalog and runtime identity gate before creating any producer workspace.
|
||||
- [ ] Create nine empty workspaces and execute each fixed caller/model row exactly once, preserving one immutable record per row with no retry/resume/recovery.
|
||||
- [ ] Fill the nine-row result table from immutable evidence; record caller-provided usage or `미제공`, never an estimate or substituted zero.
|
||||
- [ ] Assign a shuffled opaque ID after all attempts, copy each exact scorable source, and render it exactly once at desktop and mobile viewport.
|
||||
- [ ] Score each scorable opaque artifact once with locked anchors and direct source/render evidence, then verify arithmetic.
|
||||
- [ ] Write a bounded conclusion comparing only successful scorable results and separating success/time/usage from quality.
|
||||
- [ ] Run final attempt-count, render-count, placeholder, retry, secret, arithmetic, and scope checks.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Consume the Immutable Nine-Row Matrix
|
||||
|
||||
**Problem:** The archived plan required one attempt but its catalog gate never called `/v1/models`, and it described rather than specified the nine producer commands.
|
||||
|
||||
**Solution:** In one remote `zsh` session, complete the gate below, create all `row-01` through `row-09` directories, save the fixed prompt once, and then run the following row mapping exactly once in document order:
|
||||
|
||||
1. Claude / `claude-sonnet-5`
|
||||
2. Claude / `gemini-3.6-flash`
|
||||
3. OpenCode / `gemini-3.6-flash`
|
||||
4. Claude / `gpt-5.6-luna`
|
||||
5. Codex / `gpt-5.6-luna`
|
||||
6. Claude / `gemini-hybrid`
|
||||
7. OpenCode / `gemini-hybrid`
|
||||
8. Claude / `gpt-hybrid`
|
||||
9. Codex / `gpt-hybrid`
|
||||
|
||||
For each row, write `attempt.txt` before invocation with row, caller/model, version, UTC start, `attempt_count=1`, `retry=0`, `resume=0`, and workspace-empty check. Redirect caller stdout/stderr to the row's sole `producer.jsonl`, append exit/end/elapsed to `attempt.txt`, and do not invoke that row again under any outcome. Use `/opt/homebrew/bin/gtimeout 900` as an outer bound; timeout exit 124 is a final failure. Claude uses `CLAUDE_CODE_MAX_RETRIES=0`, print stream-json, bare/no persistence, and the fixed model. OpenCode uses command-scoped `OPENCODE_CONFIG_CONTENT`, `run --pure --auto --format json --dir`, with no `--continue`/`--session`. Codex uses a fresh `mktemp -d` `CODEX_HOME` outside the repository, copied auth and the repository-proven `iop-direct.config.toml`, `exec --ephemeral --json --sandbox workspace-write --cd`, and no resume command; delete that temporary home immediately after the command. Direct rows use workspace `index.html`; preset rows extract the single terminal fenced HTML block only after the invocation ends. Missing/multiple blocks are `채점 불가`, not a second attempt.
|
||||
|
||||
Use these exact caller forms, substituting only the fixed `row`, `model`, and caller from the numbered mapping above. Run each expanded block once, not a loop or repository script:
|
||||
|
||||
```bash
|
||||
# Claude rows 01, 02, 04, 06, 08
|
||||
row=row-01 model=claude-sonnet-5
|
||||
workspace="$run_root/$row/workspace"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=claude\nmodel=%s\ncaller_version=2.1.177\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
ANTHROPIC_BASE_URL="${base_url%/v1}" ANTHROPIC_AUTH_TOKEN="$iop_token" NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" CLAUDE_CODE_MAX_RETRIES=0 /opt/homebrew/bin/gtimeout 900 claude --print --output-format stream-json --bare --no-session-persistence --dangerously-skip-permissions --model "$model" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
|
||||
# OpenCode rows 03, 07
|
||||
row=row-03 model=gemini-3.6-flash
|
||||
workspace="$run_root/$row/workspace"
|
||||
export IOP_BENCH_TOKEN="$iop_token"
|
||||
export OPENCODE_CONFIG_CONTENT="$(jq -cn --arg base "${base_url%/v1}/v1" --arg model "$model" '{permission:{read:"allow",write:"allow",edit:"allow",glob:"allow",bash:"allow"},provider:{iop:{npm:"@ai-sdk/openai-compatible",options:{baseURL:$base,apiKey:"{env:IOP_BENCH_TOKEN}"},models:{($model):{name:$model}}}}}')"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=opencode\nmodel=%s\ncaller_version=1.18.3\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
NODE_EXTRA_CA_CERTS="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 opencode run --pure --auto --model "iop/$model" --agent build --format json --dir "$workspace" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
unset OPENCODE_CONFIG_CONTENT IOP_BENCH_TOKEN
|
||||
|
||||
# Codex rows 05, 09
|
||||
row=row-05 model=gpt-5.6-luna
|
||||
workspace="$run_root/$row/workspace"; codex_home="$(mktemp -d)"
|
||||
mkdir -p "$codex_home"; cp /Users/toki/.codex/auth.json "$codex_home/auth.json"; cp /Users/toki/.codex/iop-direct.config.toml "$codex_home/iop-direct.config.toml"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"; start_s="$(date +%s)"
|
||||
printf 'row=%s\ncaller=codex\nmodel=%s\ncaller_version=0.146.0\nstarted_at=%s\nattempt_count=1\nretry=0\nresume=0\nworkspace_initial_entries=%s\n' "$row" "$model" "$started" "$(find "$workspace" -mindepth 1 -maxdepth 1 | wc -l | tr -d ' ')" > "$run_root/$row/attempt.txt"
|
||||
set +e
|
||||
CODEX_HOME="$codex_home" IOP_CODEX_API_KEY="$iop_token" CODEX_CA_CERTIFICATE="$PWD/build/dev-runtime/.secrets/credential-plane/ca.pem" /opt/homebrew/bin/gtimeout 900 codex exec --ephemeral --json --sandbox workspace-write --skip-git-repo-check --cd "$workspace" --profile iop-direct --model "$model" --output-last-message "$run_root/$row/terminal.txt" "$(cat "$run_root/prompt.txt")" > "$run_root/$row/producer.jsonl" 2>&1
|
||||
status=$?
|
||||
set -e
|
||||
end_s="$(date +%s)"; printf 'ended_at=%s\nelapsed_seconds=%s\nexit_status=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$((end_s-start_s))" "$status" >> "$run_root/$row/attempt.txt"
|
||||
rm -rf "$codex_home"
|
||||
```
|
||||
|
||||
Before row 01, run `for n in {01..09}; do mkdir -p "$run_root/row-$n/workspace"; test -z "$(find "$run_root/row-$n/workspace" -mindepth 1 -maxdepth 1 -print -quit)"; done`. Before each later row, expand a fresh caller block with its fixed tuple and verify that its `attempt.txt` and `producer.jsonl` do not exist. After row 09, unset `iop_token`. From the current checkout, transfer once with `rsync -a toki@toki-labs.com:/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01/ agent-test/runs/bench-lite-01/`, then perform source extraction, opaque assignment, rendering, and tracked result editing locally.
|
||||
|
||||
For direct rows, accept `workspace/index.html` only when the row terminal contains `BENCH_LITE_01_DONE` exactly once. For preset rows, first extract the caller's single terminal text into `terminal.txt` (`jq -r 'select(.type == "result") | .result // empty'` for Claude, the final text-part event selected from the OpenCode JSONL shape observed in that sole stream, and Codex's `--output-last-message` file). Then run this one-shot strict extractor locally for each preset terminal; it succeeds only for exactly one final `html` fence and writes the bytes between fences without modifying them:
|
||||
|
||||
```bash
|
||||
ruby -e 's=File.binread(ARGV[0]); m=s.scan(/```html\r?\n(.*?)\r?\n```/m); abort("expected exactly one html fence") unless m.length==1; File.binwrite(ARGV[1],m[0][0])' terminal.txt index.html
|
||||
```
|
||||
|
||||
If the OpenCode terminal event shape cannot be selected unambiguously from its one `producer.jsonl`, record the row as `채점 불가`; do not infer text from intermediate tool events and do not rerun it.
|
||||
|
||||
After all rows, create the shuffled bijection exactly once and never regenerate it:
|
||||
|
||||
```bash
|
||||
test ! -e "$run_root/opaque-map.txt"
|
||||
ruby -e 'rows=(1..9).map { |n| format("row-%02d",n) }; ids=(1..9).map { |n| format("E%02d",n) }.shuffle; File.write(ARGV[0],rows.zip(ids).map { |r,i| "#{r} #{i}\n" }.join)' "$run_root/opaque-map.txt"
|
||||
```
|
||||
|
||||
For every mapped direct row with a valid marker/source, run `mkdir -p "$run_root/$opaque_id" && cp "$run_root/$row/workspace/index.html" "$run_root/$opaque_id/index.html"`. For every mapped preset row with an unambiguous terminal, run the strict extractor below with `"$run_root/$row/terminal.txt"` and `"$run_root/$opaque_id/index.html"`. Then record only `sha256=<digest>` in `"$run_root/$opaque_id/source.txt"` using `shasum -a 256`; never put row, caller, route, model, time, or usage in an `E*` directory. Keep the mapping closed until every score/evidence block is frozen.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: replace result placeholders with immutable row evidence.
|
||||
- [ ] `agent-test/runs/bench-lite-01/prompt.txt`: exact fixed prompt copied from the tracked document before attempts.
|
||||
- [ ] `agent-test/runs/bench-lite-01/catalog.json`: authenticated catalog body with credentials absent.
|
||||
- [ ] `agent-test/runs/bench-lite-01/runtime.txt`: redacted gate identity/output.
|
||||
- [ ] `agent-test/runs/bench-lite-01/row-01/attempt.txt` through `row-09/attempt.txt`: one immutable attempt ledger each.
|
||||
- [ ] `agent-test/runs/bench-lite-01/row-01/producer.jsonl` through `row-09/producer.jsonl`: one caller stream each.
|
||||
- [ ] `agent-test/runs/bench-lite-01/opaque-map.txt`: post-attempt row/opaque bijection.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/index.html` through `E09/index.html`: exact source only for scorable rows.
|
||||
|
||||
**Test Strategy:** No test code or common runner. The nine explicit caller commands are the measured behavior. The per-row ledger and unique stream/source paths prove single invocation without creating lifecycle automation.
|
||||
|
||||
**Verification:** Run the gate and matrix commands in `Final Verification`, then the local evidence checks. Expected: gate 200 with all five ids before workspace creation, nine attempt ledgers/streams with `attempt_count=1`, and no retry/resume/recovery marker.
|
||||
|
||||
### [TEST-2] Render, Score Once, and Conclude
|
||||
|
||||
**Problem:** The archived plan had no runnable fixed-viewport render command and did not enumerate render/evidence paths in its write boundary.
|
||||
|
||||
**Solution:** For every `E*/index.html` that exists, run Chromium exactly once per viewport with a fresh temporary profile outside the repository, `--headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw`, `--window-size=1440,900` to `desktop.png`, then `--window-size=390,844` to `mobile.png`. Record the two exact commands and exit codes in `render.txt`; do not repeat a failed render. Score only from the opaque directory's source and images. Fill A from exact selectors, B-D using only 0/1/3/5 anchors, record direct evidence and one reason per deduction, and verify `A+B+C+D`. Join route mapping only after every score/evidence block is frozen. Write the bounded conclusion without zero-substituting failures, unscorable sources, or missing usage.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-test/dev/iop-thin-agent-model-comparison.md`: score table, evidence blocks, correction notes if any, and bounded conclusion.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/desktop.png` through `E09/desktop.png`: one desktop render for each scorable source.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/mobile.png` through `E09/mobile.png`: one mobile render for each scorable source.
|
||||
- [ ] `agent-test/runs/bench-lite-01/E01/render.txt` through `E09/render.txt`: exact two commands and outcomes for each scorable source.
|
||||
|
||||
**Test Strategy:** No automated judge or browser pass/fail gate. Exact source, two immutable renders, locked anchors, and reviewer arithmetic provide the required one-pass evidence.
|
||||
|
||||
**Verification:** Run the render loop once and final checks below. Expected: every scorable ID has one source, two images, one two-entry render ledger, evidence-backed anchors, correct total, and no route/model/time/usage in opaque evidence.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `agent-test/dev/iop-thin-agent-model-comparison.md` | TEST-1, TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/prompt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/catalog.json` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/runtime.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/opaque-map.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-01/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-01/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-01/workspace/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-02/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-02/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-02/workspace/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-03/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-03/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-03/workspace/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-04/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-04/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-04/workspace/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-05/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-05/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-05/terminal.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-05/workspace/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-06/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-06/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-06/terminal.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-07/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-07/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-07/terminal.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-08/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-08/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-08/terminal.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-09/attempt.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-09/producer.jsonl` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/row-09/terminal.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E01/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E01/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E01/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E01/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E01/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E02/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E02/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E02/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E02/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E02/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E03/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E03/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E03/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E03/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E03/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E04/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E04/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E04/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E04/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E04/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E05/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E05/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E05/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E05/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E05/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E06/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E06/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E06/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E06/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E06/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E07/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E07/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E07/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E07/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E07/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E08/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E08/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E08/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E08/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E08/render.txt` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E09/index.html` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E09/source.txt` | TEST-1 |
|
||||
| `agent-test/runs/bench-lite-01/E09/desktop.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E09/mobile.png` | TEST-2 |
|
||||
| `agent-test/runs/bench-lite-01/E09/render.txt` | TEST-2 |
|
||||
| `agent-task/m-thin-agent-model-comparison-benchmark/CODE_REVIEW-cloud-G08.md` | TEST-1, TEST-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Before any producer workspace exists, run on the dev runner in one shell. This command makes the authenticated request, preserves only the credential-free body, and proves all fixed model IDs:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
cd /Users/toki/agent-work/iop-dev
|
||||
run_root=/Users/toki/agent-work/iop-dev/agent-test/runs/bench-lite-01
|
||||
test ! -e "$run_root"
|
||||
export SOPS_AGE_KEY_FILE=/Users/toki/.config/sops/age/keys.txt
|
||||
secret=/Users/toki/.config/iop/secrets/dev-openai-toki.sops.yaml
|
||||
base_url="$(/opt/homebrew/bin/sops -d --extract '["base_url"]' "$secret")"
|
||||
iop_token="$(/opt/homebrew/bin/sops -d --extract '["tokens"]["toki-dev-cline"]' "$secret")"
|
||||
test -n "$iop_token"
|
||||
tmp_catalog="$(mktemp)"
|
||||
http_code="$(curl --cacert build/dev-runtime/.secrets/credential-plane/ca.pem -sS -o "$tmp_catalog" -w '%{http_code}' -H "Authorization: Bearer $iop_token" "$base_url/v1/models")"
|
||||
test "$http_code" = 200
|
||||
for model in claude-sonnet-5 gemini-3.6-flash gpt-5.6-luna gemini-hybrid gpt-hybrid; do jq -e --arg model "$model" '.data[] | select(.id == $model)' "$tmp_catalog" >/dev/null; done
|
||||
test -z "$(git status --short)"
|
||||
test "$(git branch --show-current)" = dev
|
||||
test "$(git rev-parse HEAD)" = 16b7aba95a282b6c5d1e88d3b1849eaa1208b28a
|
||||
test "$(claude --version | head -1)" = '2.1.177 (Claude Code)'
|
||||
test "$(opencode --version)" = '1.18.3'
|
||||
codex --version | rg -x 'codex-cli 0\.146\.0'
|
||||
nc -z 127.0.0.1 18083
|
||||
nc -z 127.0.0.1 19093
|
||||
mkdir -p "$run_root"
|
||||
mv "$tmp_catalog" "$run_root/catalog.json"
|
||||
cp /dev/null "$run_root/runtime.txt"
|
||||
printf 'branch=dev\nhead=%s\nclaude=2.1.177\nopencode=1.18.3\ncodex=0.146.0\nports=18083,19093\ncatalog_http=200\n' "$(git rev-parse HEAD)" > "$run_root/runtime.txt"
|
||||
unset iop_token
|
||||
```
|
||||
|
||||
Copy the exact fixed prompt block to `prompt.txt`, create `row-01` through `row-09` before starting row 01, and execute the nine commands using the caller-specific forms fixed in TEST-1. The implementation evidence must paste each expanded command with secrets replaced by `<redacted>`, its sole exit code, and the corresponding `attempt.txt`; this is required because no shared benchmark script may be added.
|
||||
|
||||
For each scorable opaque ID, render locally with this block exactly once (replace `E01` with that ID; the block records both attempted commands and statuses and never retries):
|
||||
|
||||
```bash
|
||||
opaque_id=E01; opaque_dir="$PWD/agent-test/runs/bench-lite-01/$opaque_id"; render_log="$opaque_dir/render.txt"
|
||||
test ! -e "$render_log"; : > "$render_log"
|
||||
profile_desktop="$(mktemp -d)"
|
||||
printf 'viewport=1440x900\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_desktop" --window-size=1440,900 --screenshot="$opaque_dir/desktop.png" "file://$opaque_dir/index.html"; desktop_status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$desktop_status" >> "$render_log"
|
||||
profile_mobile="$(mktemp -d)"
|
||||
printf 'viewport=390x844\n' >> "$render_log"
|
||||
set +e; /config/.local/bin/chromium --headless --disable-gpu --hide-scrollbars --run-all-compositor-stages-before-draw --user-data-dir="$profile_mobile" --window-size=390,844 --screenshot="$opaque_dir/mobile.png" "file://$opaque_dir/index.html"; mobile_status=$?; set -e
|
||||
printf 'exit_status=%s\n' "$mobile_status" >> "$render_log"
|
||||
```
|
||||
|
||||
Run fresh final checks:
|
||||
|
||||
```bash
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name attempt.txt -type f | wc -l)" -eq 9
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name producer.jsonl -type f | wc -l)" -eq 9
|
||||
test "$(rg -l '^attempt_count=1$' agent-test/runs/bench-lite-01/row-*/attempt.txt | wc -l)" -eq 9
|
||||
test "$(cut -d' ' -f1 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9
|
||||
test "$(cut -d' ' -f2 agent-test/runs/bench-lite-01/opaque-map.txt | LC_ALL=C sort -u | wc -l)" -eq 9
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name desktop.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name mobile.png -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name render.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
test "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name source.txt -type f | wc -l)" -eq "$(find agent-test/runs/bench-lite-01 -mindepth 2 -maxdepth 2 -name index.html -type f | wc -l)"
|
||||
! rg -n '미실행|미측정|미확인|미부여-[0-9]|미채점' agent-test/dev/iop-thin-agent-model-comparison.md
|
||||
! rg -n '(retry|resume|recovery)[[:space:]]*[:=][[:space:]]*(true|yes|[1-9])' agent-test/runs/bench-lite-01
|
||||
! rg -n --hidden '(sk-|Bearer [A-Za-z0-9._-]{16,}|api[_-]?key[[:space:]]*[:=][[:space:]]*[A-Za-z0-9._-]{16,})' agent-test/dev/iop-thin-agent-model-comparison.md agent-test/runs/bench-lite-01
|
||||
! rg -n 'Claude|OpenCode|Codex|claude-sonnet|gemini|gpt|hybrid|경과|usage' agent-test/runs/bench-lite-01/E0*
|
||||
git diff --check -- agent-test/dev/iop-thin-agent-model-comparison.md agent-task/m-thin-agent-model-comparison-benchmark
|
||||
git diff --name-only -- . ':(exclude)agent-test/dev/iop-thin-agent-model-comparison.md' ':(exclude)agent-task/m-thin-agent-model-comparison-benchmark/**'
|
||||
```
|
||||
|
||||
The last command must print nothing. Reviewer inspection must also prove: gate preceded workspace creation; the nine expanded producer commands match TEST-1 and each ran once; caller usage is explicit or `미제공`; every scorable ID has one source/two renders/two-entry render ledger; route facts were unavailable during scoring; A-D evidence uses locked anchors and totals are correct; the conclusion excludes failures/unscorable rows and avoids statistical generalization.
|
||||
|
||||
After completing all work, fill implementation-owned sections in `CODE_REVIEW-cloud-G08.md`.
|
||||
Loading…
Reference in a new issue