toki/iop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(benchmark): 첫 비교 실행 준비 계약을 고정한다

Browse source 
동일 fixture와 C01-C09 matrix를 재현 가능하게 고정하고 실제 dev readiness가 검증된 뒤에만 scored 실행으로 넘어가도록 한다.
This commit is contained in:
toki 2026-08-12 04:45:08 +09:00
parent 2aeaab81fc
commit e86113f0ae
45 changed files with 2505 additions and 227 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

18
agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md
View file

@ -12,7 +12,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
## When to use
- User requests benchmark manifest validation: `validate`, `validate manifest`, `manifest 검증`
- User requests direct-route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검`
- User requests route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검`
- User requests benchmark execution: `run`, `run benchmark`, `벤치마크 실행`, `시작해`
- User requests benchmark resume: `resume`, `resume benchmark`, `재개`, `계속해`
- User requests benchmark status: `status`, `status benchmark`, `상태 확인`, `어디까지 왔어`
@ -55,14 +55,14 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
4. **Delegate preflight to the CLI**
- Run: `python3 scripts/agent_comparison_benchmark.py preflight --manifest <manifest-path>`
- The CLI validates generic preset cells locally and records only direct-cell observations in one append-only run record.
- The CLI records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order in one append-only run record.
- On exit 0, report the exact closed `ready` summary from stdout.
- On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat a generic preset contract as live readiness.
- On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat local manifest validation as live readiness.
5. **Delegate run to the CLI**
- Run: `python3 scripts/agent_comparison_benchmark.py run --manifest <manifest-path>`
- On missing or invalid manifest, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before creating execution state.
- The CLI creates one run and uses its single writer to append a fresh preflight before attempt allocation.
- The CLI creates one run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69, allocates no attempt, and preserves the run id for a later resume.
- On `ready`, it binds the exact caller, cell, fresh workspace, session, and attempt identity, then must invoke each eligible cell exactly once with the fixture task.
- Exit 0 only when every retained attempt is successful; otherwise report the exact closed execution failure summary from stderr with exit 69.
@ -70,7 +70,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
6. **Delegate resume to the CLI**
- Run: `python3 scripts/agent_comparison_benchmark.py resume --manifest <manifest-path> --run-id <run-id> [--retry-failed]`
- On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before changing the run.
- The CLI opens the exact immutable run and uses its single writer to append a fresh preflight before attempt allocation.
- The CLI opens the exact immutable run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69 and allocates no attempt.
- On `ready`, it reconciles interrupted state, skips successful slots, preserves prior attempt bytes, and allocates a new attempt only for eligible work. `--retry-failed` admits a new attempt for failed, timed-out, or cancelled slots.
- It must invoke each eligible cell exactly once with a new workspace and session identity.
@ -84,7 +84,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
8. **Delegate score to the CLI**
- Run: `python3 scripts/agent_comparison_benchmark.py score --manifest <manifest-path> --run-id <run-id> [--retry-scoring-failed]`
- The CLI classifies lifecycle or required web-gate failures as immutable `unscored`, without invoking the evaluator or assigning zero.
- Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact `landing-quality-v1` worksheet.
- Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted.
- A prior `scored` result is terminal. A prior `scoring_failed` result is retried only with `--retry-scoring-failed`, which allocates a new score id and preserves every prior byte.
- On exit 0, report the exact closed `scored`, `unscored`, `scoring_failed`, and `blocked` counts from stdout.
- On exit 69, report the exact closed counts or unavailable-state line from stderr. Never substitute evaluator route/model/effort, fabricate a worksheet, or turn failure into zero.
@ -97,7 +97,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
- [ ] The CLI command was executed and the exit code matches the documented contract.
- [ ] The reported stdout/stderr matches the CLI output exactly.
- [ ] Preflight evidence is append-only, direct-only, and its output uses only closed status/count fields.
- [ ] Preflight evidence is append-only, covers every immutable matrix cell in canonical order, and uses only closed status/count fields.
- [ ] A preflight blocker created no scored attempt and was not bypassed.
- [ ] An ineligible execution attempt became `unscored` without an evaluator invocation or a zero score.
- [ ] Each eligible score id used one opaque blind workspace and one fresh evaluator session; retry preserved prior bytes and used a new id.
@ -169,8 +169,8 @@ stderr: <verbatim closed preflight or execution failure summary>
- Durable run/attempt state and preflight evidence are persisted only under the validated run root (`agent-test/runs/<output-id>/<run-id>/`).
- Caller sessions, output workspaces, and caches are fresh and isolated for every cell, repetition, and attempt; session or cache state is never shared within a run or across runs.
- Read-only testbed/fixture inputs (such as `../iop-s2`) are not copied back or mutated; no writes occur outside the validated run root.
- Direct preflight never allocates a scored attempt. Generic preset cells are local contract validation only.
- Run/resume append a fresh direct preflight under the run writer before any attempt allocation; a blocker allocates no attempt.
- Preflight never allocates a scored attempt. Every immutable matrix cell requires its own fresh live observation.
- Run/resume append a fresh all-cell preflight under the run writer before any attempt allocation; a blocker allocates no attempt.
- Ready execution binds one exact cell and immutable attempt identity to one fresh workspace/session and one task submission.
- Scoring copies only anonymous generated files, two local images, and screenshots into an opaque run-owned blind tree; the identity mapping remains outside that tree.
- Scoring records `unscored`, `scored`, and `scoring_failed` append-only, and a retry always allocates a fresh score id/session.

2
agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md
View file

@ -57,7 +57,7 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실
- 경로: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](../../archive/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md)
- 요약: 모델·caller·prompt·반복 횟수를 manifest로 바꾸고 Claude Code, agy, Codex의 IOP 연결부터 finish/idle, 시간·token·웹 검증·익명 채점·Markdown 보고까지 같은 pipeline으로 재현한다.
- [계획] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
- [진행중] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
- 경로: [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](milestones/iop-one-shot-agent-model-comparison.md)
- 요약: route-02 정식 smoke와 benchmark pipeline 준비 뒤 dev `../iop-s2`에서 동일 정적 웹 fixture로 9개 IOP 경유 단독·하이브리드 caller 조합을 각각 한 번 실행해 속도·token·품질을 비교한다.

7
agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md
View file

@ -13,7 +13,7 @@
## 상태
[계획]
[진행중]
## 구현 잠금
@ -59,9 +59,9 @@
실행 전에 공정한 fixture와 실제 IOP route/credential 상태를 고정한다.
- [ ] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
- [x] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
- [ ] [route-readiness] dev `../iop-s2`에서 Claude Code·agy·Codex의 IOP 인증, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 preflight를 통과했는지 확인한다.
- [ ] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
- [x] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
### Epic: [comparison-runs] 9개 원샷 실행
@ -103,6 +103,7 @@
- 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2`
- 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다.
- 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다.
- 현재 차단: `route-readiness`는 clean `../iop-s2` dev HEAD의 `iop-edge` artifact가 현재 HEAD에서 빌드되지 않아 artifact freshness gate에서 fail-closed했다. testbed artifact를 현재 HEAD로 다시 빌드한 뒤 caller/version/help와 C01-C09 public preflight 전체를 재실행한다.
- 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md)
- 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md)
- 확인 필요: 없음

260
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G05_2.log Normal file
View file

@ -0,0 +1,260 @@
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
# Code Review Reference - REVIEW_API
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=2, tag=REVIEW_API
## Archive Evidence Snapshot
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`.
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`.
- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0.
- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean.
- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged.
- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence.
## For the Review Agent
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
Review completion means the following steps are finished:
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G05.md` → `code_review_cloud_G05_2.log` and `PLAN-cloud-G05.md` → `plan_cloud_G05_2.log`.
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
---
## Implementation Item Completion
| Item | Status |
|------|---------|
| REVIEW_API-1 Require complete token matches | [x] |
| REVIEW_API-2 Re-run the benchmark regression boundary | [x] |
## Implementation Checklist
- [x] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior.
- [x] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G05_2.log`.
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G05_2.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path.
- [x] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
없음. PLAN이 선택한 두 지점만 수정했고 명령 교체나 검증 생략 없이 지정된 검증을 그대로 실행했다.
## Key Design Decisions
- 공유 token helper의 의미를 별도 seed 전용 validator 없이 `Pattern.fullmatch`로 일관되게 강화했다.
- 기존 schema pattern, seeded rank, canonical digest/omission 처리, slot allocation과 fixture는 변경하지 않았다.
- terminal newline seed는 기존 schema/loader parity corpus에 추가해 양쪽 경계가 같은 값을 거부하도록 고정했다.
## Reviewer Checkpoints
- Confirm `_require_pattern` uses a complete-string match and rejects `"seed-a\n"` while valid 64-character seeds remain accepted.
- Confirm the existing schema/loader parity corpus includes the terminal-newline value and asserts `ManifestValidationError`.
- Confirm the schema, seeded rank, canonical digest/omission behavior, and `RunStore.slots` remain unchanged.
- Confirm fresh focused and combined regression commands pass without invoking external providers.
## Verification Results
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
### REVIEW_API-1 Focused Verification
```bash
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
```
_Actual output:_
```text
exit_code: 0
stdout: (none)
stderr:
.
----------------------------------------------------------------------
Ran 1 test in 0.005s
OK
```
### REVIEW_API-1 Manifest Regression
```bash
python3 -m unittest scripts.agent_benchmark.manifest_test
```
_Actual output:_
```text
exit_code: 0
stdout: (none)
stderr:
...............................................................................................................
----------------------------------------------------------------------
Ran 111 tests in 1.224s
OK
```
### REVIEW_API-2 and Final Verification
```bash
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
git diff --check
```
_Actual output:_
```text
manifest validation loop:
exit_code: 0
stdout:
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
stderr: (none)
combined unittest:
exit_code: 0
stdout: (none)
stderr:
........................................................................................................................................................................................
----------------------------------------------------------------------
Ran 184 tests in 46.261s
OK
git diff --check:
exit_code: 0
stdout: (none)
stderr: (none)
```
### Reviewer Fresh Verification (2026-08-12)
The reviewer reran the required commands from `/config/workspace/iop-s0` without invoking external providers.
```text
focused parity test:
exit_code: 0
.
----------------------------------------------------------------------
Ran 1 test in 0.008s
OK
full manifest suite:
exit_code: 0
...............................................................................................................
----------------------------------------------------------------------
Ran 111 tests in 1.381s
OK
manifest validation loop:
exit_code: 0
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
combined unittest:
exit_code: 0
........................................................................................................................................................................................
----------------------------------------------------------------------
Ran 184 tests in 50.511s
OK
git diff --check:
exit_code: 0
stdout: (none)
stderr: (none)
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.
## Section Ownership
| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
### Overall Verdict
PASS
### Dimension Assessment
| Dimension | Result | Evidence |
|-----------|--------|----------|
| Correctness | Pass | `_require_pattern` now requires `Pattern.fullmatch`, so the terminal-newline seed is rejected before ordering or digest processing. |
| Completeness | Pass | Both REVIEW_API items are implemented and the selected R1 fix is present without schema, ordering, digest, slot, or fixture changes in this follow-up. |
| Test coverage | Pass | The existing schema/loader parity corpus now includes `"seed-a\n"`; the focused test, 111 manifest tests, and 184 combined tests pass freshly. |
| API contract | Pass | Loader behavior now agrees with the tracked seed pattern's full-string interpretation while valid shipped manifests remain accepted. |
| Code quality | Pass | The fix is localized to the shared token helper and reuses the existing parity test without duplicate validation logic. |
| Implementation deviation | Pass | The implementation follows the reviewer-selected files, symbols, behavior, and verification commands with no scope expansion. |
| Verification trust | Pass | Fresh reviewer output matches the implementation-owned evidence and `git diff --check` is clean. |
| Spec conformance | Pass | The reusable seed grammar remains consistent with SDD S03 and `milestone-task=matrix-lock`; concrete C01-C09 manifest evidence remains outside this packet. |
### Findings
None. Required 0, Suggested 0, Nit 0.
### Routing Signals
- `review_rework_count=1`
- `evidence_integrity_failure=false`
### Next Step
Write `complete.log`, archive this PASS task with `milestone-task=matrix-lock` metadata preserved, and emit the runtime completion event without modifying the roadmap.

0
agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log
View file

139
agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log
View file

@ -43,42 +43,46 @@ Review completion means the following steps are finished:
| Item | Status |
|------|---------|
| API-1 Add one canonical seeded-order contract | [ ] |
| API-2 Bind slot allocation to canonical matrix order | [ ] |
| API-3 Preserve the complete benchmark baseline | [ ] |
| API-1 Add one canonical seeded-order contract | [x] |
| API-2 Bind slot allocation to canonical matrix order | [x] |
| API-3 Preserve the complete benchmark baseline | [x] |
## Implementation Checklist
- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
- [x] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
- [x] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
- [x] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
None. The implementation followed the selected owner, files, algorithm, and verification commands. Production `RunStore.slots` remains unchanged.
## Key Design Decisions
_Record key design decisions here._
- Added `execution_order_seed: str | None = None` after `Manifest.digest`, so existing constructors remain compatible.
- Reused the cell-id token grammar for loader and schema validation. The 64-character boundary is accepted; empty, oversized, non-token, and non-string values fail closed.
- Applied the exact `iop-benchmark-order-v1\0<seed>\0<cell-id>` SHA-256 rank with cell id as the collision tie-breaker.
- Included the seed in canonical serialization only when explicitly present. Omission retains pipeline-v2 lexical cell ordering and legacy canonical bytes.
- Kept slot allocation single-sourced from `Manifest.matrix`; the new regression proves repetitions remain adjacent within each seeded cell.
## Reviewer Checkpoints
@ -99,6 +103,14 @@ python3 -m unittest scripts.agent_benchmark.manifest_test
_Actual output:_
```text
...............................................................................................................
----------------------------------------------------------------------
Ran 111 tests in 1.071s
OK
```
### API-2 Verification
```bash
@ -107,6 +119,14 @@ python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_
_Actual output:_
```text
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
```
### API-3 and Final Verification
```bash
@ -117,6 +137,54 @@ git diff --check
_Actual output:_
```text
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
.......................................................................................................................................................................................
----------------------------------------------------------------------
Ran 184 tests in 46.125s
OK
```
`git diff --check` produced no output and exited 0.
### Reviewer Fail-Closed Reproducer
```bash
python3 - <<'PY'
import json
import re
import tempfile
from pathlib import Path
from scripts.agent_benchmark.manifest_test import (
_load_tmp_manifest,
_make_minimal_manifest_dict,
_write_tmp_manifest,
)
seed = "seed-a\n"
schema = json.loads(Path("scripts/fixtures/agent-comparison-benchmark-manifest.schema.json").read_text(encoding="utf-8"))
pattern = schema["properties"]["execution_order_seed"]["pattern"]
with tempfile.TemporaryDirectory() as tmp:
manifest = _load_tmp_manifest(
_write_tmp_manifest(
Path(tmp),
_make_minimal_manifest_dict(execution_order_seed=seed),
)
)
print(f"schema_fullmatch={re.fullmatch(pattern, seed) is not None}")
print(f"loader_accepted={manifest.execution_order_seed!r}")
PY
```
_Actual output:_
```text
schema_fullmatch=False
loader_accepted='seed-a\n'
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
@ -136,3 +204,40 @@ _Actual output:_
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
### Overall Verdict
FAIL
### Dimension Assessment
| Dimension | Result | Evidence |
|-----------|--------|----------|
| Correctness | Fail | A trailing-newline seed bypasses the intended bounded token grammar and reaches canonical ordering/digest code. |
| Completeness | Fail | API-1 requires invalid non-token seeds to fail closed, but that boundary remains open. |
| Test coverage | Fail | The invalid-seed corpus omits the Python `$`/trailing-newline boundary and therefore passes while the defect remains. |
| API contract | Fail | The tracked schema pattern rejects the reproduced seed under the parity test's full-string interpretation, while the loader accepts it. |
| Code quality | Pass | The seeded ordering and conditional serialization are localized and readable. |
| Implementation deviation | Pass | The implementation otherwise follows the planned owner, algorithm, files, and slot-allocation boundary. |
| Verification trust | Fail | Fresh focused and full suites pass, but the implementation claim that non-token seed values fail closed is contradicted by the reviewer reproducer. |
| Spec conformance | Pass | The explicit seed, deterministic order, digest binding, and slot consumption otherwise conform to SDD S03's reusable order contract. |
### Findings
- **Required R1 — The loader accepts an `execution_order_seed` ending in `\n`.**
- **Evidence:** The fresh reviewer reproducer reports `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. `scripts/agent_benchmark/manifest.py:729` routes the seed through `_require_pattern`, while `scripts/agent_benchmark/manifest.py:194` uses `pattern.match(s)`; Python `$` permits a match immediately before one terminal newline. This violates API-1's explicit non-token fail-closed criterion and the schema/loader parity checkpoint.
- **Root Cause:** The shared manifest token validator tests only whether a prefix matches an anchored expression instead of requiring the entire string to match. The new seed field reused that helper, and `scripts/agent_benchmark/manifest_test.py:1492` did not include the terminal-newline variant in its invalid corpus.
- **Selected Fix:** Change `_require_pattern` in `scripts/agent_benchmark/manifest.py` to use `pattern.fullmatch(s)`, then add `"seed-a\n"` to `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` in `scripts/agent_benchmark/manifest_test.py`. Keep the schema and seeded-order algorithm unchanged. Accept only after the named parity test, full manifest suite, three example validations, manifest/attempt/connectivity regression suite, and `git diff --check` all pass freshly.
No Suggested or Nit findings.
### Routing Signals
- `review_rework_count=1`
- `evidence_integrity_failure=true`
### Next Step
Create the mandatory reviewer-closed follow-up plan for Required R1, rerun isolated task routing, archive this pair, and materialize the routed follow-up pair. Do not write `complete.log`.

41
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log Normal file
View file

@ -0,0 +1,41 @@
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
# Complete - m-iop-one-shot-agent-model-comparison/01_execution_order_contract
## 완료 일시
2026-08-12
## 요약
실행 순서 seed 계약과 terminal-newline fail-closed 보완을 공식 리뷰 2회(FAIL 1회, PASS 1회)로 완료했으며 최종 판정은 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | FAIL | seeded order/digest/slot 계약은 구현됐으나 `"seed-a\n"`이 loader token 검증을 우회하는 Required R1을 확인했다. |
| `plan_cloud_G05_2.log` | `code_review_cloud_G05_2.log` | PASS | 공유 pattern helper를 full-string 검증으로 강화하고 terminal-newline parity regression을 추가해 R1을 닫았다. |
## 구현/정리 내용
- optional `execution_order_seed`가 manifest canonical order와 digest에 결정론적으로 반영되고 omission 호환성과 `RunStore.slots` 순서가 유지된다.
- `_require_pattern`이 `Pattern.fullmatch`를 사용해 seed를 포함한 manifest token 전체 문자열을 fail-closed 검증한다.
- schema/loader parity corpus에 `"seed-a\n"`을 추가했으며 schema, seeded ranking, digest, slot allocation과 shipped fixture는 follow-up 범위에서 변경하지 않았다.
## 최종 검증
- `python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` - PASS; 1 test, `OK`.
- `python3 -m unittest scripts.agent_benchmark.manifest_test` - PASS; 111 tests, `OK`.
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`.
- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 184 tests, `OK`.
- `git diff --check` - PASS; 출력 없음.
- 공개 `validate` entrypoint를 실제 fixture로 검증했다. 외부 provider 실행과 scored benchmark full-cycle은 이 parser-only packet의 검증 범위가 아니다.
## 잔여 Nit
- 없음
## 후속 작업
- dependent `04+02,03_locked_benchmark_manifest` packet이 실제 C01-C09 immutable manifest evidence를 제공해야 하며, 이 완료 로그 단독으로 `matrix-lock` 전체 완료를 주장하지 않는다.

198
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_cloud_G05_2.log Normal file
View file

@ -0,0 +1,198 @@
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
# Plan - Close Execution-Order Seed Token Validation
## For the Implementing Agent
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G05.md` is the mandatory final implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
## Background
The seeded execution-order implementation is otherwise complete, but the official review found one fail-closed boundary defect. Python's anchored `$` plus `Pattern.match` accepts a single trailing newline, so the loader can accept a seed that the declared full token grammar rejects. This follow-up applies the reviewer-selected full-string validation fix and locks the exact regression.
## Archive Evidence Snapshot
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`.
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`.
- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0.
- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean.
- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged.
- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence.
## Finding Resolution Map
| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands |
|---------|-------------------|------------|--------------|------|----------------------|---------------------|
| Required R1 | Fresh reproducer: `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. | `_require_pattern` uses `pattern.match(s)`, whose `$` can stop before one terminal newline; the invalid seed corpus omits that value. | Use `pattern.fullmatch(s)` in `scripts/agent_benchmark/manifest.py` and add `"seed-a\n"` to the invalid seed corpus in `scripts/agent_benchmark/manifest_test.py`. Keep schema and ordering behavior unchanged. | `direct-fix` | The validator implementation and regression input change, so verification no longer repeats against the defective precondition. | Named parity test; full manifest suite; three example validations; manifest/attempt/connectivity suite; `git diff --check`. |
## Analysis
### Files Read
- `scripts/agent_benchmark/manifest.py`
- `scripts/agent_benchmark/manifest_test.py`
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- `agent-spec/testing/agent-comparison-benchmark.md`
- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`
- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`
### SDD Criteria
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, lock released, no user review.
- First-line scope: `milestone-task=matrix-lock`.
- Target: S03 requires the immutable scored manifest to retain a deterministic execution-order seed; the Evidence Map links that evidence to `matrix-lock`.
- This checklist repairs the reusable seed grammar needed by S03. The dependent locked-manifest packet still owns concrete C01-C09 evidence.
### Verification Context
- No external handoff or external runner is required. Repository-native local verification is authoritative.
- Fresh review evidence: 111 manifest tests passed, the named slot test passed, three shipped manifests validated, the combined suite passed 184 tests, and `git diff --check` exited 0.
- The focused reproducer accepted `"seed-a\n"` in the loader while the schema pattern's full-string check returned false.
- Constraints: preserve valid seed behavior, omission compatibility, seeded rank, digest serialization, `RunStore.slots`, fixture bytes, and public schema.
- Confidence: high. The defect is localized to one shared pattern helper and one missing regression value; all helper patterns are intended as whole-string token validators.
- External verification preflight: not applicable.
### Test Coverage Gaps
- Existing seed tests cover empty, uppercase, dotted, oversized, boolean, and null values.
- Missing: the terminal-newline value that distinguishes `Pattern.match` from `Pattern.fullmatch`.
- Add that value to the existing schema/loader parity corpus; no new test class or external fixture is required.
### Symbol References
- No symbol is renamed or removed.
- `_require_pattern` remains the shared private validator for fixture version, model/effort/route tokens, viewport ids, cell ids, and `execution_order_seed`. The change tightens those existing whole-string grammars consistently.
### Split Judgment
This is one compact atomic bug fix: the validator semantic change and its regression must land together. It remains child `01_execution_order_contract`; no new dependency or sibling split is introduced.
### Scope Rationale
- Do not change `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`; its seed grammar is already correct.
- Do not change seeded ranking, canonical serialization/digest, `RunStore.slots`, attempt state, example manifests, contracts, living spec, or roadmap.
- Do not broaden the packet into generic path normalization; only the shared regex token helper and the exact seed regression are writable production/test targets.
### Final Routing
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`.
- Build closures: scope/context/verification/evidence/ownership/decision all true; `ownership_closed=true` through direct-fix R1.
- Build scores: `1+0+2+1+1=G05`; base `local-fit`, final `recovery-boundary` because `review_rework_count=1` and `evidence_integrity_failure=true`; lane `cloud`, catalog `worker/cloud/G05`, filename `PLAN-cloud-G05.md`.
- Review closures are all true; scores `1+0+2+1+1=G05`; route `official-review`, lane `cloud`, catalog `review/cloud/G05`, filename `CODE_REVIEW-cloud-G05.md`.
- `large_indivisible_context=false`; matched loop risks: `boundary_contract`, `structured_interpretation` (`count=2`); `risk_boundary_matched=false`; `recovery_boundary_matched=true`; no capability gap.
## Implementation Checklist
- [ ] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior.
- [ ] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [REVIEW_API-1] Require complete token matches
**Problem**
At `scripts/agent_benchmark/manifest.py:192-198`, `_require_pattern` calls `pattern.match(s)`. With the anchored patterns used by the manifest, Python accepts a match immediately before a final newline, so `execution_order_seed="seed-a\n"` reaches ordering and digest code even though it is not a valid bounded token.
**Solution**
Apply the official review's selected fix exactly.
Before (`scripts/agent_benchmark/manifest.py:192-198`):
```python
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
s = _require_str(value, field_name)
if not pattern.match(s):
raise ManifestValidationError(
f"field '{field_name}' does not match required pattern"
)
return s
```
After:
```python
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
s = _require_str(value, field_name)
if pattern.fullmatch(s) is None:
raise ManifestValidationError(
f"field '{field_name}' does not match required pattern"
)
return s
```
Add `"seed-a\n"` to the existing `invalid_seeds` tuple in `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds`. Do not change the schema pattern or add an alternate validator.
**Modified Files and Checklist**
- [ ] `scripts/agent_benchmark/manifest.py`: replace prefix-capable pattern matching with full-string matching.
- [ ] `scripts/agent_benchmark/manifest_test.py`: add the terminal-newline seed to the existing parity corpus.
**Test Strategy**
Write the regression in the existing named parity test. It must prove the tracked pattern's full-string interpretation rejects the seed and `load_manifest` raises `ManifestValidationError` for the same value. Existing valid/boundary tests cover compatibility.
**Verification**
```bash
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
python3 -m unittest scripts.agent_benchmark.manifest_test
```
Expected: both commands exit 0; the focused test rejects `"seed-a\n"` and the full manifest suite remains green.
### [REVIEW_API-2] Re-run the benchmark regression boundary
**Problem**
`_require_pattern` is shared by multiple manifest token fields, so the one-line fix must be checked against the full loader and the benchmark consumers that bind the manifest digest and matrix order.
**Solution**
Run the exact fresh commands below after REVIEW_API-1. Do not alter product behavior or replace the failing value with a sanitized seed.
**Modified Files and Checklist**
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md`: record fresh command output and implementation notes.
**Test Strategy**
No additional test file is needed beyond REVIEW_API-1. The shipped manifests and combined deterministic suite are the regression oracle; no external provider process may run.
**Verification**
```bash
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
git diff --check
```
Expected: three validation success lines, every test reports `OK`, and `git diff --check` is silent.
## Modified Files Summary
| File | Item |
|------|------|
| `scripts/agent_benchmark/manifest.py` | REVIEW_API-1 |
| `scripts/agent_benchmark/manifest_test.py` | REVIEW_API-1 |
| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md` | REVIEW_API-2 |
## Final Verification
Run from `/config/workspace/iop-s0`. Fresh output is required; cached or summarized output is not accepted.
```bash
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
python3 -m unittest scripts.agent_benchmark.manifest_test
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
git diff --check
```
Expected: all commands exit 0; the newline seed is rejected, all three manifests validate, all deterministic benchmark tests report `OK`, and the whitespace check is silent.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

0
agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log
View file

154
agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_1.log
View file

@ -43,44 +43,46 @@ Review completion means the following steps are finished:
| Item | Status |
|------|---------|
| API-1 Require exact all-cell preflight evidence | [ ] |
| API-2 Isolate live agy preflight state per cell | [ ] |
| API-3 Replace obsolete direct-only code and skill assertions | [ ] |
| API-4 Preserve append-only and redaction invariants | [ ] |
| API-1 Require exact all-cell preflight evidence | [x] |
| API-2 Isolate live agy preflight state per cell | [x] |
| API-3 Replace obsolete direct-only code and skill assertions | [x] |
| API-4 Preserve append-only and redaction invariants | [x] |
## Implementation Checklist
- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
- [ ] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
- [x] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
- [x] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
- [x] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
- [x] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`.
- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`.
- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path.
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
None.
## Key Design Decisions
_Record key design decisions here._
- Preflight collection and durable record validation use the same immutable `manifest.matrix` tuple, so cardinality, identity, and order are checked against every declared direct or execution-preset cell.
- `_LiveAdapter` clears prior admission before every cell preflight and publishes agy runtime state only after the complete live/config observation is ready. Invocation retrieves that state once by exact cell id and rejects missing or mismatched bindings.
- The project skill and its mutation tests now require affirmative all-cell live evidence while retaining closed blocker taxonomy, zero-attempt blockers, and the rule that local fixture validation alone is not live readiness.
## Reviewer Checkpoints
@ -106,6 +108,28 @@ python3 -m unittest \
_Actual output:_
```text
....
----------------------------------------------------------------------
Ran 4 tests in 2.414s
OK
```
Exit code: `0`.
_Reviewer fresh rerun:_
```text
....
----------------------------------------------------------------------
Ran 4 tests in 2.492s
OK
```
Exit code: `0`.
### API-3 Verification
```bash
@ -114,6 +138,28 @@ python3 -m unittest scripts.agent_benchmark.connectivity_integration_test script
_Actual output:_
```text
.................................................................................
----------------------------------------------------------------------
Ran 81 tests in 14.040s
OK
```
Exit code: `0`.
_Reviewer fresh rerun:_
```text
.................................................................................
----------------------------------------------------------------------
Ran 81 tests in 13.933s
OK
```
Exit code: `0`.
### API-4 and Final Verification
```bash
@ -123,6 +169,36 @@ git diff --check
_Actual output:_
```text
$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
.........................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 233 tests in 47.536s
OK
$ git diff --check
(no output)
```
Both commands exited `0`.
_Reviewer fresh rerun:_
```text
$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
.........................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 233 tests in 47.606s
OK
$ git diff --check
(no output)
```
Both commands exited `0`.
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
@ -142,3 +218,37 @@ _Actual output:_
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
### Overall Verdict
PASS
### Dimension Assessment
| Dimension | Assessment | Evidence |
|-----------|------------|----------|
| Correctness | Pass | Full-matrix collection/write/read order, blocker handling, and cell-keyed agy admission matched the planned invariants. |
| Completeness | Pass | API-1 through API-4 are implemented, and the public skill contract was updated to all-cell semantics. |
| Test coverage | Pass | Fresh focused 4-test, connectivity/skill 81-test, and full 233-test suites passed. |
| API contract | Pass | No caller wire/config contract changed; direct and execution-preset bindings remain exact and fail closed without substitution. |
| Code quality | Pass | No stale direct-only production reference remains after the review-time CLI docstring repair; `git diff --check` passed. |
| Implementation deviation | Pass | Planned implementation scope is present; unrelated sibling execution-order work was excluded from this verdict. |
| Verification trust | Pass | Reviewer reran every planned command from the current worktree and reproduced the recorded success. |
| Spec conformance | Pass | SDD S02 all-cell redacted preflight evidence and the `route-readiness` Evidence Map are satisfied by deterministic contract tests without claiming live credential readiness. |
### Findings
- Nit (repaired): `scripts/agent_comparison_benchmark.py:14` still described exit 0 as requiring every “direct preflight cell”; review changed it to “every matrix preflight cell” to match the implemented public semantics.
- Required: None.
- Suggested: None.
### Routing Signals
- `review_rework_count=0`
- `evidence_integrity_failure=false`
### Next Step
- PASS: archive this pair, write `complete.log`, and emit the milestone completion metadata for runtime aggregation.

40
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log Normal file
View file

@ -0,0 +1,40 @@
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=1 tag=API milestone-task=route-readiness -->
# Complete - m-iop-one-shot-agent-model-comparison/02_route_preflight_contract
## 완료 일시
2026-08-12
## 요약
계획 개정 2회와 공식 코드 리뷰 1회를 거쳐 전체 matrix cell preflight 및 cell별 agy 상태 격리 계약을 구현했고 최종 판정은 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | 재계획 | Epic self-review에서 project benchmark skill과 contract test 범위 누락을 발견해 구현 전 계획을 보강했다. |
| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | PASS | 전체 cell의 append-only preflight, blocker 무할당, cell별 agy 상태 격리 및 skill 계약을 확인했다. |
## 구현/정리 내용
- preflight 수집과 durable record 검증을 immutable `manifest.matrix` 전체 및 canonical order 기준으로 통일했다.
- agy live preflight/admission 상태를 cell id별로 격리하고 재-preflight 시 stale 상태를 제거했다.
- mixed/preset-only 성공·blocker·corruption·cell isolation 회귀 테스트와 project benchmark skill 계약을 all-cell 의미로 갱신했다.
- 리뷰 중 공개 CLI 종료 코드 설명의 남은 direct-only 문구를 matrix 전체 의미로 정정했다.
## 최종 검증
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state` - PASS; 4 tests, 2.492s.
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 81 tests, 13.933s.
- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 233 tests, 47.606s.
- `git diff --check` - PASS; 출력 없음.
## 잔여 Nit
- 없음
## 후속 작업
- 없음

0
agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_1.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log
View file

141
agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_1.log
View file

@ -43,44 +43,47 @@ Review completion means the following steps are finished:
| Item | Status |
|------|---------|
| API-1 Add a closed dual-version rubric catalog | [ ] |
| API-2 Bind prompt and durable validation to manifest version | [ ] |
| API-3 Preserve historical compatibility and exact new semantics | [ ] |
| API-4 Run the complete benchmark regression | [ ] |
| API-1 Add a closed dual-version rubric catalog | [x] |
| API-2 Bind prompt and durable validation to manifest version | [x] |
| API-3 Preserve historical compatibility and exact new semantics | [x] |
| API-4 Run the complete benchmark regression | [x] |
## Implementation Checklist
- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric.
- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
- [x] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
- [x] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
- [x] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric.
- [x] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path.
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
없음. 지정된 파일 범위와 검증 명령을 그대로 수행했으며, project benchmark skill과 living spec은 변경하지 않았다.
## Key Design Decisions
_Record key design decisions here._
- `RUBRIC_VERSION`과 `RUBRIC_CATEGORIES`를 legacy alias/table로 그대로 유지하고, `ONE_SHOT_RUBRIC_VERSION`, `RUBRIC_VERSIONS`, immutable `RUBRIC_CATEGORIES_BY_VERSION` catalog를 additive하게 추가했다.
- worksheet가 선언한 version의 ordered table로 먼저 검증되고, `expected_version`이 있으면 exact match를 추가로 요구하도록 해 unknown·cross-version 출력을 fail-closed했다.
- evaluator prompt와 신규/내구 score reload의 두 worksheet load path 모두 `manifest.rubric_version`을 사용한다. legacy prompt exact bytes와 canonical worksheet SHA-256도 회귀 테스트로 고정했다.
- 새-version 성공과 structurally valid legacy worksheet의 cross-version 실패를 각각 fresh score attempt로 검증했으며 실패 result에는 scored worksheet를 publish하지 않음을 확인했다.
## Reviewer Checkpoints
@ -108,6 +111,10 @@ PY
_Actual output:_
```text
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log
```
### API-1 Verification
```bash
@ -116,6 +123,14 @@ python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.
_Actual output:_
```text
......................................................................................................................
----------------------------------------------------------------------
Ran 118 tests in 1.181s
OK
```
### API-2 Verification
```bash
@ -124,6 +139,14 @@ python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manife
_Actual output:_
```text
.
----------------------------------------------------------------------
Ran 1 test in 0.448s
OK
```
### API-3 Verification
```bash
@ -132,6 +155,14 @@ python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.
_Actual output:_
```text
.......................................................................................................................................
----------------------------------------------------------------------
Ran 135 tests in 5.746s
OK
```
### API-4 and Final Verification
```bash
@ -142,6 +173,56 @@ git diff --check
_Actual output:_
```text
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 208 tests in 51.235s
OK
```
`git diff --check`: exit 0, 출력 없음.
### Reviewer Fresh Verification
2026-08-12 공식 리뷰에서 아래 검증을 현재 worktree 기준으로 다시 실행했다.
```text
go version go1.26.2 linux/arm64
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
Ran 118 tests in 1.202s
OK
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
Ran 1 test in 0.453s
OK
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
Ran 135 tests in 5.848s
OK
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
Ran 208 tests in 50.790s
OK
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
Ran 432 tests in 119.960s
OK
git diff --check
exit 0, no output
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
@ -161,3 +242,19 @@ _Actual output:_
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
- Overall Verdict: PASS
- Dimension Assessment:
- Correctness: Pass — manifest-selected rubric lookup, prompt rendering, output validation, and durable reload validation preserve one exact version/table binding.
- Completeness: Pass — API-1 through API-4 are implemented within the planned files; the dependent manifest packet remains the owner of the first tracked new-version manifest.
- Test coverage: Pass — focused dual-version tests, the planned 208-test regression, and the 432-test full benchmark discovery all pass.
- API contract: Pass — `landing-quality-v1` remains byte-compatible while the schema and loader expose only the closed two-version catalog.
- Code quality: Pass — immutable tuples plus `MappingProxyType` keep the catalog closed; no stale production literal, debug print, TODO, or whitespace error was found.
- Implementation deviation: Pass — no deviation from the revalidated plan; unrelated sibling-task worktree changes were excluded from this verdict.
- Verification trust: Pass — implementation-reported results were independently reproduced with fresh reviewer commands.
- Spec conformance: Pass — SDD S01's seven-category 100-point rubric and `fixture-lock` evidence boundary match the implementation; concrete manifest/digest evidence remains assigned to the dependent manifest packet.
- Findings: None
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock` for runtime aggregation.

43
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log Normal file
View file

@ -0,0 +1,43 @@
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=1 tag=API milestone-task=fixture-lock -->
# Complete - m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract
## 완료 일시
2026-08-12
## 요약
구현 전 ownership 충돌 재계획 1회 뒤 dual-version rubric 계약 구현을 공식 리뷰 1회로 검증했으며 최종 판정은 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_local_G08_0.log` | `code_review_cloud_G08_0.log` | 재계획 | project benchmark skill 수정 ownership 충돌을 구현 전에 제거했으며 공식 판정은 없었다. |
| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | PASS | legacy rubric을 보존한 closed dual-version catalog, manifest-selected prompt/worksheet 검증, 전체 benchmark 회귀를 확인했다. |
## 구현/정리 내용
- `landing-quality-v1`의 기존 category tuple과 canonical worksheet 의미를 유지하면서 `one-shot-agent-comparison-v1`의 SDD 승인 7개 category를 immutable catalog에 추가했다.
- manifest schema/loader, evaluator prompt, 최초 score publication과 durable score reload가 같은 manifest-selected rubric version을 사용하고 unknown/cross-version worksheet를 fail-closed한다.
- dual-version manifest/rubric/scoring regression과 legacy prompt/worksheet compatibility assertion을 추가했으며 운영 skill과 첫 tracked new-version manifest 전환은 후속 packet 범위로 유지했다.
## 최종 검증
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test` - PASS; 118 tests, `OK`.
- `python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation` - PASS; 1 test, `OK`.
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test` - PASS; 135 tests, `OK`.
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`.
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 208 tests, `OK`.
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 432 tests, `OK`.
- `git diff --check` - PASS; 출력 없음.
- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic benchmark rubric/schema packet 범위가 아니어서 실행하지 않았다.
## 잔여 Nit
- 없음
## 후속 작업
- dependent `04+02,03_locked_benchmark_manifest` packet이 첫 tracked `one-shot-agent-comparison-v1` manifest와 concrete rubric digest evidence를 제공해야 하며, 이 완료 로그 단독으로 `fixture-lock` 전체 완료를 주장하지 않는다.

0
agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_1.log
View file

135
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G06_2.log
View file

@ -43,40 +43,44 @@ Review completion means the following steps are finished:
| Item | Status |
|------|---------|
| TEST-1 Create one exact immutable readiness manifest | [ ] |
| TEST-3 Preserve the full benchmark baseline | [ ] |
| TEST-1 Create one exact immutable readiness manifest | [x] |
| TEST-3 Preserve the full benchmark baseline | [x] |
## Implementation Checklist
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
- [x] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
- [x] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`.
- [ ] Archive active `PLAN-local-G06.md` to `plan_local_G06_2.log`.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, keep parent `agent-task/m-iop-one-shot-agent-model-comparison/` because child `05+04_readiness_preflight` remains active until its own review completes.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`.
- [x] Archive active `PLAN-local-G06.md` to `plan_local_G06_2.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path.
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, keep parent `agent-task/m-iop-one-shot-agent-model-comparison/` because child `05+04_readiness_preflight` remains active until its own review completes.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
- 없음. PLAN의 파일 범위와 검증 명령을 그대로 유지했다.
- 보조 검증으로 `skill-creator`의 `quick_validate.py`를 추가 실행했으며 계획된 검증을 대체하지 않았다.
## Key Design Decisions
_Record key design decisions here._
- 새 manifest의 JSON matrix는 C01-C09 순서로 읽기 쉽게 유지하고, loader가 explicit seed로 산출한 immutable 실행 순서는 별도로 exact assertion했다.
- manifest regression은 raw JSON 전체 cell payload와 loader의 canonical payload를 모두 비교해 route kind/id, request effort, 다섯 stage 순서, cloud effort, `ornith-fast` effort 생략을 함께 잠갔다.
- scoring skill 문구는 기존 Procedure의 generic fallback-language 금지 계약을 유지하기 위해 `no substitute rubric or reinterpretation`으로 표현하고, 닫힌 두 rubric version literal과 manifest-selected semantics를 contract test로 고정했다.
- Spec update not needed: 기존 living spec의 manifest-selected rubric 및 immutable manifest 설명과 동작 범위는 그대로 정확하며 새 tracked bench-02 입력만 추가됐다.
## Reviewer Checkpoints
@ -108,6 +112,11 @@ PY
_Actual output:_
```text
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log
```
### TEST-1 Verification
```bash
@ -118,6 +127,27 @@ python3 -m unittest scripts.agent_benchmark.skill_contract_test
_Actual output:_
```text
ok: manifest is valid
.
----------------------------------------------------------------------
Ran 1 test in 0.003s
OK
..................................................
----------------------------------------------------------------------
Ran 50 tests in 1.705s
OK
```
Supplemental skill validation:
```text
$ python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/iop-agent-comparison-benchmark
Skill is valid!
```
### TEST-3 and Final Verification
```bash
@ -128,6 +158,59 @@ git diff --check
_Actual output:_
```text
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
..................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 434 tests in 118.623s
OK
git diff --check: no output (exit 0)
```
### Reviewer Fresh Verification (2026-08-12)
```text
$ python3 - <<'PY' # exact dependency verification block above
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log
$ python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
ok: manifest is valid
$ python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
.
----------------------------------------------------------------------
Ran 1 test in 0.004s
OK
$ python3 -m unittest scripts.agent_benchmark.skill_contract_test
..................................................
----------------------------------------------------------------------
Ran 50 tests in 1.716s
OK
$ for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
ok: manifest is valid
$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
----------------------------------------------------------------------
Ran 434 tests in 124.077s
OK
$ git diff --check
(no output; exit 0)
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
@ -147,3 +230,19 @@ _Actual output:_
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
- Overall Verdict: PASS
- Dimension Assessment:
- Correctness: Pass — tracked manifest fields, fixture checksum, two image paths, evaluator, C01-C09 direct/preset bindings, omitted work-stage effort, and seeded order match the approved plan and SDD.
- Completeness: Pass — TEST-1 and TEST-3 are implemented; both predecessor completion records resolve exactly once, and external readiness remains correctly isolated in child `05+04_readiness_preflight`.
- Test coverage: Pass — fresh focused manifest/skill tests, four-manifest validation, and all 434 discovered benchmark tests pass.
- API contract: Pass — direct and execution-preset route identities preserve the OpenAI/Anthropic ingress and Edge config ownership boundaries without fallback or substitution.
- Code quality: Pass — the exact regression compares raw and canonical payloads, the skill contract mutation-tests legacy-only wording, and `git diff --check` is clean.
- Implementation deviation: Pass — planned production/documentation files are present; the supplemental skill validation did not replace required verification, and predecessor/sibling worktree changes were excluded from this verdict.
- Verification trust: Pass — every applicable plan command was rerun against the current worktree and reproduced the implementation handoff; no `agent-test/runs/bench-02` execution state exists.
- Spec conformance: Pass — SDD S01/S03 and Evidence Map entries for `fixture-lock,matrix-lock` are satisfied; the current living spec already describes immutable manifest-selected rubric behavior, so a spec update is not needed.
- Findings: None
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock,matrix-lock` for runtime aggregation.

0
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G09_1.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G09_1.log
View file

45
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log Normal file
View file

@ -0,0 +1,45 @@
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=2 tag=TEST milestone-task=fixture-lock,matrix-lock -->
# Complete - m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest
## 완료 일시
2026-08-12
## 요약
두 차례 구현 전 범위 정리와 한 차례 공식 코드 리뷰를 거쳐 immutable C01-C09 manifest와 manifest-selected rubric skill 계약을 검증했으며 최종 판정은 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | 재계획 | Epic self-review에서 rubric skill 동기화와 외부 readiness gate 보강 필요를 확인했으며 구현 및 공식 판정은 없었다. |
| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | 분할 재계획 | 외부 readiness preflight를 child `05+04_readiness_preflight`로 분리하고 현재 child를 deterministic manifest 계약으로 한정했다. |
| `plan_local_G06_2.log` | `code_review_cloud_G06_2.log` | PASS | fixture checksum, C01-C09 payload와 seed 순서, dual-rubric skill 계약 및 전체 434-test 회귀를 확인했다. |
## 구현/정리 내용
- 승인된 fixture, 두 이미지, viewport, timeout, evaluator와 C01-C09 direct/execution-preset binding을 하나의 tracked bench-02 manifest로 고정했다.
- raw manifest와 canonical loader payload, explicit-seed 실행 순서, fixture checksum 및 manifest digest를 exact regression으로 잠갔다.
- benchmark project skill이 manifest-selected rubric의 닫힌 두 version만 사용하고 legacy fallback 또는 재해석을 금지하도록 contract test와 함께 동기화했다.
- 기존 living spec은 immutable manifest-selected rubric 동작을 이미 정확히 설명하므로 갱신하지 않았다.
## 최종 검증
- `python3 - <<'PY' ... PY` - PASS; 선행 `02_route_preflight_contract`, `03+01_rubric_version_contract` 완료 로그가 각각 정확히 한 건 확인됐다.
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`.
- `python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness` - PASS; 1 test, `OK`.
- `python3 -m unittest scripts.agent_benchmark.skill_contract_test` - PASS; 50 tests, `OK`.
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 네 manifest 모두 유효하다.
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 434 tests, `OK`.
- `git diff --check` - PASS; 출력 없음.
- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic manifest child 범위가 아니며 외부 readiness는 child `05+04_readiness_preflight`가 소유하므로 실행하지 않았다.
## 잔여 Nit
- 없음
## 후속 작업
- `05+04_readiness_preflight`가 이 reviewed manifest를 사용해 별도 external readiness gate를 수행한다.

0
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G06_2.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log
View file

100
agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/code_review_cloud_G09_0.log
View file

@ -42,38 +42,40 @@ Review completion means the following steps are finished:
| Item | Status |
|------|---------|
| TEST-2 Produce one redacted nine-cell readiness record | [ ] |
| TEST-2 Produce one redacted nine-cell readiness record | [x] |
## Implementation Checklist
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
- [x] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`.
- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`.
- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` and update this checklist at the final archive path.
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
- The fixed gate's unchanged setup commands were executed in dependency, testbed, and artifact segments so execution could stop at the first exact blocker. The artifact segment exited 69 at the current-HEAD freshness check, so the later caller/version/help, environment-reference, public preflight, and durable-record commands were not executed. No acceptance condition or command predicate was changed.
## Key Design Decisions
_Record key design decisions here._
- Preserved `../iop-s2` as a read-only independent testbed and captured its exact clean `dev` HEAD without changing artifacts, configuration, credentials, or route registration.
- Honored the fail-closed boundary: no caller/model/preset/effort substitution was attempted, and the public benchmark preflight was not invoked after the setup blocker.
- Made no production, manifest, fixture, benchmark runtime, adapter, skill, test, scoring, reporting, or external testbed change; this review evidence file is the only implementation write.
## Reviewer Checkpoints
@ -199,6 +201,56 @@ fi
_Actual safe output or exact blocker and resume condition:_
```text
dependency check (exit 0):
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log
testbed provenance check (exit 0):
ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
artifact check (exit 69):
blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD
```
Exact blocker: `../iop-s2/build/bin/iop-edge` did not satisfy the required modification-time freshness bound for clean testbed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`. The fixed gate stopped before accepting its Linux AArch64 ELF identity or executability as current-HEAD evidence.
Resume condition: from the exact clean `../iop-s2` `dev` HEAD, rebuild `build/bin/iop-edge` as a Linux AArch64 ELF artifact and ensure both required Edge/Node artifacts satisfy the fixed current-HEAD freshness, ELF, and host-execution checks. Then rerun the fixed gate from the dependency check. The later agy version/help, benchmark environment-reference, and public nine-cell preflight checks remain unexecuted and must also pass in that resumed gate; do not infer their current state from planning evidence.
Not executed after the first blocker:
```text
caller version/help checks: not run
benchmark environment-reference check: not run
public benchmark preflight: not invoked (no run id created)
durable nine-result and secret-absence inspection: not run
scored run/resume/score/report commands: not run
```
### Reviewer Fresh Verification (2026-08-12)
The reviewer reran the read-only dependency, testbed provenance, and artifact setup sequence. It stopped at the same first blocker without invoking the public preflight or any caller/provider:
```text
exit 69
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log
ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD
```
Focused repository checks:
```text
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
ok: manifest is valid
manifest inspection
ok: manifest_order=c02-claude-gemini-direct,c05-codex-gpt-direct,c03-agy-gemini-direct,c06-claude-gemini-hybrid,c08-claude-gpt-hybrid,c09-codex-gpt-hybrid,c01-claude-sonnet-direct,c07-agy-gemini-hybrid,c04-claude-gpt-direct
ok: cell_prefixes=c01,c02,c03,c04,c05,c06,c07,c08,c09 unique=9 agy_known_version=1.1.11
git diff --check
ok: no output
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
@ -218,3 +270,19 @@ _Actual safe output or exact blocker and resume condition:_
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
- Overall Verdict: PASS
- Dimension Assessment:
- Correctness: Pass — the fixed setup gate stopped at the first exact current-HEAD artifact blocker and did not bypass or substitute the blocked route.
- Completeness: Pass — TEST-2 explicitly accepts either nine ready results or an exact safe blocker and resume condition; the latter is complete and independently reproduced.
- Test Coverage: Pass — the reviewer reproduced the blocker, validated the locked manifest, confirmed one-to-one C01-C09 coverage, and ran `git diff --check`.
- API Contract: Pass — no caller/provider request was made after the prerequisite failure, and no route/model/effort/auth contract was substituted.
- Code Quality: Pass — this child changes only its review evidence and contains no debug artifact, raw runtime value, or unrelated production edit.
- Implementation Deviation: Pass — segmented execution preserves the plan's fixed order and fail-closed semantics; later checks were correctly not executed.
- Verification Trust: Pass — fresh reviewer output matches the recorded predecessor path, clean testbed HEAD, and exact artifact blocker.
- Spec Conformance: Pass — SDD S02 permits an exact blocker, and the evidence records the safe blocker and deterministic resume condition without asserting Milestone Task completion.
- Findings: None
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
- Next Step: PASS — write `complete.log`, archive this pair and task directory, and report `milestone-task=route-readiness` only as runtime aggregation metadata.

39
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log Normal file
View file

@ -0,0 +1,39 @@
<!-- task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight plan=0 tag=TEST milestone-task=route-readiness -->
# Complete - m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight
## 완료 일시
2026-08-12
## 요약
한 차례 구현과 공식 코드 리뷰에서 고정 readiness gate의 첫 외부 artifact blocker와 재개 조건을 secret-safe하게 확정했으며 최종 판정은 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | PASS | clean `../iop-s2` dev HEAD의 stale `iop-edge` artifact에서 fail-closed했고, exact blocker와 전체 gate 재개 조건을 독립 재현했다. |
## 구현/정리 내용
- 선행 manifest child의 완료 로그가 정확히 한 건임을 확인하고 독립 testbed의 branch, HEAD, clean 상태를 기록했다.
- current-HEAD artifact freshness gate의 첫 실패를 안전하게 기록하고 caller/version/help, 환경 reference, public preflight와 scored 명령을 실행하지 않았다.
- `route-readiness`는 Milestone 완료 선언이 아닌 기여 범위 metadata로만 보존했다.
## 최종 검증
- `TEST-2 dependency/testbed/artifact setup sequence` - BLOCKED(expected accepted outcome); exit 69, `../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD`, recorded HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 구현 evidence가 일치했다.
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`.
- `python3 - <<'PY' ... load_manifest/AGY_KNOWN_VERSION inspection ... PY` - PASS; C01-C09 prefix가 각각 정확히 한 번 존재하고 known agy version은 `1.1.11`이다.
- `git diff --check` - PASS; 출력 없음.
- 공개 benchmark preflight, repo 내부 edge-node 진단, 보조 E2E smoke와 provider full-cycle은 artifact prerequisite에서 fail-closed했으므로 실행하지 않았다.
## 잔여 Nit
- 없음
## 후속 작업
- 정확한 clean `../iop-s2` dev HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller/version/help, benchmark environment reference와 public nine-cell preflight를 포함한 고정 gate 전체를 다시 실행해야 한다.

0
agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/plan_cloud_G09_0.log
View file

30
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/work_log_0.log Normal file
View file

@ -0,0 +1,30 @@
# Milestone Work Log
> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.
| seq | time | event | task | loop | role | attempt | model | result | locator |
|---:|---|---|---|---:|---|---:|---|---|---|
| 1 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json |
| 2 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json |
| 3 | 26-08-12 03:36:10 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json |
| 4 | 26-08-12 03:36:10 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json |
| 5 | 26-08-12 03:38:25 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json |
| 6 | 26-08-12 03:38:25 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json |
| 7 | 26-08-12 03:45:17 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json |
| 8 | 26-08-12 03:46:32 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json |
| 9 | 26-08-12 03:46:32 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json |
| 10 | 26-08-12 03:50:49 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json |
| 11 | 26-08-12 03:50:50 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json |
| 12 | 26-08-12 03:57:29 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json |
| 13 | 26-08-12 03:57:29 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json |
| 14 | 26-08-12 04:06:20 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json |
| 15 | 26-08-12 04:06:21 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json |
| 16 | 26-08-12 04:14:46 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json |
| 17 | 26-08-12 04:14:46 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json |
| 18 | 26-08-12 04:24:01 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json |
| 19 | 26-08-12 04:24:01 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json |
| 20 | 26-08-12 04:31:38 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json |
| 21 | 26-08-12 04:31:38 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json |
| 22 | 26-08-12 04:34:59 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json |
| 23 | 26-08-12 04:34:59 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json |
| 24 | 26-08-12 04:41:22 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json |

32
scripts/agent_benchmark/attempts.py
View file

@ -363,7 +363,7 @@ def _connectivity_result_from_payload(
tuple(stages),
)
# Capability is deliberately not serialized. The persisted proof is
# revalidated against the exact immutable cell and its one direct route.
# revalidated against the exact immutable cell and its declared route.
capability = CallerCapability(
cell.caller, (cell.iop.route_kind,), (cell.iop.requested_effort,)
)
@ -387,11 +387,7 @@ def collect_preflight_observations(
manifest: Manifest,
adapters: Mapping[str, PreflightAdapter],
) -> dict[str, PreflightObservation]:
"""Validate the full registry, then probe direct cells in manifest order.
Execution-preset cells exercise only the local adapter capability contract in
this milestone. They never become a synthetic live-ready observation.
"""
"""Validate the full registry, then probe every cell in manifest order."""
if not isinstance(adapters, Mapping):
raise CapabilityUnavailable("capability-unavailable: caller-adapter")
required_callers = {cell.caller for cell in manifest.matrix}
@ -405,8 +401,6 @@ def collect_preflight_observations(
observations: dict[str, PreflightObservation] = {}
for cell in manifest.matrix:
if cell.iop.route_kind != "direct":
continue
observation = adapters[cell.caller].preflight(cell)
if not isinstance(observation, PreflightObservation):
raise AttemptStateError("preflight observation is invalid")
@ -512,8 +506,8 @@ class RunStore:
os.close(fd)
@staticmethod
def _direct_cells(manifest: Manifest) -> tuple[MatrixCell, ...]:
return tuple(cell for cell in manifest.matrix if cell.iop.route_kind == "direct")
def _preflight_cells(manifest: Manifest) -> tuple[MatrixCell, ...]:
return tuple(manifest.matrix)
def _preflight_root(self, run: RunIdentity, *, create: bool) -> Path:
root = Path(run.root) / "preflight"
@ -560,11 +554,11 @@ class RunStore:
or not isinstance(record["results"], list)
):
raise AttemptStateError("preflight record identity is invalid")
direct_cells = self._direct_cells(manifest)
if len(record["results"]) != len(direct_cells):
preflight_cells = self._preflight_cells(manifest)
if len(record["results"]) != len(preflight_cells):
raise AttemptStateError("preflight result set is invalid")
statuses: list[str] = []
for cell, result_payload in zip(direct_cells, record["results"]):
for cell, result_payload in zip(preflight_cells, record["results"]):
result, _, _ = _connectivity_result_from_payload(result_payload, cell)
statuses.append(result.status)
if record["status"] != _overall_preflight_status(iter(statuses)):
@ -629,11 +623,11 @@ class RunStore:
bound_run = self.open(manifest, run.run_id)
if bound_run != run:
raise AttemptStateError("run identity is invalid")
direct_cells = self._direct_cells(manifest)
if set(observations) != {cell.id for cell in direct_cells}:
preflight_cells = self._preflight_cells(manifest)
if set(observations) != {cell.id for cell in preflight_cells}:
raise AttemptStateError("preflight observation set is invalid")
results: list[dict[str, Any]] = []
for cell in direct_cells:
for cell in preflight_cells:
observation = observations[cell.id]
if not isinstance(observation, PreflightObservation):
raise AttemptStateError("preflight observation is invalid")
@ -1873,10 +1867,8 @@ def preflight_manifest(
*,
adapters: Mapping[str, PreflightAdapter],
) -> tuple[RunIdentity, dict[str, Any]]:
"""Collect direct observations, then create one run and append one record."""
"""Collect all-cell observations, then create one run and append one record."""
observations = collect_preflight_observations(manifest, adapters)
if not observations:
raise AttemptStateError("preflight requires a direct cell")
run = store.create(manifest, manifest_bytes)
return run, store.record_preflight(run, manifest, observations)
@ -1946,8 +1938,6 @@ def run_slots(
raise AttemptStateError("workspace preparer is invalid")
observations = collect_preflight_observations(manifest, adapters)
if not observations:
raise AttemptStateError("preflight requires a direct cell")
bound_run = store.open(manifest, run.run_id)
if bound_run != run:
raise AttemptStateError("run identity is invalid")

41
scripts/agent_benchmark/attempts_test.py
View file

@ -72,7 +72,12 @@ from scripts.agent_benchmark.web_validation import (
from scripts.agent_benchmark.workspace import AttemptIdentity, prepare_workspace
def _manifest(root: Path, repetitions: int = 1):
def _manifest(
root: Path,
repetitions: int = 1,
execution_order_seed: str | None = None,
cell_ids: tuple[str, ...] = ("a",),
):
fixtures = root / "scripts/fixtures"
fixtures.mkdir(parents=True, exist_ok=True)
(fixtures / "prompt.md").write_text("prompt", encoding="utf-8")
@ -89,8 +94,13 @@ def _manifest(root: Path, repetitions: int = 1):
"viewports": [{"id": "desktop", "width": 1, "height": 1}], "rubric_version": "landing-quality-v1",
"evaluator": {"caller": "codex", "iop": {"request_model": "judge", "requested_effort": "high", "route_kind": "direct", "route_id": "judge", "expected_bindings": [{"stage": "request", "model": "judge", "effort": "high"}]}},
"output_root": "agent-test/runs/a", "fixture": fixture, "repetitions": repetitions,
"matrix": [{"id": "a", "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}}],
"matrix": [
{"id": cell_id, "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}}
for cell_id in cell_ids
],
}
if execution_order_seed is not None:
data["execution_order_seed"] = execution_order_seed
path = root / "manifest.json"
raw = json.dumps(data, sort_keys=True).encode("utf-8")
path.write_bytes(raw)
@ -454,6 +464,33 @@ class AttemptBase(unittest.TestCase):
class AttemptStoreTest(AttemptBase):
def test_slots_follow_seeded_manifest_order_before_repetitions(self):
self.manifest, self.raw, self.manifest_path = _manifest(
self.root,
repetitions=2,
execution_order_seed="seed-a",
cell_ids=("cell-a", "cell-b", "cell-c"),
)
matrix_order = [cell.id for cell in self.manifest.matrix]
self.assertEqual(matrix_order, ["cell-c", "cell-a", "cell-b"])
slots = self.store.slots(self.manifest)
self.assertEqual(
[(slot.cell_id, slot.repetition) for slot in slots],
[
("cell-c", 1),
("cell-c", 2),
("cell-a", 1),
("cell-a", 2),
("cell-b", 1),
("cell-b", 2),
],
)
self.assertEqual(
[slot.cell_id for slot in slots if slot.repetition == 1],
matrix_order,
)
def test_slots_and_append_only_terminals(self):
self.manifest, self.raw, self.manifest_path = _manifest(self.root, repetitions=2)
run = self.create_run()

298
scripts/agent_benchmark/connectivity_integration_test.py
View file

@ -25,6 +25,7 @@ from scripts import agent_comparison_benchmark as benchmark_cli
from scripts.agent_benchmark import live_iop
from scripts.agent_benchmark.browser_cdp import RenderObservation, ViewportObservation
from scripts.agent_benchmark.attempts import (
AttemptStateError,
CapabilityUnavailable,
PreflightObservation,
RunBusyError,
@ -1006,14 +1007,15 @@ class ConnectivityIntegrationTest(unittest.TestCase):
self.assertFalse((run_roots[0] / "cells").exists())
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
def test_cli_mixed_manifest_never_invokes_unobserved_preset_cells(self) -> None:
def test_cli_mixed_manifest_preflights_and_invokes_every_cell(self) -> None:
self._init_testbed()
manifest, _, path = _write_manifest(
self.root,
[
_cell("direct-ready", "claude", "claude-sonnet-5", "max"),
_preset("preset-unobserved", "claude", "claude-sonnet-5", "max"),
_preset("preset-ready", "claude", "claude-sonnet-5", "max"),
],
output_id="mixed-unobserved",
output_id="mixed-ready",
)
registry = self._registry()
stdout = io.StringIO()
@ -1028,10 +1030,9 @@ class ConnectivityIntegrationTest(unittest.TestCase):
):
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
self.assertEqual(exit_code, 69)
self.assertEqual(stdout.getvalue(), "")
self.assertIn("error: benchmark execution failed", stderr.getvalue())
self.assertIn("completed=0 unresolved=2", stderr.getvalue())
self.assertEqual(exit_code, 0, stderr.getvalue())
self.assertIn("ok: run run_id=", stdout.getvalue())
self.assertEqual(stderr.getvalue(), "")
run_roots = list((self.root / manifest.output_root).glob("run-*"))
self.assertEqual(len(run_roots), 1)
preflight = json.loads(
@ -1042,11 +1043,20 @@ class ConnectivityIntegrationTest(unittest.TestCase):
self.assertEqual(preflight["status"], "ready")
self.assertEqual(
[result["cell"]["id"] for result in preflight["results"]],
["direct-ready"],
[cell.id for cell in manifest.matrix],
)
self.assertEqual(
registry["claude"].calls,
[cell.id for cell in manifest.matrix],
)
self.assertEqual(
[item[0] for item in registry["claude"].invocations],
[cell.id for cell in manifest.matrix],
)
self.assertEqual(
len(list(run_roots[0].glob("cells/*/repetition-*/attempt-*"))),
len(manifest.matrix),
)
self.assertFalse((run_roots[0] / "cells").exists())
self.assertEqual(registry["claude"].calls, ["direct-ready"])
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
def test_cli_resume_retries_append_only_and_status_is_read_only(self) -> None:
self._init_testbed()
@ -1163,8 +1173,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
)
self.assertFalse(output_root.exists())
def test_generic_preset_cells_are_local_contract_only(self) -> None:
generic, _, _ = _write_manifest(
def test_preset_only_public_preflight_appends_exact_results(self) -> None:
generic, _, path = _write_manifest(
self.root,
[
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
@ -1174,31 +1184,6 @@ class ConnectivityIntegrationTest(unittest.TestCase):
output_id="generic",
)
registry = self._registry()
observations = collect_preflight_observations(generic, registry)
self.assertEqual(observations, {})
self.assertTrue(all(adapter.calls == [] for adapter in registry.values()))
def test_generic_preset_only_public_preflight_fails_closed_without_run_state(self) -> None:
generic, raw, path = _write_manifest(
self.root,
[
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
_preset("agy-generic", "agy", "gemini-3.6-flash", "high"),
_preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"),
],
output_id="generic",
)
registry = self._registry()
output_root = self.root / generic.output_root
with self.assertRaises(Exception) as ctx:
preflight_manifest(
self.store, generic, raw, adapters=registry
)
self.assertIn("preflight requires a direct cell", str(ctx.exception))
self.assertFalse(output_root.exists())
self.assertTrue(all(adapter.calls == [] for adapter in registry.values()))
sentinel = "private_endpoint_and_token_must_not_appear"
stdout = io.StringIO()
stderr = io.StringIO()
with (
@ -1208,10 +1193,103 @@ class ConnectivityIntegrationTest(unittest.TestCase):
contextlib.redirect_stderr(stderr),
):
exit_code = benchmark_cli.main(["preflight", "--manifest", str(path)])
self.assertEqual(exit_code, 0, stderr.getvalue())
self.assertIn("status=ready", stdout.getvalue())
self.assertEqual(stderr.getvalue(), "")
run_roots = list((self.root / generic.output_root).glob("run-*"))
self.assertEqual(len(run_roots), 1)
record = json.loads(
(run_roots[0] / "preflight/preflight-000001.json").read_text(
encoding="ascii"
)
)
self.assertEqual(
[result["cell"]["id"] for result in record["results"]],
[cell.id for cell in generic.matrix],
)
self.assertFalse((run_roots[0] / "cells").exists())
self.assertEqual(
{caller: adapter.calls for caller, adapter in registry.items()},
{
"claude": ["claude-generic"],
"agy": ["agy-generic"],
"codex": ["codex-generic"],
},
)
def test_preset_blocker_appends_without_attempt_allocation(self) -> None:
generic, _, path = _write_manifest(
self.root,
[
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
_preset("agy-generic", "agy", "gemini-3.6-flash", "high"),
_preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"),
],
output_id="generic-blocked",
)
registry = self._registry(
{"agy-generic": ("credential_missing",)}
)
stdout = io.StringIO()
stderr = io.StringIO()
with (
mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root),
mock.patch.object(benchmark_cli, "build_adapter_registry", return_value=registry),
contextlib.redirect_stdout(stdout),
contextlib.redirect_stderr(stderr),
):
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
self.assertEqual(exit_code, 69)
self.assertEqual(stdout.getvalue(), "")
self.assertNotIn(sentinel, stderr.getvalue())
self.assertFalse(output_root.exists())
self.assertIn("error: preflight blocked", stderr.getvalue())
run_roots = list((self.root / generic.output_root).glob("run-*"))
self.assertEqual(len(run_roots), 1)
record = json.loads(
(run_roots[0] / "preflight/preflight-000001.json").read_text(
encoding="ascii"
)
)
self.assertEqual(record["status"], "registration_required")
self.assertEqual(
[result["cell"]["id"] for result in record["results"]],
[cell.id for cell in generic.matrix],
)
self.assertFalse((run_roots[0] / "cells").exists())
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
def test_all_cell_preflight_rejects_missing_extra_and_reordered_results(self) -> None:
manifest, raw, _ = _write_manifest(
self.root,
[
_cell("direct", "claude", "claude-sonnet-5", "max"),
_preset("preset", "claude", "claude-sonnet-5", "max"),
],
output_id="all-cell-corruption",
)
run, record = preflight_manifest(
self.store, manifest, raw, adapters=self._registry()
)
path = Path(run.root) / "preflight/preflight-000001.json"
original = path.read_bytes()
mutations = {
"missing": record["results"][:-1],
"extra": [*record["results"], record["results"][0]],
"reordered": list(reversed(record["results"])),
}
for label, results in mutations.items():
with self.subTest(label=label):
mutated = {**record, "results": results}
path.write_bytes(
json.dumps(
mutated, sort_keys=True, separators=(",", ":")
).encode("ascii")
+ b"\n"
)
with self.assertRaises(AttemptStateError):
self.store.preflights(run, manifest)
path.write_bytes(original)
self.assertEqual(self.store.preflights(run, manifest), (record,))
def test_live_registry_dereferences_secret_names_without_persisting_values(self) -> None:
sentinel = "live-token-must-not-persist"
@ -2352,6 +2430,146 @@ class ConnectivityIntegrationTest(unittest.TestCase):
)
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
def test_live_agy_multiple_cells_consume_their_own_preflight_state(self) -> None:
manifest, _, _ = _write_manifest(
self.root,
[
_cell("agy-direct", "agy", "gemini-direct", "high"),
_preset("agy-preset", "agy", "gemini-preset", "high"),
],
output_id="agy-cell-state",
)
environment = self._live_environment(manifest=manifest)
observed_models = tuple(
sorted(cell.iop.request_model for cell in manifest.matrix)
)
ready_observation = lambda _runtime: live_iop._Observation(
observed_models,
"sha256:" + "2" * 64,
True,
"agy 1.1.11",
"--print --output-format --sandbox --model --effort "
"AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
)
adapter = live_iop.build_live_adapter_registry(
environment,
observer=ready_observation,
binary_resolver=lambda _name: "/bin/true",
)["agy"]
cells = {cell.id: cell for cell in manifest.matrix}
for cell in manifest.matrix:
self.assertEqual(adapter.preflight(cell).result.status, "ready")
preflights = dict(adapter._agy_preflights) # type: ignore[attr-defined]
self.assertEqual(set(preflights), set(cells))
self.assertIsNot(preflights["agy-direct"], preflights["agy-preset"])
self.assertEqual(
preflights["agy-direct"].runtime.observation.cell_id,
"agy-direct",
)
self.assertEqual(
preflights["agy-preset"].runtime.observation.cell_id,
"agy-preset",
)
stream = CaptureStream("stdout", "", 0, 0, False)
lifecycle = InvocationResult(
True,
"success",
0,
None,
True,
True,
True,
False,
(),
stream,
replace(stream, stream="stderr"),
"",
"",
None,
"sha256:" + "a" * 64,
"2026-08-12T00:00:00+00:00",
"2026-08-12T00:00:01+00:00",
1,
(),
)
built: list[tuple[str, object]] = []
invoked: list[tuple[str, object]] = []
def build(cell, _prepared, _task, _timeout, preflight):
built.append((cell.id, preflight))
return object()
def invoke(_spec, parser, preflight, _on_started):
invoked.append((parser._cell.id, preflight))
return lifecycle
def observed_result(parser, capability, _result):
cell = parser._cell
admitted = adapter._admitted_bindings[cell.id] # type: ignore[attr-defined]
caller_capability = CallerCapability(
"agy", capability.route_kinds, capability.efforts
)
return make_result(cell, caller_capability, admitted)
adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
live_iop._DEFAULT_INVOKERS.claude,
invoke,
live_iop._DEFAULT_INVOKERS.codex,
)
with (
mock.patch.object(live_iop, "build_agy_invocation", side_effect=build),
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
mock.patch.object(
live_iop.AgyEventParser,
"observed_result",
new=observed_result,
),
):
for cell_id in ("agy-preset", "agy-direct"):
cell = cells[cell_id]
result = adapter.invoke(
cell,
object(),
object(),
"/tmp/control",
b"task",
manifest.timeout,
lambda *_args: None,
)
self.assertIs(result, lifecycle)
self.assertEqual(
built,
[
("agy-preset", preflights["agy-preset"]),
("agy-direct", preflights["agy-direct"]),
],
)
self.assertEqual(invoked, built)
adapter._observer = lambda _runtime: live_iop._Observation( # type: ignore[attr-defined]
observed_models,
"sha256:" + "2" * 64,
False,
)
self.assertEqual(
adapter.preflight(cells["agy-direct"]).result.status,
"implementation_gap",
)
with self.assertRaises(live_iop.LiveIopError) as raised:
adapter.invoke(
cells["agy-direct"],
object(),
object(),
"/tmp/control",
b"task",
manifest.timeout,
lambda *_args: None,
)
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
def test_cli_missing_live_input_fails_closed_without_secret_or_attempt(self) -> None:
sentinel = "missing-input-token"
stdout = io.StringIO()

24
scripts/agent_benchmark/live_iop.py
View file

@ -27,6 +27,7 @@ from urllib.request import Request, urlopen
from scripts.agent_benchmark.agy_iop import (
AGY_CALLER,
AgyEventParser,
AgyPreflightResult,
AgyRuntimeInputs,
AgyRuntimeObservation,
build_agy_invocation,
@ -855,12 +856,15 @@ class _LiveAdapter:
self._observer = observer
self._binary_resolver = binary_resolver
self._invokers = invokers
self._agy_preflight: Any = None
self._agy_preflights: dict[str, AgyPreflightResult] = {}
self._admitted_bindings: dict[str, RequestedEffectiveBinding] = {}
def preflight(self, cell: MatrixCell) -> PreflightObservation:
if cell.caller != self.caller:
raise LiveIopError("protocol_incompatible")
self._admitted_bindings.pop(cell.id, None)
if self.caller == AGY_CALLER:
self._agy_preflights.pop(cell.id, None)
resolution = self._runtime_resolution
runtime = resolution.runtime
if runtime is None:
@ -874,6 +878,7 @@ class _LiveAdapter:
if not observed.caller_ready:
result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible"))
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
agy_preflight: AgyPreflightResult | None = None
if self.caller == AGY_CALLER:
capability = inspect_agy_iop_capability(observed.agy_version, observed.agy_help)
agy_observation = AgyRuntimeObservation(
@ -882,7 +887,7 @@ class _LiveAdapter:
_agy_runtime_identity("credential", runtime.secret), runtime.config.identity,
)
try:
self._agy_preflight = preflight_agy_iop(
agy_preflight = preflight_agy_iop(
cell, capability,
AgyRuntimeInputs(self._binary_resolver("agy"), runtime.base_url, runtime.secret),
agy_observation,
@ -890,8 +895,8 @@ class _LiveAdapter:
except Exception:
result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible"))
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
if self._agy_preflight.issues:
result = make_result(cell, self.capability, _requested(cell), self._agy_preflight.issues)
if agy_preflight.issues:
result = make_result(cell, self.capability, _requested(cell), agy_preflight.issues)
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
binding, issues = _binding_from_config(cell, self.capability, runtime.config)
if not issues and binding.effective_model not in observed.catalog_models:
@ -899,6 +904,8 @@ class _LiveAdapter:
result = make_result(cell, self.capability, binding, issues)
if result.status == "ready":
self._admitted_bindings[cell.id] = result.binding
if agy_preflight is not None:
self._agy_preflights[cell.id] = agy_preflight
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
def invoke(
@ -933,7 +940,8 @@ class _LiveAdapter:
admitted,
)
if self.caller == AGY_CALLER:
if self._agy_preflight is None:
agy_preflight = self._agy_preflights.get(cell.id)
if agy_preflight is None or agy_preflight.binding != _requested(cell):
raise LiveIopError("stream_incompatible")
spec = _bind_live_spec(
cell,
@ -941,12 +949,12 @@ class _LiveAdapter:
attempt,
control_dir,
build_agy_invocation(
cell, prepared, task_payload, timeout, self._agy_preflight
cell, prepared, task_payload, timeout, agy_preflight
),
)
parser = AgyEventParser(cell)
result = self._invokers.agy(spec, parser, self._agy_preflight, lambda locator: on_started(locator, spec_digest(spec)))
observed = parser.observed_result(self._agy_preflight.capability, result)
result = self._invokers.agy(spec, parser, agy_preflight, lambda locator: on_started(locator, spec_digest(spec)))
observed = parser.observed_result(agy_preflight.capability, result)
if observed.status != "ready" or observed.binding != admitted:
raise LiveIopError("stream_incompatible")
return _bound_observations(result, admitted)

49
scripts/agent_benchmark/manifest.py
View file

@ -28,6 +28,8 @@ SESSION_POLICY = "fresh"
SETUP_CACHE_POLICY = "isolated"
DEFAULT_REPETITIONS = 1
RUBRIC_VERSION = "landing-quality-v1"
ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"
RUBRIC_VERSIONS = (RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION)
CALLER_ENUM = ("claude", "agy", "codex")
ROUTE_KIND_ENUM = ("direct", "execution_preset")
@ -124,6 +126,7 @@ class Manifest:
fixture: Fixture
matrix: tuple[MatrixCell, ...]
digest: str
execution_order_seed: str | None = None
# ---------------------------------------------------------------------------
@ -190,7 +193,7 @@ def _require_enum(value: Any, field_name: str, allowed: tuple[str, ...]) -> str:
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
s = _require_str(value, field_name)
if not pattern.match(s):
if pattern.fullmatch(s) is None:
raise ManifestValidationError(
f"field '{field_name}' does not match required pattern"
)
@ -310,7 +313,7 @@ def digest_manifest_and_resolved_inputs(manifest: Manifest) -> str:
def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]:
"""Convert a Manifest to a plain dict for canonical JSON serialization."""
return {
canonical = {
"pipeline_version": manifest.pipeline_version,
"environment": manifest.environment,
"testbed": manifest.testbed,
@ -377,6 +380,11 @@ def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]:
for c in manifest.matrix
],
}
# Pipeline-v2 manifests without an explicit seed keep their legacy
# canonical bytes and digest. An explicit seed is part of the contract.
if manifest.execution_order_seed is not None:
canonical["execution_order_seed"] = manifest.execution_order_seed
return canonical
# ---------------------------------------------------------------------------
@ -608,7 +616,19 @@ def _validate_evaluator(data: dict[str, Any]) -> Evaluator:
return Evaluator(caller=caller, iop=_validate_iop_cell(obj["iop"]))
def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]:
def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]:
rank = hashlib.sha256(
b"iop-benchmark-order-v1\0"
+ seed.encode("ascii")
+ b"\0"
+ cell.id.encode("ascii")
).digest()
return rank, cell.id
def _validate_matrix(
data: list[Any], execution_order_seed: str | None
) -> tuple[MatrixCell, ...]:
arr = _require_array(data, "matrix")
if len(arr) < 1:
raise ManifestValidationError("matrix must have at least 1 item")
@ -622,8 +642,13 @@ def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]:
seen_ids.add(cell.id)
cells.append(cell)
# Sort cells by id for canonical ordering
cells.sort(key=lambda c: c.id)
# No seed preserves the pipeline-v2 cell-id order. An explicit seed is
# stable across JSON input permutations and platforms.
cells.sort(
key=(lambda cell: cell.id)
if execution_order_seed is None
else (lambda cell: _execution_order_key(execution_order_seed, cell))
)
return tuple(cells)
@ -684,7 +709,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
"session_policy", "setup_cache_policy", "timeout", "viewports",
"rubric_version", "evaluator", "output_root",
}
optional_top = {"repetitions"}
optional_top = {"repetitions", "execution_order_seed"}
declared_keys = set(raw.keys())
required_present = expected_top.issubset(declared_keys)
extra = declared_keys - expected_top - optional_top
@ -701,6 +726,12 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
if "repetitions" in raw:
repetitions = _require_positive_int(raw["repetitions"], "repetitions")
execution_order_seed: str | None = None
if "execution_order_seed" in raw:
execution_order_seed = _require_pattern(
raw["execution_order_seed"], "execution_order_seed", CELL_ID_RE
)
session_policy = _require_enum(raw["session_policy"], "session_policy", (SESSION_POLICY,))
setup_cache_policy = _require_enum(
raw["setup_cache_policy"], "setup_cache_policy", (SETUP_CACHE_POLICY,)
@ -708,13 +739,13 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
timeout = _validate_timeout(raw["timeout"])
viewports = _validate_viewports(raw["viewports"])
rubric_version = _require_enum(
raw["rubric_version"], "rubric_version", (RUBRIC_VERSION,)
raw["rubric_version"], "rubric_version", RUBRIC_VERSIONS
)
evaluator = _validate_evaluator(raw["evaluator"])
output_root = _validate_output_root(raw["output_root"], repo_root)
fixture = _validate_fixture(raw["fixture"], repo_root)
matrix = _validate_matrix(raw["matrix"])
matrix = _validate_matrix(raw["matrix"], execution_order_seed)
# Compute manifest digest during load
dummy_manifest = Manifest(
@ -732,6 +763,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
fixture=fixture,
matrix=matrix,
digest="",
execution_order_seed=execution_order_seed,
)
computed_digest = digest_manifest_and_resolved_inputs(dummy_manifest)
@ -750,6 +782,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
fixture=fixture,
matrix=matrix,
digest=computed_digest,
execution_order_seed=execution_order_seed,
)

399
scripts/agent_benchmark/manifest_test.py
View file

@ -14,6 +14,7 @@ import hashlib
import inspect
import json
import os
import re
import shutil
import struct
import subprocess
@ -38,11 +39,15 @@ from scripts.agent_benchmark.manifest import (
ManifestPathError,
ManifestValidationError,
MatrixCell,
ONE_SHOT_RUBRIC_VERSION,
RUBRIC_VERSION,
RUBRIC_VERSIONS,
Timeout,
Viewport,
digest_manifest_and_resolved_inputs,
digest_workspace_inputs,
load_manifest,
_manifest_to_dict,
validate_manifest_bytes,
)
@ -159,6 +164,230 @@ def _load_tmp_manifest(path: Path) -> Manifest:
return load_manifest(path, repo_root=_REPO_ROOT)
class ManifestValidationTest(unittest.TestCase):
"""Exact tracked-manifest regressions used by benchmark readiness packets."""
def test_iop_one_shot_manifest_locks_benchmark_readiness(self):
"""The bench-02 manifest locks identical inputs and the complete C01-C09 matrix."""
manifest_path = (
_REPO_ROOT
/ "scripts"
/ "fixtures"
/ "agent-comparison-benchmark-iop-one-shot.json"
)
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
self.assertEqual(
{
key: raw[key]
for key in (
"pipeline_version",
"environment",
"testbed",
"execution_order_seed",
"repetitions",
"session_policy",
"setup_cache_policy",
"timeout",
"viewports",
"rubric_version",
"evaluator",
"output_root",
)
},
{
"pipeline_version": "2",
"environment": "dev",
"testbed": "../iop-s2",
"execution_order_seed": "bench-02-c01-c09-v1",
"repetitions": 1,
"session_policy": "fresh",
"setup_cache_policy": "isolated",
"timeout": {
"run_seconds": 300,
"idle_seconds": 30,
"quiet_seconds": 10,
"cleanup_grace_seconds": 5,
},
"viewports": [
{"id": "desktop_1080", "width": 1920, "height": 1080},
{"id": "mobile_375", "width": 375, "height": 812},
],
"rubric_version": ONE_SHOT_RUBRIC_VERSION,
"evaluator": {
"caller": "codex",
"iop": {
"request_model": "gpt-5.6-luna",
"requested_effort": "xhigh",
"route_kind": "direct",
"route_id": "gpt-5.6-luna",
"expected_bindings": [
{
"stage": "request",
"model": "gpt-5.6-luna",
"effort": "xhigh",
}
],
},
},
"output_root": "agent-test/runs/bench-02",
},
)
expected_fixture = {
"version": "landing-v1",
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
"assets": [
{
"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt",
"workspace_path": "brief/reference.txt",
},
{
"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg",
"workspace_path": "assets/aurora-grid.svg",
},
{
"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg",
"workspace_path": "assets/orbit-rings.svg",
},
],
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98",
}
self.assertEqual(raw["fixture"], expected_fixture)
self.assertEqual(
_compute_fixture_checksum(_REPO_ROOT, raw["fixture"]),
expected_fixture["checksum"],
)
self.assertEqual(
[
asset["workspace_path"]
for asset in raw["fixture"]["assets"]
if asset["source"].endswith(".svg")
],
["assets/aurora-grid.svg", "assets/orbit-rings.svg"],
)
self.assertEqual(
manifest.viewports,
(
Viewport(id="desktop_1080", width=1920, height=1080),
Viewport(id="mobile_375", width=375, height=812),
),
)
self.assertEqual(
manifest.timeout,
Timeout(
run_seconds=300,
idle_seconds=30,
quiet_seconds=10,
cleanup_grace_seconds=5,
),
)
def direct(cell_id: str, caller: str, model: str, effort: str) -> dict:
return {
"id": cell_id,
"caller": caller,
"iop": {
"request_model": model,
"requested_effort": effort,
"route_kind": "direct",
"route_id": model,
"expected_bindings": [
{"stage": "request", "model": model, "effort": effort}
],
},
}
def hybrid(
cell_id: str,
caller: str,
route_id: str,
request_effort: str,
cloud_model: str,
) -> dict:
return {
"id": cell_id,
"caller": caller,
"iop": {
"request_model": route_id,
"requested_effort": request_effort,
"route_kind": "execution_preset",
"route_id": route_id,
"expected_bindings": [
{"stage": "selector", "model": cloud_model, "effort": "high"},
{"stage": "plan", "model": cloud_model, "effort": "high"},
{"stage": "work", "model": "ornith-fast"},
{"stage": "review", "model": cloud_model, "effort": "high"},
{"stage": "repair", "model": cloud_model, "effort": "high"},
],
},
}
expected_cells = [
direct("c01-claude-sonnet-direct", "claude", "claude-sonnet-5", "max"),
direct("c02-claude-gemini-direct", "claude", "gemini-3.6-flash", "high"),
direct("c03-agy-gemini-direct", "agy", "gemini-3.6-flash", "high"),
direct("c04-claude-gpt-direct", "claude", "gpt-5.6-luna", "xhigh"),
direct("c05-codex-gpt-direct", "codex", "gpt-5.6-luna", "xhigh"),
hybrid(
"c06-claude-gemini-hybrid",
"claude",
"gemini-hybrid",
"high",
"gemini-3.6-flash",
),
hybrid(
"c07-agy-gemini-hybrid",
"agy",
"gemini-hybrid",
"high",
"gemini-3.6-flash",
),
hybrid(
"c08-claude-gpt-hybrid",
"claude",
"gpt-hybrid",
"xhigh",
"gpt-5.6-terra",
),
hybrid(
"c09-codex-gpt-hybrid",
"codex",
"gpt-hybrid",
"xhigh",
"gpt-5.6-terra",
),
]
self.assertEqual(raw["matrix"], expected_cells)
expected_order = [
"c02-claude-gemini-direct",
"c05-codex-gpt-direct",
"c03-agy-gemini-direct",
"c06-claude-gemini-hybrid",
"c08-claude-gpt-hybrid",
"c09-codex-gpt-hybrid",
"c01-claude-sonnet-direct",
"c07-agy-gemini-hybrid",
"c04-claude-gpt-direct",
]
expected_by_id = {cell["id"]: cell for cell in expected_cells}
self.assertEqual(
[cell.id for cell in manifest.matrix],
expected_order,
)
self.assertEqual(
_manifest_to_dict(manifest)["matrix"],
[expected_by_id[cell_id] for cell_id in expected_order],
)
self.assertEqual(manifest.execution_order_seed, "bench-02-c01-c09-v1")
self.assertEqual(manifest.rubric_version, ONE_SHOT_RUBRIC_VERSION)
self.assertEqual(manifest.fixture.checksum, expected_fixture["checksum"])
self.assertEqual(manifest.digest, digest_manifest_and_resolved_inputs(manifest))
self.assertRegex(manifest.digest, r"^sha256:[0-9a-f]{64}$")
class TestLoadManifestValid(unittest.TestCase):
"""Valid manifest loading tests."""
@ -374,6 +603,90 @@ class TestMatrixExtension(unittest.TestCase):
class TestDeterministicOrdering(unittest.TestCase):
"""Deterministic cell and binding ordering tests."""
@staticmethod
def _direct_cells(*cell_ids: str) -> list[dict]:
return [
{
"id": cell_id,
"caller": "claude",
"iop": {
"request_model": "claude-sonnet-4-20250514",
"requested_effort": "high",
"route_kind": "direct",
"route_id": "claude-direct",
"expected_bindings": [
{
"stage": "request",
"model": "claude-sonnet-4-20250514",
}
],
},
}
for cell_id in cell_ids
]
def test_execution_order_seed_is_canonical_and_permutation_stable(self):
"""An explicit seed produces one repeatable order for every input permutation."""
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
cells = self._direct_cells("cell-a", "cell-b", "cell-c")
seeded = _make_minimal_manifest_dict(
execution_order_seed="seed-a", matrix=cells
)
permuted = _make_minimal_manifest_dict(
execution_order_seed="seed-a", matrix=list(reversed(cells))
)
different_seed = _make_minimal_manifest_dict(
execution_order_seed="seed-b", matrix=cells
)
m1 = _load_tmp_manifest(
_write_tmp_manifest(tmp_dir, seeded, "seeded.json")
)
m2 = _load_tmp_manifest(
_write_tmp_manifest(tmp_dir, permuted, "permuted.json")
)
m3 = _load_tmp_manifest(
_write_tmp_manifest(tmp_dir, different_seed, "different.json")
)
self.assertEqual(m1.execution_order_seed, "seed-a")
self.assertEqual(
_manifest_to_dict(m1)["execution_order_seed"],
"seed-a",
)
self.assertEqual(
[cell.id for cell in m1.matrix],
["cell-c", "cell-a", "cell-b"],
)
self.assertEqual(m1.matrix, m2.matrix)
self.assertEqual(m1.digest, m2.digest)
self.assertEqual(
[cell.id for cell in m3.matrix],
["cell-a", "cell-c", "cell-b"],
)
self.assertNotEqual(m1.matrix, m3.matrix)
self.assertNotEqual(m1.digest, m3.digest)
def test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract(self):
"""Omitting the seed keeps id order and excludes a synthetic default from canonical JSON."""
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
data = _make_minimal_manifest_dict(
matrix=self._direct_cells("cell-c", "cell-a", "cell-b")
)
manifest = _load_tmp_manifest(_write_tmp_manifest(tmp_dir, data))
self.assertIsNone(manifest.execution_order_seed)
self.assertEqual(
[cell.id for cell in manifest.matrix],
["cell-a", "cell-b", "cell-c"],
)
self.assertNotIn("execution_order_seed", _manifest_to_dict(manifest))
self.assertEqual(
manifest.digest, digest_manifest_and_resolved_inputs(manifest)
)
def test_cells_sorted_by_id(self):
"""Cells are sorted by id regardless of input order."""
with tempfile.TemporaryDirectory() as tmp:
@ -951,7 +1264,7 @@ class TestEnumsAndBounds(unittest.TestCase):
_load_tmp_manifest(path)
def test_invalid_rubric_version_rejected(self):
"""Invalid rubric_version pattern is rejected."""
"""A rubric version outside the closed catalog is rejected."""
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
d = _make_minimal_manifest_dict(rubric_version="Invalid Version!")
@ -1313,6 +1626,44 @@ class TestSchemaLoaderParity(unittest.TestCase):
m = load_manifest(example_path)
self.assertEqual(m.pipeline_version, "2")
self.assertEqual(m.testbed, "../iop-s2")
self.assertEqual(m.rubric_version, RUBRIC_VERSION)
def test_schema_and_loader_accept_exact_rubric_version_catalog(self):
"""The schema and loader accept both immutable rubric versions only."""
schema_path = (
_REPO_ROOT
/ "scripts"
/ "fixtures"
/ "agent-comparison-benchmark-manifest.schema.json"
)
schema = json.loads(schema_path.read_text(encoding="utf-8"))
rubric_schema = schema["properties"]["rubric_version"]
self.assertEqual(tuple(rubric_schema["enum"]), RUBRIC_VERSIONS)
self.assertEqual(
RUBRIC_VERSIONS,
(RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION),
)
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
for index, version in enumerate(RUBRIC_VERSIONS):
with self.subTest(version=version):
path = _write_tmp_manifest(
tmp_dir,
_make_minimal_manifest_dict(rubric_version=version),
f"rubric-{index}.json",
)
self.assertEqual(_load_tmp_manifest(path).rubric_version, version)
unknown = "one-shot-agent-comparison-v2"
self.assertNotIn(unknown, rubric_schema["enum"])
path = _write_tmp_manifest(
tmp_dir,
_make_minimal_manifest_dict(rubric_version=unknown),
"rubric-unknown.json",
)
with self.assertRaises(ManifestValidationError):
_load_tmp_manifest(path)
def test_tracked_fixtures_separate_generic_contract_from_direct_preflight(self):
fixtures = _REPO_ROOT / "scripts" / "fixtures"
@ -1377,6 +1728,52 @@ class TestSchemaLoaderParity(unittest.TestCase):
with self.assertRaises(ManifestValidationError):
_load_tmp_manifest(p)
def test_schema_and_loader_reject_same_invalid_execution_order_seeds(self):
"""The tracked schema and loader enforce the same bounded seed token."""
schema_path = (
_REPO_ROOT
/ "scripts"
/ "fixtures"
/ "agent-comparison-benchmark-manifest.schema.json"
)
schema = json.loads(schema_path.read_text(encoding="utf-8"))
seed_schema = schema["properties"]["execution_order_seed"]
self.assertEqual(seed_schema["type"], "string")
self.assertEqual(seed_schema["pattern"], "^[a-z0-9][a-z0-9_-]{0,63}$")
boundary_seed = "a" * 64
self.assertIsNotNone(re.fullmatch(seed_schema["pattern"], boundary_seed))
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
data = _make_minimal_manifest_dict(
execution_order_seed=boundary_seed
)
path = _write_tmp_manifest(tmp_dir, data, "boundary-seed.json")
self.assertEqual(
_load_tmp_manifest(path).execution_order_seed,
boundary_seed,
)
invalid_seeds: tuple[object, ...] = (
"",
"Uppercase",
"contains.dot",
"seed-a\n",
"a" * 65,
True,
None,
)
pattern = re.compile(seed_schema["pattern"])
for index, seed in enumerate(invalid_seeds):
schema_accepts = isinstance(seed, str) and pattern.fullmatch(seed) is not None
self.assertFalse(schema_accepts)
with tempfile.TemporaryDirectory() as tmp:
tmp_dir = Path(tmp)
data = _make_minimal_manifest_dict(execution_order_seed=seed)
path = _write_tmp_manifest(tmp_dir, data, f"invalid-seed-{index}.json")
with self.assertRaises(ManifestValidationError):
_load_tmp_manifest(path)
def test_preset_with_request_stage_rejected(self):
"""Execution preset cell with extra request stage raises ManifestValidationError."""
with tempfile.TemporaryDirectory() as tmp:

59
scripts/agent_benchmark/rubric.py
View file

@ -1,4 +1,4 @@
"""Strict D12 landing-page worksheet contract.
"""Strict versioned benchmark worksheet contract.
Automatic web gates establish scoring eligibility; they are intentionally not
represented in this 100-point worksheet and can never contribute points.
@ -11,9 +11,13 @@ import os
import stat
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from types import MappingProxyType
from typing import Any, Mapping
from scripts.agent_benchmark.manifest import RUBRIC_VERSION
from scripts.agent_benchmark.manifest import (
ONE_SHOT_RUBRIC_VERSION,
RUBRIC_VERSION,
)
MAX_WORKSHEET_BYTES = 64 * 1024
@ -25,12 +29,39 @@ RUBRIC_CATEGORIES = (
("typography_readability", 15),
("polish_consistency", 15),
)
ONE_SHOT_RUBRIC_CATEGORIES = (
("requirements_fidelity", 25),
("visual_completeness", 25),
("responsive_accessibility", 15),
("image_detail_usage", 10),
("behavior_stability", 10),
("code_quality", 10),
("self_verification", 5),
)
RUBRIC_CATEGORIES_BY_VERSION: Mapping[str, tuple[tuple[str, int], ...]] = (
MappingProxyType(
{
RUBRIC_VERSION: RUBRIC_CATEGORIES,
ONE_SHOT_RUBRIC_VERSION: ONE_SHOT_RUBRIC_CATEGORIES,
}
)
)
class RubricError(Exception):
"""A worksheet is missing, malformed, non-canonical, or out of bounds."""
def rubric_categories(version: str) -> tuple[tuple[str, int], ...]:
"""Return the immutable ordered category table for a known rubric version."""
if not isinstance(version, str):
raise RubricError("rubric version is invalid")
try:
return RUBRIC_CATEGORIES_BY_VERSION[version]
except KeyError as exc:
raise RubricError("rubric version is invalid") from exc
@dataclass(frozen=True)
class CategoryScore:
id: str
@ -75,22 +106,28 @@ def canonical_worksheet_bytes(worksheet: Worksheet) -> bytes:
)
def validate_worksheet(value: Any) -> Worksheet:
def validate_worksheet(
value: Any, expected_version: str | None = None
) -> Worksheet:
if not isinstance(value, dict) or set(value) != {
"rubric_version", "categories", "total",
}:
raise RubricError("worksheet schema is invalid")
if value["rubric_version"] != RUBRIC_VERSION:
rubric_version = value["rubric_version"]
expected_categories = rubric_categories(rubric_version)
if expected_version is not None:
rubric_categories(expected_version)
if expected_version is not None and rubric_version != expected_version:
raise RubricError("worksheet rubric version is invalid")
raw_categories = value["categories"]
if not isinstance(raw_categories, list) or len(raw_categories) != len(
RUBRIC_CATEGORIES
expected_categories
):
raise RubricError("worksheet categories are invalid")
categories: list[CategoryScore] = []
for raw, (expected_id, expected_max) in zip(
raw_categories, RUBRIC_CATEGORIES
raw_categories, expected_categories
):
if not isinstance(raw, dict) or set(raw) != {
"id", "max_score", "score", "evidence",
@ -123,10 +160,12 @@ def validate_worksheet(value: Any) -> Worksheet:
or not 0 <= total <= 100
):
raise RubricError("worksheet total is invalid")
return Worksheet(RUBRIC_VERSION, tuple(categories), total)
return Worksheet(rubric_version, tuple(categories), total)
def load_worksheet(path: str | Path) -> Worksheet:
def load_worksheet(
path: str | Path, expected_version: str | None = None
) -> Worksheet:
target = Path(path)
flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK
if hasattr(os, "O_NOFOLLOW"):
@ -155,4 +194,4 @@ def load_worksheet(path: str | Path) -> Worksheet:
value = json.loads(bytes(raw).decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise RubricError("worksheet JSON is invalid") from exc
return validate_worksheet(value)
return validate_worksheet(value, expected_version=expected_version)

70
scripts/agent_benchmark/rubric_test.py
View file

@ -1,21 +1,29 @@
from __future__ import annotations
import hashlib
import json
import os
import tempfile
import unittest
from pathlib import Path
from scripts.agent_benchmark.manifest import (
ONE_SHOT_RUBRIC_VERSION,
RUBRIC_VERSION,
)
from scripts.agent_benchmark.rubric import (
ONE_SHOT_RUBRIC_CATEGORIES,
RUBRIC_CATEGORIES,
RUBRIC_CATEGORIES_BY_VERSION,
RubricError,
canonical_worksheet_bytes,
load_worksheet,
rubric_categories,
validate_worksheet,
)
def _worksheet() -> dict:
def _worksheet(rubric_version: str = RUBRIC_VERSION) -> dict:
categories = [
{
"id": ident,
@ -23,10 +31,10 @@ def _worksheet() -> dict:
"score": maximum,
"evidence": f"Evidence for {ident}.",
}
for ident, maximum in RUBRIC_CATEGORIES
for ident, maximum in rubric_categories(rubric_version)
]
return {
"rubric_version": "landing-quality-v1",
"rubric_version": rubric_version,
"categories": categories,
"total": 100,
}
@ -44,6 +52,62 @@ class RubricTest(unittest.TestCase):
validate_worksheet(json.loads(canonical_worksheet_bytes(worksheet))),
worksheet,
)
self.assertEqual(
hashlib.sha256(canonical_worksheet_bytes(worksheet)).hexdigest(),
"233727170226ab30409657007751c13616a16c83ca75700e15a2c8dce56a96ee",
)
def test_one_shot_rubric_exact_categories_and_total_are_accepted(self):
expected = (
("requirements_fidelity", 25),
("visual_completeness", 25),
("responsive_accessibility", 15),
("image_detail_usage", 10),
("behavior_stability", 10),
("code_quality", 10),
("self_verification", 5),
)
self.assertEqual(ONE_SHOT_RUBRIC_CATEGORIES, expected)
self.assertEqual(rubric_categories(ONE_SHOT_RUBRIC_VERSION), expected)
self.assertEqual(tuple(RUBRIC_CATEGORIES_BY_VERSION), (
RUBRIC_VERSION,
ONE_SHOT_RUBRIC_VERSION,
))
self.assertEqual(sum(maximum for _, maximum in RUBRIC_CATEGORIES), 100)
self.assertEqual(sum(maximum for _, maximum in expected), 100)
worksheet = validate_worksheet(
_worksheet(ONE_SHOT_RUBRIC_VERSION),
expected_version=ONE_SHOT_RUBRIC_VERSION,
)
self.assertEqual(worksheet.rubric_version, ONE_SHOT_RUBRIC_VERSION)
self.assertEqual(worksheet.total, 100)
self.assertEqual(
[(item.id, item.max_score) for item in worksheet.categories],
list(expected),
)
def test_unknown_and_cross_version_worksheets_are_rejected(self):
with self.assertRaises(RubricError):
rubric_categories("unknown-rubric-v1")
with self.assertRaises(RubricError):
validate_worksheet(_worksheet(), expected_version=ONE_SHOT_RUBRIC_VERSION)
with self.assertRaises(RubricError):
validate_worksheet(
_worksheet(ONE_SHOT_RUBRIC_VERSION),
expected_version=RUBRIC_VERSION,
)
unknown = _worksheet()
unknown["rubric_version"] = "unknown-rubric-v1"
with self.assertRaises(RubricError):
validate_worksheet(unknown)
def test_cross_version_worksheet_is_rejected(self):
with tempfile.TemporaryDirectory() as temp:
path = Path(temp) / "worksheet.json"
path.write_text(json.dumps(_worksheet()), encoding="utf-8")
with self.assertRaises(RubricError):
load_worksheet(path, expected_version=ONE_SHOT_RUBRIC_VERSION)
def test_missing_extra_reordered_and_out_of_range_values_fail(self):
cases = []

23
scripts/agent_benchmark/scoring.py
View file

@ -2,8 +2,8 @@
The original cell identity remains under ``cells/`` and in a run-owned mapping
that is never copied into the evaluator tree. A scorer receives only one
opaque ``blind/<id>`` directory, anonymous input bytes, the fixed D12 rubric,
and a fresh session/output pair.
opaque ``blind/<id>`` directory, anonymous input bytes, the manifest-selected
rubric, and a fresh session/output pair.
"""
from __future__ import annotations
@ -42,11 +42,11 @@ from scripts.agent_benchmark.lifecycle import (
recover_invocation,
)
from scripts.agent_benchmark.rubric import (
RUBRIC_CATEGORIES,
RubricError,
Worksheet,
canonical_worksheet_bytes,
load_worksheet,
rubric_categories,
)
from scripts.agent_benchmark.web_validation import (
GENERATED_FILES,
@ -1204,9 +1204,10 @@ def _materialize_blind(
)
def _prompt(blind: BlindWorkspace) -> bytes:
def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes:
categories = ", ".join(
f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES
f"{ident} ({maximum})"
for ident, maximum in rubric_categories(rubric_version)
)
value = (
"Evaluate the anonymous landing page using only the files under input/. "
@ -1216,7 +1217,7 @@ def _prompt(blind: BlindWorkspace) -> bytes:
"Write exactly one JSON object to output/worksheet.json with fields "
"rubric_version, categories, and total. Each category must have exactly "
"id, max_score, score, and non-empty evidence. Use rubric_version "
"landing-quality-v1, integer scores within each maximum, and total equal "
f"{rubric_version}, integer scores within each maximum, and total equal "
"to their sum. Do not add automatic gate points or any extra field."
)
return value.encode("utf-8")
@ -1654,7 +1655,8 @@ def _result_status(
raise ScoringError("scored result is invalid")
try:
worksheet = load_worksheet(
Path(run.root) / allocation["blind_path"] / "output" / "worksheet.json"
Path(run.root) / allocation["blind_path"] / "output" / "worksheet.json",
expected_version=manifest.rubric_version,
)
except RubricError as exc:
raise ScoringError("scored worksheet is invalid") from exc
@ -1886,7 +1888,7 @@ def _score_one(
)
return "scoring_failed"
prompt = _prompt(blind)
prompt = _prompt(blind, manifest.rubric_version)
identities = _identity_values(manifest, attempt)
if _contains_identity(prompt, identities):
_publish_current_failure(
@ -1984,7 +1986,10 @@ def _score_one(
)
return "scoring_failed"
try:
worksheet = load_worksheet(blind_root / "output" / "worksheet.json")
worksheet = load_worksheet(
blind_root / "output" / "worksheet.json",
expected_version=manifest.rubric_version,
)
post_tree = _blind_tree_digest(blind_root)
_publish_success(
score_root,

108
scripts/agent_benchmark/scoring_test.py
View file

@ -43,7 +43,13 @@ from scripts.agent_benchmark.lifecycle import (
run_invocation,
spec_digest,
)
from scripts.agent_benchmark.manifest import AssetMapping, digest_workspace_inputs, load_manifest
from scripts.agent_benchmark.manifest import (
AssetMapping,
ONE_SHOT_RUBRIC_VERSION,
RUBRIC_VERSION,
digest_workspace_inputs,
load_manifest,
)
from scripts.agent_benchmark.measurement import (
AttemptMeasurement,
REASON_NOT_OBSERVED,
@ -53,7 +59,7 @@ from scripts.agent_benchmark.measurement import (
publish_measurement,
unavailable,
)
from scripts.agent_benchmark.rubric import RUBRIC_CATEGORIES
from scripts.agent_benchmark.rubric import rubric_categories
from scripts.agent_benchmark import scoring as scoring_module
from scripts.agent_benchmark.scoring import (
BlindWorkspace,
@ -73,9 +79,11 @@ def _digest(data: bytes) -> str:
return "sha256:" + hashlib.sha256(data).hexdigest()
def _worksheet(total_delta: int = 0) -> dict:
def _worksheet(
total_delta: int = 0, rubric_version: str = RUBRIC_VERSION
) -> dict:
categories = []
for index, (ident, maximum) in enumerate(RUBRIC_CATEGORIES):
for index, (ident, maximum) in enumerate(rubric_categories(rubric_version)):
score = maximum - (1 if index == 0 else 0)
categories.append(
{
@ -86,7 +94,7 @@ def _worksheet(total_delta: int = 0) -> dict:
}
)
return {
"rubric_version": "landing-quality-v1",
"rubric_version": rubric_version,
"categories": categories,
"total": sum(item["score"] for item in categories) + total_delta,
}
@ -103,12 +111,14 @@ class FakeScoringAdapter:
blocked: bool = False,
modes: list[str] | None = None,
sensitive_value: str = "",
rubric_version: str = RUBRIC_VERSION,
):
self.blocked = blocked
self.modes = list(modes or ["success"])
self.preflights = 0
self.invocations: list[tuple[BlindWorkspace, bytes]] = []
self.sensitive_value = sensitive_value
self.rubric_version = rubric_version
self.last_mode = ""
def preflight(self, cell):
@ -253,7 +263,7 @@ class FakeScoringAdapter:
if mode == "malformed":
(output / "worksheet.json").write_text("{}", encoding="utf-8")
else:
worksheet = _worksheet()
worksheet = _worksheet(rubric_version=self.rubric_version)
if mode == "secret":
worksheet["categories"][0]["evidence"] = self.sensitive_value
(output / "worksheet.json").write_text(
@ -558,6 +568,22 @@ class ScoringTest(unittest.TestCase):
self.assertEqual(adapter.preflights, 1)
self.assertEqual(len(adapter.invocations), 1)
blind, prompt = adapter.invocations[0]
self.assertEqual(
prompt,
(
"Evaluate the anonymous landing page using only the files under input/. "
"Inspect index.html, styles.css, script.js, the two local images, and all "
"screenshots. Do not use network access or infer the producing system. "
"Use exactly these categories: task_fidelity (25), visual_hierarchy "
"(25), responsive_composition (20), typography_readability (15), "
"polish_consistency (15). Write exactly one JSON object to "
"output/worksheet.json with fields rubric_version, categories, and "
"total. Each category must have exactly id, max_score, score, and "
"non-empty evidence. Use rubric_version landing-quality-v1, integer "
"scores within each maximum, and total equal to their sum. Do not add "
"automatic gate points or any extra field."
).encode("utf-8"),
)
self.assertNotIn("cell-sentinel", blind.root)
self.assertNotIn("source-route", blind.root)
self.assertNotIn(str(Path(attempt.root).resolve()), blind.root)
@ -601,6 +627,76 @@ class ScoringTest(unittest.TestCase):
self.assertEqual(len(adapter.invocations), 1)
self.assertEqual(before, {path: path.read_bytes() for path in Path(attempt.root).rglob("*") if path.is_file()})
def test_manifest_selected_rubric_drives_prompt_and_worksheet_validation(self):
raw = json.loads(self.manifest_path.read_text(encoding="utf-8"))
raw["rubric_version"] = ONE_SHOT_RUBRIC_VERSION
raw["output_root"] = "agent-test/runs/new-rubric-bench"
path = self.root / "new-rubric.json"
path.write_text(json.dumps(raw), encoding="utf-8")
self.manifest_path = path
self.manifest = load_manifest(path, repo_root=self.root)
self.run = self.store.create(self.manifest, path.read_bytes())
attempt = self._attempt()
selected = FakeScoringAdapter(rubric_version=ONE_SHOT_RUBRIC_VERSION)
summary = score_run(
self.store, self.run, self.manifest, adapter=selected
)
self.assertEqual((summary.scored, summary.scoring_failed), (1, 0))
self.assertEqual(len(selected.invocations), 1)
_, prompt = selected.invocations[0]
self.assertIn(
b"Use exactly these categories: requirements_fidelity (25), "
b"visual_completeness (25), responsive_accessibility (15), "
b"image_detail_usage (10), behavior_stability (10), code_quality (10), "
b"self_verification (5).",
prompt,
)
self.assertIn(
b"Use rubric_version one-shot-agent-comparison-v1, integer scores",
prompt,
)
result_path = (
Path(attempt.root) / "scoring" / "score-000001" / "result.json"
)
result = json.loads(result_path.read_text(encoding="ascii"))
self.assertEqual(result["status"], "scored")
self.assertEqual(
result["worksheet"]["rubric_version"], ONE_SHOT_RUBRIC_VERSION
)
self.assertEqual(
[item["id"] for item in result["worksheet"]["categories"]],
[ident for ident, _ in rubric_categories(ONE_SHOT_RUBRIC_VERSION)],
)
retained = score_run(
self.store,
self.run,
self.manifest,
adapter=FakeScoringAdapter(rubric_version=ONE_SHOT_RUBRIC_VERSION),
)
self.assertEqual(retained.scored, 1)
self.run = self.store.create(self.manifest, path.read_bytes())
mismatched_attempt = self._attempt()
legacy = FakeScoringAdapter()
rejected = score_run(
self.store, self.run, self.manifest, adapter=legacy
)
self.assertEqual((rejected.scored, rejected.scoring_failed), (0, 1))
self.assertEqual(len(legacy.invocations), 1)
mismatch_result = json.loads(
(
Path(mismatched_attempt.root)
/ "scoring"
/ "score-000001"
/ "result.json"
).read_text(encoding="ascii")
)
self.assertEqual(mismatch_result["status"], "scoring_failed")
self.assertEqual(mismatch_result["reason"], "invalid_worksheet")
self.assertNotIn("worksheet", mismatch_result)
def test_ineligible_attempt_is_unscored_without_preflight_or_zero(self):
attempt = self._attempt("failed")
adapter = FakeScoringAdapter()

49
scripts/agent_benchmark/skill_contract_test.py
View file

@ -248,13 +248,13 @@ class BenchmarkSkillContractTest(unittest.TestCase):
"""Assert report is a supported operation alongside run/resume."""
procedure = self._get_section(skill_text, "Procedure")
self.assertNotIn("capability-unavailable: caller-adapter", procedure)
self.assertIn("append a fresh preflight before attempt allocation", procedure)
self.assertIn("append a fresh all-cell preflight before attempt allocation", procedure)
self.assertIn("invoke each eligible cell exactly once", procedure)
# report must be delegated, not gated
self.assertIn("report", procedure)
def _assert_preflight_contract(self, skill_text: str) -> None:
"""Require direct-only append semantics and fail-closed blocker language."""
"""Require exact all-cell append semantics and fail-closed blockers."""
procedure = self._get_section(skill_text, "Procedure")
validation = self._get_section(skill_text, "Validation")
prohibitions = self._get_section(skill_text, "Prohibitions")
@ -262,16 +262,31 @@ class BenchmarkSkillContractTest(unittest.TestCase):
"python3 scripts/agent_comparison_benchmark.py preflight --manifest <manifest-path>",
procedure,
)
self.assertIn("records only direct-cell observations", procedure)
self.assertIn("Generic preset cells are local contract validation only", skill_text)
self.assertIn(
"records one fresh live observation for every immutable matrix cell",
procedure,
)
self.assertIn("including direct and execution-preset routes", procedure)
self.assertIn("in canonical matrix order", procedure)
self.assertIn("registration_required", procedure)
self.assertIn("implementation_gap", procedure)
self.assertIn("Never bypass the blocker", procedure)
self.assertIn("substitute a route/model/effort", procedure)
self.assertIn("Preflight evidence is append-only, direct-only", validation)
self.assertIn(
"Preflight evidence is append-only, covers every immutable matrix cell in canonical order",
validation,
)
self.assertIn("created no scored attempt", validation)
self.assertIn("Do not bypass a preflight blocker", prohibitions)
self.assertIn("Do not claim execution-preset fixture validation as live readiness", prohibitions)
for obsolete in (
"records only direct-cell observations",
"Generic preset cells are local contract validation only",
"Preflight evidence is append-only, direct-only",
"Direct preflight never allocates a scored attempt",
"fresh direct preflight",
):
self.assertNotIn(obsolete, skill_text)
def _assert_scoring_contract(self, skill_text: str) -> None:
procedure = self._get_section(skill_text, "Procedure")
@ -284,6 +299,11 @@ class BenchmarkSkillContractTest(unittest.TestCase):
self.assertIn("immutable `unscored`", procedure)
self.assertIn("without invoking the evaluator or assigning zero", procedure)
self.assertIn("manifest-bound fresh Codex evaluator session", procedure)
self.assertIn("exact immutable manifest-selected rubric", procedure)
self.assertIn("closed supported catalog", procedure)
self.assertIn("`landing-quality-v1`", procedure)
self.assertIn("`one-shot-agent-comparison-v1`", procedure)
self.assertIn("no substitute rubric or reinterpretation is permitted", procedure)
self.assertIn("allocates a new score id and preserves every prior byte", procedure)
self.assertIn("`scoring_failed` used no fallback", validation)
self.assertIn("Do not retry scoring implicitly", prohibitions)
@ -441,7 +461,7 @@ class BenchmarkSkillContractTest(unittest.TestCase):
"""Run/resume must document execution rather than a capability gate."""
skill_text = _SKILL_FILE.read_text(encoding="utf-8")
self.assertNotIn("capability-unavailable: caller-adapter", skill_text)
self.assertIn("append a fresh preflight before attempt allocation", skill_text)
self.assertIn("append a fresh all-cell preflight before attempt allocation", skill_text)
def test_cli_help_exits_zero(self) -> None:
"""Skill must not expose a public prepare operation across all steps and sections."""
@ -783,11 +803,11 @@ class BenchmarkSkillContractTest(unittest.TestCase):
with self.assertRaises(AssertionError):
self._assert_full_skill_contract(mutated)
def test_mutation_claims_preset_live_readiness(self) -> None:
def test_mutation_restores_direct_only_preflight(self) -> None:
base = self._skill_base_text()
mutated = base.replace(
"Generic preset cells are local contract validation only.",
"Generic preset cells are live readiness evidence.",
"records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order",
"records only direct-cell observations",
)
self.assertNotEqual(mutated, base, "mutation fixture did not apply")
with self.assertRaises(AssertionError):
@ -803,6 +823,17 @@ class BenchmarkSkillContractTest(unittest.TestCase):
with self.assertRaises(AssertionError):
self._assert_full_skill_contract(mutated)
def test_mutation_restores_fixed_legacy_scoring_rubric(self) -> None:
"""A fixed legacy-only worksheet must fail manifest-selected scoring."""
base = self._skill_base_text()
mutated = base.replace(
"the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted",
"the exact `landing-quality-v1` worksheet",
)
self.assertNotEqual(mutated, base, "mutation fixture did not apply")
with self.assertRaises(AssertionError):
self._assert_full_skill_contract(mutated)
def test_mutation_allows_blocker_attempt_allocation(self) -> None:
base = self._skill_base_text()
mutated = base.replace(

2
scripts/agent_comparison_benchmark.py
View file

@ -11,7 +11,7 @@ Usage:
python3 scripts/agent_comparison_benchmark.py score --manifest PATH --run-id RUN_ID
Exits:
0 - manifest is valid or every direct preflight cell is ready
0 - manifest is valid or every matrix preflight cell is ready
64 - usage error (missing args, bad flags)
69 - validation/state failed or preflight is blocked
"""

178
scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json Normal file
View file

@ -0,0 +1,178 @@
{
"pipeline_version": "2",
"environment": "dev",
"testbed": "../iop-s2",
"execution_order_seed": "bench-02-c01-c09-v1",
"repetitions": 1,
"session_policy": "fresh",
"setup_cache_policy": "isolated",
"timeout": {
"run_seconds": 300,
"idle_seconds": 30,
"quiet_seconds": 10,
"cleanup_grace_seconds": 5
},
"viewports": [
{"id": "desktop_1080", "width": 1920, "height": 1080},
{"id": "mobile_375", "width": 375, "height": 812}
],
"rubric_version": "one-shot-agent-comparison-v1",
"evaluator": {
"caller": "codex",
"iop": {
"request_model": "gpt-5.6-luna",
"requested_effort": "xhigh",
"route_kind": "direct",
"route_id": "gpt-5.6-luna",
"expected_bindings": [
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
]
}
},
"output_root": "agent-test/runs/bench-02",
"fixture": {
"version": "landing-v1",
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
"assets": [
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
],
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98"
},
"matrix": [
{
"id": "c01-claude-sonnet-direct",
"caller": "claude",
"iop": {
"request_model": "claude-sonnet-5",
"requested_effort": "max",
"route_kind": "direct",
"route_id": "claude-sonnet-5",
"expected_bindings": [
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
]
}
},
{
"id": "c02-claude-gemini-direct",
"caller": "claude",
"iop": {
"request_model": "gemini-3.6-flash",
"requested_effort": "high",
"route_kind": "direct",
"route_id": "gemini-3.6-flash",
"expected_bindings": [
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
]
}
},
{
"id": "c03-agy-gemini-direct",
"caller": "agy",
"iop": {
"request_model": "gemini-3.6-flash",
"requested_effort": "high",
"route_kind": "direct",
"route_id": "gemini-3.6-flash",
"expected_bindings": [
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
]
}
},
{
"id": "c04-claude-gpt-direct",
"caller": "claude",
"iop": {
"request_model": "gpt-5.6-luna",
"requested_effort": "xhigh",
"route_kind": "direct",
"route_id": "gpt-5.6-luna",
"expected_bindings": [
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
]
}
},
{
"id": "c05-codex-gpt-direct",
"caller": "codex",
"iop": {
"request_model": "gpt-5.6-luna",
"requested_effort": "xhigh",
"route_kind": "direct",
"route_id": "gpt-5.6-luna",
"expected_bindings": [
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
]
}
},
{
"id": "c06-claude-gemini-hybrid",
"caller": "claude",
"iop": {
"request_model": "gemini-hybrid",
"requested_effort": "high",
"route_kind": "execution_preset",
"route_id": "gemini-hybrid",
"expected_bindings": [
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "work", "model": "ornith-fast"},
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
]
}
},
{
"id": "c07-agy-gemini-hybrid",
"caller": "agy",
"iop": {
"request_model": "gemini-hybrid",
"requested_effort": "high",
"route_kind": "execution_preset",
"route_id": "gemini-hybrid",
"expected_bindings": [
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "work", "model": "ornith-fast"},
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
]
}
},
{
"id": "c08-claude-gpt-hybrid",
"caller": "claude",
"iop": {
"request_model": "gpt-hybrid",
"requested_effort": "xhigh",
"route_kind": "execution_preset",
"route_id": "gpt-hybrid",
"expected_bindings": [
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "work", "model": "ornith-fast"},
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
]
}
},
{
"id": "c09-codex-gpt-hybrid",
"caller": "codex",
"iop": {
"request_model": "gpt-hybrid",
"requested_effort": "xhigh",
"route_kind": "execution_preset",
"route_id": "gpt-hybrid",
"expected_bindings": [
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "work", "model": "ornith-fast"},
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
]
}
}
]
}

8
scripts/fixtures/agent-comparison-benchmark-manifest.schema.json
View file

@ -28,11 +28,17 @@
"minimum": 1,
"default": 1
},
"execution_order_seed": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9_-]{0,63}$"
},
"session_policy": { "const": "fresh" },
"setup_cache_policy": { "const": "isolated" },
"timeout": { "$ref": "#/$defs/timeout" },
"viewports": { "$ref": "#/$defs/viewports" },
"rubric_version": { "const": "landing-quality-v1" },
"rubric_version": {
"enum": ["landing-quality-v1", "one-shot-agent-comparison-v1"]
},
"evaluator": { "$ref": "#/$defs/evaluator" },
"output_root": { "$ref": "#/$defs/output_root" },
"fixture": { "$ref": "#/$defs/fixture" },