fix(benchmark): 복구 qualification 경계를 고정한다
품질 gate를 익명 채점 evidence로 유지하면서 hard trust gate만 실행 eligibility를 차단하도록 D15 계약을 반영한다. 최종 9-cell run 전에 실패 경로만 확인할 수 있도록 300초 six-path 진단 manifest를 분리한다.
This commit is contained in:
parent
29bf84b9eb
commit
15bc8879cf
8 changed files with 706 additions and 26 deletions
|
|
@ -0,0 +1,98 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery plan=0 tag=REPAIR milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring -->
|
||||
|
||||
# Code Review Reference - REPAIR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** Complete the fixed checklist, record actual output, leave active files in place, and report ready. Do not decide the verdict, create follow-up plans, archive files, write `complete.log`, modify roadmap state, ask the user, or start the dependent final benchmark. A qualification failure must be recorded without retry; official review owns diagnosis and the next exact fix.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-13
|
||||
task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery, plan=0, tag=REPAIR
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`.
|
||||
- Preserved run: `run-20260813T081326Z-4e1ac5152c6c`, two scored/seven unscored; immutable diagnostic evidence, not completion.
|
||||
- Latest merged baseline: `origin/dev@6e2f4ff8`; deployed runtime was stale before this task.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Rerun safe deterministic checks and reconstruct the release/qualification evidence. PASS requires all six qualification paths to meet the plan's success/hard-gate oracle. If any path fails, do not PASS or let the final task start: collect the exact sanitized terminal/request-stage evidence, select one concrete root-cause fix with exact files/tests, and generate the required follow-up plan through the plan/finalize-routing flow.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| REPAIR-1 Hard/quality scoring boundary | [ ] |
|
||||
| REPAIR-2 Five-minute manifests | [ ] |
|
||||
| REPAIR-3 Central dev deployment | [ ] |
|
||||
| REPAIR-4 Six-path qualification | [ ] |
|
||||
| REPAIR-5 Recovery report | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REPAIR-1] Separate hard artifact eligibility gates from quality-scoring gates and add fail-closed regression coverage.
|
||||
- [ ] [REPAIR-2] Set the compact benchmark and diagnostic per-cell ceiling to 300 seconds and lock both manifests in tests.
|
||||
- [ ] [REPAIR-3] Commit/push the exact feature, merge it into central dev, and complete the private dev-runtime deployment procedure with a safe restart and source/build identity proof.
|
||||
- [ ] [REPAIR-4] Run exactly one fresh non-scored six-path qualification and require all six caller/route paths to produce successful terminal evidence, generated files, screenshots, and all hard gates.
|
||||
- [ ] [REPAIR-5] Publish a sanitized deterministic recovery report without secrets or claims about the not-yet-run final benchmark.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one PASS/WARN/FAIL verdict with verified routing signals.
|
||||
- [ ] Re-run focused/full tests and validate both manifests.
|
||||
- [ ] Verify the scoring split remains fail-closed for hard evidence and quality evidence reaches the evaluator.
|
||||
- [ ] Verify exact central dev release/source/build/runtime identity and safe restart evidence.
|
||||
- [ ] Verify exactly one six-path qualification, no retry/resume/score/final run, six successes, 12 screenshots, and 24 passing hard gates.
|
||||
- [ ] On live failure, diagnose and select an exact fix before generating a follow-up; do not write `complete.log`.
|
||||
- [ ] On PASS, archive the pair, preserve milestone metadata, write `complete.log`, move the task directory, and leave the dependent task ready.
|
||||
- [ ] Reconcile `.gitignore`, work log, active parent, and no duplicate execution.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Implementer: replace with `None` or exact deviation, reason, and replacement command._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Implementer: record the implemented hard/quality partition, manifest identity, release identity, and no-retry boundary._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- D14 completion is not met here; this is recovery admission only.
|
||||
- D15 partition is exact: hard=`generated_files,static_safety,images,network`; quality=`console,responsive,accessibility`.
|
||||
- Runtime warnings alone are not caller failure; terminal product/harness/process and hard artifact evidence are authoritative.
|
||||
- No raw secret, credential, provider payload, or identity map enters tracked evidence.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Focused and full deterministic tests
|
||||
|
||||
_Implementer: paste exact commands, output summaries, and exits._
|
||||
|
||||
### Central dev deployment
|
||||
|
||||
_Implementer: record sanitized feature/dev/release/tag SHA/tree, test counts, binary identities, restart/ports/nodes/providers/capacity results._
|
||||
|
||||
### Six-path qualification
|
||||
|
||||
_Implementer: record exact run id, preflight count, per-cell terminal/hard/quality/screenshot matrix, and no-retry audit._
|
||||
|
||||
### Diff and provenance audit
|
||||
|
||||
_Implementer: record manifest digests, contained report links, `git diff --check`, and status._
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section and check the final checklist item. Leave review-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---|---|---|
|
||||
| Header, Overview, Archive Snapshot, Review instructions | Fixed | Do not modify |
|
||||
| Implementation completion/checklist | Implementer | Check status only |
|
||||
| Review-Only Checklist and verdict | Reviewer | Implementer must not modify |
|
||||
| Deviations, decisions, verification results | Implementer then reviewer | Record actual evidence |
|
||||
|
|
@ -0,0 +1,267 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery plan=0 tag=REPAIR milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring -->
|
||||
|
||||
# Recover every caller path before the final nine-cell measurement
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only the closed scoring/timeout/qualification packet below. Run every verification, fill the paired `CODE_REVIEW-cloud-G10.md` with actual output, keep the active files in place, and report ready for official review. Final verdict, follow-up planning, archive moves, `complete.log`, and roadmap synchronization belong only to official review. If the six-path qualification still fails, do not retry it or start the final benchmark; record the exact sanitized terminal evidence and resume condition so the reviewer can select the next concrete fix.
|
||||
|
||||
## Background
|
||||
|
||||
The preserved run `run-20260813T081326Z-4e1ac5152c6c` ended with two scored and seven unscored cells, so it is diagnostic evidence rather than milestone completion. C02 produced a trustworthy rendered page but accessibility alone made it ineligible; the other failures are concentrated in Claude and official agy caller paths against a stale dev runtime. This task first fixes the confirmed eligibility defect, adopts the user-approved 300-second per-cell ceiling, deploys the already-merged latest dev stream-liveness baseline, and runs one non-scored six-path qualification. It must not allocate the final C01-C09 run.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`; internal verdict PASS, but its final `2 scored/7 unscored` state is not the user-level completion condition.
|
||||
- Preserved evidence: C02 has `product=succeeded`, `harness=passed`, `process=exited/0`, both screenshots, and only `accessibility=false`; C01 has `parser_error`; C03 returned Gemini `400 INVALID_ARGUMENT` after a first tool turn; C04/C06/C08 returned Claude caller errors; C07 timed out.
|
||||
- Latest central baseline merged into this feature: `origin/dev@6e2f4ff8`, including the stream terminal-liveness recovery absent from the deployed `dev-974` runtime.
|
||||
- Do not modify, resume, retry, delete, or score the preserved run.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/gemini-compatible-api.md`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/agent_benchmark/web_validation.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop_test.py`
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `apps/edge/internal/openai/gemini_handler.go`
|
||||
- `apps/edge/internal/openai/gemini_bridge.go`
|
||||
- `apps/edge/internal/openai/gemini_handler_test.go`
|
||||
- `apps/edge/internal/openai/anthropic_stream.go`
|
||||
- `apps/edge/internal/openai/single_request_anthropic_stream.go`
|
||||
- `apps/edge/internal/openai/single_request_anthropic_stream_test.go`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, `[승인됨]`, lock released.
|
||||
- First-line milestone task ids: `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring`.
|
||||
- Target scenarios: S04-S10. Evidence Map rows require per-caller lifecycle/workspace evidence, uniform hard gates/screenshots, and anonymous scoring eligibility.
|
||||
- D14 makes 9/9 successful and scored the only final completion condition. D15 makes `generated_files`, `static_safety`, `images`, and `network` hard eligibility gates while `console`, `responsive`, and `accessibility` remain quality evidence. Those rows define REPAIR-1 and the six-path admission gate in REPAIR-4.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No external handoff was supplied. Repository evidence and the preserved run were read directly.
|
||||
- Current local feature contains `origin/dev@6e2f4ff8`; live preflight is `ready=9`. The remote dev runner `/Users/toki/agent-work/iop-dev` was clean at `019500a5`, twelve commits behind `origin/dev`, with an approximately seven-hour-old Edge/Node runtime listening on 18082/18083/18084/19101.
|
||||
- External runner: `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, macOS/arm64. Before mutation, re-run the preflight in the project `dev-runtime-deploy` skill and require clean refs, exact fetched feature SHA, toolchain, runtime identity, and ports.
|
||||
- Use `agent-ops/skills/private/dev-runtime-deploy/SKILL.md` (or the project fallback when private is absent) for the complete feature-to-dev merge, release, tests, rebuild, safe restart, capacity smoke, finish, and atomic push. Do not shortcut its gates.
|
||||
- Constraint: qualification is append-only and non-scored; exactly one fresh run is allowed. Final C01-C09 execution belongs to the dependent task.
|
||||
- Gap: latest-dev live behavior has not yet been observed. Qualification failure is valid evidence, not authority for an implementation agent to choose an unplanned runtime fix. Official review must diagnose it and create a concrete follow-up plan.
|
||||
- Confidence: high for the scoring false positive and stale-runtime mismatch; medium for whether the latest baseline alone closes all six caller paths.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing scoring tests cover lifecycle and web failure eligibility, but not the D15 hard-versus-quality split; add regression cases.
|
||||
- Existing manifest tests pin `run_seconds=180`; update them to the approved 300 seconds and validate the diagnostic manifest identity.
|
||||
- Existing adapter/Edge unit tests cannot prove the deployed official Claude/agy process combination. The fresh six-path qualification supplies that external coverage.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No public symbol rename or removal.
|
||||
- Add private scoring gate constants only within `scripts/agent_benchmark/scoring.py`; `_eligibility` is the sole call site whose classification changes.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split subtask `16+15_all_cell_measurement_recovery`.
|
||||
- Predecessor `15+14_scored_benchmark_and_report` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`.
|
||||
- Stable output contract: a deployed exact dev release plus one six-path qualification where every cell has successful product/harness/process and all four hard gates. Dependent task `17+16_full_nine_cell_scored_benchmark` may start only after this task has archived PASS evidence.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Included: scoring eligibility, manifest timeout/tests, one diagnostic manifest/report, central dev deployment, and six-path live qualification.
|
||||
- Excluded: preserved run mutation, final C01-C09 allocation/scoring/reporting, fixture prompt/assets/rubric changes, caller/model substitution, credential changes, and speculative Edge/adapter edits. A remaining live defect must be owned by a reviewer-selected follow-up packet.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build and review closures are all true: scope/context/verification/evidence/ownership/decision are closed by D14-D15, the exact dev runner, and a one-run qualification oracle.
|
||||
- Build scores: scope 2, state 2, blast 2, evidence 2, verification 2 => G10; base/route basis `grade-boundary`, cloud, `worker/cloud/G10`.
|
||||
- Review scores: 2/2/2/2/2 => G10; `official-review`, cloud, `review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; matched risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product`; count 4. `review_rework_count=0`, `evidence_integrity_failure=false`.
|
||||
- Canonical files: `PLAN-cloud-G10.md`, `CODE_REVIEW-cloud-G10.md`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REPAIR-1] Separate hard artifact eligibility gates from quality-scoring gates and add fail-closed regression coverage.
|
||||
- [ ] [REPAIR-2] Set the compact benchmark and diagnostic per-cell ceiling to 300 seconds and lock both manifests in tests.
|
||||
- [ ] [REPAIR-3] Commit/push the exact feature, merge it into central dev, and complete the private dev-runtime deployment procedure with a safe restart and source/build identity proof.
|
||||
- [ ] [REPAIR-4] Run exactly one fresh non-scored six-path qualification and require all six caller/route paths to produce successful terminal evidence, generated files, screenshots, and all hard gates.
|
||||
- [ ] [REPAIR-5] Publish a sanitized deterministic recovery report without secrets or claims about the not-yet-run final benchmark.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REPAIR-1] Split eligibility from quality evidence
|
||||
|
||||
**Problem:** `scripts/agent_benchmark/scoring.py:1056-1067` appends the aggregate `web_failed` reason and every failed web gate, so a trustworthy page with only accessibility/responsive/console findings is unscored even though those dimensions belong to the rubric.
|
||||
|
||||
**Solution:** Replace aggregate web-status rejection with explicit hard-gate evaluation. Keep record-shape validation for all ordered gates; reject missing/invalid evidence and failures of `generated_files`, `static_safety`, `images`, or `network`; allow failures of `console`, `responsive`, and `accessibility` to reach the unchanged anonymous evaluator input.
|
||||
|
||||
Before (`scoring.py:1056`):
|
||||
|
||||
```python
|
||||
if web.status != "passed":
|
||||
reasons.append(f"web_{web.status}")
|
||||
reasons.extend(f"gate_{item['id']}" for item in gates if not item["passed"])
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
hard_failures = [item for item in gates if item["id"] in HARD_ELIGIBILITY_GATES and not item["passed"]]
|
||||
reasons.extend(f"gate_{item['id']}" for item in hard_failures)
|
||||
```
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: define the closed hard/quality partition and use only hard failures for eligibility.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: prove each hard failure remains unscored, each quality-only failure remains score-eligible, mixed failures reject, and malformed/missing web evidence still fails closed.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Add table-driven regression coverage in `scripts/agent_benchmark/scoring_test.py` using its existing web-validation fixture helpers. No production run is used.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test
|
||||
```
|
||||
|
||||
Expected: all tests pass; quality-only failures invoke the evaluator, hard failures do not.
|
||||
|
||||
### [REPAIR-2] Lock the five-minute compact ceiling and diagnostic matrix
|
||||
|
||||
**Problem:** `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json:10` and `scripts/agent_benchmark/manifest_test.py:207-280` still pin 180 seconds after the user approved a 300-second ceiling. There is no isolated manifest for testing only the six previously failing caller/route paths without consuming the final run.
|
||||
|
||||
**Solution:** Change only `timeout.run_seconds` to 300 in the immutable full manifest. Add a recovery qualification manifest with the same fixture checksum, viewport/rubric/session/cache policies and only C01, C03, C04, C06, C07, C08 equivalents, a distinct seed, and `output_root=agent-test/runs/bench-02-recovery`. Lock its exact matrix and non-scored purpose in manifest tests.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: set `run_seconds=300`; change nothing else.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json`: add the closed six-path diagnostic manifest.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: update the full-manifest timeout oracle and add exact diagnostic identity/matrix assertions.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Use parser round-trip tests to prove only timeout differs in the final manifest and that qualification cannot be confused with the final output root or nine-cell seed.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
```
|
||||
|
||||
Expected: tests and both validations pass.
|
||||
|
||||
### [REPAIR-3] Deploy the exact central dev baseline safely
|
||||
|
||||
**Problem:** the live dev runtime predates `origin/dev@6e2f4ff8`, while this feature already includes it. Running qualification against that process would repeat stale stream-liveness behavior.
|
||||
|
||||
**Solution:** Use `commit-push` to publish this task's exact feature SHA. Then follow `agent-ops/skills/private/dev-runtime-deploy/SKILL.md` completely: fetch the exact feature on the declared runner, merge it into `dev`, use the commit-count release, pass sequential pre/post tests, rebuild all four binaries, restart Edge then nodes, verify ports/snapshots/routes/capacity, finish only after every release gate, and atomically push main/dev/tag. Preserve secrets and ignored runtime configuration.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: record only sanitized release/source/build/process/port and qualification evidence.
|
||||
- [ ] Run the exact private deploy skill without editing central common rules, dispatcher code, or agent runtime code.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
The deployment skill's full pre/post sequential Go tests, binary provenance, config checks, listener/node/provider checks, and capacity smokes are mandatory. A failure keeps the release unfinished and blocks REPAIR-4.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com 'cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev'
|
||||
```
|
||||
|
||||
Expected after successful finish: clean `dev`, local HEAD equals `origin/dev`, and the recorded deployed binary source equals the finished release tag tree.
|
||||
|
||||
### [REPAIR-4] Qualify all previously failing caller/route families once
|
||||
|
||||
**Problem:** unit/preflight success does not prove official Claude/agy execution against the rebuilt runtime. Reusing the final manifest would spend the only authorized final cycle before recovery is proven.
|
||||
|
||||
**Solution:** With `/tmp/iop-bench-13-env` and the declared dev environment, append one all-cell preflight for the six-path diagnostic manifest and execute it exactly once. Do not score, resume, or retry it. Require `ready=6`, exactly six attempts, `product_succeeded=6`, `harness_passed=6`, `process_exited=6`, exit code zero for each, two screenshots per cell, and all four hard gates true. Quality gates may be false and must be recorded as quality evidence.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: record run id, terminal counts, hard/quality gate matrix, and contained raw-run link.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
One live qualification run only. Any failure is preserved and handed to official review; no manual workspace edits, provider substitution, resume, retry, or final benchmark start.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
```
|
||||
|
||||
Expected: one new qualification run; `ready=6`; all six terminals and hard artifacts satisfy the criteria above.
|
||||
|
||||
### [REPAIR-5] Publish bounded recovery evidence
|
||||
|
||||
**Problem:** without a deterministic summary, a later task could mistake a diagnostic run or a warning-only log for final benchmark completion.
|
||||
|
||||
**Solution:** Write `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md` with exact source/release/run identities, six rows, hard/quality gates, failures if any, and the explicit statement that it is non-scored and cannot satisfy D14.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: publish sanitized evidence and no secrets.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Reviewer reconstructs all six rows from the exact run and verifies every relative link remains inside its run root.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
git diff --check -- . ':(exclude)agent-task/archive/**'
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: no whitespace errors; only declared task changes before commit/final review.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Archived predecessor `15+14_scored_benchmark_and_report` is satisfied by the exact `complete.log` cited above.
|
||||
2. REPAIR-1 and REPAIR-2 run before any central deployment.
|
||||
3. REPAIR-3 must fully finish before REPAIR-4.
|
||||
4. REPAIR-4 is exactly once. REPAIR-5 records it. A failed qualification blocks task completion and the dependent final benchmark.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `scripts/agent_benchmark/scoring.py` | REPAIR-1 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | REPAIR-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | REPAIR-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | REPAIR-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json` | REPAIR-2, REPAIR-4 |
|
||||
| `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md` | REPAIR-3..5 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery/CODE_REVIEW-cloud-G10.md` | REPAIR-1..5 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json --run-id <EXACT_QUALIFICATION_RUN_ID>
|
||||
git diff --check -- . ':(exclude)agent-task/archive/**'
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: deterministic suite passes; final manifest remains nine cells with a 300-second ceiling; diagnostic manifest remains six cells; the one exact qualification has six successful product/harness/process outcomes, two screenshots and four hard gates per cell; no final nine-cell run was allocated.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -205,7 +205,7 @@ class ManifestValidationTest(unittest.TestCase):
|
|||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 180,
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5,
|
||||
|
|
@ -277,7 +277,7 @@ class ManifestValidationTest(unittest.TestCase):
|
|||
self.assertEqual(
|
||||
manifest.timeout,
|
||||
Timeout(
|
||||
run_seconds=180,
|
||||
run_seconds=300,
|
||||
idle_seconds=30,
|
||||
quiet_seconds=10,
|
||||
cleanup_grace_seconds=5,
|
||||
|
|
@ -387,6 +387,70 @@ class ManifestValidationTest(unittest.TestCase):
|
|||
self.assertEqual(manifest.digest, digest_manifest_and_resolved_inputs(manifest))
|
||||
self.assertRegex(manifest.digest, r"^sha256:[0-9a-f]{64}$")
|
||||
|
||||
def test_recovery_qualification_manifest_locks_non_scored_six_path_matrix(self):
|
||||
full_path = (
|
||||
_REPO_ROOT
|
||||
/ "scripts"
|
||||
/ "fixtures"
|
||||
/ "agent-comparison-benchmark-iop-one-shot.json"
|
||||
)
|
||||
recovery_path = (
|
||||
_REPO_ROOT
|
||||
/ "scripts"
|
||||
/ "fixtures"
|
||||
/ "agent-comparison-benchmark-recovery-qualification.json"
|
||||
)
|
||||
full_raw = json.loads(full_path.read_text(encoding="utf-8"))
|
||||
recovery_raw = json.loads(recovery_path.read_text(encoding="utf-8"))
|
||||
recovery = load_manifest(recovery_path, repo_root=_REPO_ROOT)
|
||||
|
||||
shared_keys = (
|
||||
"pipeline_version",
|
||||
"environment",
|
||||
"testbed",
|
||||
"repetitions",
|
||||
"session_policy",
|
||||
"setup_cache_policy",
|
||||
"timeout",
|
||||
"viewports",
|
||||
"rubric_version",
|
||||
"evaluator",
|
||||
"fixture",
|
||||
)
|
||||
self.assertEqual(
|
||||
{key: recovery_raw[key] for key in shared_keys},
|
||||
{key: full_raw[key] for key in shared_keys},
|
||||
)
|
||||
self.assertEqual(recovery_raw["timeout"]["run_seconds"], 300)
|
||||
self.assertEqual(
|
||||
recovery_raw["execution_order_seed"],
|
||||
"bench-02-recovery-qualification-v1",
|
||||
)
|
||||
self.assertNotEqual(
|
||||
recovery_raw["execution_order_seed"], full_raw["execution_order_seed"]
|
||||
)
|
||||
self.assertEqual(
|
||||
recovery_raw["output_root"],
|
||||
"agent-test/runs/bench-02-recovery",
|
||||
)
|
||||
self.assertNotEqual(recovery_raw["output_root"], full_raw["output_root"])
|
||||
|
||||
expected_ids = [
|
||||
"c01-claude-sonnet-direct",
|
||||
"c03-agy-gemini-direct",
|
||||
"c04-claude-gpt-direct",
|
||||
"c06-claude-gemini-hybrid",
|
||||
"c07-agy-gemini-hybrid",
|
||||
"c08-claude-gpt-hybrid",
|
||||
]
|
||||
full_by_id = {cell["id"]: cell for cell in full_raw["matrix"]}
|
||||
self.assertEqual(recovery_raw["matrix"], [full_by_id[item] for item in expected_ids])
|
||||
self.assertEqual(
|
||||
{cell.id for cell in recovery.matrix}, set(expected_ids)
|
||||
)
|
||||
self.assertEqual(len(recovery.matrix), 6)
|
||||
self.assertEqual(recovery.digest, digest_manifest_and_resolved_inputs(recovery))
|
||||
|
||||
|
||||
class TestLoadManifestValid(unittest.TestCase):
|
||||
"""Valid manifest loading tests."""
|
||||
|
|
|
|||
|
|
@ -71,6 +71,10 @@ INPUT_FILENAME = "input.json"
|
|||
RESULT_FILENAME = "result.json"
|
||||
RUNNER_FILENAME = "runner.json"
|
||||
SCORING_STATUSES = ("scored", "unscored", "scoring_failed", "blocked")
|
||||
HARD_ELIGIBILITY_GATES = frozenset(
|
||||
("generated_files", "static_safety", "images", "network")
|
||||
)
|
||||
QUALITY_SCORING_GATES = frozenset(("console", "responsive", "accessibility"))
|
||||
SCORING_INVOCATION_REASONS = TERMINAL_REASONS + (
|
||||
"binding_mismatch", "evaluator_failed",
|
||||
)
|
||||
|
|
@ -78,6 +82,12 @@ _POST_CLEANUP_TIMEOUT_SECONDS = 300.0
|
|||
_POST_CLEANUP_QUIET_SECONDS = 0.2
|
||||
_POST_CLEANUP_POLL_SECONDS = 0.01
|
||||
|
||||
if (
|
||||
HARD_ELIGIBILITY_GATES & QUALITY_SCORING_GATES
|
||||
or HARD_ELIGIBILITY_GATES | QUALITY_SCORING_GATES != frozenset(WEB_GATES)
|
||||
):
|
||||
raise RuntimeError("scoring web gate partition is invalid")
|
||||
|
||||
|
||||
class ScoringError(Exception):
|
||||
"""Scoring state or evaluator evidence cannot be trusted."""
|
||||
|
|
@ -1054,15 +1064,13 @@ def _eligibility(manifest: Manifest, attempt: Attempt) -> tuple[bool, tuple[str,
|
|||
web = load_web_validation(attempt.root, manifest=manifest)
|
||||
except WebValidationError as exc:
|
||||
raise ScoringError("web eligibility evidence is invalid") from exc
|
||||
if web.status != "passed":
|
||||
reasons.append(f"web_{web.status}")
|
||||
if web.record["reason"]:
|
||||
reasons.append(f"web_reason_{web.record['reason']}")
|
||||
gates = web.record["gates"]
|
||||
if [item["id"] for item in gates] != list(WEB_GATES):
|
||||
raise ScoringError("web eligibility gates are invalid")
|
||||
reasons.extend(
|
||||
f"gate_{item['id']}" for item in gates if not item["passed"]
|
||||
f"gate_{item['id']}"
|
||||
for item in gates
|
||||
if item["id"] in HARD_ELIGIBILITY_GATES and not item["passed"]
|
||||
)
|
||||
return not reasons, tuple(reasons)
|
||||
|
||||
|
|
@ -1248,7 +1256,11 @@ def _materialize_blind(
|
|||
session_identity: str,
|
||||
) -> BlindWorkspace:
|
||||
web = load_web_validation(attempt.root, manifest=manifest)
|
||||
if web.status != "passed" or not all(item["passed"] for item in web.record["gates"]):
|
||||
gates = web.record["gates"]
|
||||
if [item["id"] for item in gates] != list(WEB_GATES) or any(
|
||||
item["id"] in HARD_ELIGIBILITY_GATES and not item["passed"]
|
||||
for item in gates
|
||||
):
|
||||
raise ScoringError("execution attempt is not eligible")
|
||||
identities = _identity_values(manifest, attempt)
|
||||
if _contains_identity(str(blind_root).encode("utf-8"), identities):
|
||||
|
|
|
|||
|
|
@ -436,7 +436,14 @@ class ScoringTest(unittest.TestCase):
|
|||
self.manifest_path = self.root / "manifest.json"
|
||||
self.manifest_path.write_text(json.dumps(raw), encoding="utf-8")
|
||||
self.manifest = load_manifest(self.manifest_path, repo_root=self.root)
|
||||
tokens = iter(("123456abcdef", "234567abcdef", "345678abcdef"))
|
||||
tokens = iter(
|
||||
(
|
||||
"123456abcdef",
|
||||
"234567abcdef",
|
||||
"345678abcdef",
|
||||
*(f"{index:012x}" for index in range(4, 100)),
|
||||
)
|
||||
)
|
||||
self.store = RunStore(
|
||||
self.root,
|
||||
clock=lambda: datetime.datetime(
|
||||
|
|
@ -487,7 +494,13 @@ class ScoringTest(unittest.TestCase):
|
|||
(),
|
||||
)
|
||||
|
||||
def _view(self, root: Path, ident: str, width: int) -> ViewportObservation:
|
||||
def _view(
|
||||
self,
|
||||
root: Path,
|
||||
ident: str,
|
||||
width: int,
|
||||
web_gate_failures: frozenset[str] = frozenset(),
|
||||
) -> ViewportObservation:
|
||||
screenshot = f"screenshot-{ident}.png"
|
||||
png = b"\x89PNG\r\n\x1a\n" + ident.encode("ascii")
|
||||
(root / screenshot).write_bytes(png)
|
||||
|
|
@ -498,7 +511,7 @@ class ScoringTest(unittest.TestCase):
|
|||
"complete": True,
|
||||
"natural_width": 20,
|
||||
"natural_height": 20,
|
||||
"visible": True,
|
||||
"visible": "images" not in web_gate_failures,
|
||||
"rect": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
|
|
@ -521,7 +534,7 @@ class ScoringTest(unittest.TestCase):
|
|||
{
|
||||
"scroll_width": width,
|
||||
"client_width": width,
|
||||
"clipped": 0,
|
||||
"clipped": 1 if "responsive" in web_gate_failures else 0,
|
||||
"overlaps": 0,
|
||||
},
|
||||
{
|
||||
|
|
@ -532,7 +545,7 @@ class ScoringTest(unittest.TestCase):
|
|||
"landmarks": 1,
|
||||
"controls": [
|
||||
{
|
||||
"name": True,
|
||||
"name": "accessibility" not in web_gate_failures,
|
||||
"tab_index": 0,
|
||||
"focused": True,
|
||||
"focus_visible": True,
|
||||
|
|
@ -546,6 +559,7 @@ class ScoringTest(unittest.TestCase):
|
|||
def _attempt(
|
||||
self, state: str = "success", *, leaked: bool = False,
|
||||
rendered: bool = True, leaked_identity: str | None = None,
|
||||
web_gate_failures: frozenset[str] = frozenset(),
|
||||
):
|
||||
with self.store.writer(self.run):
|
||||
attempt = self.store.allocate(self.run, Slot("cell-sentinel", 1))
|
||||
|
|
@ -569,6 +583,14 @@ class ScoringTest(unittest.TestCase):
|
|||
(workspace / "script.js").write_text(
|
||||
"document.body.dataset.ready='1';", encoding="utf-8"
|
||||
)
|
||||
if "generated_files" in web_gate_failures:
|
||||
(workspace / "styles.css").unlink()
|
||||
elif "static_safety" in web_gate_failures:
|
||||
(workspace / "index.html").write_text(
|
||||
"<main><h1>Ready</h1><img src='https://example.invalid/a.png' "
|
||||
"alt='A'></main>",
|
||||
encoding="utf-8",
|
||||
)
|
||||
terminal_reason = "success" if state == "success" else "caller_error"
|
||||
measurement = self._measurement(attempt, terminal_reason)
|
||||
publish_measurement(attempt.root, measurement)
|
||||
|
|
@ -580,11 +602,24 @@ class ScoringTest(unittest.TestCase):
|
|||
{"kind": "local", "path": "/index.html", "allowed": True, "status": 200},
|
||||
{"kind": "local", "path": "/assets/a.svg", "allowed": True, "status": 200},
|
||||
{"kind": "local", "path": "/assets/b.svg", "allowed": True, "status": 200},
|
||||
*(
|
||||
({"kind": "external", "url_digest": _digest(b"blocked"), "allowed": False, "status": 0},)
|
||||
if "network" in web_gate_failures
|
||||
else ()
|
||||
),
|
||||
),
|
||||
(),
|
||||
(
|
||||
self._view(Path(attempt.root), "desktop", 800),
|
||||
self._view(Path(attempt.root), "mobile", 375),
|
||||
({"kind": "console", "level": "error"},)
|
||||
if "console" in web_gate_failures
|
||||
else ()
|
||||
),
|
||||
(
|
||||
self._view(
|
||||
Path(attempt.root), "desktop", 800, web_gate_failures
|
||||
),
|
||||
self._view(
|
||||
Path(attempt.root), "mobile", 375, web_gate_failures
|
||||
),
|
||||
),
|
||||
)
|
||||
else:
|
||||
|
|
@ -787,19 +822,87 @@ class ScoringTest(unittest.TestCase):
|
|||
[
|
||||
"product_failed",
|
||||
"process_nonzero_exit",
|
||||
"web_failed",
|
||||
"web_reason_render_not_run",
|
||||
"gate_images",
|
||||
"gate_network",
|
||||
"gate_console",
|
||||
"gate_responsive",
|
||||
"gate_accessibility",
|
||||
],
|
||||
)
|
||||
self.assertFalse(set(record) & {"score", "total", "worksheet"})
|
||||
score_run(self.store, self.run, self.manifest, adapter=adapter)
|
||||
self.assertEqual(path.read_bytes(), before)
|
||||
|
||||
def test_each_hard_web_gate_failure_is_unscored_without_evaluator(self):
|
||||
for gate in sorted(scoring_module.HARD_ELIGIBILITY_GATES):
|
||||
with self.subTest(gate=gate):
|
||||
self.run = self.store.create(
|
||||
self.manifest, self.manifest_path.read_bytes()
|
||||
)
|
||||
attempt = self._attempt(web_gate_failures=frozenset((gate,)))
|
||||
adapter = FakeScoringAdapter()
|
||||
summary = score_run(
|
||||
self.store, self.run, self.manifest, adapter=adapter
|
||||
)
|
||||
self.assertEqual((summary.unscored, summary.scored), (1, 0))
|
||||
self.assertEqual((adapter.preflights, adapter.invocations), (0, []))
|
||||
record = json.loads(
|
||||
(Path(attempt.root) / "scoring" / "unscored.json").read_text()
|
||||
)
|
||||
self.assertIn(f"gate_{gate}", record["reasons"])
|
||||
self.assertFalse(
|
||||
any(
|
||||
f"gate_{quality}" in record["reasons"]
|
||||
for quality in scoring_module.QUALITY_SCORING_GATES
|
||||
)
|
||||
)
|
||||
|
||||
def test_each_quality_only_web_failure_remains_score_eligible(self):
|
||||
for gate in sorted(scoring_module.QUALITY_SCORING_GATES):
|
||||
with self.subTest(gate=gate):
|
||||
self.run = self.store.create(
|
||||
self.manifest, self.manifest_path.read_bytes()
|
||||
)
|
||||
attempt = self._attempt(web_gate_failures=frozenset((gate,)))
|
||||
adapter = FakeScoringAdapter()
|
||||
summary = score_run(
|
||||
self.store, self.run, self.manifest, adapter=adapter
|
||||
)
|
||||
self.assertEqual((summary.scored, summary.unscored), (1, 0))
|
||||
self.assertEqual(adapter.preflights, 1)
|
||||
self.assertEqual(len(adapter.invocations), 1)
|
||||
self.assertFalse((Path(attempt.root) / "scoring" / "unscored.json").exists())
|
||||
|
||||
def test_mixed_hard_and_quality_web_failures_remain_unscored(self):
|
||||
attempt = self._attempt(
|
||||
web_gate_failures=frozenset(("network", "accessibility"))
|
||||
)
|
||||
adapter = FakeScoringAdapter()
|
||||
summary = score_run(self.store, self.run, self.manifest, adapter=adapter)
|
||||
self.assertEqual((summary.unscored, summary.scored), (1, 0))
|
||||
self.assertEqual((adapter.preflights, adapter.invocations), (0, []))
|
||||
record = json.loads(
|
||||
(Path(attempt.root) / "scoring" / "unscored.json").read_text()
|
||||
)
|
||||
self.assertIn("gate_network", record["reasons"])
|
||||
self.assertNotIn("gate_accessibility", record["reasons"])
|
||||
|
||||
def test_missing_or_malformed_web_evidence_fails_closed(self):
|
||||
for mode in ("missing", "malformed"):
|
||||
with self.subTest(mode=mode):
|
||||
self.run = self.store.create(
|
||||
self.manifest, self.manifest_path.read_bytes()
|
||||
)
|
||||
attempt = self._attempt()
|
||||
path = Path(attempt.root) / "web-validation.json"
|
||||
if mode == "missing":
|
||||
path.unlink()
|
||||
else:
|
||||
path.write_bytes(b"not-json\n")
|
||||
adapter = FakeScoringAdapter()
|
||||
with self.assertRaises((ScoringError, AttemptStateError)):
|
||||
score_run(
|
||||
self.store, self.run, self.manifest, adapter=adapter
|
||||
)
|
||||
self.assertEqual((adapter.preflights, adapter.invocations), (0, []))
|
||||
|
||||
def test_not_run_web_gates_are_all_unscored_without_evaluator(self):
|
||||
attempt = self._attempt(rendered=False)
|
||||
adapter = FakeScoringAdapter()
|
||||
|
|
@ -813,11 +916,12 @@ class ScoringTest(unittest.TestCase):
|
|||
web = json.loads(
|
||||
(Path(attempt.root) / "web-validation.json").read_text()
|
||||
)
|
||||
failed_gates = {
|
||||
failed_hard_gates = {
|
||||
f"gate_{item['id']}" for item in web["gates"] if not item["passed"]
|
||||
and item["id"] in scoring_module.HARD_ELIGIBILITY_GATES
|
||||
}
|
||||
self.assertTrue(failed_gates)
|
||||
self.assertTrue(failed_gates.issubset(record["reasons"]))
|
||||
self.assertTrue(failed_hard_gates)
|
||||
self.assertTrue(failed_hard_gates.issubset(record["reasons"]))
|
||||
self.assertEqual(
|
||||
[item["id"] for item in web["gates"]], list(WEB_GATES)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 180,
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
|
|
|
|||
|
|
@ -0,0 +1,135 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-recovery-qualification-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-02-recovery",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "c01-claude-sonnet-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "max",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-sonnet-5",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c03-agy-gemini-direct",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c04-claude-gpt-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c06-claude-gemini-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c07-agy-gemini-hybrid",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c08-claude-gpt-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-hybrid",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gpt-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -66,7 +66,7 @@
|
|||
|---|---|---|---|---|---|
|
||||
| cell-sentinel/r1/a1 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded |
|
||||
| cell-sentinel/r1/a2 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded |
|
||||
| cell-sentinel/r1/a3 | generated_files=pass, static_safety=pass, images=fail, network=fail, console=fail, responsive=fail, accessibility=fail | unavailable | — | unavailable | product_failed, process_nonzero_exit, web_failed, web_reason_render_not_run, gate_images, gate_network, gate_console, gate_responsive, gate_accessibility |
|
||||
| cell-sentinel/r1/a3 | generated_files=pass, static_safety=pass, images=fail, network=fail, console=fail, responsive=fail, accessibility=fail | unavailable | — | unavailable | product_failed, process_nonzero_exit, gate_images, gate_network |
|
||||
| cell-sentinel/r1/a4 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | invalid_worksheet |
|
||||
| cell-sentinel/r1/a5 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | — | unavailable | evaluator_preflight_blocked |
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue