feat(epic): evidence-report 작업을 준비한다
This commit is contained in:
parent
69e986a9b3
commit
bcfb3bb84e
23 changed files with 3795 additions and 18 deletions
|
|
@ -0,0 +1,172 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/14_timing_usage plan=1 tag=API milestone-task=timing-usage -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/14_timing_usage, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/plan_cloud_G06_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/code_review_cloud_G06_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: replace final-snapshot first-write inference with a bounded observer started before invocation; normalize fractional durations without accepting fractional token/count fields.
|
||||
- Verification evidence: all three changed example manifests validated locally; the prior pair contains no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=timing-usage`, SDD S11/Evidence Map S11.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_1.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/14_timing_usage/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Closed observation and measurement schema | [ ] |
|
||||
| API-2 — Bounded first-output/first-write observation | [ ] |
|
||||
| API-3 — Caller-specific metric integration | [ ] |
|
||||
| API-4 — Complete timing/usage verification | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the closed metric/measurement schema with source, clock, availability, overlap, calls, integer token invariants, and lossless decimal-duration normalization.
|
||||
- [ ] [API-2] Record first output and run a bounded pre-invocation workspace observer, publish/validate `attempt-measurement.json` without clobbering, and cover timeout, unavailable, tamper, non-regular, and recovery paths.
|
||||
- [ ] [API-3] Teach Claude, agy, and Codex parsers/fixtures to emit only validated observations, integrate publication into `run_slots`, and prove redaction plus three-caller evidence.
|
||||
- [ ] [API-4] Run focused/full benchmark tests and all three example-manifest validations with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G07_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G07_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/14_timing_usage/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify duration decimals normalize losslessly while token/count fields reject fractions.
|
||||
- Verify the workspace observer starts before caller invocation, stops/joins on every terminal path, and reports observation precision/source.
|
||||
- Verify no final `mtime` snapshot is presented as the true first write and no cross-clock arithmetic occurs.
|
||||
- Verify totals/intervals are not synthesized and sidecar corruption fails closed before terminal publication.
|
||||
- Verify durable evidence contains no task, endpoint, secret, config, or provider sentinel.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test
|
||||
python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips and no leaked sentinel.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused timing/usage suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
|
||||
Expected: exit 0, no failures/errors/skips, observer cleanup and credential-free evidence pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
```text
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json
|
||||
```
|
||||
|
||||
Expected: each exits 0 and prints `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,258 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/14_timing_usage plan=1 tag=API milestone-task=timing-usage -->
|
||||
|
||||
# Source-aware timing and usage evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G07.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The lifecycle records submission, terminal ordering, and total duration, but discards non-terminal output timing and stores provider metrics only as names. SDD S11 requires caller/model-aware timing and usage with clock/source provenance, explicit unavailable values, and no invented arithmetic decomposition. Self-review corrected the original end-of-run `mtime` reconstruction because a final snapshot cannot prove the first write.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/plan_cloud_G06_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/code_review_cloud_G06_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: replace final-snapshot first-write inference with a bounded observer started before invocation; normalize fractional durations without accepting fractional token/count fields.
|
||||
- Verification evidence: all three changed example manifests validated locally; the prior pair contains no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=timing-usage`, SDD S11/Evidence Map S11.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/lifecycle_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop_test.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop_test.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/plan_cloud_G06_0.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/code_review_cloud_G06_0.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md` is approved with lock released.
|
||||
- `milestone-task=timing-usage`; S11 requires submitted/first-output/first-write/model/tool/queue/total timing, calls, token categories, clock/source, overlap, and unavailable preservation.
|
||||
- Evidence Map S11 requires normalized timeline/usage fixtures and unavailable handling. API-1 through API-4 and final verification directly cover this row.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No verification handoff was supplied. Repository-native evidence is the lifecycle/attempt/caller code, fixtures, approved SDD, benchmark skill, and local testing rules.
|
||||
- Python `unittest` is uncached. Unit/integration tests use fake runners only; no caller credential, network, browser, or live provider is required.
|
||||
- The workspace observer reports its harness monotonic observation time, filesystem metadata source, and polling precision. It never labels a final `mtime` as the true first write or compares filesystem wall time with monotonic time.
|
||||
- Caller duration decimals are normalized deterministically to integer nanoseconds with `Decimal(str(value))`; token and call-count observations remain non-negative integers.
|
||||
- Confidence: high for harness observations and append-only evidence; provider omissions remain source-qualified unavailable.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No test records first non-empty output independently of terminal events.
|
||||
- No observer starts before invocation or proves first-observed write ordering under repeated writes.
|
||||
- Provider parsers do not preserve validated numeric metric payloads or token categories.
|
||||
- Attempt recovery does not validate an immutable timing/usage sidecar.
|
||||
- The three-caller integration does not prove source-aware, redacted evidence.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `parse_event` consumers are lifecycle tests, attempt tests, three caller adapters, and connectivity integration. Terminal strings remain compatible; typed metrics are additive.
|
||||
- `ExecutionAdapter.invoke` remains `InvocationResult`; `run_slots` owns observer and measurement publication around the call.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one invariant: one immutable `attempt-measurement.json` joins lifecycle events, provider observations, and the bounded workspace observer for one attempt.
|
||||
- It has no predecessor. Sibling indices 01-13 are occupied by unchanged completed work; 14 is the existing stable index.
|
||||
- Web, scoring, and reporting remain later children and consume the strict loader.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude browser/render validation, evaluator fields, scoring, reports, and benchmark skill changes.
|
||||
- Do not infer provider time by subtraction, add missing token totals, compare clocks across domains, or claim a final snapshot is the first write.
|
||||
- Do not change OpenAI/Anthropic wire contracts; only caller CLI observations are consumed.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build/review closures are all true. Scores `(scope=2,state=2,blast=1,evidence=1,verification=1)` produce G07.
|
||||
- Build base `local-fit`, route `risk-boundary`, `worker/cloud/G07`, `PLAN-cloud-G07.md`; review `official-review`, `review/cloud/G07`, `CODE_REVIEW-cloud-G07.md`.
|
||||
- `large_indivisible_context=false`; loop risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product` (5). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the closed metric/measurement schema with source, clock, availability, overlap, calls, integer token invariants, and lossless decimal-duration normalization.
|
||||
- [ ] [API-2] Record first output and run a bounded pre-invocation workspace observer, publish/validate `attempt-measurement.json` without clobbering, and cover timeout, unavailable, tamper, non-regular, and recovery paths.
|
||||
- [ ] [API-3] Teach Claude, agy, and Codex parsers/fixtures to emit only validated observations, integrate publication into `run_slots`, and prove redaction plus three-caller evidence.
|
||||
- [ ] [API-4] Run focused/full benchmark tests and all three example-manifest validations with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Define a closed observation and normalization contract
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/lifecycle.py:201-209` stores opaque event detail, and `scripts/agent_benchmark/lifecycle.py:1293-1318` discards metric values, units, clocks, and sources. Agy currently accepts integer or decimal `duration_ms`, while token/count fields require integer semantics.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/lifecycle.py:200-208`):
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class LifecycleEvent:
|
||||
kind: str
|
||||
source: str
|
||||
stream: str
|
||||
monotonic_ns: int
|
||||
source_monotonic_ns: int
|
||||
observed_at: str
|
||||
detail: str
|
||||
```
|
||||
|
||||
Add a typed `ParsedMetric` with closed name/unit/clock/source/stage/model/call id/overlap fields. Normalize finite non-negative duration integers or decimals to integer nanoseconds with `Decimal(str(value))`; reject booleans, non-finite values, precision that cannot be represented in nanoseconds, and every fractional token/call count. Keep terminal strings compatible.
|
||||
|
||||
Add `measurement.py` with strict immutable records and load/publish functions. Every required value is either `{status: "observed", value, unit, clock, source}` or `{status: "unavailable", value: null, reason, source}`. Preserve reported totals only; do not synthesize them or subtract overlapping intervals.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/lifecycle.py`: typed metrics and canonical safe events.
|
||||
- [ ] `scripts/agent_benchmark/measurement.py`: schema, normalization, no-clobber publisher, strict loader.
|
||||
- [ ] `scripts/agent_benchmark/lifecycle_test.py`: valid/invalid metrics, duration decimals, bounds, redaction, terminal compatibility.
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: observed/unavailable, clock/source, overlap, and no arithmetic synthesis.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write explicit tests for decimal duration to nanoseconds, fractional token rejection, unavailable null-not-zero, preserved provider total, and overlapping intervals with fixed clocks.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test`; expect exit 0 with no skips.
|
||||
|
||||
### [API-2] Observe first output and first workspace write during invocation
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/lifecycle.py:1263-1270` updates only a quiet timer. `scripts/agent_benchmark/attempts.py:1490-1500` has no write observer. Comparing before/after snapshots and choosing the earliest final `st_mtime_ns` cannot prove the first write because `mtime` is the last modification and transient files can disappear.
|
||||
|
||||
**Solution**
|
||||
|
||||
Record `first_output` exactly once on the first non-empty caller frame. Start a `WorkspaceWriteObserver` after the immutable baseline snapshot and before `invoke`; sample contained regular files at a bounded interval using a dedicated thread, record the first observed created/changed path with harness `monotonic_ns`, filesystem `st_mtime_ns`, source `workspace_poll`, and the configured observation precision, then stop/join it on every success/error/timeout/cancel path.
|
||||
|
||||
The observer must ignore only harness-owned paths outside the workspace, reject symlink/non-regular traversal, and report `unavailable` when no write is observed. It must never reconstruct a first-write time from the terminal snapshot. Publish one sidecar after observer shutdown and before terminal state publication; corruption or collision fails closed without mutating prior bytes.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/lifecycle.py`: exactly-once first-output event.
|
||||
- [ ] `scripts/agent_benchmark/measurement.py`: baseline scan, bounded observer, precision/source fields, sidecar coherence.
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: start/stop observer around invocation and validate measurement before terminal commit.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: lifecycle failure, timeout, cancellation, no-clobber, tamper, symlink/FIFO, recovery, thread cleanup.
|
||||
- [ ] `scripts/agent_benchmark/lifecycle_test.py`: first output ordering.
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: created/modified/repeated writes, no-write unavailable, mixed clocks, observer cleanup.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use fixed clocks and synchronization barriers so a file is written twice around another file; assert the observer preserves first-observed order rather than final `mtime` order. Test no lingering thread and prior-byte preservation on every terminal path.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test`; expect exit 0 with no skips.
|
||||
|
||||
### [API-3] Capture caller-specific metrics without leaking content
|
||||
|
||||
**Problem**
|
||||
|
||||
Claude/Codex parsers return terminal strings only, and `scripts/agent_benchmark/agy_iop.py:373-399` reduces a duration observation to `metric:duration_ms`. No caller produces normalized timing/usage evidence.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add exact per-caller allowlists. Claude consumes reported usage/durations and assistant-call counts; agy converts allowlisted duration metrics losslessly; Codex consumes reported usage, turn-call counts, and explicitly paired tool intervals. Unknown, negative, boolean, malformed, duplicate-total, binding-mismatched, or fractional count/token observations fail closed. Redactors persist only safe identifiers and numeric values.
|
||||
|
||||
Extend all three JSONL fixtures with representative observed and omitted fields. The integration must assert one strict `attempt-measurement.json` per attempt and scan durable bytes for task, endpoint, secret, config, and provider sentinels.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/codex_iop.py`: emit validated observations.
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: preserve lifecycle result while retaining typed observations.
|
||||
- [ ] `scripts/agent_benchmark/claude_iop_test.py`, `scripts/agent_benchmark/agy_iop_test.py`, `scripts/agent_benchmark/codex_iop_test.py`: normal/missing/malformed/duplicate/redaction cases.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: require redacted normalized sidecars.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl`, `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`, `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`: safe timing/usage fixtures.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Test every supported field and malformed boundary. Ensure caller-specific missing categories stay unavailable and source-qualified.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`; expect exit 0 and no durable sentinel.
|
||||
|
||||
### [API-4] Verify the complete timing/usage slice
|
||||
|
||||
**Problem**
|
||||
|
||||
Focused tests do not prove strict recovery and manifest compatibility across the benchmark package.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run the focused suite, full discovery, and all three checked-in manifests with fresh output. Do not accept reconstructed output.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: keep focused coverage module-runnable.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: keep the production-shaped fake path credential-free.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No extra file; API-1 through API-3 own unit, race, corruption, and integration coverage.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run all commands in Final Verification; every command exits 0.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/measurement.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/measurement_test.py` | API-1, API-2, API-4 |
|
||||
| `scripts/agent_benchmark/lifecycle.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/lifecycle_test.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/attempts.py` | API-2 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2 |
|
||||
| `scripts/agent_benchmark/claude_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/claude_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/agy_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/agy_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/codex_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/codex_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-3, API-4 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl` | API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/CODE_REVIEW-cloud-G07.md` | API-1, API-2, API-3, API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
- Expected: exit 0, no failures/errors/skips, observer threads cleaned up, credential-free production-path evidence passes.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with the complete benchmark suite passing; cached output is not acceptable.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
4. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- Expected for 3-5: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/14_timing_usage plan=0 tag=API milestone-task=timing-usage -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/14_timing_usage, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_0.log` and `PLAN-cloud-G06.md` → `plan_cloud_G06_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/14_timing_usage/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=timing-usage` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Closed observation and normalization contract | [ ] |
|
||||
| API-2 — First-output/first-write attempt evidence | [ ] |
|
||||
| API-3 — Caller-specific metric integration | [ ] |
|
||||
| API-4 — Complete timing/usage verification | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the closed metric observation and normalized measurement schema, including source/clock, availability, overlap, call-count, and token invariants, with normal/boundary tests.
|
||||
- [ ] [API-2] Record first output and first workspace write, publish/validate `attempt-measurement.json` without clobbering, and cover timeout, unavailable, tamper, symlink, and recovery paths.
|
||||
- [ ] [API-3] Teach Claude, agy, and Codex fixtures/parsers to emit only validated numeric observations, integrate measurement publication into `run_slots`, and prove redaction plus three-caller evidence.
|
||||
- [ ] [API-4] Run the focused and full benchmark test suites plus all three example-manifest validations with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_0.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G06.md` to `plan_cloud_G06_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/14_timing_usage/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=timing-usage` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify every observed or unavailable value carries a closed source and clock without cross-clock arithmetic.
|
||||
- Verify provider totals/intervals are never synthesized by subtraction or token addition.
|
||||
- Verify sidecar tampering/non-regular files fail closed before attempt terminal publication and preserve bytes.
|
||||
- Verify lifecycle, parser, and integration evidence contains no task, endpoint, secret, config, or provider sentinel.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test
|
||||
python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips; the integration command persists no sentinel.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused timing/usage suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
|
||||
Expected: exit 0, no failures/errors/skips, and credential-free production-path evidence passes.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0; cached output is not acceptable.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json
|
||||
```
|
||||
|
||||
Expected: each exits 0 and prints `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,343 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/14_timing_usage plan=0 tag=API milestone-task=timing-usage -->
|
||||
|
||||
# Source-aware timing and usage evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G06.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The lifecycle currently records submission, terminal ordering, and total duration, but it discards non-terminal output timing and stores provider metrics only as names. SDD S11 requires caller/model-aware timings and usage with clock/source provenance, explicit unavailable values, and no invented arithmetic decomposition.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/lifecycle_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop_test.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop_test.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`, approved and implementation lock released.
|
||||
- First-line milestone task: `timing-usage`.
|
||||
- Acceptance Scenario S11 requires prompt submission, first output, first file write, model/tool/queue/total timing, call counts, token categories, clock/source, overlap, and unavailable preservation.
|
||||
- Evidence Map S11 requires normalized timeline/usage fixtures and unavailable handling under this task. These requirements define API-1 through API-3 and all final assertions.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Repository-native evidence came from the lifecycle, attempt, three caller adapter, live integration, fixture, SDD, local test rule, and benchmark skill files listed above.
|
||||
- Existing deterministic commands are Python `unittest` modules and manifest CLI validation. `unittest` has no result cache; cached output is not acceptable.
|
||||
- Preconditions: Python 3.12+, standard library only, no live caller credentials, no network, and no browser are required for this slice.
|
||||
- Gap: current fixtures do not carry complete usage for every caller. The contract must represent absent provider values as unavailable rather than making all callers look complete.
|
||||
- Confidence: high for harness clocks and append-only evidence; provider-specific fields remain source-qualified and fail closed.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No test records first non-empty output independently of terminal events.
|
||||
- No test snapshots first workspace write or distinguishes wall-clock filesystem timestamps from monotonic harness time.
|
||||
- Provider parsers do not preserve numeric metric payloads or validate token categories.
|
||||
- Attempt recovery tests do not validate a normalized timing/usage sidecar or reject tampering/non-regular files.
|
||||
- No integration assertion proves all three callers create redacted, source-aware evidence.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `parse_event` call sites: `run_invocation` consumers in `lifecycle_test.py`, `attempts_test.py`, `claude_iop.py`, `agy_iop.py`, `codex_iop.py`, and `connectivity_integration_test.py`. String terminal returns remain compatible; typed metric returns are additive.
|
||||
- `ExecutionAdapter.invoke` call sites: `run_slots` and fake/live adapters in `attempts_test.py` and `connectivity_integration_test.py`. Its return type remains `InvocationResult`.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one stable contract: a single immutable `attempt-measurement.json` derived from lifecycle events, provider-observed metrics, and a before/after workspace snapshot.
|
||||
- PASS evidence is typed parser tests, lifecycle ordering/redaction tests, attempt corruption/recovery tests, and the three-caller integration fixture.
|
||||
- It has no predecessor. Sibling indices 01-13 are occupied by unchanged archived work; 14 is the lowest collision-free index.
|
||||
- Later web, scoring, and report children consume this evidence but are outside this write boundary.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude browser/render validation, evaluator manifest fields, scoring, report generation, and benchmark skill changes; they belong to task ids `web-validation`, `blind-score`, and `report-output`.
|
||||
- Do not infer provider time by subtracting overlapping intervals, compute missing token totals, or treat prompt-to-output latency as provider queue time.
|
||||
- Do not change outer OpenAI/Anthropic wire contracts: the implementation consumes caller CLI observations only.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision are all true. Scores `(scope=2,state=1,blast=1,evidence=1,verification=1)` produce G06; base `local-fit`, route `risk-boundary`, `worker/cloud/G06`, `PLAN-cloud-G06.md`.
|
||||
- Review closures are all true. Scores `(2,1,1,1,1)` produce `official-review`, `review/cloud/G06`, `CODE_REVIEW-cloud-G06.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the closed metric observation and normalized measurement schema, including source/clock, availability, overlap, call-count, and token invariants, with normal/boundary tests.
|
||||
- [ ] [API-2] Record first output and first workspace write, publish/validate `attempt-measurement.json` without clobbering, and cover timeout, unavailable, tamper, symlink, and recovery paths.
|
||||
- [ ] [API-3] Teach Claude, agy, and Codex fixtures/parsers to emit only validated numeric observations, integrate measurement publication into `run_slots`, and prove redaction plus three-caller evidence.
|
||||
- [ ] [API-4] Run the focused and full benchmark test suites plus all three example-manifest validations with fresh output.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Define a closed observation and normalization contract
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/lifecycle.py:201-209` records every event detail as an opaque string, while `scripts/agent_benchmark/lifecycle.py:1293-1318` accepts metric names but discards their numeric value, unit, clock, and source. There is no normalized schema that can preserve an unavailable field without inventing zero.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/lifecycle.py:200-208`):
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class LifecycleEvent:
|
||||
kind: str
|
||||
source: str
|
||||
stream: str
|
||||
monotonic_ns: int
|
||||
source_monotonic_ns: int
|
||||
observed_at: str
|
||||
detail: str
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ParsedMetric:
|
||||
name: str
|
||||
value: int
|
||||
unit: str
|
||||
clock: str
|
||||
source: str
|
||||
stage: str | None = None
|
||||
model: str | None = None
|
||||
call_id: str | None = None
|
||||
overlap: bool | None = None
|
||||
```
|
||||
|
||||
Keep terminal strings compatible, but accept `ParsedMetric` only after validating its closed identifiers, non-negative integer value, allowlisted unit/clock/source, and redaction-safe canonical JSON detail. Add `measurement.py` with immutable records and strict load/publish functions. Every required value uses `{status: "observed", value, unit, clock, source}` or `{status: "unavailable", value: null, reason, source}`. Preserve provider-reported total tokens only; never derive missing totals or subtract model/tool/queue intervals. Use only standard-library imports:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
```
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/lifecycle.py`: add `ParsedMetric`, closed validation, and canonical metric events.
|
||||
- [ ] `scripts/agent_benchmark/measurement.py`: add schema, normalization, no-clobber publisher, and strict loader.
|
||||
- [ ] `scripts/agent_benchmark/lifecycle_test.py`: test valid/invalid typed metrics, bounds, redaction, and string compatibility.
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: test observed/unavailable fields, source clocks, overlap, and no arithmetic synthesis.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write `MeasurementTest.test_unavailable_values_are_null_not_zero`, `test_source_total_is_not_recomputed`, `test_overlapping_intervals_are_preserved`, and lifecycle typed-metric boundary tests. Use fixed clocks and temporary directories.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test`; expect all tests to pass with no skips.
|
||||
|
||||
### [API-2] Bind first-output and first-write evidence to an attempt
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/lifecycle.py:1263-1270` updates only an in-memory quiet timer for arbitrary output. `scripts/agent_benchmark/attempts.py:1490-1500` invokes the caller without a workspace snapshot, so the first changed file and its filesystem clock cannot be recovered reliably.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/lifecycle.py:1263-1270`):
|
||||
|
||||
```python
|
||||
def _handle_output(self, frame: dict[str, Any]) -> None:
|
||||
stream = str(frame.get("stream", "stdout"))
|
||||
capture = self.captures.get(stream)
|
||||
if capture is None:
|
||||
return
|
||||
self.last_output_at = time.monotonic()
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
if not self.first_output_recorded and str(frame.get("data", "")):
|
||||
self.first_output_recorded = True
|
||||
self._add_event("first_output", SOURCE_CALLER_OUTPUT, stream, frame, "first_output")
|
||||
self.last_output_at = time.monotonic()
|
||||
```
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:1490-1499`):
|
||||
|
||||
```python
|
||||
lease = store.acquire_control_lease(current)
|
||||
return adapters[cell.caller].invoke(
|
||||
cell,
|
||||
prepared,
|
||||
current,
|
||||
lease.control_dir,
|
||||
manifest.fixture.prompt_content,
|
||||
manifest.timeout,
|
||||
on_started,
|
||||
)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
baseline = snapshot_workspace(prepared.workspace_dir)
|
||||
invocation = adapters[cell.caller].invoke(
|
||||
cell,
|
||||
prepared,
|
||||
current,
|
||||
lease.control_dir,
|
||||
manifest.fixture.prompt_content,
|
||||
manifest.timeout,
|
||||
on_started,
|
||||
)
|
||||
publish_attempt_measurement(
|
||||
current.root, cell, invocation, baseline,
|
||||
snapshot_workspace(prepared.workspace_dir),
|
||||
)
|
||||
return invocation
|
||||
```
|
||||
|
||||
The workspace snapshot records regular files by relative path, identity, size, digest, and `st_mtime_ns`. The normalized first write is the earliest created/changed regular file by filesystem timestamp, marked with clock `filesystem.wall_ns` and source `stat.st_mtime_ns`; it is never compared arithmetically with harness monotonic time. Ignore session/control/evidence paths, reject symlinks and non-regular entries, and publish exactly one immutable sidecar even for lifecycle failure so unavailable reasons remain visible. Extend `RunStore` terminal validation to require a coherent sidecar for newly executed attempts and to validate it before terminal state publication.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/lifecycle.py`: add exactly-once `first_output` event.
|
||||
- [ ] `scripts/agent_benchmark/measurement.py`: snapshot, first-write selection, and sidecar coherence validation.
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: snapshot around invocation and validate measurement before terminal publication.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: cover failed/timeout/unavailable publication, tamper, symlink/FIFO, no-clobber, and recovery coherence.
|
||||
- [ ] `scripts/agent_benchmark/lifecycle_test.py`: prove first output precedes finish and is not synthesized from submission.
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: cover created/modified files and mixed clock domains.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write deterministic filesystem tests with explicitly set nanosecond mtimes. Extend durable special-file and cross-record corruption tables so a malformed sidecar fails closed without changing attempt state or prior bytes.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test`; expect all tests to pass with no skips.
|
||||
|
||||
### [API-3] Capture caller-specific metrics without leaking content
|
||||
|
||||
**Problem**
|
||||
|
||||
Claude and Codex parsers in `claude_iop.py:190-255` and `codex_iop.py:250-324` return terminal strings only. Agy at `agy_iop.py:373-399` returns only `metric:duration_ms`, losing the numeric value. The live integration at `live_iop.py:427-491` therefore cannot persist model calls, provider timing, or token sources.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/agy_iop.py:387-388`):
|
||||
|
||||
```python
|
||||
if event_type == "metric" and subtype == "duration_ms":
|
||||
return "metric:duration_ms" if isinstance(event.get("value"), (int, float)) and not isinstance(event.get("value"), bool) else "malformed"
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
if event_type == "metric":
|
||||
return parse_agy_metric(event) # ParsedMetric or the fail-closed "malformed" terminal
|
||||
```
|
||||
|
||||
Add exact allowlists per caller. Claude consumes numeric `usage`, `duration_ms`, `duration_api_ms`, and observed assistant-call count when present; agy consumes allowlisted metric subtypes; Codex consumes numeric `usage`, turn-call count, and explicitly paired tool intervals. Unknown, boolean, negative, fractional token, mismatched model/stage/call-id, or duplicate total fields fail closed. Update redactors so the canonical metric event contains only safe identifiers and non-negative integers. Extend the three JSONL fixtures with representative observed and omitted fields, and make integration assert `attempt-measurement.json` contains no task, endpoint, secret, config, or provider sentinel.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/claude_iop.py`: emit validated Claude `ParsedMetric` observations.
|
||||
- [ ] `scripts/agent_benchmark/agy_iop.py`: preserve validated agy metric values and sources.
|
||||
- [ ] `scripts/agent_benchmark/codex_iop.py`: emit validated turn/tool/usage observations.
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: retain the parser path while returning the unchanged lifecycle result.
|
||||
- [ ] `scripts/agent_benchmark/claude_iop_test.py`: normal, missing, duplicate, malformed, and redaction assertions.
|
||||
- [ ] `scripts/agent_benchmark/agy_iop_test.py`: numeric metric/source and rejection assertions.
|
||||
- [ ] `scripts/agent_benchmark/codex_iop_test.py`: turn/tool/token source and pairing assertions.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: require one redacted normalized sidecar per live fixture attempt.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl`: add safe observed Claude timing/usage fields.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`: add safe observed and unavailable-category coverage.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`: add safe observed Codex usage/tool events.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write explicit parser tests for every supported field and malformed boundary. Extend the production-shaped three-caller fixture so source fields differ by caller and assert missing categories stay unavailable.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`; expect all tests to pass with no skips and no sentinel in durable bytes.
|
||||
|
||||
### [API-4] Verify the complete timing/usage slice
|
||||
|
||||
**Problem**
|
||||
|
||||
Focused tests alone do not prove strict lifecycle/recovery compatibility across the benchmark package.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run the focused suite first, the full benchmark discovery suite second, then validate all three checked-in manifests. Do not accept cached or reconstructed output.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/measurement_test.py`: keep focused coverage runnable as a module.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: keep the credential-free integration path deterministic.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional test file; API-1 through API-3 provide unit, corruption, and integration coverage.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run the commands in Final Verification; every command must exit 0 and the discovery run must report no failures or errors.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/measurement.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/measurement_test.py` | API-1, API-2, API-4 |
|
||||
| `scripts/agent_benchmark/lifecycle.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/lifecycle_test.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/attempts.py` | API-2 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2 |
|
||||
| `scripts/agent_benchmark/claude_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/claude_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/agy_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/agy_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/codex_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/codex_iop_test.py` | API-3 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-3, API-4 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl` | API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/CODE_REVIEW-cloud-G06.md` | API-1, API-2, API-3, API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.measurement_test scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
- Expected: exit 0, no failures/errors/skips, and credential-free production-path evidence passes.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with the complete benchmark suite passing. Cached output is not acceptable.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
4. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- Expected for commands 3-5: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/15+14_web_validation plan=1 tag=API milestone-task=web-validation -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/15+14_web_validation, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/plan_cloud_G08_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/code_review_cloud_G08_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor 14 through active-or-archived task evidence; block non-loopback browser requests before dispatch; distinguish the three generated root files from declared fixture inputs; verify visible images, bounded layout, focus/navigation, heading order, accessible names, and contrast.
|
||||
- Verification evidence: all three fixture manifests validated locally; no browser/provider was invoked and the prior pair has no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=web-validation`, SDD D11/S12/Evidence Map S12.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/15+14_web_validation/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Contained Chromium renderer and network denial | [ ] |
|
||||
| API-2 — Complete automatic gates and immutable record | [ ] |
|
||||
| API-3 — S12 production-path integration | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement contained loopback serving and a bounded standard-library Chromium CDP client with pre-dispatch non-loopback denial, browser identity, viewport screenshots, console/layout/accessibility observations, and cleanup tests.
|
||||
- [ ] [API-2] Implement fixture-aware generated-file checks, visible-image/no-network/no-framework/asset/console/responsive/accessibility gates, and immutable `web-validation.json` for passed/failed/blocked/not-run with corruption coverage.
|
||||
- [ ] [API-3] Integrate web validation after timing evidence, prove both images and desktop/mobile outputs in the three-caller fake path, and run the mandatory real Chromium smoke plus full suite.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G08_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G08_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/15+14_web_validation/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require predecessor 14 PASS from the exact active or dated archive `complete.log`; do not require an impossible post-PASS active-only path.
|
||||
- Verify HTTP/CDP bind loopback and CDP interception fails non-loopback requests before dispatch; prove the test makes zero outbound connection.
|
||||
- Verify exactly three generated root implementation files while declared `brief/` and `assets/` fixture inputs remain accepted and checksum-bound.
|
||||
- Verify both images are loaded and visibly rendered at both viewports, and responsive/focus/heading/name/contrast clauses have deterministic evidence.
|
||||
- Verify automatic gates remain pass/fail, browser absence is blocked, no test is skipped, and all browser/server resources are reaped.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest
|
||||
python3 -m unittest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips; the browser smoke proves two viewport PNGs and deny-before-dispatch.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Browser capability preflight
|
||||
|
||||
Command: `for candidate in chromium chromium-browser google-chrome google-chrome-stable; do command -v "$candidate" && "$candidate" --version && break; done`
|
||||
|
||||
Expected: one absolute executable path/version. If absent, record `browser_unavailable` and do not mark ready for review.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Protocol, web, attempt, and integration suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
|
||||
Expected: exit 0 with no failures/errors/skips.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Mandatory real-browser smoke
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`
|
||||
|
||||
Expected: exit 0 with no skip; two viewport PNGs, zero outbound external connection, and all valid-page gates pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh complete suite output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
|
||||
Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,214 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/15+14_web_validation plan=1 tag=API milestone-task=web-validation -->
|
||||
|
||||
# Deterministic landing-page render and web gates
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The versioned landing-page prompt, copy, and two local SVG inputs are valid, but caller success does not prove that output builds, renders, or satisfies automatic gates. SDD S12 requires deterministic desktop/mobile screenshots and separate asset, console, responsive, and accessibility evidence. Self-review added runtime network denial, fixture-input-aware file rules, and the prompt's visible-image/layout/accessibility checks.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/plan_cloud_G08_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/code_review_cloud_G08_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor 14 through active-or-archived task evidence; block non-loopback browser requests before dispatch; distinguish the three generated root files from declared fixture inputs; verify visible images, bounded layout, focus/navigation, heading order, accessible names, and contrast.
|
||||
- Verification evidence: all three fixture manifests validated locally; no browser/provider was invoked and the prior pair has no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=web-validation`, SDD D11/S12/Evidence Map S12.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/workspace.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/lifecycle_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/plan_cloud_G08_0.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/code_review_cloud_G08_0.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`; lock released; `milestone-task=web-validation`.
|
||||
- D11 fixes vanilla root outputs `index.html`, `styles.css`, `script.js`, two visible local images, no external network/framework/build tool, and versioned fixture ownership.
|
||||
- S12/Evidence Map S12 require fixture checksum, required outputs, both images, desktop/mobile screenshots, and deterministic console/asset/responsive/accessibility gates.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Python 3.12, Node, and npx are available; no supported Chromium command is present on the current host. Unit tests remain standard-library-only and browser-free.
|
||||
- External verification target is the routed cloud worker at `/config/workspace/iop-s0`, preserving the branch/worktree. Before implementation it must resolve predecessor 14 as completed through the task protocol, and it must expose one installed Chromium-family binary with recorded path/version.
|
||||
- Browser binds loopback-only HTTP/CDP ports, uses a fresh profile outside the repository, blocks every non-loopback request before network dispatch, and reaps browser/server/process-group resources on every terminal path.
|
||||
- No package/browser download may be written into the repository. A missing browser is `browser_unavailable`, not a skip or pass.
|
||||
- Confidence: high for static/schema gates; real render closure depends on the cloud browser smoke.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No bounded server/CDP client or fail-closed external request interception exists.
|
||||
- No tests distinguish declared fixture inputs from exactly three generated root implementation files.
|
||||
- No runtime proof requires both images to be visible, primary content contained, controls non-overlapping, keyboard focus visible, logical headings, accessible names, and contrast.
|
||||
- No immutable web record or corruption/recovery validation exists.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `run_slots` callers are CLI state commands and attempt/integration tests.
|
||||
- `ExecutionAdapter.invoke` remains unchanged; validation runs after it returns and after predecessor measurement publication.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one invariant: one immutable `web-validation.json` binds static gates, browser observations, and screenshot digests to the same attempt/workspace/browser session.
|
||||
- Dependency `15+14_web_validation` requires predecessor index 14. It is currently pending; runtime must accept the matching active or dated archive `complete.log`, never only the active path.
|
||||
- Static and render stages stay together because eligibility is one fail-closed record. Child 16 consumes only a complete `passed` record.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude scoring, evaluator invocation, report rendering, and aesthetic scoring.
|
||||
- Do not add a framework, package manager, browser automation dependency, or build tool.
|
||||
- Automatic gates remain pass/fail and never contribute to the 100-point rubric.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build closure has a concrete browser capability gap; cloud Chromium resolves it. Scores `(2,2,1,1,2)` produce G08 and route `capability-gap`, `worker/cloud/G08`, `PLAN-cloud-G08.md`.
|
||||
- Review closures are true on the cloud runner; `official-review`, `review/cloud/G08`, `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; loop risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product` (5). `review_rework_count=0`, `evidence_integrity_failure=false`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve predecessor index 14 using the task protocol: `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/complete.log` if active, otherwise exactly one `agent-task/archive/*/*/m-agent-comparison-benchmark-pipeline/14_timing_usage/complete.log`. It must be PASS; current preparation state is pending.
|
||||
2. Consume predecessor 14's strict measurement loader without duplicating or weakening it.
|
||||
3. Implement static/render evidence, then integrate one web record before attempt terminal commit.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement contained loopback serving and a bounded standard-library Chromium CDP client with pre-dispatch non-loopback denial, browser identity, viewport screenshots, console/layout/accessibility observations, and cleanup tests.
|
||||
- [ ] [API-2] Implement fixture-aware generated-file checks, visible-image/no-network/no-framework/asset/console/responsive/accessibility gates, and immutable `web-validation.json` for passed/failed/blocked/not-run with corruption coverage.
|
||||
- [ ] [API-3] Integrate web validation after timing evidence, prove both images and desktop/mobile outputs in the three-caller fake path, and run the mandatory real Chromium smoke plus full suite.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Build a bounded loopback Chromium renderer
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/attempts.py:1490-1500` returns after invocation. There is no contained server/browser lifecycle, and merely recording an external request after dispatch would allow generated JavaScript to contact an unapproved host during validation.
|
||||
|
||||
**Solution**
|
||||
|
||||
After predecessor measurement publication, start one loopback server and one fresh Chromium profile per attempt. Add `browser_cdp.py` using standard-library HTTP, subprocess, sockets/WebSocket framing, and JSON. Serve only contained regular workspace files, reject traversal/symlinks, bind `127.0.0.1` on OS-assigned ports, and launch Chromium with `--remote-debugging-address=127.0.0.1` plus background-networking/extensions disabled.
|
||||
|
||||
Enable CDP `Fetch` interception before navigation. Continue only same-origin loopback requests; fail every other scheme/host before dispatch and record a redacted blocked-request fact. Also collect local failures, console/log/exception events, computed layout/style facts, accessibility tree, and deterministic PNG files. Bound every wait and kill/reap the process group, server, sockets, and profile on success/error/cancel/timeout.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp.py`: server, CDP/WebSocket client, request interception, observations, screenshots, cleanup.
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp_test.py`: protocol/interleaving/malformed/timeout/request-denial/process-cleanup tests and real-browser smoke.
|
||||
- [ ] `scripts/agent_benchmark/web_validation.py`: closed renderer observation inputs.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Fake CDP verifies masked frames, event correlation, deny-before-dispatch, and cleanup. `BrowserIntegrationTest.test_two_viewports_emit_png_and_block_external_requests` uses a page that attempts dynamic external fetch and proves zero outbound connection plus a recorded failed gate.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest`; expect exit 0 without a browser.
|
||||
|
||||
### [API-2] Define complete automatic gates and immutable evidence
|
||||
|
||||
**Problem**
|
||||
|
||||
Prepared workspaces contain declared `brief/` and `assets/` inputs. The prior wording “exactly three output files” could reject those inputs, while reference-only image checks could accept hidden images and the prompt's layout/accessibility requirements were under-specified.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add strict `web-validation-v1` with status `passed|failed|blocked|not_run`, attempt/cell/manifest/measurement digests, browser identity, viewports, screenshots, blocked/local requests, and ordered gates. Lifecycle failure is `not_run`, browser absence/startup is `blocked`, and executed gate failure is `failed`.
|
||||
|
||||
Require exactly the generated root implementation files `index.html`, `styles.css`, `script.js`; declared immutable fixture inputs under `brief/` and `assets/` remain permitted and checksum-bound. Reject unexpected generated root/code/build artifacts. Static parsing rejects external/data/protocol-relative URLs, module/package imports, frameworks, and undeclared local assets.
|
||||
|
||||
At both viewports require both supplied images to load, have meaningful alt text, computed visible styles, and non-zero rectangles. `responsive-layout` checks document overflow, primary landmark containment, clipped primary content, and pairwise overlap of visible navigation/CTA controls. `accessibility-minimum` checks language, one non-empty h1, logical heading progression, landmarks, image alt, accessible interactive names, keyboard reachability, visible focus indication, and computed text/background contrast thresholds. Each result keeps source/evidence/reason; gates never produce numeric quality points.
|
||||
|
||||
Reject extra fields, escapes, invalid digests, missing screenshots, collisions, symlink/FIFO, or cross-record mismatch without mutating prior bytes.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/web_validation.py`: parser, static/runtime gates, schema, publisher, strict loader.
|
||||
- [ ] `scripts/agent_benchmark/web_validation_test.py`: fixture-aware file table, visible images, network/framework, console, layout, focus/heading/name/contrast, corruption.
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: require coherent web evidence before terminal commit.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: status/cross-record/special-file/prior-byte cases.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Build a valid page and one-fault-at-a-time fixtures. Assert declared inputs are accepted, unexpected generated files fail, hidden/missing images fail, each prompt accessibility/layout clause is exercised, and no status becomes a score.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test`; expect exit 0 with no skips.
|
||||
|
||||
### [API-3] Integrate and prove the S12 production path
|
||||
|
||||
**Problem**
|
||||
|
||||
The three-caller integration asserts lifecycle evidence only, so a successful attempt can omit page outputs or fail at render time and still be retained without web evidence.
|
||||
|
||||
**Solution**
|
||||
|
||||
Update fake callers to write the required responsive page with both fixture images. Inject renderer observations in network-free orchestration tests; keep a dedicated real Chromium test for serve/render/network denial/screenshots. Preserve execution terminal state independently from gate status; scoring eligibility later requires lifecycle success plus web `passed`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/attempts_test.py`: ordering and retained execution state.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: page outputs, two viewports/images, no sentinels.
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp_test.py`: mandatory actual Chromium smoke.
|
||||
- [ ] `scripts/agent_benchmark/web_validation_test.py`: D11/S12 gate coverage.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use injected observations for deterministic fake callers and one unskipped real-browser test. A missing supported browser records `browser_unavailable` and blocks review readiness.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`; expect two non-empty PNGs, blocked dynamic external request with zero outbound connection, and a separate valid page with all gates passing.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/browser_cdp.py` | API-1 |
|
||||
| `scripts/agent_benchmark/browser_cdp_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/web_validation.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/web_validation_test.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/attempts.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/CODE_REVIEW-cloud-G08.md` | API-1, API-2, API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `for candidate in chromium chromium-browser google-chrome google-chrome-stable; do command -v "$candidate" && "$candidate" --version && break; done`
|
||||
- Expected: one absolute executable path/version on the cloud runner; no output means `browser_unavailable` and stop.
|
||||
2. `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
- Expected: exit 0 with no failures/errors/skips.
|
||||
3. `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`
|
||||
- Expected: exit 0 with no skip; two viewport PNGs, deny-before-dispatch proof, and all valid-page gates pass.
|
||||
4. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with fresh complete suite output.
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/15+14_web_validation plan=0 tag=API milestone-task=web-validation -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/15+14_web_validation, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-cloud-G08.md` → `plan_cloud_G08_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/15+14_web_validation/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=web-validation` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Bounded loopback Chromium renderer | [ ] |
|
||||
| API-2 — Static gates and immutable web record | [ ] |
|
||||
| API-3 — S12 production-path integration | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement bounded loopback serving and a standard-library Chromium DevTools client that records browser identity, viewport screenshots, request failures, console exceptions, layout facts, and accessibility facts with cleanup tests.
|
||||
- [ ] [API-2] Implement closed static/automatic gates and immutable `web-validation.json` publication for passed, failed, blocked, and not-run attempts, with strict loader/corruption coverage.
|
||||
- [ ] [API-3] Integrate web validation after timing evidence, prove both local images and desktop/mobile outputs in the three-caller path, and run a real Chromium smoke plus the full benchmark suite.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_0.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G08.md` to `plan_cloud_G08_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/15+14_web_validation/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=web-validation` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require predecessor 14 PASS evidence and reuse its strict measurement loader without weakening it.
|
||||
- Verify the server accepts contained regular files only and browser/server process groups are reaped on every terminal path.
|
||||
- Verify automatic gates stay pass/fail and never enter the 100-point rubric.
|
||||
- Verify the real Chromium smoke is not skipped and records two PNGs, browser version, local requests, console/layout/accessibility facts.
|
||||
- Verify browser absence is `blocked` evidence, not a fabricated pass.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest
|
||||
python3 -m unittest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips; the integration command emits two viewport PNGs and all required gates pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Browser capability preflight
|
||||
|
||||
Command: `for candidate in chromium chromium-browser google-chrome google-chrome-stable; do command -v "$candidate" && "$candidate" --version && break; done`
|
||||
|
||||
Expected: one absolute executable path and version. If absent, record `browser_unavailable` and do not mark ready for review.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Protocol, web, attempt, and integration suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
|
||||
Expected: exit 0 with no failures/errors/skips.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Mandatory real-browser smoke
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`
|
||||
|
||||
Expected: exit 0 with no skip; both viewport PNGs and all required automatic gates pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0; cached output is not acceptable.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
|
||||
Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,279 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/15+14_web_validation plan=0 tag=API milestone-task=web-validation -->
|
||||
|
||||
# Deterministic landing-page render and web gates
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The benchmark now has a versioned landing-page prompt, copy, and two local SVG inputs, but successful caller execution is not yet evidence that the page builds, renders, or meets the automatic gates. SDD S12 requires deterministic desktop/mobile screenshots and separate asset, console, responsive, and accessibility evidence without external page dependencies.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/workspace.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/lifecycle_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`, approved and implementation lock released.
|
||||
- First-line milestone task: `web-validation`.
|
||||
- Decision D11 fixes vanilla `index.html`, `styles.css`, `script.js`, two local images, no network/framework/build tooling, and versioned fixture ownership.
|
||||
- Acceptance Scenario S12 requires build/serve/render at manifest desktop/mobile viewports, screenshots, asset/console evidence, and responsive/accessibility gates.
|
||||
- Evidence Map S12 requires fixture checksum, required outputs, two images, viewport screenshots, and deterministic gate results. These rows define API-1 through API-3 and final browser smoke.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Repository-native inputs are the approved SDD, manifest viewports/assets, prepared workspace, lifecycle result, attempt state, and the local test rule.
|
||||
- Current preflight: Python `/bin/python3` 3.12.3, Node v22.23.1, and npx 10.9.8 are present; `chromium`, `chromium-browser`, `google-chrome`, `google-chrome-stable`, `firefox`, and `playwright` commands are absent. There is no package manifest or lockfile.
|
||||
- External Verification Preflight: runner is the routed cloud worker; repo root/workdir `/config/workspace/iop-s0`; branch `feature/agent-comparison-benchmark-pipeline`; analyzed HEAD `69e986a9b32a501085b567bc05c0799d514b1842` with direct-small fixture changes dirty; source sync must retain those changes and completed predecessor 14. Browser path is auto-discovered from the four Chromium command names, browser version is recorded, loopback port is OS-assigned, no external host is contacted, and Linux/POSIX process-group cleanup is required. Resolution is a cloud runner with one installed Chromium-family binary; if none is found, record `browser_unavailable` and do not claim the integration smoke passed.
|
||||
- Browser/network setup must not download into the repository. Unit tests remain standard-library-only; the actual smoke is mandatory for review.
|
||||
- Confidence: high for static gates and evidence schema; actual render verification is intentionally cloud-gated by the observed local browser capability gap.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No code serves a prepared workspace on loopback or drives a browser with bounded cleanup.
|
||||
- No test rejects external URLs, frameworks/build artifacts, missing required files, or unused fixture images.
|
||||
- No renderer records asset failures, console exceptions, viewport overflow, accessibility names/landmarks, or PNG screenshots.
|
||||
- No append-only web record is bound to lifecycle/measurement evidence, and recovery does not reject tampering.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `run_slots` is called by `_cmd_state` in `scripts/agent_comparison_benchmark.py` and by `attempts_test.py`/`connectivity_integration_test.py`.
|
||||
- `ExecutionAdapter.invoke` remains unchanged; web validation runs after it returns.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one stable invariant: a terminal attempt has one immutable `web-validation.json` whose gate results and screenshot digests refer to the same served workspace and browser session.
|
||||
- Static validation and Chromium rendering remain one plan because eligibility requires their single fail-closed record; they are separated into modules and independently unit-tested.
|
||||
- Predecessor 14 is encoded by `15+14_web_validation`. `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/complete.log` is missing because the active PLAN/CODE_REVIEW pair is pending; implementation must wait for it.
|
||||
- PASS requires unit/corruption tests and one real Chromium desktop/mobile smoke. Child 16 consumes only records whose required gates pass.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude manual quality scoring, evaluator invocation, and report rendering.
|
||||
- Do not install a JS framework, browser automation package, or build tool; use Python standard library plus an externally installed Chromium executable.
|
||||
- Do not score visual aesthetics in automatic gates, and do not fold these pass/fail gates into the later 100-point rubric.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build closures: scope/context/evidence/ownership/decision are true; local verification closure is false only because no browser binary is installed. Capability gap observed by command preflight; evidence is the absent command set; cloud resolution is a Chromium-equipped runner executing the mandatory smoke. Scores `(2,2,1,1,2)` produce G08; base and route `capability-gap`, `worker/cloud/G08`, `PLAN-cloud-G08.md`.
|
||||
- Review closures are true on the cloud runner. Scores `(2,2,1,1,2)` produce `official-review`, `review/cloud/G08`, `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product` (5). `review_rework_count=0`, `evidence_integrity_failure=false`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/complete.log` must exist and PASS before implementation starts.
|
||||
2. Consume predecessor 14's strict measurement loader and sidecar; do not duplicate or weaken it.
|
||||
3. Implement browser/static validation, then integrate one web record after lifecycle and measurement publication.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement bounded loopback serving and a standard-library Chromium DevTools client that records browser identity, viewport screenshots, request failures, console exceptions, layout facts, and accessibility facts with cleanup tests.
|
||||
- [ ] [API-2] Implement closed static/automatic gates and immutable `web-validation.json` publication for passed, failed, blocked, and not-run attempts, with strict loader/corruption coverage.
|
||||
- [ ] [API-3] Integrate web validation after timing evidence, prove both local images and desktop/mobile outputs in the three-caller path, and run a real Chromium smoke plus the full benchmark suite.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Build a bounded loopback Chromium renderer
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/attempts.py:1490-1500` returns immediately after caller invocation. There is no server, browser lifecycle, console/network observation, layout query, accessibility tree, or screenshot capture.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:1490-1499`):
|
||||
|
||||
```python
|
||||
lease = store.acquire_control_lease(current)
|
||||
return adapters[cell.caller].invoke(
|
||||
cell,
|
||||
prepared,
|
||||
current,
|
||||
lease.control_dir,
|
||||
manifest.fixture.prompt_content,
|
||||
manifest.timeout,
|
||||
on_started,
|
||||
)
|
||||
```
|
||||
|
||||
After the predecessor's measurement publication, call a renderer that owns one loopback server and one fresh browser profile per attempt. Add `browser_cdp.py` using only:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
import base64
|
||||
import hashlib
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from urllib.request import Request, urlopen
|
||||
```
|
||||
|
||||
Serve only regular files contained by `workspace`, reject traversal/symlinks, bind `127.0.0.1` on an OS-assigned port, and record every request. Start the discovered browser in a temporary profile outside the repo with background networking/extensions disabled and a remote-debugging port. Implement masked WebSocket frames and request-id/event correlation for `Page`, `Runtime`, `Network`, `Log`, `Emulation`, and `Accessibility`; bound every wait and kill/reap the process group on success, error, cancel, or timeout.
|
||||
|
||||
For each manifest viewport, set exact device metrics, navigate to `/index.html`, wait for load plus bounded network quiet, collect failed/local requests and console/exception events, evaluate `scrollWidth/clientWidth`, collect the accessibility tree, and write decoded PNG bytes to deterministic `screenshots/<viewport-id>.png` without replacement. Record browser executable/version and screenshot SHA-256, never raw page content or environment.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp.py`: add server, CDP/WebSocket client, renderer, timeouts, and cleanup.
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp_test.py`: fake CDP framing/order/error tests and real-browser smoke class.
|
||||
- [ ] `scripts/agent_benchmark/web_validation.py`: define renderer observation inputs without exposing subprocess internals.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write fake socket/server tests for fragmented frames, masked writes, interleaved events, malformed JSON, disconnect, timeout, and process cleanup. Add `BrowserIntegrationTest.test_two_viewports_emit_png_and_no_external_requests`; it must fail with `browser_unavailable` rather than skip when no supported binary exists.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest`; expect all protocol/cleanup tests to pass without a browser.
|
||||
|
||||
### [API-2] Define static gates and the immutable web record
|
||||
|
||||
**Problem**
|
||||
|
||||
The prepared workspace contract copies declared assets but does not require generated `index.html`, `styles.css`, or `script.js`, verify that both images are used, or separate automatic gate failures from lifecycle success.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:1326-1330`):
|
||||
|
||||
```python
|
||||
result = invoke(attempt, lambda locator, digest: self.record_locator(attempt, locator, digest))
|
||||
terminal = self.validate_invocation_terminal(attempt, result)
|
||||
published = self.publish_terminal(
|
||||
attempt, self._state_for_reason(terminal["terminal_reason"]), result=terminal
|
||||
)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
result = invoke(
|
||||
attempt,
|
||||
lambda locator, digest: self.record_locator(attempt, locator, digest),
|
||||
)
|
||||
terminal = self.validate_invocation_terminal(attempt, result)
|
||||
web = load_and_validate_web_record(root, terminal)
|
||||
published = self.publish_terminal(
|
||||
attempt, self._state_for_reason(terminal["terminal_reason"]), result=terminal
|
||||
)
|
||||
```
|
||||
|
||||
Add a closed `web-validation-v1` record with attempt/cell/manifest/measurement digests, browser identity, viewports, screenshots, requests, and ordered gates. Status is exactly `passed`, `failed`, `blocked`, or `not_run`; lifecycle failure is `not_run`, browser absence/startup is `blocked`, and an executed failing gate is `failed`. Each gate has id, status, source, evidence, and reason; required gates are `required-files`, `two-local-images`, `no-external-network`, `no-framework-build`, `asset-load`, `console-errors`, `responsive-overflow`, and `accessibility-minimum`.
|
||||
|
||||
Parse HTML with `html.parser.HTMLParser`, resolve URLs with `urllib.parse`, and inspect CSS/JS as UTF-8 text. Require exactly the three output files, both manifest-provided images referenced through contained relative paths, no `http:`, `https:`, `//`, data URL, package CDN, framework marker, module import, or build artifact. Accessibility minimum requires document language, one non-empty h1, landmark structure, non-empty image alt, and accessible names for interactive controls from static DOM plus the browser accessibility tree. Reject extra fields, path escapes, invalid digests, missing screenshots, record collision, symlink/FIFO, and cross-record identity mismatch without mutating terminal bytes.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/web_validation.py`: add static parser, gate evaluation, schema, no-clobber publisher, and strict loader.
|
||||
- [ ] `scripts/agent_benchmark/web_validation_test.py`: normal/boundary gate table and durable corruption tests.
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: require coherent web evidence before terminal publication.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: cover all statuses, cross-record tampering, special files, and byte preservation.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Build valid and one-fault-at-a-time pages in temporary workspaces. Assert automatic status never becomes a numeric quality score and that every failed/blocked/not-run reason remains explicit.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test`; expect all tests to pass with no skips.
|
||||
|
||||
### [API-3] Integrate and prove the S12 production path
|
||||
|
||||
**Problem**
|
||||
|
||||
The current three-caller integration asserts lifecycle evidence only, so a successful attempt can omit all page outputs or fail at render time and still be retained as successful.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/connectivity_integration_test.py:613-615`):
|
||||
|
||||
```python
|
||||
for attempt_root in attempt_roots:
|
||||
self._assert_one_published_spec_digest(attempt_root)
|
||||
self._assert_sentinels_absent(published, sentinels)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
for attempt_root in attempt_roots:
|
||||
self._assert_one_published_spec_digest(attempt_root)
|
||||
self._assert_web_record_bound(attempt_root, manifest.viewports)
|
||||
self._assert_sentinels_absent(published, sentinels)
|
||||
```
|
||||
|
||||
Update fake production callers to write the required responsive page using both fixture images. Inject a fake renderer for network-free orchestration tests, while the dedicated browser integration test exercises the real server/browser path. Preserve execution terminal state independently from gate status; later scoring decides eligibility from lifecycle success plus `web.status == "passed"`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: invoke validation after predecessor measurement evidence and before terminal commit.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: prove stage ordering and retained execution state.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: create page outputs and assert two viewport records/screenshots without sentinels.
|
||||
- [ ] `scripts/agent_benchmark/browser_cdp_test.py`: run mandatory actual Chromium render smoke.
|
||||
- [ ] `scripts/agent_benchmark/web_validation_test.py`: cover fixture-specific D11 gates.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use injected renderer observations for the credential-free three-caller test and one real Chromium test for actual serve/render/screenshots. No browser test may be silently skipped.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`; expect two non-empty PNGs, zero external requests, zero console exceptions, no horizontal overflow, and required accessibility facts.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/browser_cdp.py` | API-1 |
|
||||
| `scripts/agent_benchmark/browser_cdp_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/web_validation.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/web_validation_test.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/attempts.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/CODE_REVIEW-cloud-G08.md` | API-1, API-2, API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `for candidate in chromium chromium-browser google-chrome google-chrome-stable; do command -v "$candidate" && "$candidate" --version && break; done`
|
||||
- Expected: one absolute executable path and version on the cloud runner. If none is printed, record `browser_unavailable` and stop without marking ready for review.
|
||||
2. `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserProtocolTest scripts.agent_benchmark.web_validation_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
- Expected: exit 0 with no failures/errors/skips.
|
||||
3. `python3 -m unittest scripts.agent_benchmark.browser_cdp_test.BrowserIntegrationTest`
|
||||
- Expected: exit 0 with no skip; both viewport PNGs and all required automatic gates pass.
|
||||
4. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with the complete benchmark suite passing. Cached output is not acceptable.
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,190 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/16+15_blind_score plan=1 tag=API milestone-task=blind-score -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/16+15_blind_score, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/plan_cloud_G09_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/code_review_cloud_G09_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor 15 through active-or-archived PASS evidence; allow the existing closed Codex route contract (`direct|execution_preset`) in manifest data; run the evaluator only from an opaque run-owned blind workspace whose absolute path omits cell/caller/model/route/effort identity.
|
||||
- Verification evidence: current fixture manifests validated locally; the prior pair contains no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=blind-score`, SDD D12-D14/S13/Evidence Map S13.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/16+15_blind_score/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Version-2 evaluator/rubric contract | [ ] |
|
||||
| API-2 — Eligibility and run-owned blinded inputs | [ ] |
|
||||
| API-3 — Fresh append-only scoring attempts | [ ] |
|
||||
| API-4 — Public score CLI/skill operation | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Bump the closed manifest to version 2 with required Codex evaluator binding over the existing closed route kinds and exact `landing-quality-v1` rubric, updating schema/examples/constructors and boundary tests.
|
||||
- [ ] [API-2] Implement lifecycle/web eligibility and identity-separated run-owned blind workspaces so ineligible attempts are immutable `unscored` and evaluator-visible paths/bytes contain no harness identity.
|
||||
- [ ] [API-3] Implement strict 100-point worksheets and append-only score attempts with one fresh session, provenance, `scoring_failed`, explicit new-id retry, and corruption/leak tests.
|
||||
- [ ] [API-4] Add `score --manifest --run-id [--retry-scoring-failed]`, reuse live Codex preflight without fallback, update skill/routing contracts, and run focused/full validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G09_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/16+15_blind_score/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require predecessor 15 PASS from the exact active or dated archive `complete.log`; transitive 14 is satisfied only through 15.
|
||||
- Verify manifest supports the existing closed Codex `direct|execution_preset` route contract without a direct-only schema restriction or evaluator default.
|
||||
- Verify every evaluator-visible cwd/argv/env/path/byte lives under an opaque run-owned blind path and contains no caller/cell/route/model/effort/original-path/mapping sentinel.
|
||||
- Verify automatic gates never add points and `unscored`/`scoring_failed` never become zero.
|
||||
- Verify one fresh evaluator session per score id, no implicit retry/fallback, and prior bytes survive explicit new-id retry.
|
||||
- Verify CLI/skill output contains closed counts only and durable evidence contains no runtime secret.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips, identity leak, or durable secret.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused manifest/scoring/skill suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test`
|
||||
|
||||
Expected: exit 0, no failures/errors/skips, and no harness/secret identity in evaluator-visible or durable evidence.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Score CLI help
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py score --help`
|
||||
|
||||
Expected: exit 0 and exactly `--manifest`, `--run-id`, `--retry-scoring-failed` are documented.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Version-2 example manifest validation
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json
|
||||
```
|
||||
|
||||
Expected: each exits 0 and prints `ok: manifest is valid` for version 2.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,256 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/16+15_blind_score plan=1 tag=API milestone-task=blind-score -->
|
||||
|
||||
# Blind Codex evaluator and append-only scoring attempts
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
Automatic gates establish score eligibility but do not replace the approved 100-point visual rubric. SDD D12-D14 require identity removal, manifest-bound fresh Codex evaluation through IOP, append-only retry provenance, explicit `unscored`/`scoring_failed`, and no fallback or synthetic score. Self-review corrected evaluator cwd identity leakage and preserved the SDD's data-driven route binding instead of hard-coding direct-only evaluation.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/plan_cloud_G09_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/code_review_cloud_G09_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor 15 through active-or-archived PASS evidence; allow the existing closed Codex route contract (`direct|execution_preset`) in manifest data; run the evaluator only from an opaque run-owned blind workspace whose absolute path omits cell/caller/model/route/effort identity.
|
||||
- Verification evidence: current fixture manifests validated locally; the prior pair contains no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=blind-score`, SDD D12-D14/S13/Evidence Map S13.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/workspace.py`
|
||||
- `scripts/agent_benchmark/workspace_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `agent-contract/index.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/plan_cloud_G09_0.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/code_review_cloud_G09_0.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved SDD with released lock; `milestone-task=blind-score`.
|
||||
- D12 fixes category weights 25/25/20/15/15 and excludes automatic gate points.
|
||||
- D13 permits scoring only after lifecycle success and all required web gates; others are `unscored`, never zero, and ties remain ties.
|
||||
- D14 requires a separate manifest evaluator IOP route/model/effort binding, one fresh Codex session per scoring attempt, `scoring_failed` without fallback, and explicit retry under a new id.
|
||||
- S13/Evidence Map S13 require separated mapping, strict worksheet, evaluator provenance/freshness, unscored/failure, and retry evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Current manifest version 1 lacks evaluator data; Codex already supports closed `direct` and `execution_preset` routes through the existing capability/preflight path.
|
||||
- No new outer API field is required. The evaluator uses the current Responses surface and Edge config observations.
|
||||
- Tests use fake evaluators and production-shaped Codex JSONL with no credentials/provider call. An actual `score` command remains guarded by live preflight.
|
||||
- Every evaluator-visible argv/env/cwd/path/byte must be projected from `<run-root>/blind/<opaque-blind-id>/`, not `cells/<cell-id>/...`. Durable identity mapping and score records remain outside the evaluator-visible subtree.
|
||||
- Confidence: high because eligibility/loaders, blind alias ownership, and scoring ids are explicit and fail closed.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Manifest/schema do not bind evaluator caller/route/model/effort or rubric revision.
|
||||
- No eligibility projection, blind run-root workspace, separated mapping, or leak scan exists.
|
||||
- No append-only scoring allocator, retry policy, strict worksheet, or fresh-session proof exists.
|
||||
- CLI/skill have no score operation and report remains unavailable.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `Manifest(...)` constructors in `manifest.py`, `workspace_test.py`, and fixture builders must receive evaluator after the version bump.
|
||||
- Exact CLI command/option sets in `skill_contract_test.py` must add `score` consistently.
|
||||
- `build_live_adapter_registry` consumers include CLI and integration tests; evaluator setup reuses current config/catalog proof.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- One indivisible invariant binds eligibility, blinded inputs, exactly one invocation per score id, strict worksheet, and append-only terminal status.
|
||||
- Dependency `16+15_blind_score` requires predecessor 15. It is pending; runtime must resolve its exact active or dated archive PASS `complete.log`. Transitive index 14 is satisfied through 15.
|
||||
- Report aggregation remains child 17.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude Markdown reports/rankings and any D12 weight change.
|
||||
- Do not leak harness identity, persist credentials, retry implicitly, substitute evaluator route/model/effort, or turn failure/ineligibility into zero.
|
||||
- Examples may choose a direct evaluator, but schema/runtime must preserve the approved manifest-selected closed Codex route kind.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build/review closures are true. Scores `(2,2,2,1,2)` produce G09; build `grade-boundary`, `worker/cloud/G09`, `PLAN-cloud-G09.md`; review `official-review`, `review/cloud/G09`, `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; loop risks `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product` (5). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve predecessor 15 through `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/complete.log` if active, otherwise exactly one dated archive match under `agent-task/archive/*/*/m-agent-comparison-benchmark-pipeline/15+14_web_validation/complete.log`. It must be PASS; current preparation state is pending.
|
||||
2. Use predecessor 15's strict web status/gate loader as the sole eligibility source.
|
||||
3. Land manifest/rubric contracts, then scoring storage/blinding, evaluator execution, and CLI/skill exposure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Bump the closed manifest to version 2 with required Codex evaluator binding over the existing closed route kinds and exact `landing-quality-v1` rubric, updating schema/examples/constructors and boundary tests.
|
||||
- [ ] [API-2] Implement lifecycle/web eligibility and identity-separated run-owned blind workspaces so ineligible attempts are immutable `unscored` and evaluator-visible paths/bytes contain no harness identity.
|
||||
- [ ] [API-3] Implement strict 100-point worksheets and append-only score attempts with one fresh session, provenance, `scoring_failed`, explicit new-id retry, and corruption/leak tests.
|
||||
- [ ] [API-4] Add `score --manifest --run-id [--retry-scoring-failed]`, reuse live Codex preflight without fallback, update skill/routing contracts, and run focused/full validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Version evaluator and rubric contracts
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/manifest.py:105-121` has `rubric_version` but no evaluator, and `load_manifest` rejects an evaluator field. Adding a required field without a version bump would redefine existing snapshots. The approved SDD requires a route binding but does not restrict it to direct-only.
|
||||
|
||||
**Solution**
|
||||
|
||||
Set `PIPELINE_VERSION = "2"`; add `Evaluator(caller, iop)` and require caller `codex`, an existing `codex_capability()` route kind (`direct|execution_preset`), requested model/effort, and exact expected bindings. Reuse the current IOP-cell validator rather than creating a scoring-only route dialect. Include evaluator in canonical JSON/digest. Checked-in examples use one explicit direct evaluator, without narrowing the schema.
|
||||
|
||||
Require rubric `landing-quality-v1`. Add `rubric.py` with exact categories/weights and strict worksheet validation: integer category scores within maxima, non-empty evidence, no extras/missing fields, and exact total.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/manifest_test.py`: version-2 evaluator/digest contract and boundary tests.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`, `scripts/agent_benchmark/rubric_test.py`: exact rubric/worksheet.
|
||||
- [ ] `scripts/agent_benchmark/workspace_test.py`, `scripts/agent_benchmark/attempts_test.py`, `scripts/agent_benchmark/connectivity_integration_test.py`: update constructors/builders.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` and three example manifests: version 2, evaluator, rubric.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Test caller, both supported route kinds, invalid route, model/effort/binding/revision/version/digest mutations. Do not provide evaluator defaults.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`; expect exit 0.
|
||||
|
||||
### [API-2] Materialize eligibility and truly blinded evaluator inputs
|
||||
|
||||
**Problem**
|
||||
|
||||
Execution attempts live at `cells/<cell-id>/repetition-*/attempt-*`. The prior proposal put evaluator cwd beneath that path, exposing cell identity even if copied file contents were clean.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `scoring.py`. Eligibility is exactly execution success plus web `passed` plus every required gate `passed`. Ineligible attempts get one no-clobber `scoring/unscored.json` with reason codes/digests and no score fields.
|
||||
|
||||
For eligible attempts, derive an opaque blind id from manifest digest, attempt ordinal, and harness nonce. Publish the mapping outside the evaluator-visible subtree. Materialize only `index.html`, `styles.css`, `script.js`, two local images, and screenshots under `<run-root>/blind/<opaque-blind-id>/input/`, with fresh session/output siblings. The evaluator cwd, argv, prompt, environment, tool-visible paths, and retained blind bytes must omit caller, cell, route, model, effort, original attempt path, and mapping. Refuse symlinks/non-regular files/escapes and verify digests before invocation.
|
||||
|
||||
Durable score records under the original attempt may refer only to blind id/digests and the contained run-relative blind path; they are never mounted or copied into the evaluator-visible subtree.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: eligibility, unscored record, opaque blind tree, mapping, contained copy, strict loaders.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: eligibility table, path/symlink, argv/env/cwd/byte leak scans, no-zero/no-clobber, public RunStore enumeration.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use identity sentinels in every original path/field and assert none occur in evaluator-visible paths, bytes, invocation spec, or environment. Verify each failed/blocked/not-run gate yields unscored without evaluator allocation.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.attempts_test`; expect exit 0.
|
||||
|
||||
### [API-3] Run one fresh evaluator and seal the worksheet
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/codex_iop.py:326-355` builds isolated normal attempts but has no scoring identity, evaluator preflight, worksheet, or retry ledger.
|
||||
|
||||
**Solution**
|
||||
|
||||
Extract additive Codex construction in `live_iop.py` so normal attempts remain unchanged and a `ScoringAdapter` reuses runtime secret handling, catalog/config observation, capability validation, and effective binding proof. Allocate `score-000001`, etc. under the execution attempt while holding the run writer; bind each id to one fresh blind workspace/session and immutable preflight/lifecycle/result.
|
||||
|
||||
The prompt exposes only relative blind input paths and five rubric categories, requiring one `worksheet.json`. Validate it after successful lifecycle and copy its canonical projection into durable score evidence. Missing/malformed worksheet, lifecycle failure, binding mismatch, blocker, or extra field produces `scoring_failed`. Retry only with explicit flag and a new id; prior success prevents another score.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`, `scripts/agent_benchmark/scoring_test.py`: allocation, invocation, terminal/retry/corruption/leak rules.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`, `scripts/agent_benchmark/rubric_test.py`: strict worksheet projection.
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: additive evaluator builder with existing live ownership.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: production-shaped evaluator without durable secret/identity.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Test exactly-once, fresh session, every failure, explicit retry, prior-byte preservation, route binding, and identity/secret scan.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`; expect exit 0 without live credentials.
|
||||
|
||||
### [API-4] Expose score through deterministic CLI and skill
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_comparison_benchmark.py:65-88` lacks `score`; the benchmark skill has no scoring operation.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `score --manifest --run-id [--retry-scoring-failed]`. Validate/open immutable state, run evaluator preflight and scoring under the run writer, and print closed counts for scored/unscored/scoring_failed/blocked. Failures return 69 with run id; no provider is called outside the deterministic CLI. Update project skill, project routing, and exact semantic mutation tests while keeping report unavailable for child 17.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_comparison_benchmark.py`: parser/handler/output.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: score input/procedure/output/safety.
|
||||
- [ ] `agent-ops/rules/project/rules.md`: score routing.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`, `scripts/agent_benchmark/scoring_test.py`: exact options/output/error ordering/mutations.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Cover missing state, no eligible attempt, scored/unscored/failed/blocked, retry flag, provider boundary, and read/append ownership.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.skill_contract_test`; expect exit 0.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/workspace_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/scoring.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3, API-4 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_comparison_benchmark.py` | API-4 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-4 |
|
||||
| `agent-ops/rules/project/rules.md` | API-4 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-4 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` | API-1 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/CODE_REVIEW-cloud-G09.md` | API-1, API-2, API-3, API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test`
|
||||
- Expected: exit 0, no failures/errors/skips, and no harness/secret identity in evaluator-visible evidence.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with fresh complete suite output.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py score --help`
|
||||
- Expected: exit 0 with exactly `--manifest`, `--run-id`, `--retry-scoring-failed`.
|
||||
4. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- Expected for 4-6: exit 0 and `ok: manifest is valid` for version 2.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/16+15_blind_score plan=0 tag=API milestone-task=blind-score -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/16+15_blind_score, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/16+15_blind_score/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=blind-score` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Version-2 evaluator/rubric contract | [ ] |
|
||||
| API-2 — Eligibility and blinded inputs | [ ] |
|
||||
| API-3 — Fresh append-only scoring attempts | [ ] |
|
||||
| API-4 — Public score CLI/skill operation | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Bump the closed manifest to pipeline version 2 with a required direct Codex evaluator binding and exact `landing-quality-v1` rubric, updating schema/examples/all constructor fixtures and normal/boundary tests.
|
||||
- [ ] [API-2] Implement eligibility and identity-separated anonymization records so only lifecycle-success/web-pass attempts enter scoring and every other attempt is immutable `unscored` with exact reasons.
|
||||
- [ ] [API-3] Implement strict 100-point worksheet validation plus append-only `score-000001` attempts, fresh-session invocation, provenance, `scoring_failed`, and explicit new-id retry with corruption/leak tests.
|
||||
- [ ] [API-4] Add the public `score --manifest --run-id [--retry-scoring-failed]` operation, reuse live Codex preflight without fallback, update skill/routing contracts, and run focused/full validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/16+15_blind_score/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=blind-score` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require predecessor 15 PASS and use its strict web eligibility status/gate ids only.
|
||||
- Verify pipeline version 2 is explicit and no default evaluator hides missing manifest data.
|
||||
- Verify evaluator-visible paths/bytes contain no caller, cell, route, model, effort, original path, or mapping.
|
||||
- Verify automatic gates never contribute points and `unscored`/`scoring_failed` never become zero.
|
||||
- Verify exactly one fresh evaluator session per score id, no implicit retry/fallback, and prior failed bytes survive a new-id retry.
|
||||
- Verify CLI/skill output contains closed counts only and durable evidence contains no runtime secret.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: each exits 0 with no failures/errors/skips, identity leak, or durable secret.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused manifest/scoring/skill suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test`
|
||||
|
||||
Expected: exit 0, no failures/errors/skips, and no caller/secret identity in evaluator-visible or durable evidence.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0; cached output is not acceptable.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Score CLI help
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py score --help`
|
||||
|
||||
Expected: exit 0 and exactly `--manifest`, `--run-id`, and `--retry-scoring-failed` are documented.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Version-2 example manifest validation
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json
|
||||
```
|
||||
|
||||
Expected: each exits 0 and prints `ok: manifest is valid` for pipeline version 2.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,373 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/16+15_blind_score plan=0 tag=API milestone-task=blind-score -->
|
||||
|
||||
# Blind Codex evaluator and append-only scoring attempts
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
Automatic gates can establish eligibility but cannot replace the approved 100-point visual-quality rubric. SDD D12-D14 require identity removal, a manifest-bound fresh Codex evaluator, append-only retry provenance, explicit `unscored`/`scoring_failed`, and no fallback model or synthetic score.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/workspace.py`
|
||||
- `scripts/agent_benchmark/workspace_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `agent-contract/index.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`, approved and implementation lock released.
|
||||
- First-line milestone task: `blind-score`.
|
||||
- D12 fixes weights: task fidelity 25, visual hierarchy 25, responsive composition 20, typography/readability 15, polish/consistency 15. Automatic gates are not added to the score.
|
||||
- D13 permits scoring only when lifecycle success and every required web gate pass; all other attempts are `unscored`, not zero, and ties remain ties.
|
||||
- D14 requires a separate manifest evaluator route/model/effort, one fresh Codex session per scoring attempt, `scoring_failed` on invalid/failure, no fallback, and explicit retry with a new id.
|
||||
- Acceptance Scenario/Evidence Map S13 require separated anonymization mapping, strict worksheet, evaluator provenance, freshness, unscored/failure, and retry tests. These requirements directly define API-1 through API-4.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. The current manifest is a closed version-1 schema with `rubric_version` but no evaluator. The Codex adapter already provides isolated Responses invocation and structural redaction; live IOP preflight owns route/catalog/config validation.
|
||||
- Contract context: the evaluator continues through the existing OpenAI-compatible Responses surface and existing Edge config observation; no new outer wire field is introduced.
|
||||
- Verification uses credential-free fake evaluators and production-shaped Codex JSONL. No secret discovery or live provider call is required for tests; an actual `score` command remains guarded by the same live preflight.
|
||||
- Preconditions: predecessor web evidence has a strict loader and immutable status; all scoring writes stay under the selected run/attempt root; Python `unittest` output is fresh and uncached.
|
||||
- Gap: exact current Codex terminal text shape may vary, so the evaluator contract writes `worksheet.json` inside its isolated workspace and validates that file instead of parsing free-form assistant prose.
|
||||
- Confidence: high; identity mapping and score attempts have explicit filesystem ownership, while external readiness remains fail-closed through live preflight.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Manifest/schema tests do not bind evaluator caller/route/model/effort or a supported rubric revision.
|
||||
- No eligibility projection distinguishes `unscored` from scored zero.
|
||||
- No anonymized workspace or separated mapping exists.
|
||||
- No append-only scoring attempt allocator, retry policy, worksheet validator, or fresh-session proof exists.
|
||||
- CLI/skill expose no scoring operation and still declare report output unavailable.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `Manifest(...)` constructors: `manifest.py:684`, `manifest.py:701`, `workspace_test.py:433`, and `workspace_test.py:474`; all must receive `evaluator` after the version-2 schema change.
|
||||
- Manifest fixture builders containing `rubric_version`: `manifest_test.py:70`, `attempts_test.py:73`, `connectivity_integration_test.py:210`, and `workspace_test.py:135`.
|
||||
- CLI command sets/options are asserted throughout `skill_contract_test.py`; add `score` to every exact set/loop, not just help text.
|
||||
- `build_live_adapter_registry` consumers include the public CLI and connectivity integration tests. Additive evaluator construction must reuse its config/catalog proof.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns the indivisible scoring invariant: eligibility, blinded input, exactly one evaluator invocation per scoring id, strict worksheet, and append-only terminal status must agree before a score is accepted.
|
||||
- Predecessor 15 is encoded by `16+15_blind_score`. `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/complete.log` is missing because its active pair is pending; implementation must wait for it. Transitive predecessor 14 is satisfied only through 15.
|
||||
- PASS evidence is schema/manifest tests, gate eligibility tables, anonymization leak tests, scoring retry/corruption tests, production-shaped Codex invocation, and CLI/skill parity.
|
||||
- Report aggregation remains child 17.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude Markdown report/ranking generation and any change to D12 weights.
|
||||
- Do not expose evaluator identity to the evaluator workspace, persist credentials, retry implicitly, substitute route/model/effort, or convert ineligible/failed scores to zero.
|
||||
- Do not update the OpenAI-compatible wire contract because the operation uses the existing Responses request; the new public contract is the benchmark manifest/CLI schema in this repository.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build closures are all true. Scores `(scope=2,state=2,blast=2,evidence=1,verification=2)` produce G09; base/route `grade-boundary`, `worker/cloud/G09`, `PLAN-cloud-G09.md`.
|
||||
- Review closures are all true. Scores `(2,2,2,1,2)` produce `official-review`, `review/cloud/G09`, `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, `variant_product` (5). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/complete.log` must exist and PASS before implementation starts.
|
||||
2. Use predecessor 15's web loader/status/gate ids as the sole eligibility source; do not duplicate HTML or browser checks.
|
||||
3. Land manifest/rubric contracts before scoring storage, then evaluator execution, then CLI/skill exposure.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Bump the closed manifest to pipeline version 2 with a required direct Codex evaluator binding and exact `landing-quality-v1` rubric, updating schema/examples/all constructor fixtures and normal/boundary tests.
|
||||
- [ ] [API-2] Implement eligibility and identity-separated anonymization records so only lifecycle-success/web-pass attempts enter scoring and every other attempt is immutable `unscored` with exact reasons.
|
||||
- [ ] [API-3] Implement strict 100-point worksheet validation plus append-only `score-000001` attempts, fresh-session invocation, provenance, `scoring_failed`, and explicit new-id retry with corruption/leak tests.
|
||||
- [ ] [API-4] Add the public `score --manifest --run-id [--retry-scoring-failed]` operation, reuse live Codex preflight without fallback, update skill/routing contracts, and run focused/full validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Version the evaluator and rubric contract
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/manifest.py:105-121` has only `rubric_version`; `load_manifest` at lines 635-674 rejects any evaluator field. Keeping version 1 while adding a required field would silently redefine existing snapshots.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/manifest.py:104-118`):
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Manifest:
|
||||
pipeline_version: str
|
||||
environment: str
|
||||
testbed: str
|
||||
repetitions: int
|
||||
session_policy: str
|
||||
setup_cache_policy: str
|
||||
timeout: Timeout
|
||||
viewports: tuple[Viewport, ...]
|
||||
rubric_version: str
|
||||
output_root: str
|
||||
fixture: Fixture
|
||||
matrix: tuple[MatrixCell, ...]
|
||||
digest: str
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Evaluator:
|
||||
caller: str
|
||||
iop: IopCell
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Manifest:
|
||||
pipeline_version: str
|
||||
environment: str
|
||||
testbed: str
|
||||
repetitions: int
|
||||
session_policy: str
|
||||
setup_cache_policy: str
|
||||
timeout: Timeout
|
||||
viewports: tuple[Viewport, ...]
|
||||
rubric_version: str
|
||||
evaluator: Evaluator
|
||||
output_root: str
|
||||
fixture: Fixture
|
||||
matrix: tuple[MatrixCell, ...]
|
||||
digest: str
|
||||
```
|
||||
|
||||
Set `PIPELINE_VERSION = "2"`; require top-level `evaluator`, caller exactly `codex`, direct route, one `request` expected binding matching requested model/effort, and rubric exactly `landing-quality-v1`. Include evaluator in canonical JSON/digest. Update the checked-in JSON Schema and all examples/fixture builders atomically. Add `rubric.py` with the five immutable categories and weights summing to 100; rubric validation rejects booleans, fractions, extra/missing categories, out-of-range category scores, non-string evidence, and total mismatch.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: add version-2 evaluator validation and digest projection.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: normal/boundary/version/digest tests.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`: define exact revision, categories, weights, and worksheet validator.
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: category, total, evidence, and mutation tests.
|
||||
- [ ] `scripts/agent_benchmark/workspace_test.py`: update raw and direct `Manifest` constructors.
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: update manifest builder.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: update manifest builder.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: require version-2 evaluator object.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`: bind Codex evaluator and rubric.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`: bind Codex evaluator and rubric.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`: bind Codex evaluator and rubric.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write normal and one-field-at-a-time invalid cases for caller, route, model, effort, binding, revision, version, and digest drift. Update every constructor rather than adding a default that would hide missing evaluator data.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test`; expect all tests to pass.
|
||||
|
||||
### [API-2] Materialize deterministic eligibility and blinded inputs
|
||||
|
||||
**Problem**
|
||||
|
||||
No code combines lifecycle success with predecessor web gates, and the original attempt path exposes caller/cell identity. Passing the original workspace path to an evaluator violates S13.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:1355-1360`):
|
||||
|
||||
```python
|
||||
return {
|
||||
"run_id": bound_run.run_id,
|
||||
"manifest_digest": bound_run.manifest_digest,
|
||||
"preflight": projection,
|
||||
"attempts": states,
|
||||
}
|
||||
```
|
||||
|
||||
After, scoring reads the immutable run through public `RunStore` methods and writes no field into `attempt.json`. Add `scoring.py` with:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Mapping, Protocol
|
||||
```
|
||||
|
||||
For each retained terminal attempt, eligibility is exactly lifecycle `success` plus web status `passed` plus all required gate ids `passed`. Ineligible attempts get one no-clobber `scoring/unscored.json` containing closed reason codes and evidence digests; no score fields exist.
|
||||
|
||||
For eligible attempts, derive an opaque blind id from run manifest digest + attempt ordinal + harness-owned nonce, publish `scoring/anonymization.json` outside evaluator-visible directories, and copy only `index.html`, `styles.css`, `script.js`, the two referenced local images, and predecessor screenshots into `scoring/score-N/workspace/input/`. Preserve relative asset paths needed to render but omit caller, cell, route, model, effort, original path, lifecycle capture, and mapping. Refuse symlinks/non-regular files/path escapes and validate copied digests before launch.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: eligibility, unscored record, blinded id/map, contained copy, and strict loaders.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: eligible/ineligible table, path/symlink, identity leak, no-zero/no-clobber, and execution-record non-mutation tests through public `RunStore` enumeration.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use fixed run/attempt identities and compare every evaluator-visible byte/path against forbidden caller/cell/route/model strings. Assert failed/blocked/not-run gates each produce `unscored`, never score 0 and never allocate an evaluator attempt.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.attempts_test`; expect all eligibility, containment, and append-only tests to pass.
|
||||
|
||||
### [API-3] Run one fresh evaluator and seal its worksheet
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/codex_iop.py:326-355` builds one isolated Codex invocation for a normal benchmark attempt, but there is no scoring identity, evaluator preflight, worksheet contract, or retry ledger.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/live_iop.py:473-489`):
|
||||
|
||||
```python
|
||||
if self.caller == "codex":
|
||||
invocation = build_codex_invocation(cell, prepared, runtime_from_environment({BASE_URL_ENV_KEY: runtime.base_url, SECRET_ENV_KEY: runtime.secret, "PATH": os.environ.get("PATH", "/usr/bin:/bin")}), task_payload, timeout)
|
||||
invocation = replace(
|
||||
invocation,
|
||||
spec=_bind_live_spec(
|
||||
cell,
|
||||
prepared,
|
||||
attempt,
|
||||
control_dir,
|
||||
invocation.spec,
|
||||
),
|
||||
)
|
||||
result = self._invokers.codex(invocation, lambda locator: on_started(locator, spec_digest(invocation.spec)))
|
||||
expected = (admitted.effective_route_kind, admitted.effective_route_id, admitted.effective_model, admitted.effective_effort)
|
||||
if result.effective_binding != expected:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
return result.lifecycle
|
||||
```
|
||||
|
||||
After, extract the existing construction into `_build_admitted_codex_invocation(cell, prepared, task_payload, timeout, control_binding)` and call it from both the unchanged execution branch and a new injected `ScoringAdapter`. The helper reuses `_runtime_from_environment`, catalog observation, config binding, `codex_capability`, and `build_codex_invocation`; `control_binding=None` replaces only `spec.control_dir` with `None`, while a normal attempt still passes through `_bind_live_spec`. The scoring adapter returns a closed preflight projection and invocation without exposing runtime secrets.
|
||||
|
||||
Allocate `score-000001`, `score-000002`, ... under the execution attempt while holding the run writer. Every score directory has a fresh empty session and workspace, immutable preflight, lifecycle evidence, and terminal `result.json`. The evaluator prompt names only blind input paths and the exact five rubric categories; it requires a single `worksheet.json`. Validate the worksheet after successful lifecycle. Accept only integer category scores within their weights, evidence for every category, exact total, and no identity fields. Persist `status=scored` or `status=scoring_failed`; malformed/missing worksheet, lifecycle failure, binding mismatch, preflight blocker, or extra field cannot yield a score. Do not retry unless `--retry-scoring-failed` is explicit, and then allocate a new id without changing prior bytes. A prior successful score prohibits another attempt.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: scoring attempt allocation, prompt, invocation orchestration, terminal records, retry rules, and strict validation.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: fresh-session, exactly-once, failure matrix, retry id, prior-byte, tie-neutral, and corruption tests.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`: strict worksheet projection.
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: valid/invalid worksheet cases.
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: additive Codex evaluator preflight/invocation builder using existing live ownership.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: production-shaped evaluator path with no secret/identity persistence.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Inject a fake scoring adapter for deterministic worksheets and a production-shaped Codex executable for the actual subprocess boundary. Test every terminal status, explicit retry, duplicate call, invalid worksheet, and durable secret/identity scan.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`; expect all tests to pass with no live credential.
|
||||
|
||||
### [API-4] Expose scoring through the deterministic CLI and skill
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_comparison_benchmark.py:65-88` offers only validate/preflight/run/resume/status, and the benchmark skill treats report output as unavailable without a scoring step.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_comparison_benchmark.py:80-86`):
|
||||
|
||||
```python
|
||||
for command in ("preflight", "run", "resume", "status"):
|
||||
entry = sub.add_parser(command, help=f"Safely {command} benchmark state.")
|
||||
entry.add_argument("--manifest", required=True, help="Path to the manifest JSON file.")
|
||||
if command in {"resume", "status"}:
|
||||
entry.add_argument("--run-id", required=True, help="Harness-generated run id.")
|
||||
if command == "resume":
|
||||
entry.add_argument("--retry-failed", action="store_true")
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
for command in ("preflight", "run", "resume", "status", "score"):
|
||||
entry = sub.add_parser(command, help=f"Safely {command} benchmark state.")
|
||||
entry.add_argument("--manifest", required=True)
|
||||
if command in {"resume", "status", "score"}:
|
||||
entry.add_argument("--run-id", required=True)
|
||||
if command == "score":
|
||||
entry.add_argument("--retry-scoring-failed", action="store_true")
|
||||
```
|
||||
|
||||
`score` validates/open the immutable run, obtains the live Codex scoring adapter, runs eligibility/scoring under the run writer, and prints only closed counts for `scored`, `unscored`, `scoring_failed`, and `blocked`; blocked/failed returns 69 with the run id. Update the project skill to route `score`, require run id, document explicit retry and exact output, and keep report unavailable until child 17. Update project rules and semantic mutation tests so CLI help/options and skill commands remain exact.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_comparison_benchmark.py`: add score parser, command handler, closed summary, and exit behavior.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: add supported score procedure/inputs/output/safety.
|
||||
- [ ] `agent-ops/rules/project/rules.md`: route benchmark scoring requests.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: add exact score help/options/semantic mutations while retaining report unavailable assertions.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: CLI status/output/error-order tests.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Add CLI tests for missing state, no eligible attempt, scored, unscored, failed, blocked, retry flag, and read/append boundaries. Add mutation tests that reject missing/unknown score options, implicit retry, and provider invocation outside the CLI.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.skill_contract_test`; expect exact CLI/skill parity and all semantic mutation baselines to pass.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/workspace_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/scoring.py` | API-2, API-3 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3, API-4 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-3 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_comparison_benchmark.py` | API-4 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-4 |
|
||||
| `agent-ops/rules/project/rules.md` | API-4 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-4 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` | API-1 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/CODE_REVIEW-cloud-G09.md` | API-1, API-2, API-3, API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.rubric_test scripts.agent_benchmark.workspace_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test`
|
||||
- Expected: exit 0, no failures/errors/skips, no caller/secret identity in evaluator-visible or durable score evidence.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with the complete benchmark suite passing. Cached output is not acceptable.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py score --help`
|
||||
- Expected: exit 0; exactly `--manifest`, `--run-id`, and `--retry-scoring-failed` are documented.
|
||||
4. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
5. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- Expected for commands 4-6: exit 0 and `ok: manifest is valid` for pipeline version 2.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,205 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=1 tag=API milestone-task=report-output -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor indices 14/15/16 using task-protocol active-or-dated-archive PASS `complete.log` evidence rather than impossible post-PASS active-only paths.
|
||||
- Verification evidence: current fixture manifests validated locally; prior pair has no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_1.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Strict evidence join and rankings | [ ] |
|
||||
| API-2 — Deterministic Markdown/golden | [ ] |
|
||||
| API-3 — Public report CLI/skill operation | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G07_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G07_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require exact PASS evidence for predecessors 14/15/16 from active or dated archive task-protocol paths.
|
||||
- Verify unavailable values retain reason/source and are never zero or omitted.
|
||||
- Verify equal totals retain equal rank and identity/display order does not affect rank.
|
||||
- Verify gates do not add points and unscored/scoring_failed do not enter averages/rankings.
|
||||
- Verify raw links are run-relative/contained and corrupt input leaves no partial or overwritten report.
|
||||
- Verify report never constructs adapters/providers and the skill removes only the obsolete report capability gate.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: each exits 0; golden bytes, ties, non-mutation, and CLI/skill semantics pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused report/attempt/skill suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`
|
||||
|
||||
Expected: exit 0; all statuses/ties/unavailable render and no provider is invoked.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Report CLI help
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
|
||||
Expected: exit 0 and exactly `--manifest`, `--run-id` are documented.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Public command set
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
|
||||
Expected: validate/preflight/run/resume/status/score/report are present.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Obsolete capability scan
|
||||
|
||||
Command: `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
|
||||
Expected: exit 1 with no matches.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
|
||||
Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,211 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=1 tag=API milestone-task=report-output -->
|
||||
|
||||
# Deterministic benchmark Markdown report
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G07.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
Immutable manifest, preflight, execution, measurement, web, and scoring records need one deterministic reader-facing projection with explicit missing/failure semantics and contained raw pointers. SDD S14 requires conditions, versions, timings, tokens, quality, ties, limitations, evaluator provenance, and every non-success state without fabricated values. Self-review corrected predecessor evidence lookup so PASS archives remain runnable dependencies.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log`
|
||||
- Prior review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_0.log`
|
||||
- Verdict: 없음 — implementation 전 explicit self-review replan.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; affected artifacts are the prior plan/review pair only and no implementation file was changed by that pair.
|
||||
- Correction: resolve predecessor indices 14/15/16 using task-protocol active-or-dated-archive PASS `complete.log` evidence rather than impossible post-PASS active-only paths.
|
||||
- Verification evidence: current fixture manifests validated locally; prior pair has no implementation result or official verdict.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_0.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved SDD with released lock; `milestone-task=report-output`.
|
||||
- S14 requires success/failure/blocked/unscored/scoring_failed conditions, versions, time/token/quality, ties/limits, evaluator/scoring provenance, and raw evidence pointers.
|
||||
- Evidence Map S14 requires an all-status/tie golden. API-1 through API-3 and final verification cover this row.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Report authorities are immutable run/preflight/attempt records and strict predecessor 14-16 loaders; the renderer never parses arbitrary captures or recomputes absent provider values.
|
||||
- Tests are standard-library-only, credential-free, and uncached. A synthetic run tree covers all states; one CLI integration covers idempotent `report.md` publication.
|
||||
- Generation runs under the existing run writer. Links are POSIX run-root-relative, regular, and contained.
|
||||
- Confidence: high because output is a deterministic projection with byte-for-byte golden evidence.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No strict join exists across timing, web, scoring, evaluator, and preflight records.
|
||||
- No tie ranking preserves equal totals without arbitrary rank tie-break.
|
||||
- No all-status/unavailable golden or contained raw-link test exists.
|
||||
- CLI/skill still return `capability-unavailable: report-output`.
|
||||
- No idempotent/no-clobber publication exists.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- Exact CLI command/options/capability assertions in `skill_contract_test.py` must add `report` and remove only obsolete report-unavailable assertions.
|
||||
- CLI parser/main are the only public dispatch points; benchmark project rule/skill are the only user-facing routing documents.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- One stable invariant: validated immutable evidence produces one deterministic `report.md`; corrupt required input produces no report.
|
||||
- Directory `17+14,15,16_report_output` requires exact predecessors 14/15/16. All are pending now; runtime resolves active or dated archive PASS logs. Redundant transitive indices are preserved because the directory name is the runtime source of truth.
|
||||
- PASS evidence is an all-status/tie golden, corruption/containment/idempotency tests, CLI help/output, and semantic skill mutations.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude measurement, gate, and score policy changes. Use predecessor loaders and values verbatim.
|
||||
- Do not retry execution/scoring, call providers, infer missing data, average unscored results, add gate points, or break equal score ranks by identity/order.
|
||||
- Do not write documentation outside the benchmark skill/rule and run-owned report.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build/review closures are true. Scores `(scope=2,state=1,blast=2,evidence=1,verification=1)` produce G07.
|
||||
- Build base `local-fit`, route `risk-boundary`, `worker/cloud/G07`, `PLAN-cloud-G07.md`; review `official-review`, `review/cloud/G07`, `CODE_REVIEW-cloud-G07.md`.
|
||||
- `large_indivisible_context=false`; loop risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve predecessor 14 from its exact active or dated archive `complete.log`; require PASS.
|
||||
2. Resolve predecessor 15 from its exact active or dated archive `complete.log`; require PASS.
|
||||
3. Resolve predecessor 16 from its exact active or dated archive `complete.log`; require PASS.
|
||||
4. Current preparation state: all three are pending. Implement strict projection/golden before exposing `report`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Join evidence without inference
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/attempts.py:1334-1360` exposes aggregate state counts but not measurement, web, score, or evaluator provenance. Ad hoc JSON parsing would bypass predecessor schema validation.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `reporting.py`. Open the bound run through `RunStore`, enumerate stable manifest cell/repetition/attempt order, and call strict loaders for measurement, web, anonymization, score attempts, and unscored records. Project execution/preflight status, timing/token availability+source, web gates/screenshots, scoring status/categories/total, manifest evaluator binding, score id, and contained raw paths.
|
||||
|
||||
Missing optional observations render the producer's explicit unavailable reason; required file/digest/schema contradictions raise `ReportError` and write nothing. Rank only scored attempts by descending total. Equal totals share a tie group and competition rank (`1,1,3`); display order never changes rank. Unscored/scoring_failed attempts have no rank and do not enter averages.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: strict join/projection/rank/tie/contained pointers.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: every state, unavailable/source, ties/order, missing/corrupt inputs, non-mutation.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Build a synthetic version-2 run with historical blocked then ready preflight, successful/failed attempts, observed/unavailable metrics, passed/failed/blocked web, scored/unscored/scoring_failed, and equal totals. Mutate each required input separately and assert no output.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`; expect exit 0.
|
||||
|
||||
### [API-2] Render one deterministic Markdown artifact
|
||||
|
||||
**Problem**
|
||||
|
||||
No human-readable artifact or stable raw-link contract exists. Omitting conditions or replacing unavailable with zero contradicts S14.
|
||||
|
||||
**Solution**
|
||||
|
||||
Render, in fixed order: run identity; methodology/immutable conditions; manifest/environment/fixture/rubric/evaluator versions; preflight; attempt outcomes; time/token values with unit/clock/source or unavailable reason; automatic web gates/screenshots; quality/rank; evaluator/scoring provenance; failures/unscored/scoring_failed; limitations; raw evidence index.
|
||||
|
||||
Escape Markdown cells/links, use contained POSIX run-relative paths, and refuse symlinks/escapes. Publish UTF-8 LF `<run-root>/report.md` without replacement. Repeated generation succeeds only when the existing regular file equals freshly rendered bytes; any differing byte fails closed. Add exact all-status/tie golden `scripts/fixtures/agent-comparison-benchmark-report.expected.md`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: renderer/idempotent publisher.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: full-byte golden, escaping, containment, collision/symlink, identical rerun.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-report.expected.md`: exact expected report.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Compare complete bytes, perturb input order for stability, and mutate existing output to prove no overwrite.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test`; expect exact golden equality.
|
||||
|
||||
### [API-3] Make report a supported deterministic operation
|
||||
|
||||
**Problem**
|
||||
|
||||
The CLI has no `report`, and `skill_contract_test.py:238-246` requires report capability unavailable.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `report --manifest --run-id`. Validate/open immutable state before writer acquisition, call reporter, and print only `ok: report run_id=<id> path=<run-relative-path>`. Invalid/corrupt state prints `error: benchmark report is unavailable`, exits 69, and leaves no partial output. Never construct adapters or invoke providers.
|
||||
|
||||
Update the benchmark skill description/triggers/inputs/preflight/procedure/validation/output/safety/stops/prohibitions to delegate the exact CLI and output. Update project routing and exact semantic mutation tests; supported commands include predecessor `score` and new `report`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_comparison_benchmark.py`: report parser/handler/dispatch.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: CLI error/success/idempotent/no-provider.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: executable report delegation.
|
||||
- [ ] `agent-ops/rules/project/rules.md`: report routing.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: score+report commands/options/mutations.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use a mocked reporting boundary for error ordering and the synthetic run for success/idempotency; patch adapter construction to fail if report touches it.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`; expect exit 0 and no capability-unavailable string.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/reporting.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/reporting_test.py` | API-1, API-2, API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-report.expected.md` | API-2 |
|
||||
| `scripts/agent_comparison_benchmark.py` | API-3 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 |
|
||||
| `agent-ops/rules/project/rules.md` | API-3 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G07.md` | API-1, API-2, API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`
|
||||
- Expected: exit 0, byte golden/all statuses/ties/unavailable pass, report never invokes provider.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with fresh complete suite output.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
- Expected: exit 0 with exactly `--manifest`, `--run-id`.
|
||||
4. `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
- Expected: public commands include validate/preflight/run/resume/status/score/report.
|
||||
5. `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
- Expected: exit 1 with no matches.
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,195 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=0 tag=API milestone-task=report-output -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-cloud-G07.md` → `plan_cloud_G07_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=report-output` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Strict evidence join and rankings | [ ] |
|
||||
| API-2 — Deterministic Markdown/golden | [ ] |
|
||||
| API-3 — Public report CLI/skill operation | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, version, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing and semantic contract tests, and run focused/full verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G07.md` to `code_review_cloud_G07_0.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G07.md` to `plan_cloud_G07_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=report-output` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require PASS evidence for predecessors 14, 15, and 16 and call their strict loaders rather than reparsing captures.
|
||||
- Verify unavailable values retain reason/source and are never rendered as zero or silently omitted.
|
||||
- Verify equal totals retain equal rank and no caller/cell/order tie-break affects rank.
|
||||
- Verify gates are not added to rubric totals and unscored/scoring_failed attempts do not enter averages/rankings.
|
||||
- Verify every raw link is run-relative/contained and corrupt input leaves no partial or overwritten report.
|
||||
- Verify `report` never constructs adapters or invokes a provider and the skill no longer contains the obsolete capability gate.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: each exits 0; golden bytes, tie projection, non-mutation, and CLI/skill semantics pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused report/attempt/skill suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`
|
||||
|
||||
Expected: exit 0, byte-for-byte golden passes, all statuses/ties/unavailable values render, and report never invokes a provider.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0; cached output is not acceptable.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Report CLI help
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
|
||||
Expected: exit 0 and exactly `--manifest` and `--run-id` are documented.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Public command set
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
|
||||
Expected: exit 0 and commands include validate, preflight, run, resume, status, score, report.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Obsolete capability scan
|
||||
|
||||
Command: `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
|
||||
Expected: exit 1 with no matches.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
|
||||
Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,251 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=0 tag=API milestone-task=report-output -->
|
||||
|
||||
# Deterministic benchmark Markdown report
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G07.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The pipeline's immutable manifest, preflight, execution, measurement, web, and scoring records are not useful to readers until they are joined with explicit missing/failure semantics and raw evidence pointers. SDD S14 requires a Markdown report that preserves conditions, versions, timings, tokens, quality, ties, limits, evaluator provenance, and every non-success state without fabricating values.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`, approved and implementation lock released.
|
||||
- First-line milestone task: `report-output`.
|
||||
- Acceptance Scenario S14 requires success, failure, blocked, unscored, and scoring_failed evidence to render conditions/version, time/token/quality, ties/limits, evaluator/scoring-attempt provenance, and raw pointers.
|
||||
- Evidence Map S14 requires a golden covering all statuses and ties. These rows define API-1 through API-3 and the final golden/skill verification.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Report input authorities are the immutable run snapshot/preflight/attempt records and the strict loaders from predecessors 14-16; the renderer must not parse arbitrary captures or recompute absent provider values.
|
||||
- Verification is standard-library-only and credential-free. A synthetic run tree covers every state, while one CLI integration path checks idempotent `report.md` publication.
|
||||
- Preconditions: predecessor schemas/loaders have PASS `complete.log`; report generation runs under the existing run writer; all links are run-root-relative and contained.
|
||||
- No external verification is needed. `unittest` output is uncached and must be fresh.
|
||||
- Confidence: high because the report is a deterministic projection with a checked-in byte-for-byte golden.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No module joins preflight, attempts, timing, web, and scoring by immutable identity/digest.
|
||||
- No ranking/tie algorithm preserves equal totals without arbitrary tie-break.
|
||||
- No rendering test covers unavailable tokens/timing, blocked preflight, lifecycle failure, unscored, scoring_failed, and scored rows together.
|
||||
- CLI and skill still return `capability-unavailable: report-output`.
|
||||
- No idempotent/no-clobber report publication or raw-link containment test exists.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `_CLI_HELP_COMMANDS`, `_assert_command_options`, `_assert_capabilities`, and command-specific help tests in `skill_contract_test.py` must include `report` and remove only the obsolete capability-unavailable assertions.
|
||||
- `_build_parser`/`main` in `scripts/agent_comparison_benchmark.py` are the only public command dispatch points.
|
||||
- The benchmark project rule and skill are the only user-facing report routing documents.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one stable projection contract: validated immutable evidence produces one deterministic `report.md`; input corruption produces no report.
|
||||
- Predecessors 14, 15, and 16 are encoded by `17+14,15,16_report_output`. Their `complete.log` files are all missing because their active pairs are pending; implementation must wait for all three even though 15/16 encode transitive dependencies.
|
||||
- PASS evidence is the all-status/tie golden, corrupt/missing input tests, idempotent publication, CLI help/output, and skill semantic mutation suite.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude new measurement, gate, or scoring policy. The reporter calls predecessor loaders and renders their values verbatim.
|
||||
- Do not retry execution/scoring, call providers, infer missing data, average `unscored`, count gates toward 100 points, or break equal scores by caller/name/order.
|
||||
- Do not write documentation outside the benchmark skill/rule and the run-owned report artifact.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build closures are all true. Scores `(scope=2,state=1,blast=2,evidence=1,verification=1)` produce G07; base `local-fit`, route `risk-boundary`, `worker/cloud/G07`, `PLAN-cloud-G07.md`.
|
||||
- Review closures are all true. Scores `(2,1,2,1,1)` produce `official-review`, `review/cloud/G07`, `CODE_REVIEW-cloud-G07.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `agent-task/m-agent-comparison-benchmark-pipeline/14_timing_usage/complete.log` must exist and PASS.
|
||||
2. `agent-task/m-agent-comparison-benchmark-pipeline/15+14_web_validation/complete.log` must exist and PASS.
|
||||
3. `agent-task/m-agent-comparison-benchmark-pipeline/16+15_blind_score/complete.log` must exist and PASS.
|
||||
4. Implement the read-only projection and golden before exposing `report`; update the skill only after the CLI is executable.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, version, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing and semantic contract tests, and run focused/full verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Join evidence without inference
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/attempts.py:1334-1360` exposes only aggregate state counts. It does not project attempt-level measurement, web, scoring, or evaluator provenance, and reading raw JSON ad hoc would bypass predecessor schema validation.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:1355-1360`):
|
||||
|
||||
```python
|
||||
return {
|
||||
"run_id": bound_run.run_id,
|
||||
"manifest_digest": bound_run.manifest_digest,
|
||||
"preflight": projection,
|
||||
"attempts": states,
|
||||
}
|
||||
```
|
||||
|
||||
After, leave `status()` unchanged and add `reporting.py` that opens the bound run via `RunStore`, enumerates cells/repetitions/attempts in stable manifest order, and calls predecessor strict loaders for measurement, web, anonymization, score attempts, and unscored records:
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
```
|
||||
|
||||
Define a closed projection per attempt with execution status/reason, preflight status, timing/token availability+source, web gate status, scoring status, rubric categories/total, evaluator binding, scoring attempt id, and contained raw paths. A missing optional observation renders unavailable with its producer's reason; a required file/digest/schema contradiction raises `ReportError` and writes nothing.
|
||||
|
||||
Rank only `scored` attempts by descending total. Equal totals receive the same rank and an explicit tie group; subsequent rank uses competition ranking (for example 1,1,3). Do not use caller/cell/attempt as a tie-break beyond deterministic row display order. `unscored` and `scoring_failed` have no rank and do not enter averages.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: add strict join, projections, rank/tie logic, and contained raw pointers.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: cover all states, unavailable/source display, ties, ordering, corrupt/missing inputs, and attempt/preflight non-mutation.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Construct a synthetic version-2 run with ready and blocked preflight evidence; success/failure attempts; observed/unavailable metrics; passed/failed/blocked web records; scored/unscored/scoring_failed records; and two equal scores. Assert exact projections and no output on each one-field corruption.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`; expect all join, tie, corruption, and non-mutation tests to pass.
|
||||
|
||||
### [API-2] Render one deterministic Markdown artifact
|
||||
|
||||
**Problem**
|
||||
|
||||
There is no human-readable artifact or stable raw evidence link contract. A report that omits conditions or substitutes zero for missing data would contradict S14.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md:41-42`):
|
||||
|
||||
```markdown
|
||||
2. **Handle report-readiness**
|
||||
- Return `capability-unavailable: report-output` and stop. Do not attempt to generate, render, or fabricate any report or output. Report rendering belongs to a later Epic.
|
||||
```
|
||||
|
||||
After the projection is implemented, render sections in this exact order: title/run identity; methodology and immutable conditions; manifest/environment/fixture/rubric/evaluator versions; preflight summary; attempt outcome table; timing/token table with value+unit+clock+source or explicit unavailable reason; automatic web gate table and screenshot links; quality/rank table; evaluator/scoring provenance; failures/unscored/scoring_failed; limitations; raw evidence index.
|
||||
|
||||
Escape Markdown cells/links, use POSIX run-relative paths, and refuse escapes/symlinks. Write UTF-8 LF bytes to `<run-root>/report.md` without replacement. A repeated command validates the existing regular file and succeeds only if its bytes equal the newly rendered bytes; differing existing bytes fail closed. Add `scripts/fixtures/agent-comparison-benchmark-report.expected.md` as the exact all-status/tie golden.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: add deterministic Markdown rendering and idempotent no-clobber publication.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: byte golden, escaping, path containment, collision, symlink, and identical rerun tests.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-report.expected.md`: add the all-status/tie expected report.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Compare complete bytes, not substring fragments. Mutate input order to prove stable output and mutate an existing report byte to prove no overwrite.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test`; expect byte-for-byte golden equality and all safety cases to pass.
|
||||
|
||||
### [API-3] Make report output a supported CLI/skill operation
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_comparison_benchmark.py:65-88` does not parse `report`, and `skill_contract_test.py:238-246` requires `capability-unavailable: report-output`.
|
||||
|
||||
**Solution**
|
||||
|
||||
Before (`scripts/agent_benchmark/skill_contract_test.py:243-247`):
|
||||
|
||||
```python
|
||||
def _assert_capabilities(self, skill_text: str) -> None:
|
||||
"""Assert report remains unavailable while run/resume are executable."""
|
||||
self.assertIn(_CAPABILITY_REPORT_OUTPUT, skill_text)
|
||||
procedure = self._get_section(skill_text, "Procedure")
|
||||
self.assertIn(_CAPABILITY_REPORT_OUTPUT, procedure)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _assert_capabilities(self, skill_text: str) -> None:
|
||||
procedure = self._get_section(skill_text, "Procedure")
|
||||
self.assertNotIn("capability-unavailable: report-output", skill_text)
|
||||
self.assertIn("python3 scripts/agent_comparison_benchmark.py report --manifest <manifest-path> --run-id <run-id>", procedure)
|
||||
```
|
||||
|
||||
Extend the post-16 parser with `report --manifest --run-id`. `_cmd_report` validates/opens the immutable run before acquiring the writer, calls the reporting module, and prints only `ok: report run_id=<id> path=<run-relative-path>`; invalid/corrupt state prints `error: benchmark report is unavailable` and exits 69 without partial output. It never calls adapters/providers.
|
||||
|
||||
Update the benchmark skill description, triggers, inputs, preflight, procedure, validation, output, safety, stop conditions, and prohibitions to delegate the exact command and report verbatim CLI output. Remove capability-unavailable wording. Update project rules from report readiness to report generation. Refactor semantic contract tests so supported command/option sets include the predecessor `score` and new `report`, and add mutations rejecting unknown/missing report options, fabricated output, direct provider calls, and writes outside the run root.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_comparison_benchmark.py`: add report parser/handler/dispatch and closed output.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: add CLI invalid/success/idempotent/no-provider tests.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: replace report capability gate with executable delegation.
|
||||
- [ ] `agent-ops/rules/project/rules.md`: route report generation/output requests.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: assert score+report help/options and new semantic mutations.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use a mocked reporting boundary for CLI error-order tests and the synthetic run for successful/idempotent report. Patch adapter construction to fail if called by report. Validate the full skill text and each unsafe mutation independently.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`; expect all CLI/skill/golden tests to pass and no capability-unavailable string to remain.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/reporting.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/reporting_test.py` | API-1, API-2, API-3 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-report.expected.md` | API-2 |
|
||||
| `scripts/agent_comparison_benchmark.py` | API-3 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 |
|
||||
| `agent-ops/rules/project/rules.md` | API-3 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-3 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G07.md` | API-1, API-2, API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`
|
||||
- Expected: exit 0, byte-for-byte golden passes, all statuses/ties/unavailable values render, and report never invokes a provider.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with the complete benchmark suite passing. Cached output is not acceptable.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
- Expected: exit 0; exactly `--manifest` and `--run-id` are documented.
|
||||
4. `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
- Expected: exit 0 and the public command set includes `validate`, `preflight`, `run`, `resume`, `status`, `score`, and `report`.
|
||||
5. `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
- Expected: exit 1 with no matches.
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -18,13 +18,14 @@
|
|||
"rubric_version": "v1.0",
|
||||
"output_root": "agent-test/runs/bench-01-direct-preflight",
|
||||
"fixture": {
|
||||
"version": "v1.0",
|
||||
"version": "landing-v1",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/prompt.md", "workspace_path": "workspace/prompt.md"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "workspace/reference.txt"}
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:f87b1a06dcd60687f2964a7c8e48227847acda195d3e5808710fe3f0e8149108"
|
||||
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
|
|
|
|||
|
|
@ -18,13 +18,14 @@
|
|||
"rubric_version": "v1.0",
|
||||
"output_root": "agent-test/runs/bench-01",
|
||||
"fixture": {
|
||||
"version": "v1.0",
|
||||
"version": "landing-v1",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/prompt.md", "workspace_path": "workspace/prompt.md"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "workspace/reference.txt"}
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:f87b1a06dcd60687f2964a7c8e48227847acda195d3e5808710fe3f0e8149108"
|
||||
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
|
|
|
|||
|
|
@ -18,13 +18,14 @@
|
|||
"rubric_version": "v1.0",
|
||||
"output_root": "agent-test/runs/bench-01-supported-direct",
|
||||
"fixture": {
|
||||
"version": "v1.0",
|
||||
"version": "landing-v1",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/prompt.md", "workspace_path": "workspace/prompt.md"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "workspace/reference.txt"}
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:f87b1a06dcd60687f2964a7c8e48227847acda195d3e5808710fe3f0e8149108"
|
||||
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
|
|
|
|||
42
scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg
Normal file
42
scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 800" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Aurora service grid</title>
|
||||
<desc id="desc">Abstract luminous nodes connected across a dark blue operational grid.</desc>
|
||||
<defs>
|
||||
<linearGradient id="background" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0" stop-color="#08172e"/>
|
||||
<stop offset="1" stop-color="#163764"/>
|
||||
</linearGradient>
|
||||
<radialGradient id="glow">
|
||||
<stop offset="0" stop-color="#7ef9d6" stop-opacity="0.95"/>
|
||||
<stop offset="1" stop-color="#42a5ff" stop-opacity="0"/>
|
||||
</radialGradient>
|
||||
<pattern id="grid" width="80" height="80" patternUnits="userSpaceOnUse">
|
||||
<path d="M80 0H0V80" fill="none" stroke="#b9d8ff" stroke-opacity="0.14"/>
|
||||
</pattern>
|
||||
</defs>
|
||||
<rect width="1200" height="800" rx="48" fill="url(#background)"/>
|
||||
<rect width="1200" height="800" rx="48" fill="url(#grid)"/>
|
||||
<g fill="none" stroke="#8bd6ff" stroke-width="4" stroke-opacity="0.65">
|
||||
<path d="M180 520 390 300 610 430 830 210 1030 390"/>
|
||||
<path d="M260 170 390 300 520 155 830 210 940 610"/>
|
||||
<path d="M180 520 480 650 610 430 940 610"/>
|
||||
</g>
|
||||
<g fill="url(#glow)">
|
||||
<circle cx="180" cy="520" r="120"/>
|
||||
<circle cx="390" cy="300" r="140"/>
|
||||
<circle cx="610" cy="430" r="150"/>
|
||||
<circle cx="830" cy="210" r="125"/>
|
||||
<circle cx="940" cy="610" r="135"/>
|
||||
</g>
|
||||
<g fill="#d9fff3" stroke="#0b263e" stroke-width="8">
|
||||
<circle cx="180" cy="520" r="18"/>
|
||||
<circle cx="390" cy="300" r="24"/>
|
||||
<circle cx="610" cy="430" r="28"/>
|
||||
<circle cx="830" cy="210" r="21"/>
|
||||
<circle cx="940" cy="610" r="23"/>
|
||||
<circle cx="260" cy="170" r="14"/>
|
||||
<circle cx="520" cy="155" r="16"/>
|
||||
<circle cx="1030" cy="390" r="17"/>
|
||||
<circle cx="480" cy="650" r="15"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.8 KiB |
37
scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg
Normal file
37
scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Layered orbit rings</title>
|
||||
<desc id="desc">Abstract rings and markers arranged around a bright central operating point.</desc>
|
||||
<defs>
|
||||
<linearGradient id="background" x1="0" y1="1" x2="1" y2="0">
|
||||
<stop offset="0" stop-color="#eef8ff"/>
|
||||
<stop offset="1" stop-color="#dffcf2"/>
|
||||
</linearGradient>
|
||||
<linearGradient id="ring" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0" stop-color="#1769aa"/>
|
||||
<stop offset="1" stop-color="#00a77b"/>
|
||||
</linearGradient>
|
||||
<radialGradient id="core">
|
||||
<stop offset="0" stop-color="#ffffff"/>
|
||||
<stop offset="0.55" stop-color="#79f2cb"/>
|
||||
<stop offset="1" stop-color="#1686d9"/>
|
||||
</radialGradient>
|
||||
</defs>
|
||||
<rect width="1000" height="1000" rx="72" fill="url(#background)"/>
|
||||
<g transform="translate(500 500)" fill="none" stroke="url(#ring)">
|
||||
<ellipse rx="350" ry="150" stroke-width="18" transform="rotate(-18)"/>
|
||||
<ellipse rx="350" ry="150" stroke-width="10" stroke-opacity="0.55" transform="rotate(42)"/>
|
||||
<ellipse rx="350" ry="150" stroke-width="6" stroke-opacity="0.35" transform="rotate(102)"/>
|
||||
<circle r="245" stroke-width="3" stroke-dasharray="8 18" stroke-opacity="0.42"/>
|
||||
</g>
|
||||
<circle cx="500" cy="500" r="112" fill="url(#core)"/>
|
||||
<g fill="#0a3157" stroke="#ffffff" stroke-width="12">
|
||||
<circle cx="170" cy="390" r="30"/>
|
||||
<circle cx="710" cy="245" r="25"/>
|
||||
<circle cx="820" cy="610" r="34"/>
|
||||
<circle cx="340" cy="780" r="27"/>
|
||||
</g>
|
||||
<g fill="#ffffff" opacity="0.8">
|
||||
<circle cx="470" cy="462" r="18"/>
|
||||
<circle cx="535" cy="520" r="12"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 1.7 KiB |
|
|
@ -1,3 +1,14 @@
|
|||
This is a deterministic, inert fixture prompt for the agent comparison benchmark.
|
||||
It does not contain any credentials, private endpoints, or production data.
|
||||
It is intentionally minimal so that fixture checksums remain stable across runs.
|
||||
Build a polished, responsive one-page product landing page for the fictional product described in `brief/reference.txt`.
|
||||
|
||||
Requirements:
|
||||
|
||||
- Create exactly these implementation files at the workspace root: `index.html`, `styles.css`, and `script.js`.
|
||||
- Use both provided local images, `assets/aurora-grid.svg` and `assets/orbit-rings.svg`, as visible `<img>` content with meaningful `alt` text.
|
||||
- Include a keyboard-accessible navigation, hero, feature section, workflow section, testimonial, pricing callout, and footer using the supplied copy.
|
||||
- Provide one small progressive-enhancement interaction in `script.js`; the page must remain readable when JavaScript is unavailable.
|
||||
- Support the supplied desktop and mobile viewports without horizontal overflow, clipped primary content, or overlapping controls.
|
||||
- Use semantic HTML, visible focus states, sufficient text/background contrast, a logical heading order, and labels for interactive controls.
|
||||
- Do not use external network assets, frameworks, package managers, build tools, inline data URLs, or generated replacements for the provided images.
|
||||
- Do not add credentials, private endpoints, analytics, trackers, or production data.
|
||||
|
||||
Finish only after all three required files exist and the page can be served as static files from the workspace root.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,21 @@
|
|||
This is a deterministic, inert fixture asset for the agent comparison benchmark.
|
||||
It does not contain any credentials, private endpoints, or production data.
|
||||
It is intentionally minimal so that fixture checksums remain stable across runs.
|
||||
PRODUCT: Lumen Atlas
|
||||
EYEBROW: A calmer way to understand complex systems
|
||||
HEADLINE: Turn scattered signals into a shared operating picture.
|
||||
SUMMARY: Lumen Atlas brings service health, ownership, and live operational context into one focused workspace so teams can decide with confidence.
|
||||
PRIMARY CTA: Explore the workspace
|
||||
SECONDARY CTA: See how it works
|
||||
|
||||
FEATURE 1 TITLE: One clear map
|
||||
FEATURE 1 BODY: Connect services, dependencies, and owners without flattening the details that matter.
|
||||
FEATURE 2 TITLE: Evidence in context
|
||||
FEATURE 2 BODY: Keep decisions close to live signals, recent changes, and the people responsible for the next move.
|
||||
FEATURE 3 TITLE: Calm by default
|
||||
FEATURE 3 BODY: Prioritize meaningful changes and reduce the visual noise that slows incident response.
|
||||
|
||||
WORKFLOW TITLE: From signal to shared decision
|
||||
WORKFLOW STEPS: Observe / Connect / Act
|
||||
TESTIMONIAL: “Lumen Atlas helps our team see the same system, ask better questions, and move together.”
|
||||
TESTIMONIAL ATTRIBUTION: Maya Chen, Platform Lead at Northstar Labs
|
||||
PRICING TITLE: Start with the systems you operate today.
|
||||
PRICING BODY: A focused workspace for growing platform teams, with room to expand as ownership evolves.
|
||||
FOOTER NOTE: Fictional benchmark content. No production data.
|
||||
|
|
|
|||
Loading…
Reference in a new issue