chore(epic): evidence-report 준비 결과를 검증한다
This commit is contained in:
parent
bcfb3bb84e
commit
7692bd517f
6 changed files with 672 additions and 3 deletions
|
|
@ -0,0 +1,154 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=2 tag=API milestone-task=report-output -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output, plan=2, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log.
|
||||
- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. Parent API-1/API-2 form this report-core child; parent API-3 moved to `18+17_report_cli`.
|
||||
- Verification carryover: all-status/tie golden, corruption/containment/idempotency tests, full benchmark suite; no provider call.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_2.log` and `PLAN-cloud-G06.md` → `plan_cloud_G06_2.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=report-output` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Strict evidence join and rankings | [ ] |
|
||||
| API-2 — Deterministic Markdown/golden | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G06.md` to `plan_cloud_G06_2.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=report-output` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require exact PASS evidence for predecessors 14/15/16 from active or dated archive task-protocol paths.
|
||||
- Verify unavailable values retain reason/source and are never zero or omitted.
|
||||
- Verify equal totals retain equal rank and identity/display order does not affect rank.
|
||||
- Verify gates do not add points and unscored/scoring_failed do not enter averages/rankings.
|
||||
- Verify raw links are run-relative/contained and corrupt input leaves no partial or overwritten report.
|
||||
- Verify this child does not expose a public report command or modify benchmark skill/routing.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification commands
|
||||
|
||||
Commands:
|
||||
|
||||
```text
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test
|
||||
python3 -m unittest scripts.agent_benchmark.reporting_test
|
||||
```
|
||||
|
||||
Expected: each exits 0; golden bytes, statuses, ties, containment, and non-mutation pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Focused report-core suite
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`
|
||||
|
||||
Expected: exit 0; all statuses/ties/unavailable values, corruption, containment, and non-mutation cases pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh complete suite output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output plan=2 tag=API milestone-task=report-output -->
|
||||
|
||||
# Strict benchmark report projection and Markdown artifact
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G06.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
Immutable manifest, preflight, execution, measurement, web, and scoring records need one deterministic reader-facing projection with explicit missing/failure semantics and contained raw pointers. This child implements the strict join, rank/tie projection, Markdown renderer, and immutable report artifact required by SDD S14. Public CLI and skill exposure remain a dependent rollout child so the internal report contract can independently PASS first.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log.
|
||||
- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. Parent API-1/API-2 form this report-core child; parent API-3 moved to `18+17_report_cli`.
|
||||
- Verification carryover: all-status/tie golden, corruption/containment/idempotency tests, full benchmark suite; no provider call.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`; lock released; `milestone-task=report-output`.
|
||||
- S14 requires success/failure/blocked/unscored/scoring_failed conditions, versions, time/token/quality, ties/limits, evaluator/scoring provenance, and raw evidence pointers.
|
||||
- Evidence Map S14 requires an all-status/tie golden. API-1/API-2 and final verification cover the deterministic projection and artifact rows; the dependent CLI child owns public exposure and the SDD common `git diff --check` closure.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. Report authorities are immutable run/preflight/attempt records and strict predecessor 14-16 loaders; the renderer never parses arbitrary captures or recomputes absent provider values.
|
||||
- Tests are standard-library-only, credential-free, and uncached. A synthetic run tree covers all states and report publication remains run-owned, POSIX-relative, regular, and contained.
|
||||
- No external verification is required. Confidence is high because output is a deterministic projection with byte-for-byte golden evidence.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No strict join exists across timing, web, scoring, evaluator, and preflight records.
|
||||
- No tie ranking preserves equal totals without arbitrary rank tie-break.
|
||||
- No all-status/unavailable golden, contained raw-link test, or idempotent/no-clobber publication exists.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Child 17 owns a stable contract: validated immutable evidence produces one deterministic `report.md`; corrupt required input produces no report.
|
||||
- Child `18+17_report_cli` consumes that contract and exposes it without changing report semantics.
|
||||
- Directory predecessors 14/15/16 remain exact runtime dependencies. All are pending; implementation waits for their active-or-dated-archive PASS `complete.log` evidence.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude CLI parser/dispatch, benchmark skill/rule, and semantic routing tests; child 18 owns them.
|
||||
- Exclude measurement, gate, and score policy changes. Use predecessor loaders and values verbatim.
|
||||
- Do not retry execution/scoring, call providers, infer missing data, average unscored results, add gate points, or break equal ranks by identity/order.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build/review closures are all true. Scores `(scope=2,state=1,blast=1,evidence=1,verification=1)` produce G06.
|
||||
- Build base `local-fit`, route `risk-boundary`, `worker/cloud/G06`, `PLAN-cloud-G06.md`; review `official-review`, `review/cloud/G06`, `CODE_REVIEW-cloud-G06.md`.
|
||||
- `large_indivisible_context=false`; loop risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve predecessor 14 from its exact active or dated archive `complete.log`; require PASS.
|
||||
2. Resolve predecessor 15 from its exact active or dated archive `complete.log`; require PASS.
|
||||
3. Resolve predecessor 16 from its exact active or dated archive `complete.log`; require PASS.
|
||||
4. Current preparation state: all three are pending. Implement the strict projection and golden without exposing a public command.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Join evidence without inference
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_benchmark/attempts.py:1334-1360` exposes aggregate state counts but not measurement, web, score, or evaluator provenance. Ad hoc JSON parsing would bypass predecessor schema validation.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `reporting.py`. Open the bound run through `RunStore`, enumerate stable manifest cell/repetition/attempt order, and call strict loaders for measurement, web, anonymization, score attempts, and unscored records. Project execution/preflight status, timing/token availability+source, web gates/screenshots, scoring status/categories/total, manifest evaluator binding, score id, and contained raw paths.
|
||||
|
||||
Missing optional observations render the producer's explicit unavailable reason; required file/digest/schema contradictions raise `ReportError` and write nothing. Rank only scored attempts by descending total. Equal totals share a tie group and competition rank (`1,1,3`); display order never changes rank. Unscored/scoring_failed attempts have no rank and do not enter averages.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: strict join/projection/rank/tie/contained pointers.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: every state, unavailable/source, ties/order, missing/corrupt inputs, non-mutation.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Build a synthetic version-2 run with historical blocked then ready preflight, successful/failed attempts, observed/unavailable metrics, passed/failed/blocked web, scored/unscored/scoring_failed, and equal totals. Mutate each required input separately and assert no output.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`; expect exit 0.
|
||||
|
||||
### [API-2] Render one deterministic Markdown artifact
|
||||
|
||||
**Problem**
|
||||
|
||||
No human-readable artifact or stable raw-link contract exists. Omitting conditions or replacing unavailable with zero contradicts S14.
|
||||
|
||||
**Solution**
|
||||
|
||||
Render, in fixed order: run identity; methodology/immutable conditions; manifest/environment/fixture/rubric/evaluator versions; preflight; attempt outcomes; time/token values with unit/clock/source or unavailable reason; automatic web gates/screenshots; quality/rank; evaluator/scoring provenance; failures/unscored/scoring_failed; limitations; raw evidence index.
|
||||
|
||||
Escape Markdown cells/links, use contained POSIX run-relative paths, and refuse symlinks/escapes. Publish UTF-8 LF `<run-root>/report.md` without replacement. Repeated generation succeeds only when the existing regular file equals freshly rendered bytes; any differing byte fails closed. Add exact all-status/tie golden `scripts/fixtures/agent-comparison-benchmark-report.expected.md`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/reporting.py`: renderer/idempotent publisher.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: full-byte golden, escaping, containment, collision/symlink, identical rerun.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-report.expected.md`: exact expected report.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Compare complete bytes, perturb input order for stability, and mutate existing output to prove no overwrite.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test`; expect exact golden equality.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_benchmark/reporting.py` | API-1, API-2 |
|
||||
| `scripts/agent_benchmark/reporting_test.py` | API-1, API-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-report.expected.md` | API-2 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G06.md` | API-1, API-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`
|
||||
- Expected: exit 0; all statuses/ties/unavailable values, corruption, containment, and non-mutation cases pass.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with fresh complete suite output.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -54,7 +54,7 @@ Review completion means the following steps are finished:
|
|||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
|
@ -91,6 +91,7 @@ _Record key design decisions here._
|
|||
- Verify gates do not add points and unscored/scoring_failed do not enter averages/rankings.
|
||||
- Verify raw links are run-relative/contained and corrupt input leaves no partial or overwritten report.
|
||||
- Verify report never constructs adapters/providers and the skill removes only the obsolete report capability gate.
|
||||
- Verify `git diff --check` passes after all predecessor and report changes are present.
|
||||
|
||||
## Verification Results
|
||||
|
||||
|
|
@ -184,6 +185,18 @@ Actual output:
|
|||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Epic-wide diff hygiene
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: exit 0 with no whitespace errors across the complete Epic implementation.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
|
|
@ -45,7 +45,7 @@ Immutable manifest, preflight, execution, measurement, web, and scoring records
|
|||
|
||||
- Approved SDD with released lock; `milestone-task=report-output`.
|
||||
- S14 requires success/failure/blocked/unscored/scoring_failed conditions, versions, time/token/quality, ties/limits, evaluator/scoring provenance, and raw evidence pointers.
|
||||
- Evidence Map S14 requires an all-status/tie golden. API-1 through API-3 and final verification cover this row.
|
||||
- Evidence Map S14 requires an all-status/tie golden. The SDD common closure also requires `git diff --check`; API-1 through API-3 and final verification cover both.
|
||||
|
||||
### Verification Context
|
||||
|
||||
|
|
@ -98,7 +98,7 @@ Immutable manifest, preflight, execution, measurement, web, and scoring records
|
|||
|
||||
- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests.
|
||||
- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification.
|
||||
- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Join evidence without inference
|
||||
|
|
@ -207,5 +207,7 @@ Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_be
|
|||
- Expected: exit 1 with no matches.
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
7. `git diff --check`
|
||||
- Expected: exit 0 with no whitespace errors across the complete Epic implementation.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,194 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/18+17_report_cli plan=0 tag=API milestone-task=report-output -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-11
|
||||
task=m-agent-comparison-benchmark-pipeline/18+17_report_cli, plan=0, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log.
|
||||
- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. This child is the parent API-3 strict subset and depends on completed report core child 17.
|
||||
- Verification carryover: CLI help/output, semantic skill mutation tests, full suite, example validation, and SDD common `git diff --check`.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G03.md` → `code_review_cloud_G03_0.log` and `PLAN-local-G03.md` → `plan_local_G03_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/18+17_report_cli/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve first-line `milestone-task=report-output` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 — Public report CLI/skill operation | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G03.md` to `code_review_cloud_G03_0.log`.
|
||||
- [ ] Archive active `PLAN-local-G03.md` to `plan_local_G03_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/18+17_report_cli/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task=report-output` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Require predecessor 17 PASS from the exact active or dated archive `complete.log`.
|
||||
- Verify `report` delegates to the strict reporter and never constructs caller adapters or providers.
|
||||
- Verify success/error output and exit codes are closed, and invalid state leaves no partial report.
|
||||
- Verify skill/rule changes remove only obsolete report-unavailable behavior and retain all safety boundaries.
|
||||
- Verify `git diff --check` passes after all predecessor and report changes are present.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Intermediate verification command
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`
|
||||
|
||||
Expected: exit 0; CLI/skill semantics and no-provider behavior pass.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
Expected: exit 0 with fresh complete suite output.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Report CLI help
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
|
||||
Expected: exit 0 and exactly `--manifest`, `--run-id` are documented.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Public command set
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
|
||||
Expected: validate/preflight/run/resume/status/score/report are present.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Obsolete capability scan
|
||||
|
||||
Command: `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
|
||||
Expected: exit 1 with no matches.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Example manifest validation
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
|
||||
Expected: exit 0 and `ok: manifest is valid`.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
### Epic-wide diff hygiene
|
||||
|
||||
Command: `git diff --check`
|
||||
|
||||
Expected: exit 0 with no whitespace errors across the complete Epic implementation.
|
||||
|
||||
Actual output:
|
||||
|
||||
```text
|
||||
<implementation agent records stdout/stderr here>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
<!-- task=m-agent-comparison-benchmark-pipeline/18+17_report_cli plan=0 tag=API milestone-task=report-output -->
|
||||
|
||||
# Expose deterministic report through CLI and benchmark skill
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling implementation-owned sections in `CODE_REVIEW-cloud-G03.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The strict report projection is intentionally completed before it becomes a public operation. This rollout child exposes the predecessor's deterministic `report.md` contract through the CLI and project benchmark skill, removes only the obsolete report-unavailable gate, and records the Epic-wide diff hygiene closure required by the approved SDD.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log.
|
||||
- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement.
|
||||
- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. This child is the parent API-3 strict subset and depends on completed report core child 17.
|
||||
- Verification carryover: CLI help/output, semantic skill mutation tests, full suite, example validation, and SDD common `git diff --check`.
|
||||
- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-ops/rules/project/rules.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log`
|
||||
- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`; lock released; `milestone-task=report-output`.
|
||||
- S14 requires a public Markdown report containing conditions, versions, time/token/quality, non-success states, evaluator provenance, and raw pointers without inference.
|
||||
- Evidence Map S14 requires report generation evidence. This child verifies the exact public command/skill delegation after child 17 proves report semantics, and owns the SDD common `git diff --check` closure across the complete Epic implementation.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No handoff was supplied. CLI and skill behavior are deterministic, standard-library-only, credential-free, and must never construct caller adapters or invoke a provider.
|
||||
- Verification consumes the completed child-17 report contract, exact help/output checks, semantic skill mutations, full benchmark discovery, example validation, and repository diff hygiene.
|
||||
- No external verification is required. Confidence is high because this child changes only the public delegation boundary and its exact contract tests.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- CLI has no `report` operation and benchmark skill still returns `capability-unavailable: report-output`.
|
||||
- No semantic test proves report never constructs adapters/providers or writes outside the run root.
|
||||
- The active Epic pair union has no explicit `git diff --check` evidence.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- Exact CLI command/option and capability assertions in `scripts/agent_benchmark/skill_contract_test.py` must add `report` and remove only obsolete report-unavailable assertions.
|
||||
- CLI parser/main are the only public dispatch points; the benchmark project rule and skill are the only user-facing report routing documents.
|
||||
- No symbol is renamed or removed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns one rollout contract: public `report --manifest --run-id` delegates to the completed strict reporter with closed output/error semantics and no provider path.
|
||||
- Dependency `18+17_report_cli` requires exactly one PASS `complete.log` for sibling index 17. It is currently pending.
|
||||
- Report projection/rendering remains exclusively in predecessor 17; this child does not duplicate or reinterpret it.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Exclude report join/rank/render/publish policy and golden construction; predecessor 17 owns those files and behavior.
|
||||
- Do not add execution/scoring retry, provider calls, new report values, or a second report implementation in the skill.
|
||||
- Update only the report command delegation and the smallest matching project routing/test surface.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`.
|
||||
- Build/review closures are all true. Scores `(scope=1,state=0,blast=1,evidence=0,verification=1)` produce G03.
|
||||
- Build `local-fit`, `worker/local/G03`, `PLAN-local-G03.md`; review `official-review`, `review/cloud/G03`, `CODE_REVIEW-cloud-G03.md`.
|
||||
- `large_indivisible_context=false`; loop risk `boundary_contract` (1). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve predecessor index 17 from `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/complete.log` if active, otherwise exactly one dated archive match under `agent-task/archive/*/*/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/complete.log`.
|
||||
2. Require PASS and consume its reporter API without changing projection or rendering semantics.
|
||||
3. Expose CLI/skill delegation, then run command, semantic, full-suite, manifest, and Epic-wide diff verification.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Make report a supported deterministic operation
|
||||
|
||||
**Problem**
|
||||
|
||||
`scripts/agent_comparison_benchmark.py:65-88` has no `report`, and `scripts/agent_benchmark/skill_contract_test.py:238-246` requires report capability unavailable.
|
||||
|
||||
**Solution**
|
||||
|
||||
Add `report --manifest --run-id`. Validate/open immutable state before writer acquisition, call the predecessor reporter, and print only `ok: report run_id=<id> path=<run-relative-path>`. Invalid/corrupt state prints `error: benchmark report is unavailable`, exits 69, and leaves no partial output. Never construct adapters or invoke providers.
|
||||
|
||||
Update the benchmark skill description/triggers/inputs/preflight/procedure/validation/output/safety/stops/prohibitions to delegate the exact CLI and output. Update project routing and exact semantic mutation tests; supported commands include predecessor `score` and new `report`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_comparison_benchmark.py`: report parser/handler/dispatch.
|
||||
- [ ] `scripts/agent_benchmark/reporting_test.py`: CLI error/success/idempotent/no-provider cases against the predecessor API.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: executable report delegation.
|
||||
- [ ] `agent-ops/rules/project/rules.md`: report routing.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: score+report commands/options/mutations.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Use a mocked reporting boundary for error ordering and the predecessor synthetic run for success/idempotency; patch adapter construction to fail if report touches it. Validate the full skill text and unsafe mutations independently.
|
||||
|
||||
**Verification**
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`; expect exit 0 and no report capability-unavailable string.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|------|-------|
|
||||
| `scripts/agent_comparison_benchmark.py` | API-1 |
|
||||
| `scripts/agent_benchmark/reporting_test.py` | API-1 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-1 |
|
||||
| `agent-ops/rules/project/rules.md` | API-1 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-1 |
|
||||
| `agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/CODE_REVIEW-cloud-G03.md` | API-1 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`
|
||||
- Expected: exit 0; CLI/skill semantics and no-provider behavior pass.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
- Expected: exit 0 with fresh complete suite output.
|
||||
3. `python3 scripts/agent_comparison_benchmark.py report --help`
|
||||
- Expected: exit 0 with exactly `--manifest`, `--run-id`.
|
||||
4. `python3 scripts/agent_comparison_benchmark.py --help`
|
||||
- Expected: public commands include validate/preflight/run/resume/status/score/report.
|
||||
5. `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py`
|
||||
- Expected: exit 1 with no matches.
|
||||
6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- Expected: exit 0 and `ok: manifest is valid`.
|
||||
7. `git diff --check`
|
||||
- Expected: exit 0 with no whitespace errors across the complete Epic implementation.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
Loading…
Reference in a new issue