diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G06.md b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G06.md new file mode 100644 index 00000000..79703ddf --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G06.md @@ -0,0 +1,154 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-11 +task=m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output, plan=2, tag=API + +## Archive Evidence Snapshot + +- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` +- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log. +- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement. +- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. Parent API-1/API-2 form this report-core child; parent API-3 moved to `18+17_report_cli`. +- Verification carryover: all-status/tie golden, corruption/containment/idempotency tests, full benchmark suite; no provider call. +- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_2.log` and `PLAN-cloud-G06.md` → `plan_cloud_G06_2.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve first-line `milestone-task=report-output` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 — Strict evidence join and rankings | [ ] | +| API-2 — Deterministic Markdown/golden | [ ] | + +## Implementation Checklist + +- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests. +- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`. +- [ ] Archive active `PLAN-cloud-G06.md` to `plan_cloud_G06_2.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task=report-output` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Require exact PASS evidence for predecessors 14/15/16 from active or dated archive task-protocol paths. +- Verify unavailable values retain reason/source and are never zero or omitted. +- Verify equal totals retain equal rank and identity/display order does not affect rank. +- Verify gates do not add points and unscored/scoring_failed do not enter averages/rankings. +- Verify raw links are run-relative/contained and corrupt input leaves no partial or overwritten report. +- Verify this child does not expose a public report command or modify benchmark skill/routing. + +## Verification Results + +### Intermediate verification commands + +Commands: + +```text +python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test +python3 -m unittest scripts.agent_benchmark.reporting_test +``` + +Expected: each exits 0; golden bytes, statuses, ties, containment, and non-mutation pass. + +Actual output: + +```text + +``` + +### Focused report-core suite + +Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test` + +Expected: exit 0; all statuses/ties/unavailable values, corruption, containment, and non-mutation cases pass. + +Actual output: + +```text + +``` + +### Full benchmark suite + +Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` + +Expected: exit 0 with fresh complete suite output. + +Actual output: + +```text + +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G06.md b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G06.md new file mode 100644 index 00000000..40c7b81d --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G06.md @@ -0,0 +1,161 @@ + + +# Strict benchmark report projection and Markdown artifact + +## For the Implementing Agent + +Filling implementation-owned sections in `CODE_REVIEW-cloud-G06.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`. + +## Background + +Immutable manifest, preflight, execution, measurement, web, and scoring records need one deterministic reader-facing projection with explicit missing/failure semantics and contained raw pointers. This child implements the strict join, rank/tie projection, Markdown renderer, and immutable report artifact required by SDD S14. Public CLI and skill exposure remain a dependent rollout child so the internal report contract can independently PASS first. + +## Archive Evidence Snapshot + +- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` +- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log. +- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement. +- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. Parent API-1/API-2 form this report-core child; parent API-3 moved to `18+17_report_cli`. +- Verification carryover: all-status/tie golden, corruption/containment/idempotency tests, full benchmark suite; no provider call. +- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` + +### SDD Criteria + +- Approved SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`; lock released; `milestone-task=report-output`. +- S14 requires success/failure/blocked/unscored/scoring_failed conditions, versions, time/token/quality, ties/limits, evaluator/scoring provenance, and raw evidence pointers. +- Evidence Map S14 requires an all-status/tie golden. API-1/API-2 and final verification cover the deterministic projection and artifact rows; the dependent CLI child owns public exposure and the SDD common `git diff --check` closure. + +### Verification Context + +- No handoff was supplied. Report authorities are immutable run/preflight/attempt records and strict predecessor 14-16 loaders; the renderer never parses arbitrary captures or recomputes absent provider values. +- Tests are standard-library-only, credential-free, and uncached. A synthetic run tree covers all states and report publication remains run-owned, POSIX-relative, regular, and contained. +- No external verification is required. Confidence is high because output is a deterministic projection with byte-for-byte golden evidence. + +### Test Coverage Gaps + +- No strict join exists across timing, web, scoring, evaluator, and preflight records. +- No tie ranking preserves equal totals without arbitrary rank tie-break. +- No all-status/unavailable golden, contained raw-link test, or idempotent/no-clobber publication exists. + +### Symbol References + +- No symbol is renamed or removed. + +### Split Judgment + +- Child 17 owns a stable contract: validated immutable evidence produces one deterministic `report.md`; corrupt required input produces no report. +- Child `18+17_report_cli` consumes that contract and exposes it without changing report semantics. +- Directory predecessors 14/15/16 remain exact runtime dependencies. All are pending; implementation waits for their active-or-dated-archive PASS `complete.log` evidence. + +### Scope Rationale + +- Exclude CLI parser/dispatch, benchmark skill/rule, and semantic routing tests; child 18 owns them. +- Exclude measurement, gate, and score policy changes. Use predecessor loaders and values verbatim. +- Do not retry execution/scoring, call providers, infer missing data, average unscored results, add gate points, or break equal ranks by identity/order. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`. +- Build/review closures are all true. Scores `(scope=2,state=1,blast=1,evidence=1,verification=1)` produce G06. +- Build base `local-fit`, route `risk-boundary`, `worker/cloud/G06`, `PLAN-cloud-G06.md`; review `official-review`, `review/cloud/G06`, `CODE_REVIEW-cloud-G06.md`. +- `large_indivisible_context=false`; loop risks `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (4). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve predecessor 14 from its exact active or dated archive `complete.log`; require PASS. +2. Resolve predecessor 15 from its exact active or dated archive `complete.log`; require PASS. +3. Resolve predecessor 16 from its exact active or dated archive `complete.log`; require PASS. +4. Current preparation state: all three are pending. Implement the strict projection and golden without exposing a public command. + +## Implementation Checklist + +- [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests. +- [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Join evidence without inference + +**Problem** + +`scripts/agent_benchmark/attempts.py:1334-1360` exposes aggregate state counts but not measurement, web, score, or evaluator provenance. Ad hoc JSON parsing would bypass predecessor schema validation. + +**Solution** + +Add `reporting.py`. Open the bound run through `RunStore`, enumerate stable manifest cell/repetition/attempt order, and call strict loaders for measurement, web, anonymization, score attempts, and unscored records. Project execution/preflight status, timing/token availability+source, web gates/screenshots, scoring status/categories/total, manifest evaluator binding, score id, and contained raw paths. + +Missing optional observations render the producer's explicit unavailable reason; required file/digest/schema contradictions raise `ReportError` and write nothing. Rank only scored attempts by descending total. Equal totals share a tie group and competition rank (`1,1,3`); display order never changes rank. Unscored/scoring_failed attempts have no rank and do not enter averages. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/reporting.py`: strict join/projection/rank/tie/contained pointers. +- [ ] `scripts/agent_benchmark/reporting_test.py`: every state, unavailable/source, ties/order, missing/corrupt inputs, non-mutation. + +**Test Strategy** + +Build a synthetic version-2 run with historical blocked then ready preflight, successful/failed attempts, observed/unavailable metrics, passed/failed/blocked web, scored/unscored/scoring_failed, and equal totals. Mutate each required input separately and assert no output. + +**Verification** + +Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test`; expect exit 0. + +### [API-2] Render one deterministic Markdown artifact + +**Problem** + +No human-readable artifact or stable raw-link contract exists. Omitting conditions or replacing unavailable with zero contradicts S14. + +**Solution** + +Render, in fixed order: run identity; methodology/immutable conditions; manifest/environment/fixture/rubric/evaluator versions; preflight; attempt outcomes; time/token values with unit/clock/source or unavailable reason; automatic web gates/screenshots; quality/rank; evaluator/scoring provenance; failures/unscored/scoring_failed; limitations; raw evidence index. + +Escape Markdown cells/links, use contained POSIX run-relative paths, and refuse symlinks/escapes. Publish UTF-8 LF `/report.md` without replacement. Repeated generation succeeds only when the existing regular file equals freshly rendered bytes; any differing byte fails closed. Add exact all-status/tie golden `scripts/fixtures/agent-comparison-benchmark-report.expected.md`. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/reporting.py`: renderer/idempotent publisher. +- [ ] `scripts/agent_benchmark/reporting_test.py`: full-byte golden, escaping, containment, collision/symlink, identical rerun. +- [ ] `scripts/fixtures/agent-comparison-benchmark-report.expected.md`: exact expected report. + +**Test Strategy** + +Compare complete bytes, perturb input order for stability, and mutate existing output to prove no overwrite. + +**Verification** + +Run `python3 -m unittest scripts.agent_benchmark.reporting_test`; expect exact golden equality. + +## Modified Files Summary + +| File | Items | +|------|-------| +| `scripts/agent_benchmark/reporting.py` | API-1, API-2 | +| `scripts/agent_benchmark/reporting_test.py` | API-1, API-2 | +| `scripts/fixtures/agent-comparison-benchmark-report.expected.md` | API-2 | +| `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G06.md` | API-1, API-2 | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.attempts_test` + - Expected: exit 0; all statuses/ties/unavailable values, corruption, containment, and non-mutation cases pass. +2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` + - Expected: exit 0 with fresh complete suite output. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log similarity index 96% rename from agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G07.md rename to agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log index 5dfd578f..e39de6e6 100644 --- a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/CODE_REVIEW-cloud-G07.md +++ b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log @@ -54,7 +54,7 @@ Review completion means the following steps are finished: - [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests. - [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden. -- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification. +- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification. - [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. ## Review-Only Checklist @@ -91,6 +91,7 @@ _Record key design decisions here._ - Verify gates do not add points and unscored/scoring_failed do not enter averages/rankings. - Verify raw links are run-relative/contained and corrupt input leaves no partial or overwritten report. - Verify report never constructs adapters/providers and the skill removes only the obsolete report capability gate. +- Verify `git diff --check` passes after all predecessor and report changes are present. ## Verification Results @@ -184,6 +185,18 @@ Actual output: ``` +### Epic-wide diff hygiene + +Command: `git diff --check` + +Expected: exit 0 with no whitespace errors across the complete Epic implementation. + +Actual output: + +```text + +``` + --- > **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G07.md b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log similarity index 97% rename from agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G07.md rename to agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log index 5695a475..219d67d0 100644 --- a/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/PLAN-cloud-G07.md +++ b/agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log @@ -45,7 +45,7 @@ Immutable manifest, preflight, execution, measurement, web, and scoring records - Approved SDD with released lock; `milestone-task=report-output`. - S14 requires success/failure/blocked/unscored/scoring_failed conditions, versions, time/token/quality, ties/limits, evaluator/scoring provenance, and raw evidence pointers. -- Evidence Map S14 requires an all-status/tie golden. API-1 through API-3 and final verification cover this row. +- Evidence Map S14 requires an all-status/tie golden. The SDD common closure also requires `git diff --check`; API-1 through API-3 and final verification cover both. ### Verification Context @@ -98,7 +98,7 @@ Immutable manifest, preflight, execution, measurement, web, and scoring records - [ ] [API-1] Implement strict evidence joining and deterministic status/metric/score/rank projections, preserving unavailable values, failures, evaluator/scoring provenance, and ties with corruption tests. - [ ] [API-2] Render and idempotently publish contained `report.md` with conditions, versions, tables, limitations, and relative raw evidence links; add a byte-for-byte all-status/tie golden. -- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full verification. +- [ ] [API-3] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification. - [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. ### [API-1] Join evidence without inference @@ -207,5 +207,7 @@ Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_be - Expected: exit 1 with no matches. 6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json` - Expected: exit 0 and `ok: manifest is valid`. +7. `git diff --check` + - Expected: exit 0 with no whitespace errors across the complete Epic implementation. After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/CODE_REVIEW-cloud-G03.md b/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/CODE_REVIEW-cloud-G03.md new file mode 100644 index 00000000..247413bb --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/CODE_REVIEW-cloud-G03.md @@ -0,0 +1,194 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-11 +task=m-agent-comparison-benchmark-pipeline/18+17_report_cli, plan=0, tag=API + +## Archive Evidence Snapshot + +- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` +- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log. +- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement. +- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. This child is the parent API-3 strict subset and depends on completed report core child 17. +- Verification carryover: CLI help/output, semantic skill mutation tests, full suite, example validation, and SDD common `git diff --check`. +- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G03.md` → `code_review_cloud_G03_0.log` and `PLAN-local-G03.md` → `plan_local_G03_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/18+17_report_cli/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve first-line `milestone-task=report-output` in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 — Public report CLI/skill operation | [ ] | + +## Implementation Checklist + +- [ ] [API-1] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G03.md` to `code_review_cloud_G03_0.log`. +- [ ] Archive active `PLAN-local-G03.md` to `plan_local_G03_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/` to `agent-task/archive/YYYY/MM/m-agent-comparison-benchmark-pipeline/18+17_report_cli/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task=report-output` for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-agent-comparison-benchmark-pipeline/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Require predecessor 17 PASS from the exact active or dated archive `complete.log`. +- Verify `report` delegates to the strict reporter and never constructs caller adapters or providers. +- Verify success/error output and exit codes are closed, and invalid state leaves no partial report. +- Verify skill/rule changes remove only obsolete report-unavailable behavior and retain all safety boundaries. +- Verify `git diff --check` passes after all predecessor and report changes are present. + +## Verification Results + +### Intermediate verification command + +Command: `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test` + +Expected: exit 0; CLI/skill semantics and no-provider behavior pass. + +Actual output: + +```text + +``` + +### Full benchmark suite + +Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` + +Expected: exit 0 with fresh complete suite output. + +Actual output: + +```text + +``` + +### Report CLI help + +Command: `python3 scripts/agent_comparison_benchmark.py report --help` + +Expected: exit 0 and exactly `--manifest`, `--run-id` are documented. + +Actual output: + +```text + +``` + +### Public command set + +Command: `python3 scripts/agent_comparison_benchmark.py --help` + +Expected: validate/preflight/run/resume/status/score/report are present. + +Actual output: + +```text + +``` + +### Obsolete capability scan + +Command: `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py` + +Expected: exit 1 with no matches. + +Actual output: + +```text + +``` + +### Example manifest validation + +Command: `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json` + +Expected: exit 0 and `ok: manifest is valid`. + +Actual output: + +```text + +``` + +### Epic-wide diff hygiene + +Command: `git diff --check` + +Expected: exit 0 with no whitespace errors across the complete Epic implementation. + +Actual output: + +```text + +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/PLAN-local-G03.md b/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/PLAN-local-G03.md new file mode 100644 index 00000000..01b46a8e --- /dev/null +++ b/agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/PLAN-local-G03.md @@ -0,0 +1,145 @@ + + +# Expose deterministic report through CLI and benchmark skill + +## For the Implementing Agent + +Filling implementation-owned sections in `CODE_REVIEW-cloud-G03.md` is mandatory. Run every verification command, paste actual notes/output, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only exact blocker evidence, attempted commands/output, and resume conditions in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify next state, archive logs, or write `complete.log`. + +## Background + +The strict report projection is intentionally completed before it becomes a public operation. This rollout child exposes the predecessor's deterministic `report.md` contract through the CLI and project benchmark skill, removes only the obsolete report-unavailable gate, and records the Epic-wide diff hygiene closure required by the approved SDD. + +## Archive Evidence Snapshot + +- Refined parent plan: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- Refined parent review stub: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` +- Earlier intent: `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_0.log` and matching review log. +- Verdict: 없음 — implementation 전 semantic replan 뒤 one-pass refinement. +- Findings: official Required 0, Suggested 0, Nit 0; no implementation file was changed. This child is the parent API-3 strict subset and depends on completed report core child 17. +- Verification carryover: CLI help/output, semantic skill mutation tests, full suite, example validation, and SDD common `git diff --check`. +- Roadmap carryover: `milestone-task=report-output`, SDD S14/Evidence Map S14. + +## Analysis + +### Files Read + +- `scripts/agent_comparison_benchmark.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-ops/rules/project/rules.md` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/plan_cloud_G07_1.log` +- `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/code_review_cloud_G07_1.log` + +### SDD Criteria + +- Approved SDD `agent-roadmap/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md`; lock released; `milestone-task=report-output`. +- S14 requires a public Markdown report containing conditions, versions, time/token/quality, non-success states, evaluator provenance, and raw pointers without inference. +- Evidence Map S14 requires report generation evidence. This child verifies the exact public command/skill delegation after child 17 proves report semantics, and owns the SDD common `git diff --check` closure across the complete Epic implementation. + +### Verification Context + +- No handoff was supplied. CLI and skill behavior are deterministic, standard-library-only, credential-free, and must never construct caller adapters or invoke a provider. +- Verification consumes the completed child-17 report contract, exact help/output checks, semantic skill mutations, full benchmark discovery, example validation, and repository diff hygiene. +- No external verification is required. Confidence is high because this child changes only the public delegation boundary and its exact contract tests. + +### Test Coverage Gaps + +- CLI has no `report` operation and benchmark skill still returns `capability-unavailable: report-output`. +- No semantic test proves report never constructs adapters/providers or writes outside the run root. +- The active Epic pair union has no explicit `git diff --check` evidence. + +### Symbol References + +- Exact CLI command/option and capability assertions in `scripts/agent_benchmark/skill_contract_test.py` must add `report` and remove only obsolete report-unavailable assertions. +- CLI parser/main are the only public dispatch points; the benchmark project rule and skill are the only user-facing report routing documents. +- No symbol is renamed or removed. + +### Split Judgment + +- This child owns one rollout contract: public `report --manifest --run-id` delegates to the completed strict reporter with closed output/error semantics and no provider path. +- Dependency `18+17_report_cli` requires exactly one PASS `complete.log` for sibling index 17. It is currently pending. +- Report projection/rendering remains exclusively in predecessor 17; this child does not duplicate or reinterpret it. + +### Scope Rationale + +- Exclude report join/rank/render/publish policy and golden construction; predecessor 17 owns those files and behavior. +- Do not add execution/scoring retry, provider calls, new report values, or a second report implementation in the skill. +- Update only the report command delegation and the smallest matching project routing/test surface. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; finalizer `finalize-task-policy.sh`, mode `pair`. +- Build/review closures are all true. Scores `(scope=1,state=0,blast=1,evidence=0,verification=1)` produce G03. +- Build `local-fit`, `worker/local/G03`, `PLAN-local-G03.md`; review `official-review`, `review/cloud/G03`, `CODE_REVIEW-cloud-G03.md`. +- `large_indivisible_context=false`; loop risk `boundary_contract` (1). `review_rework_count=0`, `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve predecessor index 17 from `agent-task/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/complete.log` if active, otherwise exactly one dated archive match under `agent-task/archive/*/*/m-agent-comparison-benchmark-pipeline/17+14,15,16_report_output/complete.log`. +2. Require PASS and consume its reporter API without changing projection or rendering semantics. +3. Expose CLI/skill delegation, then run command, semantic, full-suite, manifest, and Epic-wide diff verification. + +## Implementation Checklist + +- [ ] [API-1] Add `report --manifest --run-id`, replace the skill capability gate with exact CLI delegation/output, update project routing/semantic tests, and run focused/full/common verification. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Make report a supported deterministic operation + +**Problem** + +`scripts/agent_comparison_benchmark.py:65-88` has no `report`, and `scripts/agent_benchmark/skill_contract_test.py:238-246` requires report capability unavailable. + +**Solution** + +Add `report --manifest --run-id`. Validate/open immutable state before writer acquisition, call the predecessor reporter, and print only `ok: report run_id= path=`. Invalid/corrupt state prints `error: benchmark report is unavailable`, exits 69, and leaves no partial output. Never construct adapters or invoke providers. + +Update the benchmark skill description/triggers/inputs/preflight/procedure/validation/output/safety/stops/prohibitions to delegate the exact CLI and output. Update project routing and exact semantic mutation tests; supported commands include predecessor `score` and new `report`. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_comparison_benchmark.py`: report parser/handler/dispatch. +- [ ] `scripts/agent_benchmark/reporting_test.py`: CLI error/success/idempotent/no-provider cases against the predecessor API. +- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: executable report delegation. +- [ ] `agent-ops/rules/project/rules.md`: report routing. +- [ ] `scripts/agent_benchmark/skill_contract_test.py`: score+report commands/options/mutations. + +**Test Strategy** + +Use a mocked reporting boundary for error ordering and the predecessor synthetic run for success/idempotency; patch adapter construction to fail if report touches it. Validate the full skill text and unsafe mutations independently. + +**Verification** + +Run `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test`; expect exit 0 and no report capability-unavailable string. + +## Modified Files Summary + +| File | Items | +|------|-------| +| `scripts/agent_comparison_benchmark.py` | API-1 | +| `scripts/agent_benchmark/reporting_test.py` | API-1 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-1 | +| `agent-ops/rules/project/rules.md` | API-1 | +| `scripts/agent_benchmark/skill_contract_test.py` | API-1 | +| `agent-task/m-agent-comparison-benchmark-pipeline/18+17_report_cli/CODE_REVIEW-cloud-G03.md` | API-1 | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test` + - Expected: exit 0; CLI/skill semantics and no-provider behavior pass. +2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` + - Expected: exit 0 with fresh complete suite output. +3. `python3 scripts/agent_comparison_benchmark.py report --help` + - Expected: exit 0 with exactly `--manifest`, `--run-id`. +4. `python3 scripts/agent_comparison_benchmark.py --help` + - Expected: public commands include validate/preflight/run/resume/status/score/report. +5. `rg -n --sort path 'capability-unavailable: report-output' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md scripts/agent_benchmark/skill_contract_test.py` + - Expected: exit 1 with no matches. +6. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json` + - Expected: exit 0 and `ok: manifest is valid`. +7. `git diff --check` + - Expected: exit 0 with no whitespace errors across the complete Epic implementation. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.