diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md index d1e3432e..df63d5ca 100644 --- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md @@ -1,73 +1,35 @@ - + # Code Review Reference - REVIEW_TEST ## Overview date=2026-08-12 -task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=1, tag=REVIEW_TEST - -## Archive Evidence Snapshot - -- Prior logs: `plan_cloud_G09_0.log`, `code_review_cloud_G10_0.log`; verdict FAIL/R1. -- Old preflight-only id: `run-20260812T053517Z-6b4f4de67128`; never reuse. +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=2, tag=REVIEW_TEST ## Implementation Item Completion | Item | Status | |---|---| -| REVIEW_TEST-1 bootstrap test fix | [ ] | -| REVIEW_TEST-2 release qualification/deploy | [ ] | -| REVIEW_TEST-3 ready preflight | [ ] | -| REVIEW_TEST-4 scored run | [ ] | +| REVIEW_TEST-1 Claude parser | [ ] | +| REVIEW_TEST-2 agy settings | [ ] | +| REVIEW_TEST-3 OpenAI Chat bridge | [ ] | +| REVIEW_TEST-4 hybrid classification | [ ] | +| REVIEW_TEST-5 release deploy | [ ] | +| REVIEW_TEST-6 public smokes/preflight | [ ] | +| REVIEW_TEST-7 scored run/report | [ ] | -## Implementation Checklist +## Implementation Evidence -- [ ] [REVIEW_TEST-1] Apply R1 test-only lifecycle/timeout fix and pass high-count local/remote regressions. -- [ ] [REVIEW_TEST-2] Complete clean release tests, four-binary rebuild/redeploy, and 4-node healthy verification. -- [ ] [REVIEW_TEST-3] Produce a fresh public ready=9 preflight id. -- [ ] [REVIEW_TEST-4] Execute exactly one public scored run and record nine-cell terminal/workspace results. -- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. +_Fill with sanitized exact commands, identities, run ids, and terminal/web summaries._ ## Review-Only Checklist +- [ ] Verify no credential value, ambient caller config, direct provider call, manifest change, or hidden retry was introduced. +- [ ] Verify the immutable failed run is untouched and exactly one new scored run exists. +- [ ] Verify all R1-R4 regressions, clean deployment identities, ready=9, success=9, web gates, and Markdown report. - [ ] Append final verdict and routing signals. -- [ ] Freshly verify R1, release identity, ready=9, one run, and web gates. -- [ ] Archive/finalize only after all requirements pass. - -## Deviations from Plan - -_Record actual deviations or none._ - -## Key Design Decisions - -_Record actual decisions._ - -## Reviewer Checkpoints - -- Test cleanup never invokes a second Runtime Start. -- All fake register calls share the established handshake timeout. -- Deployed source is clean release HEAD; four Nodes are connected/healthy. -- Fresh preflight is ready=9; exactly one fresh scored run succeeds 9/9. - -## Verification Results - -### R1 high-count tests - -_Paste exact output._ - -### Release qualification and deployment - -_Paste exact output/identity summary._ - -### Public preflight - -_Paste exact output and id._ - -### Public scored run - -_Paste exact output, id, status, and artifact gate summary._ ## Section Ownership -Implementation status/checklist/evidence is implementation-owned. Review checklist/verdict is review-only. +Implementation status/evidence is implementation-owned. Review checklist/verdict is review-only. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md index 80c17360..ec31affe 100644 --- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md @@ -1,151 +1,56 @@ - + -# Bootstrap qualification fix and comparison rerun +# Caller/provider compatibility fixes and comparison rerun ## For the Implementing Agent -Implement the selected R1 fix exactly, run every verification, then resume the clean release deployment, ready=9 preflight, and exactly one scored run. Fill `CODE_REVIEW-cloud-G10.md`, keep active files in place, and do not create `complete.log` or archive files. +Resolve R1-R4 without changing benchmark inputs, credentials, route aliases, scoring, or the immutable failed run. Use only the public benchmark CLI for live execution. After local qualification, rebuild and deploy the clean release, run one non-scored hybrid smoke, require ready=9, and execute exactly one new scored run. ## Background -The first live-run plan correctly stopped twice: a preflight found the initial clean release lacked the feature, then the feature-bearing clean release exposed two failure modes in one bootstrap test. The reviewer closed both to stale test-only timeout/cleanup code; production runtime behavior is not changed. - -## Archive Evidence Snapshot - -- `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`: FAIL with R1 after no scored run. -- Preflight-only evidence `run-20260812T053517Z-6b4f4de67128`: all cells `model_missing` on pre-feature clean release; preserve, do not reuse. -- Release branch `release/dev-936` now contains clean feature merge `b244b6e224a9d44c09a1e7af49c0788f597c89fe`. +The clean release and ready=9 preflight succeeded. The first scored run `run-20260812T055534Z-402e1f1d5b30` completed nine cells but exposed official caller stream variants, missing isolated agy provider settings, an OpenAI Chat bridge field mismatch, and hybrid stage failures. The evidence is valid and must not be retried or rewritten. ## Finding Resolution Map -| ID | Evidence | Root Cause | Selected Fix | Mode | Changed precondition | Acceptance | -|---|---|---|---|---|---|---| -| R1 | remote full suite register timeout; package run duplicate-Start unexpectedly succeeded | obsolete 2s register bound plus cleanup duplicate Start cancels active lifetime | use `fakeNodeHandshakeTimeout` for all fake register calls and cleanup with one Stop | direct-fix | test no longer mutates lifecycle during cleanup and uses the established remote-safe bound | exact test x20, package x10, full suite, rebuild/deploy, ready=9, one scored run | +| ID | Root Cause | Selected Fix | Acceptance | +|---|---|---|---| +| R1 | Claude snapshots/result grammar is narrower than observed official output | fix active-id/result/usage parsing and replay tests | real-stream replays plus adapter suite pass | +| R2 | isolated agy HOME lacks required direct Gemini provider selector | stage exact secret-free per-session settings | invocation tests and official noninteractive smoke pass | +| R3 | Anthropic bridge uses generic `max_tokens` for OpenAI Chat; synthetic API errors become parser errors | profile-aware token field and Claude API-error classification | bridge/parser regressions plus direct GPT smoke pass | +| R4 | hybrid failures lack sufficient deterministic regression qualification | exact fail-closed classification tests and live non-scored route smokes; no hidden retry | Gemini/GPT hybrid smoke succeeds before scored run | -## Analysis +## Scope -### Files Read +- `scripts/agent_benchmark/claude_iop.py` and tests +- `scripts/agent_benchmark/agy_iop.py` and tests +- `apps/edge/internal/openai/anthropic_bridge.go` and tests +- existing single-request provider/plan/review tests only when needed for R4 classification proof +- active review and id pointer files -- `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` -- `apps/edge/internal/bootstrap/runtime.go` -- `apps/edge/internal/transport/server.go` -- `apps/edge/internal/transport/integration_test.go` -- `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log` -- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` - -### SDD Criteria - -SDD remains READY; milestone-task ids and S04-S08 are unchanged. The live evidence gate remains ready=9 followed by nine successful terminal/workspace results. - -### Verification Context - -Remote runner is `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, release `dev-936`. No scored run has started. Four Nodes were restored connected before the new release test failed. Token files remain process-local and values must not be printed. - -### Test Coverage Gaps - -R1 is covered by high-count exact/package runs. External endpoint and workspace output remain covered only by the resumed public preflight/run. - -### Symbol References - -None; no production or public symbol changes. - -### Split Judgment - -Keep the fix and resumed deployment in this follow-up because the fixed qualification is the ordered precondition for the same all-cell execution invariant. Child 07 remains satisfied by its archived `complete.log`; no new sibling dependency is introduced. - -### Scope Rationale - -Exclude production Runtime behavior, routes, model aliases, task content, scoring, and roadmap mutation. Only the broken test lifecycle/timeout, evidence pointer files, and active review may change. - -### Final Routing - -- evaluation_mode=`isolated-reassessment`; finalizer=`finalize-task-policy.sh pair` -- build closures all true; scores `2/2/2/2/2` => G10; route `grade-boundary`; `PLAN-cloud-G10.md` -- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; `CODE_REVIEW-cloud-G10.md` -- large_indivisible_context=false; risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=1; evidence_integrity_failure=false; capability gap absent. +Do not modify central common rules/skills, tokens, manifest/task/rubric, provider credentials, route aliases, or archived run evidence. ## Implementation Checklist -- [ ] [REVIEW_TEST-1] Apply R1 test-only lifecycle/timeout fix and pass high-count local/remote regressions. -- [ ] [REVIEW_TEST-2] Complete clean release tests, four-binary rebuild/redeploy, and 4-node healthy verification. -- [ ] [REVIEW_TEST-3] Produce a fresh public ready=9 preflight id. -- [ ] [REVIEW_TEST-4] Execute exactly one public scored run and record nine-cell terminal/workspace results. -- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. +- [ ] [REVIEW_TEST-1] Implement R1 Claude stream compatibility and replay regressions. +- [ ] [REVIEW_TEST-2] Implement R2 isolated agy provider settings and invocation regressions. +- [ ] [REVIEW_TEST-3] Implement R3 OpenAI Chat bridge mapping and synthetic API-error classification. +- [ ] [REVIEW_TEST-4] Add R4 single-request failure classification regressions without retries/reselection. +- [ ] [REVIEW_TEST-5] Run local full verification, commit/push, merge clean release, rebuild/deploy all runtime binaries, and verify four-node health. +- [ ] [REVIEW_TEST-6] Run fresh public non-scored direct/hybrid smokes and ready=9 preflight. +- [ ] [REVIEW_TEST-7] Execute exactly one new scored run, record the new id, and generate the human-readable Markdown report. +- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized evidence. -### [REVIEW_TEST-1] Close bootstrap test nondeterminism +## Verification -**Problem:** `runtime_refresh_node_test.go:517-524` restarts Runtime in cleanup, and register requests in the file retain a shorter timeout than the declared fake-node handshake contract. +1. Python benchmark adapter unit/integration suite passes. +2. Edge OpenAI package and dev-runtime full Go suites pass. +3. Clean release build identities match on Edge and all Nodes; Control Plane shows 4/4 connected and 8/8 healthy. +4. Direct GPT and both hybrid public smokes succeed, then public preflight reports ready=9. +5. Exactly one new scored run reports success=9 and all web gates pass; public report command writes Markdown. -**Solution:** Replace the duplicate Start cleanup with the standard Stop-only form used by neighboring tests. Replace all four fake register `2*time.Second` bounds with `fakeNodeHandshakeTimeout`; do not add retries or production changes. +## Final Routing -**Modified Files and Checklist:** - -- [ ] `apps/edge/internal/bootstrap/runtime_refresh_node_test.go`: exact R1 test-only changes. - -**Test Strategy:** Fresh high-count exact and package tests prove both observed modes are absent. - -**Verification:** `go test -count=20 -run '^TestRefreshConfigApplyNoChangeSkipsNodePush$' ./apps/edge/internal/bootstrap` and `go test -count=10 ./apps/edge/internal/bootstrap` exit 0. - -### [REVIEW_TEST-2] Release qualification and deployment - -**Problem:** The current deployed d40e release is clean but lacks the benchmark feature; b244 source has not passed qualification or been deployed. - -**Solution:** Commit/push R1, merge it into `release/dev-936`, sync the runner, run full pre/post-build suites, rebuild four binaries, config check/dry-run, deploy Edge and all four Nodes, then require connected/healthy state. - -**Modified Files and Checklist:** - -- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: exact release/deployment evidence. - -**Test Strategy:** Apply dev-runtime deploy contract; any failed required stage stops scored execution. - -**Verification:** clean source/build identity matches release HEAD and CP reports 4 connected Nodes with healthy providers. - -### [REVIEW_TEST-3] Fresh public preflight - -**Problem:** The old preflight belongs to the pre-feature release. - -**Solution:** With protected environment values, call public CLI preflight and store only emitted id. - -**Modified Files and Checklist:** - -- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact id. - -**Test Strategy:** Require `ready=9 registration_required=0 implementation_gap=0`. - -**Verification:** public preflight exits 0. - -### [REVIEW_TEST-4] One scored run - -**Problem:** No valid generated benchmark output exists. - -**Solution:** Invoke public run exactly once after ready=9, store id, and inspect immutable evidence. - -**Modified Files and Checklist:** - -- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact id. - -**Test Strategy:** Require 9 success and all web artifact gates PASS; do not retry on failure. - -**Verification:** public run exits 0 with required counts. - -## Dependencies and Execution Order - -Child 07 PASS remains satisfied. Execute REVIEW_TEST-1 through REVIEW_TEST-4 strictly in order. - -## Modified Files Summary - -| File | Item | -|---|---| -| `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` | REVIEW_TEST-1 | -| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | REVIEW_TEST-2, REVIEW_TEST-3, REVIEW_TEST-4 | -| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | REVIEW_TEST-3 | -| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | REVIEW_TEST-4 | - -## Final Verification - -1. High-count commands in REVIEW_TEST-1 exit 0. -2. Full dev-runtime pre/post suites and build identity checks pass. -3. Public preflight exits 0 with ready=9. -4. Exactly one public run exits 0 with 9 success and all artifact gates. - -After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. +- finalizer=`finalize-task-policy.sh pair` +- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10` +- review_rework_count=2; evidence_integrity_failure=false +- risks=`temporal_state,boundary_contract,variant_product,stream_protocol` diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log new file mode 100644 index 00000000..25dd7c0a --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log @@ -0,0 +1,126 @@ + + +# Code Review Reference - REVIEW_TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=1, tag=REVIEW_TEST + +## Implementation Item Completion + +| Item | Status | +|---|---| +| REVIEW_TEST-1 bootstrap test fix | [x] | +| REVIEW_TEST-2 release qualification/deploy | [x] | +| REVIEW_TEST-3 ready preflight | [x] | +| REVIEW_TEST-4 scored run | [x] | + +## Implementation Checklist + +- [x] [REVIEW_TEST-1] Test-only lifecycle/timeout fix and high-count regressions passed. +- [x] [REVIEW_TEST-2] Clean release tests, four-binary rebuild/deploy, and four-node health passed. +- [x] [REVIEW_TEST-3] Fresh public preflight produced ready=9. +- [x] [REVIEW_TEST-4] Exactly one scored run was executed and its failed immutable evidence was recorded. +- [x] Implementation evidence was filled without exposing credentials. + +## Review-Only Checklist + +- [x] Final verdict and routing signals appended. +- [x] R1, release identity, ready=9, exactly-one run, and cell gates freshly checked. +- [x] No scored retry was performed. + +## Deviations from Plan + +The bootstrap defect was closed as planned, but the first valid scored run exposed four independent runtime compatibility defects. The run remains immutable and is not retried in this packet. + +## Key Design Decisions + +Preserve the ready preflight and failed scored run as fixed evidence. Classify caller stream variants separately from upstream/provider failures, stage isolated agy settings in the already isolated session HOME, and repair the Anthropic-to-OpenAI Chat field mapping rather than changing credentials or bypassing IOP. + +## Verification Results + +### R1 high-count tests + +```text +exact bootstrap test x20: PASS +bootstrap package x10: PASS +local/remote full dev-runtime suites: PASS +repository Python/Go verification: 433 tests PASS +``` + +### Release qualification and deployment + +```text +release branch: release/dev-936 +release HEAD: cdf91830480ae158703970cfb56c535f1519323b +Edge/mac Node source identity: cdf91830 +GX10/OneXPlayer/RTX5090 rebuilt binaries: deployed +Control Plane: nodes=4 connected=4 providers=8 available_healthy=8 in_flight=0 queued=0 +``` + +### Public preflight + +```text +run_id=run-20260812T055502Z-e359d9d350a8 +ready=9 registration_required=0 implementation_gap=0 +``` + +### Public scored run + +```text +run_id=run-20260812T055534Z-402e1f1d5b30 +completed=9 unresolved=7 success=2 failed=7 timed_out=0 cancelled=0 interrupted=0 running=0 +c01 claude/sonnet direct: parser_error, exit 143 +c02 claude/gemini direct: parser_error, exit 0 +c03 agy/gemini direct: malformed_event, signal 15 +c04 claude/gpt direct: parser_error, exit 1 +c05 codex/gpt direct: lifecycle success, web failed +c06 claude/gemini hybrid: parser_error, exit 1 +c07 agy/gemini hybrid: malformed_event, signal 15 +c08 claude/gpt hybrid: parser_error, exit 1 +c09 codex/gpt hybrid: success, web passed +``` + +## Code Review Result + +- Verdict: FAIL +- Required: 4 +- Suggested: 0 +- Nit: 0 +- Dimension Assessment: + - Correctness: Fail — seven cells do not reach accepted terminal/workspace evidence. + - Completeness: Fail — the required 9/9 benchmark output is absent. + - Test coverage: Fail — real Claude stream variants, isolated agy provider settings, OpenAI Chat max-token mapping, and single-request live failure paths are not covered together. + - API contract: Pass — all provider traffic used the public benchmark/IOP surfaces and the failed run was not retried. + - Code quality: Warn — adapters fail closed, but their closed grammars omit observed official variants. + - Plan deviation: Pass — the scored failure was preserved instead of hidden. + - Verification trust: Pass — identities, preflight, run id, terminal counts, and isolated caller logs agree. +- Findings: + - R1 Required + - Evidence: Claude Sonnet reuses one assistant message id across partial/tool-result cycles; the parser marks the partial id complete on a user event and rejects its next snapshot. Claude Gemini can emit a successful result immediately after a final `stop_reason=null` snapshot, and current terminal usage contains additional documented bookkeeping fields. + - Root Cause: the Claude parser models snapshots as distinct completed messages and requires an `end_turn` assistant event before every result. + - Selected Fix: retain an active message id across tool-result cycles, admit direct successful result from an active final snapshot, allowlist current numeric usage bookkeeping while emitting only benchmark metrics, and add replay regressions. + - Disposition: direct-fix. + - R2 Required + - Evidence: both agy session logs report no isolated `settings.json`, no `modelProvider`, unauthenticated silent auth, then browser OAuth timeout despite process-local Gemini API credentials. + - Root Cause: setting HOME to the isolated session also removes the provider selector required for direct API-key mode. + - Selected Fix: create a secret-free per-attempt `~/.gemini/antigravity-cli/settings.json` with `modelProvider=gemini`, noninteractive tool permission, and telemetry disabled; never copy ambient settings. + - Disposition: direct-fix. + - R3 Required + - Evidence: Claude GPT direct receives an API-error synthetic assistant/result while Codex Responses succeeds on the same IOP GPT route. The Anthropic Chat bridge always sends `max_tokens` to the OpenAI Chat profile. + - Root Cause: the bridge does not apply the OpenAI Chat completion-token field contract and the Claude parser reports the resulting synthetic API error as a parser/model mismatch. + - Selected Fix: use `max_completion_tokens` for the built-in OpenAI Chat profile, retain `max_tokens` for compatible profiles, and classify synthetic Claude API-error terminals as process/upstream failure rather than parser corruption. + - Disposition: direct-fix. + - R4 Required + - Evidence: Gemini hybrid reached Plan and Work, then Review ended as provider failure; GPT hybrid ended in Plan validation while the later GPT hybrid cell succeeded. No retry is allowed for the immutable benchmark run. + - Root Cause: the live single-request path has no diagnostic distinction beyond its closed error class at the caller boundary, and transient provider/dispatch failures terminate the only request attempt. + - Selected Fix: add exact regression coverage for frozen-dispatch mismatch and provider-stage terminal classification, preserve fail-closed no-reselection semantics, then qualify both hybrid routes with a fresh non-scored public smoke before the next scored run. + - Disposition: verification-fix; do not add hidden provider retries. +- Routing Signals: review_rework_count=2, evidence_integrity_failure=false. +- Finalizer: grade-boundary/cloud/G10 + official-review/cloud/G10; risks=temporal_state,boundary_contract,variant_product,stream_protocol. +- Next Step: materialize the follow-up G10 pair and continue automatically. + +## Section Ownership + +Implementation status/checklist/evidence is implementation-owned. Review checklist and verdict are review-only. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log new file mode 100644 index 00000000..235c9330 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log @@ -0,0 +1,151 @@ + + +# Bootstrap qualification fix and comparison rerun + +## For the Implementing Agent + +Implement the selected R1 fix exactly, run every verification, then resume the clean release deployment, ready=9 preflight, and exactly one scored run. Fill `CODE_REVIEW-cloud-G10.md`, keep active files in place, and do not create `complete.log` or archive files. + +## Background + +The first live-run plan correctly stopped twice: a preflight found the initial clean release lacked the feature, then the feature-bearing clean release exposed two failure modes in one bootstrap test. The reviewer closed both to stale test-only timeout/cleanup code; production runtime behavior is not changed. + +## Archive Evidence Snapshot + +- `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`: FAIL with R1 after no scored run. +- Preflight-only evidence `run-20260812T053517Z-6b4f4de67128`: all cells `model_missing` on pre-feature clean release; preserve, do not reuse. +- Release branch `release/dev-936` now contains clean feature merge `b244b6e224a9d44c09a1e7af49c0788f597c89fe`. + +## Finding Resolution Map + +| ID | Evidence | Root Cause | Selected Fix | Mode | Changed precondition | Acceptance | +|---|---|---|---|---|---|---| +| R1 | remote full suite register timeout; package run duplicate-Start unexpectedly succeeded | obsolete 2s register bound plus cleanup duplicate Start cancels active lifetime | use `fakeNodeHandshakeTimeout` for all fake register calls and cleanup with one Stop | direct-fix | test no longer mutates lifecycle during cleanup and uses the established remote-safe bound | exact test x20, package x10, full suite, rebuild/deploy, ready=9, one scored run | + +## Analysis + +### Files Read + +- `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` +- `apps/edge/internal/bootstrap/runtime.go` +- `apps/edge/internal/transport/server.go` +- `apps/edge/internal/transport/integration_test.go` +- `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` + +### SDD Criteria + +SDD remains READY; milestone-task ids and S04-S08 are unchanged. The live evidence gate remains ready=9 followed by nine successful terminal/workspace results. + +### Verification Context + +Remote runner is `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, release `dev-936`. No scored run has started. Four Nodes were restored connected before the new release test failed. Token files remain process-local and values must not be printed. + +### Test Coverage Gaps + +R1 is covered by high-count exact/package runs. External endpoint and workspace output remain covered only by the resumed public preflight/run. + +### Symbol References + +None; no production or public symbol changes. + +### Split Judgment + +Keep the fix and resumed deployment in this follow-up because the fixed qualification is the ordered precondition for the same all-cell execution invariant. Child 07 remains satisfied by its archived `complete.log`; no new sibling dependency is introduced. + +### Scope Rationale + +Exclude production Runtime behavior, routes, model aliases, task content, scoring, and roadmap mutation. Only the broken test lifecycle/timeout, evidence pointer files, and active review may change. + +### Final Routing + +- evaluation_mode=`isolated-reassessment`; finalizer=`finalize-task-policy.sh pair` +- build closures all true; scores `2/2/2/2/2` => G10; route `grade-boundary`; `PLAN-cloud-G10.md` +- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; `CODE_REVIEW-cloud-G10.md` +- large_indivisible_context=false; risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=1; evidence_integrity_failure=false; capability gap absent. + +## Implementation Checklist + +- [x] [REVIEW_TEST-1] Apply R1 test-only lifecycle/timeout fix and pass high-count local/remote regressions. +- [x] [REVIEW_TEST-2] Complete clean release tests, four-binary rebuild/redeploy, and 4-node healthy verification. +- [x] [REVIEW_TEST-3] Produce a fresh public ready=9 preflight id. +- [x] [REVIEW_TEST-4] Execute exactly one public scored run and record nine-cell terminal/workspace results. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [REVIEW_TEST-1] Close bootstrap test nondeterminism + +**Problem:** `runtime_refresh_node_test.go:517-524` restarts Runtime in cleanup, and register requests in the file retain a shorter timeout than the declared fake-node handshake contract. + +**Solution:** Replace the duplicate Start cleanup with the standard Stop-only form used by neighboring tests. Replace all four fake register `2*time.Second` bounds with `fakeNodeHandshakeTimeout`; do not add retries or production changes. + +**Modified Files and Checklist:** + +- [ ] `apps/edge/internal/bootstrap/runtime_refresh_node_test.go`: exact R1 test-only changes. + +**Test Strategy:** Fresh high-count exact and package tests prove both observed modes are absent. + +**Verification:** `go test -count=20 -run '^TestRefreshConfigApplyNoChangeSkipsNodePush$' ./apps/edge/internal/bootstrap` and `go test -count=10 ./apps/edge/internal/bootstrap` exit 0. + +### [REVIEW_TEST-2] Release qualification and deployment + +**Problem:** The current deployed d40e release is clean but lacks the benchmark feature; b244 source has not passed qualification or been deployed. + +**Solution:** Commit/push R1, merge it into `release/dev-936`, sync the runner, run full pre/post-build suites, rebuild four binaries, config check/dry-run, deploy Edge and all four Nodes, then require connected/healthy state. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: exact release/deployment evidence. + +**Test Strategy:** Apply dev-runtime deploy contract; any failed required stage stops scored execution. + +**Verification:** clean source/build identity matches release HEAD and CP reports 4 connected Nodes with healthy providers. + +### [REVIEW_TEST-3] Fresh public preflight + +**Problem:** The old preflight belongs to the pre-feature release. + +**Solution:** With protected environment values, call public CLI preflight and store only emitted id. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact id. + +**Test Strategy:** Require `ready=9 registration_required=0 implementation_gap=0`. + +**Verification:** public preflight exits 0. + +### [REVIEW_TEST-4] One scored run + +**Problem:** No valid generated benchmark output exists. + +**Solution:** Invoke public run exactly once after ready=9, store id, and inspect immutable evidence. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact id. + +**Test Strategy:** Require 9 success and all web artifact gates PASS; do not retry on failure. + +**Verification:** public run exits 0 with required counts. + +## Dependencies and Execution Order + +Child 07 PASS remains satisfied. Execute REVIEW_TEST-1 through REVIEW_TEST-4 strictly in order. + +## Modified Files Summary + +| File | Item | +|---|---| +| `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` | REVIEW_TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | REVIEW_TEST-2, REVIEW_TEST-3, REVIEW_TEST-4 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | REVIEW_TEST-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | REVIEW_TEST-4 | + +## Final Verification + +1. High-count commands in REVIEW_TEST-1 exit 0. +2. Full dev-runtime pre/post suites and build identity checks pass. +3. Public preflight exits 0 with ready=9. +4. Exactly one public run exits 0 with 9 success and all artifact gates. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log new file mode 100644 index 00000000..1aaf34dc --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log @@ -0,0 +1 @@ +run-20260812T055502Z-e359d9d350a8 diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log new file mode 100644 index 00000000..c991ad74 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log @@ -0,0 +1 @@ +run-20260812T055534Z-402e1f1d5b30 diff --git a/apps/edge/internal/openai/anthropic_bridge.go b/apps/edge/internal/openai/anthropic_bridge.go index a79f341b..75be6955 100644 --- a/apps/edge/internal/openai/anthropic_bridge.go +++ b/apps/edge/internal/openai/anthropic_bridge.go @@ -96,11 +96,18 @@ func prepareAnthropicChatBridge(body []byte, target string, profile config.Concr } chat := map[string]any{ - "model": target, - "messages": messages, - "max_tokens": *req.MaxTokens, - "stream": req.Stream, + "model": target, + "messages": messages, + "stream": req.Stream, } + maxTokensField := "max_tokens" + if profile.ID == "openai" { + // OpenAI's current Chat completion models use the completion-specific + // field. Other OpenAI-compatible profiles retain their native legacy + // spelling instead of inheriting an OpenAI-only request contract. + maxTokensField = "max_completion_tokens" + } + chat[maxTokensField] = *req.MaxTokens if req.Stream { chat["stream_options"] = map[string]any{"include_usage": true} } diff --git a/apps/edge/internal/openai/anthropic_bridge_test.go b/apps/edge/internal/openai/anthropic_bridge_test.go index bd4ac7da..4b2777e0 100644 --- a/apps/edge/internal/openai/anthropic_bridge_test.go +++ b/apps/edge/internal/openai/anthropic_bridge_test.go @@ -59,7 +59,7 @@ func TestAnthropicChatBridgeMixedContentToolsAndResponse(t *testing.T) { if err := json.Unmarshal(bodies[0], &chat); err != nil { t.Fatal(err) } - if chat["model"] != "served-chat" || chat["max_tokens"] != float64(256) || chat["stream"] != false { + if chat["model"] != "served-chat" || chat["max_completion_tokens"] != float64(256) || chat["max_tokens"] != nil || chat["stream"] != false { t.Fatalf("Chat request envelope mismatch: %+v", chat) } messages := anthropicAnySlice(t, chat["messages"]) @@ -98,6 +98,27 @@ func TestAnthropicChatBridgeMixedContentToolsAndResponse(t *testing.T) { } } +func TestAnthropicChatBridgeKeepsCompatibleMaxTokensField(t *testing.T) { + profile, err := config.ResolveProtocolProfile("gemini", "", config.BuiltInProtocolProfileCatalog()) + if err != nil { + t.Fatal(err) + } + body, _, err := prepareAnthropicChatBridge( + []byte(`{"model":"claude-route","max_tokens":64,"messages":[{"role":"user","content":"hello"}]}`), + "served-chat", profile, + ) + if err != nil { + t.Fatal(err) + } + var chat map[string]any + if err := json.Unmarshal(body, &chat); err != nil { + t.Fatal(err) + } + if chat["max_tokens"] != float64(64) || chat["max_completion_tokens"] != nil { + t.Fatalf("compatible Chat max token field mismatch: %+v", chat) + } +} + func TestAnthropicChatBridgeThinkingCapabilityAndResponse(t *testing.T) { candidate := anthropicTestCandidate(t, "openai") candidate.ActualModel = "served-chat" diff --git a/apps/edge/internal/openai/single_request_provider_stage_test.go b/apps/edge/internal/openai/single_request_provider_stage_test.go index d52f7ff3..68d3b389 100644 --- a/apps/edge/internal/openai/single_request_provider_stage_test.go +++ b/apps/edge/internal/openai/single_request_provider_stage_test.go @@ -554,6 +554,10 @@ func TestSingleRequestProviderStageRejectsMismatchLimitAndContext(t *testing.T) if !errors.Is(err, errProviderStageGeneric) || !tunnel.closed { t.Fatalf("err=%v closed=%v", err, tunnel.closed) } + disposition, ok := singleRequestTerminalDisposition(err) + if !ok || disposition.Kind != edgeservice.SingleRequestTerminalError || disposition.ErrorClass != edgeservice.SingleRequestTerminalErrorValidation { + t.Fatalf("dispatch mismatch disposition=%+v ok=%v", disposition, ok) + } }) } @@ -592,6 +596,10 @@ func TestSingleRequestProviderStageRejectsMismatchLimitAndContext(t *testing.T) if !errors.Is(err, errProviderStageGeneric) { t.Fatalf("expected errProviderStageGeneric, got %v", err) } + disposition, ok := singleRequestTerminalDisposition(err) + if !ok || disposition.Kind != edgeservice.SingleRequestTerminalError || disposition.ErrorClass != edgeservice.SingleRequestTerminalErrorProvider { + t.Fatalf("provider failure disposition=%+v ok=%v", disposition, ok) + } }) t.Run("service-returns-nil-result", func(t *testing.T) { diff --git a/scripts/agent_benchmark/agy_iop.py b/scripts/agent_benchmark/agy_iop.py index 26b9f72c..3437d4f8 100644 --- a/scripts/agent_benchmark/agy_iop.py +++ b/scripts/agent_benchmark/agy_iop.py @@ -50,6 +50,12 @@ AGY_CALLER = "agy" AGY_KNOWN_VERSION = "1.1.12" AGY_ENDPOINT_ENV = "GOOGLE_GEMINI_BASE_URL" AGY_AUTH_ENV = "GEMINI_API_KEY" +AGY_SETTINGS_RELATIVE_PATH = Path(".gemini/antigravity-cli/settings.json") +_AGY_ISOLATED_SETTINGS = { + "enableTelemetry": False, + "modelProvider": "gemini", + "toolPermission": "always-proceed", +} AGY_MODEL_LABELS = { "gemini-3.6-flash": "Gemini 3.6 Flash", "gemini-hybrid": "Gemini 3.6 Flash", @@ -334,6 +340,8 @@ def build_agy_invocation( except UnicodeDecodeError as exc: raise AgyAdapterError("agy task payload must be UTF-8") from exc + _stage_agy_provider_settings(prepared.session_dir) + # The child receives a minimal environment and explicit IOP-only provider # settings. No parent agy/Gemini config or session variable is inherited. environment = { @@ -365,6 +373,29 @@ def build_agy_invocation( ) +def _stage_agy_provider_settings(session_dir: str) -> None: + """Create the exact secret-free provider selector in one isolated HOME.""" + try: + session = Path(session_dir).resolve(strict=True) + if not session.is_dir(): + raise OSError("session is not a directory") + settings = session / AGY_SETTINGS_RELATIVE_PATH + settings.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + if settings.parent.resolve(strict=True) != session / AGY_SETTINGS_RELATIVE_PATH.parent: + raise OSError("settings directory escapes session") + payload = ( + json.dumps(_AGY_ISOLATED_SETTINGS, sort_keys=True, separators=(",", ":")) + "\n" + ).encode("utf-8") + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(settings, flags, 0o600) + with os.fdopen(descriptor, "wb") as handle: + handle.write(payload) + except (OSError, ValueError) as exc: + raise AgyAdapterError("agy isolated provider settings are unavailable") from exc + + def _safe_identifier(value: Any) -> str | None: return value if isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:+-]{0,63}", value) else None diff --git a/scripts/agent_benchmark/agy_iop_test.py b/scripts/agent_benchmark/agy_iop_test.py index 6d7c7bab..ac5eb854 100644 --- a/scripts/agent_benchmark/agy_iop_test.py +++ b/scripts/agent_benchmark/agy_iop_test.py @@ -15,6 +15,7 @@ from scripts.agent_benchmark.agy_iop import ( AGY_AUTH_ENV, AGY_ENDPOINT_ENV, AGY_KNOWN_VERSION, + AGY_SETTINGS_RELATIVE_PATH, AgyAdapterError, AgyEventParser, AgyRuntimeInputs, @@ -136,6 +137,25 @@ class AgyIopTest(unittest.TestCase): self.assertEqual(environment["SSL_CERT_FILE"], "/operator/dev-ca.pem") self.assertEqual(environment["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem") self.assertEqual(spec.env_allowlist, (AGY_ENDPOINT_ENV, AGY_AUTH_ENV, "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS")) + settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH + self.assertEqual(json.loads(settings_path.read_text(encoding="utf-8")), { + "enableTelemetry": False, + "modelProvider": "gemini", + "toolPermission": "always-proceed", + }) + self.assertEqual(settings_path.stat().st_mode & 0o777, 0o600) + settings_text = settings_path.read_text(encoding="utf-8") + self.assertNotIn(self.runtime.endpoint, settings_text) + self.assertNotIn(self.runtime.credential, settings_text) + + def test_build_rejects_preexisting_isolated_settings(self) -> None: + settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH + settings_path.parent.mkdir(parents=True) + settings_path.write_text('{"modelProvider":"other"}\n', encoding="utf-8") + with self.assertRaisesRegex(AgyAdapterError, "isolated provider settings"): + build_agy_invocation( + _cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), self._preflight() + ) def test_build_rejects_missing_prepared_workspace_or_session(self) -> None: prepared = self._prepared() diff --git a/scripts/agent_benchmark/claude_iop.py b/scripts/agent_benchmark/claude_iop.py index 37293bc1..cc5cf057 100644 --- a/scripts/agent_benchmark/claude_iop.py +++ b/scripts/agent_benchmark/claude_iop.py @@ -64,6 +64,15 @@ _CLAUDE_USAGE_FIELDS = { "cache_read_input_tokens": "cached_input_tokens", "cache_creation_input_tokens": "cache_write_tokens", } +_CLAUDE_USAGE_BOOKKEEPING_FIELDS = { + "cache_creation", "inference_geo", "iterations", "output_tokens_details", + "server_tool_use", "service_tier", "speed", +} +_CLAUDE_USAGE_NESTED_COUNTS = { + "cache_creation": {"ephemeral_1h_input_tokens", "ephemeral_5m_input_tokens"}, + "output_tokens_details": {"thinking_tokens"}, + "server_tool_use": {"web_fetch_requests", "web_search_requests"}, +} class ClaudeIopError(Exception): @@ -225,6 +234,8 @@ class ClaudeStreamParser: self._assistant_messages = 0 self._active_message_id: str | None = None self._completed_message_ids: set[str] = set() + self._continuation_pending = False + self._api_error_seen = False def _require_bound_session(self, event: dict[str, Any]) -> None: if self.claude_session_id is None: @@ -247,6 +258,13 @@ class ClaudeStreamParser: message = event.get("message") if not isinstance(message, dict): raise ClaudeIopProtocolError("invalid Claude assistant event") + if event.get("is_api_error_message") is True: + if message.get("model") != "" or message.get("stop_reason") != "stop_sequence": + raise ClaudeIopProtocolError("invalid Claude API error event") + self._complete_active_message() + self._api_error_seen = True + self._phase = "await_error_result" + return None if _required_string(message, "model") != self.cell.iop.request_model: raise ClaudeIopProtocolError("Claude model binding mismatch") stop_reason = message.get("stop_reason") @@ -255,6 +273,10 @@ class ClaudeStreamParser: message_id = message.get("id") if message_id is not None and (not isinstance(message_id, str) or not message_id): raise ClaudeIopProtocolError("invalid Claude assistant message id") + if self._continuation_pending: + if message_id != self._active_message_id: + self._complete_active_message() + self._continuation_pending = False if stop_reason is None: if message_id is None: raise ClaudeIopProtocolError("unbound Claude assistant snapshot") @@ -270,14 +292,14 @@ class ClaudeStreamParser: # admissible only for the single final assistant event. if stop_reason != "end_turn" or self._active_message_id is not None: raise ClaudeIopProtocolError("unbound Claude assistant terminal") + self._assistant_messages += 1 else: if message_id in self._completed_message_ids: raise ClaudeIopProtocolError("duplicate Claude assistant message") if self._active_message_id not in (None, message_id): raise ClaudeIopProtocolError("overlapping Claude assistant messages") - self._completed_message_ids.add(message_id) - self._active_message_id = None - self._assistant_messages += 1 + self._active_message_id = message_id + self._complete_active_message() if stop_reason == "tool_use": self._phase = "await_tool_result" return None @@ -289,20 +311,48 @@ class ClaudeStreamParser: raise ClaudeIopProtocolError("out-of-order Claude user event") self._require_bound_session(event) if self._active_message_id is not None: - if self._active_message_id in self._completed_message_ids: - raise ClaudeIopProtocolError("duplicate Claude assistant message") - self._completed_message_ids.add(self._active_message_id) - self._active_message_id = None - self._assistant_messages += 1 + # Claude Code may publish cumulative snapshots with the same + # assistant message id on both sides of one or more tool results. + # The next assistant snapshot (or the result terminal) determines + # whether this message continues or a new model call begins. + self._continuation_pending = True elif self._phase != "await_tool_result": raise ClaudeIopProtocolError("unexpected Claude user event") self._phase = "await_assistant" + def _complete_active_message(self) -> None: + if self._active_message_id is None: + return + if self._active_message_id in self._completed_message_ids: + raise ClaudeIopProtocolError("duplicate Claude assistant message") + self._completed_message_ids.add(self._active_message_id) + self._active_message_id = None + self._continuation_pending = False + self._assistant_messages += 1 + def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...]: - if self._phase != "await_result": + if self._phase == "await_error_result": + self._require_bound_session(event) + if ( + not self._api_error_seen + or event.get("subtype") != "success" + or event.get("is_error") is not True + or event.get("terminal_reason") != "api_error" + ): + raise ClaudeIopProtocolError("invalid Claude API error terminal") + self._phase = "complete" + # Do not emit finish/idle for an upstream API failure. The caller's + # non-zero exit is the lifecycle terminal; an unexpected zero exit + # still fails closed as missing terminal evidence. + return () + if self._phase == "await_assistant" and self._active_message_id is not None: + if self._continuation_pending: + raise ClaudeIopProtocolError("Claude result followed an unresolved tool result") + self._complete_active_message() + elif self._phase != "await_result": raise ClaudeIopProtocolError("duplicate or out-of-order Claude result") self._require_bound_session(event) - if event.get("subtype") != "success": + if event.get("subtype") != "success" or event.get("is_error") is True: raise ClaudeIopProtocolError("invalid Claude result terminal") self._phase = "complete" return (*self._observations(event), "idle") @@ -333,13 +383,33 @@ class ClaudeStreamParser: usage = event.get("usage") if usage is None: return [] - if not isinstance(usage, dict) or not set(usage) <= set(_CLAUDE_USAGE_FIELDS): + allowed = set(_CLAUDE_USAGE_FIELDS) | _CLAUDE_USAGE_BOOKKEEPING_FIELDS + if not isinstance(usage, dict) or not set(usage) <= allowed: raise ClaudeIopProtocolError("invalid Claude usage observation") + ClaudeStreamParser._validate_usage_bookkeeping(usage) return [ count_metric(_CLAUDE_USAGE_FIELDS[field], value, model=model) - for field, value in sorted(usage.items()) + for field, value in sorted(usage.items()) if field in _CLAUDE_USAGE_FIELDS ] + @staticmethod + def _validate_usage_bookkeeping(usage: dict[str, Any]) -> None: + for field, nested_fields in _CLAUDE_USAGE_NESTED_COUNTS.items(): + if field not in usage: + continue + value = usage[field] + if not isinstance(value, dict) or not set(value) <= nested_fields: + raise ClaudeIopProtocolError("invalid Claude usage observation") + if any(not is_reported_number(item) for item in value.values()): + raise ClaudeIopProtocolError("invalid Claude usage observation") + if "iterations" in usage and usage["iterations"] != []: + raise ClaudeIopProtocolError("invalid Claude usage observation") + for field in ("inference_geo", "service_tier", "speed"): + if field in usage and ( + not isinstance(usage[field], str) or len(usage[field]) > 64 + ): + raise ClaudeIopProtocolError("invalid Claude usage observation") + def __call__(self, stream: str, raw_line: str) -> str | tuple[Any, ...] | None: if stream != "stdout": return None diff --git a/scripts/agent_benchmark/claude_iop_test.py b/scripts/agent_benchmark/claude_iop_test.py index a5a26ba7..c0e9baa5 100644 --- a/scripts/agent_benchmark/claude_iop_test.py +++ b/scripts/agent_benchmark/claude_iop_test.py @@ -245,6 +245,55 @@ class ClaudeIopTest(unittest.TestCase): metrics = {metric.name: metric.value for metric in observations[:-1]} self.assertEqual(metrics["model_calls"], 2) + def test_parser_accepts_cumulative_message_ids_and_direct_result(self) -> None: + init, _, result = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + partial = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-cumulative", "model": "claude-sonnet", + "stop_reason": None, "content": []}, + }) + user = json.dumps({ + "type": "user", "session_id": "claude-session-fixture", + "message": {"content": []}, + }) + terminal = json.loads(result) + terminal["usage"].update({ + "cache_creation": {"ephemeral_1h_input_tokens": 0, "ephemeral_5m_input_tokens": 0}, + "inference_geo": "", "iterations": [], + "output_tokens_details": {"thinking_tokens": 0}, + "server_tool_use": {"web_fetch_requests": 0, "web_search_requests": 0}, + "service_tier": "standard", "speed": "standard", + }) + parsed = [ + parser("stdout", line) + for line in (init, partial, partial, user, partial, user, user, partial) + ] + self.assertEqual(parsed, [None] * len(parsed)) + observations = parser("stdout", json.dumps(terminal)) + self.assertEqual(observations[-1], "idle") + self.assertEqual( + {metric.name: metric.value for metric in observations[:-1]}["model_calls"], 1 + ) + + def test_parser_classifies_synthetic_api_error_without_parser_failure(self) -> None: + init, _, _ = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + synthetic = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "is_api_error_message": True, + "message": {"id": "synthetic", "model": "", + "stop_reason": "stop_sequence", "content": []}, + }) + terminal = json.dumps({ + "type": "result", "session_id": "claude-session-fixture", + "subtype": "success", "is_error": True, "terminal_reason": "api_error", + }) + self.assertEqual( + [parser("stdout", line) for line in (init, synthetic, terminal)], + [None, None, ()], + ) + def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None: init, _, result = self._fixture_lines() parser = ClaudeStreamParser(self.cell, "session-fixture")