diff --git a/README.md b/README.md index 239b6b43..876658f5 100644 --- a/README.md +++ b/README.md @@ -37,5 +37,6 @@ Start with `agent-contract/index.md` for protocol and runtime contracts, and `ag Operator and client setup guides: - [Edge Local Quickstart](docs/edge-local-dev-guide.md) +- [Agent Comparison Benchmark Dev Guide](docs/agent-comparison-benchmark-dev-guide.md) - [dev OpenCode Settings Guide](docs/dev-opencode-settings-guide.md) - [dev-corp Pi Settings Guide](docs/dev-corp-pi-settings-guide.md) diff --git a/agent-contract/index.md b/agent-contract/index.md index 4e46a2b5..f1fc51e3 100644 --- a/agent-contract/index.md +++ b/agent-contract/index.md @@ -14,6 +14,7 @@ |----|-----------|-----------|------| | `iop.openai-compatible-api` | OpenAI-compatible API, Responses API, Chat Completions, legacy Completions, error envelope/SSE terminal error, `model` route, managed projection principal auth and slot-route binding, managed-versus-legacy provider credential selection, model-driven passthrough/normalized routing, provider-pool admission/unavailable error, safe credential-slot attribution, standard metadata, and provider-native extension fields such as `chat_template_kwargs` | `apps/edge/internal/openai/*`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/service/provider_tunnel.go`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/openai-compatible-api.md` | | `iop.anthropic-compatible-api` | Anthropic Messages API, count_tokens, models list, bearer or `X-Api-Key` principal auth, active managed projection auth and slot-route binding, `anthropic-version` routing, native Anthropic tunnel, Chat bridge, provider-pool-only admission, profile capability checks, managed-versus-legacy provider credentials, marked-preset single-request admission with Edge-owned internal Plan/Review template customization that leaves caller I/O unchanged, and current no-OpenAI-metric status | `apps/edge/internal/openai/anthropic_handler.go`, `apps/edge/internal/openai/anthropic_native.go`, `apps/edge/internal/openai/anthropic_bridge.go`, `apps/edge/internal/openai/anthropic_stream.go`, `apps/edge/internal/openai/anthropic_types.go`, `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/openai/provider_tunnel.go`, `apps/edge/internal/openai/provider_model_rewrite.go`, `apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_plan_stage.go`, `apps/edge/internal/openai/single_request_review_stage.go`, `packages/go/singlerequesttemplate/template.go`, `packages/go/config/protocol_profile.go` | `agent-contract/outer/anthropic-compatible-api.md` | +| `iop.gemini-compatible-api` | Gemini Developer API `streamGenerateContent`, route-qualified Gemini-native ingress, `x-goog-api-key` principal auth, `GOOGLE_GEMINI_BASE_URL`, official agy 1.1.12 API-key transport, Gemini function calls/thought signatures/SSE, and direct-versus-execution-preset binding | `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/live_iop.py` | `agent-contract/outer/gemini-compatible-api.md` | | `iop.a2a-json-rpc-api` | A2A JSON-RPC API, `message/send`, `tasks/get`, `tasks/cancel`, A2A task state, agent card, `a2a.bearer_token`, Edge A2A input surface | `apps/edge/internal/input/a2a/*`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/a2a-json-rpc-api.md` | ## Inner Contracts diff --git a/agent-contract/outer/anthropic-compatible-api.md b/agent-contract/outer/anthropic-compatible-api.md index aae2d278..c3a5a05f 100644 --- a/agent-contract/outer/anthropic-compatible-api.md +++ b/agent-contract/outer/anthropic-compatible-api.md @@ -102,9 +102,13 @@ from the operator's `execution_presets[].single_request.templates` configuration `agent-contract/inner/edge-config-runtime-refresh.md`). These templates are Edge-owned internal stage inputs only: -- The Plan template is the private Plan-stage prompt and the grammar its direct - PlanMD output is parsed against; the Review template shapes the private `review.md` - artifact rendered from the model's `checks`, `verification`, and `summary` fields. +- The Plan stage requests a stage-owned strict JSON object with a one-line `goal` + string, 2-6 non-empty one-line `steps` strings, and 1-3 non-empty one-line + `verification` strings. Edge owns the Markdown bullet/newline formatting and + renders the frozen Plan template. The Review template similarly shapes the private + `review.md` artifact rendered from the model's `checks`, `verification`, and + `summary` fields. Provider output never controls either artifact's headings or + static template text. - The caller-visible request and response schemas are unchanged. A configured template never adds, removes, or renames a Messages request field, a content block, an SSE event, a `stop_reason`, or an error shape, and the final Messages text stays @@ -281,6 +285,7 @@ anthropic-version: 2023-06-01 지원하는 `Anthropic-Beta` 값: - `advanced-tool-use-2025-11-20` +- `advisor-tool-2026-03-01` - `claude-code-20250219` - `context-management-2025-06-27` - `effort-2025-11-24` @@ -295,7 +300,7 @@ anthropic-version: 2023-06-01 지원하지 않는 beta 값을 보내면 `400 invalid_request_error`를 반환한다. Native Messages 경로는 지원 beta 헤더를 upstream으로 전달한다. Chat bridge 경로는 지원 beta 헤더를 upstream으로 전달하지 않고, 아래에 명시한 대응 field만 Chat Completions 형식으로 변환한다. -`prompt-caching-scope-2026-01-05`, `advanced-tool-use-2025-11-20`, `redact-thinking-2026-02-12`, `thinking-token-count-2026-05-13` 수용은 Claude Code 호출 호환성만 제공한다. 이 beta들은 Chat bridge에서 cache, route, stage, provider, workspace 또는 authorization 권한을 만들지 않으며 normalized Chat provider 요청으로 전달되지 않는다. +`prompt-caching-scope-2026-01-05`, `advanced-tool-use-2025-11-20`, `advisor-tool-2026-03-01`, `redact-thinking-2026-02-12`, `thinking-token-count-2026-05-13` 수용은 Claude Code 호출 호환성만 제공한다. 이 beta들은 Chat bridge에서 cache, route, stage, provider, workspace 또는 authorization 권한을 만들지 않으며 normalized Chat provider 요청으로 전달되지 않는다. ## Routes diff --git a/agent-contract/outer/gemini-compatible-api.md b/agent-contract/outer/gemini-compatible-api.md new file mode 100644 index 00000000..307c9588 --- /dev/null +++ b/agent-contract/outer/gemini-compatible-api.md @@ -0,0 +1,109 @@ +# Gemini-Compatible API Contract + +## 계약 메타 + +- id: `iop.gemini-compatible-api` +- boundary: `outer` +- status: active +- 원본 경로: + - `apps/edge/internal/openai/routes.go` + - `apps/edge/internal/openai/principal.go` + - `apps/edge/internal/openai/chat_handler.go` + - `scripts/agent_benchmark/agy_iop.py` + - `scripts/agent_benchmark/live_iop.py` +- external caller surface: official Antigravity CLI `agy` 1.1.12 Gemini API-key provider + +## 읽는 조건 + +- Gemini Developer API `streamGenerateContent`, `x-goog-api-key`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `agy` API-key provider, Gemini-native tool/function call, 또는 Gemini-native SSE ingress를 구현·검증할 때 읽는다. +- IOP execution preset을 Gemini-native caller에 노출하거나 `agy` benchmark transport를 변경할 때 읽는다. + +## 범위 + +IOP Edge가 외부 Gemini-native caller에게 제공하는 초기 호환 표면은 다음 streaming endpoint다. + +```http +POST /gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse +Content-Type: application/json +x-goog-api-key: +``` + +- `{route-id}`는 Edge가 인증된 principal에 대해 해석할 direct route 또는 virtual execution-preset id다. URL path segment 하나의 canonical token이어야 한다. +- `{caller-model}`은 caller가 선택한 Gemini 모델 id이며 관측·호환성 검증 대상이다. provider/credential 또는 execution preset 선택 권한은 갖지 않는다. +- 현재 `agy` 호환 표면은 `alt=sse`인 `streamGenerateContent`만 지원한다. `generateContent`, batch, files, cached content, tuning API는 비범위다. +- direct route와 marked single-request preset은 모두 기존 Edge route resolution, managed admission, provider-pool, preset coordinator를 사용한다. 별도 Gemini 전용 우회 dispatch를 만들지 않는다. + +## Caller 설정 + +공식 `agy` 1.1.12 API-key provider는 다음 값으로 실행한다. + +- `~/.gemini/antigravity-cli/settings.json`: `modelProvider`는 `gemini`다. +- `GEMINI_API_KEY`: upstream provider key가 아니라 IOP principal token이다. +- `GOOGLE_GEMINI_BASE_URL`: `https:///gemini/{route-id}`다. +- 사설 dev CA를 사용하는 경우 caller child에는 표준 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`만 명시적으로 전달한다. +- `--model`: 공식 CLI가 인식하는 Gemini 모델 label을 사용한다. benchmark의 Gemini 3.6 Flash 호출은 `Gemini 3.6 Flash`다. +- `--effort`는 API-key provider 호출에 전달하지 않는다. 요청된 high effort는 인증된 IOP route/preset의 effective binding으로 검증한다. + +`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 계약의 transport가 아니다. + +## Auth 및 credential 경계 + +- `x-goog-api-key`는 Gemini ingress에서 IOP caller 인증 헤더다. Edge는 이를 SHA-256 projection match에만 사용하고 raw 값을 log, metric, response, task evidence에 남기지 않는다. +- `Authorization: Bearer`를 함께 보내면 두 token은 constant-time 비교로 같아야 한다. 다르거나 malformed이면 provider dispatch 전에 Gemini 오류 envelope로 `401`을 반환한다. +- managed mode에서 provider credential은 projected slot과 sealed lease에서만 온다. inbound `x-goog-api-key`를 upstream `Authorization` 또는 upstream `x-goog-api-key`로 전달하지 않는다. +- legacy mode에서도 inbound `x-goog-api-key`를 provider credential로 재사용하지 않는다. provider auth가 별도로 필요하면 기존 명시적 legacy provider-auth 계약만 적용한다. +- marked single-request preset은 managed projection과 고정 stage authorization 없이는 fail closed한다. + +## 요청 변환 + +초기 호환 범위는 official `agy` 1.1.12가 보내는 다음 top-level field다. + +- `contents[]`: `role`, `parts[].text`, `parts[].functionCall`, `parts[].functionResponse`, optional opaque `thoughtSignature` +- `systemInstruction`: official `agy`의 `role: user`와 `parts[].text` +- `generationConfig`: `candidateCount`, `maxOutputTokens`, `stopSequences`, `temperature`, `topK`, `topP`, `thinkingConfig.includeThoughts`, `thinkingConfig.thinkingBudget`, `responseMimeType`, 그리고 상호 배타적인 `responseSchema`/`responseJsonSchema` +- `tools[].functionDeclarations[]`: `name`, `description`, 상호 배타적인 `parameters`/`parametersJsonSchema`, optional 상호 배타적인 `response`/`responseJsonSchema` +- `toolConfig.functionCallingConfig.mode` + +Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message, tool schema와 output cap으로 변환한다. `thinkingConfig`는 Gemini OpenAI-compatible upstream이 요구하는 `extra_body.google.thinking_config`로 보존한다. `responseMimeType=application/json`은 Chat `response_format`으로 변환하며 schema가 있으면 고정 이름의 strict `json_schema`, 없으면 `json_object`를 사용한다. `text/plain`은 schema를 허용하지 않는다. Gemini 3.6에서 폐기된 `temperature`, `topP`, `topK`는 형식과 범위만 검증하고 Chat upstream에는 전달하지 않는다. 함수명·JSON argument·opaque thought signature는 caller turn 사이에 의미를 바꾸지 않는다. 동의어 schema 필드가 동시에 존재하거나 schema가 JSON object가 아니면 거부한다. 지원하지 않는 content part, duplicate member, 잘못된 role, malformed function payload 또는 둘 이상의 candidate 요청도 provider dispatch 전에 `400 INVALID_ARGUMENT`으로 거부한다. + +## SSE 응답 + +- 성공 stream은 `Content-Type: text/event-stream`과 `data: ` frame을 사용한다. +- text delta는 `candidates[0].content.parts[].text`, reasoning delta는 `thought=true`인 part, 완성된 tool call은 `functionCall` part로 투영한다. +- tool-call argument fragment는 Edge가 bounded buffer에서 완성된 JSON object로 검증한 뒤 한 번만 공개한다. malformed·oversize argument는 성공 tool call로 내보내지 않는다. +- OpenAI `stop`, `length`, `tool_calls` terminal은 Gemini `finishReason`의 `STOP`, `MAX_TOKENS`, `STOP`으로 닫는다. stream 종료 뒤 별도 합성 `system idle` event를 만들지 않는다. +- provider-reported usage가 있으면 `usageMetadata.promptTokenCount`, `candidatesTokenCount`, `thoughtsTokenCount`, `cachedContentTokenCount`, `totalTokenCount`의 존재하는 값만 투영한다. 누락 값을 0으로 발명하지 않는다. +- caller disconnect는 기존 request cancellation 경계를 사용하며 이후 frame을 쓰지 않는다. + +## 공식 agy stream-json lifecycle + +`agy` 1.1.12의 각 JSONL record는 `event` discriminator와 같은 이름의 중첩 payload를 사용한다. + +- init: `{"event":"init","init":{...}}` +- step: `{"event":"step_update","step_update":{"state":...,"step_type":...,"usage":{...}}}` +- terminal: `{"event":"result","result":{"status":"SUCCESS","duration_seconds":...,"num_turns":...,"usage":{...}}}` + +benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 success terminal로 인정한다. 구조가 유효한 `result.status=ERROR`는 stream incompatibility가 아니므로 success terminal을 만들지 않고 caller process의 non-zero exit를 lifecycle failure authority로 보존한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다. + +## 오류 + +HTTP commit 전 오류는 다음 Gemini envelope 한 건으로 반환한다. + +```json +{"error":{"code":400,"message":"request is invalid","status":"INVALID_ARGUMENT"}} +``` + +- 인증 실패는 `401 UNAUTHENTICATED`, route/요청 검증 실패는 `400 INVALID_ARGUMENT`, runtime/provider 실패는 `502 UNAVAILABLE`의 고정된 caller-safe message를 사용한다. +- stream commit 뒤 오류는 Gemini `error` payload 한 건으로 끝내며 raw provider body, endpoint, route binding, credential/slot/lease id, prompt, tool argument/result를 포함하지 않는다. + +## 변경 시 확인할 코드와 테스트 + +- route/auth: `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go` +- Gemini request/SSE bridge: `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go` +- Edge regression: `apps/edge/internal/openai/gemini_handler_test.go`, existing Chat/preset/auth tests +- caller adapter: `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/agy_iop_test.py`, `scripts/agent_benchmark/live_iop.py` +- live proof: official `agy --output-format stream-json` through the dev Edge route-specific base URL, with direct and execution-preset effective binding evidence + +2026-08-12 dev 검증에서 normal/boundary/auth/tool/SSE 회귀 테스트와 공식 `agy` 1.1.12 direct·hybrid 실호출이 통과해 이 계약을 active로 전환했다. + +2026-08-12 benchmark 실측 보정에서 official agy planner의 structured-output schema field를 Chat `response_format`으로 변환하고, 구조가 유효한 ERROR result를 parser failure와 분리했다. diff --git a/agent-contract/outer/openai-compatible-api.md b/agent-contract/outer/openai-compatible-api.md index 2b6d9ed6..1b8a3238 100644 --- a/agent-contract/outer/openai-compatible-api.md +++ b/agent-contract/outer/openai-compatible-api.md @@ -54,8 +54,10 @@ Edge 설정에 `openai.principal_tokens[]`가 설정된 경우, caller는 기존 In managed mode, OpenAI-compatible routes authenticate `Authorization: Bearer ` by hashing the token and matching the projected digest. Static principal mappings and the legacy bearer are prohibited by configuration and never act as fallbacks. Unknown or removed digests, malformed headers, and expired snapshots return `401 unauthorized` before model lookup or dispatch. Expiry never returns the process to legacy behavior. When managed mode is active, model discovery (`GET /v1/models`) lists only active -projected `route_id`s for the authenticated principal. Request model resolution binds -the request strictly to one projected route's `slot_id`, `profile_id`, and `upstream_model`. +projected public route identities for the authenticated principal: `route_alias` when +it is non-empty, otherwise `route_id`. Request model resolution accepts that same +public identity and binds the request strictly to one projected route's `slot_id`, +`profile_id`, and `upstream_model`. Unknown, inactive, or cross-principal routes never fall back to legacy `model_routes`, global catalog, or single-target default. diff --git a/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md b/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md index f0838b1f..cfaea914 100644 --- a/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md +++ b/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md @@ -12,7 +12,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci ## When to use - User requests benchmark manifest validation: `validate`, `validate manifest`, `manifest 검증` -- User requests direct-route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검` +- User requests route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검` - User requests benchmark execution: `run`, `run benchmark`, `벤치마크 실행`, `시작해` - User requests benchmark resume: `resume`, `resume benchmark`, `재개`, `계속해` - User requests benchmark status: `status`, `status benchmark`, `상태 확인`, `어디까지 왔어` @@ -55,36 +55,36 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci 4. **Delegate preflight to the CLI** - Run: `python3 scripts/agent_comparison_benchmark.py preflight --manifest ` - - The CLI validates generic preset cells locally and records only direct-cell observations in one append-only run record. + - The CLI records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order in one append-only run record. - On exit 0, report the exact closed `ready` summary from stdout. - - On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat a generic preset contract as live readiness. + - On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat local manifest validation as live readiness. 5. **Delegate run to the CLI** - Run: `python3 scripts/agent_comparison_benchmark.py run --manifest ` - On missing or invalid manifest, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before creating execution state. - - The CLI creates one run and uses its single writer to append a fresh preflight before attempt allocation. + - The CLI creates one run and uses its single writer to append a fresh all-cell preflight before attempt allocation. - On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69, allocates no attempt, and preserves the run id for a later resume. - On `ready`, it binds the exact caller, cell, fresh workspace, session, and attempt identity, then must invoke each eligible cell exactly once with the fixture task. - - Exit 0 only when every retained attempt is successful; otherwise report the exact closed execution failure summary from stderr with exit 69. + - Exit 0 only when every latest slot has `product=succeeded`, `harness=passed`, `process=exited` with exit code 0 and no signal, and `artifact=passed`; otherwise report the exact independent-axis summary from stderr with exit 69. 6. **Delegate resume to the CLI** - Run: `python3 scripts/agent_comparison_benchmark.py resume --manifest --run-id [--retry-failed]` - On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before changing the run. - - The CLI opens the exact immutable run and uses its single writer to append a fresh preflight before attempt allocation. + - The CLI opens the exact immutable run and uses its single writer to append a fresh all-cell preflight before attempt allocation. - On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69 and allocates no attempt. - - On `ready`, it reconciles interrupted state, skips successful slots, preserves prior attempt bytes, and allocates a new attempt only for eligible work. `--retry-failed` admits a new attempt for failed, timed-out, or cancelled slots. + - On `ready`, it reconciles interrupted state, skips only slots whose latest product/harness/process/artifact gates all pass, preserves prior attempt bytes, and allocates a new attempt only for eligible work. `--retry-failed` admits a new attempt for any latest terminal attempt whose independent gates do not all pass. - It must invoke each eligible cell exactly once with a new workspace and session identity. 7. **Delegate status to the CLI** - Run: `python3 scripts/agent_comparison_benchmark.py status --manifest --run-id ` - On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64). - - On success, the CLI prints `ok: ` to stdout with exit 0. + - On success, the CLI prints the controller counts, product/harness/process/artifact counts, and `unresolved=` to stdout with exit 0. - Report the exact CLI output. 8. **Delegate score to the CLI** - Run: `python3 scripts/agent_comparison_benchmark.py score --manifest --run-id [--retry-scoring-failed]` - - The CLI classifies lifecycle or required web-gate failures as immutable `unscored`, without invoking the evaluator or assigning zero. - - Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact `landing-quality-v1` worksheet. + - The CLI classifies each failed product, harness, process, or required artifact gate with its own immutable `unscored` reason, without invoking the evaluator or assigning zero. + - Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted. - A prior `scored` result is terminal. A prior `scoring_failed` result is retried only with `--retry-scoring-failed`, which allocates a new score id and preserves every prior byte. - On exit 0, report the exact closed `scored`, `unscored`, `scoring_failed`, and `blocked` counts from stdout. - On exit 69, report the exact closed counts or unavailable-state line from stderr. Never substitute evaluator route/model/effort, fabricate a worksheet, or turn failure into zero. @@ -97,9 +97,10 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci - [ ] The CLI command was executed and the exit code matches the documented contract. - [ ] The reported stdout/stderr matches the CLI output exactly. -- [ ] Preflight evidence is append-only, direct-only, and its output uses only closed status/count fields. +- [ ] Preflight evidence is append-only, covers every immutable matrix cell in canonical order, and uses only closed status/count fields. - [ ] A preflight blocker created no scored attempt and was not bypassed. - [ ] An ineligible execution attempt became `unscored` without an evaluator invocation or a zero score. +- [ ] Product, harness, process, and artifact outcomes remain separately visible in run/resume/status output and report rows. - [ ] Each eligible score id used one opaque blind workspace and one fresh evaluator session; retry preserved prior bytes and used a new id. - [ ] `scoring_failed` used no fallback, synthetic worksheet, or implicit retry. - [ ] No caller or provider was invoked outside the deterministic CLI. @@ -150,7 +151,7 @@ For run/resume ready completion: ``` command: exit_code: 0 -stdout: ok: run_id= completed= unresolved=0 success= failed= timed_out= cancelled= interrupted= running=0 +stdout: ok: run_id= executed= unresolved=0 completed= timed_out= cancelled= interrupted= running=0 product_succeeded= product_failed=0 product_unknown=0 harness_passed= harness_failed=0 process_exited= process_signalled=0 process_timed_out=0 process_cancelled=0 process_not_started=0 artifact_passed= artifact_failed=0 artifact_blocked=0 artifact_not_run=0 stderr: (none) ``` @@ -169,9 +170,11 @@ stderr: - Durable run/attempt state and preflight evidence are persisted only under the validated run root (`agent-test/runs///`). - Caller sessions, output workspaces, and caches are fresh and isolated for every cell, repetition, and attempt; session or cache state is never shared within a run or across runs. - Read-only testbed/fixture inputs (such as `../iop-s2`) are not copied back or mutated; no writes occur outside the validated run root. -- Direct preflight never allocates a scored attempt. Generic preset cells are local contract validation only. -- Run/resume append a fresh direct preflight under the run writer before any attempt allocation; a blocker allocates no attempt. +- Preflight never allocates a scored attempt. Every immutable matrix cell requires its own fresh live observation. +- Run/resume append a fresh all-cell preflight under the run writer before any attempt allocation; a blocker allocates no attempt. - Ready execution binds one exact cell and immutable attempt identity to one fresh workspace/session and one task submission. +- Product, harness, process, and artifact are independent gates. Controller state `completed` only means the invocation controller reached a terminal state. +- Release qualification runs the five-cell direct manifest as one unscored canary and requires all four gates for all five cells before a fresh nine-cell preflight; it does not allocate hybrid or scored execution. - Scoring copies only anonymous generated files, two local images, and screenshots into an opaque run-owned blind tree; the identity mapping remains outside that tree. - Scoring records `unscored`, `scored`, and `scoring_failed` append-only, and a retry always allocates a fresh score id/session. - The internal workspace API (`RunStore`, `Manifest`, etc.) is not a user command. Do not expose it. diff --git a/agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md b/agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md index 87e0c151..6c9d25b3 100644 --- a/agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md +++ b/agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md @@ -57,7 +57,7 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실 - 경로: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](../../archive/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md) - 요약: 모델·caller·prompt·반복 횟수를 manifest로 바꾸고 Claude Code, agy, Codex의 IOP 연결부터 finish/idle, 시간·token·웹 검증·익명 채점·Markdown 보고까지 같은 pipeline으로 재현한다. -- [계획] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크 +- [진행중] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크 - 경로: [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](milestones/iop-one-shot-agent-model-comparison.md) - 요약: route-02 정식 smoke와 benchmark pipeline 준비 뒤 dev `../iop-s2`에서 동일 정적 웹 fixture로 9개 IOP 경유 단독·하이브리드 caller 조합을 각각 한 번 실행해 속도·token·품질을 비교한다. diff --git a/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md b/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md index 4a414a8a..bab39bfa 100644 --- a/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md +++ b/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md @@ -13,7 +13,7 @@ ## 상태 -[계획] +[진행중] ## 구현 잠금 @@ -59,9 +59,11 @@ 실행 전에 공정한 fixture와 실제 IOP route/credential 상태를 고정한다. -- [ ] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다. -- [ ] [route-readiness] dev `../iop-s2`에서 Claude Code·agy·Codex의 IOP 인증, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 preflight를 통과했는지 확인한다. -- [ ] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다. +- [x] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다. +- [x] [agy-iop-compatibility] 공식 `agy 1.1.12`의 Gemini API-key provider가 route별 `GOOGLE_GEMINI_BASE_URL`과 `GEMINI_API_KEY=`으로 Edge의 Gemini-native `streamGenerateContent` ingress를 호출하고, Gemini request/tool/SSE를 기존 direct·execution-preset 실행에 연결하며 실제 `stream-json` lifecycle/usage를 benchmark adapter가 수집하도록 구현한다. `--effort`와 비공식 custom model에 의존하지 않고 high effort는 IOP effective binding으로 검증한다. +- [x] [managed-credential-dev] dev Control Plane·Edge·Node에 CA-signed mTLS, Edge HTTPS, at-rest/issuer/recipient key material, principal projection과 Gemini/Claude/GPT slot-route를 operator-owned secret 경로로 구성한다. legacy static credential source를 제거하고 marked hybrid preset이 같은 principal의 고정 stage route를 managed lease로 실행하는지 secret-safe smoke로 확인한다. +- [x] [route-readiness] dev `../iop-s2`에서 Claude Code·Codex의 호환 ingress와 공식 agy의 Gemini API-key transport가 각각 IOP 인증 경계를 통과하고, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 실제 caller preflight를 통과했는지 확인한다. 존재하지 않는 caller 환경 변수나 합성 event fixture는 live 호환 근거로 인정하지 않는다. +- [x] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다. ### Epic: [comparison-runs] 9개 원샷 실행 @@ -103,6 +105,7 @@ - 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2` - 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다. - 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다. +- 현재 차단: readiness는 완료됐고 all-cell preflight는 `ready=9`였다. 승인된 C01-C09 `run`은 2026-08-12에 한 번 호출됐으나 첫 slot의 caller launch 전 control socket 등록이 workspace filesystem의 symlink-path `bind(2)` `EINVAL`로 중단됐다. caller는 실행되지 않았고 dangling attempt는 harness reconcile로 `interrupted=1`, `running=0`이 됐다. socket 등록과 pre-registration reconcile 결함은 회귀 테스트와 함께 수정했지만 repetitions=1·no-retry 정책 때문에 같은 scored run을 재실행하거나 resume하지 않는다. C01-C09를 다시 시작하려면 기존 실패를 덮지 않는 새 실행 승인과 SDD/manifest run 정책 결정이 필요하다. - 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md) - 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md) - 확인 필요: 없음 diff --git a/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md b/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md index 0e329567..dd72d1c6 100644 --- a/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md +++ b/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md @@ -24,6 +24,8 @@ - [x] [D08] token은 input/output/reasoning/cached/total과 source를 model/stage별로 기록하고 미제공 값을 exact로 추정하지 않는다. - [x] [D09] 결과 identity를 가린 뒤 동일 100점 rubric으로 Codex가 채점하고 자동 검증과 수동 점수를 분리한다. - [x] [D10] scored failure는 보존하고 재실행은 새 attempt로 기록하며 성공 결과만 골라 대표하지 않는다. + - [x] [D11] 공식 `agy 1.1.12`는 Gemini API-key provider의 route별 `GOOGLE_GEMINI_BASE_URL`을 IOP Edge로 지정하고 `GEMINI_API_KEY`에는 upstream key가 아닌 IOP principal token을 넣는다. `--effort`와 비공식 custom model은 사용하지 않고 high effort는 IOP effective binding으로 검증한다. + - [x] [D12] marked hybrid preset은 dev managed credential plane의 fresh projection, 고정 stage authorization과 sealed provider lease가 준비된 뒤에만 실행하며 legacy credential fallback을 허용하지 않는다. ## 문제 / 비목표 @@ -44,7 +46,7 @@ | Fixture | versioned prompt, 이미지 2장과 vanilla workspace checksum | 모든 cell의 동일 입력 기준 | | Evidence | `agent-test/runs//` | attempt별 timeline, usage, validation, screenshot와 score | | Report | `agent-test/dev/iop-one-shot-agent-comparison-.md` | 현재 프로젝트의 사람이 읽는 비교 결과 | -| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 | +| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 | | Config Contract | [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) | model route, preset, protocol profile과 credential 경계 | | User Decision | D01-D10 | 2026-08-06 확정 방향, 추가 사용자 결정 없음 | @@ -53,7 +55,7 @@ | 상태 | 진입 조건 | 다음 상태 | 근거 | |------|-----------|-----------|------| | `blocked` | `[route-02]` smoke 또는 `[bench-01]` 완료 전 | `preflighting`, 종료 | active Milestone 상태와 pipeline evidence | -| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | redacted preflight matrix | +| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | official caller transport, managed projection/lease와 redacted preflight matrix | | `ready` | fixture와 C01-C09 immutable manifest 확정 | `running`, `cancelled` | manifest/fixture/rubric digest | | `running` | seed 순서에 따라 각 cell에 사용자 작업 1회 제출 | `validating`, `failed`, `timed_out`, `cancelled` | cell/attempt event timeline | | `validating` | cell finish/complete 후 idle 확정 | `scoring`, `failed` | workspace, build/render/test evidence | @@ -75,13 +77,14 @@ State invariant: ## Interface Contract -- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) +- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) - 입력: - `fixture`: 동일 이미지 2장, one-page 요구사항, vanilla HTML/CSS/JS initial workspace와 checksum이다. - `cells`: C01-C09의 caller, IOP route/preset, expected model/stage와 effort binding이다. - `repetitions=1`, `session_policy=fresh`, `setup_cache_policy`: 초기 scored attempt 수, conversation/resume 격리와 공통 setup/cache 기준이다. - `environment=dev`, `testbed=../iop-s2`: 실제 IOP runtime 선택이다. - `completion`: caller별 finish/complete event와 idle 판정 규칙이다. + - `agy`: `modelProvider=gemini`, route별 `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY=`과 official `stream-json` event다. API-key provider가 지원하지 않는 `--effort`는 전달하지 않는다. - 측정 출력: - timestamp: submitted, first output, first file write, model/stage start/end, tool start/end, finish, idle의 monotonic 값과 observation source다. overlap과 unavailable을 명시한다. - usage: call count, input/output/reasoning/cached/total token과 source다. @@ -111,6 +114,8 @@ State invariant: | S10 | `quality-scoring` | identity가 제거된 9개 결과 | Codex rubric 평가 | 항목별 점수/근거와 총점이 자동 gate와 분리되어 기록된다. | | S11 | `performance-usage` | 모든 attempt timeline/usage | 비교 집계 | 첫 output·첫 write·model/tool/queue/total 시간의 clock/source·overlap, 호출 수와 token/source가 cell·stage별 표가 된다. | | S12 | `benchmark-report` | S01-S11 evidence | 보고서 생성 | 조건·버전·9개 결과·속도·token·품질·실패·한계와 raw evidence 링크가 Markdown에 남는다. | +| S13 | `agy-iop-compatibility` | official `agy 1.1.12`와 dev Edge | direct·hybrid route별 Gemini base URL로 실제 API-key 호출 | 두 호출 모두 `x-goog-api-key` IOP principal auth, Gemini-native request/tool/SSE, official `stream-json` finish/exit와 config-owned effective binding evidence를 남기고 upstream key 직접 호출이나 합성 event에 의존하지 않는다. | +| S14 | `managed-credential-dev` | dev Control Plane·Edge·Node와 operator-owned security material | managed credential profile로 재기동하고 slot/route를 등록 | CA-signed mTLS·Edge HTTPS·fresh projection·sealed lease가 확인되고 legacy credential source나 cross-route fallback 없이 direct와 marked preset stage가 실행된다. | ## Evidence Map @@ -128,6 +133,8 @@ State invariant: | S10 | blind mapping 분리와 Codex rubric worksheet | `agent-task/m-iop-one-shot-agent-model-comparison/quality-scoring/` | `quality-scoring` 100-point evidence | | S11 | cell/stage별 normalized timeline, calls와 token-source table | `agent-task/m-iop-one-shot-agent-model-comparison/performance-usage/` | `performance-usage` speed/token evidence | | S12 | `agent-test/dev/` Markdown과 raw run links | `agent-task/m-iop-one-shot-agent-model-comparison/benchmark-report/` | `benchmark-report` complete comparison evidence | +| S13 | official agy request-shape capture, Edge Gemini bridge tests, direct·hybrid live preflight와 sanitized lifecycle/usage | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `agy-iop-compatibility` official 1.1.12 IOP transport evidence | +| S14 | dev config check, TLS/workload identity, projection generation, slot-route/lease attribution과 post-revoke no-fallback smoke | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `managed-credential-dev` secure composition and hybrid admission evidence | 공통 완료 검증은 C01-C09 모두가 success/failure/blocked 중 하나의 terminal evidence를 가지고, 성공 결과의 자동 gate·screenshot·blind score와 모든 attempt의 timing/usage source가 보고서에 연결되는지 확인한다. 필수 credential/model이 없으면 raw secret을 요구하거나 기록하지 않고 운영 절차로 등록을 요청한다. @@ -145,6 +152,7 @@ State invariant: ## 사용자 리뷰 이력 - 2026-08-06: 사용자가 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 비교군, Claude Code·agy·Codex caller, finish/idle 원샷, 초기 1회, dev `../iop-s2`, 동일 정적 웹 fixture와 시간·token·Codex 품질 평가를 확정했다. +- 2026-08-12: 공식 `agy 1.1.12` API-key provider의 실제 Gemini-native 요청과 `stream-json` event를 확인했고, 사용자의 provider 직접 설정 지시에 따라 upstream key와 IOP principal token을 분리하며 dev managed credential plane까지 구성하는 D11-D12를 기술 보강했다. ## 작업 컨텍스트 diff --git a/agent-spec/input/openai-compatible-surface.md b/agent-spec/input/openai-compatible-surface.md index be1fe8ff..8a31da40 100644 --- a/agent-spec/input/openai-compatible-surface.md +++ b/agent-spec/input/openai-compatible-surface.md @@ -169,7 +169,7 @@ Edge가 OpenAI-compatible HTTP 요청을 받아 내부 `adapter + target` 실행 | managed projection auth | `credential_plane.enabled=true` uses the fresh Control Plane projection for inbound token auth and principal route discovery. Static principal/bearer fallback is disabled. | | managed slot route | Public model id/alias resolves to one projected route, exact slot/profile/upstream model/resource selector, and immutable revisions/generation. Unknown, cross-principal, stale, revoked, or ambiguous bindings fail closed. | | marked preset single-request admission | An authorized fixed single-request preset compiles one service-owned admission value at request start: requested public model, canonical plan/work/review bindings resolved through managed authorization, opaque workspace capability, and absolute resource caps. Later refresh cannot mutate the admitted shape. No private binding is echoed to the caller. Compiled only after every canonical reference is verified through its catalog binding for the authenticated principal; missing, duplicate, unauthorized, dynamically selected, or option-inconsistent inputs are rejected without fallback. | -| marked single-request internal templates | The admission also freezes the operator-configured effective Plan/Review Markdown templates. They are internal stage inputs only: the Plan template is the private Plan-stage prompt and PlanMD grammar, and the Review template shapes the private `review.md` artifact. Callers cannot supply, name, or select a template, and template paths, contents, and digests never appear in a response, error, log, or metric label. The caller-visible Messages request/response schema is unchanged and the final text stays the model's `decision.output`. | +| marked single-request internal templates | The admission also freezes the operator-configured effective Plan/Review Markdown templates. They are internal artifact shapes only: the Plan stage obtains a strict one-line `goal` plus bounded one-line `steps`/`verification` arrays, and Edge owns the bullet formatting and renders `plan.md`, while the Review template shapes the private `review.md` artifact. Callers cannot supply, name, or select a template, and template paths, contents, and digests never appear in a response, error, log, or metric label. The caller-visible Messages request/response schema is unchanged and the final text stays the model's `decision.output`. | | marked single-request ingress | One validated and authorized Messages POST enters the separate service coordinator capability before legacy provider/caller continuation and increments `iop_anthropic_single_request_ingress_total` once. Non-streaming returns one buffered final-only message. Streaming keeps one envelope across the coordinator lifetime, exposes only fixed plan/work/review/repair text blocks plus `event: ping`, and commits one final text/error terminal. Internal reasoning/tool wire never becomes caller `tool_use`; success is acknowledged only after the complete terminal write succeeds. | | marked single-request S11 terminal policy | The service freezes one closed `end_turn`, `length`, `error`, or `cancelled` disposition. `error` classes are provider, validation, timeout, budget, repetition, malformed, context, internal-tool, and workspace-cleanup. Buffered and SSE share one projection: `end_turn`; `max_tokens` with no private partial output; `400 invalid_request_error` for validation/context; `502 api_error` for other failures; and silent cancellation after caller disconnect. No terminal classification retries, falls back, opens a second request, or later writes success. | | marked single-request observation evidence | A single real POST links ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, and raw-free correlation without public tool protocol. `iop_anthropic_single_request_ingress_total` is unlabeled (no request_id, stage_id, provider identity, or content). Internal tool names, raw arguments, private results, and workspace references are absent from the public terminal and from log projections. Stage-pure timing, cardinality-bounded labels, and privacy semantics are documented here. SDD S12 qualified an external Claude task on an approved IOP Node with ingress `0→1`, Gemini → ornith-fast → Gemini, one `end_turn`, the exact expected workspace result, timing, cleanup, and redacted evidence. | @@ -262,10 +262,10 @@ sequenceDiagram - provider-pool model group은 capacity + priority + availability 기준으로 provider candidate를 먼저 선택하고, 선택된 provider가 OpenAI-compatible 호출 방식을 지원하면 raw tunnel passthrough로 dispatch한다. Ollama/native provider가 선택되면 normalized `RunRequest` path로 dispatch한다. - Anthropic Messages and count-tokens do not use legacy direct-route or single-target fallback. Native responses preserve provider status, allowed headers, and body/SSE bytes; bridge responses are converted between Anthropic Messages and Chat Completions shapes. - A marked single-request Messages dispatch requires the narrow service coordinator capability and never falls back to the generic provider pool. The handler copies the immutable binding and request input and counts the accepted HTTP admission once with no labels. The service projects exactly one frozen terminal candidate through both response modes: buffered/SSE `end_turn`; buffered/SSE `max_tokens` without private partial content; `invalid_request_error` for validation/context; `api_error` for provider, timeout, budget, repetition, malformed, internal-tool, and workspace-cleanup failures; or silent cancellation after caller disconnect. The streaming path maps only fixed plan/work/review/repair summaries, serializes pings and monotonic text-block indices with one terminal owner, stops and joins liveness before terminal/return, and acknowledges completion only after `message_stop`. Arbitrary progress, reasoning, tool/provider/credential/workspace data, raw failures, and internal stage terminals stay private. No classified terminal triggers retry, fallback, partial success, a second request, or a later success terminal. Count-tokens does not enter or increment this path. -- Marked single-request Plan/Review templates are Edge-owned internal stage inputs, not part of this input surface. The operator configures them in `execution_presets[].single_request.templates`; admission freezes the effective pair, so a config refresh reaches only requests admitted after it and an already running request keeps its pair. The Plan template supplies the private Plan-stage prompt and the grammar its direct PlanMD output is parsed against, and the Review template shapes the private `review.md` artifact rendered from the model's `checks`/`verification`/`summary` fields. No caller field, header, or metadata value can supply, name, select, or override a template, and no template path, content, or digest appears in a response, an error message, a log projection, or a metric label. Changing a template changes neither the Messages request schema nor the response schema: the buffered/SSE terminal projection is unchanged and the final caller-visible text remains the model's `decision.output`. +- Marked single-request Plan/Review templates are Edge-owned internal artifact shapes, not part of this input surface. The operator configures them in `execution_presets[].single_request.templates`; admission freezes the effective pair, so a config refresh reaches only requests admitted after it and an already running request keeps its pair. The Plan stage requests a closed strict JSON object containing a one-line `goal` string, a 2-6 item `steps` string array, and a 1-3 item `verification` string array. Each item must be non-empty and one-line; Edge adds the Markdown bullet prefixes and newlines and renders the frozen Plan template deterministically. The Review template shapes the private `review.md` artifact rendered from the model's `checks`/`verification`/`summary` fields. No caller field, header, or metadata value can supply, name, select, or override a template, and no template path, content, or digest appears in a response, an error message, a log projection, or a metric label. Changing a template changes neither the Messages request schema nor the response schema: the buffered/SSE terminal projection is unchanged and the final caller-visible text remains the model's `decision.output`. - Marked single-request observation evidence links ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, and raw-free correlation for one real POST. `iop_anthropic_single_request_ingress_total` is strictly unlabeled: no request_id, stage_id, provider identity, content, or workspace reference appears as a metric label. Internal tool names (`workspace_read`, `workspace_write`, etc.), raw arguments, private results, and workspace references are absent from the public terminal JSON and from log projections. Stage-pure timing, cardinality-bounded labels, and privacy semantics are documented here. SDD S12 qualifies the external Claude path on an approved IOP Node with one accepted ingress, the expected stage sequence, one terminal, exact output, timing, cleanup, and redacted evidence. - Internal workspace calls use a service-owned schema independent of caller-facing tool codecs. The five closed operation names decode into typed Node requests only after request/stage/tool identity, canonical relative path, approved operation/command/environment capability, and immutable budget checks. The loop opens once, preserves the admitted connection generation, executes one pending call at a time, accepts only correlated typed results, and returns a deep-copied raw-free result to the same executor continuation. Repeated IDs, stale responses, malformed or denied input, timeout, output/iteration exhaustion, and cancellation never become public Anthropic tool protocol or trigger a second ingress. -- Claude Code Messages requests may use adaptive thinking, `output_config.effort`, structured output, cache-control annotations, and supported beta headers. The Chat bridge consumes those headers, maps supported fields, and requires callers to replay opaque `tool_use.id` values unchanged so Gemini thought signatures can be restored on tool-result turns. +- Claude Code Messages requests may use adaptive thinking, `output_config.effort`, structured output, cache-control annotations, and supported beta headers, including the compatibility-only `advisor-tool-2026-03-01` marker emitted by the pinned official caller. The Chat bridge consumes rather than forwards those headers, maps supported fields, and requires callers to replay opaque `tool_use.id` values unchanged so Gemini thought signatures can be restored on tool-result turns. - provider capacity와 long-context slot은 model alias별이 아니라 `node_id + provider_id`별로 공유한다. queue pending 상한과 timeout은 Edge root `provider_pool` policy이며, lease 반환·refresh·disconnect/reconnect가 모든 model group waiter를 global enqueue 순서로 재평가한다. - provider가 full이면 queue policy에 따라 대기하지만 live candidate가 모두 사라지면 즉시 unavailable로 수렴한다. Chat Completions와 Responses provider-pool 표면은 새 public status/field 없이 HTTP 502 `node_dispatch_error`를 유지한다. - In legacy mode, `openai.provider_auth` stores only a forwarding rule and reads raw provider material from its request-time header; inbound IOP authorization is never reused. Managed mode rejects that rule and the caller header and uses only the sealed slot lease. @@ -326,6 +326,9 @@ sequenceDiagram ## 변경 기록 +- 2026-08-12: Admitted Claude Code's `advisor-tool-2026-03-01` beta as a consumed compatibility marker for both direct and marked-preset Messages ingress. It grants no internal capability and is not forwarded through the Chat bridge (`apps/edge/internal/openai/anthropic_types.go`). +- 2026-08-12: Replaced free-form PlanMD generation with a stage-owned strict `goal`/`steps`/`verification` JSON response and deterministic Edge rendering of the frozen Plan template. Internal artifact customization and all caller-visible Messages schemas remain unchanged (`apps/edge/internal/openai/single_request_plan_stage.go`, `packages/go/singlerequesttemplate/template.go`). +- 2026-08-12: Made the private Plan response provider-independent by representing steps and verification as bounded one-line string arrays; Edge now owns Markdown bullet and newline formatting (`apps/edge/internal/openai/single_request_plan_stage.go`, `packages/go/singlerequesttemplate/template.go`). - 2026-08-09: Separated Edge-owned internal Plan/Review stage templates from caller-visible I/O. Operator-configured `execution_presets[].single_request.templates` is frozen at admission, supplies the private Plan prompt/PlanMD grammar and the private `review.md` artifact shape, and is not selectable, supplyable, or observable by callers. The Messages request/response schema, terminal projection, and final `decision.output` text are unchanged (`apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_review_stage.go`, `agent-contract/outer/anthropic-compatible-api.md`). - 2026-08-09: Qualified S12 `claude-smoke` with one supervised external Claude task on an approved IOP Node: ingress `0→1`, Gemini → ornith-fast → Gemini, one `end_turn`, exact 42-byte workspace output, timing, cleanup, and redacted evidence. - 2026-08-07: Implemented and documented S11 `error-cancel`: one closed service terminal disposition, request-local typed failure/no-progress classification, shared buffered/SSE `end_turn`/`max_tokens`/`invalid_request_error`/`api_error` mapping, silent disconnect, private-partial suppression, and deterministic one-ingress/one-terminal/no-second-request evidence. S12 external qualification remains pending. diff --git a/agent-spec/runtime/edge-node-execution.md b/agent-spec/runtime/edge-node-execution.md index 4452edd2..e8fb0451 100644 --- a/agent-spec/runtime/edge-node-execution.md +++ b/agent-spec/runtime/edge-node-execution.md @@ -80,16 +80,16 @@ source_evidence: notes: Exact-generation dispatch, frozen request-authority construction, closed artifact response validation, bounds, and stable failure translation - type: code path: apps/edge/internal/openai/single_request_plan_stage.go - notes: Private fixed Plan stage runner, frozen-template prompt and direct PlanMD parsing, and PLAN artifact write + notes: Private fixed Plan stage runner, strict structured field response, deterministic frozen-template rendering, and PLAN artifact write - type: test path: apps/edge/internal/openai/single_request_plan_stage_test.go notes: Deterministic Plan request/options/envelope/artifact evidence - type: code path: packages/go/singlerequesttemplate/template.go - notes: Bounded Plan/Review template grammar with exact standalone heading and PASS lines, closed placeholder inventory, PlanMD parsing, and Review rendering + notes: Bounded Plan/Review template grammar with exact standalone heading and PASS lines, closed placeholder inventory, deterministic Plan rendering, legacy PlanMD parsing, and Review rendering - type: test path: packages/go/singlerequesttemplate/template_test.go - notes: Grammar, size-boundary, delimiter-residue, and PlanMD/Review render evidence + notes: Grammar, size-boundary, delimiter-residue, and deterministic Plan/Review render evidence - type: code path: packages/go/config/load.go notes: Relative-only plan_file/review_file resolution against the edge.yaml directory with per-file default fallback and filesystem-kind, size, and UTF-8 rejection @@ -228,7 +228,7 @@ The shared `packages/go/execution` package contains provider lifecycle, registry | workspace tool executor | A validated `darwin|linux` Node catalog owns opened root and directory handles only when every entry platform matches the host exactly. Windows, unknown hosts, and cross-platform catalogs fail before root open; empty catalogs remain compatible. Go 1.24-compatible no-follow file primitives provide bounded read, bounded list, structured write, and non-recursive delete. Exact operator-owned command templates run through an inherited-root `fchdir`/`exec` shim with minimal allowlisted environment, shared stdout/stderr bounds, process-group timeout/cancel, and stable typed results. OS is runtime evidence rather than a caller-visible selector. | | internal workspace tool loop | The service decodes only `workspace_read`, `workspace_list`, `workspace_write`, `workspace_delete`, and `workspace_command`, opens the admitted workspace once, dispatches one call at a time on the frozen generation, and delivers one deep-copied typed result to the emitting executor continuation. Unique request/stage/tool correlation, per-stage iteration/output/deadline limits, request wall-clock budget, and typed cancel fail closed without external continuation or reselection. | | request-owned internal artifacts | `SingleRequestController` exposes closed plan/review read/write operations. Artifact calls and model workspace tools share one serialized lazy `WorkspaceOpen`, the exact admitted Node generation, the active stage deadline, the immutable output bound, in-flight work accounting, and one terminal cleanup. Node alone maps selectors to `plan.md` and `review.md`, and inventoried descriptor-relative reads fail closed on identity replacement. | -| Plan stage | The Plan runner emits the `planning` envelope, sends the immutable task through the frozen Gemini Chat binding with `reasoning_effort=high` and the frozen effective Plan template as the prompt, parses direct PlanMD output against that same template, and writes deterministic bounded Markdown through `SingleRequestArtifactPlan`. | +| Plan stage | The Plan runner validates the frozen effective template, emits the `planning` envelope, sends the immutable task through the frozen Plan binding with `reasoning_effort=high` and a stage-owned strict JSON schema for one-line `goal` plus bounded one-line `steps`/`verification` arrays, validates the fields, adds Markdown bullets, renders the template deterministically inside Edge, and writes the resulting Markdown through `SingleRequestArtifactPlan`. | | single-request effective templates | `execution_presets[].single_request.templates` optionally loads `plan_file`/`review_file` as bounded Markdown relative to the directory containing `edge.yaml`; absolute and empty paths, non-regular files, oversize (`>8192` bytes), non-UTF-8, and invalid grammar fail closed at load, and each file falls back to its built-in default independently. Admission freezes the effective Plan/Review pair into the binding, so a later refresh reaches only newly admitted requests. Templates select internal stage input and internal artifact shape only; caller request/response schemas are unchanged. | | Work stage | The `ornith-fast` Work runner reads the closed PLAN artifact, projects only the admitted workspace tools, and resumes the same frozen provider route after exactly correlated Node results. It rejects any Work `reasoning_effort`, malformed or multiple tool calls, and empty completion or verification evidence. | | request-owned cleanup | Node creates and inventories only `.iop/job/` internal state, cancels and waits for all active command groups, validates the exact tree without following entries, and removes matching artifacts deepest-first with non-recursive descriptor operations. Symlinks, special files, foreign devices, identity replacements, and unowned entries fail closed. User results and sibling request state are preserved. Concurrent cleanup callers receive one bounded cached typed result. | @@ -255,7 +255,7 @@ The shared `packages/go/execution` package contains provider lifecycle, registry - The service freezes the first public terminal candidate. Legacy successful results normalize to `end_turn`; output limits produce `length`; caller disconnect produces silent `cancelled`; validation/context become `invalid_request_error`; other errors become `api_error`. Buffered and SSE projectors share that policy, emit at most one terminal, and never expose private partial stage content for `length`. This completes deterministic S11 `error-cancel` evidence without changing the Edge-Node protobuf wire. S12 external Claude qualification on an approved IOP Node is complete. - The request-local internal tool loop is implemented between the coordinator and the dedicated workspace wire. Strict decode and capability checks happen before wire effects; Node results are accepted only for the one pending call and return only bounded typed fields to the same optional executor continuation. Repeated or stale identities, malformed/denied calls, exhausted immutable budgets, and cancellation terminate internally without selecting another Node or involving the HTTP caller. - Request-owned plan and review artifact access is implemented between the controller and the same dedicated workspace wire. Only `SingleRequestArtifactPlan` and `SingleRequestArtifactReview` are accepted. Artifact and model-tool callers share one serialized open attempt and the same opened cleanup gate; terminal and cancellation paths wait for in-flight artifact work before issuing exactly one cleanup. Edge bounds writes before dispatch and reads before acceptance, validates the echoed kind/operation and canonical terminal, and never reselects after a generation mismatch. Node maps the closed selectors to `plan.md` and `review.md`, validates the inventoried parent/file identity with descriptor-relative no-follow reads, and never grants the public workspace tool surface access to `.iop`. -- The private Plan stage is installed in the composite single-request executor at Edge input startup (`apps/edge/internal/input/manager.go`). Its provider codec accepts only frozen Plan options and selected dispatch facts, uses the admitted stage deadline and exact output limit, accepts only `RESPONSE_START`, zero or more `BODY`, then `END`, and projects all provider failures to a generic internal failure. The Plan prompt embeds the request's frozen effective Plan template, and the stage parses the provider's direct PlanMD output against that same frozen template before the closed PLAN artifact is written: required headings must be exact standalone lines, the documented `{{goal}}`/`{{steps}}`/`{{verification}}` inventory is closed, unresolved delimiters are rejected, and the bounded single-line goal, 2-6 step lines, and 1-3 verification lines are enforced. Caller request fields cannot select, supply, or override the admitted template. +- The private Plan stage is installed in the composite single-request executor at Edge input startup (`apps/edge/internal/input/manager.go`). Its provider codec accepts only frozen Plan options and selected dispatch facts, uses the admitted stage deadline and exact output limit, accepts only `RESPONSE_START`, zero or more `BODY`, then `END`, and projects all provider failures to a generic internal failure. The stage owns a closed strict JSON response schema with exactly a string `goal`, a string-array `steps`, and a string-array `verification`; unknown, duplicate, missing, or mistyped fields fail malformed. It enforces a single-line goal, 2-6 non-empty one-line step items, and 1-3 non-empty one-line verification items. Edge, rather than the provider, adds Markdown bullet prefixes and newlines before substituting the values into the frozen effective Plan template. Required headings remain exact standalone lines, the documented placeholder inventory is closed, and unresolved delimiters are rejected. Provider output therefore cannot vary headings, bullet formatting, or static template text, and caller request fields cannot select, supply, or override the admitted template. - The Review stage renders its internal REVIEW artifact from the request's frozen effective Review template, substituting only the model's `checks`, `verification`, and `summary` fields into the documented placeholder inventory. The template selects the internal artifact shape only: the caller-visible final response remains the model's `decision.output`, so replacing the Review template never changes the public Messages response schema. - The private Work stage is installed in the composite single-request executor at Edge input startup (`apps/edge/internal/input/manager.go`). It reads only `SingleRequestArtifactPlan`, retains only request/stage/tool identifiers while waiting for the coordinator-owned continuation, and sends no `reasoning_effort` field in an initial or resumed provider request. Its provider messages contain the immutable task, PLAN, admitted tool schemas, and bounded typed tool results; Review/repair and composite installation are active, and S12 (`claude-smoke`) qualifies the external Claude path. - The Node-private workspace request/result wire is implemented, including catalog delivery, parser registration, optional handler behavior, stable typed failures, generation-fenced dispatch, context-cancel propagation, and request cleanup. Before ready, a non-empty catalog requires a supported `darwin|linux` host and exact entry/host matching before any root open; unsupported and cross-platform catalogs fail closed while empty catalogs remain compatible. The Node installs the workspace handler before ready and cleans active requests before closing workspace authority ahead of session/store teardown. Request authority is immutable and request-local. File operations reserve `.iop`, reject symlink/mount/replaced-parent/special-file paths before effects, process bounded list batches with deterministic truncation, and use a same-parent structured write. Command execution resolves only admitted ids to fixed templates, enters the already-opened root descriptor through `fchdir`, provides only allowlisted environment entries, shares one output cap across drained stdout/stderr, and owns the complete process group through exit, timeout, context cancel, exact request/tool cancel, or request cleanup. @@ -344,7 +344,7 @@ Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 l - `go test -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — deterministic Edge overlay evidence/transition with closed label values and identity exclusion; `TestProviderHealthObservabilityDoesNotExposeSentinels` covers the sentinel/prohibited-value guard. - `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAILivenessObservationSink|TestOpenAILivenessRecoveryObservability)$'` — deterministic OpenAI recovery eligibility/results with closed label values and identifier exclusion. - `go test -count=1 ./apps/edge/internal/openai -run 'TestAnthropicSingleRequestObservation'` — deterministic single-request observation evidence: ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation, and unlabeled metric assertion. -- `go test -count=1 ./apps/edge/internal/openai -run 'TestSingleRequest(ProviderStage|PlanStage)'` — deterministic frozen provider codec and Plan stage evidence, including high reasoning, frozen-template prompt and parse authority, ordered tunnel frames, direct PlanMD validation, planning envelope, and `plan.md` artifact selection. +- `go test -count=1 ./apps/edge/internal/openai -run 'TestSingleRequest(ProviderStage|PlanStage)'` — deterministic frozen provider codec and Plan stage evidence, including high reasoning, the closed structured response schema, exact JSON field admission, ordered tunnel frames, deterministic frozen-template rendering, planning envelope, and `plan.md` artifact selection. - `go test -count=1 ./packages/go/singlerequesttemplate ./packages/go/config` — exact standalone heading/`PASS` grammar, closed placeholder inventory, the `8192`-byte boundary, and relative-only `plan_file`/`review_file` loading with per-file fallback and filesystem-kind rejection. - `go test -count=1 ./apps/edge/internal/service -run 'TestSingleRequestBinding'` and `go test -count=1 ./apps/edge/internal/openai -run 'TestSingleRequest(PresetBinding|ReviewStage)'` — admitted effective-template snapshot through clone and workspace revalidation, refresh isolation for admitted versus newly admitted requests, and the custom-template REVIEW artifact with unchanged caller-visible output. - `go test -race -count=1 ./apps/edge/internal/openai -run 'TestSingleRequestWork(Stage|ToolBridge)'` — deterministic ornith-fast Work tool loop, correlation isolation, cancellation cleanup, strict completion evidence, and Work reasoning-option absence. @@ -365,6 +365,8 @@ Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 l ## 변경 기록 +- 2026-08-12: Replaced nondeterministic free-form PlanMD generation with a stage-owned strict `goal`/`steps`/`verification` JSON response. Edge rejects unknown, duplicate, missing, mistyped, or out-of-bound fields and deterministically renders the already-frozen operator Plan template, preserving template customization and every caller-visible schema (`apps/edge/internal/openai/single_request_plan_stage.go`, `packages/go/singlerequesttemplate/template.go`). +- 2026-08-12: Replaced Markdown-bearing Plan string fields with bounded one-line string arrays and moved all bullet/newline rendering into Edge, removing the remaining provider-specific formatting dependency without changing the public Messages surface (`apps/edge/internal/openai/single_request_plan_stage.go`, `packages/go/singlerequesttemplate/template.go`). - 2026-08-09: Replaced the Plan stage's structured JSON response contract with operator-configurable bounded Markdown templates. `execution_presets[].single_request.templates` resolves `plan_file`/`review_file` relative to the directory containing `edge.yaml` only, admission freezes the effective Plan/Review pair so a refresh reaches only newly admitted requests, the Plan stage prompts with and parses direct PlanMD against the frozen template, and the Review stage renders the internal REVIEW artifact from the frozen Review template while the caller-visible final response stays the model's `decision.output` (`packages/go/singlerequesttemplate/template.go`, `packages/go/config/load.go`, `apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_plan_stage.go`, `apps/edge/internal/openai/single_request_review_stage.go`). - 2026-08-09: Qualified S12 `claude-smoke` on an approved IOP Node. One supervised Claude task with SDK retry and session-title generation disabled produced ingress `0→1`, Gemini → ornith-fast → Gemini, one `end_turn`, the exact 42-byte workspace result, stage/total timing, cleanup, and redacted evidence. - 2026-08-08: Expanded workspace runtime admission to the closed `darwin|linux` implementation set with exact catalog/host matching before root open, preserved empty-catalog compatibility, and kept Windows/unknown hosts fail-closed. diff --git a/agent-spec/testing/agent-comparison-benchmark.md b/agent-spec/testing/agent-comparison-benchmark.md index 4b24239c..b5d35c44 100644 --- a/agent-spec/testing/agent-comparison-benchmark.md +++ b/agent-spec/testing/agent-comparison-benchmark.md @@ -68,11 +68,11 @@ Claude Code, agy, Codex가 IOP를 경유해 수행하는 동일 과업을 설정 |------|------| | manifest 검증 | caller, IOP direct/preset route, model, effort, fixture, 반복 횟수, timeout, evaluator와 `agent-test/runs/` 경로를 검증하고 canonical digest를 만든다. | | 연결 preflight | Claude Code, agy, Codex의 binary/config와 IOP endpoint·auth·model·effort·stream binding을 확인하고 `ready`, `registration_required`, `implementation_gap`으로 분류한다. | -| 격리 실행과 재개 | 각 cell/repetition을 동일 checksum의 clean workspace와 fresh caller session에서 실행하며, 종료·idle·timeout·cancel·cleanup을 bounded하게 처리한다. 실패한 attempt는 덮어쓰지 않고 명시적 재개 시 새 attempt로 남긴다. | +| 격리 실행과 재개 | 각 cell/repetition을 동일 checksum의 clean workspace와 fresh caller session에서 실행하며, 제품 결과·harness 정합성·process 종료를 독립 결과로 보존한다. top-level state는 `running`, `completed`, `timed_out`, `cancelled`, `interrupted`의 controller 상태만 나타내며, 실패한 attempt는 덮어쓰지 않는다. | | 측정과 evidence | 제출, 첫 출력, 첫 파일 쓰기, model/tool/queue, finish/idle 시간을 관측 source와 함께 정규화한다. token은 보고 주체와 미제공 상태를 보존하며 임의 추정값을 authoritative 값과 섞지 않는다. | -| 웹 자동 검증 | 필수 HTML/CSS/JS와 로컬 이미지, 외부 asset 금지, desktop/mobile render, console/asset 오류, 반응형·접근성 gate와 screenshot을 확인한다. | +| 웹 자동 검증 | product/harness 성공 여부와 무관하게 모든 terminal workspace에서 필수 HTML/CSS/JS와 로컬 이미지, 외부 asset 금지, desktop/mobile render, console/asset 오류, 반응형·접근성 gate와 screenshot을 확인한다. | | 익명 품질 채점 | 필수 자동 gate를 통과한 결과만 identity를 가린 뒤 manifest에 고정된 fresh evaluator로 100점 rubric을 평가한다. 부적격 결과는 `unscored`, 평가 실패는 `scoring_failed`로 남기며 retry는 새 scoring attempt id를 사용한다. | -| 상태와 보고 | `validate`, `preflight`, `run`, `resume`, `status`, `score`, `report` CLI를 제공하고, 성공·실패·blocked·unscored·scoring_failed·동점을 raw evidence 포인터와 함께 deterministic Markdown으로 만든다. | +| 상태와 보고 | `validate`, `preflight`, `run`, `resume`, `status`, `score`, `report` CLI를 제공하고, controller/product/harness/process/artifact/scoring 축과 동점을 raw evidence 포인터와 함께 deterministic Markdown으로 만든다. | ## 범위 @@ -105,6 +105,10 @@ flowchart LR - manifest의 matrix cell은 stable id, caller, IOP route kind, requested/effective model과 effort를 가진다. unsupported alias나 effort는 다른 값으로 대체하지 않고 fail-closed한다. - run state는 `agent-test/runs///` 아래에 격리되며 manifest digest가 다른 상태를 재개하지 않는다. - preflight는 scored attempt가 아니며, 실행 중 실패·timeout·cancel과 scoring 실패는 기존 attempt를 수정하지 않고 보존한다. +- caller parser는 raw terminal 문자열 대신 `CallerEvent(finish|idle)`, `CallerTerminal(succeeded|failed)`와 typed metric만 반환한다. Claude result가 마지막 active assistant snapshot을 직접 완성하면 adapter가 typed finish와 idle을 함께 투영하고, assistant가 이미 finish를 냈으면 result는 idle만 투영한다. synthetic API error와 agy ERROR result는 `product=failed`, `harness=passed`가 될 수 있으며 parser malformed는 `product=unknown`, `harness=failed`로 구분한다. +- durable lifecycle/measurement/attempt evidence는 `product`, `harness`, `process` 객체를 그대로 저장한다. `run` exit 0과 scoring eligibility는 product succeeded, harness passed, process exited/exit 0/no signal, artifact passed를 모두 요구한다. +- 배포 qualification은 동일 clean source에서 5-cell direct manifest를 unscored canary로 한 번 실행해 네 gate 5/5를 확인한 뒤 fresh C01-C09 preflight `ready=9`까지만 수행한다. hybrid 또는 scored C01-C09 실행은 후속 승인 전에는 할당하지 않는다. +- lifecycle supervisor는 exit watcher와 출력 reader를 join한 뒤 하나의 child return code를 동결해 lifecycle result와 cleanup receipt가 동일한 exit/signal을 갖게 한다. 불일치 evidence는 resume에서 fail-closed한다. - raw credential과 private endpoint는 tracked manifest, event, log, screenshot과 report에 기록하지 않는다. - report는 run state의 canonical evidence에서 생성되며 성공하지 않은 결과를 0점으로 변환하거나 동점에 임의 순위를 부여하지 않는다. @@ -125,4 +129,6 @@ flowchart LR ## 변경 기록 +- 2026-08-12: caller terminal을 closed typed observation으로 바꾸고 product/harness/process 결과, failure-inclusive artifact gate, 독립 CLI/report/scoring gate와 direct-first qualification을 구현했다. +- 2026-08-12: official Claude result-direct/API-error 및 agy ERROR terminal을 lifecycle 계약에 맞게 분리하고, timeout cleanup result/receipt가 같은 child exit snapshot을 사용하도록 동기화했다. - 2026-08-12: `[bench-01]` 종료 감사에서 확인한 421개 benchmark test, manifest/CLI 계약과 구현 evidence를 기준으로 생성했다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G05_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G05_2.log new file mode 100644 index 00000000..86dc909a --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G05_2.log @@ -0,0 +1,260 @@ + + +# Code Review Reference - REVIEW_API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=2, tag=REVIEW_API + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`. +- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0. +- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean. +- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged. +- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G05.md` → `code_review_cloud_G05_2.log` and `PLAN-cloud-G05.md` → `plan_cloud_G05_2.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| REVIEW_API-1 Require complete token matches | [x] | +| REVIEW_API-2 Re-run the benchmark regression boundary | [x] | + +## Implementation Checklist + +- [x] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior. +- [x] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G05_2.log`. +- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G05_2.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path. +- [x] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +없음. PLAN이 선택한 두 지점만 수정했고 명령 교체나 검증 생략 없이 지정된 검증을 그대로 실행했다. + +## Key Design Decisions + +- 공유 token helper의 의미를 별도 seed 전용 validator 없이 `Pattern.fullmatch`로 일관되게 강화했다. +- 기존 schema pattern, seeded rank, canonical digest/omission 처리, slot allocation과 fixture는 변경하지 않았다. +- terminal newline seed는 기존 schema/loader parity corpus에 추가해 양쪽 경계가 같은 값을 거부하도록 고정했다. + +## Reviewer Checkpoints + +- Confirm `_require_pattern` uses a complete-string match and rejects `"seed-a\n"` while valid 64-character seeds remain accepted. +- Confirm the existing schema/loader parity corpus includes the terminal-newline value and asserts `ManifestValidationError`. +- Confirm the schema, seeded rank, canonical digest/omission behavior, and `RunStore.slots` remain unchanged. +- Confirm fresh focused and combined regression commands pass without invoking external providers. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### REVIEW_API-1 Focused Verification + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds +``` + +_Actual output:_ + +```text +exit_code: 0 +stdout: (none) +stderr: +. +---------------------------------------------------------------------- +Ran 1 test in 0.005s + +OK +``` + +### REVIEW_API-1 Manifest Regression + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +```text +exit_code: 0 +stdout: (none) +stderr: +............................................................................................................... +---------------------------------------------------------------------- +Ran 111 tests in 1.224s + +OK +``` + +### REVIEW_API-2 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +```text +manifest validation loop: +exit_code: 0 +stdout: +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +stderr: (none) + +combined unittest: +exit_code: 0 +stdout: (none) +stderr: +........................................................................................................................................................................................ +---------------------------------------------------------------------- +Ran 184 tests in 46.261s + +OK + +git diff --check: +exit_code: 0 +stdout: (none) +stderr: (none) +``` + +### Reviewer Fresh Verification (2026-08-12) + +The reviewer reran the required commands from `/config/workspace/iop-s0` without invoking external providers. + +```text +focused parity test: +exit_code: 0 +. +---------------------------------------------------------------------- +Ran 1 test in 0.008s + +OK + +full manifest suite: +exit_code: 0 +............................................................................................................... +---------------------------------------------------------------------- +Ran 111 tests in 1.381s + +OK + +manifest validation loop: +exit_code: 0 +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid + +combined unittest: +exit_code: 0 +........................................................................................................................................................................................ +---------------------------------------------------------------------- +Ran 184 tests in 50.511s + +OK + +git diff --check: +exit_code: 0 +stdout: (none) +stderr: (none) +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +### Overall Verdict + +PASS + +### Dimension Assessment + +| Dimension | Result | Evidence | +|-----------|--------|----------| +| Correctness | Pass | `_require_pattern` now requires `Pattern.fullmatch`, so the terminal-newline seed is rejected before ordering or digest processing. | +| Completeness | Pass | Both REVIEW_API items are implemented and the selected R1 fix is present without schema, ordering, digest, slot, or fixture changes in this follow-up. | +| Test coverage | Pass | The existing schema/loader parity corpus now includes `"seed-a\n"`; the focused test, 111 manifest tests, and 184 combined tests pass freshly. | +| API contract | Pass | Loader behavior now agrees with the tracked seed pattern's full-string interpretation while valid shipped manifests remain accepted. | +| Code quality | Pass | The fix is localized to the shared token helper and reuses the existing parity test without duplicate validation logic. | +| Implementation deviation | Pass | The implementation follows the reviewer-selected files, symbols, behavior, and verification commands with no scope expansion. | +| Verification trust | Pass | Fresh reviewer output matches the implementation-owned evidence and `git diff --check` is clean. | +| Spec conformance | Pass | The reusable seed grammar remains consistent with SDD S03 and `milestone-task=matrix-lock`; concrete C01-C09 manifest evidence remains outside this packet. | + +### Findings + +None. Required 0, Suggested 0, Nit 0. + +### Routing Signals + +- `review_rework_count=1` +- `evidence_integrity_failure=false` + +### Next Step + +Write `complete.log`, archive this PASS task with `milestone-task=matrix-lock` metadata preserved, and emit the runtime completion event without modifying the roadmap. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log new file mode 100644 index 00000000..35b3327d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log @@ -0,0 +1,131 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=0, tag=API + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-local-G08.md` → `plan_local_G08_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Add one canonical seeded-order contract | [ ] | +| API-2 Bind slot allocation to canonical matrix order | [ ] | +| API-3 Preserve the complete benchmark baseline | [ ] | + +## Implementation Checklist + +- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases. +- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source. +- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_0.log`. +- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm omission preserves legacy id ordering and does not inject a default into canonical bytes. +- Confirm explicit seeds use the documented domain-separated rank and stable cell-id tie-breaker. +- Confirm the JSON schema and Python loader reject the same invalid seed shapes. +- Confirm the slot allocator has no independent sorting rule. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### API-1 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +### API-2 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +``` + +_Actual output:_ + +### API-3 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log new file mode 100644 index 00000000..4075fb4c --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log @@ -0,0 +1,243 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=1, tag=API + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found one stale dependency statement: it named a nonexistent integration child and incorrectly implied that child depended directly on both indices 01 and 02. +- Replan keeps the implementation contract unchanged and corrects the DAG to `01 -> 03+01`, `02` independent, and `(02,03) -> 04+02,03`. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-local-G08.md` → `plan_local_G08_1.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Add one canonical seeded-order contract | [x] | +| API-2 Bind slot allocation to canonical matrix order | [x] | +| API-3 Preserve the complete benchmark baseline | [x] | + +## Implementation Checklist + +- [x] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases. +- [x] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source. +- [x] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`. +- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +None. The implementation followed the selected owner, files, algorithm, and verification commands. Production `RunStore.slots` remains unchanged. + +## Key Design Decisions + +- Added `execution_order_seed: str | None = None` after `Manifest.digest`, so existing constructors remain compatible. +- Reused the cell-id token grammar for loader and schema validation. The 64-character boundary is accepted; empty, oversized, non-token, and non-string values fail closed. +- Applied the exact `iop-benchmark-order-v1\0\0` SHA-256 rank with cell id as the collision tie-breaker. +- Included the seed in canonical serialization only when explicitly present. Omission retains pipeline-v2 lexical cell ordering and legacy canonical bytes. +- Kept slot allocation single-sourced from `Manifest.matrix`; the new regression proves repetitions remain adjacent within each seeded cell. + +## Reviewer Checkpoints + +- Confirm omission preserves legacy id ordering and does not inject a default into canonical bytes. +- Confirm explicit seeds use the documented domain-separated rank and stable cell-id tie-breaker. +- Confirm the JSON schema and Python loader reject the same invalid seed shapes. +- Confirm the slot allocator has no independent sorting rule. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### API-1 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +```text +............................................................................................................... +---------------------------------------------------------------------- +Ran 111 tests in 1.071s + +OK +``` + +### API-2 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +``` + +_Actual output:_ + +```text +. +---------------------------------------------------------------------- +Ran 1 test in 0.001s + +OK +``` + +### API-3 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +```text +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +....................................................................................................................................................................................... +---------------------------------------------------------------------- +Ran 184 tests in 46.125s + +OK +``` + +`git diff --check` produced no output and exited 0. + +### Reviewer Fail-Closed Reproducer + +```bash +python3 - <<'PY' +import json +import re +import tempfile +from pathlib import Path +from scripts.agent_benchmark.manifest_test import ( + _load_tmp_manifest, + _make_minimal_manifest_dict, + _write_tmp_manifest, +) +seed = "seed-a\n" +schema = json.loads(Path("scripts/fixtures/agent-comparison-benchmark-manifest.schema.json").read_text(encoding="utf-8")) +pattern = schema["properties"]["execution_order_seed"]["pattern"] +with tempfile.TemporaryDirectory() as tmp: + manifest = _load_tmp_manifest( + _write_tmp_manifest( + Path(tmp), + _make_minimal_manifest_dict(execution_order_seed=seed), + ) + ) +print(f"schema_fullmatch={re.fullmatch(pattern, seed) is not None}") +print(f"loader_accepted={manifest.execution_order_seed!r}") +PY +``` + +_Actual output:_ + +```text +schema_fullmatch=False +loader_accepted='seed-a\n' +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +### Overall Verdict + +FAIL + +### Dimension Assessment + +| Dimension | Result | Evidence | +|-----------|--------|----------| +| Correctness | Fail | A trailing-newline seed bypasses the intended bounded token grammar and reaches canonical ordering/digest code. | +| Completeness | Fail | API-1 requires invalid non-token seeds to fail closed, but that boundary remains open. | +| Test coverage | Fail | The invalid-seed corpus omits the Python `$`/trailing-newline boundary and therefore passes while the defect remains. | +| API contract | Fail | The tracked schema pattern rejects the reproduced seed under the parity test's full-string interpretation, while the loader accepts it. | +| Code quality | Pass | The seeded ordering and conditional serialization are localized and readable. | +| Implementation deviation | Pass | The implementation otherwise follows the planned owner, algorithm, files, and slot-allocation boundary. | +| Verification trust | Fail | Fresh focused and full suites pass, but the implementation claim that non-token seed values fail closed is contradicted by the reviewer reproducer. | +| Spec conformance | Pass | The explicit seed, deterministic order, digest binding, and slot consumption otherwise conform to SDD S03's reusable order contract. | + +### Findings + +- **Required R1 — The loader accepts an `execution_order_seed` ending in `\n`.** + - **Evidence:** The fresh reviewer reproducer reports `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. `scripts/agent_benchmark/manifest.py:729` routes the seed through `_require_pattern`, while `scripts/agent_benchmark/manifest.py:194` uses `pattern.match(s)`; Python `$` permits a match immediately before one terminal newline. This violates API-1's explicit non-token fail-closed criterion and the schema/loader parity checkpoint. + - **Root Cause:** The shared manifest token validator tests only whether a prefix matches an anchored expression instead of requiring the entire string to match. The new seed field reused that helper, and `scripts/agent_benchmark/manifest_test.py:1492` did not include the terminal-newline variant in its invalid corpus. + - **Selected Fix:** Change `_require_pattern` in `scripts/agent_benchmark/manifest.py` to use `pattern.fullmatch(s)`, then add `"seed-a\n"` to `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` in `scripts/agent_benchmark/manifest_test.py`. Keep the schema and seeded-order algorithm unchanged. Accept only after the named parity test, full manifest suite, three example validations, manifest/attempt/connectivity regression suite, and `git diff --check` all pass freshly. + +No Suggested or Nit findings. + +### Routing Signals + +- `review_rework_count=1` +- `evidence_integrity_failure=true` + +### Next Step + +Create the mandatory reviewer-closed follow-up plan for Required R1, rerun isolated task routing, archive this pair, and materialize the routed follow-up pair. Do not write `complete.log`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log new file mode 100644 index 00000000..a5914b84 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log @@ -0,0 +1,41 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/01_execution_order_contract + +## 완료 일시 + +2026-08-12 + +## 요약 + +실행 순서 seed 계약과 terminal-newline fail-closed 보완을 공식 리뷰 2회(FAIL 1회, PASS 1회)로 완료했으며 최종 판정은 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | FAIL | seeded order/digest/slot 계약은 구현됐으나 `"seed-a\n"`이 loader token 검증을 우회하는 Required R1을 확인했다. | +| `plan_cloud_G05_2.log` | `code_review_cloud_G05_2.log` | PASS | 공유 pattern helper를 full-string 검증으로 강화하고 terminal-newline parity regression을 추가해 R1을 닫았다. | + +## 구현/정리 내용 + +- optional `execution_order_seed`가 manifest canonical order와 digest에 결정론적으로 반영되고 omission 호환성과 `RunStore.slots` 순서가 유지된다. +- `_require_pattern`이 `Pattern.fullmatch`를 사용해 seed를 포함한 manifest token 전체 문자열을 fail-closed 검증한다. +- schema/loader parity corpus에 `"seed-a\n"`을 추가했으며 schema, seeded ranking, digest, slot allocation과 shipped fixture는 follow-up 범위에서 변경하지 않았다. + +## 최종 검증 + +- `python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` - PASS; 1 test, `OK`. +- `python3 -m unittest scripts.agent_benchmark.manifest_test` - PASS; 111 tests, `OK`. +- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`. +- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 184 tests, `OK`. +- `git diff --check` - PASS; 출력 없음. +- 공개 `validate` entrypoint를 실제 fixture로 검증했다. 외부 provider 실행과 scored benchmark full-cycle은 이 parser-only packet의 검증 범위가 아니다. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- dependent `04+02,03_locked_benchmark_manifest` packet이 실제 C01-C09 immutable manifest evidence를 제공해야 하며, 이 완료 로그 단독으로 `matrix-lock` 전체 완료를 주장하지 않는다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_cloud_G05_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_cloud_G05_2.log new file mode 100644 index 00000000..2694ae3b --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_cloud_G05_2.log @@ -0,0 +1,198 @@ + + +# Plan - Close Execution-Order Seed Token Validation + +## For the Implementing Agent + +Filling the implementation-owned sections in `CODE_REVIEW-cloud-G05.md` is the mandatory final implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The seeded execution-order implementation is otherwise complete, but the official review found one fail-closed boundary defect. Python's anchored `$` plus `Pattern.match` accepts a single trailing newline, so the loader can accept a seed that the declared full token grammar rejects. This follow-up applies the reviewer-selected full-string validation fix and locks the exact regression. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`. +- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0. +- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean. +- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged. +- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence. + +## Finding Resolution Map + +| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands | +|---------|-------------------|------------|--------------|------|----------------------|---------------------| +| Required R1 | Fresh reproducer: `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. | `_require_pattern` uses `pattern.match(s)`, whose `$` can stop before one terminal newline; the invalid seed corpus omits that value. | Use `pattern.fullmatch(s)` in `scripts/agent_benchmark/manifest.py` and add `"seed-a\n"` to the invalid seed corpus in `scripts/agent_benchmark/manifest_test.py`. Keep schema and ordering behavior unchanged. | `direct-fix` | The validator implementation and regression input change, so verification no longer repeats against the defective precondition. | Named parity test; full manifest suite; three example validations; manifest/attempt/connectivity suite; `git diff --check`. | + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log` +- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, lock released, no user review. +- First-line scope: `milestone-task=matrix-lock`. +- Target: S03 requires the immutable scored manifest to retain a deterministic execution-order seed; the Evidence Map links that evidence to `matrix-lock`. +- This checklist repairs the reusable seed grammar needed by S03. The dependent locked-manifest packet still owns concrete C01-C09 evidence. + +### Verification Context + +- No external handoff or external runner is required. Repository-native local verification is authoritative. +- Fresh review evidence: 111 manifest tests passed, the named slot test passed, three shipped manifests validated, the combined suite passed 184 tests, and `git diff --check` exited 0. +- The focused reproducer accepted `"seed-a\n"` in the loader while the schema pattern's full-string check returned false. +- Constraints: preserve valid seed behavior, omission compatibility, seeded rank, digest serialization, `RunStore.slots`, fixture bytes, and public schema. +- Confidence: high. The defect is localized to one shared pattern helper and one missing regression value; all helper patterns are intended as whole-string token validators. +- External verification preflight: not applicable. + +### Test Coverage Gaps + +- Existing seed tests cover empty, uppercase, dotted, oversized, boolean, and null values. +- Missing: the terminal-newline value that distinguishes `Pattern.match` from `Pattern.fullmatch`. +- Add that value to the existing schema/loader parity corpus; no new test class or external fixture is required. + +### Symbol References + +- No symbol is renamed or removed. +- `_require_pattern` remains the shared private validator for fixture version, model/effort/route tokens, viewport ids, cell ids, and `execution_order_seed`. The change tightens those existing whole-string grammars consistently. + +### Split Judgment + +This is one compact atomic bug fix: the validator semantic change and its regression must land together. It remains child `01_execution_order_contract`; no new dependency or sibling split is introduced. + +### Scope Rationale + +- Do not change `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`; its seed grammar is already correct. +- Do not change seeded ranking, canonical serialization/digest, `RunStore.slots`, attempt state, example manifests, contracts, living spec, or roadmap. +- Do not broaden the packet into generic path normalization; only the shared regex token helper and the exact seed regression are writable production/test targets. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all true; `ownership_closed=true` through direct-fix R1. +- Build scores: `1+0+2+1+1=G05`; base `local-fit`, final `recovery-boundary` because `review_rework_count=1` and `evidence_integrity_failure=true`; lane `cloud`, catalog `worker/cloud/G05`, filename `PLAN-cloud-G05.md`. +- Review closures are all true; scores `1+0+2+1+1=G05`; route `official-review`, lane `cloud`, catalog `review/cloud/G05`, filename `CODE_REVIEW-cloud-G05.md`. +- `large_indivisible_context=false`; matched loop risks: `boundary_contract`, `structured_interpretation` (`count=2`); `risk_boundary_matched=false`; `recovery_boundary_matched=true`; no capability gap. + +## Implementation Checklist + +- [ ] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior. +- [ ] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [REVIEW_API-1] Require complete token matches + +**Problem** + +At `scripts/agent_benchmark/manifest.py:192-198`, `_require_pattern` calls `pattern.match(s)`. With the anchored patterns used by the manifest, Python accepts a match immediately before a final newline, so `execution_order_seed="seed-a\n"` reaches ordering and digest code even though it is not a valid bounded token. + +**Solution** + +Apply the official review's selected fix exactly. + +Before (`scripts/agent_benchmark/manifest.py:192-198`): + +```python +def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str: + s = _require_str(value, field_name) + if not pattern.match(s): + raise ManifestValidationError( + f"field '{field_name}' does not match required pattern" + ) + return s +``` + +After: + +```python +def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str: + s = _require_str(value, field_name) + if pattern.fullmatch(s) is None: + raise ManifestValidationError( + f"field '{field_name}' does not match required pattern" + ) + return s +``` + +Add `"seed-a\n"` to the existing `invalid_seeds` tuple in `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds`. Do not change the schema pattern or add an alternate validator. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/manifest.py`: replace prefix-capable pattern matching with full-string matching. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add the terminal-newline seed to the existing parity corpus. + +**Test Strategy** + +Write the regression in the existing named parity test. It must prove the tracked pattern's full-string interpretation rejects the seed and `load_manifest` raises `ManifestValidationError` for the same value. Existing valid/boundary tests cover compatibility. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +Expected: both commands exit 0; the focused test rejects `"seed-a\n"` and the full manifest suite remains green. + +### [REVIEW_API-2] Re-run the benchmark regression boundary + +**Problem** + +`_require_pattern` is shared by multiple manifest token fields, so the one-line fix must be checked against the full loader and the benchmark consumers that bind the manifest digest and matrix order. + +**Solution** + +Run the exact fresh commands below after REVIEW_API-1. Do not alter product behavior or replace the failing value with a sanitized seed. + +**Modified Files and Checklist** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md`: record fresh command output and implementation notes. + +**Test Strategy** + +No additional test file is needed beyond REVIEW_API-1. The shipped manifests and combined deterministic suite are the regression oracle; no external provider process may run. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: three validation success lines, every test reports `OK`, and `git diff --check` is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/manifest.py` | REVIEW_API-1 | +| `scripts/agent_benchmark/manifest_test.py` | REVIEW_API-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md` | REVIEW_API-2 | + +## Final Verification + +Run from `/config/workspace/iop-s0`. Fresh output is required; cached or summarized output is not accepted. + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds +python3 -m unittest scripts.agent_benchmark.manifest_test +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: all commands exit 0; the newline seed is rejected, all three manifests validate, all deterministic benchmark tests report `OK`, and the whitespace check is silent. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. + diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log new file mode 100644 index 00000000..b9e6d07a --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log @@ -0,0 +1,230 @@ + + +# Plan - Seeded Benchmark Execution Order Contract + +## For the Implementing Agent + +Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The approved benchmark SDD requires an immutable execution-order seed for C01-C09, but the loader currently canonicalizes every matrix only by cell id. This packet adds a backward-compatible optional seed whose deterministic order is part of the normalized manifest and is consumed by the existing slot allocator. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=matrix-lock`. +- Target scenario: S03 requires `repetitions=1`, an execution-order seed, fresh session/setup-cache policy, timeout, and expected bindings in an immutable scored manifest. +- Evidence Map row: S03 requires an immutable scored manifest and order seed, linked to `matrix-lock` evidence. +- This packet implements only the reusable seed/order contract. The dependent locked-manifest packet supplies the concrete nine-cell S03 evidence. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the loader/schema/tests above and from fresh commands run at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`. +- Current baseline passed all three example `validate` commands and `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` (`Ran 178 tests`, `OK`). +- `Manifest.matrix` is sorted by id at `manifest.py:625-627`; `RunStore.slots` iterates that tuple at `attempts.py:679-681`. Therefore the loader can own the seeded order without a second scheduler or changes to run-state code. +- The manifest schema is a public tracked config contract, so normal, omission-compatibility, permutation, invalid-token, digest, and slot-order tests are required. Fresh test output is required; cached output is not accepted. +- No external verification is required for this packet. + +### Test Coverage Gaps + +- Existing tests cover cell-id canonicalization and repetitions, but no field represents an execution-order seed. +- No test proves that two differently ordered JSON inputs with the same explicit seed produce the same matrix order and digest. +- No test proves that omitted seed behavior and existing example digests remain backward compatible. +- No test binds seeded loader order to the slot sequence. + +### Symbol References + +- No symbol is renamed or removed. +- `Manifest(...)` constructor call sites in `manifest.py` and tests must be updated only as required by the new defaulted field; keep the field optional to avoid unrelated call-site churn. + +### Split Judgment + +- This is split child `01_execution_order_contract`. Its stable contract is: one optional seed deterministically canonicalizes `Manifest.matrix`, enters the digest only when explicitly present, and is consumed unchanged by `RunStore.slots`. +- PASS evidence is loader/schema regression coverage plus the slot-order test. +- It is independent of `02_route_preflight_contract`; neither packet claims the other's files. `03+01,02_locked_benchmark_manifest` depends on both. + +### Scope Rationale + +- Do not create the C01-C09 manifest here; the dependent integration packet owns it. +- Do not change preflight observation coverage, caller adapters, run-record schemas, pipeline version, fixture bytes, contracts, or IOP configuration. +- Do not reorder manifests that omit the field and do not rewrite the existing example manifests solely to exercise an optional field. + +### Final Routing + +- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`. +- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Implementation Checklist + +- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases. +- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source. +- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Add one canonical seeded-order contract + +**Problem** + +`Manifest` has no seed field (`manifest.py:112-126`), the top-level allowlist accepts only `repetitions` as optional (`manifest.py:681-694`), and `_validate_matrix` always sorts by id (`manifest.py:611-627`). Consequently S03 cannot persist or reproduce a deliberately seeded C01-C09 order. + +**Solution** + +Keep pipeline version 2 backward compatible. Add `execution_order_seed: str | None = None` to `Manifest`, accept it as an optional top-level field using the existing cell-id token constraints, and add the same optional constraint to the JSON schema. When absent, preserve exact cell-id sorting and omit the field from canonical serialization so prior normalized digests remain unchanged. When present, order cells by a fixed domain-separated SHA-256 rank with cell id as the collision tie-breaker, include the seed in canonical serialization, and therefore bind it into `digest_manifest_and_resolved_inputs`. + +Before (`scripts/agent_benchmark/manifest.py:625-627`): + +```python +# Sort cells by id for canonical ordering +cells.sort(key=lambda c: c.id) +return tuple(cells) +``` + +After: + +```python +def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]: + rank = hashlib.sha256( + b"iop-benchmark-order-v1\0" + seed.encode("ascii") + b"\0" + cell.id.encode("ascii") + ).digest() + return rank, cell.id + +# No seed preserves the pipeline-v2 cell-id order. An explicit seed is stable +# across JSON input permutations and platforms. +cells.sort( + key=(lambda cell: cell.id) + if execution_order_seed is None + else (lambda cell: _execution_order_key(execution_order_seed, cell)) +) +``` + +Construct the canonical dict first, then conditionally add `execution_order_seed`; never serialize a synthetic default. Add comments documenting the domain string and compatibility behavior. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/manifest.py`: model, validate, normalize, order, and digest the explicit seed. +- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: accept only the optional bounded seed token while retaining `additionalProperties: false`. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add exact regression tests named `test_execution_order_seed_is_canonical_and_permutation_stable`, `test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract`, and schema/loader invalid-seed parity coverage. + +**Test Strategy** + +Write tests. Use at least three cells whose SHA-256 seeded order differs from lexical id order. Assert input permutation independence, same-seed repeatability, different-seed digest/order differentiation, invalid empty/oversize/non-token rejection, and omission compatibility. Existing example validation must remain unchanged. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +Expected: all manifest loader/schema/digest tests pass freshly. + +### [API-2] Bind slot allocation to the canonical matrix order + +**Problem** + +`RunStore.slots` already iterates `manifest.matrix` (`attempts.py:679-681`), but the current test only checks repetitions for a one-cell manifest (`attempts_test.py:457`). There is no regression proof that a seeded multi-cell order reaches attempt allocation without being re-sorted. + +**Solution** + +Do not modify `RunStore.slots`. Extend the attempts-test manifest helper to accept an explicit seed and multiple cells, then assert the exact sequence is the loader's seeded `Manifest.matrix` order with repetitions `1..N` adjacent for each cell. + +Before (`scripts/agent_benchmark/attempts.py:679-681`): + +```python +@staticmethod +def slots(manifest: Manifest) -> tuple[Slot, ...]: + return tuple(Slot(cell.id, repetition) for cell in manifest.matrix for repetition in range(1, manifest.repetitions + 1)) +``` + +After: keep this production code unchanged; lock its behavior with `test_slots_follow_seeded_manifest_order_before_repetitions`. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/attempts_test.py`: add the seeded multi-cell slot-order regression without changing production attempts code. + +**Test Strategy** + +Write the named test. Assert both `(cell_id, repetition)` tuples and that the first occurrence order equals `manifest.matrix`. This prevents a future alphabetical sort in the store from silently defeating the seed. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +``` + +Expected: the exact seeded slot sequence passes. + +### [API-3] Preserve the complete benchmark baseline + +**Problem** + +The field touches canonical JSON and a public schema; focused tests alone could miss example or integration drift. + +**Solution** + +Run fresh benchmark suites and all shipped validation fixtures after the code and tests are complete. Do not accept cached output. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G08.md`: record actual implementation decisions and full command output. + +**Test Strategy** + +No additional test file beyond API-1/API-2. The full existing suite is the regression oracle. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: three `ok: manifest is valid` lines, all tests `OK`, and no whitespace errors. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/manifest.py` | API-1 | +| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 | +| `scripts/agent_benchmark/manifest_test.py` | API-1 | +| `scripts/agent_benchmark/attempts_test.py` | API-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md` | API-3 | + +## Final Verification + +Run from `/config/workspace/iop-s0`; fresh output is required and cached test output is not acceptable. + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: every command exits 0, the three manifests validate, all benchmark tests report `OK`, and `git diff --check` is silent. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log new file mode 100644 index 00000000..40d01991 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log @@ -0,0 +1,237 @@ + + +# Plan - Seeded Benchmark Execution Order Contract + +## For the Implementing Agent + +Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The approved benchmark SDD requires an immutable execution-order seed for C01-C09, but the loader currently canonicalizes every matrix only by cell id. This packet adds a backward-compatible optional seed whose deterministic order is part of the normalized manifest and is consumed by the existing slot allocator. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found one stale dependency statement: it named a nonexistent integration child and incorrectly implied that child depended directly on both indices 01 and 02. +- Replan keeps the implementation contract unchanged and corrects the DAG to `01 -> 03+01`, `02` independent, and `(02,03) -> 04+02,03`. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=matrix-lock`. +- Target scenario: S03 requires `repetitions=1`, an execution-order seed, fresh session/setup-cache policy, timeout, and expected bindings in an immutable scored manifest. +- Evidence Map row: S03 requires an immutable scored manifest and order seed, linked to `matrix-lock` evidence. +- This packet implements only the reusable seed/order contract. The dependent locked-manifest packet supplies the concrete nine-cell S03 evidence. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the loader/schema/tests above and from fresh commands run at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`. +- Current baseline passed all three example `validate` commands and `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` (`Ran 178 tests`, `OK`). +- `Manifest.matrix` is sorted by id at `manifest.py:625-627`; `RunStore.slots` iterates that tuple at `attempts.py:679-681`. Therefore the loader can own the seeded order without a second scheduler or changes to run-state code. +- The manifest schema is a public tracked config contract, so normal, omission-compatibility, permutation, invalid-token, digest, and slot-order tests are required. Fresh test output is required; cached output is not accepted. +- No external verification is required for this packet. + +### Test Coverage Gaps + +- Existing tests cover cell-id canonicalization and repetitions, but no field represents an execution-order seed. +- No test proves that two differently ordered JSON inputs with the same explicit seed produce the same matrix order and digest. +- No test proves that omitted seed behavior and existing example digests remain backward compatible. +- No test binds seeded loader order to the slot sequence. + +### Symbol References + +- No symbol is renamed or removed. +- `Manifest(...)` constructor call sites in `manifest.py` and tests must be updated only as required by the new defaulted field; keep the field optional to avoid unrelated call-site churn. + +### Split Judgment + +- This is split child `01_execution_order_contract`. Its stable contract is: one optional seed deterministically canonicalizes `Manifest.matrix`, enters the digest only when explicitly present, and is consumed unchanged by `RunStore.slots`. +- PASS evidence is loader/schema regression coverage plus the slot-order test. +- It is independent of `02_route_preflight_contract`; neither packet claims the other's files. Only `03+01_rubric_version_contract` depends directly on this packet. The final `04+02,03_locked_benchmark_manifest` packet depends on indices 02 and 03. + +### Scope Rationale + +- Do not create the C01-C09 manifest here; the dependent integration packet owns it. +- Do not change preflight observation coverage, caller adapters, run-record schemas, pipeline version, fixture bytes, contracts, or IOP configuration. +- Do not reorder manifests that omit the field and do not rewrite the existing example manifests solely to exercise an optional field. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`. +- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Implementation Checklist + +- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases. +- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source. +- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Add one canonical seeded-order contract + +**Problem** + +`Manifest` has no seed field (`manifest.py:112-126`), the top-level allowlist accepts only `repetitions` as optional (`manifest.py:681-694`), and `_validate_matrix` always sorts by id (`manifest.py:611-627`). Consequently S03 cannot persist or reproduce a deliberately seeded C01-C09 order. + +**Solution** + +Keep pipeline version 2 backward compatible. Append `execution_order_seed: str | None = None` after the existing required `digest` field in `Manifest` so dataclass default ordering stays valid, accept it as an optional top-level field using the existing cell-id token constraints, and add the same optional constraint to the JSON schema. Pass an explicit seed into both loader-created `Manifest` values before digesting. When absent, preserve exact cell-id sorting and omit the field from canonical serialization so prior normalized digests remain unchanged. When present, order cells by a fixed domain-separated SHA-256 rank with cell id as the collision tie-breaker, include the seed in canonical serialization, and therefore bind it into `digest_manifest_and_resolved_inputs`. + +Before (`scripts/agent_benchmark/manifest.py:625-627`): + +```python +# Sort cells by id for canonical ordering +cells.sort(key=lambda c: c.id) +return tuple(cells) +``` + +After: + +```python +def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]: + rank = hashlib.sha256( + b"iop-benchmark-order-v1\0" + seed.encode("ascii") + b"\0" + cell.id.encode("ascii") + ).digest() + return rank, cell.id + +# No seed preserves the pipeline-v2 cell-id order. An explicit seed is stable +# across JSON input permutations and platforms. +cells.sort( + key=(lambda cell: cell.id) + if execution_order_seed is None + else (lambda cell: _execution_order_key(execution_order_seed, cell)) +) +``` + +Construct the canonical dict first, then conditionally add `execution_order_seed`; never serialize a synthetic default. Add comments documenting the domain string and compatibility behavior. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/manifest.py`: model, validate, normalize, order, and digest the explicit seed. +- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: accept only the optional bounded seed token while retaining `additionalProperties: false`. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add exact regression tests named `test_execution_order_seed_is_canonical_and_permutation_stable`, `test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract`, and schema/loader invalid-seed parity coverage. + +**Test Strategy** + +Write tests. Use at least three cells whose SHA-256 seeded order differs from lexical id order. Assert input permutation independence, same-seed repeatability, different-seed digest/order differentiation, invalid empty/oversize/non-token rejection, and omission compatibility. Existing example validation must remain unchanged. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +``` + +Expected: all manifest loader/schema/digest tests pass freshly. + +### [API-2] Bind slot allocation to the canonical matrix order + +**Problem** + +`RunStore.slots` already iterates `manifest.matrix` (`attempts.py:679-681`), but the current test only checks repetitions for a one-cell manifest (`attempts_test.py:457`). There is no regression proof that a seeded multi-cell order reaches attempt allocation without being re-sorted. + +**Solution** + +Do not modify `RunStore.slots`. Extend the attempts-test manifest helper to accept an explicit seed and multiple cells, then assert the exact sequence is the loader's seeded `Manifest.matrix` order with repetitions `1..N` adjacent for each cell. + +Before (`scripts/agent_benchmark/attempts.py:679-681`): + +```python +@staticmethod +def slots(manifest: Manifest) -> tuple[Slot, ...]: + return tuple(Slot(cell.id, repetition) for cell in manifest.matrix for repetition in range(1, manifest.repetitions + 1)) +``` + +After: keep this production code unchanged; lock its behavior with `test_slots_follow_seeded_manifest_order_before_repetitions`. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/attempts_test.py`: add the seeded multi-cell slot-order regression without changing production attempts code. + +**Test Strategy** + +Write the named test. Assert both `(cell_id, repetition)` tuples and that the first occurrence order equals `manifest.matrix`. This prevents a future alphabetical sort in the store from silently defeating the seed. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +``` + +Expected: the exact seeded slot sequence passes. + +### [API-3] Preserve the complete benchmark baseline + +**Problem** + +The field touches canonical JSON and a public schema; focused tests alone could miss example or integration drift. + +**Solution** + +Run fresh benchmark suites and all shipped validation fixtures after the code and tests are complete. Do not accept cached output. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G08.md`: record actual implementation decisions and full command output. + +**Test Strategy** + +No additional test file beyond API-1/API-2. The full existing suite is the regression oracle. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: three `ok: manifest is valid` lines, all tests `OK`, and no whitespace errors. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/manifest.py` | API-1 | +| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 | +| `scripts/agent_benchmark/manifest_test.py` | API-1 | +| `scripts/agent_benchmark/attempts_test.py` | API-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md` | API-3 | + +## Final Verification + +Run from `/config/workspace/iop-s0`; fresh output is required and cached test output is not acceptable. + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test +python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: every command exits 0, the three manifests validate, all benchmark tests report `OK`, and `git diff --check` is silent. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log new file mode 100644 index 00000000..214be5ae --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log @@ -0,0 +1,136 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract, plan=0, tag=API + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Require exact all-cell preflight evidence | [ ] | +| API-2 Isolate live agy preflight state per cell | [ ] | +| API-3 Replace obsolete direct-only assertions | [ ] | +| API-4 Preserve append-only and redaction invariants | [ ] | + +## Implementation Checklist + +- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers. +- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state. +- [ ] [API-3] Replace direct-only contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions. +- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`. +- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm the durable result set equals all `manifest.matrix` cells in canonical order on both write and read. +- Confirm preset blockers publish exact closed taxonomy and allocate no attempt. +- Confirm agy runtime state is keyed by cell id and a blocked re-preflight clears stale ready state. +- Confirm no adapter-specific endpoint, token, or raw caller value enters evidence. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### API-1/API-2 Focused Verification + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +``` + +_Actual output:_ + +### API-3 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test +``` + +_Actual output:_ + +### API-4 and Final Verification + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_1.log new file mode 100644 index 00000000..1cbc2c2d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_1.log @@ -0,0 +1,254 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract, plan=1, tag=API + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found that it changed public preflight semantics but omitted the project benchmark skill and its contract tests, both of which explicitly require the obsolete direct-only behavior. +- Replan makes this packet the sole owner of all-cell preflight documentation/tests, adds the concrete agy preflight type source to analysis, and leaves rubric-selection wording for the dependent final-manifest packet. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Require exact all-cell preflight evidence | [x] | +| API-2 Isolate live agy preflight state per cell | [x] | +| API-3 Replace obsolete direct-only code and skill assertions | [x] | +| API-4 Preserve append-only and redaction invariants | [x] | + +## Implementation Checklist + +- [x] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers. +- [x] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state. +- [x] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions. +- [x] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`. +- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path. +- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +None. + +## Key Design Decisions + +- Preflight collection and durable record validation use the same immutable `manifest.matrix` tuple, so cardinality, identity, and order are checked against every declared direct or execution-preset cell. +- `_LiveAdapter` clears prior admission before every cell preflight and publishes agy runtime state only after the complete live/config observation is ready. Invocation retrieves that state once by exact cell id and rejects missing or mismatched bindings. +- The project skill and its mutation tests now require affirmative all-cell live evidence while retaining closed blocker taxonomy, zero-attempt blockers, and the rule that local fixture validation alone is not live readiness. + +## Reviewer Checkpoints + +- Confirm the durable result set equals all `manifest.matrix` cells in canonical order on both write and read. +- Confirm preset blockers publish exact closed taxonomy and allocate no attempt. +- Confirm agy runtime state is keyed by cell id and a blocked re-preflight clears stale ready state. +- Confirm the project benchmark skill and contract tests require one observation for every immutable matrix cell and contain no direct-only/preset-local exception. +- Confirm no adapter-specific endpoint, token, or raw caller value enters evidence. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### API-1/API-2 Focused Verification + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +``` + +_Actual output:_ + +```text +.... +---------------------------------------------------------------------- +Ran 4 tests in 2.414s + +OK +``` + +Exit code: `0`. + +_Reviewer fresh rerun:_ + +```text +.... +---------------------------------------------------------------------- +Ran 4 tests in 2.492s + +OK +``` + +Exit code: `0`. + +### API-3 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +``` + +_Actual output:_ + +```text +................................................................................. +---------------------------------------------------------------------- +Ran 81 tests in 14.040s + +OK +``` + +Exit code: `0`. + +_Reviewer fresh rerun:_ + +```text +................................................................................. +---------------------------------------------------------------------- +Ran 81 tests in 13.933s + +OK +``` + +Exit code: `0`. + +### API-4 and Final Verification + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +_Actual output:_ + +```text +$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +......................................................................................................................................................................................................................................... +---------------------------------------------------------------------- +Ran 233 tests in 47.536s + +OK + +$ git diff --check +(no output) +``` + +Both commands exited `0`. + +_Reviewer fresh rerun:_ + +```text +$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +......................................................................................................................................................................................................................................... +---------------------------------------------------------------------- +Ran 233 tests in 47.606s + +OK + +$ git diff --check +(no output) +``` + +Both commands exited `0`. + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +### Overall Verdict + +PASS + +### Dimension Assessment + +| Dimension | Assessment | Evidence | +|-----------|------------|----------| +| Correctness | Pass | Full-matrix collection/write/read order, blocker handling, and cell-keyed agy admission matched the planned invariants. | +| Completeness | Pass | API-1 through API-4 are implemented, and the public skill contract was updated to all-cell semantics. | +| Test coverage | Pass | Fresh focused 4-test, connectivity/skill 81-test, and full 233-test suites passed. | +| API contract | Pass | No caller wire/config contract changed; direct and execution-preset bindings remain exact and fail closed without substitution. | +| Code quality | Pass | No stale direct-only production reference remains after the review-time CLI docstring repair; `git diff --check` passed. | +| Implementation deviation | Pass | Planned implementation scope is present; unrelated sibling execution-order work was excluded from this verdict. | +| Verification trust | Pass | Reviewer reran every planned command from the current worktree and reproduced the recorded success. | +| Spec conformance | Pass | SDD S02 all-cell redacted preflight evidence and the `route-readiness` Evidence Map are satisfied by deterministic contract tests without claiming live credential readiness. | + +### Findings + +- Nit (repaired): `scripts/agent_comparison_benchmark.py:14` still described exit 0 as requiring every “direct preflight cell”; review changed it to “every matrix preflight cell” to match the implemented public semantics. +- Required: None. +- Suggested: None. + +### Routing Signals + +- `review_rework_count=0` +- `evidence_integrity_failure=false` + +### Next Step + +- PASS: archive this pair, write `complete.log`, and emit the milestone completion metadata for runtime aggregation. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log new file mode 100644 index 00000000..d1f7cbc4 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log @@ -0,0 +1,40 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/02_route_preflight_contract + +## 완료 일시 + +2026-08-12 + +## 요약 + +계획 개정 2회와 공식 코드 리뷰 1회를 거쳐 전체 matrix cell preflight 및 cell별 agy 상태 격리 계약을 구현했고 최종 판정은 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | 재계획 | Epic self-review에서 project benchmark skill과 contract test 범위 누락을 발견해 구현 전 계획을 보강했다. | +| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | PASS | 전체 cell의 append-only preflight, blocker 무할당, cell별 agy 상태 격리 및 skill 계약을 확인했다. | + +## 구현/정리 내용 + +- preflight 수집과 durable record 검증을 immutable `manifest.matrix` 전체 및 canonical order 기준으로 통일했다. +- agy live preflight/admission 상태를 cell id별로 격리하고 재-preflight 시 stale 상태를 제거했다. +- mixed/preset-only 성공·blocker·corruption·cell isolation 회귀 테스트와 project benchmark skill 계약을 all-cell 의미로 갱신했다. +- 리뷰 중 공개 CLI 종료 코드 설명의 남은 direct-only 문구를 matrix 전체 의미로 정정했다. + +## 최종 검증 + +- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state` - PASS; 4 tests, 2.492s. +- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 81 tests, 13.933s. +- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 233 tests, 47.606s. +- `git diff --check` - PASS; 출력 없음. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- 없음 diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log new file mode 100644 index 00000000..c90b16f6 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log @@ -0,0 +1,276 @@ + + +# Plan - All-Cell Route Preflight Contract + +## For the Implementing Agent + +Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The benchmark currently validates preset capability locally but deliberately omits preset cells from durable preflight. A mixed direct/preset manifest therefore records a misleading ready subset and allocates no attempts, while the live `agy` adapter retains only the last cell's preflight state. Route readiness for C01-C09 requires every declared cell to be observed and bound independently before any attempt starts. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/agent_benchmark/manifest.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=route-readiness`. +- Target scenario: S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker. +- Evidence Map row: S02 requires a redacted C01-C09 preflight matrix with auth/route/effort/terminal evidence. +- The checklist therefore replaces direct-only subset semantics with exact all-cell evidence and adds per-cell caller-state regressions. Live credentials remain the dependent manifest packet's external gate. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the attempts/live adapter paths and network-free integration tests above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three example manifests validated and the 178-test manifest/attempts/connectivity suite passed. +- Current tests explicitly lock the old limitation: `test_cli_mixed_manifest_never_invokes_unobserved_preset_cells`, `test_generic_preset_cells_are_local_contract_only`, and `test_generic_preset_only_public_preflight_fails_closed_without_run_state` (`connectivity_integration_test.py:1009-1214`). These must be replaced, not preserved. +- `collect_preflight_observations` skips non-direct cells (`attempts.py:386-423`), durable read/write validates against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`), and public preflight rejects an empty direct subset (`attempts.py:1869-1881`). +- `_LiveAdapter` stores one `_agy_preflight` value for every cell (`live_iop.py:843-902`) and reuses it at invocation (`live_iop.py:935-950`), so multiple agy routes can cross-bind. +- This packet uses only network-free adapters and mocks. Fresh output is required; cached output is not accepted. + +### Test Coverage Gaps + +- Existing tests cover direct cells and intentionally assert that preset cells are omitted. +- No mixed manifest test expects both direct and preset results in the same append-only record and then executes both. +- No preset-only public preflight test expects a durable ready or blocker record. +- No live-adapter test preflights two agy cells and proves each invocation receives its own preflight capability/runtime binding. +- Existing corruption and secret-redaction coverage is broad and must continue to pass after the result-set invariant changes. + +### Symbol References + +- Rename private `RunStore._direct_cells` to `_preflight_cells` or remove the helper. Its only references are `attempts.py:563` and `attempts.py:632`; update both together. +- Rename private `_LiveAdapter._agy_preflight` to `_agy_preflights`; all references are `live_iop.py:858`, `885`, `893-894`, and `936-949`. +- No public import or CLI symbol is renamed. + +### Split Judgment + +- This is split child `02_route_preflight_contract`. Its stable invariant is: the preflight result set equals the entire immutable matrix, and every admitted live invocation consumes the preflight state for its own cell id. +- PASS evidence is all-cell append/read corruption coverage, mixed/preset-only CLI coverage, and a two-cell agy state test. +- It is independent of `01_execution_order_contract`; it consumes `manifest.matrix` order without owning that order. `03+01,02_locked_benchmark_manifest` depends on both. + +### Scope Rationale + +- Do not add the final nine-cell manifest or an execution-order field here. +- Do not change caller wire formats, IOP contracts, provider config, credentials, retry policy, attempt allocation semantics, or scoring. +- Do not synthesize readiness for missing routes. Every preset must pass the same live config/catalog/capability checks and retain exact `registration_required` or `implementation_gap` taxonomy. + +### Final Routing + +- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+2+2+1+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`. +- Review closures: all closed. Scores `2+2+2+1+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`. +- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but does not replace grade basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Implementation Checklist + +- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers. +- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state. +- [ ] [API-3] Replace direct-only contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions. +- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Require exact all-cell preflight evidence + +**Problem** + +The collector skips every `execution_preset` (`attempts.py:390-410`), and both record validation and publication compare against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`). A mixed manifest can publish `status=ready` for only its direct subset, after which `run_slots` correctly refuses to execute because the record does not cover the matrix. + +**Solution** + +Validate capabilities and call `adapter.preflight(cell)` for every cell in canonical `manifest.matrix` order. Rename the private helper to `_preflight_cells` returning the full matrix, and use it for durable result cardinality, identity, order, and publication. Remove the direct-cell-empty special case from `preflight_manifest`; a valid manifest is already non-empty. Preserve canonical evidence validation and closed aggregate taxonomy. + +Before (`scripts/agent_benchmark/attempts.py:386-423`): + +```python +"""Validate the full registry, then probe direct cells in manifest order.""" +... +for cell in manifest.matrix: + if cell.iop.route_kind != "direct": + continue + observation = adapters[cell.caller].preflight(cell) +``` + +After: + +```python +"""Validate the full registry, then probe every cell in manifest order.""" +... +for cell in manifest.matrix: + observation = adapters[cell.caller].preflight(cell) + ... + observations[cell.id] = observation +``` + +Use the exact same full tuple on write and read so missing, extra, reordered, or foreign results fail closed. If any cell is blocked, publish the all-cell record, allocate zero attempts, and return the existing closed CLI status. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/attempts.py`: collect, write, read, and aggregate the exact full matrix. +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: update the public preflight/CLI contract tests for all-cell behavior. + +**Test Strategy** + +Write/replace tests. Assert preset-only ready records are durable, preset-only blockers are durable with zero cells directory, mixed records include both cells in matrix order, a ready mixed run invokes both exactly once, and missing/extra/reordered record payloads remain rejected. + +**Verification** + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation +``` + +Expected: all named all-cell tests pass. + +### [API-2] Isolate live agy preflight state per cell + +**Problem** + +`_LiveAdapter._agy_preflight` is overwritten on each preflight (`live_iop.py:858`, `885`) and the last value is used for every agy invocation (`live_iop.py:935-949`). C03 and C07 in the same matrix can therefore use the wrong route observation/capability state. + +**Solution** + +Replace the scalar with `dict[str, AgyIopPreflight]` (use the concrete existing return type if exported; otherwise retain `Any` only at the value boundary). At the start of every cell preflight, remove that cell's prior agy state and admitted binding. Store both only after the current result is ready. At invocation, retrieve by `cell.id` once and pass that same object to `build_agy_invocation`, the invoker, and `observed_result`; reject absent state before building a spec. + +Before (`scripts/agent_benchmark/live_iop.py:935-950`): + +```python +if self.caller == AGY_CALLER: + if self._agy_preflight is None: + raise LiveIopError("stream_incompatible") + ... + result = self._invokers.agy(spec, parser, self._agy_preflight, ...) + observed = parser.observed_result(self._agy_preflight.capability, result) +``` + +After: + +```python +if self.caller == AGY_CALLER: + agy_preflight = self._agy_preflights.get(cell.id) + if agy_preflight is None: + raise LiveIopError("stream_incompatible") + ... + result = self._invokers.agy(spec, parser, agy_preflight, ...) + observed = parser.observed_result(agy_preflight.capability, result) +``` + +Do not persist raw endpoint or credential values; the new map remains process-local runtime state. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/live_iop.py`: clear, store, and retrieve agy preflight/admission by exact cell id. +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: add `test_live_agy_multiple_cells_consume_their_own_preflight_state` with two distinct route kinds/ids and mocked invokers. + +**Test Strategy** + +Write the named network-free regression. Preflight direct and preset agy cells with distinguishable observations, invoke in reverse preflight order, and assert each invocation receives its own object and binding. Add a re-preflight blocker case proving stale ready state cannot be invoked. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +``` + +Expected: both reverse-order isolation and stale-state rejection pass. + +### [API-3] Replace the obsolete direct-only assertions + +**Problem** + +Three integration tests at `connectivity_integration_test.py:1009-1214` intentionally require preset omission. Leaving them unchanged would preserve the exact S02 gap this packet closes. + +**Solution** + +Replace those tests with the all-cell cases named in API-1 and update exact adapter call/result assertions everywhere affected. Retain the existing three-caller direct case, registration-vs-implementation taxonomy, append-only corruption, missing-adapter, CLI redaction, and zero-attempt blocker assertions. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove only obsolete local-only expectations and add full-matrix expectations. + +**Test Strategy** + +Write tests, do not skip. All expected result ids must be derived from `manifest.matrix` and compared in stable order. No test may invent a ready observation without going through the typed fake or live adapter boundary. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test +``` + +Expected: all network-free public connectivity tests pass. + +### [API-4] Preserve append-only and redaction invariants + +**Problem** + +Expanding the record set changes durable cardinality and exercises more caller variants; regressions could appear outside the focused cases. + +**Solution** + +Run the complete manifest/attempt/connectivity suites and `git diff --check`. Record full output in the review stub. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G09.md`: record implementation decisions, deviations, and actual command output. + +**Test Strategy** + +No further files. Existing corruption, retry/resume, measurement, web-validation, secret-redaction, and live-adapter tests provide the whole-suite oracle. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: all tests report `OK`; whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/attempts.py` | API-1 | +| `scripts/agent_benchmark/live_iop.py` | API-2 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-2, API-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md` | API-4 | + +## Final Verification + +Run from `/config/workspace/iop-s0`; fresh output is required and cached output is not acceptable. + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: every command exits 0, all benchmark suites report `OK`, and no secret/raw runtime values appear in durable-record assertions. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_1.log new file mode 100644 index 00000000..b5d7d4aa --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_1.log @@ -0,0 +1,292 @@ + + +# Plan - All-Cell Route Preflight Contract + +## For the Implementing Agent + +Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The benchmark currently validates preset capability locally but deliberately omits preset cells from durable preflight. A mixed direct/preset manifest therefore records a misleading ready subset and allocates no attempts, while the live `agy` adapter retains only the last cell's preflight state. Route readiness for C01-C09 requires every declared cell to be observed and bound independently before any attempt starts. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found that it changed public preflight semantics but omitted the project benchmark skill and its contract tests, both of which explicitly require the obsolete direct-only behavior. +- Replan makes this packet the sole owner of all-cell preflight documentation/tests, adds the concrete agy preflight type source to analysis, and leaves rubric-selection wording for the dependent final-manifest packet. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `scripts/agent_benchmark/manifest.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=route-readiness`. +- Target scenario: S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker. +- Evidence Map row: S02 requires a redacted C01-C09 preflight matrix with auth/route/effort/terminal evidence. +- The checklist therefore replaces direct-only subset semantics with exact all-cell evidence and adds per-cell caller-state regressions. Live credentials remain the dependent manifest packet's external gate. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the attempts/live adapter paths and network-free integration tests above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three example manifests validated and the 178-test manifest/attempts/connectivity suite passed. +- Current tests explicitly lock the old limitation: `test_cli_mixed_manifest_never_invokes_unobserved_preset_cells`, `test_generic_preset_cells_are_local_contract_only`, and `test_generic_preset_only_public_preflight_fails_closed_without_run_state` (`connectivity_integration_test.py:1009-1214`). These must be replaced, not preserved. +- `collect_preflight_observations` skips non-direct cells (`attempts.py:386-423`), durable read/write validates against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`), and public preflight rejects an empty direct subset (`attempts.py:1869-1881`). +- `_LiveAdapter` stores one `_agy_preflight` value for every cell (`live_iop.py:843-902`) and reuses it at invocation (`live_iop.py:935-950`), so multiple agy routes can cross-bind. +- This packet uses only network-free adapters and mocks. Fresh output is required; cached output is not accepted. + +### Test Coverage Gaps + +- Existing tests cover direct cells and intentionally assert that preset cells are omitted. +- No mixed manifest test expects both direct and preset results in the same append-only record and then executes both. +- No preset-only public preflight test expects a durable ready or blocker record. +- No live-adapter test preflights two agy cells and proves each invocation receives its own preflight capability/runtime binding. +- Existing corruption and secret-redaction coverage is broad and must continue to pass after the result-set invariant changes. +- The project benchmark skill and `skill_contract_test.py` explicitly describe/require direct-only preflight, so they would contradict production immediately after this packet unless changed in the same boundary. + +### Symbol References + +- Rename private `RunStore._direct_cells` to `_preflight_cells` or remove the helper. Its only references are `attempts.py:563` and `attempts.py:632`; update both together. +- Rename private `_LiveAdapter._agy_preflight` to `_agy_preflights`; all references are `live_iop.py:858`, `885`, `893-894`, and `936-949`. +- No public import or CLI symbol is renamed. + +### Split Judgment + +- This is split child `02_route_preflight_contract`. Its stable invariant is: the preflight result set equals the entire immutable matrix, and every admitted live invocation consumes the preflight state for its own cell id. +- PASS evidence is all-cell append/read corruption coverage, mixed/preset-only CLI coverage, and a two-cell agy state test. +- It is independent of `01_execution_order_contract`; it consumes `manifest.matrix` order without owning that order. The final `04+02,03_locked_benchmark_manifest` packet depends on this packet and index 03. + +### Scope Rationale + +- Do not add the final nine-cell manifest or an execution-order field here. +- Do not change caller wire formats, IOP contracts, provider config, credentials, retry policy, attempt allocation semantics, or scoring. +- Do not synthesize readiness for missing routes. Every preset must pass the same live config/catalog/capability checks and retain exact `registration_required` or `implementation_gap` taxonomy. +- Update only the skill's preflight/run/resume/validation/safety wording here. Do not add the new rubric-selection wording; the dependent final-manifest packet owns that operational transition after the rubric implementation lands. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+2+2+1+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`. +- Review closures: all closed. Scores `2+2+2+1+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`. +- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but does not replace grade basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Implementation Checklist + +- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers. +- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state. +- [ ] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions. +- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Require exact all-cell preflight evidence + +**Problem** + +The collector skips every `execution_preset` (`attempts.py:390-410`), and both record validation and publication compare against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`). A mixed manifest can publish `status=ready` for only its direct subset, after which `run_slots` correctly refuses to execute because the record does not cover the matrix. + +**Solution** + +Validate capabilities and call `adapter.preflight(cell)` for every cell in canonical `manifest.matrix` order. Rename the private helper to `_preflight_cells` returning the full matrix, and use it for durable result cardinality, identity, order, and publication. Remove the direct-cell-empty special case from `preflight_manifest`; a valid manifest is already non-empty. Preserve canonical evidence validation and closed aggregate taxonomy. + +Before (`scripts/agent_benchmark/attempts.py:386-423`): + +```python +"""Validate the full registry, then probe direct cells in manifest order.""" +... +for cell in manifest.matrix: + if cell.iop.route_kind != "direct": + continue + observation = adapters[cell.caller].preflight(cell) +``` + +After: + +```python +"""Validate the full registry, then probe every cell in manifest order.""" +... +for cell in manifest.matrix: + observation = adapters[cell.caller].preflight(cell) + ... + observations[cell.id] = observation +``` + +Use the exact same full tuple on write and read so missing, extra, reordered, or foreign results fail closed. If any cell is blocked, publish the all-cell record, allocate zero attempts, and return the existing closed CLI status. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/attempts.py`: collect, write, read, and aggregate the exact full matrix. +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: update the public preflight/CLI contract tests for all-cell behavior. + +**Test Strategy** + +Write/replace tests. Assert preset-only ready records are durable, preset-only blockers are durable with zero cells directory, mixed records include both cells in matrix order, a ready mixed run invokes both exactly once, and missing/extra/reordered record payloads remain rejected. + +**Verification** + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation +``` + +Expected: all named all-cell tests pass. + +### [API-2] Isolate live agy preflight state per cell + +**Problem** + +`_LiveAdapter._agy_preflight` is overwritten on each preflight (`live_iop.py:858`, `885`) and the last value is used for every agy invocation (`live_iop.py:935-949`). C03 and C07 in the same matrix can therefore use the wrong route observation/capability state. + +**Solution** + +Replace the scalar with `dict[str, AgyPreflightResult]`, importing the concrete exported dataclass from `agy_iop.py`. At the start of every cell preflight, remove that cell's prior agy state and admitted binding. Store both only after the current result is ready. At invocation, retrieve by `cell.id` once and pass that same object to `build_agy_invocation`, the invoker, and `observed_result`; reject absent state before building a spec. + +Before (`scripts/agent_benchmark/live_iop.py:935-950`): + +```python +if self.caller == AGY_CALLER: + if self._agy_preflight is None: + raise LiveIopError("stream_incompatible") + ... + result = self._invokers.agy(spec, parser, self._agy_preflight, ...) + observed = parser.observed_result(self._agy_preflight.capability, result) +``` + +After: + +```python +if self.caller == AGY_CALLER: + agy_preflight = self._agy_preflights.get(cell.id) + if agy_preflight is None: + raise LiveIopError("stream_incompatible") + ... + result = self._invokers.agy(spec, parser, agy_preflight, ...) + observed = parser.observed_result(agy_preflight.capability, result) +``` + +Do not persist raw endpoint or credential values; the new map remains process-local runtime state. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/live_iop.py`: clear, store, and retrieve agy preflight/admission by exact cell id. +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: add `test_live_agy_multiple_cells_consume_their_own_preflight_state` with two distinct route kinds/ids and mocked invokers. + +**Test Strategy** + +Write the named network-free regression. Preflight direct and preset agy cells with distinguishable observations, invoke in reverse preflight order, and assert each invocation receives its own object and binding. Add a re-preflight blocker case proving stale ready state cannot be invoked. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +``` + +Expected: both reverse-order isolation and stale-state rejection pass. + +### [API-3] Replace the obsolete direct-only code and skill assertions + +**Problem** + +Three integration tests at `connectivity_integration_test.py:1009-1214` intentionally require preset omission. The project benchmark skill repeats that contract in its preflight procedure, validation, and safety sections, and `_assert_preflight_contract` plus its mutation at `skill_contract_test.py:256-271,786-794` enforce it. Leaving either surface unchanged would preserve or publish the exact S02 gap this packet closes. + +**Solution** + +Replace those integration tests with the all-cell cases named in API-1 and update exact adapter call/result assertions everywhere affected. Update the project skill so preflight, run, and resume describe a fresh append-only observation for every immutable matrix cell, while retaining closed blocker taxonomy and zero-attempt behavior. Rewrite `_assert_preflight_contract` and its mutation to reject any return of direct-only/preset-local language. Retain the existing three-caller direct case, registration-vs-implementation taxonomy, append-only corruption, missing-adapter, CLI redaction, and zero-attempt blocker assertions. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove only obsolete local-only expectations and add full-matrix expectations. +- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: replace direct-only preflight wording with the exact all-cell durable contract. +- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require all-cell skill wording and reject restoration of preset-local semantics. + +**Test Strategy** + +Write tests, do not skip. All expected result ids must be derived from `manifest.matrix` and compared in stable order. No test may invent a ready observation without going through the typed fake or live adapter boundary. Skill tests must assert the affirmative all-cell contract and mutation-fail on the former direct-only statements. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +``` + +Expected: all network-free public connectivity tests pass. + +### [API-4] Preserve append-only and redaction invariants + +**Problem** + +Expanding the record set changes durable cardinality and exercises more caller variants; regressions could appear outside the focused cases. + +**Solution** + +Run the complete manifest/attempt/connectivity/skill-contract suites and `git diff --check`. Record full output in the review stub. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G09.md`: record implementation decisions, deviations, and actual command output. + +**Test Strategy** + +No further files. Existing corruption, retry/resume, measurement, web-validation, secret-redaction, and live-adapter tests provide the whole-suite oracle. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +Expected: all tests report `OK`; whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/attempts.py` | API-1 | +| `scripts/agent_benchmark/live_iop.py` | API-2 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-2, API-3 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 | +| `scripts/agent_benchmark/skill_contract_test.py` | API-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md` | API-4 | + +## Final Verification + +Run from `/config/workspace/iop-s0`; fresh output is required and cached output is not acceptable. + +```bash +python3 -m unittest \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \ + scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state +python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +Expected: every command exits 0, all benchmark suites report `OK`, and no secret/raw runtime values appear in durable-record assertions. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log new file mode 100644 index 00000000..f69d943d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log @@ -0,0 +1,157 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract, plan=0, tag=API + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-local-G08.md` → `plan_local_G08_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Add a closed dual-version rubric catalog | [ ] | +| API-2 Bind prompt and durable validation to manifest version | [ ] | +| API-3 Preserve historical compatibility and exact new semantics | [ ] | +| API-4 Run the complete benchmark regression | [ ] | + +## Implementation Checklist + +- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior. +- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output. +- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility, and update the project benchmark skill to the manifest-selected contract. +- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_0.log`. +- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm the legacy version/category tuple remains exact and reloadable. +- Confirm the new seven categories and maxima exactly match the approved SDD and total 100. +- Confirm prompt and both worksheet load paths use `manifest.rubric_version`. +- Confirm unknown and cross-version worksheets fail before scored result publication. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### Dependency Verification + +```bash +python3 - <<'PY' +from pathlib import Path +active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log") +archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, candidates +print(f"ok: predecessor complete {candidates[0]}") +PY +``` + +_Actual output:_ + +### API-1 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +### API-2 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +``` + +_Actual output:_ + +### API-3 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md +``` + +_Actual output:_ + +### API-4 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_1.log new file mode 100644 index 00000000..c1ff0afb --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_1.log @@ -0,0 +1,260 @@ + + +# Code Review Reference - API + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract, plan=1, tag=API + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found a semantic contradiction: Scope Rationale prohibited a project-skill edit while API-3 and the file summary required that same edit, creating ownership overlap with the route-preflight packet. +- Replan keeps this packet on the dormant additive rubric implementation and dual-version code tests. The dependent final-manifest packet owns the operational skill transition when it selects the new rubric. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-local-G08.md` → `plan_local_G08_1.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| API-1 Add a closed dual-version rubric catalog | [x] | +| API-2 Bind prompt and durable validation to manifest version | [x] | +| API-3 Preserve historical compatibility and exact new semantics | [x] | +| API-4 Run the complete benchmark regression | [x] | + +## Implementation Checklist + +- [x] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior. +- [x] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output. +- [x] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric. +- [x] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`. +- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path. +- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +없음. 지정된 파일 범위와 검증 명령을 그대로 수행했으며, project benchmark skill과 living spec은 변경하지 않았다. + +## Key Design Decisions + +- `RUBRIC_VERSION`과 `RUBRIC_CATEGORIES`를 legacy alias/table로 그대로 유지하고, `ONE_SHOT_RUBRIC_VERSION`, `RUBRIC_VERSIONS`, immutable `RUBRIC_CATEGORIES_BY_VERSION` catalog를 additive하게 추가했다. +- worksheet가 선언한 version의 ordered table로 먼저 검증되고, `expected_version`이 있으면 exact match를 추가로 요구하도록 해 unknown·cross-version 출력을 fail-closed했다. +- evaluator prompt와 신규/내구 score reload의 두 worksheet load path 모두 `manifest.rubric_version`을 사용한다. legacy prompt exact bytes와 canonical worksheet SHA-256도 회귀 테스트로 고정했다. +- 새-version 성공과 structurally valid legacy worksheet의 cross-version 실패를 각각 fresh score attempt로 검증했으며 실패 result에는 scored worksheet를 publish하지 않음을 확인했다. + +## Reviewer Checkpoints + +- Confirm the legacy version/category tuple remains exact and reloadable. +- Confirm the new seven categories and maxima exactly match the approved SDD and total 100. +- Confirm prompt and both worksheet load paths use `manifest.rubric_version`. +- Confirm unknown and cross-version worksheets fail before scored result publication. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### Dependency Verification + +```bash +python3 - <<'PY' +from pathlib import Path +active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log") +archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, candidates +print(f"ok: predecessor complete {candidates[0]}") +PY +``` + +_Actual output:_ + +```text +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log +``` + +### API-1 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +```text +...................................................................................................................... +---------------------------------------------------------------------- +Ran 118 tests in 1.181s + +OK +``` + +### API-2 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +``` + +_Actual output:_ + +```text +. +---------------------------------------------------------------------- +Ran 1 test in 0.448s + +OK +``` + +### API-3 Verification + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +``` + +_Actual output:_ + +```text +....................................................................................................................................... +---------------------------------------------------------------------- +Ran 135 tests in 5.746s + +OK +``` + +### API-4 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +```text +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +................................................................................................................................................................................................................ +---------------------------------------------------------------------- +Ran 208 tests in 51.235s + +OK +``` + +`git diff --check`: exit 0, 출력 없음. + +### Reviewer Fresh Verification + +2026-08-12 공식 리뷰에서 아래 검증을 현재 worktree 기준으로 다시 실행했다. + +```text +go version go1.26.2 linux/arm64 +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log + +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +Ran 118 tests in 1.202s +OK + +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +Ran 1 test in 0.453s +OK + +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +Ran 135 tests in 5.848s +OK + +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid + +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +Ran 208 tests in 50.790s +OK + +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +Ran 432 tests in 119.960s +OK + +git diff --check +exit 0, no output +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +- Overall Verdict: PASS +- Dimension Assessment: + - Correctness: Pass — manifest-selected rubric lookup, prompt rendering, output validation, and durable reload validation preserve one exact version/table binding. + - Completeness: Pass — API-1 through API-4 are implemented within the planned files; the dependent manifest packet remains the owner of the first tracked new-version manifest. + - Test coverage: Pass — focused dual-version tests, the planned 208-test regression, and the 432-test full benchmark discovery all pass. + - API contract: Pass — `landing-quality-v1` remains byte-compatible while the schema and loader expose only the closed two-version catalog. + - Code quality: Pass — immutable tuples plus `MappingProxyType` keep the catalog closed; no stale production literal, debug print, TODO, or whitespace error was found. + - Implementation deviation: Pass — no deviation from the revalidated plan; unrelated sibling-task worktree changes were excluded from this verdict. + - Verification trust: Pass — implementation-reported results were independently reproduced with fresh reviewer commands. + - Spec conformance: Pass — SDD S01's seven-category 100-point rubric and `fixture-lock` evidence boundary match the implementation; concrete manifest/digest evidence remains assigned to the dependent manifest packet. +- Findings: None +- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false` +- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock` for runtime aggregation. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log new file mode 100644 index 00000000..fb889f60 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log @@ -0,0 +1,43 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract + +## 완료 일시 + +2026-08-12 + +## 요약 + +구현 전 ownership 충돌 재계획 1회 뒤 dual-version rubric 계약 구현을 공식 리뷰 1회로 검증했으며 최종 판정은 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_local_G08_0.log` | `code_review_cloud_G08_0.log` | 재계획 | project benchmark skill 수정 ownership 충돌을 구현 전에 제거했으며 공식 판정은 없었다. | +| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | PASS | legacy rubric을 보존한 closed dual-version catalog, manifest-selected prompt/worksheet 검증, 전체 benchmark 회귀를 확인했다. | + +## 구현/정리 내용 + +- `landing-quality-v1`의 기존 category tuple과 canonical worksheet 의미를 유지하면서 `one-shot-agent-comparison-v1`의 SDD 승인 7개 category를 immutable catalog에 추가했다. +- manifest schema/loader, evaluator prompt, 최초 score publication과 durable score reload가 같은 manifest-selected rubric version을 사용하고 unknown/cross-version worksheet를 fail-closed한다. +- dual-version manifest/rubric/scoring regression과 legacy prompt/worksheet compatibility assertion을 추가했으며 운영 skill과 첫 tracked new-version manifest 전환은 후속 packet 범위로 유지했다. + +## 최종 검증 + +- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test` - PASS; 118 tests, `OK`. +- `python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation` - PASS; 1 test, `OK`. +- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test` - PASS; 135 tests, `OK`. +- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`. +- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 208 tests, `OK`. +- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 432 tests, `OK`. +- `git diff --check` - PASS; 출력 없음. +- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic benchmark rubric/schema packet 범위가 아니어서 실행하지 않았다. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- dependent `04+02,03_locked_benchmark_manifest` packet이 첫 tracked `one-shot-agent-comparison-v1` manifest와 concrete rubric digest evidence를 제공해야 하며, 이 완료 로그 단독으로 `fixture-lock` 전체 완료를 주장하지 않는다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log new file mode 100644 index 00000000..b68d0759 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log @@ -0,0 +1,304 @@ + + +# Plan - Versioned One-Shot Benchmark Rubric Contract + +## For the Implementing Agent + +Do not start until exactly one active or archived `01_execution_order_contract/complete.log` is resolved by the dependency command below. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The approved one-shot comparison SDD fixes seven rubric categories totaling 100, while the existing `landing-quality-v1` worksheet contains five different categories. Reusing that version name would mutate historical meaning, so this packet preserves legacy runs and adds a manifest-selected version for the approved benchmark rubric. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/rubric.py` +- `scripts/agent_benchmark/rubric_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=fixture-lock`. +- Target scenario: S01 requires the same fixture, viewport, rubric checksum/version for every cell. The approved rubric at SDD lines 90-91 is 요구사항 충족 25, 시각 완성도 25, 반응형·접근성 15, 이미지 활용·디테일 10, 동작 안정성 10, 코드 품질 10, 자체 검증 완결성 5. +- Evidence Map row: S01 requires fixture prompt/assets/workspace/rubric digest evidence. +- The checklist therefore creates a distinct immutable rubric version, retains legacy validation, and binds prompt/output validation to the manifest-selected version. The final manifest packet supplies the concrete version/digest evidence. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the complete manifest, rubric, scoring, and test files above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all example manifests validated and the 178-test manifest/attempt/connectivity suite passed. `rubric_test` and `scoring_test` are additional required fresh suites for this packet. +- Current `RUBRIC_CATEGORIES` is five entries totaling 100 (`rubric.py:21-27`). `validate_worksheet` accepts only global `RUBRIC_VERSION` (`rubric.py:82-126`), and `_prompt` hardcodes those categories and `landing-quality-v1` (`scoring.py:1207-1222`). +- The safe compatibility rule is additive: keep `landing-quality-v1` and its five-category table readable; add `one-shot-agent-comparison-v1` with the approved seven-category table; reject unknown and cross-version worksheets. +- Fresh output is required; cached output is not accepted. No external runner is required. + +### Test Coverage Gaps + +- Existing rubric tests validate only the five-category legacy table. +- Existing scoring tests generate only legacy worksheets and cannot prove prompt/worksheet selection from the manifest. +- Existing manifest/schema tests accept only one rubric constant. +- No test rejects a valid legacy worksheet when the allocation requires the new rubric, or vice versa. + +### Symbol References + +- `manifest.RUBRIC_VERSION` is referenced only by `manifest.py:711` and `rubric.py:16,83,126`. Replace internal single-version validation with a closed version catalog while preserving a legacy alias if needed for compatibility. +- `rubric.RUBRIC_CATEGORIES` is imported by `scoring.py:44-49`, `scoring_test.py:56`, and `rubric_test.py:10`. Preserve it as the legacy tuple for existing callers, add a version lookup, and migrate production scoring to the lookup. +- `_prompt` is private and called at `scoring.py:1879`; change both definition and call together. +- `load_worksheet` call sites are `scoring.py:1641` and `scoring.py:1972`; both must pass the manifest-selected expected version. + +### Split Judgment + +- This is dependent child `03+01_rubric_version_contract`. Its stable invariant is: rubric version selects exactly one immutable ordered category table from manifest load through evaluator prompt, worksheet validation, durable result, and historical reload. +- PASS evidence is dual-version loader/rubric/scoring tests plus full regression. +- Predecessor index 01 is currently active at `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` but has no `complete.log`; implementation is dependency-blocked until that exact file exists. The dependency is required because both packets modify `manifest.py`, `manifest_test.py`, and the schema. + +### Scope Rationale + +- Do not delete, rename, or reinterpret `landing-quality-v1`; old manifests and immutable run snapshots must still load and score under the old table. +- Do not change automatic web gate eligibility, evaluator identity blinding, score persistence, caller bindings, fixture bytes, or report layout. +- Do not update the project benchmark skill or living spec in this packet; their current general 100-point description remains true, and operational selection remains manifest-owned. + +### Final Routing + +- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`. +- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve exactly one predecessor `complete.log`: the active sibling path or one matching `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log`. +2. Implement this packet against the reviewed 01 source. Do not infer completion from an active PLAN or review stub. +3. On PASS, `04+02,03_locked_benchmark_manifest` may proceed only after both index 02 and this index 03 have `complete.log`. + +## Implementation Checklist + +- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior. +- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output. +- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility, and update the project benchmark skill to the manifest-selected contract. +- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Add a closed dual-version rubric catalog + +**Problem** + +`manifest.py:30` exposes one allowed rubric version, and `rubric.py:21-27` gives that version a five-category table that differs from the approved SDD. Changing the existing tuple in place would silently rewrite historical `landing-quality-v1` meaning. + +**Solution** + +Keep the legacy string and tuple unchanged. Add `ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"`, a closed allowed-version tuple in `manifest.py`, and an ordered category mapping in `rubric.py`. The new table is exactly: + +```python +ONE_SHOT_RUBRIC_CATEGORIES = ( + ("requirements_fidelity", 25), + ("visual_completeness", 25), + ("responsive_accessibility", 15), + ("image_detail_usage", 10), + ("behavior_stability", 10), + ("code_quality", 10), + ("self_verification", 5), +) +``` + +Add `rubric_categories(version)` that fails closed for unknown versions. Update the manifest schema from a single `const` to an exact two-value enum. `validate_worksheet` must read the worksheet version, optionally require an `expected_version`, and validate against only that version's ordered table. + +Before (`scripts/agent_benchmark/rubric.py:21-27`): + +```python +RUBRIC_CATEGORIES = ( + ("task_fidelity", 25), + ("visual_hierarchy", 25), + ("responsive_composition", 20), + ("typography_readability", 15), + ("polish_consistency", 15), +) +``` + +After: retain that exact tuple as the legacy value and add the new tuple/mapping without mutation. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/manifest.py`: accept the closed legacy/new rubric version catalog. +- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: mirror the exact two-version enum. +- [ ] `scripts/agent_benchmark/rubric.py`: provide versioned category lookup and expected-version worksheet validation. +- [ ] `scripts/agent_benchmark/rubric_test.py`: verify exact tables, totals, ordering, unknown versions, and cross-version rejection. +- [ ] `scripts/agent_benchmark/manifest_test.py`: verify loader/schema parity for both allowed versions and an unknown version. + +**Test Strategy** + +Write tests. Preserve every existing legacy assertion. Add `test_one_shot_rubric_exact_categories_and_total_are_accepted`, `test_cross_version_worksheet_is_rejected`, and manifest/schema two-version parity. Assert both category sums equal exactly 100. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +``` + +Expected: both versions pass their exact table; malformed, unknown, reordered, and cross-version cases fail closed. + +### [API-2] Bind prompt and durable score validation to the manifest version + +**Problem** + +`_prompt` uses global categories and a literal version (`scoring.py:1207-1222`), while both `load_worksheet` calls (`scoring.py:1641`, `1972`) validate only the global legacy version. A new manifest version could therefore ask for or accept the wrong worksheet. + +**Solution** + +Change `_prompt` to accept `rubric_version`, look up its exact table, render that version string, and call it with `manifest.rubric_version`. Extend `load_worksheet`/`validate_worksheet` with `expected_version` and pass `manifest.rubric_version` at both scoring call sites. Keep allocation/result binding to the manifest version and reject a worksheet whose internal version differs before publication. + +Before (`scripts/agent_benchmark/scoring.py:1207-1222`): + +```python +def _prompt(blind: BlindWorkspace) -> bytes: + categories = ", ".join( + f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES + ) + ... + "landing-quality-v1, integer scores within each maximum ..." +``` + +After: + +```python +def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes: + categories = ", ".join( + f"{ident} ({maximum})" + for ident, maximum in rubric_categories(rubric_version) + ) + ... + f"{rubric_version}, integer scores within each maximum ..." +``` + +The `blind` parameter may remain for interface stability even though it is not interpolated. Do not expose producer identity. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/scoring.py`: select prompt categories/version and worksheet validation from `manifest.rubric_version`. +- [ ] `scripts/agent_benchmark/scoring_test.py`: add a new-version scoring success and cross-version failure test. + +**Test Strategy** + +Write `test_manifest_selected_rubric_drives_prompt_and_worksheet_validation`. Build a new-version manifest, emit the exact seven-category worksheet, and assert success plus prompt category/version content. On a fresh score attempt, emit a structurally valid legacy worksheet and assert `invalid_worksheet` with no scored worksheet publication. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +``` + +Expected: selected-version success and cross-version rejection both pass. + +### [API-3] Preserve historical compatibility and exact new semantics + +**Problem** + +An additive version is safe only if old manifests, prompts, worksheets, and immutable score reloads retain byte-level meaning. + +**Solution** + +Keep existing example manifests on `landing-quality-v1`. Make test helpers accept a version instead of replacing their legacy default. Add explicit assertions that legacy canonical worksheet bytes and prompts remain unchanged, while the new version has seven ordered categories and total 100. Update the project benchmark skill's scoring phase so it requires the exact rubric selected by the immutable manifest and names both supported versions; do not leave its current line 87 fixed to the legacy version. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/rubric_test.py`: dual-version canonical and boundary assertions. +- [ ] `scripts/agent_benchmark/scoring_test.py`: legacy default plus new-version selected behavior. +- [ ] `scripts/agent_benchmark/manifest_test.py`: both versions and shipped examples. +- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact rubric behavior and the closed supported version set. + +**Test Strategy** + +Write tests; do not mass-replace legacy version literals in unrelated fixtures. The new final manifest is added only by the dependent packet. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md +``` + +Expected: all legacy and new rubric tests report `OK`. + +### [API-4] Run the complete benchmark regression + +**Problem** + +Rubric validation is used by scoring recovery and durable result reads, so focused tests are not enough. + +**Solution** + +Run all rubric/scoring and the existing benchmark suites, validate shipped legacy examples, and check whitespace. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G08.md`: record exact decisions, deviations, and all actual command output. + +**Test Strategy** + +No further files. Existing recovery, redaction, scoring retry, manifest digest, and connectivity suites are the full oracle. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: three legacy manifests validate, all tests report `OK`, and whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/manifest.py` | API-1 | +| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 | +| `scripts/agent_benchmark/rubric.py` | API-1 | +| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 | +| `scripts/agent_benchmark/manifest_test.py` | API-1, API-3 | +| `scripts/agent_benchmark/scoring.py` | API-2 | +| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md` | API-4 | + +## Final Verification + +Run from `/config/workspace/iop-s0` after predecessor 01 is complete; fresh output is required and cached output is not acceptable. + +```bash +python3 - <<'PY' +from pathlib import Path +active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log") +archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, candidates +print(f"ok: predecessor complete {candidates[0]}") +PY +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: dependency check and every command exit 0, legacy examples validate, all suites report `OK`, and `git diff --check` is silent. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_1.log new file mode 100644 index 00000000..e04087bf --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_1.log @@ -0,0 +1,307 @@ + + +# Plan - Versioned One-Shot Benchmark Rubric Contract + +## For the Implementing Agent + +Do not start until exactly one active or archived `01_execution_order_contract/complete.log` is resolved by the dependency command below. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The approved one-shot comparison SDD fixes seven rubric categories totaling 100, while the existing `landing-quality-v1` worksheet contains five different categories. Reusing that version name would mutate historical meaning, so this packet preserves legacy runs and adds a manifest-selected version for the approved benchmark rubric. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found a semantic contradiction: Scope Rationale prohibited a project-skill edit while API-3 and the file summary required that same edit, creating ownership overlap with the route-preflight packet. +- Replan keeps this packet on the dormant additive rubric implementation and dual-version code tests. The dependent final-manifest packet owns the operational skill transition when it selects the new rubric. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/rubric.py` +- `scripts/agent_benchmark/rubric_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=fixture-lock`. +- Target scenario: S01 requires the same fixture, viewport, rubric checksum/version for every cell. The approved rubric at SDD lines 90-91 is 요구사항 충족 25, 시각 완성도 25, 반응형·접근성 15, 이미지 활용·디테일 10, 동작 안정성 10, 코드 품질 10, 자체 검증 완결성 5. +- Evidence Map row: S01 requires fixture prompt/assets/workspace/rubric digest evidence. +- The checklist therefore creates a distinct immutable rubric version, retains legacy validation, and binds prompt/output validation to the manifest-selected version. The final manifest packet supplies the concrete version/digest evidence. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the complete manifest, rubric, scoring, and test files above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all example manifests validated and the 178-test manifest/attempt/connectivity suite passed. `rubric_test` and `scoring_test` are additional required fresh suites for this packet. +- Current `RUBRIC_CATEGORIES` is five entries totaling 100 (`rubric.py:21-27`). `validate_worksheet` accepts only global `RUBRIC_VERSION` (`rubric.py:82-126`), and `_prompt` hardcodes those categories and `landing-quality-v1` (`scoring.py:1207-1222`). +- The safe compatibility rule is additive: keep `landing-quality-v1` and its five-category table readable; add `one-shot-agent-comparison-v1` with the approved seven-category table; reject unknown and cross-version worksheets. +- Fresh output is required; cached output is not accepted. No external runner is required. + +### Test Coverage Gaps + +- Existing rubric tests validate only the five-category legacy table. +- Existing scoring tests generate only legacy worksheets and cannot prove prompt/worksheet selection from the manifest. +- Existing manifest/schema tests accept only one rubric constant. +- No test rejects a valid legacy worksheet when the allocation requires the new rubric, or vice versa. + +### Symbol References + +- `manifest.RUBRIC_VERSION` is referenced only by `manifest.py:711` and `rubric.py:16,83,126`. Replace internal single-version validation with a closed version catalog while preserving a legacy alias if needed for compatibility. +- `rubric.RUBRIC_CATEGORIES` is imported by `scoring.py:44-49`, `scoring_test.py:56`, and `rubric_test.py:10`. Preserve it as the legacy tuple for existing callers, add a version lookup, and migrate production scoring to the lookup. +- `_prompt` is private and called at `scoring.py:1879`; change both definition and call together. +- `load_worksheet` call sites are `scoring.py:1641` and `scoring.py:1972`; both must pass the manifest-selected expected version. + +### Split Judgment + +- This is dependent child `03+01_rubric_version_contract`. Its stable invariant is: rubric version selects exactly one immutable ordered category table from manifest load through evaluator prompt, worksheet validation, durable result, and historical reload. +- PASS evidence is dual-version loader/rubric/scoring tests plus full regression. +- Predecessor index 01 is currently active at `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` but has no `complete.log`; implementation is dependency-blocked until that exact file exists. The dependency is required because both packets modify `manifest.py`, `manifest_test.py`, and the schema. + +### Scope Rationale + +- Do not delete, rename, or reinterpret `landing-quality-v1`; old manifests and immutable run snapshots must still load and score under the old table. +- Do not change automatic web gate eligibility, evaluator identity blinding, score persistence, caller bindings, fixture bytes, or report layout. +- Do not update the project benchmark skill or living spec in this packet. The currently tracked manifests still select the legacy rubric; dependent packet 04 owns the atomic skill transition when it adds the first tracked new-version manifest. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`. +- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve exactly one predecessor `complete.log`: the active sibling path or one matching `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log`. +2. Implement this packet against the reviewed 01 source. Do not infer completion from an active PLAN or review stub. +3. On PASS, `04+02,03_locked_benchmark_manifest` may proceed only after both index 02 and this index 03 have `complete.log`. + +## Implementation Checklist + +- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior. +- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output. +- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric. +- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Add a closed dual-version rubric catalog + +**Problem** + +`manifest.py:30` exposes one allowed rubric version, and `rubric.py:21-27` gives that version a five-category table that differs from the approved SDD. Changing the existing tuple in place would silently rewrite historical `landing-quality-v1` meaning. + +**Solution** + +Keep the legacy string and tuple unchanged. Add `ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"`, a closed allowed-version tuple in `manifest.py`, and an ordered category mapping in `rubric.py`. The new table is exactly: + +```python +ONE_SHOT_RUBRIC_CATEGORIES = ( + ("requirements_fidelity", 25), + ("visual_completeness", 25), + ("responsive_accessibility", 15), + ("image_detail_usage", 10), + ("behavior_stability", 10), + ("code_quality", 10), + ("self_verification", 5), +) +``` + +Add `rubric_categories(version)` that fails closed for unknown versions. Update the manifest schema from a single `const` to an exact two-value enum. `validate_worksheet` must read the worksheet version, optionally require an `expected_version`, and validate against only that version's ordered table. + +Before (`scripts/agent_benchmark/rubric.py:21-27`): + +```python +RUBRIC_CATEGORIES = ( + ("task_fidelity", 25), + ("visual_hierarchy", 25), + ("responsive_composition", 20), + ("typography_readability", 15), + ("polish_consistency", 15), +) +``` + +After: retain that exact tuple as the legacy value and add the new tuple/mapping without mutation. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/manifest.py`: accept the closed legacy/new rubric version catalog. +- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: mirror the exact two-version enum. +- [ ] `scripts/agent_benchmark/rubric.py`: provide versioned category lookup and expected-version worksheet validation. +- [ ] `scripts/agent_benchmark/rubric_test.py`: verify exact tables, totals, ordering, unknown versions, and cross-version rejection. +- [ ] `scripts/agent_benchmark/manifest_test.py`: verify loader/schema parity for both allowed versions and an unknown version. + +**Test Strategy** + +Write tests. Preserve every existing legacy assertion. Add `test_one_shot_rubric_exact_categories_and_total_are_accepted`, `test_cross_version_worksheet_is_rejected`, and manifest/schema two-version parity. Assert both category sums equal exactly 100. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +``` + +Expected: both versions pass their exact table; malformed, unknown, reordered, and cross-version cases fail closed. + +### [API-2] Bind prompt and durable score validation to the manifest version + +**Problem** + +`_prompt` uses global categories and a literal version (`scoring.py:1207-1222`), while both `load_worksheet` calls (`scoring.py:1641`, `1972`) validate only the global legacy version. A new manifest version could therefore ask for or accept the wrong worksheet. + +**Solution** + +Change `_prompt` to accept `rubric_version`, look up its exact table, render that version string, and call it with `manifest.rubric_version`. Extend `load_worksheet`/`validate_worksheet` with `expected_version` and pass `manifest.rubric_version` at both scoring call sites. Keep allocation/result binding to the manifest version and reject a worksheet whose internal version differs before publication. + +Before (`scripts/agent_benchmark/scoring.py:1207-1222`): + +```python +def _prompt(blind: BlindWorkspace) -> bytes: + categories = ", ".join( + f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES + ) + ... + "landing-quality-v1, integer scores within each maximum ..." +``` + +After: + +```python +def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes: + categories = ", ".join( + f"{ident} ({maximum})" + for ident, maximum in rubric_categories(rubric_version) + ) + ... + f"{rubric_version}, integer scores within each maximum ..." +``` + +The `blind` parameter may remain for interface stability even though it is not interpolated. Do not expose producer identity. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/scoring.py`: select prompt categories/version and worksheet validation from `manifest.rubric_version`. +- [ ] `scripts/agent_benchmark/scoring_test.py`: add a new-version scoring success and cross-version failure test. + +**Test Strategy** + +Write `test_manifest_selected_rubric_drives_prompt_and_worksheet_validation`. Build a new-version manifest, emit the exact seven-category worksheet, and assert success plus prompt category/version content. On a fresh score attempt, emit a structurally valid legacy worksheet and assert `invalid_worksheet` with no scored worksheet publication. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +``` + +Expected: selected-version success and cross-version rejection both pass. + +### [API-3] Preserve historical compatibility and exact new semantics + +**Problem** + +An additive version is safe only if old manifests, prompts, worksheets, and immutable score reloads retain byte-level meaning. + +**Solution** + +Keep existing example manifests on `landing-quality-v1`. Make test helpers accept a version instead of replacing their legacy default. Add explicit assertions that legacy canonical worksheet bytes and prompts remain unchanged, while the new version has seven ordered categories and total 100. Do not change the project benchmark skill in this dependency packet: all currently tracked manifests still select the legacy version, and the dependent final-manifest packet owns the atomic operational transition to manifest-selected wording. + +**Modified Files and Checklist** + +- [ ] `scripts/agent_benchmark/rubric_test.py`: dual-version canonical and boundary assertions. +- [ ] `scripts/agent_benchmark/scoring_test.py`: legacy default plus new-version selected behavior. +- [ ] `scripts/agent_benchmark/manifest_test.py`: both versions and shipped examples. + +**Test Strategy** + +Write tests; do not mass-replace legacy version literals in unrelated fixtures. The new final manifest is added only by the dependent packet. + +**Verification** + +```bash +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +``` + +Expected: all legacy and new rubric tests report `OK`. + +### [API-4] Run the complete benchmark regression + +**Problem** + +Rubric validation is used by scoring recovery and durable result reads, so focused tests are not enough. + +**Solution** + +Run all rubric/scoring and the existing benchmark suites, validate shipped legacy examples, and check whitespace. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G08.md`: record exact decisions, deviations, and all actual command output. + +**Test Strategy** + +No further files. Existing recovery, redaction, scoring retry, manifest digest, and connectivity suites are the full oracle. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: three legacy manifests validate, all tests report `OK`, and whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/agent_benchmark/manifest.py` | API-1 | +| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 | +| `scripts/agent_benchmark/rubric.py` | API-1 | +| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 | +| `scripts/agent_benchmark/manifest_test.py` | API-1, API-3 | +| `scripts/agent_benchmark/scoring.py` | API-2 | +| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md` | API-4 | + +## Final Verification + +Run from `/config/workspace/iop-s0` after predecessor 01 is complete; fresh output is required and cached output is not acceptable. + +```bash +python3 - <<'PY' +from pathlib import Path +active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log") +archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, candidates +print(f"ok: predecessor complete {candidates[0]}") +PY +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test +python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: dependency check and every command exit 0, legacy examples validate, all suites report `OK`, and `git diff --check` is silent. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G06_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G06_2.log new file mode 100644 index 00000000..26aff3a9 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G06_2.log @@ -0,0 +1,248 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=2, tag=TEST + +## Archive Evidence Snapshot + +- Prior plan/review before Epic self-review: `plan_local_G07_0.log`, `code_review_cloud_G07_0.log`. +- Refinement source pair: `plan_cloud_G09_1.log`, `code_review_cloud_G09_1.log`. +- Neither prior pair contains implementation evidence or an official verdict. The refinement preserves the original manifest, skill-contract, and regression scope while moving only the external readiness preflight into child `05+04_readiness_preflight`. +- Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b` in the source plan evidence. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_2.log` and `PLAN-local-G06.md` → `plan_local_G06_2.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1 Create one exact immutable readiness manifest | [x] | +| TEST-3 Preserve the full benchmark baseline | [x] | + +## Implementation Checklist + +- [x] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog. +- [x] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`. +- [x] Archive active `PLAN-local-G06.md` to `plan_local_G06_2.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path. +- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [x] If PASS for split work, keep parent `agent-task/m-iop-one-shot-agent-model-comparison/` because child `05+04_readiness_preflight` remains active until its own review completes. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +- 없음. PLAN의 파일 범위와 검증 명령을 그대로 유지했다. +- 보조 검증으로 `skill-creator`의 `quick_validate.py`를 추가 실행했으며 계획된 검증을 대체하지 않았다. + +## Key Design Decisions + +- 새 manifest의 JSON matrix는 C01-C09 순서로 읽기 쉽게 유지하고, loader가 explicit seed로 산출한 immutable 실행 순서는 별도로 exact assertion했다. +- manifest regression은 raw JSON 전체 cell payload와 loader의 canonical payload를 모두 비교해 route kind/id, request effort, 다섯 stage 순서, cloud effort, `ornith-fast` effort 생략을 함께 잠갔다. +- scoring skill 문구는 기존 Procedure의 generic fallback-language 금지 계약을 유지하기 위해 `no substitute rubric or reinterpretation`으로 표현하고, 닫힌 두 rubric version literal과 manifest-selected semantics를 contract test로 고정했다. +- Spec update not needed: 기존 living spec의 manifest-selected rubric 및 immutable manifest 설명과 동작 범위는 그대로 정확하며 새 tracked bench-02 입력만 추가됐다. + +## Reviewer Checkpoints + +- Confirm both predecessor completion files exist before reviewing implementation. +- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact. +- Confirm hybrid routes use route kind `execution_preset`, include `repair`, bind every cloud stage to its exact model with high effort, and bind work to `ornith-fast` with no invented effort. +- Confirm project-skill scoring uses the exact manifest-selected rubric, names the closed two-version set, and preserves predecessor 02's all-cell preflight contract. +- Confirm no public live preflight or scored run/score/report command was executed in this child. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. + +### Dependency Verification + +```bash +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +``` + +_Actual output:_ + +```text +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log +``` + +### TEST-1 Verification + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +``` + +_Actual output:_ + +```text +ok: manifest is valid +. +---------------------------------------------------------------------- +Ran 1 test in 0.003s + +OK +.................................................. +---------------------------------------------------------------------- +Ran 50 tests in 1.705s + +OK +``` + +Supplemental skill validation: + +```text +$ python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/iop-agent-comparison-benchmark +Skill is valid! +``` + +### TEST-3 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +git diff --check +``` + +_Actual output:_ + +```text +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +.................................................................................................................................................................................................................................................................................................................................................................................................................................................. +---------------------------------------------------------------------- +Ran 434 tests in 118.623s + +OK +git diff --check: no output (exit 0) +``` + +### Reviewer Fresh Verification (2026-08-12) + +```text +$ python3 - <<'PY' # exact dependency verification block above +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log + +$ python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +ok: manifest is valid + +$ python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +. +---------------------------------------------------------------------- +Ran 1 test in 0.004s + +OK + +$ python3 -m unittest scripts.agent_benchmark.skill_contract_test +.................................................. +---------------------------------------------------------------------- +Ran 50 tests in 1.716s + +OK + +$ for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid +ok: manifest is valid + +$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +---------------------------------------------------------------------- +Ran 434 tests in 124.077s + +OK + +$ git diff --check +(no output; exit 0) +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +- Overall Verdict: PASS +- Dimension Assessment: + - Correctness: Pass — tracked manifest fields, fixture checksum, two image paths, evaluator, C01-C09 direct/preset bindings, omitted work-stage effort, and seeded order match the approved plan and SDD. + - Completeness: Pass — TEST-1 and TEST-3 are implemented; both predecessor completion records resolve exactly once, and external readiness remains correctly isolated in child `05+04_readiness_preflight`. + - Test coverage: Pass — fresh focused manifest/skill tests, four-manifest validation, and all 434 discovered benchmark tests pass. + - API contract: Pass — direct and execution-preset route identities preserve the OpenAI/Anthropic ingress and Edge config ownership boundaries without fallback or substitution. + - Code quality: Pass — the exact regression compares raw and canonical payloads, the skill contract mutation-tests legacy-only wording, and `git diff --check` is clean. + - Implementation deviation: Pass — planned production/documentation files are present; the supplemental skill validation did not replace required verification, and predecessor/sibling worktree changes were excluded from this verdict. + - Verification trust: Pass — every applicable plan command was rerun against the current worktree and reproduced the implementation handoff; no `agent-test/runs/bench-02` execution state exists. + - Spec conformance: Pass — SDD S01/S03 and Evidence Map entries for `fixture-lock,matrix-lock` are satisfied; the current living spec already describes immutable manifest-selected rubric behavior, so a spec update is not needed. +- Findings: None +- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false` +- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock,matrix-lock` for runtime aggregation. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log new file mode 100644 index 00000000..1f33f3ac --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log @@ -0,0 +1,209 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=0, tag=TEST + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-local-G07.md` → `plan_local_G07_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1 Create one exact immutable readiness manifest | [ ] | +| TEST-2 Produce one redacted nine-cell readiness record | [ ] | +| TEST-3 Preserve the full benchmark baseline | [ ] | + +## Implementation Checklist + +- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order. +- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G07.md` to `code_review_cloud_G07_0.log`. +- [ ] Archive active `PLAN-local-G07.md` to `plan_local_G07_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm both predecessor completion files exist before reviewing implementation. +- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact. +- Confirm hybrid routes include `repair`, cloud stages use high, and work uses `ornith-fast` with no invented effort. +- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value. +- Confirm no scored run/score/report command was executed in this Epic. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. TEST-2's full command block is fixed in the plan and must be copied with its actual output or exact blocker here. + +### Dependency Verification + +```bash +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +``` + +_Actual output:_ + +### TEST-1 Verification + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +``` + +_Actual output:_ + +### TEST-2 External Preflight + +```bash +set -euo pipefail +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" +test -z "$(git -C ../iop-s2 status --short)" +test -x ../iop-s2/build/bin/iop-edge +test -x ../iop-s2/build/dev/iop-node +test -f ../iop-s2/configs/edge.yaml +claude --version +agy --version +codex --version +agy_help="$(agy --help 2>&1)"; for token in --print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json; do grep -F -- "$token" <<<"$agy_help" >/dev/null; done +python3 - <<'PY' +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)" +printf '%s\n' "$preflight_output" +case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +python3 - "$run_id" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert record["status"] == "ready" +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine ready results are manifest-bound and runtime values are absent") +PY +``` + +_Actual output or exact blocker and resume condition:_ + +### TEST-3 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +_Actual output:_ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G09_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G09_1.log new file mode 100644 index 00000000..b6bd5e24 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G09_1.log @@ -0,0 +1,243 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=1, tag=TEST + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found that the final manifest would select the new rubric while the public project skill remained fixed to `landing-quality-v1`; it also found an incomplete agy resume gate and ambiguous testbed source provenance. +- Replan atomically assigns rubric skill/contract-test synchronization to this dependent packet, requires the adapter-known agy version plus its documented transport surface, and records the independent read-only testbed's exact clean commit without requiring equality to the orchestration repository. A fresh read-only probe also found the current Edge artifact is Mach-O and both runtime artifacts predate that commit, so operator rebuild is an explicit resume condition. Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1 Create one exact immutable readiness manifest | [ ] | +| TEST-2 Produce one redacted nine-cell readiness record | [ ] | +| TEST-3 Preserve the full benchmark baseline | [ ] | + +## Implementation Checklist + +- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog. +- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`. +- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm both predecessor completion files exist before reviewing implementation. +- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact. +- Confirm hybrid routes include `repair`, cloud stages use high, and work uses `ornith-fast` with no invented effort. +- Confirm project-skill scoring uses the exact manifest-selected rubric, names the closed two-version set, and preserves predecessor 02's all-cell preflight contract. +- Confirm the external gate captures the clean independent `../iop-s2` dev HEAD and accepts only the production adapter's exact known agy version/help contract. +- Confirm both iop-s2 artifacts are current-HEAD AArch64 ELF executables before accepting endpoint readiness; `test -x` alone is insufficient. +- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value. +- Confirm no scored run/score/report command was executed in this Epic. + +## Verification Results + +Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. TEST-2's full command block is fixed in the plan and must be copied with its actual output or exact blocker here. + +### Dependency Verification + +```bash +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +``` + +_Actual output:_ + +### TEST-1 Verification + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +``` + +_Actual output:_ + +### TEST-2 External Preflight + +```bash +set -euo pipefail +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" +test -z "$(git -C ../iop-s2 status --short)" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +test -n "$testbed_head" +printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head" +command -v readelf >/dev/null +test -x ../iop-s2/build/bin/iop-edge +test -x ../iop-s2/build/dev/iop-node +test -f ../iop-s2/configs/edge.yaml +commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test "$(stat -c %Y "$binary")" -ge "$commit_epoch" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null +done +../iop-s2/build/bin/iop-edge --help >/dev/null +test -n "$(../iop-s2/build/dev/iop-node version)" +printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n' +claude --version +agy --version +codex --version +python3 - <<'PY' +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +version = (version_run.stdout + version_run.stderr).strip() +help_text = help_run.stdout + help_run.stderr +capability = inspect_agy_iop_capability(version, help_text) +assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION) +assert capability.iop_transport_supported, capability +print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented") +PY +python3 - <<'PY' +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)" +printf '%s\n' "$preflight_output" +case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +python3 - "$run_id" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert record["status"] == "ready" +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine ready results are manifest-bound and runtime values are absent") +PY +``` + +_Actual output or exact blocker and resume condition:_ + +### TEST-3 and Final Verification + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +_Actual output:_ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log new file mode 100644 index 00000000..b4172523 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log @@ -0,0 +1,45 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest + +## 완료 일시 + +2026-08-12 + +## 요약 + +두 차례 구현 전 범위 정리와 한 차례 공식 코드 리뷰를 거쳐 immutable C01-C09 manifest와 manifest-selected rubric skill 계약을 검증했으며 최종 판정은 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | 재계획 | Epic self-review에서 rubric skill 동기화와 외부 readiness gate 보강 필요를 확인했으며 구현 및 공식 판정은 없었다. | +| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | 분할 재계획 | 외부 readiness preflight를 child `05+04_readiness_preflight`로 분리하고 현재 child를 deterministic manifest 계약으로 한정했다. | +| `plan_local_G06_2.log` | `code_review_cloud_G06_2.log` | PASS | fixture checksum, C01-C09 payload와 seed 순서, dual-rubric skill 계약 및 전체 434-test 회귀를 확인했다. | + +## 구현/정리 내용 + +- 승인된 fixture, 두 이미지, viewport, timeout, evaluator와 C01-C09 direct/execution-preset binding을 하나의 tracked bench-02 manifest로 고정했다. +- raw manifest와 canonical loader payload, explicit-seed 실행 순서, fixture checksum 및 manifest digest를 exact regression으로 잠갔다. +- benchmark project skill이 manifest-selected rubric의 닫힌 두 version만 사용하고 legacy fallback 또는 재해석을 금지하도록 contract test와 함께 동기화했다. +- 기존 living spec은 immutable manifest-selected rubric 동작을 이미 정확히 설명하므로 갱신하지 않았다. + +## 최종 검증 + +- `python3 - <<'PY' ... PY` - PASS; 선행 `02_route_preflight_contract`, `03+01_rubric_version_contract` 완료 로그가 각각 정확히 한 건 확인됐다. +- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`. +- `python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness` - PASS; 1 test, `OK`. +- `python3 -m unittest scripts.agent_benchmark.skill_contract_test` - PASS; 50 tests, `OK`. +- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 네 manifest 모두 유효하다. +- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 434 tests, `OK`. +- `git diff --check` - PASS; 출력 없음. +- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic manifest child 범위가 아니며 외부 readiness는 child `05+04_readiness_preflight`가 소유하므로 실행하지 않았다. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- `05+04_readiness_preflight`가 이 reviewed manifest를 사용해 별도 external readiness gate를 수행한다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log new file mode 100644 index 00000000..d766498c --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log @@ -0,0 +1,363 @@ + + +# Plan - Locked C01-C09 Benchmark Readiness Manifest + +## For the Implementing Agent + +Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact live route evidence. This packet creates that final tracked input and runs the public preflight without substituting missing callers, credentials, models, or presets. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log`. +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log`. +- The prior pair contains no implementation evidence or official verdict. Epic self-review found that the final manifest would select the new rubric while the public project skill remained fixed to `landing-quality-v1`; it also found an incomplete agy resume gate and ambiguous testbed source provenance. +- Replan atomically assigns rubric skill/contract-test synchronization to this dependent packet, requires the adapter-known agy version plus its documented transport surface, and records the independent read-only testbed's exact clean commit without requiring equality to the orchestration repository. A fresh read-only probe also found the current Edge artifact is Mach-O and both runtime artifacts predate that commit, so operator rebuild is an explicit resume condition. Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `scripts/agent_benchmark/rubric.py` +- `scripts/agent_benchmark/rubric_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `scripts/fixtures/agent-comparison-benchmark/prompt.md` +- `scripts/fixtures/agent-comparison-benchmark/reference.txt` +- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg` +- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=fixture-lock,route-readiness,matrix-lock`. +- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version; S02 requires a redacted all-cell preflight or exact blocker; S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings. +- Evidence Map rows S01-S03 drive the static manifest assertions, secret-safe external preflight, and durable nine-result inspection below. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the complete fixture, manifest/schema, public CLI, adapter/store tests, SDD, spec, and contracts listed above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three shipped manifests validated; `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` ran 178 tests and passed. +- Fixture evidence: prompt requires exactly root `index.html`, `styles.css`, `script.js`, vanilla HTML/CSS/JS, both local images, responsive desktop/mobile behavior, accessibility, and no external dependency. The assets resolve to exactly two images plus one reference file; checksum is `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`. Viewports are 1920x1080 and 375x812. +- The direct example already fixes C01-C05 model/effort pairs. The approved SDD names the hybrid scenarios `gemini-hybrid` and `gpt-hybrid`; this packet uses those exact strings as both virtual request model and preset route id. Missing registration must remain `registration_required`, never fallback. +- Planned seeded order for `bench-02-c01-c09-v1` under the reviewed domain-separated algorithm is: C02, C05, C03, C06, C08, C09, C01, C07, C04. +- Fresh output is required; cached output is not accepted. + +#### External Verification Preflight + +- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`. +- Testbed: `/config/workspace/iop-s2`, branch `dev`, exact observed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean at planning time. It is the manifest-selected independent read-only runtime, so equality with the iop-s0 orchestration HEAD is neither expected nor a valid sync test; execution must capture its exact current `dev` HEAD and clean state as provenance before using existing artifacts. +- Artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`, and `../iop-s2/configs/edge.yaml` exist, but existence is insufficient. The Edge magic bytes are Mach-O (`cf fa ed fe`) and it exits 126 with `Exec format error` on Linux; Node is AArch64 ELF. Both artifact mtimes predate current testbed HEAD (`2026-08-10T00:09:38+09:00`), so they are stale and Edge is host-incompatible. The tracked example config exposes only local example models and no benchmark route/preset catalog; do not treat it as live registration. +- Callers: Claude Code `2.1.227`, agy `1.1.12`, Codex CLI `0.147.0` were installed. Claude/Codex help probes completed. The production adapter accepts only `agy` `1.1.11`; current `1.1.12` also lacks the required `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, and `AGY_OPENAI_API_KEY` help tokens, so it is definitively incompatible rather than merely missing a few tokens. +- Runtime/config: every `IOP_BENCH_{CLAUDE,AGY,CODEX}_{BASE_URL,SECRET_ENV}` and `IOP_BENCH_CONFIG_OBSERVATION_ENV` was missing. Consequently runtime identity, endpoint hosts/ports, live catalog, auth, and route registration could not be observed. `ss` is unavailable; the public preflight's endpoint probe is the authoritative reachability check once URLs exist. +- Exact resume condition: an operator must rebuild both iop-s2 artifacts from its exact clean `dev` HEAD for Linux AArch64, provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run those rebuilt Edge/Node endpoints, and install adapter-known `agy` `1.1.11` whose help satisfies every documented transport token. Execution proves both artifacts are AArch64 ELF, are not older than the captured commit, and expose their help/version commands. If adapter version support changes in a reviewed predecessor, use that code's exact `AGY_KNOWN_VERSION` and capability parser instead. No raw value is written to the repository or review file. +- If the public preflight returns `registration_required` or `implementation_gap`, preserve its run record, record only the closed summary and run id in the review evidence, and stop. Do not edit `../iop-s2`, install tools, substitute models, or invoke callers outside the benchmark CLI. + +### Test Coverage Gaps + +- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version. +- Existing generic preset examples do not express Gemini/GPT plan→ornith-fast work→review/repair stage bindings. +- No static regression asserts the fixture checksum, exact two-image paths, policy fields, cell map, seeded order, and hybrid binding table together. +- The project benchmark skill still hardcodes the legacy worksheet, and its contract test does not require the closed version catalog or manifest-selected scoring language. +- Live readiness is currently blocked by stale/host-incompatible testbed artifacts, missing environment/config, and incompatible agy version/help evidence; the plan contains the exact secret-safe resume/preflight command. + +### Symbol References + +- No symbol is renamed or removed. +- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced. + +### Split Judgment + +- Direct-small classification produced zero code changes. The existing fixture bytes already meet the prompt/image/workspace requirements, but their final version/checksum evidence is inseparable from the planned seeded/rubric manifest and all-cell preflight. Editing a partial manifest before those contracts land would collide with planned work. +- This is dependent child `04+02,03_locked_benchmark_manifest`. Its stable invariant is one tracked manifest whose fixture, policies, nine exact cells, order, rubric, and durable preflight all agree. +- Predecessor index 02 is active at `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` without `complete.log`; predecessor index 03 is active at `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` without `complete.log`. Both are currently unsatisfied; do not implement until both exact completion files exist. + +### Scope Rationale + +- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations. +- Update only the project skill's scoring wording and matching contract assertions here; preserve the all-cell preflight wording delivered by predecessor 02. +- Do not run scored `run`, `resume`, `score`, or `report`; this Epic ends at immutable input plus readiness preflight. +- Do not use the generic example aliases, omit `repair`, lower efforts, or replace unavailable models/presets. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+2+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`. +- Review closures: all closed. Scores `2+1+2+2+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`. +- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but grade remains the basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. External rebuild/registration is a named execution precondition, not a cloud-resolvable planning gap. + +## Dependencies and Execution Order + +1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`. +2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`; its own index-01 dependency is already encoded there. +3. Create and statically validate the locked manifest and regression test. +4. Run the secret-safe external preflight once. Stop on its exact closed blocker; do not proceed to scored execution. + +## Implementation Checklist + +- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog. +- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Create one exact immutable readiness manifest + +**Problem** + +The generic fixture manifest has only three homogeneous presets (`agent-comparison-benchmark-manifest.example.json:42-91`), while the direct preflight example has only C01-C05 equivalents (`agent-comparison-benchmark-direct-preflight.example.json:42-108`). Neither can satisfy S01-S03. + +**Solution** + +Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields: + +```json +{ + "pipeline_version": "2", + "environment": "dev", + "testbed": "../iop-s2", + "execution_order_seed": "bench-02-c01-c09-v1", + "repetitions": 1, + "session_policy": "fresh", + "setup_cache_policy": "isolated", + "timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5}, + "rubric_version": "one-shot-agent-comparison-v1", + "output_root": "agent-test/runs/bench-02" +} +``` + +Reuse the existing evaluator, fixture paths/checksum, and viewports byte-for-byte: `desktop_1080` at 1920x1080 and `mobile_375` at 375x812. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort. + +| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings | +|----|---------|--------|---------------|----------------------|-------------------| +| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request Sonnet/max | +| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high | +| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high | +| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh | +| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh | +| C06 | `c06-claude-gemini-hybrid` | claude | preset / `gemini-hybrid` | `gemini-hybrid` / high | Gemini/high, Gemini/high, ornith-fast, Gemini/high, Gemini/high | +| C07 | `c07-agy-gemini-hybrid` | agy | preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 | +| C08 | `c08-claude-gpt-hybrid` | claude | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | GPT/high, GPT/high, ornith-fast, GPT/high, GPT/high | +| C09 | `c09-codex-gpt-hybrid` | codex | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 | + +Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order: + +```text +c02-claude-gemini-direct +c05-codex-gpt-direct +c03-agy-gemini-direct +c06-claude-gemini-hybrid +c08-claude-gpt-hybrid +c09-codex-gpt-hybrid +c01-claude-sonnet-direct +c07-agy-gemini-hybrid +c04-claude-gpt-direct +``` + +In the same dependent change, replace the project skill's fixed `landing-quality-v1` scoring sentence with a requirement to use the exact immutable manifest-selected rubric. Name the closed supported set (`landing-quality-v1`, `one-shot-agent-comparison-v1`) and forbid fallback or reinterpretation. Extend `skill_contract_test.py` so its base contract requires that wording and both version literals, while a mutation back to fixed-legacy wording fails. Preserve predecessor 02's all-cell preflight assertions unchanged. + +**Modified Files and Checklist** + +- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs. +- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact scoring and the closed two-version rubric set. +- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require that scoring contract without weakening the all-cell preflight contract. + +**Test Strategy** + +Write the named manifest test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters. In the skill contract test, assert both exact supported rubric versions and manifest selection, then mutation-test the legacy-only regression. + +**Verification** + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +``` + +Expected: the manifest validates, the exact lock test passes, and the skill contract suite accepts only manifest-selected rubric wording. + +### [TEST-2] Produce one redacted nine-cell readiness record + +**Problem** + +Planning-time external preflight is blocked: benchmark environment references are missing, the dev runtime identity is unknown, and agy help lacks three adapter-required tokens. Static fixtures cannot claim live readiness. + +**Solution** + +First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them. + +If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G09.md` and stop. Do not mutate external state. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G09.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence. + +**Test Strategy** + +No additional unit test; this is required external execution evidence. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly. + +**Verification** + +```bash +set -euo pipefail +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" +test -z "$(git -C ../iop-s2 status --short)" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +test -n "$testbed_head" +printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head" +command -v readelf >/dev/null +test -x ../iop-s2/build/bin/iop-edge +test -x ../iop-s2/build/dev/iop-node +test -f ../iop-s2/configs/edge.yaml +commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test "$(stat -c %Y "$binary")" -ge "$commit_epoch" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null +done +../iop-s2/build/bin/iop-edge --help >/dev/null +test -n "$(../iop-s2/build/dev/iop-node version)" +printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n' +claude --version +agy --version +codex --version +python3 - <<'PY' +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +version = (version_run.stdout + version_run.stderr).strip() +help_text = help_run.stdout + help_run.stderr +capability = inspect_agy_iop_capability(version, help_text) +assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION) +assert capability.iop_transport_supported, capability +print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented") +PY +python3 - <<'PY' +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)" +printf '%s\n' "$preflight_output" +case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +python3 - "$run_id" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert record["status"] == "ready" +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine ready results are manifest-bound and runtime values are absent") +PY +``` + +Expected: every setup command exits 0; preflight prints one ready summary with nine results; durable inspection prints its safe success line. At planning time this block is expected to stop at agy/environment setup until the recorded resume condition is satisfied. + +### [TEST-3] Preserve the full benchmark baseline + +**Problem** + +The final file consumes the new order, rubric, and all-cell contracts. It must not regress legacy fixtures or other benchmark behavior. + +**Solution** + +Validate all four manifests, run the focused static test and complete rubric/scoring/manifest/attempt/connectivity suites, and check the diff. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G09.md`: record final command output after live readiness succeeds; if TEST-2 is blocked, leave this item unchecked and record the resume condition. + +**Test Strategy** + +No additional files beyond TEST-1. Existing suites are the regression oracle; external preflight output is not cached. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +Expected: four manifests validate, all suites report `OK`, and whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 | +| `scripts/agent_benchmark/manifest_test.py` | TEST-1 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | TEST-1 | +| `scripts/agent_benchmark/skill_contract_test.py` | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G09.md` | TEST-2, TEST-3 | + +## Final Verification + +Run from `/config/workspace/iop-s0` in dependency order. Fresh output is required; cached output is not acceptable. Run the full TEST-2 command block exactly once after its setup checks pass, then run: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test +git diff --check +``` + +Expected: static validation/test pass, TEST-2 has one nine-ready durable preflight record with no runtime values, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No scored benchmark command is run. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G06_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G06_2.log new file mode 100644 index 00000000..f2495de9 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G06_2.log @@ -0,0 +1,244 @@ + + +# Plan - Locked C01-C09 Benchmark Manifest + +## For the Implementing Agent + +Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G06.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact policies. This packet creates and statically validates that tracked input before the dependent readiness-preflight packet contacts any external caller or runtime. + +## Archive Evidence Snapshot + +- Prior plan/review before Epic self-review: `plan_local_G07_0.log`, `code_review_cloud_G07_0.log`. +- Refinement source pair: `plan_cloud_G09_1.log`, `code_review_cloud_G09_1.log`. +- Neither prior pair contains implementation evidence or an official verdict. The refinement preserves the original manifest, skill-contract, and regression scope while moving only the external readiness preflight into child `05+04_readiness_preflight`. +- Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b` in the source plan evidence. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/rubric.py` +- `scripts/agent_benchmark/rubric_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `scripts/fixtures/agent-comparison-benchmark/prompt.md` +- `scripts/fixtures/agent-comparison-benchmark/reference.txt` +- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg` +- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` + +### SDD Criteria + +- SDD status is `[승인됨]`, implementation lock is released, and no unresolved user review exists. +- First-line scope is narrowed to `milestone-task=fixture-lock,matrix-lock`. +- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version. S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings. +- The static manifest assertions and complete benchmark regression provide this child’s S01/S03 evidence. Live S02 evidence belongs to child 05. + +### Verification Context + +- The source plan fixed fixture checksum `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`, viewports 1920x1080 and 375x812, and seeded order C02, C05, C03, C06, C08, C09, C01, C07, C04. +- The approved hybrid route ids are `gemini-hybrid` and `gpt-hybrid`; stage order is selector, plan, work, review, repair. Cloud stages use `high`, while `ornith-fast` work omits effort. +- The manifest/schema route-kind contract accepts only `direct` and `execution_preset`; the locked hybrid cells must use `execution_preset` exactly. +- Fresh static validation and regression output is required; cached output is not accepted. No external runner is required for this child. + +### Test Coverage Gaps + +- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version. +- No static regression asserts fixture checksum, exact two-image paths, policies, full cell map, seeded order, and hybrid binding table together. +- The project benchmark skill still hardcodes the legacy worksheet, and its contract test does not require manifest-selected scoring with the closed rubric catalog. + +### Symbol References + +- No symbol is renamed or removed. +- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced. + +### Split Judgment + +- This child owns the complete production/documentation change and every repository regression required for its PASS. +- Child `05+04_readiness_preflight` depends on this reviewed manifest and owns only the additional external closure verification. It does not modify the manifest or duplicate the static regression. +- Existing indices 01-04 remain fixed because their task directories contain logs; the new closure child uses the next free index 05. + +### Scope Rationale + +- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations. +- Update only the project skill's scoring wording and matching contract assertions; preserve predecessor 02's all-cell preflight wording. +- Do not run public live preflight or any scored `run`, `resume`, `score`, or `report` command in this child. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures are all closed. Scores `2+0+2+1+1=G06`; base/final route `local-fit`, lane `local`, catalog `worker/local/G06`, filename `PLAN-local-G06.md`. +- Review closures are all closed. Scores `2+0+2+1+1=G06`; route `official-review`, lane `cloud`, catalog `review/cloud/G06`, filename `CODE_REVIEW-cloud-G06.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`. +2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`. +3. Create and statically validate the locked manifest and project-skill scoring contract. +4. On PASS, child `05+04_readiness_preflight` may run the external readiness gate. + +## Implementation Checklist + +- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog. +- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Create one exact immutable readiness manifest + +**Problem** + +The generic fixture manifest has only three homogeneous presets (`scripts/fixtures/agent-comparison-benchmark-manifest.example.json:44`), while the direct preflight example has only five direct cells (`scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json:44`). Neither can satisfy S01 and S03, and the current route-kind contract accepts `direct` or `execution_preset` only (`scripts/agent_benchmark/manifest.py:33`). + +**Solution** + +Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields: + +```json +{ + "pipeline_version": "2", + "environment": "dev", + "testbed": "../iop-s2", + "execution_order_seed": "bench-02-c01-c09-v1", + "repetitions": 1, + "session_policy": "fresh", + "setup_cache_policy": "isolated", + "timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5}, + "rubric_version": "one-shot-agent-comparison-v1", + "output_root": "agent-test/runs/bench-02" +} +``` + +Reuse the existing evaluator, fixture paths/checksum, and viewports byte-for-byte. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort. + +| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings | +|----|---------|--------|---------------|----------------------|-------------------| +| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request `claude-sonnet-5`/max | +| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / high | request `gemini-3.6-flash`/high | +| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / high | request `gemini-3.6-flash`/high | +| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / xhigh | request `gpt-5.6-luna`/xhigh | +| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / xhigh | request `gpt-5.6-luna`/xhigh | +| C06 | `c06-claude-gemini-hybrid` | claude | execution_preset / `gemini-hybrid` | `gemini-hybrid` / high | selector=`gemini-3.6-flash`/high; plan=`gemini-3.6-flash`/high; work=`ornith-fast`/omitted; review=`gemini-3.6-flash`/high; repair=`gemini-3.6-flash`/high | +| C07 | `c07-agy-gemini-hybrid` | agy | execution_preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 | +| C08 | `c08-claude-gpt-hybrid` | claude | execution_preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | selector=`gpt-5.6-terra`/high; plan=`gpt-5.6-terra`/high; work=`ornith-fast`/omitted; review=`gpt-5.6-terra`/high; repair=`gpt-5.6-terra`/high | +| C09 | `c09-codex-gpt-hybrid` | codex | execution_preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 | + +Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order: + +```text +c02-claude-gemini-direct +c05-codex-gpt-direct +c03-agy-gemini-direct +c06-claude-gemini-hybrid +c08-claude-gpt-hybrid +c09-codex-gpt-hybrid +c01-claude-sonnet-direct +c07-agy-gemini-hybrid +c04-claude-gpt-direct +``` + +In the same change, replace the project skill's fixed `landing-quality-v1` scoring sentence with a requirement to use the exact immutable manifest-selected rubric. Name the closed supported set (`landing-quality-v1`, `one-shot-agent-comparison-v1`) and forbid fallback or reinterpretation. Extend `skill_contract_test.py` so its base contract requires that wording and both version literals, while a mutation back to fixed-legacy wording fails. Preserve predecessor 02's all-cell preflight assertions unchanged. + +**Modified Files and Checklist** + +- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs. +- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact scoring and the closed two-version rubric set. +- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require that scoring contract without weakening the all-cell preflight contract. + +**Test Strategy** + +Write the named manifest test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters. In the skill contract test, assert both exact supported rubric versions and manifest selection, then mutation-test the legacy-only regression. + +**Verification** + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +``` + +Expected: the manifest validates, the exact lock test passes, and the skill contract suite accepts only manifest-selected rubric wording. + +### [TEST-3] Preserve the full benchmark baseline + +**Problem** + +The final file consumes the new order, rubric, and all-cell contracts through manifest loading (`scripts/agent_benchmark/manifest.py:651`), run storage/execution (`scripts/agent_benchmark/attempts.py:426`), scoring (`scripts/agent_benchmark/scoring.py:2006`), and the public CLI (`scripts/agent_comparison_benchmark.py:273`). It must not regress legacy fixtures or any other benchmark behavior. + +**Solution** + +Validate all four manifests, run the focused static test and the full benchmark discovery suite, and check the diff. Do not run live preflight in this child. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G06.md`: record final static command output. + +**Test Strategy** + +No additional files beyond TEST-1. Existing suites are the regression oracle. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +git diff --check +``` + +Expected: four manifests validate, the complete discovered benchmark suite reports `OK`, and whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 | +| `scripts/agent_benchmark/manifest_test.py` | TEST-1 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | TEST-1 | +| `scripts/agent_benchmark/skill_contract_test.py` | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md` | TEST-3 | + +## Final Verification + +Run from `/config/workspace/iop-s0` after both predecessors complete; fresh output is required and cached output is not acceptable. + +```bash +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +python3 -m unittest scripts.agent_benchmark.skill_contract_test +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +git diff --check +``` + +Expected: dependency check and every command exit 0, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No external preflight or scored benchmark command is run. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log new file mode 100644 index 00000000..dfd5229a --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log @@ -0,0 +1,320 @@ + + +# Plan - Locked C01-C09 Benchmark Readiness Manifest + +## For the Implementing Agent + +Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G07.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact live route evidence. This packet creates that final tracked input and runs the public preflight without substituting missing callers, credentials, models, or presets. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/manifest_test.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/agent_benchmark/rubric.py` +- `scripts/agent_benchmark/rubric_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json` +- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` +- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` +- `scripts/fixtures/agent-comparison-benchmark/prompt.md` +- `scripts/fixtures/agent-comparison-benchmark/reference.txt` +- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg` +- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review. +- First-line scope: `milestone-task=fixture-lock,route-readiness,matrix-lock`. +- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version; S02 requires a redacted all-cell preflight or exact blocker; S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings. +- Evidence Map rows S01-S03 drive the static manifest assertions, secret-safe external preflight, and durable nine-result inspection below. + +### Verification Context + +- No separate handoff was supplied. Repository-native evidence came from the complete fixture, manifest/schema, public CLI, adapter/store tests, SDD, spec, and contracts listed above. +- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three shipped manifests validated; `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` ran 178 tests and passed. +- Fixture evidence: prompt requires exactly root `index.html`, `styles.css`, `script.js`, vanilla HTML/CSS/JS, both local images, responsive desktop/mobile behavior, accessibility, and no external dependency. The assets resolve to exactly two images plus one reference file; checksum is `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`. Viewports are 1920x1080 and 375x812. +- The direct example already fixes C01-C05 model/effort pairs. The approved SDD names the hybrid scenarios `gemini-hybrid` and `gpt-hybrid`; this packet uses those exact strings as both virtual request model and preset route id. Missing registration must remain `registration_required`, never fallback. +- Planned seeded order for `bench-02-c01-c09-v1` under the reviewed domain-separated algorithm is: C02, C05, C03, C06, C08, C09, C01, C07, C04. +- Fresh output is required; cached output is not accepted. + +#### External Verification Preflight + +- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`. +- Testbed: `/config/workspace/iop-s2`, branch `dev`, observed HEAD `1f2f7f...`, clean at planning time. Source sync beyond the clean dev checkout was not asserted. +- Artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`, and `../iop-s2/configs/edge.yaml` exist. The tracked example config exposes only local example models and no benchmark route/preset catalog; do not treat it as live registration. +- Callers: Claude Code `2.1.227`, agy `1.1.12`, Codex CLI `0.147.0` were installed. Claude/Codex help probes completed. Current agy help contains `--print`, `--output-format`, `--sandbox`, `--model`, `--effort`, and `stream-json`, but lacks the adapter-required `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, and `AGY_OPENAI_API_KEY` tokens. +- Runtime/config: every `IOP_BENCH_{CLAUDE,AGY,CODEX}_{BASE_URL,SECRET_ENV}` and `IOP_BENCH_CONFIG_OBSERVATION_ENV` was missing. Consequently runtime identity, endpoint hosts/ports, live catalog, auth, and route registration could not be observed. `ss` is unavailable; the public preflight's endpoint probe is the authoritative reachability check once URLs exist. +- Exact resume condition: an operator must provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run the dev Edge/Node endpoints, and install an agy build whose help satisfies the current adapter capability tokens. No raw value is written to the repository or review file. +- If the public preflight returns `registration_required` or `implementation_gap`, preserve its run record, record only the closed summary and run id in the review evidence, and stop. Do not edit `../iop-s2`, install tools, substitute models, or invoke callers outside the benchmark CLI. + +### Test Coverage Gaps + +- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version. +- Existing generic preset examples do not express Gemini/GPT plan→ornith-fast work→review/repair stage bindings. +- No static regression asserts the fixture checksum, exact two-image paths, policy fields, cell map, seeded order, and hybrid binding table together. +- Live readiness is currently blocked by missing environment/config and incompatible agy help evidence; the plan contains the exact secret-safe resume/preflight command. + +### Symbol References + +- No symbol is renamed or removed. +- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced. + +### Split Judgment + +- Direct-small classification produced zero code changes. The existing fixture bytes already meet the prompt/image/workspace requirements, but their final version/checksum evidence is inseparable from the planned seeded/rubric manifest and all-cell preflight. Editing a partial manifest before those contracts land would collide with planned work. +- This is dependent child `04+02,03_locked_benchmark_manifest`. Its stable invariant is one tracked manifest whose fixture, policies, nine exact cells, order, rubric, and durable preflight all agree. +- Predecessor index 02 is active at `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` without `complete.log`; predecessor index 03 is active at `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` without `complete.log`. Both are currently unsatisfied; do not implement until both exact completion files exist. + +### Scope Rationale + +- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations. +- Do not run scored `run`, `resume`, `score`, or `report`; this Epic ends at immutable input plus readiness preflight. +- Do not use the generic example aliases, omit `repair`, lower efforts, or replace unavailable models/presets. + +### Final Routing + +- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `1+1+1+2+2=G07`; base/final route `local-fit`, lane `local`, catalog `worker/local/G07`, filename `PLAN-local-G07.md`. +- Review closures: all closed. Scores `1+1+1+2+2=G07`; route `official-review`, lane `cloud`, catalog `review/cloud/G07`, filename `CODE_REVIEW-cloud-G07.md`. +- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. External registration is a named execution precondition, not a cloud-resolvable planning gap. + +## Dependencies and Execution Order + +1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`. +2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`; its own index-01 dependency is already encoded there. +3. Create and statically validate the locked manifest and regression test. +4. Run the secret-safe external preflight once. Stop on its exact closed blocker; do not proceed to scored execution. + +## Implementation Checklist + +- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order. +- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Create one exact immutable readiness manifest + +**Problem** + +The generic fixture manifest has only three homogeneous presets (`agent-comparison-benchmark-manifest.example.json:42-91`), while the direct preflight example has only C01-C05 equivalents (`agent-comparison-benchmark-direct-preflight.example.json:42-108`). Neither can satisfy S01-S03. + +**Solution** + +Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields: + +```json +{ + "pipeline_version": "2", + "environment": "dev", + "testbed": "../iop-s2", + "execution_order_seed": "bench-02-c01-c09-v1", + "repetitions": 1, + "session_policy": "fresh", + "setup_cache_policy": "isolated", + "timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5}, + "rubric_version": "one-shot-agent-comparison-v1", + "output_root": "agent-test/runs/bench-02" +} +``` + +Reuse the existing evaluator, fixture paths/checksum, and desktop/mobile viewports byte-for-byte. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort. + +| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings | +|----|---------|--------|---------------|----------------------|-------------------| +| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request Sonnet/max | +| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high | +| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high | +| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh | +| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh | +| C06 | `c06-claude-gemini-hybrid` | claude | preset / `gemini-hybrid` | `gemini-hybrid` / high | Gemini/high, Gemini/high, ornith-fast, Gemini/high, Gemini/high | +| C07 | `c07-agy-gemini-hybrid` | agy | preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 | +| C08 | `c08-claude-gpt-hybrid` | claude | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | GPT/high, GPT/high, ornith-fast, GPT/high, GPT/high | +| C09 | `c09-codex-gpt-hybrid` | codex | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 | + +Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order: + +```text +c02-claude-gemini-direct +c05-codex-gpt-direct +c03-agy-gemini-direct +c06-claude-gemini-hybrid +c08-claude-gpt-hybrid +c09-codex-gpt-hybrid +c01-claude-sonnet-direct +c07-agy-gemini-hybrid +c04-claude-gpt-direct +``` + +**Modified Files and Checklist** + +- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest. +- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs. + +**Test Strategy** + +Write the named test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters. + +**Verification** + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +``` + +Expected: the manifest validates and the exact lock test passes. + +### [TEST-2] Produce one redacted nine-cell readiness record + +**Problem** + +Planning-time external preflight is blocked: benchmark environment references are missing, the dev runtime identity is unknown, and agy help lacks three adapter-required tokens. Static fixtures cannot claim live readiness. + +**Solution** + +First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them. + +If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G07.md` and stop. Do not mutate external state. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G07.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence. + +**Test Strategy** + +No additional unit test; this is required external execution evidence. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly. + +**Verification** + +```bash +set -euo pipefail +python3 - <<'PY' +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"): + active = root / group / subtask / "complete.log" + archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) + candidates = [path for path in (active, *archived) if path.is_file()] + assert len(candidates) == 1, (subtask, candidates) + print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" +test -z "$(git -C ../iop-s2 status --short)" +test -x ../iop-s2/build/bin/iop-edge +test -x ../iop-s2/build/dev/iop-node +test -f ../iop-s2/configs/edge.yaml +claude --version +agy --version +codex --version +agy_help="$(agy --help 2>&1)"; for token in --print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json; do grep -F -- "$token" <<<"$agy_help" >/dev/null; done +python3 - <<'PY' +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)" +printf '%s\n' "$preflight_output" +case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +python3 - "$run_id" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert record["status"] == "ready" +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine ready results are manifest-bound and runtime values are absent") +PY +``` + +Expected: every setup command exits 0; preflight prints one ready summary with nine results; durable inspection prints its safe success line. At planning time this block is expected to stop at agy/environment setup until the recorded resume condition is satisfied. + +### [TEST-3] Preserve the full benchmark baseline + +**Problem** + +The final file consumes the new order, rubric, and all-cell contracts. It must not regress legacy fixtures or other benchmark behavior. + +**Solution** + +Validate all four manifests, run the focused static test and complete rubric/scoring/manifest/attempt/connectivity suites, and check the diff. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G07.md`: record final command output after live readiness succeeds; if TEST-2 is blocked, leave this item unchecked and record the resume condition. + +**Test Strategy** + +No additional files beyond TEST-1. Existing suites are the regression oracle; external preflight output is not cached. + +**Verification** + +```bash +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: four manifests validate, all suites report `OK`, and whitespace check is silent. + +## Modified Files Summary + +| File | Item | +|------|------| +| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 | +| `scripts/agent_benchmark/manifest_test.py` | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G07.md` | TEST-2, TEST-3 | + +## Final Verification + +Run from `/config/workspace/iop-s0` in dependency order. Fresh output is required; cached output is not acceptable. Run the full TEST-2 command block exactly once after its setup checks pass, then run: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness +for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done +python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test +git diff --check +``` + +Expected: static validation/test pass, TEST-2 has one nine-ready durable preflight record with no runtime values, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No scored benchmark command is run. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/code_review_cloud_G09_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/code_review_cloud_G09_0.log new file mode 100644 index 00000000..bd0ac46d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/code_review_cloud_G09_0.log @@ -0,0 +1,288 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight, plan=0, tag=TEST + +## Archive Evidence Snapshot + +- Refinement source pair: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log` and matching review log. +- The source pair contains no implementation evidence or official verdict. This child preserves only its TEST-2 external readiness scope and depends on the reviewed static manifest child. +- Source-plan read-only evidence found the current testbed blocked by host-incompatible/stale artifacts, missing benchmark environment references, and an incompatible agy version/help surface. Those facts are planning evidence, not a substitute for execution-time checks. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare the execution evidence against the fixed gate. Rerun only when the plan's setup prerequisites are present and a fresh readiness observation is required; do not run scored benchmark commands. If the gate is blocked, verify that the exact safe blocker and resume condition are recorded without leaking runtime values. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-2 Produce one redacted nine-cell readiness record | [x] | + +## Implementation Checklist + +- [x] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`. +- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` and update this checklist at the final archive path. +- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +- The fixed gate's unchanged setup commands were executed in dependency, testbed, and artifact segments so execution could stop at the first exact blocker. The artifact segment exited 69 at the current-HEAD freshness check, so the later caller/version/help, environment-reference, public preflight, and durable-record commands were not executed. No acceptance condition or command predicate was changed. + +## Key Design Decisions + +- Preserved `../iop-s2` as a read-only independent testbed and captured its exact clean `dev` HEAD without changing artifacts, configuration, credentials, or route registration. +- Honored the fail-closed boundary: no caller/model/preset/effort substitution was attempted, and the public benchmark preflight was not invoked after the setup blocker. +- Made no production, manifest, fixture, benchmark runtime, adapter, skill, test, scoring, reporting, or external testbed change; this review evidence file is the only implementation write. + +## Reviewer Checkpoints + +- Confirm predecessor 04 has exactly one active or archived `complete.log` before accepting any readiness evidence. +- Confirm the external gate captures the clean independent `../iop-s2` dev HEAD and accepts only the production adapter's exact known agy version/help contract. +- Confirm both iop-s2 artifacts are current-HEAD AArch64 ELF executables before accepting endpoint readiness; `test -x` alone is insufficient. +- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value. +- Confirm every locked manifest cell is represented exactly once and all nine are ready. +- Confirm no tracked production/config/test file and no scored run/score/report surface was modified or executed in this child. + +## Verification Results + +Record actual safe stdout/stderr below. If a command changes, document the replacement and reason in `Deviations from Plan`. The full external block is fixed and must be recorded with actual safe output or the exact blocker and resume condition. + +### TEST-2 External Preflight + +```bash +set -euo pipefail +blocked() { + printf 'blocked: %s\n' "$1" >&2 + exit 69 +} +python3 - <<'PY' || blocked "predecessor 04 must have exactly one active or archived complete.log" +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +subtask = "04+02,03_locked_benchmark_manifest" +active = root / group / subtask / "complete.log" +archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, (subtask, candidates) +print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" || blocked "../iop-s2 must be on branch dev" +test -z "$(git -C ../iop-s2 status --short)" || blocked "../iop-s2 must be clean" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" || blocked "../iop-s2 HEAD must resolve" +test -n "$testbed_head" || blocked "../iop-s2 HEAD must be non-empty" +printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head" +command -v readelf >/dev/null || blocked "readelf must be installed" +test -x ../iop-s2/build/bin/iop-edge || blocked "current iop-edge artifact must be executable" +test -x ../iop-s2/build/dev/iop-node || blocked "current iop-node artifact must be executable" +test -f ../iop-s2/configs/edge.yaml || blocked "../iop-s2/configs/edge.yaml must exist" +commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" || blocked "testbed commit time must resolve" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test "$(stat -c %Y "$binary")" -ge "$commit_epoch" || blocked "$binary must be built from the current testbed HEAD" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute on this host" +test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute on this host" +printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n' +claude --version || blocked "claude caller must be installed" +agy --version || blocked "agy caller must be installed" +codex --version || blocked "codex caller must be installed" +python3 - <<'PY' || blocked "agy must match the adapter-known version and transport help contract" +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +version = (version_run.stdout + version_run.stderr).strip() +help_text = help_run.stdout + help_run.stderr +capability = inspect_agy_iop_capability(version, help_text) +assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION) +assert capability.iop_transport_supported, capability +print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented") +PY +python3 - <<'PY' || blocked "benchmark endpoint, secret, and config-observation references must be present" +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +set +e +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json 2>&1)" +preflight_status=$? +set -e +printf '%s\n' "$preflight_output" +case "$preflight_status" in + 0) case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac ;; + 69) case "$preflight_output" in *"error: preflight blocked "*) ;; *) exit 1 ;; esac ;; + *) exit "$preflight_status" ;; +esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +test -n "$run_id" +python3 - "$run_id" "$preflight_status" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 +if sys.argv[2] == "0": + assert record["status"] == "ready" and all(item["status"] == "ready" for item in record["results"]) +else: + assert record["status"] in {"registration_required", "implementation_gap"} + assert any(item["status"] != "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine preflight results are manifest-bound and runtime values are absent") +for item in record["results"]: + issues = ",".join(f'{issue["code"]}:{issue["resume_code"]}' for issue in item["issues"]) or "none" + print(f'cell={item["cell"]["id"]} status={item["status"]} issues={issues}') +PY +if [ "$preflight_status" -eq 69 ]; then + exit 69 +fi +``` + +_Actual safe output or exact blocker and resume condition:_ + +```text +dependency check (exit 0): +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log + +testbed provenance check (exit 0): +ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true + +artifact check (exit 69): +blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD +``` + +Exact blocker: `../iop-s2/build/bin/iop-edge` did not satisfy the required modification-time freshness bound for clean testbed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`. The fixed gate stopped before accepting its Linux AArch64 ELF identity or executability as current-HEAD evidence. + +Resume condition: from the exact clean `../iop-s2` `dev` HEAD, rebuild `build/bin/iop-edge` as a Linux AArch64 ELF artifact and ensure both required Edge/Node artifacts satisfy the fixed current-HEAD freshness, ELF, and host-execution checks. Then rerun the fixed gate from the dependency check. The later agy version/help, benchmark environment-reference, and public nine-cell preflight checks remain unexecuted and must also pass in that resumed gate; do not infer their current state from planning evidence. + +Not executed after the first blocker: + +```text +caller version/help checks: not run +benchmark environment-reference check: not run +public benchmark preflight: not invoked (no run id created) +durable nine-result and secret-absence inspection: not run +scored run/resume/score/report commands: not run +``` + +### Reviewer Fresh Verification (2026-08-12) + +The reviewer reran the read-only dependency, testbed provenance, and artifact setup sequence. It stopped at the same first blocker without invoking the public preflight or any caller/provider: + +```text +exit 69 +ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log +ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true +blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD +``` + +Focused repository checks: + +```text +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +ok: manifest is valid + +manifest inspection +ok: manifest_order=c02-claude-gemini-direct,c05-codex-gpt-direct,c03-agy-gemini-direct,c06-claude-gemini-hybrid,c08-claude-gpt-hybrid,c09-codex-gpt-hybrid,c01-claude-sonnet-direct,c07-agy-gemini-hybrid,c04-claude-gpt-direct +ok: cell_prefixes=c01,c02,c03,c04,c05,c06,c07,c08,c09 unique=9 agy_known_version=1.1.11 + +git diff --check +ok: no output +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +- Overall Verdict: PASS +- Dimension Assessment: + - Correctness: Pass — the fixed setup gate stopped at the first exact current-HEAD artifact blocker and did not bypass or substitute the blocked route. + - Completeness: Pass — TEST-2 explicitly accepts either nine ready results or an exact safe blocker and resume condition; the latter is complete and independently reproduced. + - Test Coverage: Pass — the reviewer reproduced the blocker, validated the locked manifest, confirmed one-to-one C01-C09 coverage, and ran `git diff --check`. + - API Contract: Pass — no caller/provider request was made after the prerequisite failure, and no route/model/effort/auth contract was substituted. + - Code Quality: Pass — this child changes only its review evidence and contains no debug artifact, raw runtime value, or unrelated production edit. + - Implementation Deviation: Pass — segmented execution preserves the plan's fixed order and fail-closed semantics; later checks were correctly not executed. + - Verification Trust: Pass — fresh reviewer output matches the recorded predecessor path, clean testbed HEAD, and exact artifact blocker. + - Spec Conformance: Pass — SDD S02 permits an exact blocker, and the evidence records the safe blocker and deterministic resume condition without asserting Milestone Task completion. +- Findings: None +- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false` +- Next Step: PASS — write `complete.log`, archive this pair and task directory, and report `milestone-task=route-readiness` only as runtime aggregation metadata. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log new file mode 100644 index 00000000..62dcabb4 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log @@ -0,0 +1,39 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight + +## 완료 일시 + +2026-08-12 + +## 요약 + +한 차례 구현과 공식 코드 리뷰에서 고정 readiness gate의 첫 외부 artifact blocker와 재개 조건을 secret-safe하게 확정했으며 최종 판정은 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | PASS | clean `../iop-s2` dev HEAD의 stale `iop-edge` artifact에서 fail-closed했고, exact blocker와 전체 gate 재개 조건을 독립 재현했다. | + +## 구현/정리 내용 + +- 선행 manifest child의 완료 로그가 정확히 한 건임을 확인하고 독립 testbed의 branch, HEAD, clean 상태를 기록했다. +- current-HEAD artifact freshness gate의 첫 실패를 안전하게 기록하고 caller/version/help, 환경 reference, public preflight와 scored 명령을 실행하지 않았다. +- `route-readiness`는 Milestone 완료 선언이 아닌 기여 범위 metadata로만 보존했다. + +## 최종 검증 + +- `TEST-2 dependency/testbed/artifact setup sequence` - BLOCKED(expected accepted outcome); exit 69, `../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD`, recorded HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 구현 evidence가 일치했다. +- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`. +- `python3 - <<'PY' ... load_manifest/AGY_KNOWN_VERSION inspection ... PY` - PASS; C01-C09 prefix가 각각 정확히 한 번 존재하고 known agy version은 `1.1.11`이다. +- `git diff --check` - PASS; 출력 없음. +- 공개 benchmark preflight, repo 내부 edge-node 진단, 보조 E2E smoke와 provider full-cycle은 artifact prerequisite에서 fail-closed했으므로 실행하지 않았다. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- 정확한 clean `../iop-s2` dev HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller/version/help, benchmark environment reference와 public nine-cell preflight를 포함한 고정 gate 전체를 다시 실행해야 한다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/plan_cloud_G09_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/plan_cloud_G09_0.log new file mode 100644 index 00000000..d975d4f4 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/plan_cloud_G09_0.log @@ -0,0 +1,239 @@ + + +# Plan - Redacted Nine-Cell Readiness Preflight + +## For the Implementing Agent + +Do not start until the dependency command resolves exactly one active or archived `04+02,03_locked_benchmark_manifest/complete.log`. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run the fixed external gate exactly once after its setup checks pass, record safe output or the exact blocker and resume condition, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The reviewed static child creates the immutable C01-C09 input. This packet performs the remaining execution-day readiness closure through the public benchmark preflight without substituting missing callers, credentials, models, presets, runtime artifacts, or effort settings. + +## Archive Evidence Snapshot + +- Refinement source pair: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log` and matching review log. +- The source pair contains no implementation evidence or official verdict. This child preserves only its TEST-2 external readiness scope and depends on the reviewed static manifest child. +- Source-plan read-only evidence found the current testbed blocked by host-incompatible/stale artifacts, missing benchmark environment references, and an incompatible agy version/help surface. Those facts are planning evidence, not a substitute for execution-time checks. + +## Analysis + +### Files Read + +- `scripts/agent_comparison_benchmark.py` +- `scripts/agent_benchmark/manifest.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/connectivity.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/claude_iop.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/codex_iop.py` +- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-contract/outer/anthropic-compatible-api.md` +- `agent-contract/outer/openai-compatible-api.md` +- `agent-contract/inner/edge-config-runtime-refresh.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` + +### SDD Criteria + +- SDD status is `[승인됨]`, implementation lock is released, and no unresolved user review exists. +- First-line scope is `milestone-task=route-readiness`. +- S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker. +- The required evidence is one redacted C01-C09 preflight matrix with auth/route/effort/terminal status, or the exact safe blocker and resume condition. + +### Verification Context + +- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`. +- Testbed: `/config/workspace/iop-s2`, branch `dev`; execution must capture its exact current clean HEAD as independent read-only provenance. +- Source-plan evidence found `../iop-s2/build/bin/iop-edge` to be Mach-O and both runtime artifacts older than the observed testbed HEAD. It also found agy `1.1.12` while the reviewed adapter accepted `1.1.11`, and all benchmark environment references were missing. +- Exact resume condition remains: rebuild both iop-s2 artifacts from its exact clean `dev` HEAD for Linux AArch64, provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run the rebuilt endpoints, and install the adapter-known agy version whose help satisfies every documented transport token. +- No raw endpoint, secret, or config-observation value may be written to the repository or review file. + +### Test Coverage Gaps + +- Static fixtures cannot prove execution-day runtime identity, artifact compatibility, credentials, route registration, caller transport, or finish/idle readiness. +- No durable nine-cell all-ready record exists for the locked manifest. + +### Symbol References + +- No production symbol or tracked config is changed. +- The public `preflight` command and durable record are the only acceptance boundary; never invoke a caller/provider directly. + +### Split Judgment + +- This is the allowed closure-verification child from the source pair. It has no production write set and consumes the exact reviewed manifest from child 04. +- Static manifest construction and all repository regressions remain in predecessor 04; this child does not duplicate them. +- No child created in this refinement pass is split again. + +### Scope Rationale + +- Do not edit `../iop-s2`, install tools, change credentials/registrations, or substitute models, presets, callers, or effort. +- Do not modify the locked manifest, fixture bytes, benchmark runtime, adapters, project skill, tests, scoring, or reporting. +- Do not run scored `run`, `resume`, `score`, or `report`; this child ends at readiness preflight evidence. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`. +- Build closures are all closed. Scores `2+1+2+2+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`. +- Review closures are all closed. Scores `2+1+2+2+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`. +- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`; grade remains the route basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. + +## Dependencies and Execution Order + +1. Resolve exactly one active-sibling or matching archived `complete.log` for `04+02,03_locked_benchmark_manifest`. +2. Run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks in the fixed order. +3. Only after every setup check passes, invoke the public benchmark `preflight` once and inspect its durable nine-result record. +4. Stop on the first exact blocker or after recording the all-ready redacted evidence. Do not proceed to scored execution. + +## Implementation Checklist + +- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-2] Produce one redacted nine-cell readiness record + +**Problem** + +Static validation cannot establish live readiness. The public CLI returns closed preflight blockers with exit 69 (`scripts/agent_comparison_benchmark.py:208`), while canonical per-cell issue/resume evidence and preflight collection are owned by `scripts/agent_benchmark/connectivity.py:406` and `scripts/agent_benchmark/attempts.py:386`. Planning-time external preflight was blocked by stale/host-incompatible artifacts, missing environment references, and an incompatible agy capability surface. + +**Solution** + +First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them. + +If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G09.md` and stop. The shell must capture preflight stderr and status without allowing `set -e` to exit before the blocker can be recorded. Do not mutate external state. + +**Modified Files and Checklist** + +- [ ] `CODE_REVIEW-cloud-G09.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence. + +**Test Strategy** + +No unit test is added. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly. + +**Verification** + +```bash +set -euo pipefail +blocked() { + printf 'blocked: %s\n' "$1" >&2 + exit 69 +} +python3 - <<'PY' || blocked "predecessor 04 must have exactly one active or archived complete.log" +from pathlib import Path +root = Path("agent-task") +group = "m-iop-one-shot-agent-model-comparison" +subtask = "04+02,03_locked_benchmark_manifest" +active = root / group / subtask / "complete.log" +archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log")) +candidates = [path for path in (active, *archived) if path.is_file()] +assert len(candidates) == 1, (subtask, candidates) +print(f"ok: predecessor complete {candidates[0]}") +PY +test "$(git -C ../iop-s2 branch --show-current)" = "dev" || blocked "../iop-s2 must be on branch dev" +test -z "$(git -C ../iop-s2 status --short)" || blocked "../iop-s2 must be clean" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" || blocked "../iop-s2 HEAD must resolve" +test -n "$testbed_head" || blocked "../iop-s2 HEAD must be non-empty" +printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head" +command -v readelf >/dev/null || blocked "readelf must be installed" +test -x ../iop-s2/build/bin/iop-edge || blocked "current iop-edge artifact must be executable" +test -x ../iop-s2/build/dev/iop-node || blocked "current iop-node artifact must be executable" +test -f ../iop-s2/configs/edge.yaml || blocked "../iop-s2/configs/edge.yaml must exist" +commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" || blocked "testbed commit time must resolve" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test "$(stat -c %Y "$binary")" -ge "$commit_epoch" || blocked "$binary must be built from the current testbed HEAD" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute on this host" +test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute on this host" +printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n' +claude --version || blocked "claude caller must be installed" +agy --version || blocked "agy caller must be installed" +codex --version || blocked "codex caller must be installed" +python3 - <<'PY' || blocked "agy must match the adapter-known version and transport help contract" +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +version = (version_run.stdout + version_run.stderr).strip() +help_text = help_run.stdout + help_run.stderr +capability = inspect_agy_iop_capability(version, help_text) +assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION) +assert capability.iop_transport_supported, capability +print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented") +PY +python3 - <<'PY' || blocked "benchmark endpoint, secret, and config-observation references must be present" +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references present") +PY +set +e +preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json 2>&1)" +preflight_status=$? +set -e +printf '%s\n' "$preflight_output" +case "$preflight_status" in + 0) case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac ;; + 69) case "$preflight_output" in *"error: preflight blocked "*) ;; *) exit 1 ;; esac ;; + *) exit "$preflight_status" ;; +esac +run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}" +test -n "$run_id" +python3 - "$run_id" "$preflight_status" <<'PY' +import json, os, sys +from pathlib import Path +from scripts.agent_benchmark.manifest import load_manifest +manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json") +manifest = load_manifest(manifest_path, repo_root=Path.cwd()) +run_root = Path(manifest.output_root) / sys.argv[1] +record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii")) +assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix] +assert len(record["results"]) == 9 +if sys.argv[2] == "0": + assert record["status"] == "ready" and all(item["status"] == "ready" for item in record["results"]) +else: + assert record["status"] in {"registration_required", "implementation_gap"} + assert any(item["status"] != "ready" for item in record["results"]) +sensitive = [] +for caller in ("CLAUDE", "AGY", "CODEX"): + sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode()) + sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode()) +config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"] +sensitive.append(os.environ[config_ref].encode()) +durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file()) +assert all(value and value not in durable for value in sensitive) +print("ok: nine preflight results are manifest-bound and runtime values are absent") +for item in record["results"]: + issues = ",".join(f'{issue["code"]}:{issue["resume_code"]}' for issue in item["issues"]) or "none" + print(f'cell={item["cell"]["id"]} status={item["status"]} issues={issues}') +PY +if [ "$preflight_status" -eq 69 ]; then + exit 69 +fi +``` + +Expected: every setup command exits 0. A ready preflight prints one ready summary, verifies nine ready results, and exits 0. A closed preflight blocker is captured despite `set -e`, verifies and prints only per-cell status plus canonical issue/resume codes from the durable record, and exits 69 so implementation records that exact safe blocker and the resume condition before stopping. + +## Modified Files Summary + +| File | Item | +|------|------| +| `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md` | TEST-2 | + +## Final Verification + +Run from `/config/workspace/iop-s0` after predecessor 04 completes. Run the TEST-2 verification block exactly once after its setup checks pass. No other validation or scored benchmark command belongs to this child. + +Expected: dependency and setup checks pass, one public preflight reports nine ready results, and durable evidence contains none of the exact runtime values. Otherwise the review stub records the exact safe blocker and resume condition. + +After completing the readiness check, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G06_5.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G06_5.log new file mode 100644 index 00000000..3625a9d9 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G06_5.log @@ -0,0 +1,200 @@ + + +# Code Review Reference - REVIEW_TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, run public benchmark `run|resume`, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=5, tag=REVIEW_TEST + +## Archive Evidence Snapshot + +- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다. +- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다. +- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다. +- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다. +- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_5.log` and `PLAN-local-G06.md` → `plan_local_G06_5.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|---|---| +| FIX-1: Current Codex usage schema | [x] | +| FIX-2: Config-owned effective binding consumption | [x] | + +## Implementation Checklist + +- [x] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals. +- [x] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G06_5.log`. +- [x] Archive active `PLAN-*-G??.md` to `plan_local_G06_5.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent or verify it was kept due to remaining siblings/files. +- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +없음. public benchmark `run|resume|score`, caller/provider 직접 호출, run tree 수정 없이 plan의 다섯 source/test/fixture 파일만 수정했다. + +## Key Design Decisions + +- Codex의 `cache_write_input_tokens`는 caller가 명시적으로 보고한 값만 canonical `cache_write_tokens`로 보존하고 `total_tokens`는 합성하지 않는다. +- independently validated config observation을 execution/scoring의 canonical binding으로 사용한다. optional caller binding은 absent일 수 있지만, 보고된 non-`None` 값은 admitted tuple과 정확히 일치해야 한다. +- scoring result에는 admitted tuple을 기록해 downstream이 caller observation 유무와 무관하게 검증된 effective binding을 소비하도록 했다. + +## Reviewer Checkpoints + +- Verify `_CODEX_USAGE_FIELDS` accepts only the current explicit keys and maps `cache_write_input_tokens` to `cache_write_tokens` without deriving `total_tokens`. +- Verify the tracked Codex fixture no longer relies on a fabricated `iop_effective_binding` event for production execution correctness. +- Verify execution and scoring both accept `None` caller observation only after ready config admission and still reject any contradictory non-`None` observation. +- Verify no attempt recovery, run tree, provider, remote runtime, or unrelated caller behavior changed. +- Verify no public `run|resume|score` or direct caller/provider command was executed in this follow-up. + +## Verification Results + +### Focused parser and integration coverage + +Command: + +```bash +python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test +``` + +Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection. + +Actual stdout/stderr: + +```text +......................................... +---------------------------------------------------------------------- +Ran 41 tests in 14.500s + +OK +``` + +### Full benchmark suite + +Command: + +```bash +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +``` + +Expected: all benchmark tests pass with no skipped required test or network/provider call. + +Actual stdout/stderr: + +```text +............................................................................................................................................................................................................................................................................................................................................................................. +---------------------------------------------------------------------- +Ran 429 tests in 115.180s + +OK +``` + +### Manifest and repository diff + +Command: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Expected: `ok: manifest is valid`; `git diff --check` has no output. + +Actual stdout/stderr: + +```text +ok: manifest is valid +``` + +`git diff --check` stdout/stderr: `(none)` + +Reviewer fresh rerun: + +```text +focused: Ran 41 tests in 14.154s — OK +full: Ran 429 tests in 116.096s — OK +manifest: ok: manifest is valid +git diff --check: (none) +``` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---|---|---| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns applicable commands and records fresh output | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +- Overall Verdict: FAIL +- Dimension Assessment: + - Correctness: Pass — R1 current usage parsing and R2 config-owned execution/scoring binding now match production behavior while preserving mismatch rejection. + - Completeness: Fail — no replacement C01-C09 scored execution exists after the retained plan=4 failure. + - Test coverage: Pass — production-shaped usage, absent observation success, non-None mismatch failure, execution, and scoring paths have meaningful regression coverage. + - API contract: Pass — optional caller observation and canonical config admission now have one consistent ownership contract. + - Code quality: Pass — changes are bounded to the selected parser/consumer/test/fixture paths with no debug or dead code. + - Implementation deviation: Pass — implementation followed R1/R2 exactly and did not invoke a caller/provider or mutate retained run state. + - Verification trust: Pass — reviewer fresh reruns match implementation evidence: focused 41/OK, full 429/OK, manifest valid, diff check clean. + - Spec conformance: Fail — S04-S08 still require nine retained C01-C09 terminal attempts from an authorized scored execution. +- Findings: + - Required R3 — Authorized replacement scored execution is still required. + - Evidence: source-fix reviewer verification passes, but read-only public status for `run-20260812T040619Z-00a5e6664764` remains `failed=1, running=1`; `run_id.log` is absent and no replacement execution was invoked by plan=5. The benchmark skill requires stopping after a retained execution failure until the user explicitly authorizes another stateful execution. + - Root Cause: the single scored run authorized for plan=4 was consumed before R1/R2 were known; append-only policy forbids rewriting that attempt, and plan=5 intentionally repaired only deterministic source/test boundaries. + - Selected Fix: after explicit user authorization, create a new execution plan that rebinds the protected local token/CA references, requires fresh public preflight `ready=9`, invokes `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once, stores only its CLI-emitted run id, and accepts only nine retained terminal attempts with zero `running|interrupted`. Preserve every older run and do not call direct callers/providers or edit state. +- Routing Signals: review_rework_count=2 evidence_integrity_failure=false +- Next Step: USER_REVIEW external-execution — obtain explicit authorization for exactly one new public C01-C09 scored run after the retained failure; then re-enter the plan skill for this exact task path. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_0.log new file mode 100644 index 00000000..4ee87808 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_0.log @@ -0,0 +1,208 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=0, tag=TEST + +## Archive Evidence Snapshot + +- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`다. 미해결 Required/Suggested는 기재되지 않았고 잔여 Nit은 없다. +- 선행 작업은 source를 변경하지 않았고, clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 stale `../iop-s2/build/bin/iop-edge` 사이의 freshness blocker를 exit 69로 재현했다. Manifest validate는 `ok: manifest is valid`였다. +- Roadmap carryover는 위 exact HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller version/help, secret-safe environment reference, public nine-cell preflight를 포함한 gate를 다시 통과하는 것이다. 구현자는 archive 전체를 탐색하지 말고 추가 세부가 꼭 필요할 때만 위 `complete.log` 한 건을 읽는다. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1 | [ ] | + +## Implementation Checklist + +- [ ] TEST-1: Confirm the fixed external preconditions without exposing secrets, execute the immutable C01-C09 `run` exactly once, persist its canonical run id, and verify all nine attempts reach success with unresolved 0; if the gate blocks or an attempt fails, do not retry and record the exact evidence and resume condition. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_0.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_0.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm `run_id.log` contains the id emitted by the one and only `run` invocation and was not fabricated or replaced. +- Confirm verbatim execution output reports `completed=9`, `unresolved=0`, `success=9`, and zero failed/timed_out/cancelled/interrupted/running attempts; independently rerun only `status`, never `run` or `resume`. +- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid`, with timing/usage/workspace and hybrid stage evidence validated by the run. +- Confirm no secret, raw private endpoint/config payload, caller substitution, retry, score, or report was introduced and `../iop-s2` stayed clean. + +## Verification Results + +> The implementing agent must replace each placeholder below with the exact command result and verbatim stdout/stderr. If blocked, retain unchecked completion boxes and record the exit code, exact public error, and resume condition. The review agent reruns only commands marked repeatable; the one-time provider execution must be evaluated from its preserved evidence and public `status`. + +### Static manifest and harness tests — repeatable + +Command: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test +``` + +Expected: manifest valid; 91 tests pass. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### External execution preconditions — repeatable and secret-safe + +Command: + +```bash +set -eu +test "$(uname -s)" = Linux +test "$(uname -m)" = aarch64 +test "$(git -C ../iop-s2 branch --show-current)" = dev +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 +test -z "$(git -C ../iop-s2 status --porcelain=v1)" +for tool in python3 claude agy codex git; do command -v "$tool"; done +python3 - <<'PY' +import os + +ok = True +for caller in ("CLAUDE", "AGY", "CODEX"): + base = bool(os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")) + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + secret = bool(ref and os.environ.get(ref)) + print(f"{caller}: base_reference={base} secret_reference={bool(ref)} referenced_secret={secret}") + ok = ok and base and bool(ref) and secret +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +config = bool(config_ref and os.environ.get(config_ref)) +print(f"CONFIG: reference={bool(config_ref)} referenced_value={config}") +ok = ok and bool(config_ref) and config +raise SystemExit(0 if ok else 69) +PY +``` + +Expected: exact clean testbed branch/HEAD; five binaries; every printed boolean is `True`. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### C01-C09 run — implementation-only, do not repeat in review + +Command: + +```bash +set -eu +bench_stdout="$(mktemp)" +bench_stderr="$(mktemp)" +trap 'rm -f "$bench_stdout" "$bench_stderr"' EXIT +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$bench_stdout" 2>"$bench_stderr" +bench_exit=$? +set -e +printf 'exit=%s\n' "$bench_exit" +printf '%s\n' '--- stdout ---' +cat "$bench_stdout" +printf '%s\n' '--- stderr ---' +cat "$bench_stderr" +bench_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$bench_stdout" "$bench_stderr" | sed -n '1p')" +test -n "$bench_run_id" +printf '%s\n' "$bench_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" +test "$bench_exit" -eq 0 +``` + +Expected: exit 0; completed 9, unresolved 0, success 9, all other counts 0; one canonical `run_id.log`. + +Actual stdout/stderr: + +_Fill with verbatim output, including the exit line and both labeled streams._ + +### Recorded run status and worktree — repeatable, no provider invocation + +Command: + +```bash +set -eu +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" +git diff --check +``` + +Expected: valid run id; status success 9 and every other count 0; `git diff --check` has no output. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_1.log new file mode 100644 index 00000000..36d55740 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_1.log @@ -0,0 +1,273 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=1, tag=TEST + +## Archive Evidence Snapshot + +- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 readiness 확인 절차가 exact blocker를 보존했다는 뜻이며 `route-readiness` 완료 선언이 아니다. +- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`에서 Edge/Node artifact freshness, Linux AArch64 identity, caller version/help, environment reference를 모두 통과한 뒤에만 live benchmark를 시작하도록 요구한다. +- 최초 active pair는 공식 verdict 없이 `plan_cloud_G09_0.log`와 `code_review_cloud_G10_0.log`로 보존했다. 그 pair의 success-only acceptance와 불완전한 artifact/caller preflight는 이 generation이 대체하며 구현자는 이전 pair를 다시 읽지 않는다. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker. +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`. +3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill. +4. If PASS and task group is `m-`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1 | [ ] | + +## Implementation Checklist + +- [ ] TEST-1: Prove every fixed external precondition, invoke the immutable C01-C09 `run` exactly once, preserve its canonical run id and verbatim CLI result, and accept only either nine success results or nine retained non-interrupted terminal results; never retry, resume, substitute, or treat preflight/partial execution as completion. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_1.log`. +- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_1.log`. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores local `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files. +- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path. +- [ ] If PASS and task group is `m-`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files. +- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm every fixed gate passed before the one and only `run` invocation; in particular both artifacts are current-HEAD Linux AArch64 ELF, agy matches the adapter-known version/help contract, and environment references were dereferenced without disclosure. +- Confirm `run_id.log` names the one exact newly created run and matches any CLI-emitted id; it must not point to the preparation-only preflight run. +- Confirm public status validates exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted`. Retained failures are valid execution evidence; zero or partial attempts are not. +- Confirm the CLI exit/output and state classification agree, no retry/resume/score/report was invoked, no secret or raw private endpoint/config payload was recorded, and `../iop-s2` stayed clean. +- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid` under the immutable manifest. + +## Verification Results + +> The implementing agent must replace each placeholder below with the exact command result and verbatim stdout/stderr. If a fixed gate or partial execution blocks, retain unchecked completion boxes and record the exact public error and resume condition. The review agent reruns only commands marked repeatable; the one-time provider execution must be evaluated from preserved evidence and public `status`. + +### Static manifest and harness tests — repeatable + +Command: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test +``` + +Expected: manifest valid; 91 tests pass. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### Fixed external execution preconditions — repeatable and secret-safe + +Command: + +```bash +set -euo pipefail +blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; } +test "$(uname -s)" = Linux || blocked "benchmark host must be Linux" +test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64" +test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev" +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed" +test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean" +command -v readelf >/dev/null || blocked "readelf must be installed" +command -v go >/dev/null || blocked "go must be installed" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test -x "$binary" || blocked "$binary must be executable" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD" +import subprocess, sys +expected = sys.argv[1] +for binary in sys.argv[2:]: + output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout + build = {} + for line in output.splitlines(): + fields = line.strip().split("\t", 1) + if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]: + key, value = fields[1].split("=", 1) + build[key] = value + assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision")) + assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified")) + assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH")) +print("ok: Edge/Node build identities match the clean testbed HEAD") +PY +../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute" +test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute" +for tool in python3 claude agy codex git; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done +python3 - <<'PY' || blocked "agy must match the adapter-known version and transport contract" +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +capability = inspect_agy_iop_capability((version_run.stdout + version_run.stderr).strip(), help_run.stdout + help_run.stderr) +assert capability.version == AGY_KNOWN_VERSION +assert capability.iop_transport_supported +print("ok: agy IOP transport capability") +PY +python3 - <<'PY' || blocked "benchmark environment references must be present" +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references") +PY +``` + +Expected: all checks exit 0. Any blocker stops before the scored command. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### C01-C09 run — implementation-only, do not repeat in review + +Command: + +```bash +set -euo pipefail +run_root=agent-test/runs/bench-02 +before="$(mktemp)"; after="$(mktemp)"; new_runs="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)" +trap 'rm -f "$before" "$after" "$new_runs" "$out" "$err"' EXIT +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before" +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err" +bench_exit=$? +set -e +printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out" +printf '%s\n' 'stderr:'; cat "$err" +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' | sort >"$after" +comm -13 "$before" "$after" >"$new_runs" +test "$(wc -l <"$new_runs")" -eq 1 +discovered_run_id="$(cat "$new_runs")" +emitted_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$out" "$err" | sed -n '1p')" +if test -n "$emitted_run_id"; then test "$emitted_run_id" = "$discovered_run_id"; fi +printf '%s\n' "$discovered_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$discovered_run_id")" +printf 'status_stdout:\n%s\n' "$status_output" +BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"} +assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values()) +bench_exit = int(os.environ["BENCH_EXIT"]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0: + assert bench_exit == (0 if states["success"] == 9 else 69) + print(f"classification=execution_complete states={states}") + raise SystemExit(0) +if sum(states.values()) == 0 and bench_exit == 69: + print(f"classification=blocked_preflight states={states}") +else: + print(f"classification=blocked_partial states={states}") +raise SystemExit(69) +PY +``` + +Expected: exactly one new run id and `classification=execution_complete`. The original CLI exit may be 69 only when all nine non-interrupted terminal attempts are retained and public status proves their states. + +Actual stdout/stderr: + +_Fill with verbatim output, including command, exit code, both streams, status and classification._ + +### Recorded run status and worktree — repeatable, no provider invocation + +Command: + +```bash +set -euo pipefail +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")" +printf '%s\n' "$status_output" +STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states +print(f"ok: nine retained terminal attempts states={states}") +PY +git diff --check +``` + +Expected: valid run id; nine retained accepted terminal attempts; zero interrupted/running; no diff whitespace errors. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_2.log new file mode 100644 index 00000000..34d3bab9 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_2.log @@ -0,0 +1,265 @@ + + +# Code Review Reference - C01-C09 원샷 비교 실행 + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=2, tag=TEST + +## Archive Evidence Snapshot + +- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` proves task-protocol dependency completion while preserving a fail-closed live-readiness blocker; it does not mark Milestone `route-readiness` complete. +- Generation 0/1 are preserved as `plan_cloud_G09_0.log`, `code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`, `code_review_cloud_G10_1.log`, all without an official verdict. Active plan=2 replaces their semantic defects; do not execute archived commands. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. Never repeat the one-time provider execution in Final Verification step 3. If implementation is present, repair missing or stale repeatable verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating a follow-up plan. + +Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-{review_lane}-{review_grade}.md` and `PLAN-{build_lane}-{build_grade}.md` to the next canonical log indices. +3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/{task_name}/`. If WARN/FAIL, write the next filesystem state required by the code-review skill. +4. If PASS, preserve the first-line `milestone-task` metadata and report it for runtime aggregation. Roadmap evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|--------| +| TEST-1: Immutable C01-C09 comparison run | [ ] | + +## Implementation Checklist + +- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [ ] Run applicable required repeatable verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan. +- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan. +- [ ] Archive active `CODE_REVIEW-*-G??.md` and `PLAN-*-G??.md` to the next canonical log indices. +- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` from the canonical template and leave no active `.md` files. +- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/{task_name}/` and update this checklist at the final archive path. +- [ ] If PASS, preserve and report the five exact `milestone-task` ids for runtime aggregation without modifying roadmap or directly calling `update-roadmap`. +- [ ] If PASS for split work, remove an empty active parent or verify it remains because siblings/files exist. +- [ ] If WARN/FAIL, write the next filesystem state matching the verdict and do not write `complete.log`. + +## Deviations from Plan + +_Record any deviations from the plan and the rationale here._ + +## Key Design Decisions + +_Record key design decisions here._ + +## Reviewer Checkpoints + +- Confirm active PLAN/review headers both use plan=2 and the exact five-task union: `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`. +- Confirm active index `06+05`, exact archived predecessor `05+04/.../complete.log`, and the preserved readiness blocker; do not infer that `route-readiness` is complete. +- Confirm S04-S08 map C01-C09 to those five tasks and source/spec/contract/SDD/Milestone remained outside the write set. +- Confirm the implementer did not directly invoke a caller/provider, inspect secret values, duplicate live-preflight policy, rebuild/mutate `../iop-s2`, or invoke `resume`, retry, score, evaluator, or report. +- Confirm step 2 proved fixed host/testbed/artifact identity and command presence before any public `run` call. +- Confirm step 3 was invoked at most once, the pointer came only from one unique CLI-emitted id, that id did not preexist, and its exact run directory exists. Ignore unrelated concurrent run directories. +- Confirm public status validates exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted`. Retained failure is valid execution evidence; zero/partial/invalid/unidentified state is not. +- Confirm CLI exit/output, state classification and checklist agree, no secret/raw private endpoint/config payload was recorded, and `../iop-s2` stayed clean. +- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid`. + +## Verification Results + +> The implementing agent must replace each placeholder with exact command results and verbatim stdout/stderr. If a fixed gate or execution state blocks, retain unchecked completion boxes and record the exact public error and resume condition. The review agent reruns only sections marked repeatable; the one-time provider execution is evaluated from preserved evidence and public `status`. + +### Static manifest and full benchmark suite — repeatable + +Command: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +``` + +Expected: manifest valid and all discovered benchmark tests pass. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### Fixed host/testbed/artifact gate — repeatable, no caller/provider invocation + +Command: + +```bash +set -euo pipefail +blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; } +test "$(uname -s)" = Linux || blocked "benchmark host must be Linux" +test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64" +test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev" +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed" +test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean" +for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test -x "$binary" || blocked "$binary must be executable" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD" +import subprocess, sys +expected = sys.argv[1] +for binary in sys.argv[2:]: + output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout + build = {} + for line in output.splitlines(): + fields = line.strip().split("\t", 1) + if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]: + key, value = fields[1].split("=", 1) + build[key] = value + assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision")) + assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified")) + assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH")) +print("ok: Edge/Node build identities match the clean testbed HEAD") +PY +``` + +Expected: all checks exit 0. Any blocker stops before the scored command. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +### C01-C09 run — implementation-only, do not repeat in review + +Command: + +```bash +set -euo pipefail +task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs +run_root=agent-test/runs/bench-02 +test ! -e "$task_dir/run_id.log" +before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)" +trap 'rm -f "$before" "$out" "$err"' EXIT +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before" +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err" +bench_exit=$? +set -e +printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out" +printf '%s\n' 'stderr:'; cat "$err" +emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)" +emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)" +if test "$emitted_count" -ne 1; then + printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count" + exit 69 +fi +bench_run_id="$(printf '%s\n' "$emitted_ids")" +if grep -Fxq "$bench_run_id" "$before"; then + printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id" + exit 69 +fi +test -d "$run_root/$bench_run_id" +printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log" +set +e +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)" +status_exit=$? +set -e +printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output" +if test "$status_exit" -ne 0; then + printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id" + exit 69 +fi +BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"} +assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values()) +bench_exit = int(os.environ["BENCH_EXIT"]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0: + assert bench_exit == (0 if states["success"] == 9 else 69) + print(f"classification=execution_complete states={states}") + raise SystemExit(0) +if sum(states.values()) == 0 and bench_exit == 69: + print(f"classification=blocked_preflight states={states}") +else: + print(f"classification=blocked_partial states={states}") +raise SystemExit(69) +PY +``` + +Expected: one CLI-emitted new run id and `classification=execution_complete`. Exit 69 is accepted only when all nine non-interrupted terminal attempts are retained. + +Actual stdout/stderr: + +_Fill with verbatim output, including command, exit code, both streams, status and classification._ + +### Recorded run status and worktrees — repeatable, no provider invocation + +Command: + +```bash +set -euo pipefail +pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +test "$(wc -l <"$pointer")" -eq 1 +bench_run_id="$(tr -d '\n' <"$pointer")" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")" +printf '%s\n' "$status_output" +STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states +print(f"ok: nine retained terminal attempts states={states}") +PY +test -z "$(git -C ../iop-s2 status --porcelain=v1)" +git diff --check +``` + +Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed and no whitespace errors. + +Actual stdout/stderr: + +_Fill with verbatim output._ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry | +| Code Review Result | Review agent appends | Not included in stub | diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log new file mode 100644 index 00000000..aba4a0d3 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log @@ -0,0 +1,206 @@ + + +# Code Review Reference - official agy IOP 호환, managed dev 배포와 C01-C09 실행 + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Finalization (`Code Review Result`, archive, `complete.log`) is review-agent-only. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=3, tag=API + +## Archive Evidence Snapshot + +- Generation 0/1/2 are preserved in the same task directory and have no official verdict. Active plan=3 replaces the invalid synthetic agy/legacy credential premise while preserving the immutable one-run rule. +- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` satisfies the index dependency but does not remove the current live-readiness blocker. + +## For the Review Agent + +Compare every implementation item with source and fresh repeatable verification. Never repeat the one-time C01-C09 provider run. Append verdict and routing signals, archive the active pair, and create `complete.log` only on PASS. + +## Implementation Item Completion + +| Item | Status | +|------|--------| +| API-1: Gemini-native Edge ingress | [x] | +| API-2: official agy 1.1.12 adapter | [x] | +| API-3: managed dev deployment and live readiness | [x] | +| API-4: immutable C01-C09 run | [ ] | + +## Implementation Checklist + +- [x] API-1: Implement and test the route-qualified Gemini request/auth/SSE bridge through existing route and preset admission. +- [x] API-2: Use official agy 1.1.12 Gemini API-key transport and real stream-json lifecycle with config-owned binding evidence. +- [x] API-3: Deploy fresh artifacts and a complete managed credential/TLS composition to dev, proving direct/hybrid readiness without legacy fallback. +- [ ] API-4: Invoke the public C01-C09 run exactly once after readiness and retain exactly nine non-interrupted terminal attempts. +- [x] Fill all implementation-owned sections below with actual notes and verification output. + +## Review-Only Checklist + +- [ ] Append one verdict and verified `review_rework_count` / `evidence_integrity_failure` signals. +- [ ] Run all applicable repeatable verification and compare it with implementation evidence. +- [ ] Confirm no inbound principal token can become provider authorization and no secret appears in tracked evidence. +- [ ] Confirm official agy argv/env/events and config-owned effective binding match the contract. +- [ ] Confirm dev has CA-signed TLS, projection, slot-route authorization and no legacy credential fallback. +- [ ] Confirm the scored run occurred at most once and has exactly nine retained terminal attempts. +- [ ] Archive the active pair and write `complete.log` only for PASS. + +## Deviations from Plan + +- 공식 `agy` 1.1.12의 실제 JSONL은 flat `status`/`usage`가 아니라 `event`와 같은 이름의 `init`/`step_update`/`result` 중첩 payload를 사용했다. adapter, fixture와 integration seam을 실제 출력 구조로 교정했다. +- `/tmp`는 실행 불가라 deterministic credential smoke의 임시 실행 경로를 ignored `build/test-tmp`로 옮겼다. 이 과정에서 smoke가 내부 route id를 기다리던 오래된 판정을 찾아, 계약대로 public route alias를 기다리도록 수정했다. +- 공식 agy hybrid와 raw Chat hybrid에서 plan output의 마지막 LF 생략, work/review tool path와 review JSON strictness 문제가 드러나 parser의 최종 LF 경계, strict tool schema/한 번의 bounded correction, review response schema를 보완했다. +- 공개 `run`은 all-cell preflight 뒤 정확히 한 번 호출했지만 첫 caller launch 전에 shared filesystem이 symlink 경유 Unix socket bind를 `EINVAL`로 거부했다. 상대 basename bind와 0700 parent fallback, pre-registration reconcile을 구현하고 회귀 테스트를 추가했다. one-run 원칙에 따라 `run`/`resume`은 다시 호출하지 않았다. + +## Key Design Decisions + +- Gemini ingress는 별도 provider 우회 경로를 만들지 않고 인증된 내부 Chat request로 변환해 기존 route/preset admission, managed credential projection과 stage authorization을 그대로 사용한다. +- inbound `x-goog-api-key`는 IOP principal token으로만 사용하고 provider credential은 slot/lease projection에서만 가져온다. caller token, provider token과 private key는 tracked evidence에 기록하지 않는다. +- official caller child에는 최소 환경을 유지하되 private dev CA가 필요한 경우 표준 `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS` 두 값만 명시적으로 전달한다. +- pre-registration interruption은 invocation identity가 존재하지 않으므로 불가능한 measurement/web sidecar를 발명하지 않는다. locator 또는 일부 sidecar가 존재하는 경우에는 기존 엄격 검증을 유지한다. + +## Reviewer Checkpoints + +- Confirm `/gemini/{route}/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse` is authenticated using the IOP principal, while provider credentials still come only from managed projection. +- Confirm malformed/conflicting auth and unsupported Gemini bodies fail before dispatch, and stream cancellation/error/tool/usage paths remain bounded and caller-safe. +- Confirm agy uses `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`, official model label and no `--effort` or invented `AGY_*` variables/events. +- Confirm remote config and live evidence prove managed credentials, HTTPS/mTLS, exact direct/hybrid routes and no static provider credential source. +- Confirm C01-C09 is invoked only after all-cell preflight; no retry/resume/manual result editing is allowed. + +## Verification Results + +### Focused local suites — repeatable + +Command: + +```bash +go test -count=1 ./apps/edge/internal/openai +python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test +``` + +Actual stdout/stderr: + +```text +ok iop/apps/edge/internal/openai 8.326s +ok iop/apps/edge/internal/service 8.252s +ok iop/packages/go/singlerequesttemplate 0.006s + +Ran 62 tests in 19.224s +OK +``` + +### Full local suites and contract hygiene — repeatable + +Command: + +```bash +go test -p 1 -count=1 ./... +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Actual stdout/stderr: + +```text +go test -p 1 -count=1 ./...: PASS (all packages) +Ran 429 tests in 115.981s +OK +ok: manifest is valid +git diff --check: PASS +``` + +첫 full Go 병렬 검증에서는 unrelated transport duplicate-registration test 한 건이 45초 timeout이었으나 단독 0.01초, package 4.766초에 통과했고 부하 없이 반복한 전체 `-p 1` run은 모두 통과했다. + +### Managed credential deterministic qualification — repeatable + +Command: + +```bash +credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)" +TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke +rmdir "$credential_smoke_parent" +``` + +Actual stdout/stderr: + +```text +mode=deterministic +profiles=seulgi_chat,seulgi_messages +same_model_two_slot=true +exact_auth.chat=2 +exact_auth.messages=1 +ciphertext_only=true +tls_negative_matrix=passed +post_revoke.counters_unchanged=true +post_revoke.no_fallback=true +result=success +``` + +실제 slot/route 식별자는 secret-safe evidence 요구에 따라 생략했다. + +### Remote managed dev and live caller readiness — repeatable except provisioning + +Record sanitized source/artifact/config identity, runtime health, official agy direct/hybrid smoke and public preflight output. Do not record secrets, raw request content, private keys, slot aliases or lease ids. + +Actual stdout/stderr: + +```text +edge_sha256=c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc +edge_pid=172 +ports=18082,18083,18084,19093,19101 all_up +nodes=4 connected=4 providers=8 provider_states=available/healthy +nodes=mac-codex-node,gx10-vllm-node,onexplayer-lemonade-node,rtx5090-lemonade-node + +raw gemini-hybrid smoke #1: HTTP 200, finish=stop, errors=0, done=true +raw gemini-hybrid smoke #2: HTTP 200, finish=stop, errors=0, done=true +official agy direct: exit=0, result.status=SUCCESS, stderr_bytes=0 +official agy hybrid: exit=0, result.status=SUCCESS, workspace_unchanged=true, stderr_bytes=0 +ok: preflight status=ready ready=9 registration_required=0 implementation_gap=0 +``` + +### C01-C09 run — implementation-only, never repeat in review + +Record the single public `run` command exit/output, its one CLI-emitted run id, and public `status` classification. + +Actual stdout/stderr: + +```text +$ python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +error: benchmark state is unavailable +exit=69 +``` + +CLI-emitted run id는 없었으므로 `run_id.log`는 생성하지 않았다. Controller가 만든 state는 첫 slot에서 `caller_launched=false`, `terminal_reason=supervisor_error`, cleanup 완료였다. 원인은 symlink path Unix socket `bind(2)`의 `EINVAL`이며 수정 후 symlink control socket 및 pre-registration reconcile 회귀 테스트가 통과했다. 같은 run을 재시도하거나 resume하지 않았다. + +### Recorded run status and worktrees — repeatable + +Record public status for the stored pointer, nine terminal attempts, zero running/interrupted, and `git diff --check`. + +Actual stdout/stderr: + +```text +controller_state=run-20260812T031657Z-0a24376c2414 +status before reconcile: running=1, all other terminal counts=0 +status after harness reconcile: interrupted=1, running=0, success=0, failed=0, timed_out=0, cancelled=0 +caller_launched=false +live_supervisor=false +control_alias_count=0 +run_id.log=absent +testbed=../iop-s2 branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true +git diff --check: PASS +``` + +API-4 acceptance인 nine terminal attempts/zero interrupted는 충족하지 못했다. resume/retry 없이 새 실행 정책 승인이 있어야 C01-C09를 다시 시작할 수 있다. + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section and check only completed items. Leave review-only sections unchanged. + +## Code Review Result + +_Review-agent only._ diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log new file mode 100644 index 00000000..cfc75663 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log @@ -0,0 +1,267 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected scope and state boundary as written. Do not choose another owner, narrow/expand the write boundary, or replace the run with an alternate caller path. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume. +> Follow the ownership table at the bottom of this file for which sections you own. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=4, tag=TEST + +## Archive Evidence Snapshot + +- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다. +- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다. +- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다. +- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section. + +Compare implementation of each item against the active Plan and immutable run state. Repeat only local tests, manifest validation, secret-safe preflight and public `status`; never repeat `run`, invoke `resume`, use `--retry-failed`, call a caller/provider directly, or edit a run tree. Review completion means the following steps are finished: + +1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals. +2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_4.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_4.log`. +3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; if WARN/FAIL, write the next state required by the code-review skill. +4. Preserve first-line `milestone-task` metadata for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`. +5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting. + +--- + +## Implementation Item Completion + +| Item | Status | +|------|---------| +| TEST-1: Protected runtime binding and fresh readiness | [x] | +| TEST-2: One replacement C01-C09 execution run | [ ] | + +## Implementation Checklist + +- [x] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output. +- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent. +> Implementing agents must not modify or check this section. + +- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`. +- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match. +- [x] Run applicable repeatable verification and record fresh output; do not repeat the execution run. +- [x] Confirm prior interrupted/preflight-only run roots remain append-only and no stored pointer was written for the non-emitted execution id. +- [x] Confirm public status is `failed=1, running=1` rather than the required nine terminal attempts and classify the gap as Required. +- [x] Confirm no secret value, raw prompt/response, private key, slot alias or lease id appears in tracked evidence. +- [x] Archive active `CODE_REVIEW-cloud-G10.md` to `code_review_cloud_G10_4.log`. +- [x] Archive active `PLAN-cloud-G10.md` to `plan_cloud_G10_4.log`. +- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`. +- [ ] If PASS, write `complete.log` and leave no active `.md` files. +- [ ] If PASS, move the active task directory to the archive path and update this checklist at the final archive path. +- [ ] If PASS, preserve and report `milestone-task` metadata without modifying roadmap directly. +- [x] If WARN/FAIL, write the next filesystem state matching the code-review verdict and do not write `complete.log`. + +## Deviations from Plan + +- Planning 중 보호된 runtime reference가 없는 셸에서 공개 preflight를 호출해 `implementation_gap=9`가 발생했다. attempt는 생성되지 않았으며 `token/.iop-bench`, private dev CA와 배포 config observation을 process-local environment에 연결한 뒤 `ready=9`를 확인했다. +- 최초 execution wrapper는 임시파일 cleanup 구문이 host safety policy에서 process 시작 전에 거부됐다. public benchmark CLI는 호출되지 않았고 active Plan/Review의 command를 direct CLI output 방식으로 교정한 뒤 실행했다. +- 교정 후 public `run`은 정확히 한 번 호출됐다. 첫 read-only status는 `failed=1,running=1`이었고 CLI는 이후 `error: benchmark state is unavailable`, exit 69로 종료했다. CLI-emitted run id가 없고 최종 public status도 `failed=1,running=1`이라 TEST-2 acceptance를 충족하지 못했다. +- observed controller directory는 `run-20260812T040619Z-00a5e6664764`지만 CLI-emitted canonical id가 아니므로 `run_id.log`를 생성하지 않았다. 해당 run을 resume/retry/reconcile하거나 run tree를 편집하지 않았다. + +## Key Design Decisions + +- preflight/run/status는 모두 public deterministic benchmark CLI만 사용했다. caller/provider 직접 호출이나 route/model/effort 대체는 하지 않았다. +- retained failure와 dangling status를 성공으로 재분류하지 않는다. 실행 process가 없는 것을 read-only로 확인했지만 내부 `RunStore`나 reconcile API로 상태를 바꾸지 않는다. +- TEST-2는 exact blocker 상태로 남긴다. 재개 조건은 official review가 `benchmark state is unavailable`의 root cause와 one selected fix를 확정하고, append-only evidence를 보존하는 follow-up plan을 생성하는 것이다. + +## Reviewer Checkpoints + +- Confirm the no-reference `implementation_gap=9` preflight and the corrected `ready=9` preflight allocated no scored attempts. +- Confirm TEST-2 uses one new public `run` identity and does not resume or modify `run-20260812T031657Z-0a24376c2414`. +- Confirm each C01-C09 cell has one retained accepted terminal attempt under the new run, with fresh workspace/session identity and IOP route binding. +- Confirm `run_id.log` contains exactly the CLI-emitted new execution run id and public `status` matches the recorded evidence. +- Never use execution success alone to complete the third Epic; validation/scoring/report remain separately planned S09-S12 work. + +## Verification Results + +### Fresh local contract checks — repeatable + +Command: + +```bash +python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output. + +Actual stdout/stderr: + +```text +..................................................................... +---------------------------------------------------------------------- +Ran 69 tests in 59.949s + +OK +ok: manifest is valid +``` + +`git diff --check` stdout/stderr: `(none)` + +### Protected all-cell preflight — repeatable, no scored attempt + +Command: + +```bash +set -euo pipefail +read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench +export IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083 +export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083 +export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1 +export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem" +export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem" +export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG +BENCH_CONFIG="$(python3 - <<'PY' +import json +routes = [ + {"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]}, + {"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]}, + {"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]}, +] +print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":"))) +PY +)" +export BENCH_CONFIG +for tool in python3 claude agy codex git; do command -v "$tool"; done +python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`. + +Actual stdout/stderr: + +```text +/bin/python3 +/config/.npm-global/bin/claude +/config/.local/bin/agy +/config/.npm-global/bin/codex +/bin/git +ok: preflight run_id=run-20260812T040619Z-1ed70fffdbda status=ready ready=9 registration_required=0 implementation_gap=0 +``` + +### C01-C09 execution — implementation-only, never repeat in review + +Command: + +```bash +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: the CLI emits exactly one canonical execution id. Record its exit code and stdout/stderr verbatim; public status is checked separately. A retained failure is preserved and never retried. + +Actual stdout/stderr: + +```text +command: run +exit_code: 69 +stdout: +benchmark_exit_code=69 +stderr: +error: benchmark state is unavailable +cli_emitted_run_id=(none) +``` + +Intermediate public status while the command was active: + +```text +ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0} +``` + +### Stored execution status — repeatable, provider-free + +Command: + +```bash +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id" +git diff --check +``` + +Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error. + +Actual stdout/stderr: + +The planned stored-pointer command was not run because `run_id.log` was correctly absent. Read-only blocker checks were: + +```text +$ python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id run-20260812T040619Z-00a5e6664764 +ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0} + +$ protected path/process check +run_id.log=absent +live_execution_process=false +``` + +`git diff --check` stdout/stderr: `(none)` + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?** +> If anything is blank, go back and fill it in before saving this file. +> Leave review-agent-only sections unchanged. + +## Section Ownership + +| Section | Owner | Note | +|---------|-------|------| +| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) | +| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly | +| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only | +| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only | +| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section | +| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content | +| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan | +| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns only repeatable commands and never the execution run | +| Code Review Result | Review agent appends | Not included in stub | + +## Code Review Result + +- Overall Verdict: FAIL +- Dimension Assessment: + - Correctness: Fail — production Codex output is rejected and the public run aborts with one dead attempt retained as `running`. + - Completeness: Fail — TEST-2 requires nine non-interrupted terminal attempts, but public status is `failed=1, running=1` after only two allocated cells. + - Test coverage: Fail — the fixture and integration seam omit a production usage key and synthesize a caller binding event that the real Codex CLI does not emit. + - API contract: Fail — parser and consumers disagree about both the current Codex JSONL usage schema and the documented optional caller binding observation. + - Code quality: Pass — no unrelated debug output, TODO, or formatting defect was found in the reviewed scope. + - Implementation deviation: Pass — the implementation preserved the failed run, did not write `run_id.log`, and did not resume, retry, or edit state. + - Verification trust: Pass — recorded exit/status evidence matches fresh read-only status, process inspection, lifecycle replay, and retained attempt files. + - Spec conformance: Fail — S04-S08 still lack the required C01-C09 terminal execution evidence. +- Findings: + - Required R1 — Current Codex usage JSONL is rejected. + - Evidence: replaying the retained C05 `lifecycle-result.json` stdout through `CodexJSONLParser` fails on line 26 with `CodexJSONLError: invalid Codex usage observation`; the `turn.completed.usage` keys are `cache_write_input_tokens`, `cached_input_tokens`, `input_tokens`, `output_tokens`, and `reasoning_output_tokens`. `scripts/agent_benchmark/codex_iop.py:79-85,334-345` does not admit `cache_write_input_tokens`. Fresh `python3 -m unittest scripts.agent_benchmark.codex_iop_test` still reports 9 tests/OK, proving the current test set misses the production shape. + - Root Cause: the Codex adapter's exact usage-key mapping predates the caller's `cache_write_input_tokens` field, while the tracked fixture and parser unit test still use the older subset. + - Selected Fix: add `cache_write_input_tokens -> cache_write_tokens` to `_CODEX_USAGE_FIELDS`; update `scripts/agent_benchmark/codex_iop_test.py` and `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` with the retained production-shaped usage object and assertions that the value is preserved without reconstructing totals. Acceptance commands: `python3 -m unittest scripts.agent_benchmark.codex_iop_test` and the focused lifecycle/integration suite in the follow-up plan. + - Required R2 — Config-owned binding admission is contradicted by caller-owned enforcement. + - Evidence: `CodexJSONLParser.effective_binding` is explicitly optional and `test_bridge_proves_finish_then_idle_after_child_exit` asserts `None`, but `scripts/agent_benchmark/live_iop.py:992-995` rejects every live invocation unless the caller emits an exact `iop_effective_binding`. The retained real Codex stdout contains no such synthetic event. The live scoring consumer at `scripts/agent_benchmark/live_iop.py:1123-1131` applies the same impossible requirement. `scripts/agent_benchmark/connectivity_integration_test.py:217-227` currently injects the synthetic event and `:2403-2436` asserts that absence is an error, hiding the production mismatch. + - Root Cause: preflight moved effective route/model/stage ownership to the independently validated config observation, but execution and scoring consumers retained the earlier caller-event equality gate and their seams continued fabricating that event. + - Selected Fix: in `_LiveAdapter.invoke` and `_LiveScoringAdapter.invoke`, use the admitted config binding as the canonical result; treat an optional caller observation as an additional exact-match check only when present, and still reject any non-`None` mismatch. Remove the synthetic Codex binding from the integration fixture and add execution plus scoring assertions that absent observation succeeds from admitted config while mismatches fail. Acceptance commands: the focused `codex_iop_test` and named connectivity integration tests, followed by the full Python benchmark suite and `git diff --check`. +- Routing Signals: review_rework_count=1 evidence_integrity_failure=false +- Next Step: Run the mandatory plan-skill WARN/FAIL follow-up for Required R1 and R2; after deterministic fixes pass, a new scored execution remains authorization-gated because this retained run must not be retried implicitly. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_6.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_6.log new file mode 100644 index 00000000..9e4c778a --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_6.log @@ -0,0 +1,101 @@ + + +# Code Review Reference - TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=6, tag=TEST + +## Archive Evidence Snapshot + +- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained execution defect evidence. +- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 fixed with 41/429 tests passing. +- `user_review_0.log`: continuing execution authorization resolved. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| TEST-1 deterministic regression | [x] | +| TEST-2 protected ready=9 preflight | [x] | +| TEST-3 new C01-C09 public run | [ ] | + +## Implementation Checklist + +- [x] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh. +- [x] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`. +- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output. + +## Review-Only Checklist + +- [ ] Append verdict and routing signals. +- [ ] Run repeatable verification without repeating scored execution. +- [ ] Archive active pair to `plan_cloud_G10_6.log` and `code_review_cloud_G10_6.log`. +- [ ] If PASS, write complete.log and archive task; if non-PASS, materialize the required next state. + +## Deviations from Plan + +TEST-3의 public `run`은 계획대로 정확히 한 번 호출했으나 exit 69로 종료됐다. CLI가 canonical id를 출력하지 않아 `run_id.log`를 쓰지 않았고, 새 run은 read-only blocker evidence로만 식별했다. resume/retry/direct caller/provider/state edit는 수행하지 않았다. + +## Key Design Decisions + +Codex C05 success가 종결된 뒤 C03 agy의 ordinary nonzero lifecycle에서 controller exception이 발생한 사실을 보존했다. retained run은 재개하지 않고, ordinary lifecycle failure를 terminal attempt로 처리하는 source fix를 후속 plan에서 적용한다. + +## Verification Results + +### TEST-1 deterministic regression + +```text +focused: Ran 41 tests in 15.084s — OK +full: Ran 429 tests in 118.691s — OK +manifest: ok: manifest is valid +git diff --check: (none) +``` + +### TEST-2 protected preflight + +```text +ok: preflight run_id=run-20260812T043640Z-91c437c5c7cd status=ready ready=9 registration_required=0 implementation_gap=0 +``` + +### TEST-3 one public run and status + +```text +command: run +exit_code: 69 +stdout: (none) +stderr: error: benchmark state is unavailable +cli_emitted_run_id: (none) +identified_new_run: run-20260812T043704Z-041320764a11 +public_status: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 1, 'timed_out': 0} +attempts: c02=failed, c05=success, c03=running +``` + +C03 retained lifecycle is `success=false`, `terminal_reason=nonzero_exit`, `exit_code=1`, `cleanup_complete=true`, `process_group_alive=false`, but attempt-measurement/web-validation are absent and `attempt.json` remains `running`. + +## Section Ownership + +- Implementation owns completion status, deviations, decisions, and verification output. +- Review owns verdict, archive, complete.log, next state, and roadmap completion event. + +## Code Review Result + +- Overall Verdict: FAIL +- Dimension Assessment: + - Correctness: Fail — an ordinary agy nonzero lifecycle is converted to an adapter exception and leaves a dead attempt running. + - Completeness: Fail — only three attempts were allocated and C01-C09 terminal evidence is incomplete. + - Test coverage: Fail — current live integration coverage does not assert that an agy failed lifecycle is returned for terminal publication and that later slots continue. + - API contract: Fail — execution adapter treats caller execution failure as connectivity-contract failure after preflight was already admitted. + - Code quality: Pass — retained evidence and prior Codex fix are consistent and bounded. + - Implementation deviation: Pass — exactly one run was called and no forbidden recovery path was used. + - Verification trust: Pass — public status and retained lifecycle/sidecar evidence agree. + - Spec conformance: Fail — S04-S08 still lack nine terminal attempts. +- Findings: + - Required R4 — agy ordinary failed lifecycle aborts the whole run instead of terminalizing the attempt. + - Evidence: C03 `lifecycle-result.json` is a clean `nonzero_exit` with process cleanup complete; `live_iop.py` calls `parser.observed_result`, receives non-ready because lifecycle success is false, then raises `LiveIopError`. This occurs before `run_slots.invoke_bound` publishes measurement/web validation, so `attempt.json` stays `running` and the CLI returns unavailable state. C05 immediately before it is `success`, proving the earlier Codex fix works. + - Root Cause: `_LiveAdapter.invoke` conflates post-admission execution outcome with preflight binding verification. It applies the successful-stream `observed_result` gate even when lifecycle already has a valid terminal failure. + - Selected Fix: in the agy branch, return `_bound_observations(result, admitted)` immediately for any `result.success == false`; apply `parser.observed_result` exact binding checks only to successful lifecycle results. Add credential-free integration coverage for failed agy lifecycle terminal publication and later-slot continuation, then run full benchmark tests, fresh ready=9 preflight, and one new public run under the user's continuing authorization. +- Routing Signals: review_rework_count=3 evidence_integrity_failure=false +- Next Step: Run the mandatory plan-skill FAIL follow-up for R4, including one authorized new public run after the deterministic fix; preserve this run without resume/retry/edit. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_7.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_7.log new file mode 100644 index 00000000..6308544b --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_7.log @@ -0,0 +1,97 @@ + + +# Code Review Reference - REVIEW_TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=7 + +## Archive Evidence Snapshot + +- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: R4 evidence and retained run. +- `user_review_0.log`: continuing execution authorization. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| FIX-1 agy failure terminalization | [x] | +| FIX-2 regression coverage | [x] | +| TEST-1 deterministic verification | [x] | +| TEST-2 ready=9 and one new run | [x] | + +## Implementation Checklist + +- [x] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams. +- [x] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots. +- [x] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check. +- [x] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted. +- [x] Fill implementation evidence. + +## Review-Only Checklist + +- [x] Append PASS verdict/routing signals, run repeatable verification, archive to `_7.log`, write `complete.log`, move the task directory, and preserve milestone completion metadata. + +## Deviations from Plan + +없음. 새 public run은 정확히 한 번 호출했으며 exit 69은 retained failed cells 때문이었다. CLI-emitted canonical id만 `run_id.log`에 기록했고 old run resume/retry/state edit 또는 direct caller/provider 호출은 없었다. + +## Key Design Decisions + +- agy lifecycle 자체가 실패하면 connectivity binding을 재판정하지 않고 admitted binding에 대해 typed observation만 검증해 RunStore가 정상 terminal publication을 수행하게 했다. +- 성공 lifecycle에만 `observed_result` exact config-binding gate를 유지해 contract mismatch는 계속 fail-closed한다. +- 개별 failed 결과를 재시도하지 않고 D10 retained evidence로 수용했다. + +## Verification Results + +### Source/tests + +```text +focused: Ran 75 tests in 51.600s — OK +full: Ran 430 tests in 122.446s — OK +manifest: ok: manifest is valid +git diff --check: (none) +``` + +### Protected preflight/run/status + +```text +preflight: ok: preflight run_id=run-20260812T044743Z-40dd65f85a35 status=ready ready=9 registration_required=0 implementation_gap=0 +run exit_code: 69 +run stderr: error: benchmark execution failed run_id=run-20260812T044800Z-412e05fc80df completed=9 unresolved=7 success=2 failed=7 timed_out=0 cancelled=0 interrupted=0 running=0 +status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0} +attempt_count: 9 +all_terminal: True +all_sidecars: True +``` + +## Section Ownership + +Implementation owns status/evidence; review owns verdict/archive/complete/next state. + +## Code Review Result + +- Overall Verdict: PASS +- Dimension Assessment: + - Correctness: Pass — agy failed lifecycles now terminalize and later slots continue; successful binding checks remain exact. + - Completeness: Pass — C01-C09 all have retained terminal attempts, required measurement/web sidecars, and zero running/interrupted. + - Test coverage: Pass — focused 75 and full 430 tests include adapter failure return and later-slot continuation. + - API contract: Pass — post-admission execution failure is distinct from successful-stream binding validation. + - Code quality: Pass — the source change is a bounded branch guard with focused regression coverage. + - Implementation deviation: Pass — one fresh run, no resume/retry/substitution/state edit/direct caller/provider. + - Verification trust: Pass — reviewer fresh tests, manifest, diff, pointer format, and public status all agree. + - Spec conformance: Pass — S04-S08/D06/D10 accept the complete nine-cell success/failure terminal matrix with preserved failures. +- Findings: None +- Routing Signals: review_rework_count=3 evidence_integrity_failure=false +- Next Step: PASS — archive the pair, write complete.log, move the task directory, and emit milestone completion metadata for runtime aggregation. + +### Reviewer Fresh Verification + +```text +focused: Ran 75 tests in 45.609s — OK +full: Ran 430 tests in 117.951s — OK +manifest: ok: manifest is valid +status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0} +git diff --check: (none) +``` diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log new file mode 100644 index 00000000..6e53243f --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log @@ -0,0 +1,47 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs + +## 완료 일시 + +2026-08-12 + +## 요약 + +8개 plan/review revision과 지속 실행 승인 1건을 거쳐 C01-C09의 fresh one-shot terminal evidence를 완성했고 최종 verdict는 PASS다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|---|---|---|---| +| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | 초기 실행 준비 revision. | +| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | readiness 정합화 revision. | +| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | runtime/provider 준비 revision. | +| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, managed dev, all-cell readiness와 pre-launch controller fix. | +| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Codex current usage와 config-owned binding 결함 발견. | +| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | Codex 결함 수정 및 실행 지속 승인 대기. | +| `plan_cloud_G10_6.log` | `code_review_cloud_G10_6.log` | FAIL | Codex success 확인 후 agy failed-lifecycle terminalization 결함 발견. | +| `plan_cloud_G10_7.log` | `code_review_cloud_G10_7.log` | PASS | R4 수정, 9개 terminal attempt 및 sidecar evidence 완성. | + +## 구현/정리 내용 + +- Gemini-native Edge ingress, official agy transport, managed credential dev binding과 all-cell readiness를 정합화했다. +- Codex `cache_write_input_tokens`를 canonical metric으로 수용하고 config-owned binding을 execution/scoring에서 일관되게 적용했다. +- agy ordinary failed lifecycle을 terminal attempt로 보존해 이후 슬롯 실행이 계속되도록 수정했다. +- canonical run `run-20260812T044800Z-412e05fc80df`에 C01-C09 9개 attempt, `success=2`, `failed=7`, `running=0`, `interrupted=0`과 모든 measurement/web-validation sidecar를 보존했다. + +## 최종 검증 + +- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test` - PASS; reviewer fresh 75 tests/OK. +- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; reviewer fresh 430 tests/OK. +- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; manifest valid. +- public `status --run-id run-20260812T044800Z-412e05fc80df` - PASS; 9 terminal, running/interrupted 0. +- `git diff --check` - PASS; output 없음. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- 다음 Epic의 objective validation, blind scoring, aggregation/reporting은 별도 task로 진행한다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_0.log new file mode 100644 index 00000000..7ec8e546 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_0.log @@ -0,0 +1,201 @@ + + +# Plan - C01-C09 원샷 비교 실행 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 아래 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다. + +## Background + +승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행하도록 정의한다. `run` 경로가 단일 writer 안에서 fresh all-cell preflight 뒤 모든 slot을 순회하므로 caller/model별로 분할 실행하면 동일 조건과 실패 보존 계약이 깨진다. 이 작업은 외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아니며, 준비된 harness를 한 번 실행하고 검증 evidence를 고정하는 하나의 large slice다. + +## Archive Evidence Snapshot + +- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`다. 미해결 Required/Suggested는 기재되지 않았고 잔여 Nit은 없다. +- 선행 작업은 source를 변경하지 않았고, clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 stale `../iop-s2/build/bin/iop-edge` 사이의 freshness blocker를 exit 69로 재현했다. Manifest validate는 `ok: manifest is valid`였다. +- Roadmap carryover는 위 exact HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller version/help, secret-safe environment reference, public nine-cell preflight를 포함한 gate를 다시 통과하는 것이다. 구현자는 archive 전체를 탐색하지 말고 추가 세부가 꼭 필요할 때만 위 `complete.log` 한 건을 읽는다. + +## Analysis + +### Files Read + +- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md` +- Runner source: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/codex_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/workspace.py` +- Tests/rules: `scripts/agent_benchmark/attempts_test.py`, `scripts/agent_benchmark/skill_contract_test.py`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md` +- Fixture: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- Prior evidence: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` + +### SDD Criteria + +- SDD는 `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`이며 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다. +- 첫 줄 metadata는 `milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`를 그대로 보존한다. +- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다 (`SDD.md:105-109`). +- Evidence Map S04-S08은 각각 event/timing/usage/workspace evidence와 hybrid stage/terminal evidence를 요구한다 (`SDD.md:122-126`). 공통 완료 조건은 모든 cell의 terminal evidence 및 성공 결과의 gate/screenshot, 모든 attempt의 timing/usage source다 (`SDD.md:132`). +- 따라서 checklist는 C01-C09를 한 번만 제출하고 성공/실패 evidence를 보존하도록 구성했으며, final verification은 canonical `run_id`의 9개 success와 unresolved 0을 public `status`로 확인한다. Scoring/report evidence인 S09-S12는 이 Epic 범위가 아니다. + +### Verification Context + +- 별도 handoff는 없었다. 위 Files Read의 runner, tests, fixture, SDD, spec, contracts 및 exact predecessor `complete.log`에서 실행 계약을 재구성했다. +- 적용 기준은 local/dev testing smoke, 동일 fixture/checksum, repetitions 1, fresh session, isolated setup cache, fixed seed, no manual retry다. +- 준비 단계에서 `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`은 `ok: manifest is valid`, `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`는 91 tests/OK, `git diff --check`는 출력 없이 통과했다. Cached test output은 허용하지 않으며 구현·review에서 fresh 실행한다. +- Repository-native fallback evidence는 manifest validator, 91개 unit/contract test, public `run`/`status` CLI다. Provider 호출 성공을 정적 test로 대체하지 않는다. +- 제약: secret 값과 raw private endpoint/config payload를 출력하지 않고, `../iop-s2`는 clean/read-only fixture source로 유지하며, run state는 CLI가 검증한 `agent-test/runs/bench-02//` 아래에만 생성한다. 한 번 시작한 `run`을 재실행하거나 `resume --retry-failed`로 실패 evidence를 대체하지 않는다. +- Gap은 현재 준비 host의 `IOP_BENCH_*` runtime reference가 unset이고 선행 evidence의 artifact freshness blocker가 아직 해소됐다는 증거가 없다는 점이다. 이는 제품/범위 결정이 아니라 실행 환경 선행조건이며, 해소되지 않으면 `run`을 호출하지 않고 review evidence에 exact blocker와 resume condition을 남긴다. +- Confidence는 medium이다. Repository-native 실행/검증 계약은 명확하지만 실제 provider와 current-HEAD artifact readiness는 외부 runtime이 준비된 실행 시점에만 증명된다. Maintenance mode 전환은 필요 없다. + +#### External Verification Preflight + +- Runner/repo: local Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, starting/current HEAD `e86113f0ae3faecf8dfe715990e4755cdc42bedf`, 준비 시 clean. +- Testbed: `/config/workspace/iop-s2`, branch `dev`, HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, 준비 시 clean. Source sync는 수행하지 않았고 이 exact HEAD를 고정한다. +- Binaries: `/bin/python3` (3.12.3), `/config/.npm-global/bin/claude`, `/config/.local/bin/agy`, `/config/.npm-global/bin/codex`, `/bin/git`. Caller version/help는 public run preflight가 secret-safe하게 검사한다. +- Artifact/config/runtime: stale blocker는 `../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD`다. Exact setup 조건은 위 testbed HEAD에서 Linux AArch64 Edge/Node artifact를 runtime owner가 재빌드하고, 세 caller의 `BASE_URL`/`SECRET_ENV` reference 및 `IOP_BENCH_CONFIG_OBSERVATION_ENV` reference가 유효한 환경에서 실행하는 것이다. Config는 file path가 아니라 environment-owned JSON reference다. +- Ports/process/external hosts: raw 값은 benchmark evidence 계약상 출력하지 않는다. Public all-cell preflight가 endpoint/catalog/config identity를 hash로 관측하고 실패 시 exit 69로 닫는다. + +### Test Coverage Gaps + +- Source behavior change는 없다. Existing tests는 run 생성, single writer, fresh preflight, terminal state, retry 보존, skill command/report contract를 다룬다. +- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 실제 C01-C09 run 대상이다. +- 새 test는 추가하지 않는다. 검증된 production harness의 외부 실행이 이 slice의 변화이며, 별도 mock test는 acceptance evidence를 늘리지 못한다. + +### Symbol References + +None. Rename/remove/change하는 symbol이 없다. + +### Split Judgment + +- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가지므로 작은 독립 변경 기준을 충족하지 않는다. +- 다섯 task id는 `execution_order_seed=bench-02-c01-c09-v1`, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유한다 (`agent-comparison-benchmark-iop-one-shot.json:5-8,43-177`, `attempts.py:1928-1969`). 따라서 하나의 plan으로만 실행한다. +- 이 split subtask의 predecessor index `05`는 archived `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` 한 건으로 만족한다. 다만 그 PASS가 보존한 current-HEAD artifact freshness 조건은 TEST-1의 pre-execution gate로 남는다. + +### Scope Rationale + +- `scripts/**`, `agent-spec/**`, `agent-contract/**`, SDD/Milestone 문서는 변경하지 않는다. Harness/manifest/계약은 이미 확정돼 있고 이 Epic은 실제 실행만 소유한다. +- `score`, blind evaluator, aggregate/report, Milestone task S09-S12는 후속 Epic 범위이므로 호출하거나 생성하지 않는다. +- Caller/model/effort/preset 대체, extra repetition, manual edit, retry/resume, testbed source 변경은 immutable comparison과 failure preservation을 깨므로 제외한다. + +### Final Routing + +- `evaluation_mode=pair`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh`. +- Build target: closure=closed, grade=`G09`, route=`cloud`, canonical filename=`PLAN-cloud-G09.md`. +- Review target: closure=closed, grade=`G10`, route=`cloud`, canonical filename=`CODE_REVIEW-cloud-G10.md`. +- `large_indivisible_context=false`; positive loop risks는 `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product`의 4개다. +- Recovery signals는 `review_rework_count=0`, `evidence_integrity_failure=false`다. +- Capability gap은 없다. 실행 환경이 미준비이면 exact external blocker로 닫는 방법과 재개 조건이 정해져 있고, 실행/검증은 repository-native CLI로 완결된다. + +## Dependencies and Execution Order + +1. 디렉터리명 `06+05_comparison_runs`의 유일한 runtime dependency는 predecessor `05`다. `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`가 이를 충족한다. +2. 위 predecessor가 보존한 exact testbed HEAD artifact rebuild와 secret-safe environment references가 준비된 뒤 TEST-1을 시작한다. 이것은 새 task dependency가 아니라 TEST-1 자체의 fail-closed execution precondition이다. + +## Implementation Checklist + +- [ ] TEST-1: Confirm the fixed external preconditions without exposing secrets, execute the immutable C01-C09 `run` exactly once, persist its canonical run id, and verify all nine attempts reach success with unresolved 0; if the gate blocks or an attempt fails, do not retry and record the exact evidence and resume condition. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Immutable C01-C09 comparison run + +#### Problem + +`scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json:43-177` binds C01-C09 to five Milestone tasks, but no canonical scored run exists yet. `scripts/agent_comparison_benchmark.py:124-176` creates one run and reports unresolved slots, while `scripts/agent_benchmark/attempts.py:1928-1969` appends one fresh all-cell preflight and serializes every slot under one writer. Per-cell plans or repeated commands would break the SDD S04-S08 same-condition and failure-preservation invariants. + +#### Solution + +After the exact external preconditions are available, run the repository CLI once and save only the emitted canonical run id to deterministic task evidence. Let the CLI create all dynamic run/attempt artifacts; do not edit them. On exit 69 or any nonzero attempt result, retain the run id and public status, do not invoke `run` or `resume` again, and record the blocker in the review stub. + +No before/after code snippet applies because this item changes no source: it performs one stateful CLI execution. The state transition is “no canonical C01-C09 run pointer” → “one `run_id.log` pointing to the append-only C01-C09 run and a public status proving its terminal result.” + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one CLI-emitted `run-...` id; never fabricate or replace it. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: paste verbatim stdout/stderr and record deviations/design decisions without secrets. +- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02//` artifacts or `../iop-s2`. + +#### Test Strategy + +No test file is added because there is no source/API change. Fresh existing unit/contract tests verify the harness, and the actual one-shot run plus public status supplies the provider/caller acceptance evidence unavailable to mocks. + +#### Verification + +Run the external-precondition command in Final Verification step 2. Only when it passes, run the one-time execution command in Final Verification step 3 exactly once. Expected success is exit 0, `completed=9`, `unresolved=0`, `success=9`, every other terminal/nonterminal count 0, and one valid `run_id.log`; any other result is preserved blocker evidence, not a retry trigger. + +## Modified Files Summary + +| File | Item | +|------|------| +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1 | + +## Final Verification + +1. Validate the immutable manifest and run fresh repository-native tests. Expected: manifest valid, 91 tests pass. + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test +``` + +2. Prove the fixed checkout/tool/environment assumptions without printing secret or endpoint values. Expected: Linux/AArch64, exact clean testbed branch/HEAD, all binaries found, and every boolean `True`. If the artifact has not been rebuilt from this HEAD, stop before step 3 and record the predecessor blocker and resume condition. + +```bash +set -eu +test "$(uname -s)" = Linux +test "$(uname -m)" = aarch64 +test "$(git -C ../iop-s2 branch --show-current)" = dev +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 +test -z "$(git -C ../iop-s2 status --porcelain=v1)" +for tool in python3 claude agy codex git; do command -v "$tool"; done +python3 - <<'PY' +import os + +ok = True +for caller in ("CLAUDE", "AGY", "CODEX"): + base = bool(os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")) + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + secret = bool(ref and os.environ.get(ref)) + print(f"{caller}: base_reference={base} secret_reference={bool(ref)} referenced_secret={secret}") + ok = ok and base and bool(ref) and secret +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +config = bool(config_ref and os.environ.get(config_ref)) +print(f"CONFIG: reference={bool(config_ref)} referenced_value={config}") +ok = ok and bool(config_ref) and config +raise SystemExit(0 if ok else 69) +PY +``` + +3. Execute the nine cells exactly once, preserve stdout/stderr, save the emitted run id, and inspect that same run. This command is implementation-only and non-repeatable; the review agent verifies step 4 instead of rerunning it. Expected success: exit 0, `completed=9 unresolved=0 success=9 failed=0 timed_out=0 cancelled=0 interrupted=0 running=0`, followed by valid status. + +```bash +set -eu +bench_stdout="$(mktemp)" +bench_stderr="$(mktemp)" +trap 'rm -f "$bench_stdout" "$bench_stderr"' EXIT +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$bench_stdout" 2>"$bench_stderr" +bench_exit=$? +set -e +printf 'exit=%s\n' "$bench_exit" +printf '%s\n' '--- stdout ---' +cat "$bench_stdout" +printf '%s\n' '--- stderr ---' +cat "$bench_stderr" +bench_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$bench_stdout" "$bench_stderr" | sed -n '1p')" +test -n "$bench_run_id" +printf '%s\n' "$bench_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" +test "$bench_exit" -eq 0 +``` + +4. Revalidate the recorded run without provider re-execution and check the worktree. Expected: one well-formed id, public status with success 9 and all other counts 0, and `git diff --check` passes. The active PLAN/review pair and `run_id.log` are intentional changes. + +```bash +set -eu +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" +git diff --check +``` + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_1.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_1.log new file mode 100644 index 00000000..0504d869 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_1.log @@ -0,0 +1,264 @@ + + +# Plan - C01-C09 원샷 비교 실행 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 아래 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다. + +## Background + +승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행하도록 정의한다. 이 generation은 최초 계획의 두 semantic defect를 바로잡는다. scored 실행 전에 current-HEAD Edge/Node artifact와 caller capability를 모두 증명하고, 실제 제출이 9개 모두 terminal이면 성공뿐 아니라 보존된 실패·timeout·cancel도 SDD에 맞는 비교 실행 evidence로 인정한다. 외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아닌 하나의 indivisible execution slice다. + +## Archive Evidence Snapshot + +- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 readiness 확인 절차가 exact blocker를 보존했다는 뜻이며 `route-readiness` 완료 선언이 아니다. +- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`에서 Edge/Node artifact freshness, Linux AArch64 identity, caller version/help, environment reference를 모두 통과한 뒤에만 live benchmark를 시작하도록 요구한다. +- 최초 active pair는 공식 verdict 없이 `plan_cloud_G09_0.log`와 `code_review_cloud_G10_0.log`로 보존했다. 그 pair의 success-only acceptance와 불완전한 artifact/caller preflight는 이 generation이 대체하며 구현자는 이전 pair를 다시 읽지 않는다. + +## Analysis + +### Files Read + +- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md` +- Runner/fixture: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- Tests/rules: `scripts/agent_benchmark/attempts_test.py`, `scripts/agent_benchmark/skill_contract_test.py`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- Exact prior evidence: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`, its directly connected `plan_cloud_G09_0.log` and `code_review_cloud_G09_0.log`, and the current task's archived generation-0 pair. + +### SDD Criteria + +- SDD는 `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`이며 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다. +- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다. Evidence Map은 각 cell의 event/timing/usage/workspace evidence와 hybrid stage/terminal evidence를 요구한다. +- D10과 state invariant는 scored failure를 보존하고 성공 결과만 골라 대표하지 않도록 요구한다. 따라서 preflight 차단이나 1~8개 partial attempt는 미완료지만, fresh run에서 C01-C09 각각 정확히 한 번 제출되어 9개 모두 `success|failed|timed_out|cancelled` terminal이면 이 execution Epic의 evidence는 완결된다. 실패가 있더라도 `resume --retry-failed`를 호출하지 않는다. +- Scoring/report evidence인 S09-S12는 이 Epic 범위가 아니다. + +### Verification Context + +- 별도 handoff는 없었다. repository-native CLI, manifest, SDD, spec/contracts, focused tests와 exact predecessor evidence에서 실행 계약을 재구성했다. +- 2026-08-12 self-review에서 manifest validate는 `ok: manifest is valid`, focused unit/contract suite는 `Ran 91 tests ... OK`, `git diff --check`는 출력 없이 통과했다. Cached output은 구현·review evidence로 재사용하지 않는다. +- 같은 self-review의 secret-safe preflight snapshot은 local Linux/AArch64, exact clean testbed branch/HEAD, `iop-edge` build metadata가 exact HEAD/clean, `iop-node`가 다른 revision의 dirty build, agy observed `1.1.12`/adapter-known `1.1.11`/IOP transport unsupported, 세 caller와 config observation environment reference 모두 absent였다. 현재는 scored run을 시작할 수 없다. +- `agent-test/runs/bench-02/run-20260811T210154Z-925c0f26af88/`은 기준 커밋 뒤 생성된 preparation-only preflight state다. public status는 모든 attempt count 0이고 preflight는 `implementation_gap=9`다. Append-only evidence로 보존하되 readiness나 C01-C09 실행 evidence로 재사용하지 않는다. +- 제약: secret 값과 raw private endpoint/config payload를 출력하지 않고, `../iop-s2`는 clean/read-only fixture source로 유지하며, dynamic run state는 CLI가 검증한 `agent-test/runs/bench-02//` 아래에만 둔다. `run_id.log`는 그 canonical state를 가리키는 task-local pointer일 뿐 run data를 복제하지 않는다. +- Confidence는 high다. 현재 blocker와 실행/terminal 분류가 deterministic하며 provider 호출은 fixed gate가 모두 통과한 뒤 public `run` 한 번으로만 발생한다. + +#### External Verification Preflight + +- Runner/repo: local Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, source HEAD `e86113f0ae3faecf8dfe715990e4755cdc42bedf`. +- Testbed: `/config/workspace/iop-s2`, branch `dev`, fixed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean/read-only. +- Required artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`; both must be executable current-HEAD Linux AArch64 ELF artifacts and execute their safe help/version probes. +- Required callers: `claude`, `agy`, `codex`; agy must equal the production adapter's `AGY_KNOWN_VERSION` and expose its IOP transport help contract. +- Required environment: each caller's base-url reference, secret environment-name reference and dereferenced non-empty secret, plus valid config-observation JSON reference. Commands print only booleans or fixed success text. +- Ports/process/external hosts are observed by the public all-cell preflight inside `run`; raw values remain out of task evidence. + +### Test Coverage Gaps + +- Source behavior change는 없다. Existing tests는 run 생성, single writer, fresh preflight, terminal state, retry 보존과 skill command contract를 다룬다. +- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, workspace/web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 실제 C01-C09 run 대상이다. +- 새 test는 추가하지 않는다. Fresh focused tests와 actual one-shot run/public status가 이 slice의 acceptance oracle이다. + +### Symbol References + +None. Rename/remove/change하는 symbol이 없다. + +### Split Judgment + +- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가진다. +- 다섯 task id는 fixed seed, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유하므로 하나의 plan으로 실행한다. +- 디렉터리 predecessor index `05`는 archived `05+04_readiness_preflight/complete.log`로 task-protocol상 충족한다. 그 로그가 보존한 live readiness blocker는 TEST-1의 fail-closed precondition이며 아직 해소되지 않았다. + +### Scope Rationale + +- `scripts/**`, `agent-spec/**`, `agent-contract/**`, SDD/Milestone 문서는 변경하지 않는다. 이 Epic은 준비된 harness의 actual run만 소유한다. +- `score`, evaluator, report, S09-S12는 후속 Epic 범위다. +- Caller/model/effort/preset 대체, extra repetition, manual edit, `resume`, retry, testbed source/config mutation은 제외한다. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh` mode=`pair`. +- Build: `scope/context/verification/evidence/ownership/decision=true`, capability gap=none, grade scores `1/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G09`, filename=`PLAN-cloud-G09.md`, catalog route=`worker/cloud/G09`. +- Review: `scope/context/verification/evidence/ownership/decision=true`, capability gap=none, grade scores `2/2/2/2/2`, route basis=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog route=`review/cloud/G10`. +- `large_indivisible_context=false`; positive loop risks=`temporal_state,concurrent_consistency,boundary_contract,variant_product` (4); `review_rework_count=0`; `evidence_integrity_failure=false`. + +## Dependencies and Execution Order + +1. `06+05_comparison_runs`의 runtime dependency는 predecessor `05`이며 exact archived `complete.log`가 task-protocol dependency를 충족한다. +2. TEST-1의 fixed gate가 current-HEAD artifacts, caller capability와 environment references를 모두 통과해야 scored `run`을 시작할 수 있다. 현재 blocker가 하나라도 남으면 run을 호출하지 않고 active pair를 유지한다. +3. Gate 통과 뒤 public `run`을 정확히 한 번 호출한다. 그 fresh run의 9개 terminal attempt를 확인한 뒤에만 TEST-1을 완료로 표시한다. + +## Implementation Checklist + +- [ ] TEST-1: Prove every fixed external precondition, invoke the immutable C01-C09 `run` exactly once, preserve its canonical run id and verbatim CLI result, and accept only either nine success results or nine retained non-interrupted terminal results; never retry, resume, substitute, or treat preflight/partial execution as completion. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Immutable C01-C09 comparison run + +#### Problem + +The generation-0 plan omitted the predecessor's full Edge/Node artifact and agy capability gate, so it could allocate a blocked run against stale or unsupported runtime inputs. It also required `success=9` even though SDD D10 requires retained scored failures to remain valid comparison evidence. `scripts/agent_comparison_benchmark.py` returns exit 69 for both preflight blockers and retained failures, so acceptance must distinguish zero/partial attempts from nine terminal attempts without rerunning. + +#### Solution + +Run the complete fixed precondition command first. Only after it exits 0, snapshot existing run roots and invoke the public CLI exactly once. Capture its stdout/stderr and exit code verbatim, resolve the new run id from the CLI output or the one exact newly created run root, and write only that id to `run_id.log`. + +Use the public read-only `status` command on that exact run. A preflight blocker with zero attempts or any partial/running/interrupted state leaves TEST-1 unchecked and records a resume condition. Exactly nine attempts in `success|failed|timed_out|cancelled`, with zero `running` and `interrupted`, completes the execution slice; exit 0 must correspond to nine successes and exit 69 to retained non-success terminals. Do not invoke `run` or `resume` again. + +No before/after code snippet applies because this item changes no source. The durable transition is no scored run → one append-only C01-C09 run plus one task-local canonical pointer. + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one resolved `run-...` id; never fabricate or replace it. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim command output, classification, deviations and blocker/resume evidence without secrets. +- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02//` artifacts or `../iop-s2`. + +#### Test Strategy + +No test file is added because there is no source/API change. Fresh existing unit/contract tests verify the harness; the actual one-shot run and public status supply the external acceptance evidence unavailable to mocks. + +#### Verification + +Run Final Verification steps 1 and 2. Only if step 2 passes, run step 3 exactly once. Mark TEST-1 complete only when step 3 classifies `execution_complete`; otherwise keep it pending with the exact blocker and resume condition. The review agent reruns steps 1, 2 and 4 only, never step 3. + +## Modified Files Summary + +| File | Item | +|------|------| +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1 | + +## Final Verification + +1. Validate the immutable manifest and run fresh focused tests. Expected: manifest valid; 91 tests pass. + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test +``` + +2. Prove the fixed testbed, artifact, caller and secret-reference assumptions without printing secret or endpoint values. Expected: every command succeeds. If any check fails, stop before step 3. + +```bash +set -euo pipefail +blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; } +test "$(uname -s)" = Linux || blocked "benchmark host must be Linux" +test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64" +test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev" +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed" +test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean" +command -v readelf >/dev/null || blocked "readelf must be installed" +command -v go >/dev/null || blocked "go must be installed" +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test -x "$binary" || blocked "$binary must be executable" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD" +import subprocess, sys +expected = sys.argv[1] +for binary in sys.argv[2:]: + output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout + build = {} + for line in output.splitlines(): + fields = line.strip().split("\t", 1) + if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]: + key, value = fields[1].split("=", 1) + build[key] = value + assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision")) + assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified")) + assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH")) +print("ok: Edge/Node build identities match the clean testbed HEAD") +PY +../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute" +test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute" +for tool in python3 claude agy codex git; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done +python3 - <<'PY' || blocked "agy must match the adapter-known version and transport contract" +import subprocess +from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability +version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True) +help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True) +capability = inspect_agy_iop_capability((version_run.stdout + version_run.stderr).strip(), help_run.stdout + help_run.stderr) +assert capability.version == AGY_KNOWN_VERSION +assert capability.iop_transport_supported +print("ok: agy IOP transport capability") +PY +python3 - <<'PY' || blocked "benchmark environment references must be present" +import json, os, re +name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") +for caller in ("CLAUDE", "AGY", "CODEX"): + assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL") + ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "") + assert name.fullmatch(ref) and os.environ.get(ref) +config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "") +assert name.fullmatch(config_ref) and os.environ.get(config_ref) +value = json.loads(os.environ[config_ref]) +assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list) +print("ok: benchmark environment references") +PY +``` + +3. Execute the nine cells exactly once. This block is implementation-only and non-repeatable. Expected task completion is `classification=execution_complete`; the recorded benchmark exit remains 0 for all-success or 69 for retained failures. + +```bash +set -euo pipefail +run_root=agent-test/runs/bench-02 +before="$(mktemp)"; after="$(mktemp)"; new_runs="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)" +trap 'rm -f "$before" "$after" "$new_runs" "$out" "$err"' EXIT +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before" +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err" +bench_exit=$? +set -e +printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out" +printf '%s\n' 'stderr:'; cat "$err" +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' | sort >"$after" +comm -13 "$before" "$after" >"$new_runs" +test "$(wc -l <"$new_runs")" -eq 1 +discovered_run_id="$(cat "$new_runs")" +emitted_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$out" "$err" | sed -n '1p')" +if test -n "$emitted_run_id"; then test "$emitted_run_id" = "$discovered_run_id"; fi +printf '%s\n' "$discovered_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$discovered_run_id")" +printf 'status_stdout:\n%s\n' "$status_output" +BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"} +assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values()) +bench_exit = int(os.environ["BENCH_EXIT"]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0: + assert bench_exit == (0 if states["success"] == 9 else 69) + print(f"classification=execution_complete states={states}") + raise SystemExit(0) +if sum(states.values()) == 0 and bench_exit == 69: + print(f"classification=blocked_preflight states={states}") +else: + print(f"classification=blocked_partial states={states}") +raise SystemExit(69) +PY +``` + +4. Revalidate only the recorded run and worktree; do not invoke a provider. Expected after completed execution: one valid id, public status with nine accepted terminals and zero interrupted/running, `git diff --check` success. + +```bash +set -euo pipefail +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")" +printf '%s\n' "$status_output" +STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states +print(f"ok: nine retained terminal attempts states={states}") +PY +git diff --check +``` + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_2.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_2.log new file mode 100644 index 00000000..8a1a3d33 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_2.log @@ -0,0 +1,248 @@ + + +# Plan - C01-C09 원샷 비교 실행 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 검증 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다. + +## Background + +승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행한다. checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`의 generation 0 의도인 “fresh preflight 뒤 public run 한 번, retry/resume 없음, 한 run의 아홉 cell 보존”을 유지한다. Generation 1의 caller 직접 호출·preflight 정책 중복과 global run-directory 차집합 의존을 제거한다. Caller capability, environment, live route readiness는 public `run`의 fresh all-cell preflight만 판정하며, CLI가 방출한 run id만 canonical pointer로 채택한다. + +외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아닌 하나의 indivisible execution slice다. Scored failure도 SDD D10에 따라 보존하므로 아홉 cell이 모두 non-interrupted terminal이면 이 Epic의 실행 evidence는 완결된다. + +## Archive Evidence Snapshot + +- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 exact `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 blocker를 보존한 readiness 절차의 완료이지 Milestone `route-readiness` 완료 선언이 아니다. +- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 current-HEAD Edge/Node artifact를 요구한다. 이 dependency는 TEST-1에서 fail-closed로 다시 확인한다. +- Generation 0은 `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`, generation 1은 `plan_cloud_G09_1.log` / `code_review_cloud_G10_1.log`로 공식 verdict 없이 보존됐다. Generation 0의 atomic intent는 유지하고 success-only acceptance를 바로잡았으며, generation 1의 직접 caller probe와 concurrent run 오인 가능성을 이 generation이 대체한다. 구현자는 이 plan과 active review만 실행하며 archive를 다시 읽지 않는다. + +## Analysis + +### Files Read + +- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md` +- Runner/fixture: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/workspace.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg` +- Rules/skills: `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- Exact prior evidence: predecessor `complete.log`와 직접 연결된 plan/review log, 그리고 current task의 generation 0/1 pair + +### SDD Criteria and Task Union + +- SDD 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다. +- 이 pair의 Milestone task union은 정확히 `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다. S04는 C01, S05는 C02-C03, S06은 C04-C05, S07은 C06-C07, S08은 C08-C09에 대응한다. 다른 Milestone task는 소유하지 않는다. +- 각 cell은 event/timing/usage/workspace evidence를 보존하고 hybrid cell은 stage/terminal evidence도 보존해야 한다. +- D10과 state invariant는 scored failure를 보존하고 성공 결과만 선별하지 않도록 요구한다. 따라서 preflight 차단이나 1~8개 partial attempt는 미완료지만, fresh run에서 C01-C09가 각각 정확히 한 번 제출되어 9개 모두 `success|failed|timed_out|cancelled`이면 실행 Epic evidence는 완결된다. `resume --retry-failed`는 호출하지 않는다. +- S09-S12의 score/evaluator/report evidence는 이 Epic 범위가 아니다. + +### Verification Context + +- 별도 handoff는 없었다. repository-native CLI, manifest, SDD, living spec/contracts, current source와 exact predecessor evidence에서 실행 계약을 재구성했다. +- 현재 repository HEAD는 checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`다. 제품 source는 직전 `e86113f0ae3faecf8dfe715990e4755cdc42bedf` 이후 변경되지 않았고 checkpoint에는 task pair만 추가됐다. +- 2026-08-12 read-only audit에서 `../iop-s2`는 clean `dev`의 fixed HEAD이고 `iop-edge` metadata는 그 exact HEAD/clean과 일치하지만, `iop-node` metadata는 revision `9b2f...`, `vcs.modified=true`로 stale다. 현재는 step 2가 fail-closed해야 하며 scored `run`을 호출하면 안 된다. +- Generic dev runner 경로보다 SDD/immutable manifest의 task-specific local sibling `../iop-s2`가 이 benchmark의 고정 testbed 계약을 우선한다. Testbed source/config는 read-only다. +- Caller/provider를 public CLI 밖에서 실행하지 않는다. `command -v`는 설치 여부만 확인하고 caller version/help, secret dereference, endpoint/process/config observation과 live connectivity는 public `run`의 fresh all-cell preflight가 단독 소유한다. +- Secret 값과 raw private endpoint/config payload를 출력하지 않는다. Dynamic state는 CLI가 검증한 `agent-test/runs/bench-02//` 아래에만 쓰고, `run_id.log`는 그 exact state를 가리키는 task-local pointer다. + +### Test Coverage Gaps + +- Source behavior change는 없다. Repository full benchmark suite는 manifest, run allocation, single writer, fresh preflight, terminal state, retry 보존, caller registry와 skill command contract를 검증한다. +- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, workspace/web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 C01-C09 run 대상이다. +- 새 test는 추가하지 않는다. Fresh full benchmark suite와 actual one-shot run/public status가 acceptance oracle이다. + +### Symbol References + +None. Rename/remove/change하는 symbol이 없다. + +### Split Judgment + +- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가진다. +- 다섯 task id는 fixed seed, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유하므로 하나의 plan으로 실행한다. +- 디렉터리 predecessor index `05`는 archived `05+04_readiness_preflight/complete.log`로 task-protocol상 충족한다. 보존된 readiness blocker는 TEST-1의 fail-closed precondition이며 아직 해소되지 않았다. + +### Scope and Write Ownership + +- Source/spec/contract/SDD/Milestone는 변경하지 않는다. Caller/model/effort/preset 대체, extra repetition, manual run-state edit, `resume`, retry, score, evaluator, report, testbed rebuild/source/config mutation은 제외한다. +- Implementer-owned write: task-local `run_id.log`와 active `CODE_REVIEW-cloud-G10.md`의 implementation-owned evidence/checklist sections. +- CLI-owned dynamic write: public `run`이 방출한 exact `agent-test/runs/bench-02//**`. Implementer는 이 tree를 직접 편집하지 않는다. +- Review-owned write/archive는 review verdict 이후 절차이며 이 implementation slice가 실행하지 않는다. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh`, mode=`pair`, status=`routed`. +- Build closures는 모두 true, capability gap은 none, grade scores=`1/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G09`, filename=`PLAN-cloud-G09.md`, catalog route=`worker/cloud/G09`. +- Review closures는 모두 true, capability gap은 none, grade scores=`2/2/2/2/2`, route basis=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog route=`review/cloud/G10`. +- `large_indivisible_context=false`; matched risks=`temporal_state,concurrent_consistency,boundary_contract,variant_product` (4); `review_rework_count=0`; `evidence_integrity_failure=false`. Grade boundary가 build cloud route를 결정한다. + +## Dependencies and Execution Order + +1. Active index `06+05`의 predecessor는 archived index `05+04`이고 exact `complete.log`가 task-protocol dependency를 충족한다. +2. Step 2가 host/testbed/artifact identity와 tool presence를 모두 통과해야 step 3을 시작할 수 있다. 현재 stale `iop-node`가 남아 있으면 stop하고 TEST-1을 unchecked로 유지한다. +3. Step 3의 public `run`은 caller/environment/live readiness를 fresh all-cell preflight로 판정하고, gate 통과 시 C01-C09를 동일 run에서 각각 한 번 제출한다. 이 명령은 이 plan에서 정확히 한 번만 실행한다. +4. CLI가 한 canonical run id를 방출하고 exact run root가 새로 생긴 경우에만 pointer를 쓴다. Id가 없거나 ambiguous하면 global directory 차집합으로 추측하지 않고 `blocked_unidentified`로 중단하며 이 plan에서 rerun하지 않는다. +5. Public status가 아홉 non-interrupted terminal attempt를 증명한 뒤에만 TEST-1을 완료로 표시한다. + +## Implementation Checklist + +- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Immutable C01-C09 comparison run + +#### Problem + +Generation 1 directly invoked `agy --version/--help`, dereferenced benchmark environment references, and duplicated policy already owned by the public all-cell preflight. It also inferred identity from “exactly one new global run directory,” which can reject the correct non-repeatable run when another actor concurrently creates an unrelated run. These defects violate the benchmark skill boundary and weaken exactly-once evidence. + +#### Solution + +Use a pre-run gate only for task-specific host/testbed/artifact identity and command presence. Let the public `run` exclusively invoke callers and assess capability, secrets, endpoints, processes, config observation and connectivity. Invoke it once, preserve both streams and exit code, and extract exactly one unique `run-...` id only from CLI output. Confirm that id was absent before the command and its exact directory now exists; unrelated concurrent run roots are ignored. + +Write `run_id.log` only after those checks. Use public read-only `status` on that exact id. Zero attempts, partial attempts, `running`, `interrupted`, invalid state, or an unidentified run leave TEST-1 unchecked with an exact resume condition. Exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted` complete this execution slice. Exit 0 must mean nine successes; retained non-success terminal results require exit 69. Never invoke `run` or `resume` again under this plan. + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one CLI-emitted canonical `run-...` id; never infer, fabricate, or replace it. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim command output, classification, deviations and blocker/resume evidence without secrets. +- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02//**` or anything under `../iop-s2`. + +#### Test Strategy + +No test file is added because there is no source/API change. Fresh repository benchmark tests verify the harness. The actual one-shot run and public status supply the external acceptance evidence unavailable to mocks. + +#### Verification + +Run Final Verification steps 1 and 2. Only if step 2 passes, run step 3 exactly once. Mark TEST-1 complete only when step 3 prints `classification=execution_complete`; otherwise retain unchecked items and exact blocker/resume evidence. The review agent reruns steps 1, 2 and 4 only, never step 3. + +## Modified Files Summary + +| Path | Owner | Item | +|------|-------|------| +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | Implementer | TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | Implementer evidence fields | TEST-1 | +| `agent-test/runs/bench-02//**` | Public benchmark CLI | TEST-1 | + +## Final Verification + +1. Validate the immutable manifest and run the full repository benchmark suite. Expected: manifest valid and all discovered benchmark tests pass. + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +``` + +2. Prove only the fixed host/testbed/artifact and command-presence assumptions. This command does not invoke a caller/provider or reproduce public live-preflight policy. Expected: every check succeeds. If any check fails, stop before step 3. + +```bash +set -euo pipefail +blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; } +test "$(uname -s)" = Linux || blocked "benchmark host must be Linux" +test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64" +test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev" +test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed" +test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean" +for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done +testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" +for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do + test -x "$binary" || blocked "$binary must be executable" + readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact" +done +python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD" +import subprocess, sys +expected = sys.argv[1] +for binary in sys.argv[2:]: + output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout + build = {} + for line in output.splitlines(): + fields = line.strip().split("\t", 1) + if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]: + key, value = fields[1].split("=", 1) + build[key] = value + assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision")) + assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified")) + assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH")) +print("ok: Edge/Node build identities match the clean testbed HEAD") +PY +``` + +3. Execute the nine cells exactly once. This block is implementation-only and non-repeatable. Expected task completion is `classification=execution_complete`; the benchmark exit remains 0 for all-success or 69 for retained failures. + +```bash +set -euo pipefail +task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs +run_root=agent-test/runs/bench-02 +test ! -e "$task_dir/run_id.log" +before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)" +trap 'rm -f "$before" "$out" "$err"' EXIT +find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before" +set +e +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err" +bench_exit=$? +set -e +printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out" +printf '%s\n' 'stderr:'; cat "$err" +emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)" +emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)" +if test "$emitted_count" -ne 1; then + printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count" + exit 69 +fi +bench_run_id="$(printf '%s\n' "$emitted_ids")" +if grep -Fxq "$bench_run_id" "$before"; then + printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id" + exit 69 +fi +test -d "$run_root/$bench_run_id" +printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log" +set +e +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)" +status_exit=$? +set -e +printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output" +if test "$status_exit" -ne 0; then + printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id" + exit 69 +fi +BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"} +assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values()) +bench_exit = int(os.environ["BENCH_EXIT"]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0: + assert bench_exit == (0 if states["success"] == 9 else 69) + print(f"classification=execution_complete states={states}") + raise SystemExit(0) +if sum(states.values()) == 0 and bench_exit == 69: + print(f"classification=blocked_preflight states={states}") +else: + print(f"classification=blocked_partial states={states}") +raise SystemExit(69) +PY +``` + +4. Re-read the recorded run without invoking a provider and check both worktrees. Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed, no whitespace errors. + +```bash +set -euo pipefail +pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log +test "$(wc -l <"$pointer")" -eq 1 +bench_run_id="$(tr -d '\n' <"$pointer")" +printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")" +printf '%s\n' "$status_output" +STATUS_OUTPUT="$status_output" python3 - <<'PY' +import ast, os +text = os.environ["STATUS_OUTPUT"] +assert text.startswith("ok: "), text +states = ast.literal_eval(text[4:]) +accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled")) +assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states +print(f"ok: nine retained terminal attempts states={states}") +PY +test -z "$(git -C ../iop-s2 status --porcelain=v1)" +git diff --check +``` diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log new file mode 100644 index 00000000..89d4a057 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log @@ -0,0 +1,294 @@ + + +# Plan - official agy IOP 호환, managed dev 배포와 C01-C09 실행 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 설계와 검증을 그대로 구현하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행한 명령/출력과 재개 조건만 evidence 필드에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다. + +## Background + +기존 active plan은 이미 준비됐다고 가정하고 C01-C09만 실행하도록 했으나, official `agy 1.1.12`의 실제 Gemini API-key transport와 dev managed credential runtime이 그 전제와 맞지 않았다. 실제 CLI는 Gemini-native `streamGenerateContent`, `x-goog-api-key`, `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`과 `event=init|step_update|result` stream JSON을 사용하며 API-key mode에서 `--effort`를 거부한다. Edge에는 이 ingress가 없고 dev는 legacy credential mode라 marked hybrid preset admission도 불가능하다. 이 plan은 caller→Edge→preset/provider→official lifecycle 전체가 한 번의 실제 preflight에서 닫힌 뒤에만 immutable 9-cell run을 시작한다. + +## Archive Evidence Snapshot + +- `plan_cloud_G09_0.log`/`code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`/`code_review_cloud_G10_1.log`, `plan_cloud_G09_2.log`/`code_review_cloud_G10_2.log`는 공식 verdict 없이 종료된 이전 실행 계획이다. 그중 plan=2의 one-run 원칙은 API-4에 유지하지만 “source/contract/dev 변경 없음” 전제는 폐기한다. +- predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 index `05` 의존성을 충족하지만 live blocker를 완료로 바꾸지 않는다. +- remote dev checkout은 `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이며 runtime config는 `build/dev-runtime/edge.yaml`, `control-plane.yaml`, `node-*.yaml`이다. public Edge는 현재 legacy HTTP `:18083`이다. + +## Analysis + +### Files Read + +- Rules: `agent-ops/rules/project/domain/control-plane/rules.md`, `agent-ops/rules/project/domain/edge/rules.md`, `agent-ops/rules/project/domain/node/rules.md`, `agent-ops/rules/project/domain/platform-common/rules.md`, `agent-test/local/rules.md`, `agent-test/dev/rules.md` +- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- Contracts/docs: `agent-contract/index.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`, `docs/edge-local-dev-guide.md` +- Edge: `apps/edge/internal/openai/routes.go`, `principal.go`, `server.go`, `route_resolution.go`, `dispatch_context.go`, `chat_handler.go`, `provider_tunnel.go`, `anthropic_bridge.go`, `single_request_preset_binding.go`, `apps/edge/internal/service/provider_tunnel.go` +- Config/Node/Control Plane: `packages/go/config/protocol_profile.go`, `edge_types.go`, `node_types.go`, `validate.go`, `credential_plane_config_test.go`, `apps/node/internal/node/tunnel_handler.go`, `apps/control-plane/cmd/control-plane/main.go`, `credential_commands.go`, `credential_http_handlers.go`, `secure_delivery_integration_test.go`, `configs/control-plane.yaml`, `scripts/e2e-credential-slot-smoke.sh` +- Benchmark: `scripts/agent_benchmark/agy_iop.py`, `agy_iop_test.py`, `live_iop.py`, `manifest.py`, `lifecycle.py`, `scripts/agent_comparison_benchmark.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` +- Active/prior evidence: current task plan=0/1/2 logs and active pair, exact predecessor `complete.log` + +### SDD Criteria + +- SDD는 `[승인됨]`, 잠금 `해제`다. 이 pair의 first-line task ids는 `agy-iop-compatibility`, `managed-credential-dev`, `route-readiness`, `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다. +- S13은 official agy 1.1.12의 route별 Gemini base URL, IOP principal auth, Gemini request/tool/SSE와 실제 stream-json lifecycle을 요구하므로 API-1/API-2를 정한다. +- S14는 CA-signed mTLS, Edge HTTPS, projection, slot-route와 sealed lease/no-fallback evidence를 요구하므로 API-3을 정한다. +- S02는 모든 caller의 auth/route/effort/terminal live preflight를 요구하고, S04-S08은 C01-C09 one-submission evidence를 요구하므로 API-3 통과 후에만 API-4를 실행한다. + +### Verification Context + +- handoff는 없으며 repository source, contracts, official CLI 실행과 remote read-only preflight를 직접 확인했다. +- official `agy 1.1.12` known model은 `Gemini 3.6 Flash`다. API-key call은 `POST /v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse`, `x-goog-api-key`와 `contents/generationConfig/systemInstruction/toolConfig/tools`를 보냈다. `GOOGLE_GEMINI_BASE_URL` invalid endpoint가 network failure를 만들었고 `GEMINI_BASE_URL`은 override가 아니었다. +- official stream JSON은 `event=init`, `event=step_update`와 `event=result`; result는 `status`, `duration_seconds`, `num_turns`, `usage(input_tokens/output_tokens/thinking_tokens/cache_read_tokens/total_tokens)`를 가진다. 합성 `iop effective_binding`, metric, `system idle` event는 없다. +- remote dev는 clean release checkout과 `build/dev-runtime` config/artifacts를 갖지만 `credential_plane`/TLS material이 없고 legacy static provider credential이 남아 있다. 모든 raw provider token은 workspace `token/.gemini`, `.claude`, `.gpt`에서만 읽어 credential HTTPS stdin으로 등록하며 출력·명령 인자·tracked file에 넣지 않는다. +- external target: benchmark host Linux/AArch64의 current checkout, Mac dev Edge/Control Plane/local Node, inventory의 GX10/OneXPlayer/RTX5090 Nodes. Source sync→fresh build→config check→bounded restart→health/identity→direct/hybrid smoke 순으로 진행한다. + +### Test Coverage Gaps + +- Gemini-native ingress/auth/body conversion/SSE projection은 기존 test가 없다. normal text, tool, usage, invalid path/body, conflicting auth, route denial, stream error/cancel regression을 새 Go test로 작성한다. +- agy adapter test는 fake 1.1.11 help와 합성 events라 실제 1.1.12를 가리지 못한다. official help/event fixtures와 exact invocation env/argv tests로 교체하고 live registry integration test를 갱신한다. +- managed credential code 자체는 deterministic full-cycle smoke가 있다. gap은 실제 dev security material, slot/route와 multi-node rollout이므로 config check와 live secret-safe smoke가 필요하다. +- C01-C09 결과는 unit test로 대체할 수 없다. API-4의 public run/status가 유일한 scored evidence다. + +### Symbol References + +삭제·rename symbol은 없다. 새 Gemini handler/bridge symbol은 `routes.go` 한 곳에서 등록되고, benchmark 상수 변경 call site는 `agy_iop.py`, `agy_iop_test.py`, `live_iop.py`와 connectivity integration tests다. + +### Split Judgment + +- API translator, official event parser, managed route authorization과 actual caller smoke는 각각 독립 test가 있지만, 이 task의 correctness 조건은 `GEMINI_API_KEY`가 upstream key가 아닌 projected IOP token이고 route-specific base가 direct/hybrid binding으로 연결된다는 하나의 cross-boundary invariant다. +- API-4는 그 invariant가 live에서 닫힌 후 exactly once여야 하므로 별도 활성 sibling으로 이동하면 기존 fixed index와 non-repeatable state coordination을 분리할 수 없다. 하나의 순차 plan으로 유지한다. +- index `05` predecessor는 archived `complete.log`로 충족됐다. + +### Scope Rationale + +- Gemini non-stream/batch/files/cache/tuning API, arbitrary third-party Gemini SDK, unofficial agy custom model, new protocol-profile driver, pricing과 report/scoring task는 제외한다. +- 기존 OpenAI/Anthropic caller schema, preset stage semantics와 provider OpenAI-compatible upstream은 변경하지 않는다. Gemini ingress는 이를 재사용하는 outer translation만 소유한다. +- remote dev config와 operator secret material은 repository 밖 external state다. raw token/cert private key/slot alias/lease id는 plan/review/run evidence에 쓰지 않는다. +- scored run 전 preflight가 실패하면 API-4는 실행하지 않는다. 한 번 시작한 public run은 retry/resume하지 않는다. + +### Final Routing + +- `evaluation_mode=first-pass`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`. +- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`. +- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`. +- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`. + +## Dependencies and Execution Order + +1. API-1 Go tests와 API-2 Python tests/full local suite가 통과해야 API-3 remote build/deploy를 시작한다. +2. API-3는 current source의 fresh artifacts, managed config validation, all runtime health, official agy direct/hybrid smoke와 all-cell public preflight가 모두 통과해야 완료다. +3. API-3 완료 전 API-4 public `run`을 호출하지 않는다. API-4는 exactly once이며 CLI-emitted run id만 보존한다. + +## Implementation Checklist + +- [x] API-1: Implement the route-qualified Gemini `streamGenerateContent` ingress, `x-goog-api-key` principal authentication, bounded Gemini↔Chat/tool/SSE translation, caller-safe errors and full Edge regression tests without bypassing existing route/preset admission. +- [x] API-2: Replace the invented agy 1.1.11/OpenAI env/event contract with the official agy 1.1.12 Gemini API-key invocation and stream-json parser, config-owned effective binding evidence, exact redaction and updated unit/integration fixtures. +- [x] API-3: Build and deploy fresh artifacts to dev, provision the complete operator-owned managed credential/TLS composition and projected provider routes, then pass deterministic credential qualification plus official agy direct/hybrid and all-caller live preflight without secret leakage or legacy fallback. +- [ ] API-4: Invoke the immutable public C01-C09 run exactly once only after API-3, preserve its CLI-emitted canonical run id and all nine non-interrupted terminal attempts, and never retry, resume, substitute, or manually edit run state. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [API-1] Gemini-native Edge ingress + +#### Problem + +`apps/edge/internal/openai/routes.go:11-19` registers only OpenAI, Anthropic and Ollama surfaces. `principal.go:164-183` understands bearer and Anthropic `X-Api-Key` but not official agy `x-goog-api-key`. Sending agy's native body into `chat_handler.go:55-190` is impossible without a schema and streaming translation, so current official calls either bypass IOP or fail before route/preset admission. + +#### Solution + +Register `/gemini/` under the shared auth wrapper. Parse only `/gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse`, freeze the bounded body, validate the contract fields, and translate to a synthetic internal streaming Chat request whose `model` is `{route-id}`. Call the existing Chat handler directly with the already-authenticated context and a streaming response writer that maps Chat SSE text/reasoning/tool/usage/terminal/error frames to Gemini SSE. Accept `x-goog-api-key` as a surface-specific IOP token, require equality with bearer if both exist, and never forward it as provider auth. + +Before: + +```go +// apps/edge/internal/openai/routes.go:11-19 +mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions)) +s.registerAnthropicRoutes(mux) +``` + +After: + +```go +mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions)) +mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent)) +s.registerAnthropicRoutes(mux) +``` + +#### Modified Files and Checklist + +- [x] `apps/edge/internal/openai/routes.go`: register Gemini route and Gemini-native auth/error selection. +- [x] `apps/edge/internal/openai/principal.go`: add constant-time `x-goog-api-key` principal extraction/conflict checks. +- [x] `apps/edge/internal/openai/gemini_types.go`: define bounded request/response/error DTOs for the contracted subset. +- [x] `apps/edge/internal/openai/gemini_handler.go`: validate path/query/body, construct internal Chat request and preserve cancellation. +- [x] `apps/edge/internal/openai/gemini_bridge.go`: incrementally translate Chat SSE to Gemini SSE with bounded tool assembly and usage mapping. +- [x] `apps/edge/internal/openai/gemini_handler_test.go`: cover normal, boundary, auth, route, tool, usage, error and cancellation behavior. + +#### Test Strategy + +Write table-driven tests using existing mock run/provider/preset services. Assert zero dispatch for malformed path/body/conflicting key, direct and preset route identity, no inbound key in provider headers/log body, streaming text/tool/usage shape, one terminal and cancellation propagation. + +#### Verification + +Run `gofmt -w` on changed Go files, then `go test -count=1 ./apps/edge/internal/openai`. Expected: all package tests pass fresh. + +### [API-2] official agy 1.1.12 benchmark adapter + +#### Problem + +`scripts/agent_benchmark/agy_iop.py:48-66` hard-codes 1.1.11, nonexistent `AGY_*` OpenAI transport and synthetic duration events. `build_agy_invocation` adds unsupported `--effort`, while `AgyEventParser` waits for events official agy never emits. `live_iop.py:881-908` therefore classifies the installed valid CLI as an implementation gap. + +#### Solution + +Use `GEMINI_API_KEY` and `GOOGLE_GEMINI_BASE_URL`; map `gemini-3.6-flash` to official label `Gemini 3.6 Flash`; omit `--effort`. Version-gate 1.1.12 and its real help options. Parse `init`, `step_update`, `result`; preserve result/step usage as caller-reported count metrics and duration seconds as a duration metric, derive finish+idle from one successful result plus process exit/quiescence, and classify non-success as failure. Effective route/stage binding comes only from the independently hashed config observation validated before launch, never a fabricated caller event. Structural redaction persists only event/state/step type/status and numeric metrics. + +#### Modified Files and Checklist + +- [x] `scripts/agent_benchmark/agy_iop.py`: replace constants, invocation, capability and official event parser/redactor. +- [x] `scripts/agent_benchmark/agy_iop_test.py`: replace fake transport/synthetic event tests with captured official 1.1.12 shapes and boundary cases. +- [x] `scripts/agent_benchmark/live_iop.py`: derive catalog root from route-qualified Gemini base and bind config-owned effective route before agy invocation. +- [x] `scripts/agent_benchmark/connectivity_integration_test.py`: update live agy seams and exact admitted binding assertions. +- [x] `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`: replace synthetic fixture with content-free official event shapes. + +#### Test Strategy + +Tests assert exact env allowlist/argv, no `--effort`, known-model rejection, 1.1.12 help/version gate, official success/error/result order, usage/duration units, malformed/duplicate result rejection, config-binding mismatch and durable secret/content redaction. No unit test contacts a provider. + +#### Verification + +Run `python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test`. Expected: all tests pass. + +### [API-3] Managed dev deployment and live readiness + +#### Problem + +Remote `build/dev-runtime/edge.yaml` serves legacy HTTP and contains static provider auth. Marked presets require managed principal projection/stage authorization, and the new Gemini ingress contract requires HTTPS plus `x-goog-api-key` as an IOP token. Source compatibility alone cannot prove C03/C07 or hybrid readiness. + +#### Solution + +First run the repository deterministic credential-slot smoke. Build current Control Plane/Edge/Node artifacts for each target and stage them on the clean remote release checkout. Under remote `build/dev-runtime/.secrets/credential-plane/` generate one CA, role/name-bound CP/Edge/Node certs, Edge HTTP cert for `toki-labs.com`, at-rest keyring, issuer Ed25519 and per-node X25519 recipient keys with restrictive modes. Create managed candidate configs that remove all legacy bearer/principal/provider header/env values, enable CP credential HTTPS/edge-wire mTLS, Edge CP/Node mTLS and public HTTPS, and Node lease crypto. Bootstrap one benchmark principal into local protected `token/.iop-bench`, register `.gemini`, `.claude`, `.gpt` through credential HTTPS stdin as slots/routes for the canonical direct and stage models, then bounded-restart CP→Edge→Nodes. Verify fresh projection, advertised route ids, safe slot revision/lease attribution, direct agy and hybrid agy through route-qualified bases, Claude/Codex compatibility and the public all-cell preflight. Update the Gemini contract status to active and the human dev guide only after these pass. + +#### Modified Files and Checklist + +- [x] `agent-contract/outer/gemini-compatible-api.md`: change planned→active and record final implemented constraints only after live proof. +- [x] `agent-contract/index.md`: keep source pointers and trigger terms aligned with implemented files. +- [x] `docs/edge-local-dev-guide.md`: add secret-safe route-qualified official agy and managed dev verification commands without token values. +- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record sanitized deterministic/remote/live outputs and exact external config paths. + +#### Test Strategy + +Reuse `make test-credential-slot-smoke` for deterministic security lifecycle. Remote checks use each binary's `config check` where available, TLS health, authenticated model discovery, official agy direct/hybrid minimal workspace calls and benchmark `preflight`; raw provider responses and credentials are not retained. + +#### Verification + +Run the Final Verification local suite, deterministic credential smoke, remote clean/source/artifact/config checks, then official agy direct/hybrid and `agent_comparison_benchmark.py preflight`. Expected: all cells ready, no legacy source accepted, and sanitized evidence only. + +### [API-4] Immutable C01-C09 run + +#### Problem + +The benchmark must retain one attempt per cell and scored failures, but invoking it before API-3 or rerunning after partial output would violate the approved manifest and D10. + +#### Solution + +After API-3, invoke public `run` exactly once. Extract exactly one CLI-emitted new run id, write only that id to `run_id.log`, and use public `status` to require exactly nine `success|failed|timed_out|cancelled` with zero running/interrupted. Never call `resume`, retry failed cells, directly invoke scored callers, or edit the run tree. + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store one CLI-emitted canonical run id after the one-time command. +- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim sanitized run/status output and classification. + +#### Test Strategy + +No additional test file. The immutable manifest validation, full benchmark suite, fresh all-cell preflight, one-time public run and read-only status are the acceptance oracle. + +#### Verification + +Run Final Verification step 5 exactly once. Mark API-4 complete only for nine retained non-interrupted terminal attempts; otherwise record blocker and do not rerun. + +## Modified Files Summary + +| Path | Item | +|------|------| +| `apps/edge/internal/openai/routes.go` | API-1 | +| `apps/edge/internal/openai/principal.go` | API-1 | +| `apps/edge/internal/openai/gemini_types.go` | API-1 | +| `apps/edge/internal/openai/gemini_handler.go` | API-1 | +| `apps/edge/internal/openai/gemini_bridge.go` | API-1 | +| `apps/edge/internal/openai/gemini_handler_test.go` | API-1 | +| `scripts/agent_benchmark/agy_iop.py` | API-2 | +| `scripts/agent_benchmark/agy_iop_test.py` | API-2 | +| `scripts/agent_benchmark/claude_iop.py` | API-2 | +| `scripts/agent_benchmark/claude_iop_test.py` | API-2 | +| `scripts/agent_benchmark/codex_iop.py` | API-2 | +| `scripts/agent_benchmark/codex_iop_test.py` | API-2 | +| `scripts/agent_benchmark/lifecycle.py` | API-2, API-4 recovery | +| `scripts/agent_benchmark/lifecycle_test.py` | API-2, API-4 recovery | +| `scripts/agent_benchmark/attempts.py` | API-4 recovery | +| `scripts/agent_benchmark/attempts_test.py` | API-4 recovery | +| `scripts/agent_benchmark/live_iop.py` | API-2 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | API-2 | +| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-2 | +| `scripts/e2e-credential-slot-smoke.sh` | API-3 | +| `agent-contract/outer/openai-compatible-api.md` | API-3 | +| `agent-contract/outer/gemini-compatible-api.md` | API-3 | +| `agent-contract/index.md` | API-3 | +| `docs/edge-local-dev-guide.md` | API-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | API-4 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | API-1, API-2, API-3, API-4 evidence | + +## Final Verification + +1. Format and run fresh focused suites: + +```bash +gofmt -w apps/edge/internal/openai/routes.go apps/edge/internal/openai/principal.go apps/edge/internal/openai/gemini_types.go apps/edge/internal/openai/gemini_handler.go apps/edge/internal/openai/gemini_bridge.go apps/edge/internal/openai/gemini_handler_test.go +go test -count=1 ./apps/edge/internal/openai +python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test +``` + +Expected: all focused tests pass. + +2. Run the full local suites and contract hygiene: + +```bash +go test -p 1 -count=1 ./... +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Expected: all packages/tests pass, manifest is valid, no whitespace error. + +3. Run deterministic managed credential qualification outside the repo temp tree: + +```bash +credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)" +TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke +rmdir "$credential_smoke_parent" +``` + +Expected: CA/mTLS, two-slot, rotation, revoke/no-fallback and Messages/Chat qualification pass. + +4. After fresh dev deploy, run the sanitized live readiness commands documented in `docs/edge-local-dev-guide.md`: official agy direct and hybrid minimal calls through `https://toki-labs.com:18083/gemini/`, authenticated model discovery, and: + +```bash +python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: C01-C09 all report ready with official caller version, managed route/stage binding and no secret output. + +5. Implementation-only, exactly once after step 4: run the immutable public benchmark and preserve its CLI-emitted id. + +```bash +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Then write the emitted `run-...` id to `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` and run: + +```bash +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +``` + +Expected: exactly nine retained non-interrupted terminal attempts. Do not rerun step 5. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log new file mode 100644 index 00000000..8025a651 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log @@ -0,0 +1,203 @@ + + +# Plan - pre-scored interruption 이후 C01-C09 replacement 실행 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 명령과 상태 경계를 그대로 실행하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다. + +## Background + +plan=3의 제품 코드, managed dev 배포와 caller readiness는 완료됐지만 최초 public `run`은 caller launch 전에 control socket `EINVAL`로 중단됐다. 그 결함과 pre-registration reconcile은 수정·회귀 검증됐고, SDD D10은 실패 evidence를 보존한 새 attempt를 허용한다. 이 revision은 이전 interruption을 덮어쓰지 않고 새 public run identity에서 C01-C09를 실행하는 작업만 소유한다. + +## Archive Evidence Snapshot + +- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다. +- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다. +- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다. +- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다. + +## Analysis + +### Files Read + +- Rules/skills: `agent-ops/rules/project/rules.md`, `agent-ops/rules/common/rules-roadmap.md`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-ops/skills/common/router.md`, `agent-ops/skills/common/plan/SKILL.md`, `agent-ops/skills/common/finalize-task-routing/SKILL.md`, `agent-ops/skills/common/update-test/SKILL.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- Test rules: `agent-test/local/rules.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md` +- Roadmap/SDD: `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- Benchmark/runtime: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/codex_iop.py`, `docs/edge-local-dev-guide.md` +- Active evidence: `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log`, `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, 상태 `[승인됨]`, 잠금 `해제`, 사용자 리뷰 없음. +- `milestone-task`는 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`다. +- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다. Evidence Map은 caller별 event/timing/usage/workspace와 hybrid stage/terminal evidence를 요구한다. +- D06은 cell별 repetitions=1과 fresh session을, D10은 실패 보존과 새 attempt 기록을 요구한다. 따라서 TEST-1은 S02 readiness를 재확인하고 TEST-2는 S04-S08의 새 run evidence를 append-only로 생성한다. + +### Verification Context + +- Environment: dev. Rules state는 usable이며 matching profile은 testing/edge/node smoke다. 파일 수정 없는 `resolve-context`로 local current checkout, remote dev runtime과 public benchmark CLI를 판정했다. +- Local runner: Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, HEAD `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`, 현재 task 구현 변경 때문에 dirty다. `../iop-s2`는 branch `dev`, HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean이다. +- Remote runtime: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, Darwin/arm64, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이다. `build/dev-runtime/bin/edge` hash는 위 snapshot과 일치하고 PID 172, ports 18082/18083/18084/19093/19101은 모두 up이다. +- CLI: `scripts/agent_comparison_benchmark.py`는 validate/preflight/run/resume/status/score/report를 제공하고 manifest validate는 `ok: manifest is valid`다. caller binary/version/help와 model catalog는 fresh public preflight가 실제로 확인했다. +- Runtime references: principal은 보호된 `token/.iop-bench`, private dev CA는 `token/iop-dev-ca.pem`에서 같은 process environment로만 읽는다. caller base/secret reference와 config observation 값만 child allowlist에 전달하며 값은 출력·tracked 파일·run metadata에 기록하지 않는다. +- Read-only preflight: 무설정 호출은 `implementation_gap=9`로 fail-closed했고 attempt를 만들지 않았다. reference 주입 뒤 `ok: preflight run_id=run-20260812T035838Z-4c4a056c63b1 status=ready ready=9 registration_required=0 implementation_gap=0`으로 통과했다. +- Constraints: benchmark CLI 밖 caller/provider 호출, route/model/effort 대체, old run resume, `--retry-failed`, run tree 편집, testbed 쓰기, raw secret 출력은 금지한다. 새 execution run이 retained failure로 끝나면 status만 기록하고 재실행하지 않는다. +- Gaps: 없음. Confidence는 high이며 test rule 유지보수는 필요 없다. + +### Test Coverage Gaps + +- 새 제품 동작이나 source 변경은 없다. control socket symlink와 pre-registration reconcile 회귀는 plan=3에서 lifecycle/attempt tests 및 전체 Python suite로 검증됐다. +- 남은 gap은 실제 C01-C09 결과뿐이며 unit test로 대체할 수 없다. public preflight/run/status가 acceptance oracle이다. + +### Symbol References + +- 없음. symbol rename, remove, dependency 변경이 없다. + +### Split Judgment + +- 새 run identity의 preflight, attempt allocation과 terminal status는 하나의 append-only state invariant다. 별도 sibling으로 나누면 runtime identity와 one-submission evidence가 분리되므로 같은 task path의 단일 plan으로 유지한다. + +### Scope Rationale + +- Edge/Node/Control Plane, caller adapter, manifest, SDD, contract와 dev config를 수정하지 않는다. 현재 ready runtime을 소비하는 검증 실행만 범위다. +- objective validation, blind scoring, performance aggregation과 report는 세 번째 Epic S09-S12의 후속 plan으로 남긴다. +- old interrupted run resume/retry/reconcile, 실패 cell 재시도, provider 직접 호출은 제외한다. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`. +- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`. +- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`. +- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`. + +## Dependencies and Execution Order + +1. TEST-1의 manifest validation과 fresh all-cell preflight가 `ready=9`여야 TEST-2를 시작한다. +2. TEST-2는 public `run`을 한 번만 호출한다. CLI가 만든 새 canonical run id를 기록한 뒤에는 read-only `status`만 사용한다. + +## Implementation Checklist + +- [ ] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output. +- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Protected runtime binding and fresh readiness + +#### Problem + +`scripts/agent_benchmark/live_iop.py:1-5,259-266` requires caller base/secret references and an operator-owned config observation. A fresh shell without those references produced preflight-only `implementation_gap=9`, even though the deployed runtime itself remained healthy. Starting scored work before a closed `ready=9` violates S02 and the benchmark skill stop condition. + +#### Solution + +Read the existing principal and CA only inside the execution process, bind Claude/agy/Codex to their deployed HTTPS surfaces, and provide the already-verified route/preset observation. Run manifest validation, the focused lifecycle regression, caller version/help checks and public preflight. No repository source or configuration is changed. + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record exact secret-safe validation/preflight output. + +#### Test Strategy + +No new test file is required because there is no behavior change. Run fresh lifecycle/attempt regressions and the public preflight; cached output is not accepted. + +#### Verification + +Run the Final Verification readiness block. Expected: manifest valid, focused tests pass, caller commands resolve, environment reference booleans are true, and preflight reports `ready=9` with no registration or implementation gap. + +### [TEST-2] One replacement C01-C09 execution run + +#### Problem + +`code_review_cloud_G10_3.log:159-197` records that the former execution command ended before caller launch and retained no C01-C09 result. S04-S08 therefore remain incomplete even though the infrastructure fault is fixed and readiness is green. + +#### Solution + +With TEST-1 references still bound, call the public `run` exactly once to create a distinct execution run. Preserve all older run roots, store only the CLI-emitted canonical execution id in `run_id.log`, and use public `status` to require nine `success|failed|timed_out|cancelled`, zero `interrupted`, and zero `running`. A retained failed/timed-out/cancelled cell remains evidence and is not retried. + +#### Modified Files and Checklist + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store exactly one CLI-emitted execution run id. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record the verbatim run output, exit code, canonical id and public status. + +#### Test Strategy + +No new test file is required. The deterministic CLI's append-only run/status contract is the only valid live acceptance path; callers are never invoked separately. + +#### Verification + +Run the Final Verification execution block once. Expected: exactly one new execution run id and nine retained accepted terminal attempts with zero interrupted/running. Do not invoke `run` or `resume` again in implementation or review. + +## Modified Files Summary + +| Path | Item | +|------|------| +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2 | + +## Final Verification + +1. Run fresh local contract checks: + +```bash +python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output. + +2. In one protected shell, bind runtime references and run fresh all-cell preflight. Read token values only into variables; never print them: + +```bash +set -euo pipefail +read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench +export IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083 +export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083 +export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1 +export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem" +export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem" +export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG +BENCH_CONFIG="$(python3 - <<'PY' +import json +routes = [ + {"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]}, + {"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]}, + {"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]}, +] +print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":"))) +PY +)" +export BENCH_CONFIG +for tool in python3 claude agy codex git; do command -v "$tool"; done +python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`. If it does not, stop before TEST-2. + +3. In the same protected shell, invoke the execution command exactly once and preserve its direct CLI output. A nonzero exit is acceptable only when public status later proves nine retained accepted terminal attempts: + +```bash +python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: the CLI emits one canonical execution id. Record the command exit code and stdout/stderr verbatim, then write that exact emitted id to `run_id.log` and use step 4 for status. Never rerun this block. + +4. Verify the stored pointer without invoking a provider: + +```bash +test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1 +execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" +printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$' +python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id" +git diff --check +``` + +Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_6.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_6.log new file mode 100644 index 00000000..4740fc36 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_6.log @@ -0,0 +1,86 @@ + + +# Plan - 지속 승인 기반 C01-C09 fresh scored execution + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md` 구현 섹션을 채우는 것이 필수 마지막 단계다. protected local references를 process environment에서만 읽고, public benchmark CLI를 통해 preflight와 새 `run`을 수행한다. caller/provider 직접 호출, old run resume, `--retry-failed`, run tree 편집, secret 출력은 금지한다. 새 run이 retained failure로 끝나더라도 사용자의 지속 승인은 이후 새 plan 진행 권한으로 유지되지만, 이 plan 안에서는 `run`을 한 번만 호출하고 status evidence를 기록한 뒤 멈춘다. + +## Background + +plan=5에서 Codex production usage parser와 config-owned execution/scoring binding 결함을 수정했고 focused 41 tests와 전체 benchmark 429 tests가 통과했다. 이후 `USER_REVIEW.md`는 사용자가 “이후는 계속 승인 상태이니 이어서 진행”이라고 명시해 external-execution 조건이 해소됐으며 `user_review_0.log`로 보존됐다. 이 revision은 모든 기존 run을 보존한 distinct new run identity에서 C01-C09를 실행한다. + +## Archive Evidence Snapshot + +- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained run `run-20260812T040619Z-00a5e6664764`의 parser/binding failure와 FAIL evidence. +- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 source fix, focused 41/OK, full 429/OK와 새 execution authorization gate. +- `user_review_0.log`: 사용자 지속 승인으로 exact new public run 실행 조건이 RESOLVED 됐다. +- 이전 run은 어떤 방식으로도 resume/retry/reconcile/edit하지 않는다. 새 execution CLI가 출력한 id만 `run_id.log`에 기록한다. + +## Analysis + +### SDD Criteria + +- Approved/unlocked SDD의 S04-S08은 C01-C09 one-submission, fresh session, caller lifecycle/usage/workspace 및 hybrid terminal evidence를 요구한다. +- D06은 repetitions=1, D10은 실패 evidence 보존과 새 attempt identity를 요구한다. +- milestone-task는 기존 다섯 id를 그대로 유지한다. + +### Verification Context + +- Local source regression: focused 41/OK, full 429/OK, manifest valid, diff check clean. +- Dev runtime: prior fresh public preflight가 `ready=9`; existing protected principal과 dev CA는 `token/.iop-bench`, `token/iop-dev-ca.pem`에 있다. +- Execution environment binds Claude/agy/Codex base URLs and independently verified config observation in one process only. Raw values are never printed or persisted in tracked task evidence. +- Constraints: deterministic public CLI only; no direct caller/provider, no route/model/effort substitution, no old run resume/retry, no run tree mutation. +- User authorization is continuing for subsequent necessary new executions. Each individual plan still calls public `run` at most once and preserves failure before another plan. + +### Test Coverage Gaps + +- Deterministic source tests are closed. Remaining gap is real C01-C09 scored evidence, which only public `run/status` can produce. + +### Split Judgment + +All-cell preflight, immutable run creation, nine attempt allocations and terminal status form one append-only execution invariant. Splitting would separate readiness and the exact authorized run identity, so one plan is retained. + +### Scope Rationale + +- No product/runtime/config/source change is planned. +- This plan owns only fresh verification, protected preflight, one new public run, stored pointer and status evidence. +- score/report and objective validation remain later Epic work. + +### Final Routing + +- isolated reassessment; closures true; capability gap none. +- Build scores `2/2/2/2/2`, risks all five, rework=2, integrity=false; route `grade-boundary`, cloud G10, `PLAN-cloud-G10.md`. +- Review scores `2/2/2/2/2`; `official-review`, cloud G10, `CODE_REVIEW-cloud-G10.md`. + +## Dependencies and Execution Order + +1. Fresh deterministic tests and manifest validation pass. +2. Protected public preflight reports exactly `ready=9` and zero gaps. +3. Call public `run` exactly once. +4. Store only a CLI-emitted canonical run id; use read-only status to verify nine accepted terminal attempts, `running=0`, `interrupted=0`. + +## Implementation Checklist + +- [ ] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh. +- [ ] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`. +- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output. + +## Modified Files Summary + +| Path | Item | +|---|---| +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 evidence | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test` +2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` +3. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` and `git diff --check` +4. In one secret-safe process, read `token/.iop-bench`, bind the existing dev endpoints/CA/config observation, and call public `preflight`. Require `ready=9 registration_required=0 implementation_gap=0`. +5. In that protected process, call `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once. Record verbatim stdout/stderr and exit code. Write `run_id.log` only if the CLI emits one canonical id. +6. If an id was emitted, call public `status`. Completion requires nine retained terminal attempts, zero running/interrupted. If no id was emitted, identify the newly created run root read-only for blocker evidence but do not fabricate `run_id.log`. + +After completing, fill CODE_REVIEW implementation-owned sections and keep the active pair for review. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_7.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_7.log new file mode 100644 index 00000000..d9b001ec --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_7.log @@ -0,0 +1,79 @@ + + +# Plan - agy failed lifecycle terminalization 및 fresh C01-C09 run + +## For the Implementing Agent + +R4의 선택된 수정만 구현하고 credential-free tests를 통과한 뒤 protected preflight와 public `run`을 각 계획대로 수행한다. 원인 재조사, old run resume, retry-failed, caller/provider 직접 호출, state tree 편집, secret 출력은 금지한다. 새 run은 이 plan에서 정확히 한 번 호출한다. + +## Background + +plan=6 new run `run-20260812T043704Z-041320764a11`은 C05 Codex를 success로 종결해 R1/R2 fix를 검증했지만, C03 agy의 valid `nonzero_exit`를 live adapter가 `stream_incompatible` 예외로 바꿔 attempt를 running으로 남겼다. 사용자의 이후 실행 지속 승인은 유효하다. 기존 run은 보존하고 R4 fix 이후 distinct new run을 수행한다. + +## Archive Evidence Snapshot + +- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: ready=9, one public run, C02 failed/C05 success/C03 running과 R4 FAIL evidence. +- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: Codex R1/R2 fixes and 429/OK. +- `user_review_0.log`: subsequent execution remains authorized. + +## Finding Resolution Map + +| Finding | Evidence | Root Cause | Selected Fix | Mode | Acceptance | +|---|---|---|---|---|---| +| Required R4 | C03 lifecycle is clean nonzero_exit/cleanup complete, but measurement/web absent and attempt running; agy branch raises after `observed_result` | post-admission execution failure is incorrectly reclassified as connectivity failure | return bound failed lifecycle directly; apply observed binding gate only to successful lifecycle; add integration test proving failure terminalization/continuation | direct-fix | focused integration, full 429+, ready=9, one new run with nine terminal attempts | + +## Analysis + +### SDD Criteria + +S04-S08/D06/D10 and the five milestone-task ids remain unchanged. Failed cells are accepted terminal evidence; `running|interrupted` are not. + +### Verification Context + +- Local/dev context and protected `token/.iop-bench`, `token/iop-dev-ca.pem` bindings are unchanged. +- Previous fresh ready=9 and Codex success prove runtime and R1/R2. +- New gap is deterministic in `_LiveAdapter.invoke` and can be tested without provider access. +- Continuing authorization permits a new run after the fix; each plan still invokes run once. + +### Test Coverage Gaps + +No test proves agy non-success lifecycle is returned to RunStore and later slots continue. Add the smallest live-registry/run-slots regression using invoker seams and existing production sidecar assertions. + +### Split Judgment + +R4 terminalization and its verification in a new nine-cell run are one correctness boundary; separating them would leave S04-S08 unverified. + +### Scope Rationale + +Modify only `live_iop.py`, its integration test, active evidence and `run_id.log`. Do not change agy transport/parser, dev runtime, generic recovery, other callers, scoring/report. + +### Final Routing + +isolated reassessment, closures true, scores 2/2/2/2/2, five loop risks, rework=3, integrity=false; build grade-boundary cloud G10, review official cloud G10. + +## Implementation Checklist + +- [ ] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams. +- [ ] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots. +- [ ] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check. +- [ ] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted. +- [ ] Fill CODE_REVIEW implementation-owned evidence. + +## Modified Files Summary + +| Path | Item | +|---|---| +| `scripts/agent_benchmark/live_iop.py` | FIX-1 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | evidence | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test` +2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` +3. manifest validate and `git diff --check`. +4. Protected public preflight must report ready=9. +5. Public `run` exactly once. Store only CLI-emitted id and use status. PASS requires nine terminal, running=0, interrupted=0. + +After completing, fill active CODE_REVIEW evidence and stop for review. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_local_G06_5.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_local_G06_5.log new file mode 100644 index 00000000..4f8044cc --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_local_G06_5.log @@ -0,0 +1,178 @@ + + +# Plan - Codex production stream 및 config-owned binding 정합화 + +## For the Implementing Agent + +`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 R1/R2의 선택된 수정만 구현하고 검증 결과를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 원인을 재조사하거나 다른 해결책을 선택하지 않는다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, public benchmark `run|resume`, provider 직접 호출, verdict·archive·`complete.log` 작성은 하지 않는다. + +## Background + +plan=4의 새 public run `run-20260812T040619Z-00a5e6664764`은 C02를 failed로 종결한 뒤 C05 Codex 프로세스가 실제로 끝났음에도 attempt를 `running`으로 남기고 exit 69로 중단됐다. 리뷰가 retained lifecycle stdout을 동일 parser에 재생해 current Codex usage key 누락과 config-owned binding/caller-owned enforcement 모순을 확정했다. 이 revision은 그 두 source defect와 회귀 테스트만 수정한다. 실패 run은 append-only evidence로 보존하며 재개·retry·state 편집·새 scored 실행을 하지 않는다. + +## Archive Evidence Snapshot + +- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다. +- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다. +- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다. +- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다. +- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다. + +## Finding Resolution Map + +| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands | +|---|---|---|---|---|---|---| +| Required R1 | C05 retained stdout replay가 `cache_write_input_tokens`에서 `CodexJSONLError`; 기존 9 parser tests는 모두 통과해 coverage gap 확인 | `_CODEX_USAGE_FIELDS`와 tracked fixture가 current caller usage schema보다 오래됨 | `cache_write_input_tokens -> cache_write_tokens`를 exact mapping에 추가하고 unit fixture/assertion을 production shape로 갱신하며 total은 합성하지 않음 | direct-fix | 실제 `turn.completed.usage`가 parser terminal success 및 typed cache-write metric으로 보존됨 | `python3 -m unittest scripts.agent_benchmark.codex_iop_test`; focused integration/full benchmark tests | +| Required R2 | parser는 absent binding을 허용하지만 `_LiveAdapter.invoke`와 `_LiveScoringAdapter.invoke`는 absent를 mismatch로 거부; integration fake가 synthetic event 주입 | config observation으로 admission ownership을 이전한 뒤 두 consumer와 fixture가 구 caller-event gate를 유지함 | admitted config binding을 canonical result로 사용하고 optional non-None caller observation만 exact mismatch 검사; execution/scoring 테스트에서 absent PASS와 mismatch FAIL 검증 | direct-fix | real Codex의 absent observation이 admitted binding으로 진행되고 실제 mismatch만 fail-closed함 | named connectivity integration tests; full benchmark tests; `git diff --check` | + +## Analysis + +### Files Read + +- Reviewer evidence: archived plan/review pair 위 두 exact 경로와 retained run의 C05 `attempt.json`, `lifecycle-result.json`, `cleanup-receipt.json`. +- Source/tests: `scripts/agent_benchmark/codex_iop.py`, `codex_iop_test.py`, `live_iop.py`, `connectivity_integration_test.py`, `lifecycle.py`, `attempts.py`, `scoring.py`, tracked Codex JSONL fixture. +- Rules/skills: project/testing rules, local test rules, `code-review`, `plan`, `finalize-task-routing`, project benchmark skill. +- Roadmap/SDD: active comparison Milestone와 approved/unlocked SDD S04-S08/D06/D10. + +### SDD Criteria + +- `milestone-task`는 기존과 동일한 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`를 유지한다. +- S04-S08은 C01-C09 retained terminal evidence를 요구한다. 이 source-fix revision은 실패한 실행 경계를 복구하지만 그 scored evidence 자체를 대체하지 않는다. +- D10에 따라 failed/incomplete run을 수정하거나 재사용하지 않는다. 이후 새 scored 실행은 별도 명시적 authorization과 새 plan에서만 수행한다. + +### Verification Context + +- Environment는 local source/test다. provider, remote dev, protected token이 필요하지 않다. +- Reviewer가 production retained output으로 exact parser failure를 재현했고 current test가 gap을 놓치는 것도 확인했다. +- 변경 후 credential-free unit/integration/full Python suite와 manifest validation을 fresh 실행한다. cached output은 허용하지 않는다. +- Constraints: archived/run evidence 수정 금지, public `run|resume` 금지, caller/provider 직접 호출 금지, optional observation mismatch의 fail-closed 성질 유지. +- Confidence는 high이고 verification context gap은 없다. + +### Test Coverage Gaps + +- parser test/fixture에 `cache_write_input_tokens`가 없다. +- live integration fake는 실제 Codex가 내보내지 않는 `iop_effective_binding`을 합성한다. +- execution consumer의 absent-observation PASS와 scoring consumer의 absent-observation PASS/non-None mismatch FAIL 조합이 없다. + +### Symbol References + +- `_CODEX_USAGE_FIELDS`: exact accepted caller usage schema. +- `CodexJSONLParser._turn_observations`: typed metric conversion and unknown-key fail-closed boundary. +- `_LiveAdapter.invoke`: execution-time config admission consumer. +- `_LiveScoringAdapter.invoke`: scoring-time config admission consumer. +- `CodexInvocationResult.effective_binding`: optional caller observation; canonical config binding과 동일한 소유물이 아니다. + +### Split Judgment + +R1 parser terminal success와 R2 admitted binding 소비는 하나의 real Codex invocation이 lifecycle result로 반환되는 compact boundary다. 분리하면 parser가 성공한 직후 동일 caller-binding 모순으로 다시 중단되므로 한 plan에서 함께 수정한다. predecessor index 05는 기존 archived completion evidence로 이미 충족되며 새 dependency는 없다. + +### Scope Rationale + +- Codex parser, live execution/scoring consumer, 해당 tests/fixture만 수정한다. +- generic attempt recovery, CLI exception policy, Claude의 실제 tool-use failure, agy/Edge/provider/runtime config는 변경하지 않는다. +- scored result 생성·재개·채점·report와 roadmap 완료 갱신은 제외한다. + +### Final Routing + +- `evaluation_mode=isolated-reassessment`, finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`. +- Build closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06, risks=`boundary_contract,structured_interpretation`, `large_indivisible_context=false`, `review_rework_count=1`, `evidence_integrity_failure=false`; route=`local-fit`, lane=`local`, filename=`PLAN-local-G06.md`, catalog=`worker/local/G06`. +- Review closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06; route=`official-review`, lane=`cloud`, filename=`CODE_REVIEW-cloud-G06.md`, catalog=`review/cloud/G06`. + +## Dependencies and Execution Order + +1. FIX-1이 production usage shape를 terminal success로 parse해야 한다. +2. FIX-2가 그 lifecycle result를 optional caller observation 없이 admitted config binding으로 실행·채점 소비하되 non-None mismatch는 거부해야 한다. +3. focused tests가 통과한 뒤 전체 Python benchmark suite와 manifest validation을 실행한다. + +## Implementation Checklist + +- [ ] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals. +- [ ] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [FIX-1] Current Codex usage schema + +#### Problem + +`CodexJSONLParser._turn_observations` validates usage keys as an exact subset of `_CODEX_USAGE_FIELDS`. The production `turn.completed` adds `cache_write_input_tokens`; omission converts an otherwise completed invocation into `parser_error`. + +#### Solution + +Add only the observed key with canonical metric name `cache_write_tokens`. Update the unit event and tracked stream fixture to include the retained production-shaped five fields, assert every exact reported count, and keep `total_tokens` unavailable when the caller omits it. + +#### Modified Files and Checklist + +- [ ] `scripts/agent_benchmark/codex_iop.py`: extend exact usage mapping. +- [ ] `scripts/agent_benchmark/codex_iop_test.py`: assert production-shaped cache-write parsing and no derived total. +- [ ] `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`: align deterministic fixture with current caller output. + +#### Test Strategy + +The parser unit must fail before the mapping and pass after it. Existing unknown/fractional/negative-key/value cases continue proving fail-closed behavior. + +#### Verification + +Run the focused parser suite in Final Verification step 1. + +### [FIX-2] Config-owned effective binding consumption + +#### Problem + +Preflight admits exact route/model/stage from independently validated config. Real Codex reports no custom binding event, but execution/scoring consumers currently require one, making all real successful Codex invocations impossible. + +#### Solution + +For execution, allow `result.effective_binding is None`, reject only a present value different from admitted, then retain the lifecycle observations already checked against admitted models/stages. For scoring, apply the same optional mismatch check and return the expected admitted tuple as the canonical scoring result. Remove the synthetic event from the integration fake; assert absent observation succeeds and present mismatch fails in both consumer paths. + +#### Modified Files and Checklist + +- [ ] `scripts/agent_benchmark/live_iop.py`: align execution and scoring consumers with config-owned admission. +- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove synthetic Codex event and cover absent/mismatch execution and scoring behavior. + +#### Test Strategy + +Use credential-free invoker seams returning real-shaped `CodexInvocationResult`. Verify absent optional observation reaches success using admitted binding, while a contradictory non-None tuple remains fail-closed. + +#### Verification + +Run the focused integration suite in Final Verification step 1, then the full suite in step 2. + +## Modified Files Summary + +| Path | Item | +|---|---| +| `scripts/agent_benchmark/codex_iop.py` | FIX-1 | +| `scripts/agent_benchmark/codex_iop_test.py` | FIX-1 | +| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | FIX-1 | +| `scripts/agent_benchmark/live_iop.py` | FIX-2 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G06.md` | FIX-1, FIX-2 evidence | + +## Final Verification + +1. Run focused parser and integration coverage: + +```bash +python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test +``` + +Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection. + +2. Run the fresh full benchmark suite: + +```bash +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +``` + +Expected: all benchmark tests pass with no skipped required test or network/provider call. + +3. Validate manifest and repository diff: + +```bash +python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +``` + +Expected: `ok: manifest is valid`; `git diff --check` has no output. Do not run public `run`, `resume`, `score`, caller, or provider commands in this follow-up. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log new file mode 100644 index 00000000..071b61e5 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log @@ -0,0 +1 @@ +run-20260812T044800Z-412e05fc80df diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/user_review_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/user_review_0.log new file mode 100644 index 00000000..ab99eaea --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/user_review_0.log @@ -0,0 +1,54 @@ +# User Review Required - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs + +## Requested At + +2026-08-12 + +## Status + +RESOLVED + +## Reason + +- Type: external-execution +- Target: `scripts/agent_comparison_benchmark.py run` against `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` using the existing protected `token/.iop-bench` and `token/iop-dev-ca.pem` dev binding +- Current review number: 5 +- Final verdict: FAIL +- Summary: R1/R2 source defects are fixed and all deterministic tests pass, but the one scored run previously authorized was retained as `failed=1,running=1`. Benchmark policy requires explicit user authorization before another stateful scored execution. + +## Loop History + +| Plan | Review | Verdict | Note | +|---|---|---|---| +| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | Initial execution preparation; no official verdict. | +| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | Readiness revision; no official verdict. | +| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | Runtime/provider readiness revision; no official verdict. | +| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, official agy, managed dev and caller readiness completed; first controller run interrupted before scored caller launch. | +| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Replacement run exposed current Codex usage parser and config-binding consumer defects. | +| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | R1/R2 fixed; focused 41 and full 429 tests pass, but a new scored execution remains authorization-gated. | + +## Blocking Evidence + +- Problem: S04-S08 still lack one C01-C09 run containing exactly nine retained terminal attempts with zero `running|interrupted`. +- Current archived plan: `plan_local_G06_5.log` +- Current archived review: `code_review_cloud_G06_5.log` +- Verification command: `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test && python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' && python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json && git diff --check` +- Actual output: focused `Ran 41 tests in 14.154s — OK`; full `Ran 429 tests in 116.096s — OK`; `ok: manifest is valid`; diff check `(none)`. Existing run status remains `{'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}`. +- Blocking rationale: the declared CLI and protected runtime binding are available, but another scored run repeats stateful/cost-bearing model execution after a retained failure. The benchmark skill forbids implicit retry or replacement execution without explicit user authorization. + +## Required User Action + +- [x] Explicitly authorize exactly one new public C01-C09 scored `run` after the retained failure. Resolved by the user's instruction that subsequent execution remains continuously authorized; no token value or provider key handoff is needed. + +## Resume Condition + +- The user explicitly states that one new public C01-C09 scored run is authorized. The next plan must preserve all old run roots, use fresh public preflight `ready=9`, call public `run` once, and never call `resume`, `--retry-failed`, direct callers/providers, or edit run state. + +## Next Execution Hint + +- Re-enter the `plan` skill for `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; archive this file as `user_review_0.log`, create a fresh routed execution pair, consume `token/.iop-bench` and `token/iop-dev-ca.pem` only inside the protected process environment, and use the deterministic public CLI. + +## Closure Rules + +- If the recorded user action and evidence resolve this stop as complete/PASS, update `USER_REVIEW.md` to the resolved state, write `complete.log` from `agent-ops/skills/common/code-review/templates/complete-log-template.md`, and move the task directory to the archive. +- If new implementation is required, the `plan` skill archives `USER_REVIEW.md` as `user_review_N.log` before writing a new `PLAN-*-G??.md` / `CODE_REVIEW-*-G??.md` pair. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/code_review_cloud_G07_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/code_review_cloud_G07_0.log new file mode 100644 index 00000000..b2b8235d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/code_review_cloud_G07_0.log @@ -0,0 +1,148 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run verification, keep active files in place, and report ready for review. Finalization, archive moves, and `complete.log` are review-only. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts, plan=0, tag=TEST + +## Archive Evidence Snapshot + +- Predecessor terminal run: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. +- Failed-result report: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md` (`success=2`, `failed=7`, no required workspace files). + +## Implementation Item Completion + +| Item | Status | +|---|---| +| TEST-1 Claude write tools/parser | [x] | +| TEST-2 Codex sandbox/metrics | [x] | +| TEST-3 agy HOME | [x] | +| TEST-4 guide/full tests | [x] | + +## Implementation Checklist + +- [x] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage. +- [x] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage. +- [x] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage. +- [x] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +- [x] Append PASS/WARN/FAIL and routing signals after fresh review verification. +- [x] Verify finding classifications and dimension assessment. +- [x] Archive active plan/review, write `complete.log` on PASS, and move the task directory to the dated archive. +- [x] Preserve milestone-task metadata and keep the active parent because child 08 remains. + +## Deviations from Plan + +없음. 계획에 명시된 파일과 동작 경계 안에서 구현했다. + +## Key Design Decisions + +- Claude는 shell/network를 열지 않고 `Read,Write,Edit`만 허용하며 같은 목록만 사전 승인한다. +- Claude partial snapshot은 message id로 중복 제거하고 user tool-result 경계에서 미완료 tool-use turn을 확정한다. +- Codex tool completion count와 선택적 duration metric을 분리하고, 검증 실패 전에는 count 상태를 바꾸지 않는다. +- agy는 prepared session directory를 HOME으로 사용하며 workspace/session/attempt 경로가 모두 존재해야 실행한다. + +## Reviewer Checkpoints + +- Claude has only Read/Write/Edit and accepts actual tool-use stream lifecycle without accepting contradictory terminals. +- Codex always uses workspace-write and tool counts are not coupled to optional duration. +- agy receives the fresh prepared HOME and no ambient user config. +- No secrets or task contents enter captures. + +## Verification Results + +### Focused adapter tests + +Command: `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` + +```text +................................. +---------------------------------------------------------------------- +Ran 33 tests in 6.742s + +OK +``` + +exit status: 0 + +### Complete benchmark tests + +Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` + +```text +................................................................................................................................................................................................................................................................................................................................................................................................................................................. +---------------------------------------------------------------------- +Ran 433 tests in 119.205s + +OK +``` + +exit status: 0 + +### Contract search + +Command: `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md` + +```text +scripts/agent_benchmark/agy_iop.py:344: "HOME": prepared.session_dir, +scripts/agent_benchmark/claude_iop.py:433: "--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit", +scripts/agent_benchmark/codex_iop.py:228: *executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config", +scripts/agent_benchmark/codex_iop.py:243: "HOME": prepared.session_dir, +docs/agent-comparison-benchmark-dev-guide.md:176:| Claude Code | Anthropic-compatible Messages | ... `--tools Read,Write,Edit --allowedTools Read,Write,Edit` ... | +docs/agent-comparison-benchmark-dev-guide.md:177:| agy | Gemini-native `streamGenerateContent` | fresh session `HOME` ... | +docs/agent-comparison-benchmark-dev-guide.md:178:| Codex | OpenAI-compatible Responses | fresh session `HOME` ... `exec --sandbox workspace-write` ... | +``` + +exit status: 0 + +## Section Ownership + +Implementation item status, checklist, deviations, decisions, and verification output are implementation-owned. Review checklist and verdict are review-only. + +## Reviewer Fresh Verification + +```text +$ python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test +................................. +---------------------------------------------------------------------- +Ran 33 tests in 6.763s + +OK + +$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +................................................................................................................................................................................................................................................................................................................................................................................................................................................. +---------------------------------------------------------------------- +Ran 433 tests in 118.282s + +OK + +$ git diff --check -- +(no output) +``` + +All commands exited 0. + +## Code Review Result + +- Verdict: PASS +- Required: 0 +- Suggested: 0 +- Nit: 0 +- Dimension Assessment: + - Correctness: Pass — restricted capabilities and parser state transitions are explicit and fail closed on contradictions. + - Completeness: Pass — all four plan items and documentation are implemented. + - Test coverage: Pass — launch argv, partial/tool-use streams, durationless Codex tools, and missing agy paths have regressions. + - API contract: Pass — existing builder/parser call sites remain compatible; documented CLI contracts match argv. + - Code quality: Pass — no debug output, placeholder, or ambient-secret inheritance added. + - Plan deviation: Pass — no out-of-scope implementation change. + - Verification trust: Pass — reviewer reran 33 focused and 433 complete tests plus diff check. +- Findings: 없음. +- Routing Signals: review_rework_count=0, evidence_integrity_failure=false. +- Next Step: PASS finalization and dependent child 08 execution. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log new file mode 100644 index 00000000..cdedc2a7 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log @@ -0,0 +1,38 @@ + + +# Complete - m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts + +## 완료 일시 + +2026-08-12 + +## 요약 + +Claude/Codex/agy의 격리 workspace 쓰기 계약과 parser/metric 회귀를 1회 루프 PASS로 정리했다. + +## 루프 이력 + +| Plan | Review | Verdict | 메모 | +|------|--------|---------|------| +| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | PASS | 33 focused/433 full tests와 diff check 통과 | + +## 구현/정리 내용 + +- Claude에 `Read,Write,Edit`만 허용하고 partial/tool-use/user-result stream lifecycle을 처리한다. +- Codex에 `workspace-write` sandbox를 지정하고 duration 없는 완료 tool도 정확히 집계한다. +- agy에 prepared session HOME을 제공하고 모든 prepared 경로 존재를 검증한다. +- dev guide의 세 caller 실행 계약과 격리 경계를 현재 구현에 맞췄다. + +## 최종 검증 + +- `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` - PASS; 33 tests. +- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 433 tests. +- `git diff --check -- ` - PASS; no output. + +## 잔여 Nit + +- 없음 + +## 후속 작업 + +- `08+07_comparison_rerun`에서 managed dev runtime과 fresh 9-cell execution을 검증한다. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/plan_local_G07_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/plan_local_G07_0.log new file mode 100644 index 00000000..ac0df25d --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/plan_local_G07_0.log @@ -0,0 +1,163 @@ + + +# Caller workspace-write contract remediation + +## For the Implementing Agent + +Implement every item, run the specified verification, and fill the implementation-owned sections of `CODE_REVIEW-cloud-G07.md` with actual output. Keep both active files in place and report ready for review. If blocked, record only the exact blocker, attempted commands/output, and resume condition. Do not ask the user, create control-plane stop files, archive logs, or write `complete.log`; finalization is code-review-skill only. + +## Background + +The first terminal 9-cell smoke run ended with seven failed cells and no generated workspace files. The adapters launched real callers, but Claude had all tools disabled and rejected normal stream snapshots, Codex did not grant workspace-write sandbox access, and agy omitted the isolated HOME required for its runtime configuration. + +## Archive Evidence Snapshot + +- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log` (PASS for terminal matrix execution). +- Canonical failed-result evidence: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md`; success=2, failed=7, all web gates failed because required files were absent. +- The predecessor review accepted terminal-evidence collection, not successful benchmark artifacts. This plan repairs the concrete caller launch/parser contracts exposed by that run. + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/claude_iop.py` +- `scripts/agent_benchmark/claude_iop_test.py` +- `scripts/agent_benchmark/codex_iop.py` +- `scripts/agent_benchmark/codex_iop_test.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/agy_iop_test.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `docs/agent-comparison-benchmark-dev-guide.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` + +### SDD Criteria + +The approved SDD is `READY`; this plan contributes milestone-task ids `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`. S04-S08 and Evidence Map rows require terminal evidence plus generated workspace results for C01-C09. The checklist therefore repairs all three caller write paths and requires deterministic adapter regression tests before any new scored run. + +### Verification Context + +No separate handoff was supplied. Repository source, archived predecessor `complete.log`, its canonical run report, and official caller CLI documentation were used. The old run is immutable evidence; no direct caller/provider invocation is permitted. Confidence is high for the launch-contract defects and medium for complete external recovery until the dependent live run is executed. + +### Test Coverage Gaps + +- Claude tests assert the empty `--tools=` launch and only a one-shot `end_turn`; they do not cover partial snapshots or tool-use cycles. +- Codex tests do not require `--sandbox workspace-write` and count a tool only when duration is reported. +- agy tests do not require isolated HOME or prepared-session directory availability. + +### Symbol References + +No public symbols are renamed or removed. Existing call sites of the modified builders/parsers remain unchanged. + +### Split Judgment + +This is child 07 of a split remediation. It has a stable contract: every caller receives the minimum isolated filesystem capability and its parser accepts the caller's documented event lifecycle. PASS is the focused/full Python test suite. Predecessor 06 is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Child 08 depends on this PASS and owns runtime recovery/scored execution. + +### Scope Rationale + +Do not change IOP routing, provider credentials, benchmark task content, scoring, roadmap state, or archived runs. This packet is limited to caller launch/parser contracts, regression tests, and the human dev guide. + +### Final Routing + +- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair` +- build closures: all true; scores `2/1/1/2/1` => G07; base/route `local-fit`; filename `PLAN-local-G07.md` +- review closures: all true; scores `2/1/1/2/1` => G07; route `official-review`; filename `CODE_REVIEW-cloud-G07.md` +- large_indivisible_context=false; positive risks=`boundary_contract,structured_interpretation,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent. + +## Implementation Checklist + +- [ ] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage. +- [ ] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage. +- [ ] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage. +- [ ] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Claude write-tool and stream lifecycle + +**Problem:** `scripts/agent_benchmark/claude_iop.py:220-238` rejects assistant snapshots whose `stop_reason` is null/tool-use, and `:351-356` passes `--tools=` while the task requires file creation. + +**Solution:** Launch Claude with exact `Read,Write,Edit` tools, pre-approve only those tools, retain `dontAsk`, and make the parser deduplicate message ids while accepting partial snapshots, tool-use completion, user tool results, then one final `end_turn` and successful result. + +**Modified Files and Checklist:** + +- [ ] `scripts/agent_benchmark/claude_iop.py`: restricted tool argv and multi-message lifecycle. +- [ ] `scripts/agent_benchmark/claude_iop_test.py`: exact argv and partial/tool-use regression cases. + +**Test Strategy:** Add normal, duplicate-terminal, session/model mismatch, and final result ordering assertions to the existing credential-free unit tests. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.claude_iop_test` exits 0. + +### [TEST-2] Codex workspace sandbox and tool metrics + +**Problem:** `scripts/agent_benchmark/codex_iop.py:221-224` starts ephemeral Codex without a writable sandbox. `:330-351` discards real completed command items when `duration_ms` is absent, reporting zero tool calls. + +**Solution:** Add `--sandbox workspace-write` to the isolated exec argv. Count a completed tool item once by safe id/type regardless of duration; publish a duration metric only when the caller explicitly reports one. + +**Modified Files and Checklist:** + +- [ ] `scripts/agent_benchmark/codex_iop.py`: sandbox argv and independent completed-tool count. +- [ ] `scripts/agent_benchmark/codex_iop_test.py`: argv and durationless command regressions. + +**Test Strategy:** Extend the existing fixture/unit coverage for exact sandbox arguments, durationless tool completion, uniqueness, and explicit-duration preservation. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.codex_iop_test` exits 0. + +### [TEST-3] agy isolated HOME + +**Problem:** `scripts/agent_benchmark/agy_iop.py` constructs a minimal environment without HOME even though agy stores runtime configuration beneath the user's home, causing startup to exit before JSONL output. + +**Solution:** Require both prepared workspace and session directories and pass `HOME=prepared.session_dir` without treating it as a secret-bearing allowlist key. + +**Modified Files and Checklist:** + +- [ ] `scripts/agent_benchmark/agy_iop.py`: prepared-path validation and HOME environment. +- [ ] `scripts/agent_benchmark/agy_iop_test.py`: create session fixture and assert HOME/path rejection. + +**Test Strategy:** Extend the builder tests; no network or live caller execution. + +**Verification:** `python3 -m unittest scripts.agent_benchmark.agy_iop_test` exits 0. + +### [TEST-4] Contract guide and full local tests + +**Problem:** `docs/agent-comparison-benchmark-dev-guide.md` still documents the broken launch contracts. + +**Solution:** Document the minimum write-capability arguments, isolated HOME, and failure classification so future setup does not reintroduce the smoke-only state. + +**Modified Files and Checklist:** + +- [ ] `docs/agent-comparison-benchmark-dev-guide.md`: align caller launch and troubleshooting guidance. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md`: record actual implementation evidence. + +**Test Strategy:** Run the complete benchmark module suite fresh; documentation is checked by exact searches in review. + +**Verification:** `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` exits 0. + +## Dependencies and Execution Order + +Predecessor `06+05_comparison_runs` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Complete TEST-1 through TEST-4 before child `08+07_comparison_rerun` starts. + +## Modified Files Summary + +| File | Item | +|---|---| +| `scripts/agent_benchmark/claude_iop.py` | TEST-1 | +| `scripts/agent_benchmark/claude_iop_test.py` | TEST-1 | +| `scripts/agent_benchmark/codex_iop.py` | TEST-2 | +| `scripts/agent_benchmark/codex_iop_test.py` | TEST-2 | +| `scripts/agent_benchmark/agy_iop.py` | TEST-3 | +| `scripts/agent_benchmark/agy_iop_test.py` | TEST-3 | +| `docs/agent-comparison-benchmark-dev-guide.md` | TEST-4 | +| `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md` | TEST-4 | + +## Final Verification + +1. `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` — exits 0. +2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` — exits 0 with fresh execution; cached output is not accepted. +3. `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md` — shows only the intended restricted/write-enabled contracts. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/work_log_0.log b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/work_log_0.log new file mode 100644 index 00000000..6b4bedd3 --- /dev/null +++ b/agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/work_log_0.log @@ -0,0 +1,30 @@ +# Milestone Work Log + +> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file. + +| seq | time | event | task | loop | role | attempt | model | result | locator | +|---:|---|---|---|---:|---|---:|---|---|---| +| 1 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json | +| 2 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json | +| 3 | 26-08-12 03:36:10 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json | +| 4 | 26-08-12 03:36:10 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json | +| 5 | 26-08-12 03:38:25 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json | +| 6 | 26-08-12 03:38:25 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json | +| 7 | 26-08-12 03:45:17 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json | +| 8 | 26-08-12 03:46:32 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json | +| 9 | 26-08-12 03:46:32 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json | +| 10 | 26-08-12 03:50:49 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json | +| 11 | 26-08-12 03:50:50 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json | +| 12 | 26-08-12 03:57:29 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json | +| 13 | 26-08-12 03:57:29 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json | +| 14 | 26-08-12 04:06:20 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json | +| 15 | 26-08-12 04:06:21 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json | +| 16 | 26-08-12 04:14:46 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json | +| 17 | 26-08-12 04:14:46 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json | +| 18 | 26-08-12 04:24:01 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json | +| 19 | 26-08-12 04:24:01 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json | +| 20 | 26-08-12 04:31:38 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json | +| 21 | 26-08-12 04:31:38 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json | +| 22 | 26-08-12 04:34:59 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json | +| 23 | 26-08-12 04:34:59 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json | +| 24 | 26-08-12 04:41:22 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json | diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md new file mode 100644 index 00000000..2de59bc9 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md @@ -0,0 +1,147 @@ + + +# Code Review Reference - REVIEW_REFACTOR + +> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** +> The task is NOT complete until every implementation-owned section below is filled in. +> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving. +> Fill implementation-owned sections, then stop with active files in place and report ready for review. +> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt. +> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields. +> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state. +> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=4, tag=REVIEW_REFACTOR + +## Archive Evidence Snapshot + +- Previous active packet is preserved at `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log` and `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log`; it had implemented R1-R6 but had not completed full local verification, deployment, public smoke, or a fresh scored run. +- R1-R6 remain selected, committed compatibility fixes. In particular, they cover Claude result-direct/API-error shapes, agy structured output and ERROR terminal handling, supervisor exit/receipt coherence, deterministic Plan rendering, the pinned Claude beta, and provider-independent Plan arrays. +- Retained run evidence already showed that a single lifecycle result conflates product and harness causes: valid caller errors became missing/malformed lifecycle evidence, parser/order defects became attempt failures, and web validation was skipped solely because lifecycle was non-success. +- The interrupted test left only `/tmp/iop-s0-interrupted-test-tmp7hc4t2o6`; no benchmark, unittest, deploy, or target Codex process remains active. + +## For the Review Agent + +> **[REVIEW AGENT ONLY]** Compare every item with source and rerun applicable commands fresh. Verify the implementation preserved R1-R6 and did not allocate a hybrid or scored nine-cell run. Finalization, verdict, log renames, `complete.log`, archive moves, and any next-state classification are review-agent only. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| REVIEW_REFACTOR-1 typed invocation outcomes | [ ] | +| REVIEW_REFACTOR-2 durable projections, artifact gate, and reporting | [ ] | +| REVIEW_REFACTOR-3 deterministic and clean-build qualification | [ ] | +| REVIEW_REFACTOR-4 direct-first live convergence gate | [ ] | + +## Implementation Checklist + +- [ ] [REVIEW_REFACTOR-1] Replace caller/parser string terminals and the overloaded lifecycle result with closed typed caller, product, harness, and process outcomes. +- [ ] [REVIEW_REFACTOR-2] Migrate attempt, measurement, web validation, scoring, CLI, report, project skill, guide, and living spec to preserve the three outcome axes and validate every terminal workspace. +- [ ] [REVIEW_REFACTOR-3] Run the complete deterministic benchmark suite, manifest validation, diff checks, and clean release source/build verification. +- [ ] [REVIEW_REFACTOR-4] Deploy the same clean source ref and pass the five-cell direct canary plus fresh C01-C09 ready=9 without allocating a hybrid or scored run. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +> **[REVIEW AGENT ONLY]** Implementing agents must not modify or check this section. + +- [ ] Verify caller product error, parser/order error, process terminal, cleanup, and artifact result cannot overwrite one another. +- [ ] Verify no compatibility `success`/`terminal_reason` alias remains available to new consumers. +- [ ] Verify failure workspaces receive uniform automatic gates and screenshots when renderable. +- [ ] Verify scoring eligibility requires product, harness/process, and artifact gates independently. +- [ ] Verify R1-R6, manifest, fixture, rubric, routes, credentials, and retry policy were not weakened. +- [ ] Verify local tests are fresh, clean deployment source/build identities match, direct canary is 5/5, and ready=9 is fresh. +- [ ] Verify no hybrid or new nine-cell scored run was allocated and no old failed run was retried/rewritten. +- [ ] Append one verdict and verified `review_rework_count` / `evidence_integrity_failure` signals. +- [ ] Archive the active review to `code_review_cloud_G10_4.log` and plan to `plan_cloud_G10_4.log` only through the code-review skill. +- [ ] If PASS, write `complete.log`, preserve milestone-task metadata, and move the task directory to its dated archive path; otherwise write the required next filesystem state. + +## Deviations from Plan + +_Record any deviations and rationale here._ + +## Key Design Decisions + +_Record the exact closed vocabularies, precedence rules, schema version decisions, and compatibility decisions here._ + +## Reviewer Checkpoints + +- Product success comes only from one caller-declared typed success terminal plus its required finish/idle evidence. +- A valid caller-declared product error may coexist with harness `passed`; malformed/contradictory output yields product `unknown` and harness `failed`. +- Timeout, cancellation, nonzero/signal exit, not-started, and cleanup failure remain independently queryable. +- Artifact validation runs for every terminal workspace; renderer/workspace unavailability is explicit. +- CLI/report rows and scoring reasons expose the independent axes without a misleading aggregate success. +- Live verification stops after the direct canary and ready=9; hybrid/scored execution requires a later authorized state. + +## Verification Results + +### REVIEW_REFACTOR-1 focused caller/lifecycle suite + +```bash +python3 -m unittest \ + scripts.agent_benchmark.lifecycle_test \ + scripts.agent_benchmark.claude_iop_test \ + scripts.agent_benchmark.agy_iop_test \ + scripts.agent_benchmark.codex_iop_test \ + scripts.agent_benchmark.connectivity_integration_test +``` + +_Paste actual stdout/stderr and exit code._ + +### REVIEW_REFACTOR-2 projection/validation/scoring/report suite + +```bash +python3 -m unittest \ + scripts.agent_benchmark.attempts_test \ + scripts.agent_benchmark.measurement_test \ + scripts.agent_benchmark.web_validation_test \ + scripts.agent_benchmark.scoring_test \ + scripts.agent_benchmark.reporting_test \ + scripts.agent_benchmark.skill_contract_test +``` + +_Paste actual stdout/stderr and exit code._ + +### REVIEW_REFACTOR-3 complete deterministic verification + +```bash +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate \ + --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +git status --short --branch +``` + +_Paste actual stdout/stderr, exit codes, commit/push identity, clean remote source state, build identities, and deployment health._ + +### REVIEW_REFACTOR-4 direct-first live convergence + +```bash +python3 scripts/agent_comparison_benchmark.py preflight \ + --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json +python3 scripts/agent_comparison_benchmark.py run \ + --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json +python3 scripts/agent_comparison_benchmark.py preflight \ + --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +_Paste sanitized exact CLI output, direct run id/status with five independent gates, ready=9 evidence, and proof that no hybrid or scored nine-cell run was allocated._ + +--- + +> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section, then leave active files in place. + +## Section Ownership + +| Section | Owner | Note | +|---|---|---| +| Header, Overview, Archive Evidence, Review Agent Instructions | Fixed | Do not modify | +| Implementation Item Completion and Implementation Checklist | Implementing agent checks status only | Text/order fixed | +| Review-Only Checklist | Review agent | Implementer must not modify | +| Deviations, Key Design Decisions | Implementing agent | Replace placeholders with actual content | +| Reviewer Checkpoints | Fixed | Plan-derived acceptance | +| Verification Results | Implementing agent, then reviewer | Record actual output; command changes require a deviation entry | +| Code Review Result | Review agent appends | Not present in stub | diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md new file mode 100644 index 00000000..ef9ac2b7 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md @@ -0,0 +1,343 @@ + + +# Benchmark outcome boundaries and direct-first convergence + +## For the Implementing Agent + +Implement the selected result-boundary migration exactly as written. Preserve commit `58fdb322` and all earlier R1-R6 compatibility fixes; do not deploy or start another nine-cell scored run until the new gates pass. Run every verification command, fill the implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with actual output, keep both active files in place, and report ready for review. If blocked, record only the exact blocker, attempted command/output, and resume condition in the review evidence; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`. + +## Background + +The stopped session completed and pushed R6 as `58fdb322`, then was interrupted while running the full Python benchmark suite. The current design still projects caller-declared product failure, parser/order failure, process failure, cleanup failure, timeout, and cancellation through one `terminal_reason` and one `success` boolean; attempt state, web validation, scoring, CLI summaries, and the report then reuse that mixed value. This packet replaces that overloaded boundary with three independent results—product execution, harness integrity, and artifact validation—and requires a five-cell direct canary before any hybrid or scored matrix execution. + +## Archive Evidence Snapshot + +- Previous active packet is preserved at `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log` and `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log`; it had implemented R1-R6 but had not completed full local verification, deployment, public smoke, or a fresh scored run. +- R1-R6 remain selected, committed compatibility fixes. In particular, they cover Claude result-direct/API-error shapes, agy structured output and ERROR terminal handling, supervisor exit/receipt coherence, deterministic Plan rendering, the pinned Claude beta, and provider-independent Plan arrays. +- Retained run evidence already showed that a single lifecycle result conflates product and harness causes: valid caller errors became missing/malformed lifecycle evidence, parser/order defects became attempt failures, and web validation was skipped solely because lifecycle was non-success. +- The interrupted test left only `/tmp/iop-s0-interrupted-test-tmp7hc4t2o6`; no benchmark, unittest, deploy, or target Codex process remains active. + +## Finding Resolution Map + +| ID | Evidence | Root Cause | Selected Fix | Mode | Acceptance | +|---|---|---|---|---|---| +| R7 | Claude API error and agy `status=ERROR` are structurally valid product failures, yet adapters currently emit no success terminal and lifecycle falls through to `nonzero_exit`/`missing_idle`; parser defects share the same reason field | parser output is an open union of strings, metrics, and tuples, and `InvocationResult.success` is computed from the harness terminal reason | replace string terminals with closed typed caller observations and freeze separate `ProductOutcome`, `HarnessOutcome`, and `ProcessOutcome` values in one invocation result | direct-fix | success, upstream product error, malformed stream, nonzero exit, timeout, cancellation, cleanup failure, duplicate and out-of-order cases preserve the correct independent axes | +| R8 | `attempt.json`, measurement, web validation, scoring eligibility, CLI summaries, skill prose, and Markdown report all treat lifecycle/attempt success as the one outcome | consumers copy the overloaded lifecycle reason instead of projecting independent gates | migrate durable schemas and consumers atomically; artifact validation runs against every terminal workspace and scoring requires product success + harness pass + artifact pass | direct-fix | deterministic fixtures expose separate product/harness/process/artifact columns and never label a parser-clean product error as harness failure or suppress artifact evidence because product failed | +| R9 | previous loops deployed after deterministic tests but discovered new direct-path caller variants only in scored execution | deployment qualification had no immutable direct-only all-caller gate and jumped from unit tests/ad-hoc smokes to the nine-cell scored run | after local and clean-build verification, deploy once, run the existing five-cell direct manifest as an unscored canary, require all five product/harness/artifact gates, then refresh the nine-cell preflight; hybrid execution and scored C01-C09 remain review-gated | direct-fix | no hybrid or scored run is allocated until direct canary 5/5, fresh ready=9, and source/build identity evidence all pass | + +## Analysis + +### Files Read + +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/lifecycle_test.py` +- `scripts/agent_benchmark/claude_iop.py` +- `scripts/agent_benchmark/claude_iop_test.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/agy_iop_test.py` +- `scripts/agent_benchmark/codex_iop.py` +- `scripts/agent_benchmark/codex_iop_test.py` +- `scripts/agent_benchmark/live_iop.py` +- `scripts/agent_benchmark/connectivity_integration_test.py` +- `scripts/agent_benchmark/attempts.py` +- `scripts/agent_benchmark/attempts_test.py` +- `scripts/agent_benchmark/measurement.py` +- `scripts/agent_benchmark/measurement_test.py` +- `scripts/agent_benchmark/web_validation.py` +- `scripts/agent_benchmark/web_validation_test.py` +- `scripts/agent_benchmark/scoring.py` +- `scripts/agent_benchmark/scoring_test.py` +- `scripts/agent_benchmark/reporting.py` +- `scripts/agent_benchmark/reporting_test.py` +- `scripts/agent_comparison_benchmark.py` +- `scripts/agent_benchmark/skill_contract_test.py` +- `scripts/fixtures/agent-comparison-benchmark-report.expected.md` +- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` +- `docs/agent-comparison-benchmark-dev-guide.md` +- `agent-spec/testing/agent-comparison-benchmark.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-test/local/rules.md` +- `agent-test/dev/rules.md` +- `agent-test/local/edge-smoke.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `agent-ops/rules/project/domain/edge/rules.md` +- `agent-ops/skills/project/dev-runtime-deploy/SKILL.md` +- `agent-ops/skills/project/e2e-smoke/SKILL.md` +- `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log` +- `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log` + +### SDD Criteria + +- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`; status `승인됨`, lock `해제`, user review `없음`. +- First-line milestone tasks: `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`. +- Target scenarios: S04-S08 require each caller/direct/hybrid product result plus terminal/timing/usage evidence; S09 explicitly requires automatic validation for both success and failure workspaces; S10 requires scoring separate from automatic gates; S11-S12 require source-labelled measurements and a complete failure-inclusive report; S13-S14 retain official agy and managed credential constraints. +- Evidence Map rows S04-S08 and the common completion rule drive the product/harness separation; S09 drives failure-workspace artifact validation; S10-S12 drive separate eligibility/report projections. Therefore the checklist migrates the result schema and every consumer before any new live execution. + +### Verification Context + +- No neutral verification handoff was supplied. Repository-native evidence came from the source/tests above, the approved SDD, local/dev test rules, the retained plan/review pair, and direct inspection of the stopped session/process tree. +- Preconditions: HEAD and `origin/feature/iop-one-shot-agent-model-comparison` are both `58fdb322`; tracked worktree was clean before this plan write; no benchmark/test/deploy process remained; archived predecessor `07+06_caller_write_contracts/complete.log` satisfies index 07. +- Constraints: preserve immutable failed runs, fixture/manifest/rubric/routes/credentials/retry policy, R1-R6, and public caller paths; never print secrets; use only the benchmark CLI for live canaries/scored work. +- Gap and confidence: deterministic fixtures cover every selected semantic axis, but live caller/provider compatibility remains external and must be proven after a same-ref clean deployment. Confidence is high for the structural root cause and moderate for field convergence until the direct canary passes; hybrid compatibility stays an explicit later gate. + +#### External Verification Preflight + +- Runner/workdir: `ssh toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, as declared by `agent-test/dev/rules.md`; native dev-runtime, not the compose profile. +- Source state: fetch `origin/dev`, `origin/main`, and tags; require a clean checkout at the selected release ref and record branch, HEAD, dirty state, ancestry, and source sync before build. +- Artifacts/config: `build/dev-runtime/bin/edge`, mac/Linux ARM64/Windows AMD64 Node binaries, and `build/dev-runtime/edge.yaml`; require `go version -m`, SHA-256, config check, refresh help/dry-run, and identical source identity. +- Commands/runtime: record `python3`, `git`, `go`, `claude --version`, `agy --version`, `codex --version`, caller help surfaces, managed Edge/CP/Node processes and ports `18082`, `18083`, `18084`, `19093`, `19101`; confirm 4 connected Nodes and all expected provider snapshots. +- Hosts/OS: remote runner macOS/ARM64, GX10 Linux/ARM64, OneXPlayer and RTX5090 Windows/AMD64 per inventory. A mismatch requires the documented clean sync/rebuild/redeploy/restart step; do not reuse a stale binary. + +### Test Coverage Gaps + +- Typed caller observations: current tests cover caller-specific success variants and malformed input but do not assert a common closed product terminal type across all three callers. +- Independent outcomes: current lifecycle tests assert only `success` and `terminal_reason`; add Cartesian boundary tests for product error vs parser failure vs process/cleanup failure. +- Durable migration: current attempt/measurement validation hard-codes the old result field set and `success == terminal_reason == success`; add canonical round-trip and tamper tests for all three axes. +- Failure-workspace validation: current web test explicitly expects `not_run` for every non-success lifecycle; replace it with generated/static/browser evidence tests independent of product outcome. +- CLI/report contract: current summaries and golden report expose `execution` and `terminal`; add independent columns/counts and skill contract assertions. +- Live convergence: no deterministic test substitutes for exact current official caller/provider behavior; require the direct canary in this packet and keep hybrid execution as a later reviewed gate before scored execution. + +### Symbol References + +- Replace parser string terminals `"finish"`, `"idle"`, and `"malformed"` at `ClaudeStreamParser`, `AgyEventParser`, `CodexJSONLParser`, lifecycle test parser lambdas, and connectivity fixtures with closed observation values. +- Replace `InvocationResult.success`, `InvocationResult.terminal_reason`, and `InvocationResult.finish_then_idle_then_quiet` call sites found in `agy_iop.py`, `live_iop.py`, `attempts.py`, `measurement.py`, `web_validation.py`, `scoring.py`, `reporting.py`, their tests, and CLI projections. Do not retain compatibility properties that let new consumers silently collapse the axes again. +- `AttemptMeasurement.terminal_reason` and report `execution/terminal` projections are schema migrations; update all constructor, serialization, loader, tamper, golden fixture, and skill-documentation references in the modified-file boundary. +- Top-level attempt state changes from an overloaded success result to controller lifecycle only: `running | completed | timed_out | cancelled | interrupted`. Retry and unresolved decisions consume the nested product/harness/process/artifact gates, not the state name. + +### Split Judgment + +Keep one atomic plan. The stable persisted attempt invariant is: one invocation publishes one coherent product outcome, harness outcome, process outcome, measurement, artifact outcome, eligibility decision, CLI status, and report row. Splitting the producer types from durable validation/consumer migration would either require compatibility aliases that preserve the defect or create an invalid intermediate schema; live rollout is included as integration evidence only after the atomic deterministic migration passes. + +### Scope Rationale + +- No Edge/Node product code, outer API contract, route binding, credential plane, provider retry, manifest, fixture, rubric, or SDD decision changes are allowed; R1-R6 already own those compatibility fixes. +- Historical run trees and `preflight_id.log`/`run_id.log` remain immutable. A new scored run is explicitly excluded from implementation ownership; after canaries and ready=9, the reviewer verifies the gate and decides the next authorized state. +- Common Agent-Ops rules/skills remain untouched. Only the project benchmark skill is updated because its public CLI outcome contract changes. + +### Final Routing + +- `evaluation_mode=first-pass`; all build/review closures are true, with no capability gap. +- finalizer=`finalize-task-policy.sh pair`; build=`grade-boundary/cloud/G10`, catalog=`worker/cloud/G10`, filename=`PLAN-cloud-G10.md`; review=`official-review/cloud/G10`, catalog=`review/cloud/G10`, filename=`CODE_REVIEW-cloud-G10.md`. +- Grade scores: build `2/2/2/2/2`, review `2/2/2/2/2` for scope/state/blast/evidence/verification. +- `large_indivisible_context=true`; positive loop risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5). +- Recovery signals: `review_rework_count=3`, `evidence_integrity_failure=true`; both risk and recovery boundaries match, while the route basis remains grade-boundary. + +## Implementation Checklist + +- [ ] [REVIEW_REFACTOR-1] Replace caller/parser string terminals and the overloaded lifecycle result with closed typed caller, product, harness, and process outcomes. +- [ ] [REVIEW_REFACTOR-2] Migrate attempt, measurement, web validation, scoring, CLI, report, project skill, guide, and living spec to preserve the three outcome axes and validate every terminal workspace. +- [ ] [REVIEW_REFACTOR-3] Run the complete deterministic benchmark suite, manifest validation, diff checks, and clean release source/build verification. +- [ ] [REVIEW_REFACTOR-4] Deploy the same clean source ref and pass the five-cell direct canary plus fresh C01-C09 ready=9 without allocating a hybrid or scored run. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [REVIEW_REFACTOR-1] Typed invocation outcomes + +**Problem:** `scripts/agent_benchmark/lifecycle.py:262-334,1552-1654,1718-1839` admits `str | ParsedMetric | tuple` parser values, treats unknown values as harness reasons, and computes `success` from the same reason used for process/cleanup. `claude_iop.py:280-361`, `agy_iop.py:439-496`, and `codex_iop.py:314-336` encode valid product errors by omitting terminal evidence or raising parser errors. + +**Solution:** Introduce frozen closed observations in `lifecycle.py`: `CallerEvent(kind=finish|idle)`, `CallerTerminal(status=succeeded|failed, reason=)`, plus existing `ParsedMetric`. Parser return type becomes `CallerObservation | tuple[CallerObservation, ...] | None`; raw strings are rejected. Freeze three results in `InvocationResult`: + +```python +# before: scripts/agent_benchmark/lifecycle.py:313-334 +class InvocationResult: + success: bool + terminal_reason: str + exit_code: Optional[int] + signal: Optional[int] + finish_then_idle_then_quiet: bool + +# after +class ProductOutcome: + status: str # succeeded | failed | unknown + reason: str # caller_success | caller_error | unavailable + +class HarnessOutcome: + status: str # passed | failed + reason: str # success | parser_error | malformed_event | ... + ordered_terminal: bool + cleanup_complete: bool + +class ProcessOutcome: + status: str # exited | signalled | timed_out | cancelled | not_started + exit_code: Optional[int] + signal: Optional[int] + +class InvocationResult: + product: ProductOutcome + harness: HarnessOutcome + process: ProcessOutcome + ... +``` + +The exact reason vocabularies must be constants validated at construction and serialization. A valid caller failure emits `CallerTerminal(failed, caller_error)` and may still have a clean parser/order/cleanup; malformed or contradictory caller output sets `product=unknown`, `harness=failed`; timeout/cancel/launch/cleanup remain process/harness results and cannot fabricate a product failure. Product success requires exactly one succeeded caller terminal plus the caller-specific finish/idle sequence; process exit alone never creates product success. Update each official caller adapter to emit the same types and keep metrics independently. + +**Modified Files and Checklist:** + +- [ ] `scripts/agent_benchmark/lifecycle.py`: add closed types/vocabularies, typed parser application, independent result construction, and canonical evidence records. +- [ ] `scripts/agent_benchmark/claude_iop.py`: emit typed finish/idle and success/API-error terminal observations. +- [ ] `scripts/agent_benchmark/agy_iop.py`: emit typed success/ERROR terminals; reserve parser failure for malformed structure. +- [ ] `scripts/agent_benchmark/codex_iop.py`: emit typed turn success and bridge idle; explicitly type unsuccessful terminal turns. +- [ ] `scripts/agent_benchmark/live_iop.py`: bind metrics and connectivity using typed outcomes without collapsing them. +- [ ] `scripts/agent_benchmark/lifecycle_test.py`, `claude_iop_test.py`, `agy_iop_test.py`, `codex_iop_test.py`, `connectivity_integration_test.py`: replace constructors/expectations and add independent-axis regressions. + +**Test Strategy:** Add named tests `test_product_error_can_have_clean_harness_and_process`, `test_parser_failure_leaves_product_unknown`, `test_timeout_cancel_and_cleanup_do_not_fabricate_product`, and caller-specific valid-error/success tests. Use local fake executables and current JSONL fixtures; no network. + +**Verification:** + +```bash +python3 -m unittest \ + scripts.agent_benchmark.lifecycle_test \ + scripts.agent_benchmark.claude_iop_test \ + scripts.agent_benchmark.agy_iop_test \ + scripts.agent_benchmark.codex_iop_test \ + scripts.agent_benchmark.connectivity_integration_test +``` + +Expected: exit 0; success and product-error fixtures have harness `passed`, malformed fixtures have product `unknown` and harness `failed`, and timeout/cancel/process failure retain their own process status. + +### [REVIEW_REFACTOR-2] Durable projections, artifact gate, and reporting + +**Problem:** `attempts.py:1192-1214,1368-1464,1608-1870`, `measurement.py:478-541,675-716`, `web_validation.py:491-579`, `scoring.py:943-964`, `agent_comparison_benchmark.py:145-176`, and `reporting.py:349-494` reuse lifecycle success as execution, validation eligibility, scoring eligibility, and report truth. This violates S09 for failed workspaces and makes the report unable to distinguish product defects from harness defects. + +**Solution:** Bump the internal lifecycle/journal, attempt-result, measurement, web-validation, and report fixture schema versions where defined; new writers emit only the new version and never rewrite historical evidence. A versioned read-only legacy decoder may project an old proven success as product succeeded/harness passed, but every legacy non-success must keep product `unknown` and preserve only the harness/process fact actually evidenced—never infer a product failure. Persist exact nested `product`, `harness`, and `process` objects in lifecycle result/terminal, measurement, and attempt record. Make top-level attempt state controller-only: `running | completed | timed_out | cancelled | interrupted`; retry/unresolved logic reads the independent nested gates. Run generated/static/browser validation for every terminal workspace, including product/harness failure, and mark only genuinely unavailable renderer/workspace evidence as `not_run`/`blocked`. Scoring eligibility requires product `succeeded`, harness `passed`, process acceptable, and web `passed`, with a separate reason for each failed gate. + +Replace the CLI and report projection with explicit fields: + +```text +product=succeeded|failed|unknown +harness=passed|failed +process=exited|signalled|timed_out|cancelled|not_started +artifact=passed|failed|blocked|not_run +scoring=scored|unscored|scoring_failed|blocked +``` + +`run` exit 0 still requires the latest attempt of every slot to have product succeeded, harness passed, acceptable process cleanup, and artifact passed. `status` and Markdown must display each axis; no aggregate named `success` may hide which axis failed. Update the project skill, dev guide, golden report, and living spec to document the same exact contract. + +**Modified Files and Checklist:** + +- [ ] `scripts/agent_benchmark/attempts.py`, `attempts_test.py`: migrate canonical attempt/result validation, terminal transitions, status projection, tamper and recovery tests. +- [ ] `scripts/agent_benchmark/measurement.py`, `measurement_test.py`: persist all three outcomes and their sources. +- [ ] `scripts/agent_benchmark/web_validation.py`, `web_validation_test.py`: validate every terminal workspace independent of product/harness status. +- [ ] `scripts/agent_benchmark/scoring.py`, `scoring_test.py`: use explicit gate reasons and invoke evaluator only when all eligibility axes pass. +- [ ] `scripts/agent_benchmark/reporting.py`, `reporting_test.py`, `scripts/fixtures/agent-comparison-benchmark-report.expected.md`: render independent outcome columns and failure-inclusive artifact evidence. +- [ ] `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/skill_contract_test.py`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: align public summaries and exact output contract. +- [ ] `docs/agent-comparison-benchmark-dev-guide.md`, `agent-spec/testing/agent-comparison-benchmark.md`: document the new source of truth and direct-first gate. +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record actual migration and verification evidence. + +**Test Strategy:** Add canonical round-trip/tamper tests for each nested result, web validation of a failed-product workspace that still produces screenshots/gates, scoring exclusions for each independent axis, CLI status count assertions, and a golden report row for product failure with harness pass plus parser failure with product unknown. + +**Verification:** + +```bash +python3 -m unittest \ + scripts.agent_benchmark.attempts_test \ + scripts.agent_benchmark.measurement_test \ + scripts.agent_benchmark.web_validation_test \ + scripts.agent_benchmark.scoring_test \ + scripts.agent_benchmark.reporting_test \ + scripts.agent_benchmark.skill_contract_test +``` + +Expected: exit 0; durable tampering is rejected, failed-product workspaces receive artifact evidence, evaluator invocation occurs only for all-pass attempts, and the golden report shows the separate axes. + +### [REVIEW_REFACTOR-3] Deterministic and clean-build qualification + +**Problem:** The stopped session was interrupted during the complete suite, so commit `58fdb322` and this migration have no complete fresh deterministic result. A live deployment before this gate would repeat the prior loop. + +**Solution:** Run the entire Python suite from the clean feature tree, validate the fixed manifest, run diff checks, commit/push the migration, then follow `dev-runtime-deploy` through clean remote sync, sequential relevant Go tests, rebuild of all four binaries, config checks, and source/build identity capture. Do not start a caller canary until every binary is proven to contain the same clean ref. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record full local output and clean release/build identities without secrets. + +**Test Strategy:** No new test file; this item executes the complete existing suite fresh (`unittest` has no result cache) and validates the actual manifest plus release build boundary. + +**Verification:** + +```bash +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate \ + --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +git status --short --branch +``` + +Expected: all tests and validation exit 0, `git diff --check` is silent, and only the intentional plan/review plus implementation changes exist before commit; after commit/push, the remote release clean-build procedure proves matching source and binary identities. + +### [REVIEW_REFACTOR-4] Direct-first live convergence gate + +**Problem:** The prior packet planned ad-hoc direct/hybrid smoke after deployment but did not define a single deterministic gate whose failure prevents scored execution. Consequently new exact caller variants were discovered inside the costly nine-cell run. + +**Solution:** After the same-ref deployment and 4/4 Node plus provider health checks, use only the public benchmark CLI with the existing `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` five-cell direct manifest. Run `preflight`, then exactly one unscored canary `run`; require five latest attempts with product succeeded, harness passed, acceptable process, artifact passed, no running/interrupted state, and no secret leakage. Only after the direct canary passes may the one-shot manifest receive a fresh `preflight` with ready=9. Stop there: do not call any hybrid canary or the scored nine-cell `run` in this implementation packet; those start only from a later reviewed/authorized packet. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record sanitized deployment, five direct cell outcomes, hybrid caller/route outcomes, and ready=9 preflight id/output. + +**Test Strategy:** External integration evidence is mandatory because current official CLI/provider versions are the compatibility boundary. The direct canary is unscored, uses fresh workspaces/sessions, and creates append-only run evidence; no retry or route substitution is allowed. Hybrid/scored execution is deliberately deferred rather than represented by an undefined ad-hoc command. + +**Verification:** + +```bash +python3 scripts/agent_comparison_benchmark.py preflight \ + --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json +python3 scripts/agent_comparison_benchmark.py run \ + --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json +python3 scripts/agent_comparison_benchmark.py preflight \ + --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +Expected: direct preflight is ready=5, the direct canary returns product/harness/artifact 5/5 with no unresolved attempt, and final one-shot preflight is ready=9. No hybrid or nine-cell scored run id is allocated. + +## Modified Files Summary + +| File | Items | +|---|---| +| `scripts/agent_benchmark/lifecycle.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/lifecycle_test.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/claude_iop.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/claude_iop_test.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/agy_iop.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/agy_iop_test.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/codex_iop.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/codex_iop_test.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/live_iop.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/connectivity_integration_test.py` | REVIEW_REFACTOR-1 | +| `scripts/agent_benchmark/attempts.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/attempts_test.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/measurement.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/measurement_test.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/web_validation.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/web_validation_test.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/scoring.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/scoring_test.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/reporting.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/reporting_test.py` | REVIEW_REFACTOR-2 | +| `scripts/fixtures/agent-comparison-benchmark-report.expected.md` | REVIEW_REFACTOR-2 | +| `scripts/agent_comparison_benchmark.py` | REVIEW_REFACTOR-2 | +| `scripts/agent_benchmark/skill_contract_test.py` | REVIEW_REFACTOR-2 | +| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | REVIEW_REFACTOR-2 | +| `docs/agent-comparison-benchmark-dev-guide.md` | REVIEW_REFACTOR-2 | +| `agent-spec/testing/agent-comparison-benchmark.md` | REVIEW_REFACTOR-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | REVIEW_REFACTOR-2, REVIEW_REFACTOR-3, REVIEW_REFACTOR-4 | + +## Dependencies and Execution Order + +- Predecessor `07+06_caller_write_contracts` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log`. +- Execute items 1 and 2 as one schema migration, then item 3. Item 4 starts only after item 3 and same-ref deployment pass. The direct canary precedes ready=9. Hybrid and scored execution are outside this packet. + +## Final Verification + +```bash +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate \ + --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +git diff --check +git status --short --branch +``` + +Expected local outcome: all tests/validation pass, diff check is silent, and repository state contains only intentional changes. Cached test output is not accepted; rerun fresh. + +After clean same-ref build/deploy, record 4/4 connected Nodes, expected healthy providers, direct preflight ready=5, direct product/harness/artifact 5/5, and final ready=9. Verify no hybrid or scored C01-C09 `run` was allocated by this packet. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log new file mode 100644 index 00000000..fcd4613a --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log @@ -0,0 +1,115 @@ + + +# Code Review Reference - TEST + +> **[IMPLEMENTING AGENT — READ FIRST]** Start only after child 07 PASS. Fill implementation evidence, leave active files in place, and report ready for review. Finalization is review-only. + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=0, tag=TEST + +## Archive Evidence Snapshot + +- Preserve `run-20260812T044800Z-412e05fc80df` and `run-20260812T050756Z-18293db0b83f`; do not reuse them. +- Dependency: archived PASS `07+06_caller_write_contracts/complete.log`. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| TEST-1 runtime recovery | [ ] | +| TEST-2 ready=9 preflight | [ ] | +| TEST-3 one scored run | [ ] | + +## Implementation Checklist + +- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source. +- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9. +- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +## Review-Only Checklist + +- [ ] Append verdict/routing signals after fresh verification. +- [ ] Verify no direct caller/provider invocation or scored retry occurred. +- [ ] Archive active files and write/move `complete.log` only on PASS. +- [ ] Preserve milestone-task metadata. + +## Deviations from Plan + +Clean release source was required before a scored run. The first rebuilt release lacked the feature source and public preflight `run-20260812T053517Z-6b4f4de67128` correctly stopped with 9 `model_missing` cells. After committing/merging the feature into `release/dev-936`, the mandatory pre-build suite exposed a new deterministic test defect, so no scored run was started. + +## Key Design Decisions + +Preserve both preflight-only runs. Do not bypass the failed pre-build suite or deploy `+dirty` source. Repair the exact bootstrap test lifecycle/timeout defect, rerun the full release qualification, then resume preflight and the single scored run. + +## Reviewer Checkpoints + +- Runtime recovery used the existing managed deployment and did not expose secrets. +- Preflight is ready=9 before run. +- Exactly one new scored run id exists, with 9 success and all web gates passing. + +## Verification Results + +### Dependency and runtime recovery + +```text +release source before merge: d40e4494e767e3fec7796c670f58130a9a194e80 +first rebuilt Edge source: d40e4494e767 (clean) +ports: 18082/18083/18084/19093/19101 listening +Control Plane: 4/4 nodes connected; 8 providers healthy/available; in_flight=0 queued=0 + +fresh release b244b6e224a9d44c09a1e7af49c0788f597c89fe pre-build test: +--- FAIL: TestRefreshConfigApplyNoChangeSkipsNodePush (2.02s) + runtime_refresh_node_test.go:575: register: request timeout for nonce 1 + +isolated probes: +3 x exact test: PASS +full bootstrap package: +--- FAIL: TestRefreshConfigApplyNoChangeSkipsNodePush (0.00s) + runtime_refresh_node_test.go:518: expected already started error +``` + +### Public preflight + +Command: `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` + +```text +error: preflight blocked run_id=run-20260812T053517Z-6b4f4de67128 status=registration_required ready=0 registration_required=9 implementation_gap=0 +``` + +The immutable preflight JSON classifies every cell as `model_missing`. This came from the clean d40e release not yet containing the feature; it is not accepted as ready evidence. + +### Public scored run + +Command: `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` + +Not run. The mandatory clean-release pre-build suite failed before the scored-run gate. + +## Code Review Result + +- Verdict: FAIL +- Required: 1 +- Suggested: 0 +- Nit: 0 +- Dimension Assessment: + - Correctness: Fail — bootstrap qualification is nondeterministic under its own test lifecycle. + - Completeness: Fail — ready=9 and scored execution are not yet produced. + - Test coverage: Fail — the intended handshake timeout constant is not used by register calls and cleanup asserts an unsupported duplicate Start lifecycle. + - API contract: Pass — public preflight stopped without substitution. + - Code quality: Warn — one test contains a destructive duplicate Start in cleanup. + - Plan deviation: Pass — the new source defect was preserved and routed instead of bypassed. + - Verification trust: Pass — both distinct failure modes were reproduced on the declared remote release runner. +- Findings: + - R1 Required + - Evidence: `apps/edge/internal/bootstrap/runtime_refresh_node_test.go:517-524` calls `Start` again in cleanup; `Runtime.newLifetimeContext` cancels the live lifetime before the second transport bind. The same test's registration at `:569-575` still uses `2*time.Second` while `fakeNodeHandshakeTimeout=5*time.Second` exists for this remote/race stall class. Remote full suite produced register timeout; isolated package produced duplicate-Start success. + - Root Cause: stale test-only lifecycle and timeout code; the duplicate Start can cancel/rebind the active runtime, and the register handshake retains the obsolete short bound. + - Selected Fix: replace the cleanup with one `rt.Stop()` call and replace all fake-node register `2*time.Second` bounds in this file with `fakeNodeHandshakeTimeout`. Add no production lifecycle change. Accept with exact test `-count=20`, package `-count=10`, full dev-runtime suite, then rebuild/redeploy and resume ready=9 plus one scored run. + - Disposition: direct-fix. +- Routing Signals: review_rework_count=1, evidence_integrity_failure=false. +- Next Step: materialize REVIEW_TEST follow-up and continue automatically. + +## Section Ownership + +Implementation status/checklist/evidence is implementation-owned. Review checklist and verdict are review-only. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log new file mode 100644 index 00000000..25dd7c0a --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_1.log @@ -0,0 +1,126 @@ + + +# Code Review Reference - REVIEW_TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=1, tag=REVIEW_TEST + +## Implementation Item Completion + +| Item | Status | +|---|---| +| REVIEW_TEST-1 bootstrap test fix | [x] | +| REVIEW_TEST-2 release qualification/deploy | [x] | +| REVIEW_TEST-3 ready preflight | [x] | +| REVIEW_TEST-4 scored run | [x] | + +## Implementation Checklist + +- [x] [REVIEW_TEST-1] Test-only lifecycle/timeout fix and high-count regressions passed. +- [x] [REVIEW_TEST-2] Clean release tests, four-binary rebuild/deploy, and four-node health passed. +- [x] [REVIEW_TEST-3] Fresh public preflight produced ready=9. +- [x] [REVIEW_TEST-4] Exactly one scored run was executed and its failed immutable evidence was recorded. +- [x] Implementation evidence was filled without exposing credentials. + +## Review-Only Checklist + +- [x] Final verdict and routing signals appended. +- [x] R1, release identity, ready=9, exactly-one run, and cell gates freshly checked. +- [x] No scored retry was performed. + +## Deviations from Plan + +The bootstrap defect was closed as planned, but the first valid scored run exposed four independent runtime compatibility defects. The run remains immutable and is not retried in this packet. + +## Key Design Decisions + +Preserve the ready preflight and failed scored run as fixed evidence. Classify caller stream variants separately from upstream/provider failures, stage isolated agy settings in the already isolated session HOME, and repair the Anthropic-to-OpenAI Chat field mapping rather than changing credentials or bypassing IOP. + +## Verification Results + +### R1 high-count tests + +```text +exact bootstrap test x20: PASS +bootstrap package x10: PASS +local/remote full dev-runtime suites: PASS +repository Python/Go verification: 433 tests PASS +``` + +### Release qualification and deployment + +```text +release branch: release/dev-936 +release HEAD: cdf91830480ae158703970cfb56c535f1519323b +Edge/mac Node source identity: cdf91830 +GX10/OneXPlayer/RTX5090 rebuilt binaries: deployed +Control Plane: nodes=4 connected=4 providers=8 available_healthy=8 in_flight=0 queued=0 +``` + +### Public preflight + +```text +run_id=run-20260812T055502Z-e359d9d350a8 +ready=9 registration_required=0 implementation_gap=0 +``` + +### Public scored run + +```text +run_id=run-20260812T055534Z-402e1f1d5b30 +completed=9 unresolved=7 success=2 failed=7 timed_out=0 cancelled=0 interrupted=0 running=0 +c01 claude/sonnet direct: parser_error, exit 143 +c02 claude/gemini direct: parser_error, exit 0 +c03 agy/gemini direct: malformed_event, signal 15 +c04 claude/gpt direct: parser_error, exit 1 +c05 codex/gpt direct: lifecycle success, web failed +c06 claude/gemini hybrid: parser_error, exit 1 +c07 agy/gemini hybrid: malformed_event, signal 15 +c08 claude/gpt hybrid: parser_error, exit 1 +c09 codex/gpt hybrid: success, web passed +``` + +## Code Review Result + +- Verdict: FAIL +- Required: 4 +- Suggested: 0 +- Nit: 0 +- Dimension Assessment: + - Correctness: Fail — seven cells do not reach accepted terminal/workspace evidence. + - Completeness: Fail — the required 9/9 benchmark output is absent. + - Test coverage: Fail — real Claude stream variants, isolated agy provider settings, OpenAI Chat max-token mapping, and single-request live failure paths are not covered together. + - API contract: Pass — all provider traffic used the public benchmark/IOP surfaces and the failed run was not retried. + - Code quality: Warn — adapters fail closed, but their closed grammars omit observed official variants. + - Plan deviation: Pass — the scored failure was preserved instead of hidden. + - Verification trust: Pass — identities, preflight, run id, terminal counts, and isolated caller logs agree. +- Findings: + - R1 Required + - Evidence: Claude Sonnet reuses one assistant message id across partial/tool-result cycles; the parser marks the partial id complete on a user event and rejects its next snapshot. Claude Gemini can emit a successful result immediately after a final `stop_reason=null` snapshot, and current terminal usage contains additional documented bookkeeping fields. + - Root Cause: the Claude parser models snapshots as distinct completed messages and requires an `end_turn` assistant event before every result. + - Selected Fix: retain an active message id across tool-result cycles, admit direct successful result from an active final snapshot, allowlist current numeric usage bookkeeping while emitting only benchmark metrics, and add replay regressions. + - Disposition: direct-fix. + - R2 Required + - Evidence: both agy session logs report no isolated `settings.json`, no `modelProvider`, unauthenticated silent auth, then browser OAuth timeout despite process-local Gemini API credentials. + - Root Cause: setting HOME to the isolated session also removes the provider selector required for direct API-key mode. + - Selected Fix: create a secret-free per-attempt `~/.gemini/antigravity-cli/settings.json` with `modelProvider=gemini`, noninteractive tool permission, and telemetry disabled; never copy ambient settings. + - Disposition: direct-fix. + - R3 Required + - Evidence: Claude GPT direct receives an API-error synthetic assistant/result while Codex Responses succeeds on the same IOP GPT route. The Anthropic Chat bridge always sends `max_tokens` to the OpenAI Chat profile. + - Root Cause: the bridge does not apply the OpenAI Chat completion-token field contract and the Claude parser reports the resulting synthetic API error as a parser/model mismatch. + - Selected Fix: use `max_completion_tokens` for the built-in OpenAI Chat profile, retain `max_tokens` for compatible profiles, and classify synthetic Claude API-error terminals as process/upstream failure rather than parser corruption. + - Disposition: direct-fix. + - R4 Required + - Evidence: Gemini hybrid reached Plan and Work, then Review ended as provider failure; GPT hybrid ended in Plan validation while the later GPT hybrid cell succeeded. No retry is allowed for the immutable benchmark run. + - Root Cause: the live single-request path has no diagnostic distinction beyond its closed error class at the caller boundary, and transient provider/dispatch failures terminate the only request attempt. + - Selected Fix: add exact regression coverage for frozen-dispatch mismatch and provider-stage terminal classification, preserve fail-closed no-reselection semantics, then qualify both hybrid routes with a fresh non-scored public smoke before the next scored run. + - Disposition: verification-fix; do not add hidden provider retries. +- Routing Signals: review_rework_count=2, evidence_integrity_failure=false. +- Finalizer: grade-boundary/cloud/G10 + official-review/cloud/G10; risks=temporal_state,boundary_contract,variant_product,stream_protocol. +- Next Step: materialize the follow-up G10 pair and continue automatically. + +## Section Ownership + +Implementation status/checklist/evidence is implementation-owned. Review checklist and verdict are review-only. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log new file mode 100644 index 00000000..b9a757db --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log @@ -0,0 +1,69 @@ + + +# Code Review Reference - REVIEW_TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=2, tag=REVIEW_TEST + +## Implementation Item Completion + +| Item | Status | +|---|---| +| REVIEW_TEST-1 Claude parser | [x] | +| REVIEW_TEST-2 agy settings | [x] | +| REVIEW_TEST-3 OpenAI Chat bridge | [x] | +| REVIEW_TEST-4 hybrid classification | [x] | +| REVIEW_TEST-5 release deploy | [x] | +| REVIEW_TEST-6 public smokes/preflight | [x] | +| REVIEW_TEST-7 scored run/report | [ ] | + +## Implementation Evidence + +- Feature commit `731482ee` passed the 436-test Python adapter suite, the Edge OpenAI package, and the sequential dev-runtime Go suites before release merge. +- Release `04f7c39372ae526a51c5583926aefaf4bdc76394` was clean-built and deployed to Edge and all four Nodes. Control Plane reported `nodes=4`, `connected=4`, `providers=8`, `available_healthy=8`. +- Public preflight `run-20260812T074527Z-b3b7f8bbdde4` reported `ready=9`, `registration_required=0`, `implementation_gap=0`. +- The one plan-2 scored run is `run-20260812T074548Z-d15500c16009`. It retained two successes, four parser/lifecycle failures, and a C01 timeout whose terminal publication could not be reconciled. A public `resume` without `--retry-failed` appended preflight sequence 3 with `ready=9` but correctly rejected the internally inconsistent terminal evidence. +- Exact retained outcomes: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 lifecycle `timed_out` with result exit `143` versus cleanup-receipt exit `null`; C05/C09 `success`. + +## Review-Only Checklist + +- [x] Verify no credential value, ambient caller config, direct provider call, manifest change, or hidden retry was introduced. +- [x] Verify the immutable failed run is untouched and exactly one new scored run exists. +- [ ] Verify all R1-R4 regressions, clean deployment identities, ready=9, success=9, web gates, and Markdown report. +- [x] Append final verdict and routing signals. + +## Section Ownership + +Implementation status/evidence is implementation-owned. Review checklist/verdict is review-only. + +## Code Review Result + +### Overall Verdict + +FAIL + +### Findings + +- Required R1 — Claude terminal projection remains inconsistent with the lifecycle contract. + - Evidence: retained C02 emitted metrics followed by `idle` without `finish`, producing `out_of_order_event`; retained C06/C08 returned an empty parser tuple for synthetic API-error terminals, which the lifecycle correctly classified as `malformed_event`. + - Root Cause: `ClaudeStreamParser._consume_result` cannot distinguish a result that completes an active assistant snapshot from one already preceded by an assistant `finish`, and uses `()` where the parser contract requires `None` for no observation. + - Selected Fix: in `scripts/agent_benchmark/claude_iop.py`, emit `finish,idle` only for result-direct completion, retain `idle` after an already emitted finish, and return `None` for a valid API-error terminal. Add lifecycle-backed regressions for all three sequences. +- Required R2 — agy 1.1.12 official `stream-json` is rejected before the model call can complete. + - Evidence: retained C03 begins with top-level `{"event":"init"}`, then top-level `step_update/state/step_type` events and a `result/status`; the parser requires a nested object at `item[event]` and terminates on the first line. + - Root Cause: `AgyEventParser.parse` and `redact_agy_event` implement a nested payload grammar that does not match the pinned caller's observed top-level stream grammar. + - Selected Fix: in `scripts/agent_benchmark/agy_iop.py`, validate the exact top-level init/step/result fields emitted by 1.1.12, keep content structurally redacted, preserve fail-closed status/usage handling, and add official-shape lifecycle regressions. +- Required R3 — timeout cleanup evidence can be durably self-contradictory. + - Evidence: C01's immutable `lifecycle-result.json` records `terminal_reason=timed_out`, `exit_code=143`, while its authenticated cleanup receipt records the same terminal with `exit_code=null`; public resume therefore returns `benchmark state is unavailable`. + - Root Cause: `_Supervisor.finish` snapshots `child.returncode` before joining the concurrent exit watcher, then writes the receipt; the controller can subsequently consume a later exited frame and publish a different result exit value. + - Selected Fix: in `scripts/agent_benchmark/lifecycle.py`, join the watcher before taking the authoritative child exit snapshot for the receipt, and add a deterministic delayed-watcher timeout/recovery regression proving result/receipt equality. + +### Routing Signals + +- `review_rework_count=3` +- `evidence_integrity_failure=true` + +### Next Step + +Archive this failed pair and materialize the prepared `REVIEW_REVIEW_TEST` follow-up pair. Do not retry or rewrite `run-20260812T074548Z-d15500c16009`. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log new file mode 100644 index 00000000..f3426709 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_3.log @@ -0,0 +1,57 @@ + + +# Code Review Reference - REVIEW_REVIEW_TEST + +## Overview + +date=2026-08-12 +task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=3, tag=REVIEW_REVIEW_TEST + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log` +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log` +- Verdict: `FAIL`; Required R1 Claude result-direct/API-error terminal projection, Required R2 agy top-level official JSONL, Required R3 timeout result/receipt exit-code race. +- Retained run evidence: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 `timed_out` with lifecycle exit `143` and receipt exit `null`; C05/C09 successful. Public resume appended a ready=9 preflight but correctly rejected the inconsistent terminal and did not retry a failed cell. + +## Implementation Item Completion + +| Item | Status | +|---|---| +| REVIEW_REVIEW_TEST-1 Claude terminal projection | [x] | +| REVIEW_REVIEW_TEST-2 agy request/error compatibility | [x] | +| REVIEW_REVIEW_TEST-3 lifecycle exit coherence | [x] | +| REVIEW_REVIEW_TEST-4 deterministic Plan rendering | [x] | +| REVIEW_REVIEW_TEST-5 pinned Claude beta compatibility | [x] | +| REVIEW_REVIEW_TEST-6 provider-independent Plan arrays | [x] | +| REVIEW_REVIEW_TEST-7 complete local verification | [ ] | +| REVIEW_REVIEW_TEST-8 clean release deployment | [ ] | +| REVIEW_REVIEW_TEST-9 public smokes/preflight | [ ] | +| REVIEW_REVIEW_TEST-10 scored run/report | [ ] | + +## Implementation Evidence + +- `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.lifecycle_test` passed 55 tests after R1-R3 implementation. +- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` passed 437 tests. +- `go test -count=1 ./apps/edge/internal/openai` passed with official agy structured-output request coverage. +- Sequential `go test -count=1` passed every package returned by `go list ./apps/control-plane/... ./apps/edge/... ./apps/node/... ./packages/go/...`, excluding only the declared `agenttask` boundary. +- Sanitized replay of retained C02/C06/C08 streams now projects C02 as `finish,idle` and both synthetic API-error cells as no success terminal without parser failure. +- Post-deploy official agy direct and `gemini-hybrid` smokes reached `result.status=SUCCESS`; Claude direct reached `result/subtype=success`. The first Claude hybrid smoke did not repeat the immediate API error, but raw-free Edge metrics exposed three Plan validation failures and the call exceeded the 180-second non-scored boundary, creating R4 before scored execution. +- R4 replaces free-form PlanMD generation with a strict stage-owned `goal`/`steps`/`verification` JSON schema, exact closed-object decoding (including duplicate-key rejection), bounded field validation, and deterministic rendering of the frozen operator template. Focused Plan/template/executor regressions and the complete `apps/edge/internal/openai` package pass. +- The current feature tree passes all 437 Python benchmark tests, every relevant Go package under Control Plane/Edge/Node/shared runtime (excluding only the declared `agenttask` boundary), `git diff --check`, and manifest validation. Clean release-tree repetition remains pending commit/merge. +- After deploying release `1f748bd9bd7fcc843074847a57ed523a28db75fd`, official Claude Code `2.1.228` direct smoke exposed R5 before any scored run: its nominal `result/subtype=success` carried `is_error=true` and a sanitized unsupported-beta 400 for `advisor-tool-2026-03-01`. The scored-run allowance remains unused. +- R5 adds the exact pinned-caller beta to the closed admission inventory and consumes it at the Chat bridge without forwarding it or creating capability authority. Direct Claude Code mapping, marked single-request admission, and unknown-beta rejection regressions pass. +- After the R5 release, official Claude GPT hybrid reached the configured `gpt-5.6-terra` Plan provider twice but returned a caller-visible server error. Raw-free Edge evidence classified both Plan terminals as malformed, proving routing and credential selection succeeded while the remaining R4 Markdown-in-JSON string shape failed provider-independent decoding. +- R6 changes only the private Plan response contract: `steps` and `verification` are bounded arrays of non-empty one-line strings, while Edge deterministically adds Markdown bullets and newlines. It does not change routes, credentials, retries, templates, or any caller-visible schema. + +## Review-Only Checklist + +- [ ] Verify every R1-R6 regression exercises the exact production lifecycle path and remains fail-closed for unknown data. +- [ ] Verify no credential value, ambient caller config, direct provider call, manifest change, hidden retry, or provider reselection was introduced. +- [ ] Verify the prior run is immutable, exactly one plan-3 scored run exists, and all nine attempts plus web gates succeed. +- [ ] Verify clean build/deployment identities, 4/4 node and 8/8 provider health, ready=9, and report pointer consistency. +- [ ] Append final verdict and routing signals, then complete the mandated archive/next-state action. + +## Section Ownership + +Implementation status/evidence is implementation-owned. Review checklist/verdict is review-only. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G09_0.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G09_0.log new file mode 100644 index 00000000..5811a211 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G09_0.log @@ -0,0 +1,137 @@ + + +# Dev runtime recovery and scored comparison rerun + +## For the Implementing Agent + +Start only after child 07 has archived PASS evidence. Restore the existing managed dev runtime, use only the public benchmark CLI, store emitted ids in the exact pointer files, and fill `CODE_REVIEW-cloud-G10.md`. Do not invoke callers/providers directly, reuse a scored run id, overwrite evidence, ask the user, archive logs, or write `complete.log`. + +## Background + +The adapter fixes are not proven until the managed dev endpoint is available and a fresh preflight reports all nine cells ready. A new append-only scored run must then prove every caller can create the required website artifacts. + +## Archive Evidence Snapshot + +- Old terminal run remains at `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df` and must not be reused or overwritten. +- A later read-only preflight created `run-20260812T050756Z-18293db0b83f` with ready=0 because the current shell had no runtime endpoint variables and no local managed listener. +- Child 07 must supply the corrected caller contracts before this plan begins. + +## Analysis + +### Files Read + +- `scripts/agent_comparison_benchmark.py` +- `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` +- `docs/agent-comparison-benchmark-dev-guide.md` +- `agent-test/local/rules.md` +- `agent-test/local/testing-smoke.md` +- `agent-ops/rules/project/domain/testing/rules.md` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` +- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md` + +### SDD Criteria + +SDD is `READY`; milestone-task ids are all five comparison tasks. S04-S08 and Evidence Map rows require C01-C09 event/timing/usage/workspace evidence, so PASS requires ready=9 followed by exactly one fresh scored run whose nine cells pass terminal and web gates. + +### Verification Context + +Declared runner is this local checkout; managed dev runtime inventory resolves to SSH `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, branch `dev`, CP status port 18001 and edge ports 18082-18084/19093. Current local preflight is blocked by unavailable endpoint environment. Before mutation, inspect remote branch/HEAD/dirty state, process/service ownership, config, listeners, and health without exposing secret values. Restore the existing deployed runtime/config rather than changing IOP source. The public CLI is the only allowed benchmark/provider execution surface. + +### External Verification Preflight + +- Runner/workdir: `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, dirty feature worktree; caller binaries are installed for Linux aarch64. +- Remote: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`; exact current HEAD/dirty/service state must be captured before restart. +- Credentials/CA: protected files under `token/`; never print values. Export only in the benchmark command process according to the dev guide. +- Resume gate: remote health succeeds and public CLI preflight emits `ready=9`, `registration_required=0`, `implementation_gap=0`. + +### Test Coverage Gaps + +Local tests cannot prove external endpoint health, caller installation, TLS, or generated files. This plan closes those gaps with preflight and one fresh scored run. + +### Symbol References + +None; this packet changes no production symbols. + +### Split Judgment + +This is dependent child 08. Its stable contract is external full-cycle evidence and its PASS oracle is a fresh all-success run. Directory dependency `+07` requires archived `07+06_caller_write_contracts/complete.log`; implementation must not start while it is missing. + +### Scope Rationale + +Do not deploy new IOP source, alter routes/models/task content, invoke direct callers/providers, retry a scored run, score subjective quality, or modify roadmap state. If the new run exposes a new code defect, preserve evidence and return it to review for a concrete follow-up. + +### Final Routing + +- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair` +- build closures all true; scores `1/2/2/2/2` => G09; base/route `grade-boundary`; filename `PLAN-cloud-G09.md` +- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; filename `CODE_REVIEW-cloud-G10.md` +- large_indivisible_context=false; positive risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent. + +## Implementation Checklist + +- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source. +- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9. +- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results. +- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [TEST-1] Runtime recovery + +**Problem:** The current public preflight reports ready=0 because the declared managed dev endpoint/config is unavailable to the runner. + +**Solution:** Verify the child-07 completion log, inspect remote git/runtime state read-only, then restart/rebuild only the existing deployed managed services needed to restore health. Bind local command environment from protected `token/` files without logging values. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record remote preflight/recovery evidence. + +**Test Strategy:** Use remote health/listener and public CLI preflight only; no direct provider call. + +**Verification:** Remote health succeeds and the public preflight gate can execute. + +### [TEST-2] Fresh ready=9 preflight + +**Problem:** A scored run is forbidden until every matrix cell has proven connectivity/effective binding. + +**Solution:** Execute the manifest preflight once runtime inputs are present and store only the emitted id in `preflight_id.log`. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact emitted preflight id. + +**Test Strategy:** Assert summary `ready=9 registration_required=0 implementation_gap=0`. + +**Verification:** `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0. + +### [TEST-3] One scored 9-cell rerun + +**Problem:** The old run proves process termination but not successful benchmark artifacts. + +**Solution:** Execute exactly one new public `run`, store its id, and inspect immutable result/workspace validation evidence. Do not retry that id. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact emitted scored run id. + +**Test Strategy:** Require 9 success, 0 failed/interrupted/running and all required HTML/CSS/JS workspace gates PASS. + +**Verification:** `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0 and emitted run evidence satisfies the stated counts. + +## Dependencies and Execution Order + +`07+06_caller_write_contracts` must first produce an archived `complete.log`. Then perform TEST-1, TEST-2, and TEST-3 in order. No dependency beyond index 07 is implied. + +## Modified Files Summary + +| File | Item | +|---|---| +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | TEST-2 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | TEST-3 | + +## Final Verification + +1. `test -f agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log` — exits 0. +2. Public preflight command above — exits 0 with ready=9. +3. Public run command above — executes exactly once and exits 0 with nine passing terminal/web results. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log new file mode 100644 index 00000000..235c9330 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_1.log @@ -0,0 +1,151 @@ + + +# Bootstrap qualification fix and comparison rerun + +## For the Implementing Agent + +Implement the selected R1 fix exactly, run every verification, then resume the clean release deployment, ready=9 preflight, and exactly one scored run. Fill `CODE_REVIEW-cloud-G10.md`, keep active files in place, and do not create `complete.log` or archive files. + +## Background + +The first live-run plan correctly stopped twice: a preflight found the initial clean release lacked the feature, then the feature-bearing clean release exposed two failure modes in one bootstrap test. The reviewer closed both to stale test-only timeout/cleanup code; production runtime behavior is not changed. + +## Archive Evidence Snapshot + +- `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`: FAIL with R1 after no scored run. +- Preflight-only evidence `run-20260812T053517Z-6b4f4de67128`: all cells `model_missing` on pre-feature clean release; preserve, do not reuse. +- Release branch `release/dev-936` now contains clean feature merge `b244b6e224a9d44c09a1e7af49c0788f597c89fe`. + +## Finding Resolution Map + +| ID | Evidence | Root Cause | Selected Fix | Mode | Changed precondition | Acceptance | +|---|---|---|---|---|---|---| +| R1 | remote full suite register timeout; package run duplicate-Start unexpectedly succeeded | obsolete 2s register bound plus cleanup duplicate Start cancels active lifetime | use `fakeNodeHandshakeTimeout` for all fake register calls and cleanup with one Stop | direct-fix | test no longer mutates lifecycle during cleanup and uses the established remote-safe bound | exact test x20, package x10, full suite, rebuild/deploy, ready=9, one scored run | + +## Analysis + +### Files Read + +- `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` +- `apps/edge/internal/bootstrap/runtime.go` +- `apps/edge/internal/transport/server.go` +- `apps/edge/internal/transport/integration_test.go` +- `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_0.log` +- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` + +### SDD Criteria + +SDD remains READY; milestone-task ids and S04-S08 are unchanged. The live evidence gate remains ready=9 followed by nine successful terminal/workspace results. + +### Verification Context + +Remote runner is `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, release `dev-936`. No scored run has started. Four Nodes were restored connected before the new release test failed. Token files remain process-local and values must not be printed. + +### Test Coverage Gaps + +R1 is covered by high-count exact/package runs. External endpoint and workspace output remain covered only by the resumed public preflight/run. + +### Symbol References + +None; no production or public symbol changes. + +### Split Judgment + +Keep the fix and resumed deployment in this follow-up because the fixed qualification is the ordered precondition for the same all-cell execution invariant. Child 07 remains satisfied by its archived `complete.log`; no new sibling dependency is introduced. + +### Scope Rationale + +Exclude production Runtime behavior, routes, model aliases, task content, scoring, and roadmap mutation. Only the broken test lifecycle/timeout, evidence pointer files, and active review may change. + +### Final Routing + +- evaluation_mode=`isolated-reassessment`; finalizer=`finalize-task-policy.sh pair` +- build closures all true; scores `2/2/2/2/2` => G10; route `grade-boundary`; `PLAN-cloud-G10.md` +- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; `CODE_REVIEW-cloud-G10.md` +- large_indivisible_context=false; risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=1; evidence_integrity_failure=false; capability gap absent. + +## Implementation Checklist + +- [x] [REVIEW_TEST-1] Apply R1 test-only lifecycle/timeout fix and pass high-count local/remote regressions. +- [x] [REVIEW_TEST-2] Complete clean release tests, four-binary rebuild/redeploy, and 4-node healthy verification. +- [x] [REVIEW_TEST-3] Produce a fresh public ready=9 preflight id. +- [x] [REVIEW_TEST-4] Execute exactly one public scored run and record nine-cell terminal/workspace results. +- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output. + +### [REVIEW_TEST-1] Close bootstrap test nondeterminism + +**Problem:** `runtime_refresh_node_test.go:517-524` restarts Runtime in cleanup, and register requests in the file retain a shorter timeout than the declared fake-node handshake contract. + +**Solution:** Replace the duplicate Start cleanup with the standard Stop-only form used by neighboring tests. Replace all four fake register `2*time.Second` bounds with `fakeNodeHandshakeTimeout`; do not add retries or production changes. + +**Modified Files and Checklist:** + +- [ ] `apps/edge/internal/bootstrap/runtime_refresh_node_test.go`: exact R1 test-only changes. + +**Test Strategy:** Fresh high-count exact and package tests prove both observed modes are absent. + +**Verification:** `go test -count=20 -run '^TestRefreshConfigApplyNoChangeSkipsNodePush$' ./apps/edge/internal/bootstrap` and `go test -count=10 ./apps/edge/internal/bootstrap` exit 0. + +### [REVIEW_TEST-2] Release qualification and deployment + +**Problem:** The current deployed d40e release is clean but lacks the benchmark feature; b244 source has not passed qualification or been deployed. + +**Solution:** Commit/push R1, merge it into `release/dev-936`, sync the runner, run full pre/post-build suites, rebuild four binaries, config check/dry-run, deploy Edge and all four Nodes, then require connected/healthy state. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: exact release/deployment evidence. + +**Test Strategy:** Apply dev-runtime deploy contract; any failed required stage stops scored execution. + +**Verification:** clean source/build identity matches release HEAD and CP reports 4 connected Nodes with healthy providers. + +### [REVIEW_TEST-3] Fresh public preflight + +**Problem:** The old preflight belongs to the pre-feature release. + +**Solution:** With protected environment values, call public CLI preflight and store only emitted id. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact id. + +**Test Strategy:** Require `ready=9 registration_required=0 implementation_gap=0`. + +**Verification:** public preflight exits 0. + +### [REVIEW_TEST-4] One scored run + +**Problem:** No valid generated benchmark output exists. + +**Solution:** Invoke public run exactly once after ready=9, store id, and inspect immutable evidence. + +**Modified Files and Checklist:** + +- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact id. + +**Test Strategy:** Require 9 success and all web artifact gates PASS; do not retry on failure. + +**Verification:** public run exits 0 with required counts. + +## Dependencies and Execution Order + +Child 07 PASS remains satisfied. Execute REVIEW_TEST-1 through REVIEW_TEST-4 strictly in order. + +## Modified Files Summary + +| File | Item | +|---|---| +| `apps/edge/internal/bootstrap/runtime_refresh_node_test.go` | REVIEW_TEST-1 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | REVIEW_TEST-2, REVIEW_TEST-3, REVIEW_TEST-4 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | REVIEW_TEST-3 | +| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | REVIEW_TEST-4 | + +## Final Verification + +1. High-count commands in REVIEW_TEST-1 exit 0. +2. Full dev-runtime pre/post suites and build identity checks pass. +3. Public preflight exits 0 with ready=9. +4. Exactly one public run exits 0 with 9 success and all artifact gates. + +After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`. diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log new file mode 100644 index 00000000..ec31affe --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log @@ -0,0 +1,56 @@ + + +# Caller/provider compatibility fixes and comparison rerun + +## For the Implementing Agent + +Resolve R1-R4 without changing benchmark inputs, credentials, route aliases, scoring, or the immutable failed run. Use only the public benchmark CLI for live execution. After local qualification, rebuild and deploy the clean release, run one non-scored hybrid smoke, require ready=9, and execute exactly one new scored run. + +## Background + +The clean release and ready=9 preflight succeeded. The first scored run `run-20260812T055534Z-402e1f1d5b30` completed nine cells but exposed official caller stream variants, missing isolated agy provider settings, an OpenAI Chat bridge field mismatch, and hybrid stage failures. The evidence is valid and must not be retried or rewritten. + +## Finding Resolution Map + +| ID | Root Cause | Selected Fix | Acceptance | +|---|---|---|---| +| R1 | Claude snapshots/result grammar is narrower than observed official output | fix active-id/result/usage parsing and replay tests | real-stream replays plus adapter suite pass | +| R2 | isolated agy HOME lacks required direct Gemini provider selector | stage exact secret-free per-session settings | invocation tests and official noninteractive smoke pass | +| R3 | Anthropic bridge uses generic `max_tokens` for OpenAI Chat; synthetic API errors become parser errors | profile-aware token field and Claude API-error classification | bridge/parser regressions plus direct GPT smoke pass | +| R4 | hybrid failures lack sufficient deterministic regression qualification | exact fail-closed classification tests and live non-scored route smokes; no hidden retry | Gemini/GPT hybrid smoke succeeds before scored run | + +## Scope + +- `scripts/agent_benchmark/claude_iop.py` and tests +- `scripts/agent_benchmark/agy_iop.py` and tests +- `apps/edge/internal/openai/anthropic_bridge.go` and tests +- existing single-request provider/plan/review tests only when needed for R4 classification proof +- active review and id pointer files + +Do not modify central common rules/skills, tokens, manifest/task/rubric, provider credentials, route aliases, or archived run evidence. + +## Implementation Checklist + +- [ ] [REVIEW_TEST-1] Implement R1 Claude stream compatibility and replay regressions. +- [ ] [REVIEW_TEST-2] Implement R2 isolated agy provider settings and invocation regressions. +- [ ] [REVIEW_TEST-3] Implement R3 OpenAI Chat bridge mapping and synthetic API-error classification. +- [ ] [REVIEW_TEST-4] Add R4 single-request failure classification regressions without retries/reselection. +- [ ] [REVIEW_TEST-5] Run local full verification, commit/push, merge clean release, rebuild/deploy all runtime binaries, and verify four-node health. +- [ ] [REVIEW_TEST-6] Run fresh public non-scored direct/hybrid smokes and ready=9 preflight. +- [ ] [REVIEW_TEST-7] Execute exactly one new scored run, record the new id, and generate the human-readable Markdown report. +- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized evidence. + +## Verification + +1. Python benchmark adapter unit/integration suite passes. +2. Edge OpenAI package and dev-runtime full Go suites pass. +3. Clean release build identities match on Edge and all Nodes; Control Plane shows 4/4 connected and 8/8 healthy. +4. Direct GPT and both hybrid public smokes succeed, then public preflight reports ready=9. +5. Exactly one new scored run reports success=9 and all web gates pass; public report command writes Markdown. + +## Final Routing + +- finalizer=`finalize-task-policy.sh pair` +- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10` +- review_rework_count=2; evidence_integrity_failure=false +- risks=`temporal_state,boundary_contract,variant_product,stream_protocol` diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log new file mode 100644 index 00000000..d344dfdb --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_3.log @@ -0,0 +1,113 @@ + + +# Official caller terminal compatibility and evidence coherence + +## For the Implementing Agent + +Implement R1-R6 exactly as selected below. Do not alter benchmark inputs, route aliases, credentials, scoring, or retained run evidence. Use only the public benchmark CLI or its documented official-caller smoke path for live caller/provider execution. Run deterministic local qualification first, then clean-build and deploy the release, run non-scored direct/hybrid smokes, require ready=9, and execute exactly one fresh scored run. Fill the implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with sanitized actual evidence and leave both active files in place for review. + +## Background + +Plan 2 fixed the first live compatibility set and deployed clean release `04f7c39372ae526a51c5583926aefaf4bdc76394`. Public preflight was ready=9, but the single scored run `run-20260812T074548Z-d15500c16009` exposed three narrower boundary defects: Claude result terminal sequencing, official agy planner request compatibility plus error-terminal classification, and a supervisor exit-watcher/receipt race. The run is immutable and must not be retried or rewritten. + +## Archive Evidence Snapshot + +- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log` +- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log` +- Verdict: `FAIL`; Required R1 Claude result-direct/API-error terminal projection, Required R2 agy top-level official JSONL, Required R3 timeout result/receipt exit-code race. +- Retained run evidence: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 `timed_out` with lifecycle exit `143` and receipt exit `null`; C05/C09 successful. Public resume appended a ready=9 preflight but correctly rejected the inconsistent terminal and did not retry a failed cell. + +## Finding Resolution Map + +| ID | Reviewer Evidence | Root Cause | Selected Fix | Mode | Acceptance | +|---|---|---|---|---|---| +| R1 | C02 emitted `idle` without `finish`; C06/C08 valid API-error terminals returned an empty tuple | Claude result handling does not distinguish result-direct completion and violates the no-observation parser contract | emit `finish,idle` for direct completion, `idle` after prior finish, and `None` for a valid API-error terminal; add lifecycle-backed tests | direct-fix | Claude adapter tests and retained-shape replay pass without out-of-order/malformed classification | +| R2 | C03's structurally redacted evidence initially showed only top-level keys, but the retained raw stream and caller log prove the payload is nested and the model request first failed with Edge `400 INVALID_ARGUMENT: request body is invalid` | Edge's pinned Gemini request decoder lacks the official agy planner's structured-output request fields; the adapter then maps the caller's nested `result.status=ERROR` to `malformed_event` instead of preserving the upstream/process failure | extend the Gemini bridge's exact structured-output request contract and tests; treat a valid agy ERROR result as non-success terminal evidence without parser failure | direct-fix | Gemini bridge structured-output regression and agy error lifecycle regression pass; non-scored official agy direct smoke succeeds | +| R3 | C01 result exit `143` differs from authenticated receipt exit `null` | supervisor snapshots returncode before joining the concurrent exit watcher | join watcher/IO, refresh the authoritative child returncode, then write the one receipt; add deterministic race/recovery regression | direct-fix | timeout/recovery test proves result and receipt exit/signal equality and public resume can reconcile | +| R4 | Post-deploy raw-free metrics recorded three `plan/validation` failures among six marked single-request calls; Claude hybrid did not complete before the non-scored 180-second boundary | Plan stage asks the provider for free-form PlanMD and rejects harmless format drift after the model call | request a stage-owned strict JSON object (`goal`, `steps`, `verification`), validate the bounded fields, and render the configured PlanMD template deterministically inside Edge | direct-fix | repeated plan-stage fixtures cannot create malformed PlanMD; official Claude hybrid smoke reaches a success terminal within the scored timeout budget | +| R5 | After the R4 release deployment, pinned Claude Code `2.1.228` returned `result/subtype=success` with `is_error=true` and `API Error: 400 unsupported anthropic-beta "advisor-tool-2026-03-01"` before either direct or hybrid model execution | the exact official caller now emits a compatibility beta absent from Edge's closed admission set | admit and consume `advisor-tool-2026-03-01` without forwarding it or granting any route/tool/workspace authority; retain rejection for every unknown beta | direct-fix | direct and marked-preset header regressions pass; official Claude direct and Gemini hybrid smokes terminate with `is_error=false` | +| R6 | After the R5 release deployment, official Claude `gpt-hybrid` reached the configured GPT Plan provider twice but returned `is_error=true`; raw-free Edge terminal evidence classified both Plan responses as `malformed` | R4 still makes the model serialize Markdown bullet prefixes and newline layout inside two JSON string fields, leaving a provider-format dependency after structured output succeeds | make `steps` and `verification` bounded arrays of non-empty one-line strings in the closed response schema, then let Edge own bullet prefixes and newline rendering | direct-fix | array-schema/template regressions pass; official Claude GPT hybrid terminates with `is_error=false` without a retry or route change | + +## Modified Files Summary + +- `scripts/agent_benchmark/claude_iop.py` +- `scripts/agent_benchmark/claude_iop_test.py` +- `scripts/agent_benchmark/agy_iop.py` +- `scripts/agent_benchmark/agy_iop_test.py` +- `scripts/agent_benchmark/lifecycle.py` +- `scripts/agent_benchmark/lifecycle_test.py` and/or the existing attempts recovery test module containing the closest deterministic oracle +- `apps/edge/internal/openai/gemini_types.go` +- `apps/edge/internal/openai/gemini_handler.go` +- `apps/edge/internal/openai/gemini_handler_test.go` +- `apps/edge/internal/openai/single_request_plan_stage.go` +- `apps/edge/internal/openai/single_request_plan_stage_test.go` +- `apps/edge/internal/openai/single_request_provider_stage.go` +- `apps/edge/internal/openai/single_request_executor_test.go` +- `packages/go/singlerequesttemplate/template.go` +- `packages/go/singlerequesttemplate/template_test.go` +- `apps/edge/internal/openai/anthropic_types.go` +- `apps/edge/internal/openai/anthropic_bridge_test.go` +- `apps/edge/internal/openai/single_request_handler_test.go` +- this active review plus existing run/preflight/report pointer files after successful fresh execution + +## Implementation Checklist + +- [x] [REVIEW_REVIEW_TEST-1] Implement R1 Claude terminal projection and lifecycle-backed regressions. +- [x] [REVIEW_REVIEW_TEST-2] Implement R2 official agy structured-output request compatibility and error-terminal classification regressions. +- [x] [REVIEW_REVIEW_TEST-3] Implement R3 authoritative supervisor exit snapshot and timeout/recovery regression. +- [x] [REVIEW_REVIEW_TEST-4] Implement R4 schema-bound Plan fields and deterministic configured-template rendering. +- [x] [REVIEW_REVIEW_TEST-5] Implement R5 pinned Claude beta compatibility without forwarding or widening internal authority. +- [x] [REVIEW_REVIEW_TEST-6] Implement R6 provider-independent array fields and Edge-owned Plan bullet rendering. +- [ ] [REVIEW_REVIEW_TEST-7] Run the complete Python benchmark suite and relevant Go/runtime regressions from a clean tree. +- [ ] [REVIEW_REVIEW_TEST-8] Commit/push, merge the clean release branch, rebuild/deploy Edge and every Node, and verify 4/4 nodes plus 8/8 healthy providers. +- [ ] [REVIEW_REVIEW_TEST-9] Run public non-scored direct and hybrid smoke coverage and a fresh ready=9 preflight. +- [ ] [REVIEW_REVIEW_TEST-10] Execute exactly one fresh scored run, require success=9 and all web gates, update existing id pointers, and generate the human-readable Markdown report. +- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized commands, release identities, deployment health, run id, terminal summary, and report path. + +## Verification + +1. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` passes. +2. Targeted Claude/agy/lifecycle/attempt recovery regressions pass on the benchmark runner's supported Python. +3. Clean release validation/build tests pass and all deployed binaries report the same clean VCS revision. +4. Control Plane reports four connected nodes and eight available healthy providers. +5. Public non-scored direct/hybrid smokes succeed, then public preflight reports ready=9. +6. Exactly one plan-3 scored run reports success=9 with all required web validations, and the public report command writes Markdown. + +## Constraints and Exclusions + +- Never print, persist, or pass provider source keys to callers. Read the managed benchmark principal only from `token/.iop-bench` into a process-local variable. +- Do not mutate `agent-test/runs/bench-02/run-20260812T074548Z-d15500c16009` or use `--retry-failed` on it. +- Do not change manifest/task/rubric/checksums, route models/efforts/stages, retry policy, or failure classification to manufacture success. +- Do not add a retry, provider reselection, alternate route, or direct provider invocation. +- Parser compatibility must remain exact to the pinned official caller version and fail closed on unknown structures. + +## Analysis + +### Outcome and Acceptance + +The task is complete only when deterministic regressions prove the six boundary corrections, the clean release is deployed across the managed dev runtime, public readiness is 9/9, and one fresh immutable scored run is 9/9 with a generated Markdown report. + +### Scope and Ownership + +The caller adapters own their exact JSONL interpretation and redaction. The Gemini ingress owns the exact official agy request-to-Chat conversion. The lifecycle supervisor owns the authoritative process exit projection shared by its result and receipt. Deployment and benchmark commands only consume these contracts; they do not reinterpret them. + +### Split Judgment + +Keep one plan because the scored-run acceptance depends on one indivisible evidence invariant: every caller terminal must project through the shared lifecycle into a coherent immutable attempt before the same release can be accepted. Splitting would require multiple scored reruns or accept an invalid intermediate evidence state. + +### Routing + +- finalizer=`finalize-task-policy.sh`, mode=`pair` +- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10` +- risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` +- `review_rework_count=3`; `evidence_integrity_failure=true` + +## Dependencies and Execution Order + +R1-R6 are independent code fixes but must all pass local regressions before one clean release is built. Deployment must finish before non-scored smokes; smokes and ready=9 must pass before the only fresh scored run. + +## Final Routing + +- finalizer=`finalize-task-policy.sh pair` +- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10` +- catalog routes=`worker/cloud/G10`, `review/cloud/G10` diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log new file mode 100644 index 00000000..1aaf34dc --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log @@ -0,0 +1 @@ +run-20260812T055502Z-e359d9d350a8 diff --git a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log new file mode 100644 index 00000000..c991ad74 --- /dev/null +++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log @@ -0,0 +1 @@ +run-20260812T055534Z-402e1f1d5b30 diff --git a/apps/edge/internal/bootstrap/runtime_refresh_node_test.go b/apps/edge/internal/bootstrap/runtime_refresh_node_test.go index ee154f6a..f464b4e0 100644 --- a/apps/edge/internal/bootstrap/runtime_refresh_node_test.go +++ b/apps/edge/internal/bootstrap/runtime_refresh_node_test.go @@ -183,7 +183,7 @@ nodes: regResp, err := toki.SendRequestTyped[*iop.RegisterRequest, *iop.RegisterResponse]( &fakeNode.Communicator, &iop.RegisterRequest{Token: "tok-refresh"}, - 2*time.Second, + fakeNodeHandshakeTimeout, ) if err != nil { t.Fatalf("register: %v", err) @@ -402,7 +402,7 @@ nodes: regResp, err := toki.SendRequestTyped[*iop.RegisterRequest, *iop.RegisterResponse]( &fakeNode.Communicator, &iop.RegisterRequest{Token: "tok-conn"}, - 2*time.Second, + fakeNodeHandshakeTimeout, ) if err != nil { t.Fatalf("register: %v", err) @@ -514,13 +514,8 @@ nodes: t.Fatalf("Start: %v", err) } defer func() { - if err := rt.Start(context.Background()); err == nil { - t.Fatal("expected already started error") - } else { - // runtime is running, stop it - if err := rt.Stop(); err != nil { - t.Fatalf("Stop: %v", err) - } + if err := rt.Stop(); err != nil { + t.Fatalf("Stop: %v", err) } }() @@ -569,7 +564,7 @@ nodes: regResp, err := toki.SendRequestTyped[*iop.RegisterRequest, *iop.RegisterResponse]( &fakeNode.Communicator, &iop.RegisterRequest{Token: "tok-noc-change"}, - 2*time.Second, + fakeNodeHandshakeTimeout, ) if err != nil { t.Fatalf("register: %v", err) @@ -878,7 +873,7 @@ nodes: regResp, err := toki.SendRequestTyped[*iop.RegisterRequest, *iop.RegisterResponse]( &fakeNode.Communicator, &iop.RegisterRequest{Token: "tok-rt"}, - 2*time.Second, + fakeNodeHandshakeTimeout, ) if err != nil { t.Fatalf("register: %v", err) diff --git a/apps/edge/internal/openai/anthropic_bridge.go b/apps/edge/internal/openai/anthropic_bridge.go index a79f341b..75be6955 100644 --- a/apps/edge/internal/openai/anthropic_bridge.go +++ b/apps/edge/internal/openai/anthropic_bridge.go @@ -96,11 +96,18 @@ func prepareAnthropicChatBridge(body []byte, target string, profile config.Concr } chat := map[string]any{ - "model": target, - "messages": messages, - "max_tokens": *req.MaxTokens, - "stream": req.Stream, + "model": target, + "messages": messages, + "stream": req.Stream, } + maxTokensField := "max_tokens" + if profile.ID == "openai" { + // OpenAI's current Chat completion models use the completion-specific + // field. Other OpenAI-compatible profiles retain their native legacy + // spelling instead of inheriting an OpenAI-only request contract. + maxTokensField = "max_completion_tokens" + } + chat[maxTokensField] = *req.MaxTokens if req.Stream { chat["stream_options"] = map[string]any{"include_usage": true} } diff --git a/apps/edge/internal/openai/anthropic_bridge_test.go b/apps/edge/internal/openai/anthropic_bridge_test.go index bd4ac7da..975e237f 100644 --- a/apps/edge/internal/openai/anthropic_bridge_test.go +++ b/apps/edge/internal/openai/anthropic_bridge_test.go @@ -59,7 +59,7 @@ func TestAnthropicChatBridgeMixedContentToolsAndResponse(t *testing.T) { if err := json.Unmarshal(bodies[0], &chat); err != nil { t.Fatal(err) } - if chat["model"] != "served-chat" || chat["max_tokens"] != float64(256) || chat["stream"] != false { + if chat["model"] != "served-chat" || chat["max_completion_tokens"] != float64(256) || chat["max_tokens"] != nil || chat["stream"] != false { t.Fatalf("Chat request envelope mismatch: %+v", chat) } messages := anthropicAnySlice(t, chat["messages"]) @@ -98,6 +98,27 @@ func TestAnthropicChatBridgeMixedContentToolsAndResponse(t *testing.T) { } } +func TestAnthropicChatBridgeKeepsCompatibleMaxTokensField(t *testing.T) { + profile, err := config.ResolveProtocolProfile("gemini", "", config.BuiltInProtocolProfileCatalog()) + if err != nil { + t.Fatal(err) + } + body, _, err := prepareAnthropicChatBridge( + []byte(`{"model":"claude-route","max_tokens":64,"messages":[{"role":"user","content":"hello"}]}`), + "served-chat", profile, + ) + if err != nil { + t.Fatal(err) + } + var chat map[string]any + if err := json.Unmarshal(body, &chat); err != nil { + t.Fatal(err) + } + if chat["max_tokens"] != float64(64) || chat["max_completion_tokens"] != nil { + t.Fatalf("compatible Chat max token field mismatch: %+v", chat) + } +} + func TestAnthropicChatBridgeThinkingCapabilityAndResponse(t *testing.T) { candidate := anthropicTestCandidate(t, "openai") candidate.ActualModel = "served-chat" @@ -379,6 +400,7 @@ func TestAnthropicChatBridgeClaudeCodeRequest(t *testing.T) { req := newAnthropicRequest(http.MethodPost, "/v1/messages", body) req.Header.Set(anthropicBetaHeader, strings.Join([]string{ "advanced-tool-use-2025-11-20", + "advisor-tool-2026-03-01", "claude-code-20250219", "context-management-2025-06-27", "interleaved-thinking-2025-05-14", diff --git a/apps/edge/internal/openai/anthropic_types.go b/apps/edge/internal/openai/anthropic_types.go index 80d6bf73..b2439270 100644 --- a/apps/edge/internal/openai/anthropic_types.go +++ b/apps/edge/internal/openai/anthropic_types.go @@ -18,6 +18,7 @@ const ( var supportedAnthropicBetas = map[string]struct{}{ "advanced-tool-use-2025-11-20": {}, + "advisor-tool-2026-03-01": {}, "claude-code-20250219": {}, "context-management-2025-06-27": {}, "effort-2025-11-24": {}, diff --git a/apps/edge/internal/openai/chat_decode.go b/apps/edge/internal/openai/chat_decode.go index 5ef76bf3..41c56399 100644 --- a/apps/edge/internal/openai/chat_decode.go +++ b/apps/edge/internal/openai/chat_decode.go @@ -16,7 +16,7 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest) } for key := range raw { switch key { - case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning": + case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body": default: return fmt.Errorf("%s is not supported for /v1/chat/completions", key) } @@ -40,6 +40,9 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest) if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) { return fmt.Errorf("top_p must be between 0 and 1") } + if req.TopK != nil && *req.TopK <= 0 { + return fmt.Errorf("top_k must be greater than zero") + } if err := validateThinkControl(req); err != nil { return err } @@ -83,7 +86,7 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe } for key := range raw { switch key { - case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning": + case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body": default: // Unknown fields are tolerated for provider-pool passthrough. } @@ -108,6 +111,9 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) { return fmt.Errorf("top_p must be between 0 and 1") } + if req.TopK != nil && *req.TopK <= 0 { + return fmt.Errorf("top_k must be greater than zero") + } if err := validateThinkControl(req); err != nil { return err } diff --git a/apps/edge/internal/openai/chat_handler.go b/apps/edge/internal/openai/chat_handler.go index 6aae441b..12024bfd 100644 --- a/apps/edge/internal/openai/chat_handler.go +++ b/apps/edge/internal/openai/chat_handler.go @@ -102,6 +102,20 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusBadRequest, "invalid_request_error", "messages are required") return } + if dispatch.SingleRequest != nil { + capability, ok := s.service.(singleRequestService) + if !ok { + writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable") + return + } + recordSingleRequestIngress() + if req.Stream { + s.handleChatSingleRequestStream(w, r, capability, dispatch, rawBody) + } else { + s.handleChatSingleRequest(w, r, capability, dispatch, rawBody) + } + return + } outputPolicy := s.resolveOutputPolicy(basePrompt) // The mutable model-catalog generation policy is applied here, at the single // ingress point, before req is frozen into the dispatch context. Every stage diff --git a/apps/edge/internal/openai/chat_types.go b/apps/edge/internal/openai/chat_types.go index fa78bc51..52fce4c8 100644 --- a/apps/edge/internal/openai/chat_types.go +++ b/apps/edge/internal/openai/chat_types.go @@ -14,6 +14,7 @@ type chatCompletionRequest struct { MaxCompletionTokens *int `json:"max_completion_tokens,omitempty"` Temperature *float64 `json:"temperature,omitempty"` TopP *float64 `json:"top_p,omitempty"` + TopK *int `json:"top_k,omitempty"` PresencePenalty *float64 `json:"presence_penalty,omitempty"` FrequencyPenalty *float64 `json:"frequency_penalty,omitempty"` Seed *int `json:"seed,omitempty"` @@ -28,6 +29,7 @@ type chatCompletionRequest struct { ReasoningEffort *string `json:"reasoning_effort,omitempty"` ThinkingTokenBudget *int `json:"thinking_token_budget,omitempty"` IncludeReasoning *bool `json:"include_reasoning,omitempty"` + ExtraBody any `json:"extra_body,omitempty"` } type chatMessage struct { @@ -76,6 +78,7 @@ func (req chatCompletionRequest) runInput(prompt string, messages []chatMessage, } setOptionFloat(options, "temperature", req.Temperature) setOptionFloat(options, "top_p", req.TopP) + setOptionInt(options, "top_k", req.TopK) setOptionFloat(options, "presence_penalty", req.PresencePenalty) setOptionFloat(options, "frequency_penalty", req.FrequencyPenalty) setOptionInt(options, "seed", req.Seed) diff --git a/apps/edge/internal/openai/gemini_bridge.go b/apps/edge/internal/openai/gemini_bridge.go new file mode 100644 index 00000000..c5aacef9 --- /dev/null +++ b/apps/edge/internal/openai/gemini_bridge.go @@ -0,0 +1,310 @@ +package openai + +import ( + "bytes" + "encoding/json" + "fmt" + "net/http" + "sort" + "strings" +) + +type geminiBridgeResponseWriter struct { + target http.ResponseWriter + header http.Header + status int + committed bool + buffer bytes.Buffer + stream *geminiBridgeStream +} + +func newGeminiBridgeResponseWriter(target http.ResponseWriter, model string) *geminiBridgeResponseWriter { + return &geminiBridgeResponseWriter{ + target: target, header: make(http.Header), stream: newGeminiBridgeStream(target, model), + } +} + +func (w *geminiBridgeResponseWriter) Header() http.Header { return w.header } + +func (w *geminiBridgeResponseWriter) WriteHeader(status int) { + if w.status == 0 { + w.status = status + } +} + +func (w *geminiBridgeResponseWriter) Write(payload []byte) (int, error) { + if w.status == 0 { + w.status = http.StatusOK + } + if w.status >= http.StatusBadRequest { + return w.buffer.Write(payload) + } + w.commit() + if err := w.stream.Feed(payload); err != nil { + _ = w.stream.Error("upstream stream could not be translated") + return 0, err + } + return len(payload), nil +} + +func (w *geminiBridgeResponseWriter) Flush() { + if w.status == 0 { + w.status = http.StatusOK + } + if w.status < http.StatusBadRequest { + w.commit() + } + if flusher, ok := w.target.(http.Flusher); ok { + flusher.Flush() + } +} + +func (w *geminiBridgeResponseWriter) commit() { + if w.committed { + return + } + w.target.Header().Set("Content-Type", "text/event-stream") + w.target.Header().Set("Cache-Control", "no-cache") + w.target.Header().Del("Content-Length") + w.target.WriteHeader(http.StatusOK) + w.committed = true +} + +func (w *geminiBridgeResponseWriter) Finish() { + if w.status == 0 { + writeGeminiError(w.target, http.StatusBadGateway, "UNAVAILABLE", "runtime request failed") + return + } + if w.status >= http.StatusBadRequest { + status, code := geminiHTTPError(w.status) + writeGeminiError(w.target, status, code, geminiSafeErrorMessage(status)) + return + } + w.commit() + if err := w.stream.Finish(); err != nil { + _ = w.stream.Error("upstream stream could not be translated") + } + if flusher, ok := w.target.(http.Flusher); ok { + flusher.Flush() + } +} + +func geminiHTTPError(status int) (int, string) { + switch { + case status == http.StatusUnauthorized || status == http.StatusForbidden: + return http.StatusUnauthorized, "UNAUTHENTICATED" + case status >= 400 && status < 500: + return http.StatusBadRequest, "INVALID_ARGUMENT" + default: + return http.StatusBadGateway, "UNAVAILABLE" + } +} + +func geminiSafeErrorMessage(status int) string { + if status == http.StatusUnauthorized { + return "authentication failed" + } + if status == http.StatusBadRequest { + return "request is invalid" + } + return "runtime request failed" +} + +type geminiBridgeToolState struct { + name string + thoughtSignature string + arguments strings.Builder +} + +type geminiBridgeStream struct { + w http.ResponseWriter + model string + pendingSSE []byte + tools map[int]*geminiBridgeToolState + finish string + usage map[string]int + stopped bool + errored bool +} + +func newGeminiBridgeStream(w http.ResponseWriter, model string) *geminiBridgeStream { + return &geminiBridgeStream{w: w, model: model, tools: make(map[int]*geminiBridgeToolState), usage: make(map[string]int)} +} + +func (s *geminiBridgeStream) Feed(chunk []byte) error { + if s.stopped { + return nil + } + s.pendingSSE = append(s.pendingSSE, chunk...) + s.pendingSSE = bytes.ReplaceAll(s.pendingSSE, []byte("\r\n"), []byte("\n")) + for { + index := bytes.Index(s.pendingSSE, []byte("\n\n")) + if index < 0 { + return nil + } + event := append([]byte(nil), s.pendingSSE[:index]...) + s.pendingSSE = s.pendingSSE[index+2:] + if err := s.consumeSSEEvent(event); err != nil { + return err + } + if s.stopped { + s.pendingSSE = nil + return nil + } + } +} + +func (s *geminiBridgeStream) consumeSSEEvent(event []byte) error { + var lines [][]byte + for _, line := range bytes.Split(event, []byte("\n")) { + line = bytes.TrimSpace(line) + if bytes.HasPrefix(line, []byte("data:")) { + lines = append(lines, bytes.TrimSpace(bytes.TrimPrefix(line, []byte("data:")))) + } + } + if len(lines) == 0 { + return nil + } + payload := bytes.Join(lines, []byte("\n")) + if bytes.Equal(payload, []byte("[DONE]")) { + return s.Finish() + } + var chunk geminiChatStreamChunk + if err := json.Unmarshal(payload, &chunk); err != nil { + return fmt.Errorf("decode Chat SSE: %w", err) + } + if chunk.Error != nil { + return s.Error("upstream request failed") + } + if chunk.Usage != nil { + setGeminiUsage(s.usage, "promptTokenCount", chunk.Usage.PromptTokens) + setGeminiUsage(s.usage, "candidatesTokenCount", chunk.Usage.CompletionTokens) + setGeminiUsage(s.usage, "totalTokenCount", chunk.Usage.TotalTokens) + if chunk.Usage.PromptTokensDetails != nil { + setGeminiUsage(s.usage, "cachedContentTokenCount", chunk.Usage.PromptTokensDetails.CachedTokens) + } + if chunk.Usage.CompletionTokensDetails != nil { + setGeminiUsage(s.usage, "thoughtsTokenCount", chunk.Usage.CompletionTokensDetails.ReasoningTokens) + } + } + for _, choice := range chunk.Choices { + reasoning := choice.Delta.ReasoningContent + if reasoning == "" { + reasoning = choice.Delta.Reasoning + } + if reasoning != "" { + if err := s.emitParts([]any{map[string]any{"text": reasoning, "thought": true}}, ""); err != nil { + return err + } + } + if choice.Delta.Content != "" { + if err := s.emitParts([]any{map[string]any{"text": choice.Delta.Content}}, ""); err != nil { + return err + } + } + for _, delta := range choice.Delta.ToolCalls { + state := s.tools[delta.Index] + if state == nil { + state = &geminiBridgeToolState{} + s.tools[delta.Index] = state + } + if delta.Function.Name != "" { + state.name = delta.Function.Name + } + if delta.ExtraContent.Google != nil && delta.ExtraContent.Google.ThoughtSignature != "" { + state.thoughtSignature = delta.ExtraContent.Google.ThoughtSignature + } + if state.arguments.Len()+len(delta.Function.Arguments) > geminiToolArgumentLimit { + return fmt.Errorf("tool arguments exceed limit") + } + state.arguments.WriteString(delta.Function.Arguments) + } + if choice.FinishReason != nil { + s.finish = *choice.FinishReason + } + } + return nil +} + +func setGeminiUsage(target map[string]int, key string, value *int) { + if value != nil { + target[key] = *value + } +} + +func (s *geminiBridgeStream) emitParts(parts []any, finish string) error { + candidate := map[string]any{"content": map[string]any{"role": "model", "parts": parts}} + if finish != "" { + candidate["finishReason"] = finish + } + payload := map[string]any{"candidates": []any{candidate}, "modelVersion": s.model} + if len(s.usage) > 0 { + payload["usageMetadata"] = s.usage + } + return writeGeminiSSE(s.w, payload) +} + +func (s *geminiBridgeStream) emitTools() error { + if len(s.tools) == 0 { + return nil + } + indices := make([]int, 0, len(s.tools)) + for index := range s.tools { + indices = append(indices, index) + } + sort.Ints(indices) + parts := make([]any, 0, len(indices)) + for _, index := range indices { + state := s.tools[index] + if !geminiPathToken.MatchString(state.name) { + return fmt.Errorf("tool name is invalid") + } + var args map[string]any + if json.Unmarshal([]byte(state.arguments.String()), &args) != nil { + return fmt.Errorf("tool arguments are invalid") + } + part := map[string]any{"functionCall": map[string]any{"name": state.name, "args": args}} + if state.thoughtSignature != "" { + part["thoughtSignature"] = state.thoughtSignature + } + parts = append(parts, part) + } + return s.emitParts(parts, "") +} + +func (s *geminiBridgeStream) Finish() error { + if s.stopped { + return nil + } + if len(bytes.TrimSpace(s.pendingSSE)) > 0 { + return fmt.Errorf("truncated Chat SSE") + } + if err := s.emitTools(); err != nil { + return err + } + finish := "STOP" + if s.finish == "length" { + finish = "MAX_TOKENS" + } + s.stopped = true + return s.emitParts([]any{}, finish) +} + +func (s *geminiBridgeStream) Error(message string) error { + if s.errored || s.stopped { + return nil + } + s.errored, s.stopped = true, true + return writeGeminiSSE(s.w, geminiErrorResponse{Error: geminiErrorBody{ + Code: http.StatusBadGateway, Message: message, Status: "UNAVAILABLE", + }}) +} + +func writeGeminiSSE(w http.ResponseWriter, value any) error { + payload, err := json.Marshal(value) + if err != nil { + return err + } + _, err = fmt.Fprintf(w, "data: %s\n\n", payload) + return err +} diff --git a/apps/edge/internal/openai/gemini_handler.go b/apps/edge/internal/openai/gemini_handler.go new file mode 100644 index 00000000..2a57c496 --- /dev/null +++ b/apps/edge/internal/openai/gemini_handler.go @@ -0,0 +1,462 @@ +package openai + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "net/http" + "regexp" + "strings" +) + +var geminiPathToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`) + +func isGeminiRequest(r *http.Request) bool { + return r != nil && strings.HasPrefix(r.URL.Path, geminiPathPrefix) +} + +func writeGeminiError(w http.ResponseWriter, status int, code, message string) { + writeJSON(w, status, geminiErrorResponse{Error: geminiErrorBody{ + Code: status, Message: message, Status: code, + }}) +} + +func (s *Server) handleGeminiStreamGenerateContent(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + writeGeminiError(w, http.StatusMethodNotAllowed, "INVALID_ARGUMENT", "method not allowed") + return + } + routeID, callerModel, err := parseGeminiStreamPath(r) + if err != nil { + writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request path is invalid") + return + } + defer r.Body.Close() + body, err := readOpenAIIngressBody(w, r, s.maxIngressSnapshotBytes()) + if err != nil { + writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") + return + } + chatBody, err := prepareGeminiChatBridge(body, routeID) + if err != nil { + writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") + return + } + + internal := r.Clone(r.Context()) + internal.URL.Path = "/v1/chat/completions" + internal.URL.RawPath = "" + internal.URL.RawQuery = "" + internal.RequestURI = "/v1/chat/completions" + internal.Body = io.NopCloser(bytes.NewReader(chatBody)) + internal.ContentLength = int64(len(chatBody)) + internal.Header = r.Header.Clone() + internal.Header.Del("Authorization") + internal.Header.Del("X-Goog-Api-Key") + internal.Header.Set("Content-Type", "application/json") + + bridge := newGeminiBridgeResponseWriter(w, callerModel) + s.handleChatCompletions(bridge, internal) + bridge.Finish() +} + +func parseGeminiStreamPath(r *http.Request) (string, string, error) { + if r == nil || r.URL == nil { + return "", "", fmt.Errorf("missing URL") + } + parts := strings.Split(strings.TrimPrefix(r.URL.Path, geminiPathPrefix), "/") + if len(parts) != 4 || parts[1] != "v1beta" || parts[2] != "models" { + return "", "", fmt.Errorf("unexpected path") + } + const suffix = ":streamGenerateContent" + if !strings.HasSuffix(parts[3], suffix) { + return "", "", fmt.Errorf("unexpected method") + } + routeID := parts[0] + callerModel := strings.TrimSuffix(parts[3], suffix) + if !geminiPathToken.MatchString(routeID) || !geminiPathToken.MatchString(callerModel) { + return "", "", fmt.Errorf("invalid path token") + } + query := r.URL.Query() + if len(query) != 1 || len(query["alt"]) != 1 || query.Get("alt") != "sse" { + return "", "", fmt.Errorf("alt=sse is required") + } + return routeID, callerModel, nil +} + +func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) { + if err := validateJSONMembers(body); err != nil { + return nil, err + } + decoder := json.NewDecoder(bytes.NewReader(body)) + decoder.DisallowUnknownFields() + var req geminiRequest + if err := decoder.Decode(&req); err != nil { + return nil, err + } + if err := requireGeminiJSONEOF(decoder); err != nil { + return nil, err + } + if len(req.Contents) == 0 { + return nil, fmt.Errorf("contents are required") + } + chat := map[string]any{ + "model": routeID, "stream": true, + "stream_options": map[string]any{"include_usage": true}, + } + messages := make([]map[string]any, 0, len(req.Contents)+1) + if req.SystemInstruction != nil { + text, err := geminiTextOnly(*req.SystemInstruction) + if err != nil || strings.TrimSpace(text) == "" { + return nil, fmt.Errorf("systemInstruction is invalid") + } + messages = append(messages, map[string]any{"role": "system", "content": text}) + } + pendingCalls := make(map[string][]string) + for contentIndex, content := range req.Contents { + converted, err := geminiContentToChat(content, contentIndex, pendingCalls) + if err != nil { + return nil, err + } + messages = append(messages, converted...) + } + chat["messages"] = messages + if config := req.GenerationConfig; config != nil { + if config.CandidateCount != nil && *config.CandidateCount != 1 { + return nil, fmt.Errorf("candidateCount must be one") + } + if config.MaxOutputTokens != nil { + if *config.MaxOutputTokens <= 0 { + return nil, fmt.Errorf("maxOutputTokens must be positive") + } + chat["max_tokens"] = *config.MaxOutputTokens + } + if len(config.StopSequences) > 0 { + chat["stop"] = config.StopSequences + } + if config.Temperature != nil { + if *config.Temperature < 0 || *config.Temperature > 2 { + return nil, fmt.Errorf("temperature is invalid") + } + } + if config.TopK != nil { + if *config.TopK <= 0 { + return nil, fmt.Errorf("topK is invalid") + } + } + if config.TopP != nil { + if *config.TopP < 0 || *config.TopP > 1 { + return nil, fmt.Errorf("topP is invalid") + } + } + if thinking := config.ThinkingConfig; thinking != nil { + googleThinking := make(map[string]any) + if thinking.IncludeThoughts != nil { + googleThinking["include_thoughts"] = *thinking.IncludeThoughts + } + if thinking.ThinkingBudget != nil { + if *thinking.ThinkingBudget < -1 { + return nil, fmt.Errorf("thinkingBudget is invalid") + } + googleThinking["thinking_budget"] = *thinking.ThinkingBudget + } + if len(googleThinking) > 0 { + // Gemini's OpenAI-compatible endpoint accepts native options only + // below extra_body.google; generic think/include_reasoning fields + // are rejected by that endpoint. + chat["extra_body"] = map[string]any{ + "google": map[string]any{"thinking_config": googleThinking}, + } + } + } + responseSchema, err := geminiExclusiveSchema(config.ResponseSchema, config.ResponseJSONSchema) + if err != nil { + return nil, fmt.Errorf("response schema is invalid") + } + switch config.ResponseMIMEType { + case "": + if responseSchema != nil { + return nil, fmt.Errorf("response MIME type is required") + } + case "text/plain": + if responseSchema != nil { + return nil, fmt.Errorf("text response schema is invalid") + } + case "application/json": + if responseSchema == nil { + chat["response_format"] = map[string]any{"type": "json_object"} + } else { + chat["response_format"] = map[string]any{ + "type": "json_schema", + "json_schema": map[string]any{ + "name": "agy_response", "strict": true, "schema": responseSchema, + }, + } + } + default: + return nil, fmt.Errorf("response MIME type is invalid") + } + } + if len(req.Tools) > 0 { + tools := make([]map[string]any, 0) + for _, group := range req.Tools { + if len(group.FunctionDeclarations) == 0 { + return nil, fmt.Errorf("functionDeclarations are required") + } + for _, declaration := range group.FunctionDeclarations { + if !geminiPathToken.MatchString(declaration.Name) { + return nil, fmt.Errorf("function declaration is invalid") + } + schema, err := geminiExclusiveSchema( + declaration.Parameters, declaration.ParametersJSONSchema, + ) + if err != nil || schema == nil { + return nil, fmt.Errorf("function schema is invalid") + } + if _, err := geminiExclusiveSchema( + declaration.Response, declaration.ResponseJSONSchema, + ); err != nil { + return nil, fmt.Errorf("function response schema is invalid") + } + function := map[string]any{"name": declaration.Name, "parameters": schema} + if declaration.Description != "" { + function["description"] = declaration.Description + } + tools = append(tools, map[string]any{"type": "function", "function": function}) + } + } + chat["tools"] = tools + } + if req.ToolConfig != nil && req.ToolConfig.FunctionCallingConfig != nil { + switch strings.ToUpper(req.ToolConfig.FunctionCallingConfig.Mode) { + case "", "AUTO": + chat["tool_choice"] = "auto" + case "ANY": + chat["tool_choice"] = "required" + case "NONE": + chat["tool_choice"] = "none" + default: + return nil, fmt.Errorf("function calling mode is invalid") + } + } + return json.Marshal(chat) +} + +func geminiExclusiveSchema(first, second json.RawMessage) (map[string]any, error) { + if len(first) > 0 && len(second) > 0 { + return nil, fmt.Errorf("schema alternatives conflict") + } + raw := first + if len(raw) == 0 { + raw = second + } + if len(raw) == 0 { + return nil, nil + } + var schema map[string]any + if err := json.Unmarshal(raw, &schema); err != nil || schema == nil { + return nil, fmt.Errorf("schema is not an object") + } + if err := normalizeGeminiSchemaTypes(schema); err != nil { + return nil, err + } + return schema, nil +} + +func normalizeGeminiSchemaTypes(value any) error { + switch typed := value.(type) { + case map[string]any: + for key, item := range typed { + if key == "type" { + name, ok := item.(string) + if !ok { + return fmt.Errorf("schema type is invalid") + } + normalized := strings.ToLower(name) + switch normalized { + case "null", "boolean", "object", "array", "number", "integer", "string": + typed[key] = normalized + default: + return fmt.Errorf("schema type is invalid") + } + continue + } + if err := normalizeGeminiSchemaTypes(item); err != nil { + return err + } + } + case []any: + for _, item := range typed { + if err := normalizeGeminiSchemaTypes(item); err != nil { + return err + } + } + } + return nil +} + +func geminiTextOnly(content geminiContent) (string, error) { + // Gemini represents systemInstruction as Content and official agy 1.1.12 + // labels that Content with the API-native "user" role. + if content.Role != "" && content.Role != "user" { + return "", fmt.Errorf("role is invalid") + } + texts := make([]string, 0, len(content.Parts)) + for _, part := range content.Parts { + if part.Text == nil || part.FunctionCall != nil || part.FunctionResponse != nil || part.Thought || part.ThoughtSignature != "" { + return "", fmt.Errorf("only text is supported") + } + texts = append(texts, *part.Text) + } + return strings.Join(texts, "\n"), nil +} + +func geminiContentToChat(content geminiContent, contentIndex int, pending map[string][]string) ([]map[string]any, error) { + role := strings.ToLower(strings.TrimSpace(content.Role)) + if role != "user" && role != "model" { + return nil, fmt.Errorf("content role is invalid") + } + if len(content.Parts) == 0 { + return nil, fmt.Errorf("content parts are required") + } + var texts []string + var reasoning []string + var toolCalls []any + var toolMessages []map[string]any + for partIndex, part := range content.Parts { + set := 0 + if part.Text != nil { + set++ + } + if part.FunctionCall != nil { + set++ + } + if part.FunctionResponse != nil { + set++ + } + if set != 1 { + return nil, fmt.Errorf("content part is invalid") + } + if part.Text != nil { + if role == "model" && part.Thought { + reasoning = append(reasoning, *part.Text) + } else if part.Thought { + return nil, fmt.Errorf("user thought is invalid") + } else { + texts = append(texts, *part.Text) + } + continue + } + if part.FunctionCall != nil { + if role != "model" || !geminiPathToken.MatchString(part.FunctionCall.Name) { + return nil, fmt.Errorf("functionCall is invalid") + } + var args map[string]any + if json.Unmarshal(part.FunctionCall.Args, &args) != nil { + return nil, fmt.Errorf("functionCall args are invalid") + } + callID := fmt.Sprintf("gemini_call_%d_%d", contentIndex, partIndex) + pending[part.FunctionCall.Name] = append(pending[part.FunctionCall.Name], callID) + call := map[string]any{"id": callID, "type": "function", "function": map[string]any{"name": part.FunctionCall.Name, "arguments": string(part.FunctionCall.Args)}} + if part.ThoughtSignature != "" { + call["extra_content"] = openAIChatThoughtSignature(part.ThoughtSignature) + } + toolCalls = append(toolCalls, call) + continue + } + response := part.FunctionResponse + if role != "user" || !geminiPathToken.MatchString(response.Name) { + return nil, fmt.Errorf("functionResponse is invalid") + } + ids := pending[response.Name] + if len(ids) == 0 || !json.Valid(response.Response) { + return nil, fmt.Errorf("functionResponse has no matching call") + } + callID := ids[0] + pending[response.Name] = ids[1:] + toolMessages = append(toolMessages, map[string]any{"role": "tool", "tool_call_id": callID, "tool_name": response.Name, "content": string(response.Response)}) + } + if role == "model" { + message := map[string]any{"role": "assistant", "content": strings.Join(texts, "\n")} + if len(reasoning) > 0 { + message["reasoning_content"] = strings.Join(reasoning, "") + } + if len(toolCalls) > 0 { + message["tool_calls"] = toolCalls + } + return []map[string]any{message}, nil + } + if len(texts) > 0 { + toolMessages = append(toolMessages, map[string]any{"role": "user", "content": strings.Join(texts, "\n")}) + } + if len(toolMessages) == 0 { + return nil, fmt.Errorf("user content is empty") + } + return toolMessages, nil +} + +func validateJSONMembers(raw []byte) error { + dec := json.NewDecoder(bytes.NewReader(raw)) + if err := validateGeminiJSONValue(dec); err != nil { + return err + } + return requireGeminiJSONEOF(dec) +} + +func validateGeminiJSONValue(dec *json.Decoder) error { + token, err := dec.Token() + if err != nil { + return err + } + delim, ok := token.(json.Delim) + if !ok { + return nil + } + switch delim { + case '{': + seen := map[string]struct{}{} + for dec.More() { + keyToken, err := dec.Token() + if err != nil { + return err + } + key, ok := keyToken.(string) + if !ok { + return fmt.Errorf("object key is invalid") + } + if _, duplicate := seen[key]; duplicate { + return fmt.Errorf("duplicate member") + } + seen[key] = struct{}{} + if err := validateGeminiJSONValue(dec); err != nil { + return err + } + } + closeToken, err := dec.Token() + if err != nil || closeToken != json.Delim('}') { + return fmt.Errorf("object is invalid") + } + case '[': + for dec.More() { + if err := validateGeminiJSONValue(dec); err != nil { + return err + } + } + closeToken, err := dec.Token() + if err != nil || closeToken != json.Delim(']') { + return fmt.Errorf("array is invalid") + } + default: + return fmt.Errorf("JSON delimiter is invalid") + } + return nil +} + +func requireGeminiJSONEOF(dec *json.Decoder) error { + var extra any + if err := dec.Decode(&extra); err != io.EOF { + return fmt.Errorf("trailing JSON value") + } + return nil +} diff --git a/apps/edge/internal/openai/gemini_handler_test.go b/apps/edge/internal/openai/gemini_handler_test.go new file mode 100644 index 00000000..ed8b2757 --- /dev/null +++ b/apps/edge/internal/openai/gemini_handler_test.go @@ -0,0 +1,216 @@ +package openai + +import ( + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "iop/packages/go/config" + iop "iop/proto/gen/iop" +) + +func TestGeminiIngressAuthenticatesAndStreamsThroughChatRoute(t *testing.T) { + fake := &fakeRunService{events: bufferedRunEvents( + &iop.RunEvent{Type: "delta", Delta: "hello"}, + &iop.RunEvent{Type: "complete", Usage: &iop.Usage{InputTokens: 3, OutputTokens: 2}, Metadata: map[string]string{"finish_reason": "length"}}, + )} + srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama", Target: "provider-model"}, fake, nil) + body := `{ + "systemInstruction":{"role":"user","parts":[{"text":"be concise"}]}, + "contents":[{"role":"user","parts":[{"text":"say hello"}]}], + "generationConfig":{"candidateCount":1,"maxOutputTokens":16,"temperature":0.2,"topK":8,"topP":0.9,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}} + }` + req := httptest.NewRequest(http.MethodPost, "/gemini/gemini-direct/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", strings.NewReader(body)) + req.Header.Set("X-Goog-Api-Key", "iop-principal") + w := httptest.NewRecorder() + srv.routes().ServeHTTP(w, req) + + if w.Code != http.StatusOK || w.Header().Get("Content-Type") != "text/event-stream" { + t.Fatalf("status/header: %d %q body=%s", w.Code, w.Header().Get("Content-Type"), w.Body.String()) + } + if fake.req.ModelGroupKey != "gemini-direct" || fake.req.Target != "provider-model" { + t.Fatalf("route bypassed Chat admission: %+v", fake.req) + } + if !strings.Contains(fake.req.Prompt, "system: be concise") || !strings.Contains(fake.req.Prompt, "user: say hello") { + t.Fatalf("prompt conversion mismatch: %q", fake.req.Prompt) + } + options := fake.req.Input["options"].(map[string]any) + if options["max_tokens"] != 16 { + t.Fatalf("generation config mismatch: input=%+v", fake.req.Input) + } + if _, exists := options["top_k"]; exists { + t.Fatalf("deprecated Gemini sampling option reached Chat upstream: input=%+v", fake.req.Input) + } + response := w.Body.String() + for _, want := range []string{`"text":"hello"`, `"finishReason":"MAX_TOKENS"`} { + if !strings.Contains(response, want) { + t.Fatalf("missing %s in %s", want, response) + } + } + if strings.Contains(response, "iop-principal") { + t.Fatal("principal token leaked to response") + } +} + +func TestGeminiIngressRejectsAuthenticationAndShapeBeforeDispatch(t *testing.T) { + base := `{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}` + for _, tc := range []struct { + name, path, body, bearer, key string + status int + }{ + {"missing key", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "", "", 401}, + {"conflicting auth", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "Bearer other", "iop-principal", 401}, + {"wrong query", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent", base, "", "iop-principal", 400}, + {"duplicate member", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[],"contents":[]}`, "", "iop-principal", 400}, + {"two candidates", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"candidateCount":2}}`, "", "iop-principal", 400}, + {"invalid thinking budget", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"thinkingConfig":{"thinkingBudget":-2}}}`, "", "iop-principal", 400}, + } { + t.Run(tc.name, func(t *testing.T) { + fake := &fakeRunService{} + srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama"}, fake, nil) + req := httptest.NewRequest(http.MethodPost, tc.path, strings.NewReader(tc.body)) + if tc.bearer != "" { + req.Header.Set("Authorization", tc.bearer) + } + if tc.key != "" { + req.Header.Set("X-Goog-Api-Key", tc.key) + } + w := httptest.NewRecorder() + srv.routes().ServeHTTP(w, req) + if w.Code != tc.status { + t.Fatalf("status=%d body=%s", w.Code, w.Body.String()) + } + var failure geminiErrorResponse + if json.Unmarshal(w.Body.Bytes(), &failure) != nil || failure.Error.Code != tc.status { + t.Fatalf("not a Gemini error: %s", w.Body.String()) + } + if fake.req.ModelGroupKey != "" { + t.Fatalf("unexpected dispatch: %+v", fake.req) + } + }) + } +} + +func TestGeminiRequestBridgePreservesToolsAndThoughtSignature(t *testing.T) { + body := []byte(`{ + "contents":[ + {"role":"model","parts":[{"text":"thinking","thought":true},{"functionCall":{"name":"lookup","args":{"q":"x"}},"thoughtSignature":"opaque"}]}, + {"role":"user","parts":[{"functionResponse":{"name":"lookup","response":{"value":1}}}]}, + {"role":"user","parts":[{"text":"continue"}]} + ], + "tools":[{"functionDeclarations":[{"name":"lookup","description":"find","parametersJsonSchema":{"type":"object"}}]}], + "toolConfig":{"functionCallingConfig":{"mode":"ANY"}} + }`) + converted, err := prepareGeminiChatBridge(body, "preset-gemini-hybrid") + if err != nil { + t.Fatal(err) + } + text := string(converted) + for _, want := range []string{`"model":"preset-gemini-hybrid"`, `"reasoning_content":"thinking"`, `"thought_signature":"opaque"`, `"tool_call_id":"gemini_call_0_1"`, `"tool_choice":"required"`} { + if !strings.Contains(text, want) { + t.Fatalf("missing %s in %s", want, text) + } + } +} + +func TestGeminiRequestBridgeUsesProviderNativeThinkingEnvelope(t *testing.T) { + converted, err := prepareGeminiChatBridge([]byte(`{ + "contents":[{"role":"user","parts":[{"text":"hello"}]}], + "generationConfig":{"temperature":1,"topK":50,"topP":1,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}} + }`), "gemini-3.6-flash") + if err != nil { + t.Fatal(err) + } + var body map[string]any + if json.Unmarshal(converted, &body) != nil { + t.Fatal("converted body is invalid") + } + for _, forbidden := range []string{"temperature", "top_k", "top_p", "think", "include_reasoning", "thinking_token_budget"} { + if _, exists := body[forbidden]; exists { + t.Fatalf("unsupported field %q in converted body: %s", forbidden, converted) + } + } + extra := body["extra_body"].(map[string]any) + google := extra["google"].(map[string]any) + thinking := google["thinking_config"].(map[string]any) + if thinking["include_thoughts"] != true || thinking["thinking_budget"] != float64(-1) { + t.Fatalf("thinking envelope mismatch: %+v", thinking) + } +} + +func TestGeminiRequestBridgeAcceptsOfficialAgyPlannerStructuredOutput(t *testing.T) { + converted, err := prepareGeminiChatBridge([]byte(`{ + "contents":[{"role":"user","parts":[{"text":"plan"}]}], + "generationConfig":{ + "responseMimeType":"application/json", + "responseJsonSchema":{"type":"OBJECT","properties":{"steps":{"type":"ARRAY"}},"required":["steps"]} + }, + "tools":[{"functionDeclarations":[{ + "name":"read_file","parameters":{"type":"OBJECT","properties":{"path":{"type":"STRING"}}}, + "responseJsonSchema":{"type":"object"} + }]}] + }`), "gemini-3.6-flash") + if err != nil { + t.Fatal(err) + } + var body map[string]any + if json.Unmarshal(converted, &body) != nil { + t.Fatal("converted body is invalid") + } + format := body["response_format"].(map[string]any) + jsonSchema := format["json_schema"].(map[string]any) + schema := jsonSchema["schema"].(map[string]any) + if format["type"] != "json_schema" || jsonSchema["name"] != "agy_response" || jsonSchema["strict"] != true || schema["type"] != "object" { + t.Fatalf("structured output mismatch: %+v", format) + } + tools := body["tools"].([]any) + function := tools[0].(map[string]any)["function"].(map[string]any) + if function["name"] != "read_file" { + t.Fatalf("function conversion mismatch: %+v", function) + } +} + +func TestGeminiRequestBridgeRejectsConflictingStructuredSchemas(t *testing.T) { + for _, body := range []string{ + `{"contents":[{"role":"user","parts":[{"text":"x"}]}],"generationConfig":{"responseMimeType":"application/json","responseSchema":{"type":"object"},"responseJsonSchema":{"type":"object"}}}`, + `{"contents":[{"role":"user","parts":[{"text":"x"}]}],"tools":[{"functionDeclarations":[{"name":"f","parameters":{"type":"object"},"parametersJsonSchema":{"type":"object"}}]}]}`, + } { + if _, err := prepareGeminiChatBridge([]byte(body), "gemini-3.6-flash"); err == nil { + t.Fatal("conflicting schema alternatives must fail") + } + } +} + +func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) { + w := httptest.NewRecorder() + stream := newGeminiBridgeStream(w, "gemini-3.6-flash") + input := "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\"}}]}}]}\n\n" + + "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"x\\\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\n" + + "data: [DONE]\n\n" + if err := stream.Feed([]byte(input)); err != nil { + t.Fatal(err) + } + if strings.Count(w.Body.String(), `"functionCall"`) != 1 || !strings.Contains(w.Body.String(), `"finishReason":"STOP"`) { + t.Fatalf("tool terminal mismatch: %s", w.Body.String()) + } + usageWriter := httptest.NewRecorder() + usageStream := newGeminiBridgeStream(usageWriter, "m") + if err := usageStream.Feed([]byte("data: {\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":2,\"total_tokens\":5,\"prompt_tokens_details\":{\"cached_tokens\":1},\"completion_tokens_details\":{\"reasoning_tokens\":1}}}\n\ndata: [DONE]\n\n")); err != nil { + t.Fatal(err) + } + for _, want := range []string{`"promptTokenCount":3`, `"candidatesTokenCount":2`, `"cachedContentTokenCount":1`, `"thoughtsTokenCount":1`, `"totalTokenCount":5`} { + if !strings.Contains(usageWriter.Body.String(), want) { + t.Fatalf("missing usage %s in %s", want, usageWriter.Body.String()) + } + } + oversize := newGeminiBridgeStream(httptest.NewRecorder(), "m") + state := &geminiBridgeToolState{name: "lookup"} + state.arguments.WriteString(strings.Repeat("x", geminiToolArgumentLimit)) + oversize.tools[0] = state + chunk := `data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"x"}}]}}]}` + "\n\n" + if err := oversize.Feed([]byte(chunk)); err == nil { + t.Fatal("oversize tool arguments must fail") + } +} diff --git a/apps/edge/internal/openai/gemini_types.go b/apps/edge/internal/openai/gemini_types.go new file mode 100644 index 00000000..b21bdbc1 --- /dev/null +++ b/apps/edge/internal/openai/gemini_types.go @@ -0,0 +1,123 @@ +package openai + +import "encoding/json" + +const ( + geminiPathPrefix = "/gemini/" + geminiToolArgumentLimit = 1 << 20 +) + +type geminiRequest struct { + Contents []geminiContent `json:"contents"` + SystemInstruction *geminiContent `json:"systemInstruction,omitempty"` + GenerationConfig *geminiGenerationConfig `json:"generationConfig,omitempty"` + Tools []geminiTool `json:"tools,omitempty"` + ToolConfig *geminiToolConfig `json:"toolConfig,omitempty"` +} + +type geminiContent struct { + Role string `json:"role,omitempty"` + Parts []geminiPart `json:"parts"` +} + +type geminiPart struct { + Text *string `json:"text,omitempty"` + Thought bool `json:"thought,omitempty"` + ThoughtSignature string `json:"thoughtSignature,omitempty"` + FunctionCall *geminiFunctionCall `json:"functionCall,omitempty"` + FunctionResponse *geminiFunctionResponse `json:"functionResponse,omitempty"` +} + +type geminiFunctionCall struct { + Name string `json:"name"` + Args json.RawMessage `json:"args"` +} + +type geminiFunctionResponse struct { + Name string `json:"name"` + Response json.RawMessage `json:"response"` +} + +type geminiGenerationConfig struct { + CandidateCount *int `json:"candidateCount,omitempty"` + MaxOutputTokens *int `json:"maxOutputTokens,omitempty"` + StopSequences []string `json:"stopSequences,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopK *int `json:"topK,omitempty"` + TopP *float64 `json:"topP,omitempty"` + ThinkingConfig *geminiThinkingConfig `json:"thinkingConfig,omitempty"` + ResponseMIMEType string `json:"responseMimeType,omitempty"` + ResponseSchema json.RawMessage `json:"responseSchema,omitempty"` + ResponseJSONSchema json.RawMessage `json:"responseJsonSchema,omitempty"` +} + +type geminiThinkingConfig struct { + IncludeThoughts *bool `json:"includeThoughts,omitempty"` + ThinkingBudget *int `json:"thinkingBudget,omitempty"` +} + +type geminiTool struct { + FunctionDeclarations []geminiFunctionDeclaration `json:"functionDeclarations"` +} + +type geminiFunctionDeclaration struct { + Name string `json:"name"` + Description string `json:"description,omitempty"` + Parameters json.RawMessage `json:"parameters,omitempty"` + ParametersJSONSchema json.RawMessage `json:"parametersJsonSchema,omitempty"` + Response json.RawMessage `json:"response,omitempty"` + ResponseJSONSchema json.RawMessage `json:"responseJsonSchema,omitempty"` +} + +type geminiToolConfig struct { + FunctionCallingConfig *geminiFunctionCallingConfig `json:"functionCallingConfig,omitempty"` +} + +type geminiFunctionCallingConfig struct { + Mode string `json:"mode"` +} + +type geminiErrorResponse struct { + Error geminiErrorBody `json:"error"` +} + +type geminiErrorBody struct { + Code int `json:"code"` + Message string `json:"message"` + Status string `json:"status"` +} + +type geminiChatStreamChunk struct { + Choices []struct { + Delta struct { + Content string `json:"content"` + ReasoningContent string `json:"reasoning_content"` + Reasoning string `json:"reasoning"` + ToolCalls []struct { + Index int `json:"index"` + ID string `json:"id"` + ExtraContent openAIChatToolExtraContent `json:"extra_content,omitempty"` + Function struct { + Name string `json:"name"` + Arguments string `json:"arguments"` + } `json:"function"` + } `json:"tool_calls"` + } `json:"delta"` + FinishReason *string `json:"finish_reason"` + } `json:"choices"` + Usage *struct { + PromptTokens *int `json:"prompt_tokens,omitempty"` + CompletionTokens *int `json:"completion_tokens,omitempty"` + TotalTokens *int `json:"total_tokens,omitempty"` + PromptTokensDetails *struct { + CachedTokens *int `json:"cached_tokens,omitempty"` + } `json:"prompt_tokens_details,omitempty"` + CompletionTokensDetails *struct { + ReasoningTokens *int `json:"reasoning_tokens,omitempty"` + } `json:"completion_tokens_details,omitempty"` + } `json:"usage,omitempty"` + Error *struct { + Type string `json:"type"` + Message string `json:"message"` + } `json:"error,omitempty"` +} diff --git a/apps/edge/internal/openai/principal.go b/apps/edge/internal/openai/principal.go index a011525b..8102275c 100644 --- a/apps/edge/internal/openai/principal.go +++ b/apps/edge/internal/openai/principal.go @@ -167,6 +167,8 @@ func principalTokenFromRequest(r *http.Request) (string, bool) { apiKey := "" if isAnthropicRequest(r) { apiKey = strings.TrimSpace(r.Header.Get("X-Api-Key")) + } else if isGeminiRequest(r) { + apiKey = strings.TrimSpace(r.Header.Get("X-Goog-Api-Key")) } if authorization != "" && bearer == "" { return "", false diff --git a/apps/edge/internal/openai/principal_routes.go b/apps/edge/internal/openai/principal_routes.go index 8ee7467d..96185e57 100644 --- a/apps/edge/internal/openai/principal_routes.go +++ b/apps/edge/internal/openai/principal_routes.go @@ -54,7 +54,10 @@ func (s *Server) advertisedModelsForPrincipal(ctx context.Context) ([]advertised } for _, r := range routes { - id := strings.TrimSpace(r.RouteID) + id := strings.TrimSpace(r.RouteAlias) + if id == "" { + id = strings.TrimSpace(r.RouteID) + } addModel(id, id) } @@ -241,8 +244,10 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M if selector == "" { return managedCatalogBinding{}, ErrRouteNotFound } + routeAlias := strings.TrimSpace(route.RouteAlias) explicit := !strings.EqualFold(selector, "default") var matches []managedCatalogBinding + var aliasMatches []managedCatalogBinding for _, entry := range catalog { group := strings.TrimSpace(entry.ID) if group == "" { @@ -260,9 +265,23 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M binding.ProviderID = strings.TrimSpace(providerID) } matches = append(matches, binding) + if routeAlias != "" && routeAlias == group { + aliasMatches = append(aliasMatches, binding) + } break } } + // A principal-owned public alias that exactly names a compatible catalog + // group is an explicit disambiguator. This is required when two catalog + // groups intentionally expose the same provider/upstream pair (for example + // a general model and a preset-pinned fast alias). It never bypasses the + // resource selector or upstream-model checks above. + if len(aliasMatches) == 1 { + return aliasMatches[0], nil + } + if len(aliasMatches) > 1 { + return managedCatalogBinding{}, ErrRouteNotFound + } if len(matches) != 1 { return managedCatalogBinding{}, ErrRouteNotFound } diff --git a/apps/edge/internal/openai/principal_routes_test.go b/apps/edge/internal/openai/principal_routes_test.go index 1979bd79..591a520f 100644 --- a/apps/edge/internal/openai/principal_routes_test.go +++ b/apps/edge/internal/openai/principal_routes_test.go @@ -72,7 +72,7 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) { "token-p1": "principal-1", "token-p2": "principal-2", }, map[string]authprojection.Route{ - "r1": {RouteID: "route-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"}, + "r1": {RouteID: "route-p1", RouteAlias: "model-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"}, "r2": {RouteID: "route-p2", PrincipalRef: "principal-2", CredentialSlotRef: "slot-2", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"}, }) if err := cache.Apply(proj); err != nil { @@ -96,8 +96,8 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) { if err := json.Unmarshal(wP1.Body.Bytes(), &respP1); err != nil { t.Fatal(err) } - if len(respP1.Data) != 1 || respP1.Data[0].ID != "route-p1" { - t.Fatalf("P1 models: expected [route-p1], got %+v", respP1.Data) + if len(respP1.Data) != 1 || respP1.Data[0].ID != "model-p1" { + t.Fatalf("P1 models: expected public alias [model-p1], got %+v", respP1.Data) } // Call for P2 @@ -123,7 +123,7 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) { proj := makeTestProjection(1, now, time.Hour, map[string]string{ "token-p1": "principal-1", }, map[string]authprojection.Route{ - "r1": {RouteID: "claude-route-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"}, + "r1": {RouteID: "claude-route-1", RouteAlias: "claude-public-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"}, }) if err := cache.Apply(proj); err != nil { t.Fatal(err) @@ -148,8 +148,8 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) { if err := json.Unmarshal(w.Body.Bytes(), &anthropicResp); err != nil { t.Fatal(err) } - if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-route-1" { - t.Fatalf("expected [claude-route-1], got %+v", anthropicResp.Data) + if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-public-1" { + t.Fatalf("expected public alias [claude-public-1], got %+v", anthropicResp.Data) } } @@ -337,9 +337,15 @@ func TestManagedCatalogBindingExplicitSelectorAndNoFallback(t *testing.T) { t.Fatalf("missing selector err=%v", err) } route.ResourceSelector = "default" - if _, err := resolveManagedCatalogBinding(route, append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}})); !errors.Is(err, ErrRouteNotFound) { + ambiguous := append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}}) + if _, err := resolveManagedCatalogBinding(route, ambiguous); !errors.Is(err, ErrRouteNotFound) { t.Fatalf("ambiguous catalog err=%v", err) } + route.RouteAlias = "another-group" + binding, err = resolveManagedCatalogBinding(route, ambiguous) + if err != nil || binding.ModelGroupKey != "another-group" || binding.ProviderID != "" { + t.Fatalf("alias-disambiguated binding=%+v err=%v", binding, err) + } } func TestMetadataSpoofIsOverwrittenByManagedBinding(t *testing.T) { diff --git a/apps/edge/internal/openai/request_identity_ingress.go b/apps/edge/internal/openai/request_identity_ingress.go index 84c68b73..05603353 100644 --- a/apps/edge/internal/openai/request_identity_ingress.go +++ b/apps/edge/internal/openai/request_identity_ingress.go @@ -386,6 +386,12 @@ func (s *Server) compilePresetArtifactBinding(dispatch routeDispatch, protocol s preset = found } } + // Marked single-request presets use the operator-owned internal workspace + // capability compiled into dispatch.SingleRequest. They intentionally reject + // legacy caller workspace_tools, so artifact binding must not run here. + if preset.SingleRequest != nil || dispatch.SingleRequest != nil { + return nil, false, nil + } if !isModeAllowed(preset, modeLight) { return nil, false, nil } diff --git a/apps/edge/internal/openai/routes.go b/apps/edge/internal/openai/routes.go index 2b3d59f6..6517cc88 100644 --- a/apps/edge/internal/openai/routes.go +++ b/apps/edge/internal/openai/routes.go @@ -14,6 +14,7 @@ func (s *Server) routes() *http.ServeMux { mux.HandleFunc("/v1/models", s.withAuth(s.handleModels)) mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions)) mux.HandleFunc("/v1/responses", s.withAuth(s.handleResponses)) + mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent)) s.registerAnthropicRoutes(mux) mux.HandleFunc("/api/", s.withAuth(s.handleOllamaAPI)) return mux @@ -54,6 +55,10 @@ func (s *Server) writeCallerProviderCredentialRejection(w http.ResponseWriter, r writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", message) return } + if isGeminiRequest(r) { + writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", message) + return + } writeError(w, http.StatusBadRequest, "invalid_request_error", message) } @@ -63,6 +68,10 @@ func (s *Server) writeAuthenticationFailure(w http.ResponseWriter, r *http.Reque writeAnthropicError(w, http.StatusUnauthorized, "authentication_error", "authentication failed") return } + if isGeminiRequest(r) { + writeGeminiError(w, http.StatusUnauthorized, "UNAUTHENTICATED", "authentication failed") + return + } writeError(w, http.StatusUnauthorized, "unauthorized", "unauthorized") } diff --git a/apps/edge/internal/openai/single_request_chat.go b/apps/edge/internal/openai/single_request_chat.go new file mode 100644 index 00000000..d19c344d --- /dev/null +++ b/apps/edge/internal/openai/single_request_chat.go @@ -0,0 +1,366 @@ +package openai + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" + + edgeservice "iop/apps/edge/internal/service" +) + +type singleRequestChatPolicy struct { + status int + finishReason string + errorType string + message string + silent bool + error bool +} + +func singleRequestChatTerminalPolicy(disposition edgeservice.SingleRequestTerminalDisposition) singleRequestChatPolicy { + if disposition.Kind == "" { + disposition.Kind = edgeservice.SingleRequestTerminalEndTurn + } + if disposition.Validate() != nil { + disposition = edgeservice.SingleRequestTerminalDisposition{ + Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider, + } + } + switch disposition.Kind { + case edgeservice.SingleRequestTerminalEndTurn: + return singleRequestChatPolicy{status: http.StatusOK, finishReason: "stop"} + case edgeservice.SingleRequestTerminalLength: + return singleRequestChatPolicy{status: http.StatusOK, finishReason: "length"} + case edgeservice.SingleRequestTerminalCancelled: + return singleRequestChatPolicy{silent: true} + case edgeservice.SingleRequestTerminalError: + switch disposition.ErrorClass { + case edgeservice.SingleRequestTerminalErrorValidation, edgeservice.SingleRequestTerminalErrorContext: + return singleRequestChatPolicy{status: http.StatusBadRequest, errorType: "invalid_request_error", message: "single-request execution was rejected", error: true} + case edgeservice.SingleRequestTerminalErrorTimeout: + return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution timed out", error: true} + default: + return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true} + } + default: + return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true} + } +} + +func (s *Server) handleChatSingleRequestStream( + w http.ResponseWriter, + r *http.Request, + capability singleRequestService, + dispatch routeDispatch, + body []byte, +) { + requestID, err := newLogicalRequestRandomID() + if err != nil { + writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable") + return + } + requestID = "req_" + requestID + stream, err := newSingleRequestChatStream(w, requestID, dispatch.SingleRequest.PublicModel) + if err != nil { + writeError(w, http.StatusInternalServerError, "run_error", "single-request streaming is unavailable") + return + } + execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{ + RequestID: requestID, + Binding: dispatch.SingleRequest.Clone(), + Prompt: string(append([]byte(nil), body...)), + }) + if err != nil || execution == nil { + if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) { + writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable") + return + } + writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started") + return + } + defer execution.Cancel() + _ = pumpSingleRequestChatStream(r.Context(), execution, stream, newWallClockSingleRequestAnthropicTicker) +} + +func (s *Server) handleChatSingleRequest( + w http.ResponseWriter, + r *http.Request, + capability singleRequestService, + dispatch routeDispatch, + body []byte, +) { + requestID, err := newLogicalRequestRandomID() + if err != nil { + writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable") + return + } + requestID = "req_" + requestID + execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{ + RequestID: requestID, + Binding: dispatch.SingleRequest.Clone(), + Prompt: string(append([]byte(nil), body...)), + }) + if err != nil || execution == nil { + if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) { + writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable") + return + } + writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started") + return + } + defer execution.Cancel() + for { + select { + case <-r.Context().Done(): + execution.Cancel() + return + case progress, ok := <-execution.Progress(): + if !ok { + if r.Context().Err() == nil && execution.State() != edgeservice.SingleRequestStateCancelled { + writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed") + } + return + } + switch progress.Stage { + case edgeservice.SingleRequestStateFinalizing: + if progress.Result == nil { + _ = execution.AcknowledgeTerminal(false) + writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed") + return + } + writeErr := writeChatSingleRequestTerminal(w, requestID, dispatch.SingleRequest.PublicModel, *progress.Result) + _ = execution.AcknowledgeTerminal(writeErr == nil) + return + case edgeservice.SingleRequestStateFailed: + policy := singleRequestChatTerminalPolicy(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})) + writeError(w, policy.status, policy.errorType, policy.message) + return + case edgeservice.SingleRequestStateCancelled: + return + } + } + } +} + +func writeChatSingleRequestTerminal(w http.ResponseWriter, requestID, model string, result edgeservice.SingleRequestResult) error { + policy := singleRequestChatTerminalPolicy(result.Terminal) + if policy.silent || policy.error || policy.finishReason == "" { + return errors.New("single-request result has no Chat terminal") + } + content := result.Output + if policy.finishReason == "length" { + content = "" + } + response := chatCompletionResponse{ + ID: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), Object: "chat.completion", + Created: time.Now().Unix(), Model: model, + Choices: []chatCompletionChoice{{Index: 0, Message: chatMessage{Role: "assistant", Content: content}, FinishReason: policy.finishReason}}, + } + encoded, err := json.Marshal(response) + if err != nil { + return err + } + encoded = append(encoded, '\n') + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(policy.status) + n, err := w.Write(encoded) + if err != nil { + return err + } + if n != len(encoded) { + return io.ErrShortWrite + } + return nil +} + +type singleRequestChatStream struct { + mu sync.Mutex + w http.ResponseWriter + flusher http.Flusher + id string + model string + created int64 + started bool + terminal bool + terminalErr error +} + +func newSingleRequestChatStream(w http.ResponseWriter, requestID, model string) (*singleRequestChatStream, error) { + flusher, ok := w.(http.Flusher) + if !ok || strings.TrimSpace(requestID) == "" || strings.TrimSpace(model) == "" { + return nil, errors.New("single-request Chat stream is unavailable") + } + return &singleRequestChatStream{w: w, flusher: flusher, id: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), model: model, created: time.Now().Unix()}, nil +} + +func (s *singleRequestChatStream) startLocked() error { + if s.started { + return nil + } + s.w.Header().Set("Content-Type", "text/event-stream") + s.w.Header().Set("Cache-Control", "no-cache") + s.w.WriteHeader(http.StatusOK) + s.flusher.Flush() + s.started = true + return nil +} + +func (s *singleRequestChatStream) Start() error { + s.mu.Lock() + defer s.mu.Unlock() + if s.terminal { + return s.terminalErr + } + return s.startLocked() +} + +func (s *singleRequestChatStream) Ping() error { + s.mu.Lock() + defer s.mu.Unlock() + if s.terminal { + return s.terminalErr + } + if err := s.startLocked(); err != nil { + return err + } + _, err := io.WriteString(s.w, ": ping\n\n") + if err == nil { + s.flusher.Flush() + } + return err +} + +func (s *singleRequestChatStream) writeSSELocked(value any) error { + payload, err := json.Marshal(value) + if err != nil { + return err + } + if _, err = fmt.Fprintf(s.w, "data: %s\n\n", payload); err != nil { + return err + } + s.flusher.Flush() + return nil +} + +func (s *singleRequestChatStream) Final(result edgeservice.SingleRequestResult) error { + s.mu.Lock() + defer s.mu.Unlock() + if s.terminal { + return s.terminalErr + } + if err := s.startLocked(); err != nil { + return err + } + policy := singleRequestChatTerminalPolicy(result.Terminal) + if policy.silent || policy.error || policy.finishReason == "" { + return errors.New("single-request result has no Chat stream terminal") + } + s.terminal = true + if policy.finishReason != "length" && result.Output != "" { + if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{Content: result.Output}}}}); err != nil { + s.terminalErr = err + return err + } + } + if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{}, FinishReason: policy.finishReason}}}); err != nil { + s.terminalErr = err + return err + } + if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil { + s.terminalErr = err + return err + } + s.flusher.Flush() + return nil +} + +func (s *singleRequestChatStream) TerminalError(disposition edgeservice.SingleRequestTerminalDisposition) error { + s.mu.Lock() + defer s.mu.Unlock() + if s.terminal { + return s.terminalErr + } + policy := singleRequestChatTerminalPolicy(disposition) + if policy.silent { + s.terminal = true + return nil + } + if !policy.error { + policy = singleRequestChatTerminalPolicy(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}) + } + if err := s.startLocked(); err != nil { + return err + } + s.terminal = true + if err := s.writeSSELocked(errorResponse{Error: errorBody{Type: policy.errorType, Message: policy.message}}); err != nil { + s.terminalErr = err + return err + } + if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil { + s.terminalErr = err + return err + } + s.flusher.Flush() + return nil +} + +func pumpSingleRequestChatStream(ctx context.Context, execution edgeservice.SingleRequestExecution, stream *singleRequestChatStream, tickerFactory singleRequestAnthropicTickerFactory) error { + if execution == nil || stream == nil || tickerFactory == nil { + return errors.New("single-request Chat stream is unavailable") + } + if err := ctx.Err(); err != nil { + execution.Cancel() + return err + } + if err := stream.Start(); err != nil { + execution.Cancel() + return err + } + ticker := tickerFactory() + if ticker == nil { + execution.Cancel() + return errors.New("single-request Chat stream is unavailable") + } + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + execution.Cancel() + return ctx.Err() + case <-ticker.Ticks(): + if err := stream.Ping(); err != nil { + execution.Cancel() + return err + } + case progress, ok := <-execution.Progress(): + if !ok { + if ctx.Err() != nil { + return ctx.Err() + } + if execution.State() == edgeservice.SingleRequestStateCompleted { + return nil + } + return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}) + } + switch progress.Stage { + case edgeservice.SingleRequestStateFinalizing: + if progress.Result == nil { + err := stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}) + return errors.Join(err, execution.AcknowledgeTerminal(false)) + } + err := stream.Final(*progress.Result) + return errors.Join(err, execution.AcknowledgeTerminal(err == nil)) + case edgeservice.SingleRequestStateFailed: + return stream.TerminalError(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})) + case edgeservice.SingleRequestStateCancelled: + return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalCancelled}) + } + } + } +} diff --git a/apps/edge/internal/openai/single_request_chat_test.go b/apps/edge/internal/openai/single_request_chat_test.go new file mode 100644 index 00000000..36e0a4b5 --- /dev/null +++ b/apps/edge/internal/openai/single_request_chat_test.go @@ -0,0 +1,96 @@ +package openai + +import ( + "context" + "errors" + "net/http/httptest" + "strings" + "testing" + + edgeservice "iop/apps/edge/internal/service" +) + +func TestSingleRequestChatStreamPumpProjectsOnlyFinalResult(t *testing.T) { + execution := startSingleRequestAnthropicTestExecution(t, func( + _ context.Context, + req edgeservice.SingleRequestRequest, + ctrl edgeservice.SingleRequestController, + ) error { + for index, stage := range []edgeservice.SingleRequestState{ + edgeservice.SingleRequestStatePlanning, + edgeservice.SingleRequestStateWorking, + edgeservice.SingleRequestStateReviewing, + edgeservice.SingleRequestStateFinalizing, + } { + envelope := edgeservice.SingleRequestEnvelope{ + RequestID: req.RequestID, Sequence: uint64(index + 1), Stage: stage, + } + if stage == edgeservice.SingleRequestStateFinalizing { + envelope.Result = &edgeservice.SingleRequestResult{Output: "safe final"} + } + if err := ctrl.SubmitEnvelope(envelope); err != nil { + return err + } + } + return nil + }) + w := httptest.NewRecorder() + stream, err := newSingleRequestChatStream(w, "req_chat", "virtual-model") + if err != nil { + t.Fatal(err) + } + ticker := newManualSingleRequestAnthropicTicker() + if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil { + t.Fatal(err) + } + body := w.Body.String() + if !strings.Contains(body, `"content":"safe final"`) || !strings.Contains(body, `"finish_reason":"stop"`) || strings.Count(body, "data: [DONE]") != 1 { + t.Fatalf("unexpected Chat stream: %s", body) + } + for _, private := range []string{"Planning the requested work", "Executing the requested work", "Reviewing the completed work"} { + if strings.Contains(body, private) { + t.Fatalf("internal progress leaked: %s", body) + } + } +} + +func TestSingleRequestChatStreamSanitizesExecutorFailure(t *testing.T) { + execution := startSingleRequestAnthropicTestExecution(t, func( + _ context.Context, + req edgeservice.SingleRequestRequest, + ctrl edgeservice.SingleRequestController, + ) error { + if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: 1, Stage: edgeservice.SingleRequestStatePlanning}); err != nil { + return err + } + return errors.New("PRIVATE_EXECUTOR_FAILURE") + }) + w := httptest.NewRecorder() + stream, err := newSingleRequestChatStream(w, "req_chat_error", "virtual-model") + if err != nil { + t.Fatal(err) + } + ticker := newManualSingleRequestAnthropicTicker() + if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil { + t.Fatal(err) + } + body := w.Body.String() + if strings.Contains(body, "PRIVATE_EXECUTOR_FAILURE") || !strings.Contains(body, "single-request execution failed") || strings.Count(body, "data: [DONE]") != 1 { + t.Fatalf("unexpected sanitized Chat error: %s", body) + } +} + +func TestMarkedSingleRequestSkipsLegacyWorkspaceToolBinding(t *testing.T) { + dispatch := routeDispatch{ + Preset: validSingleRequestPreset(), + SingleRequest: newSingleRequestAnthropicTestBinding(t), + } + binding, pinArtifact, err := (&Server{}).compilePresetArtifactBinding( + dispatch, + "openai", + []byte(`{"model":"virtual-model","messages":[{"role":"user","content":"task"}]}`), + ) + if err != nil || binding != nil || pinArtifact { + t.Fatalf("marked single request entered legacy workspace binding: binding=%+v pin=%t err=%v", binding, pinArtifact, err) + } +} diff --git a/apps/edge/internal/openai/single_request_executor_test.go b/apps/edge/internal/openai/single_request_executor_test.go index 18c65d6c..b330f318 100644 --- a/apps/edge/internal/openai/single_request_executor_test.go +++ b/apps/edge/internal/openai/single_request_executor_test.go @@ -93,8 +93,12 @@ func waitExecutionResult(exec edgeservice.SingleRequestExecution) (edgeservice.S } func executorPlanBody(goal, verification string) []byte { - planMD := fmt.Sprintf("# Plan\n\n## Goal\n%s\n\n## Steps\n- Step one.\n- Step two.\n\n## Verification\n- %s\n", goal, verification) - return successBodyWithThoughtSignature(planMD) + b, _ := json.Marshal(map[string]any{ + "goal": goal, + "steps": []string{"Step one.", "Step two."}, + "verification": []string{verification}, + }) + return successBodyWithThoughtSignature(string(b)) } func executorWorkBody(completion, verification string) []byte { @@ -289,7 +293,7 @@ func TestSingleRequestExecutorConcurrentToolIsolation(t *testing.T) { bodyStr := string(reqBody) var resp []byte - if strings.Contains(bodyStr, "Produce exactly one Markdown document matching this template") { + if strings.Contains(bodyStr, "Create a concise plan for the task") { resp = executorPlanBody(fmt.Sprintf("Plan for %s", reqID), fmt.Sprintf("Verify plan for %s", reqID)) } else if strings.Contains(bodyStr, "Read the supplied plan") { if !strings.Contains(bodyStr, "typed-result-") { @@ -756,7 +760,7 @@ func TestSingleRequestExecutorTerminalWaiterCleanup(t *testing.T) { bodyStr := string(reqBody) var resp []byte - if strings.Contains(bodyStr, "Produce exactly one Markdown document matching this template") { + if strings.Contains(bodyStr, "Create a concise plan for the task") { resp = executorPlanBody("Plan step", "Verify plan") } else if strings.Contains(bodyStr, "Read the supplied plan") { if !strings.Contains(bodyStr, "colliding-tool-id") { @@ -810,7 +814,7 @@ func TestSingleRequestExecutorTerminalWaiterCleanup(t *testing.T) { bodyStr := string(reqBody) var resp []byte - if strings.Contains(bodyStr, "Produce exactly one Markdown document matching this template") { + if strings.Contains(bodyStr, "Create a concise plan for the task") { resp = executorPlanBody("Plan step", "Verify plan") } else if strings.Contains(bodyStr, "Read the supplied plan") { if !strings.Contains(bodyStr, "colliding-tool-id") { @@ -872,7 +876,7 @@ func TestSingleRequestExecutorTerminalWaiterCleanup(t *testing.T) { } var resp []byte - if strings.Contains(bodyStr, "Produce exactly one Markdown document matching this template") { + if strings.Contains(bodyStr, "Create a concise plan for the task") { resp = executorPlanBody("Plan step for "+reqID, "Verify plan") } else if strings.Contains(bodyStr, "Read the supplied plan") { if !strings.Contains(bodyStr, "colliding-tool-id") { diff --git a/apps/edge/internal/openai/single_request_handler_test.go b/apps/edge/internal/openai/single_request_handler_test.go index 18319ffd..332489b8 100644 --- a/apps/edge/internal/openai/single_request_handler_test.go +++ b/apps/edge/internal/openai/single_request_handler_test.go @@ -281,7 +281,7 @@ func TestAnthropicSingleRequestUsesOnePost(t *testing.T) { defer cancel() body := `{"model":"` + testSingleRequestModel + `","max_tokens":128,"messages":[{"role":"user","content":"` + privatePrompt + `"}],"tools":[{"name":"caller_tool","input_schema":{"type":"object"}}]}` req := newAnthropicSingleRequestHTTPReq(t, ctx, httpServer.URL, "/v1/messages", body) - req.Header.Set(anthropicBetaHeader, "thinking-token-count-2026-05-13") + req.Header.Set(anthropicBetaHeader, "advisor-tool-2026-03-01,thinking-token-count-2026-05-13") response, err := httpServer.Client().Do(req) if err != nil { t.Fatalf("POST /v1/messages: %v", err) diff --git a/apps/edge/internal/openai/single_request_plan_stage.go b/apps/edge/internal/openai/single_request_plan_stage.go index 7c3b872f..343a0208 100644 --- a/apps/edge/internal/openai/single_request_plan_stage.go +++ b/apps/edge/internal/openai/single_request_plan_stage.go @@ -2,15 +2,14 @@ package openai import ( "context" + "encoding/json" "errors" edgeservice "iop/apps/edge/internal/service" "iop/packages/go/singlerequesttemplate" ) -func singleRequestPlanPrompt(template string) string { - return "Produce exactly one Markdown document matching this template:\n\n" + template + "\n\nRequirements:\n- Keep the goal to a single non-empty line.\n- Provide between 2 and 6 step lines, each starting with '- '.\n- Provide between 1 and 3 verification lines, each starting with '- '.\n- Match all headings and static text exactly with no unresolved tokens." -} +const singleRequestPlanPrompt = "Create a concise plan for the task. Return exactly one JSON object with a non-empty one-line goal string, a steps array containing 2 to 6 non-empty one-line strings, and a verification array containing 1 to 3 non-empty one-line strings. Do not include Markdown bullets, headings, or template text; Edge owns all Plan document formatting." var errSingleRequestPlanStage = errors.New("single-request plan stage: failed") @@ -32,6 +31,49 @@ type singleRequestPlanStageRequest struct { Quality *singleRequestQualityGate } +type singleRequestPlanFields struct { + Goal string `json:"goal"` + Steps []string `json:"steps"` + Verification []string `json:"verification"` +} + +func (v *singleRequestPlanFields) UnmarshalJSON(data []byte) error { + if err := validateSingleRequestObjectFields(data, "goal", "steps", "verification"); err != nil { + return err + } + type alias singleRequestPlanFields + var decoded alias + if err := json.Unmarshal(data, &decoded); err != nil { + return err + } + *v = singleRequestPlanFields(decoded) + return nil +} + +func singleRequestPlanResponseFormat() *singleRequestProviderResponseFormat { + return &singleRequestProviderResponseFormat{ + Type: "json_schema", + JSONSchema: singleRequestProviderResponseJSONSchema{ + Name: "single_request_plan", + Strict: true, + Schema: singleRequestProviderOutputSchema{ + Type: "object", + Properties: map[string]singleRequestProviderOutputProperty{ + "goal": {Type: "string", MinLength: 1}, + "steps": { + Type: "array", Items: &singleRequestProviderOutputProperty{Type: "string", MinLength: 1}, MinItems: 2, MaxItems: 6, + }, + "verification": { + Type: "array", Items: &singleRequestProviderOutputProperty{Type: "string", MinLength: 1}, MinItems: 1, MaxItems: 3, + }, + }, + Required: []string{"goal", "steps", "verification"}, + AdditionalProperties: false, + }, + }, + } +} + func (s *singleRequestPlanStage) run(ctx context.Context, req singleRequestPlanStageRequest, ctrl edgeservice.SingleRequestController) ([]byte, error) { quality := singleRequestQualityGateOrNew(req.Quality) if s == nil || s.provider == nil || ctrl == nil || req.RequestID == "" || req.Task == "" || req.Sequence == 0 { @@ -41,20 +83,30 @@ func (s *singleRequestPlanStage) run(ctx context.Context, req singleRequestPlanS if binding == nil || binding.Templates.Plan == "" { return nil, quality.validation(errSingleRequestPlanStage) } + if err := singlerequesttemplate.ValidatePlanTemplate(binding.Templates.Plan); err != nil { + return nil, quality.validation(errSingleRequestPlanStage) + } if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: req.Sequence, Stage: edgeservice.SingleRequestStatePlanning}); err != nil { return nil, quality.serviceFailure(ctx, err, errSingleRequestPlanStage) } response, err := s.provider.submit(ctx, singleRequestProviderStageRequest{ StageBinding: req.StageBinding, Limits: req.Limits, NodeRef: req.NodeRef, SessionID: req.SessionID, UsageAttribution: req.UsageAttribution, Quality: quality, Messages: []chatMessage{ - {Role: "system", Content: singleRequestPlanPrompt(binding.Templates.Plan)}, + {Role: "system", Content: singleRequestPlanPrompt}, {Role: "user", Content: req.Task}, }, + ResponseFormat: singleRequestPlanResponseFormat(), }) if err != nil { return nil, quality.reclassify(err, errSingleRequestPlanStage) } - content, err := singlerequesttemplate.ParsePlan(binding.Templates.Plan, response.Output, req.Limits.MaxOutputBytes) + var fields singleRequestPlanFields + if err := json.Unmarshal([]byte(response.Output), &fields); err != nil { + return nil, quality.malformed(errSingleRequestPlanStage) + } + content, err := singlerequesttemplate.RenderPlan(binding.Templates.Plan, singlerequesttemplate.PlanFields{ + Goal: fields.Goal, Steps: fields.Steps, Verification: fields.Verification, + }, req.Limits.MaxOutputBytes) if err != nil { return nil, quality.malformed(errSingleRequestPlanStage) } diff --git a/apps/edge/internal/openai/single_request_plan_stage_test.go b/apps/edge/internal/openai/single_request_plan_stage_test.go index df9071e8..da5f7be9 100644 --- a/apps/edge/internal/openai/single_request_plan_stage_test.go +++ b/apps/edge/internal/openai/single_request_plan_stage_test.go @@ -4,6 +4,7 @@ import ( "context" "encoding/json" "errors" + "reflect" "strings" "testing" @@ -65,8 +66,9 @@ func validPlanStageRequest() singleRequestPlanStageRequest { func TestSingleRequestPlanStageWritesArtifact(t *testing.T) { d := matchingDispatch() + planJSON := `{"goal":"Inspect the target.","steps":["Step one.","Step two."],"verification":["Run focused tests."]}` planMD := "# Plan\n\n## Goal\nInspect the target.\n\n## Steps\n- Step one.\n- Step two.\n\n## Verification\n- Run focused tests.\n" - tunnel := &mockTunnel{frames: framesFor(successBodyWithThoughtSignature(planMD))} + tunnel := &mockTunnel{frames: framesFor(successBodyWithThoughtSignature(planJSON))} var captured edgeservice.ProviderPoolDispatchRequest provider := newSingleRequestProviderStage(&mockService{submit: func(_ context.Context, r edgeservice.ProviderPoolDispatchRequest) (*edgeservice.ProviderPoolDispatchResult, error) { captured = r @@ -85,21 +87,27 @@ func TestSingleRequestPlanStageWritesArtifact(t *testing.T) { t.Fatalf("controller=%+v", ctrl) } body, _ := captured.Tunnel.BuildBody("gemini-3.6-flash") - if !containsAll(string(body), "Fix immutable task", "Produce exactly one Markdown document", "reasoning_effort", "high") { + if !containsAll(string(body), "Fix immutable task", "Return exactly one JSON object", "reasoning_effort", "high") { t.Fatalf("body=%s", body) } var decoded map[string]any if err := json.Unmarshal(body, &decoded); err != nil { t.Fatal(err) } - if decoded["response_format"] != nil { - t.Fatalf("response_format should be omitted, got %#v", decoded["response_format"]) + encodedFormat, _ := json.Marshal(singleRequestPlanResponseFormat()) + var expectedFormat any + if err := json.Unmarshal(encodedFormat, &expectedFormat); err != nil { + t.Fatal(err) + } + if got := decoded["response_format"]; !reflect.DeepEqual(got, expectedFormat) { + t.Fatalf("response_format=%#v, want %#v", got, expectedFormat) } } func TestSingleRequestPlanStageCustomTemplate(t *testing.T) { d := matchingDispatch() customTmpl := "# Plan\n\nCustom Header\n\n## Goal\n{{goal}}\n\n## Steps\n{{steps}}\n\n## Verification\n{{verification}}\n" + planJSON := `{"goal":"Inspect custom target.","steps":["Custom step 1.","Custom step 2."],"verification":["Custom verify."]}` planMD := "# Plan\n\nCustom Header\n\n## Goal\nInspect custom target.\n\n## Steps\n- Custom step 1.\n- Custom step 2.\n\n## Verification\n- Custom verify.\n" binding, err := edgeservice.NewSingleRequestBindingWithTemplates("virtual-model", "ws-ref", validStageBinding(), validStageBinding(), validStageBinding(), validLimits(), edgeservice.SingleRequestTemplateBinding{ @@ -110,7 +118,7 @@ func TestSingleRequestPlanStageCustomTemplate(t *testing.T) { t.Fatal(err) } - tunnel := &mockTunnel{frames: framesFor(successBodyWithThoughtSignature(planMD))} + tunnel := &mockTunnel{frames: framesFor(successBodyWithThoughtSignature(planJSON))} provider := newSingleRequestProviderStage(&mockService{submit: func(_ context.Context, _ edgeservice.ProviderPoolDispatchRequest) (*edgeservice.ProviderPoolDispatchResult, error) { return &edgeservice.ProviderPoolDispatchResult{Path: edgeservice.ProviderPoolPathTunnel, Tunnel: tunnel, DispatchInfo: d}, nil }}) @@ -130,15 +138,20 @@ func TestSingleRequestPlanStageFailsClosed(t *testing.T) { raw string }{ {"empty-string", ""}, - {"step-count-1-too-few", "# Plan\n\n## Goal\nFix bug.\n\n## Steps\n- Step 1\n\n## Verification\n- Verify 1\n"}, - {"step-count-7-too-many", "# Plan\n\n## Goal\nFix bug.\n\n## Steps\n- S1\n- S2\n- S3\n- S4\n- S5\n- S6\n- S7\n\n## Verification\n- Verify 1\n"}, - {"verif-count-0-too-few", "# Plan\n\n## Goal\nFix bug.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n"}, - {"verif-count-4-too-many", "# Plan\n\n## Goal\nFix bug.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- V1\n- V2\n- V3\n- V4\n"}, - {"multiline-goal", "# Plan\n\n## Goal\nLine 1\nLine 2\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- Verify 1\n"}, - {"altered-heading", "# Plan Mismatch\n\n## Goal\nFix bug.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- Verify 1\n"}, - {"unresolved-token", "# Plan\n\n## Goal\nFix {{goal}} bug.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- Verify 1\n"}, - {"trailing-content", "# Plan\n\n## Goal\nFix bug.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- Verify 1\n\nExtra Trailing"}, - {"not-markdown", "just plain text"}, + {"not-json", "just plain text"}, + {"unknown-field", `{"goal":"Fix bug.","steps":["Step 1","Step 2"],"verification":["Verify 1"],"extra":"no"}`}, + {"duplicate-field", `{"goal":"Fix bug.","goal":"Again.","steps":["Step 1","Step 2"],"verification":["Verify 1"]}`}, + {"missing-field", `{"goal":"Fix bug.","steps":["Step 1","Step 2"]}`}, + {"wrong-field-type", `{"goal":"Fix bug.","steps":"Step 1 and Step 2","verification":["Verify 1"]}`}, + {"step-count-1-too-few", `{"goal":"Fix bug.","steps":["Step 1"],"verification":["Verify 1"]}`}, + {"step-count-7-too-many", `{"goal":"Fix bug.","steps":["S1","S2","S3","S4","S5","S6","S7"],"verification":["Verify 1"]}`}, + {"step-empty-item", `{"goal":"Fix bug.","steps":["Step 1",""],"verification":["Verify 1"]}`}, + {"step-multiline-item", `{"goal":"Fix bug.","steps":["Step 1\ncontinued","Step 2"],"verification":["Verify 1"]}`}, + {"verif-count-0-too-few", `{"goal":"Fix bug.","steps":["Step 1","Step 2"],"verification":[]}`}, + {"verif-count-4-too-many", `{"goal":"Fix bug.","steps":["Step 1","Step 2"],"verification":["V1","V2","V3","V4"]}`}, + {"verification-multiline-item", `{"goal":"Fix bug.","steps":["Step 1","Step 2"],"verification":["Verify 1\ncontinued"]}`}, + {"multiline-goal", `{"goal":"Line 1\nLine 2","steps":["Step 1","Step 2"],"verification":["Verify 1"]}`}, + {"unresolved-token", `{"goal":"Fix {{goal}} bug.","steps":["Step 1","Step 2"],"verification":["Verify 1"]}`}, } for _, tt := range malformedTests { @@ -210,8 +223,8 @@ func TestSingleRequestPlanStageFailsClosed(t *testing.T) { t.Run("artifact-write-failure-rejects", func(t *testing.T) { d := matchingDispatch() - planMD := "# Plan\n\n## Goal\nGoal text.\n\n## Steps\n- Step 1\n- Step 2\n\n## Verification\n- Verify 1\n" - tunnel := &mockTunnel{frames: framesFor(successBody(planMD))} + planJSON := `{"goal":"Goal text.","steps":["Step 1","Step 2"],"verification":["Verify 1"]}` + tunnel := &mockTunnel{frames: framesFor(successBody(planJSON))} provider := newSingleRequestProviderStage(&mockService{submit: func(context.Context, edgeservice.ProviderPoolDispatchRequest) (*edgeservice.ProviderPoolDispatchResult, error) { return &edgeservice.ProviderPoolDispatchResult{Path: edgeservice.ProviderPoolPathTunnel, Tunnel: tunnel, DispatchInfo: d}, nil }}) diff --git a/apps/edge/internal/openai/single_request_provider_stage.go b/apps/edge/internal/openai/single_request_provider_stage.go index 7d9b84b8..ca295141 100644 --- a/apps/edge/internal/openai/single_request_provider_stage.go +++ b/apps/edge/internal/openai/single_request_provider_stage.go @@ -185,8 +185,13 @@ type singleRequestProviderOutputSchema struct { } type singleRequestProviderOutputProperty struct { - Type string `json:"type"` - Description string `json:"description"` + Type string `json:"type"` + Description string `json:"description,omitempty"` + Enum []string `json:"enum,omitempty"` + MinLength int `json:"minLength,omitempty"` + Items *singleRequestProviderOutputProperty `json:"items,omitempty"` + MinItems int `json:"minItems,omitempty"` + MaxItems int `json:"maxItems,omitempty"` } // buildSingleRequestChatBody owns all request authority. Stage options are @@ -320,6 +325,7 @@ func validateSingleRequestObjectFields(data []byte, allowed ...string) error { for _, a := range allowed { allowedMap[a] = true } + seen := make(map[string]bool, len(allowed)) for decoder.More() { keyTok, err := decoder.Token() if err != nil { @@ -329,9 +335,10 @@ func validateSingleRequestObjectFields(data []byte, allowed ...string) error { if !ok { return errors.New("json: object key must be string") } - if !allowedMap[key] { + if !allowedMap[key] || seen[key] { return errors.New("json: unknown or non-canonical field: " + key) } + seen[key] = true var val json.RawMessage if err := decoder.Decode(&val); err != nil { return err diff --git a/apps/edge/internal/openai/single_request_provider_stage_test.go b/apps/edge/internal/openai/single_request_provider_stage_test.go index d52f7ff3..68d3b389 100644 --- a/apps/edge/internal/openai/single_request_provider_stage_test.go +++ b/apps/edge/internal/openai/single_request_provider_stage_test.go @@ -554,6 +554,10 @@ func TestSingleRequestProviderStageRejectsMismatchLimitAndContext(t *testing.T) if !errors.Is(err, errProviderStageGeneric) || !tunnel.closed { t.Fatalf("err=%v closed=%v", err, tunnel.closed) } + disposition, ok := singleRequestTerminalDisposition(err) + if !ok || disposition.Kind != edgeservice.SingleRequestTerminalError || disposition.ErrorClass != edgeservice.SingleRequestTerminalErrorValidation { + t.Fatalf("dispatch mismatch disposition=%+v ok=%v", disposition, ok) + } }) } @@ -592,6 +596,10 @@ func TestSingleRequestProviderStageRejectsMismatchLimitAndContext(t *testing.T) if !errors.Is(err, errProviderStageGeneric) { t.Fatalf("expected errProviderStageGeneric, got %v", err) } + disposition, ok := singleRequestTerminalDisposition(err) + if !ok || disposition.Kind != edgeservice.SingleRequestTerminalError || disposition.ErrorClass != edgeservice.SingleRequestTerminalErrorProvider { + t.Fatalf("provider failure disposition=%+v ok=%v", disposition, ok) + } }) t.Run("service-returns-nil-result", func(t *testing.T) { diff --git a/apps/edge/internal/openai/single_request_review_stage.go b/apps/edge/internal/openai/single_request_review_stage.go index c1434b6f..f6b581b8 100644 --- a/apps/edge/internal/openai/single_request_review_stage.go +++ b/apps/edge/internal/openai/single_request_review_stage.go @@ -15,7 +15,7 @@ import ( ) const ( - singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." + singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." singleRequestReviewStageID = "review" ) @@ -110,7 +110,9 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev {Role: "user", Content: "Task:\n" + strings.TrimSpace(req.Task) + "\n\nPLAN:\n" + string(plan) + "\n\nWORK COMPLETION:\n" + strings.TrimSpace(req.Work.Completion) + "\n\nWORK VERIFICATION:\n" + strings.TrimSpace(req.Work.Verification)}, } repairRequired := false - for attempts := 0; attempts <= req.Limits.MaxToolIterations; attempts++ { + invalidToolCorrections := 0 + toolAttempts := 0 + for { response, err := s.submit(ctx, req, messages, tools, repairRequired) if err != nil { return nil, quality.reclassify(err, errSingleRequestReviewStage) @@ -132,7 +134,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev } return result, nil } - if response.call == nil || attempts == req.Limits.MaxToolIterations { + if response.call == nil || toolAttempts >= req.Limits.MaxToolIterations { if response.call != nil { return nil, quality.budget(errSingleRequestReviewStage) } @@ -142,6 +144,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev if err != nil { return nil, quality.malformed(errSingleRequestReviewStage) } + arguments = normalizeSingleRequestProviderToolArguments(response.call.Function.Name, arguments) isInspection := response.call.Function.Name == edgeservice.InternalWorkspaceToolRead || response.call.Function.Name == edgeservice.InternalWorkspaceToolList if !isInspection && !isSingleRequestReviewRepairTool(response.call.Function.Name) { return nil, quality.malformed(errSingleRequestReviewStage) @@ -168,12 +171,24 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev } } key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestReviewStageID, toolCallID: response.call.ID} + call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments} + if err := edgeservice.ValidateInternalWorkspaceToolCall(call); err != nil { + if invalidToolCorrections >= 1 { + return nil, quality.malformed(errSingleRequestReviewStage) + } + invalidToolCorrections++ + messages = append(messages, + chatMessage{Role: "assistant", ToolCalls: []any{response.call.asChatToolCall()}}, + chatMessage{Role: "tool", ToolCallID: response.call.ID, ToolName: response.call.Function.Name, Content: singleRequestWorkToolResultContent(edgeservice.InternalWorkspaceToolResult{Status: "invalid", ErrorCode: "invalid_request"}, req.Limits.MaxOutputBytes)}, + ) + continue + } + toolAttempts++ resultCh, err := s.bridge.register(key) if err != nil { return nil, quality.internalTool(errSingleRequestReviewStage) } sequence++ - call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments} if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: sequence, Stage: edgeservice.SingleRequestStateInternalTool, SavedStage: stage, ToolCall: call}); err != nil { s.bridge.unregister(key) return nil, quality.serviceFailure(ctx, err, errSingleRequestReviewStage) @@ -249,7 +264,7 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any if repairRequired { toolChoice = "required" } - body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "stream": false} + body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "response_format": singleRequestReviewResponseFormat(), "stream": false} for key, value := range options { folded := strings.ToLower(key) if isSingleRequestReviewReservedOption(folded) && key != folded { @@ -269,13 +284,35 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any func isSingleRequestReviewReservedOption(key string) bool { switch key { - case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "stream", "credential", "credential_binding", "reasoning_effort": + case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "response_format", "stream", "credential", "credential_binding", "reasoning_effort": return true default: return false } } +func singleRequestReviewResponseFormat() *singleRequestProviderResponseFormat { + return &singleRequestProviderResponseFormat{ + Type: "json_schema", + JSONSchema: singleRequestProviderResponseJSONSchema{ + Name: "single_request_review", + Strict: true, + Schema: singleRequestProviderOutputSchema{ + Type: "object", + Properties: map[string]singleRequestProviderOutputProperty{ + "decision": {Type: "string", Enum: []string{"pass"}}, + "output": {Type: "string", MinLength: 1}, + "checks": {Type: "string", MinLength: 1}, + "verification": {Type: "string", MinLength: 1}, + "summary": {Type: "string", MinLength: 1}, + }, + Required: []string{"decision", "output", "checks", "verification", "summary"}, + AdditionalProperties: false, + }, + }, + } +} + type singleRequestReviewProviderEnvelope struct { ID string `json:"id"` Object string `json:"object"` diff --git a/apps/edge/internal/openai/single_request_review_stage_test.go b/apps/edge/internal/openai/single_request_review_stage_test.go index b3671f61..2dee3c3d 100644 --- a/apps/edge/internal/openai/single_request_review_stage_test.go +++ b/apps/edge/internal/openai/single_request_review_stage_test.go @@ -383,6 +383,52 @@ func TestSingleRequestReviewStageInspectionAndRepairRemainInLegalStates(t *testi }) } +func TestSingleRequestReviewStageBoundedInvalidToolCorrection(t *testing.T) { + t.Run("one malformed provider call is corrected before coordinator admission", func(t *testing.T) { + bridge := newSingleRequestWorkToolBridge() + ctrl := newReviewController(t, bridge) + var bodies [][]byte + responses := [][]byte{ + reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`), + reviewToolBody("inspect-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":"result.txt"}`), + reviewPassBody("Approved.", "Corrected inspection passed."), + } + stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge) + if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); err != nil { + t.Fatal(err) + } + if len(bodies) != 3 || !strings.Contains(string(bodies[1]), "invalid_request") { + t.Fatalf("provider correction bodies=%d", len(bodies)) + } + want := []edgeservice.SingleRequestState{edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateInternalTool, edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateFinalizing} + if len(ctrl.envelopes) != len(want) { + t.Fatalf("envelopes=%+v", ctrl.envelopes) + } + for index, state := range want { + if ctrl.envelopes[index].Stage != state { + t.Fatalf("envelopes=%+v", ctrl.envelopes) + } + } + }) + + t.Run("second malformed provider call fails closed", func(t *testing.T) { + bridge := newSingleRequestWorkToolBridge() + ctrl := newReviewController(t, bridge) + var bodies [][]byte + responses := [][]byte{ + reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`), + reviewToolBody("invalid-2", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`), + } + stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge) + if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); !errors.Is(err, errSingleRequestReviewStage) { + t.Fatalf("err=%v, want review stage failure", err) + } + if len(bodies) != 2 || len(ctrl.envelopes) != 1 || ctrl.envelopes[0].Stage != edgeservice.SingleRequestStateReviewing || bridge.pendingCount() != 0 { + t.Fatalf("bodies=%d envelopes=%+v pending=%d", len(bodies), ctrl.envelopes, bridge.pendingCount()) + } + }) +} + func TestSingleRequestReviewStageFailsClosed(t *testing.T) { for _, raw := range []string{ `{"decision":"pass","output":"x","checks":"c","verification":"v","summary":"y","extra":1}`, @@ -471,11 +517,41 @@ func TestSingleRequestReviewBodyRejectsOptionAliases(t *testing.T) { if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "low"}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) { t.Fatalf("err=%v", err) } + if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "high", "Response_Format": map[string]any{"type": "caller"}}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) { + t.Fatalf("response format alias err=%v", err) + } if _, _, err := renderSingleRequestReview(singlerequesttemplate.DefaultReviewTemplate, singleRequestReviewDecision{Decision: "pass", Output: strings.Repeat("x", 10), Checks: "c", Verification: "v", Summary: "summary"}, 9); !errors.Is(err, errSingleRequestReviewStage) { t.Fatalf("err=%v", err) } } +func TestSingleRequestReviewResponseFormatIsClosed(t *testing.T) { + format := singleRequestReviewResponseFormat() + if format.Type != "json_schema" || format.JSONSchema.Name != "single_request_review" || !format.JSONSchema.Strict { + t.Fatalf("response format=%+v", format) + } + schema := format.JSONSchema.Schema + if schema.Type != "object" || schema.AdditionalProperties || len(schema.Properties) != 5 || len(schema.Required) != 5 { + t.Fatalf("review schema=%+v", schema) + } + if decision := schema.Properties["decision"]; !reflect.DeepEqual(decision.Enum, []string{"pass"}) { + t.Fatalf("decision schema=%+v", decision) + } + for _, name := range []string{"output", "checks", "verification", "summary"} { + if schema.Properties[name].MinLength != 1 { + t.Fatalf("property %q schema=%+v", name, schema.Properties[name]) + } + } +} + +func TestSingleRequestReviewPromptRequiresCanonicalWorkspaceRelativePaths(t *testing.T) { + for _, requirement := range []string{"canonical", "workspace-relative", "README.md", "./README.md", "absolute path", "parent traversal"} { + if !strings.Contains(singleRequestReviewPrompt, requirement) { + t.Fatalf("review prompt does not describe %q path requirement", requirement) + } + } +} + // customReviewStageTemplate is an operator-authored effective Review template // that differs from the built-in default, so a rendered artifact cannot pass by // accidentally falling back. @@ -645,7 +721,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "messages": [ { "role": "system", - "content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." + "content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." }, { "role": "user", @@ -664,7 +740,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -681,7 +759,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -698,7 +778,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path", "content"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." }, "content": { "type": "string" @@ -718,7 +800,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -764,7 +848,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "messages": [ { "role": "system", - "content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." + "content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call." }, { "role": "user", @@ -808,7 +892,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -825,7 +911,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -842,7 +930,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path", "content"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." }, "content": { "type": "string" @@ -862,7 +952,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { "required": ["relative_path"], "properties": { "relative_path": { - "type": "string" + "type": "string", + "minLength": 1, + "description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." } } } @@ -900,6 +992,15 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any { } var res map[string]any _ = json.Unmarshal([]byte(raw), &res) + for _, rawTool := range res["tools"].([]any) { + tool := rawTool.(map[string]any) + function := tool["function"].(map[string]any) + function["strict"] = true + } + encodedFormat, _ := json.Marshal(singleRequestReviewResponseFormat()) + var expectedFormat any + _ = json.Unmarshal(encodedFormat, &expectedFormat) + res["response_format"] = expectedFormat return res } diff --git a/apps/edge/internal/openai/single_request_work_stage.go b/apps/edge/internal/openai/single_request_work_stage.go index 8a51da20..79be259f 100644 --- a/apps/edge/internal/openai/single_request_work_stage.go +++ b/apps/edge/internal/openai/single_request_work_stage.go @@ -16,8 +16,9 @@ import ( ) const ( - singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification." - singleRequestWorkStageID = "work" + singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification. Every relative_path argument and every workspace path mentioned in the completion or verification must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal." + singleRequestWorkStageID = "work" + singleRequestCanonicalRelativePathDescription = "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md." ) var errSingleRequestWorkStage = errors.New("single-request work stage: failed") @@ -240,6 +241,7 @@ func (s *singleRequestWorkStage) run(ctx context.Context, req singleRequestWorkS if err != nil { return nil, quality.malformed(errSingleRequestWorkStage) } + arguments = normalizeSingleRequestProviderToolArguments(call.Function.Name, arguments) key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestWorkStageID, toolCallID: call.ID} resultCh, err := s.bridge.register(key) if err != nil { @@ -283,14 +285,15 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding return false } tools := make([]any, 0, len(workspace.OperationIDs)) - path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}}} + pathProperty := map[string]any{"type": "string", "minLength": 1, "description": singleRequestCanonicalRelativePathDescription} + path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": pathProperty}} for _, pair := range []struct{ operation, name string }{{"read", edgeservice.InternalWorkspaceToolRead}, {"list", edgeservice.InternalWorkspaceToolList}, {"write", edgeservice.InternalWorkspaceToolWrite}, {"delete", edgeservice.InternalWorkspaceToolDelete}} { if !has(pair.operation) { continue } parameters := path if pair.operation == "write" { - parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}, "content": map[string]any{"type": "string"}}} + parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": pathProperty, "content": map[string]any{"type": "string"}}} } tools = append(tools, singleRequestWorkToolSchema(pair.name, parameters)) } @@ -316,7 +319,7 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding } func singleRequestWorkToolSchema(name string, parameters map[string]any) map[string]any { - return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "parameters": parameters}} + return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "strict": true, "parameters": parameters}} } type singleRequestWorkProviderResponse struct { @@ -514,6 +517,30 @@ func decodeSingleRequestWorkToolArguments(arguments string) (json.RawMessage, er return append(json.RawMessage(nil), bytes.TrimSpace(raw)...), nil } +// normalizeSingleRequestProviderToolArguments canonicalizes the one +// containment-safe provider convention that differs from the internal tool +// contract: an empty workspace_list path means the workspace root. All other +// operations and non-canonical paths remain unchanged and are rejected by the +// service-owned decoder. +func normalizeSingleRequestProviderToolArguments(name string, arguments json.RawMessage) json.RawMessage { + if name != edgeservice.InternalWorkspaceToolList { + return arguments + } + var pathArguments struct { + RelativePath *string `json:"relative_path"` + } + decoder := json.NewDecoder(bytes.NewReader(arguments)) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&pathArguments); err != nil || pathArguments.RelativePath == nil || *pathArguments.RelativePath != "" { + return arguments + } + var trailing any + if err := decoder.Decode(&trailing); err != io.EOF { + return arguments + } + return json.RawMessage(`{"relative_path":"."}`) +} + func decodeSingleRequestWorkProviderResponse(body []byte, maximum int) (*singleRequestWorkProviderResponse, error) { if len(body) == 0 || len(body) > maximum || validateSingleRequestJSON(body) != nil { return nil, errSingleRequestWorkStage diff --git a/apps/edge/internal/openai/single_request_work_stage_test.go b/apps/edge/internal/openai/single_request_work_stage_test.go index 25389b2c..035ec0b1 100644 --- a/apps/edge/internal/openai/single_request_work_stage_test.go +++ b/apps/edge/internal/openai/single_request_work_stage_test.go @@ -877,6 +877,29 @@ func TestSingleRequestWorkStageRejectsMalformedResponsesAndOptions(t *testing.T) } } +func TestNormalizeSingleRequestProviderToolArguments(t *testing.T) { + tests := []struct { + name string + tool string + arguments string + want string + }{ + {name: "empty list root becomes canonical dot", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":""}`, want: `{"relative_path":"."}`}, + {name: "empty read remains invalid for service", tool: edgeservice.InternalWorkspaceToolRead, arguments: `{"relative_path":""}`, want: `{"relative_path":""}`}, + {name: "dot-prefixed list remains invalid for service", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"./"}`, want: `{"relative_path":"./"}`}, + {name: "canonical list path is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"src"}`, want: `{"relative_path":"src"}`}, + {name: "unknown list field is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"","extra":true}`, want: `{"relative_path":"","extra":true}`}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + got := normalizeSingleRequestProviderToolArguments(test.tool, json.RawMessage(test.arguments)) + if string(got) != test.want { + t.Fatalf("normalized arguments=%s, want %s", got, test.want) + } + }) + } +} + func TestSingleRequestWorkStageRejectsReservedOptionAliases(t *testing.T) { messageSets := map[string][]chatMessage{ "initial": {{Role: "user", Content: "immutable task"}}, @@ -977,6 +1000,37 @@ func TestSingleRequestWorkStageProjectsClosedEnvironmentSchema(t *testing.T) { } } +func TestSingleRequestWorkspaceToolSchemasDescribeCanonicalRelativePaths(t *testing.T) { + tools, err := singleRequestWorkTools(workBinding(t).Workspace) + if err != nil { + t.Fatal(err) + } + pathTools := 0 + for _, raw := range tools { + tool, _ := raw.(map[string]any) + function, _ := tool["function"].(map[string]any) + if function["strict"] != true { + t.Fatalf("tool %q strict=%v, want true", function["name"], function["strict"]) + } + parameters, _ := function["parameters"].(map[string]any) + properties, _ := parameters["properties"].(map[string]any) + pathProperty, hasPath := properties["relative_path"].(map[string]any) + if !hasPath { + continue + } + pathTools++ + if pathProperty["description"] != singleRequestCanonicalRelativePathDescription { + t.Fatalf("tool %q relative_path description=%v", function["name"], pathProperty["description"]) + } + if pathProperty["minLength"] != 1 { + t.Fatalf("tool %q relative_path minLength=%v, want 1", function["name"], pathProperty["minLength"]) + } + } + if pathTools == 0 { + t.Fatal("workspace tools did not expose a relative_path schema") + } +} + func TestSingleRequestWorkToolBridgeCorrelatesAndCleansUp(t *testing.T) { b := newSingleRequestWorkToolBridge() keys := []singleRequestWorkToolKey{{"request-a", "working", "one"}, {"request-b", "working", "two"}} diff --git a/apps/edge/internal/service/single_request_tool_types.go b/apps/edge/internal/service/single_request_tool_types.go index db363816..21671883 100644 --- a/apps/edge/internal/service/single_request_tool_types.go +++ b/apps/edge/internal/service/single_request_tool_types.go @@ -81,6 +81,15 @@ func (r InternalWorkspaceToolResult) Clone() InternalWorkspaceToolResult { return r } +// ValidateInternalWorkspaceToolCall applies the service-owned closed decoder +// without executing the call. Private provider stages use it to reject or +// boundedly correct malformed model output before the coordinator enters an +// internal-tool state. +func ValidateInternalWorkspaceToolCall(call *InternalWorkspaceToolCall) error { + _, err := decodeInternalWorkspaceToolCall(call.Clone()) + return err +} + // SingleRequestToolContinuation is optional. An executor that emits an // internal workspace call must implement it so the coordinator can deliver the // correlated Node result without involving an HTTP caller. diff --git a/apps/edge/internal/service/single_request_tool_types_test.go b/apps/edge/internal/service/single_request_tool_types_test.go index 05822db7..062c5c7e 100644 --- a/apps/edge/internal/service/single_request_tool_types_test.go +++ b/apps/edge/internal/service/single_request_tool_types_test.go @@ -53,6 +53,18 @@ func TestInternalWorkspaceToolDecodeClosedOperations(t *testing.T) { } } +func TestValidateInternalWorkspaceToolCall(t *testing.T) { + if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolList, `{"relative_path":"."}`)); err != nil { + t.Fatalf("canonical call rejected: %v", err) + } + if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolRead, `{"relative_path":""}`)); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) { + t.Fatalf("empty read error=%v, want invalid call", err) + } + if err := ValidateInternalWorkspaceToolCall(nil); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) { + t.Fatalf("nil call error=%v, want invalid call", err) + } +} + func TestInternalWorkspaceToolDecodeRejectsMalformed(t *testing.T) { const rawSentinel = "RAW-ARGUMENT-SENTINEL-DO-NOT-LEAK" tests := map[string]*InternalWorkspaceToolCall{ diff --git a/apps/node/internal/node/workspace_handler_test.go b/apps/node/internal/node/workspace_handler_test.go index c1ab069d..bc5caa68 100644 --- a/apps/node/internal/node/workspace_handler_test.go +++ b/apps/node/internal/node/workspace_handler_test.go @@ -37,7 +37,9 @@ func TestNodeWorkspaceCommandHelperProcess(t *testing.T) { if err := os.WriteFile(os.Getenv("IOP_NODE_START_FILE"), []byte("started"), 0o600); err != nil { os.Exit(21) } - select {} + for { + time.Sleep(time.Hour) + } default: os.Exit(22) } diff --git a/apps/node/internal/workspace/command_executor_test.go b/apps/node/internal/workspace/command_executor_test.go index b87bfbda..b343c458 100644 --- a/apps/node/internal/workspace/command_executor_test.go +++ b/apps/node/internal/workspace/command_executor_test.go @@ -53,7 +53,9 @@ func TestWorkspaceCommandHelperProcess(t *testing.T) { if err := os.WriteFile(os.Getenv("IOP_START_FILE"), []byte("started"), 0o600); err != nil { os.Exit(10) } - select {} + for { + time.Sleep(time.Hour) + } case "group": cmd := exec.Command(os.Args[0], "-test.run=^TestWorkspaceCommandGrandchild$") cmd.Env = []string{"IOP_WORKSPACE_GRANDCHILD=1"} @@ -64,7 +66,9 @@ func TestWorkspaceCommandHelperProcess(t *testing.T) { _ = cmd.Process.Kill() os.Exit(12) } - select {} + for { + time.Sleep(time.Hour) + } case "sentinel": if err := os.WriteFile(os.Getenv("IOP_SENTINEL_FILE"), []byte("target-started"), 0o600); err != nil { os.Exit(13) @@ -79,7 +83,9 @@ func TestWorkspaceCommandGrandchild(t *testing.T) { if os.Getenv("IOP_WORKSPACE_GRANDCHILD") == "" { return } - select {} + for { + time.Sleep(time.Hour) + } } func newCommandRuntime(t *testing.T, root string, outputLimit int64) *Runtime { diff --git a/docs/agent-comparison-benchmark-dev-guide.md b/docs/agent-comparison-benchmark-dev-guide.md new file mode 100644 index 00000000..45161186 --- /dev/null +++ b/docs/agent-comparison-benchmark-dev-guide.md @@ -0,0 +1,580 @@ +# Agent Comparison Benchmark Dev Guide + +이 문서는 IOP one-shot agent/model comparison benchmark의 현재 구현, dev 환경 구성, caller 연결 방식, managed credential 경계, 실행 절차와 장애 대응 기준을 한곳에 모은 운영 가이드다. + +raw token, provider API key, private key, slot alias, lease id와 개인 endpoint는 이 문서에 기록하지 않는다. 실제 host, checkout, Node/provider endpoint와 최신 process 상태는 아래 source of truth에서 확인한다. + +- benchmark manifest: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` +- public CLI: `scripts/agent_comparison_benchmark.py` +- dev environment: `agent-test/dev/rules.md` +- machine-readable dev inventory: `agent-test/inventory-dev.yaml` +- Edge/Node verification: `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md` +- API contracts: `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md` +- benchmark SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md` + +이 문서는 위 계약과 manifest를 설명할 뿐 덮어쓰지 않는다. 값이 다르면 manifest, contract, inventory, environment rule 순서로 최신 상태를 확인한다. + +## 1. 현재 상태 + +2026-08-12 기준 상태는 다음과 같다. + +| 영역 | 상태 | +|---|---| +| Gemini-native Edge ingress | 구현 완료. route-qualified `streamGenerateContent`, `x-goog-api-key` IOP principal auth, request/tool/SSE 변환을 지원한다. | +| official agy adapter | 구현 완료. `agy 1.1.12`, Gemini API-key provider, real `init/step_update/result` JSONL을 사용한다. | +| managed dev credential runtime | 구성 완료. Control Plane projection, credential slot/route, sealed lease, Edge HTTPS, CP↔Edge/Edge↔Node mTLS를 사용한다. | +| caller readiness | 최근 완료된 readiness evidence에서 C01-C09 `ready=9`를 확인했다. 실제 실행일에는 fresh preflight가 다시 필요하다. | +| controller recovery | control socket symlink와 caller launch 전 interruption 회귀 수정 및 테스트가 완료됐다. | +| Codex production JSONL | `cache_write_input_tokens` 수용과 config-owned effective binding 정합화가 완료됐다. | +| deterministic tests | focused 41 tests, 전체 benchmark 429 tests가 통과했다. 숫자는 당시 snapshot이며 현재 suite 결과는 fresh 실행으로 판단한다. | +| C01-C09 scored execution | 아직 완료되지 않았다. 과거 incomplete/failed run은 append-only evidence로 보존하며 정상 결과로 간주하지 않는다. | +| blind scoring/report | 유효한 9-cell execution이 생긴 뒤 수행하는 후속 단계다. | + +과거 실행 실패는 새 run을 정당화하는 완료 evidence가 아니다. 이전 run tree를 수정하거나 old run을 암묵적으로 `resume`/retry하지 않는다. 새 scored execution은 그 실행을 명시적으로 소유하는 현재 plan과 사용자 권한이 있을 때만 한 번 수행한다. 이 문서 자체는 지속적 실행 승인을 부여하지 않는다. + +## 2. 시스템 구성 + +```text +Benchmark runner + ├─ Claude Code ── Anthropic Messages ───────────────┐ + ├─ agy ───────── Gemini streamGenerateContent ─────┤ + └─ Codex ─────── OpenAI Responses ─────────────────┤ + v + IOP Edge HTTPS + │ + ┌───────────────────────────┴───────────────────────────┐ + │ │ + direct model route execution preset + │ selector/plan/work/review/repair + └───────────────────────────┬───────────────────────────┘ + v + managed credential lease + Control Plane projection + Node sealed lease + │ + v + Node-owned provider +``` + +핵심 보안 경계는 다음과 같다. + +- benchmark caller는 하나의 IOP principal token으로 Edge만 인증한다. +- caller token은 upstream provider credential이 아니다. +- provider credential은 Control Plane에 등록된 slot과 Node 대상 sealed lease에서만 온다. +- Edge는 provider credential을 복호화하지 않고 caller가 보낸 token을 upstream auth로 재사용하지 않는다. +- managed mode는 legacy static principal/provider credential과 혼용하지 않는다. +- config observation은 secret이 아닌 route/model/stage binding의 독립 증거다. + +## 3. 환경 프로필 + +### 3.1 Benchmark runner + +현재 검증된 runner class는 Linux/AArch64다. 명령은 repository root에서 실행한다. + +필수 command: + +```bash +command -v python3 +command -v git +command -v claude +command -v agy +command -v codex +``` + +2026-08-12 확인 snapshot: + +| Tool | 확인된 버전 | 정책 | +|---|---|---| +| Claude Code | `2.1.228` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`/`--help`를 확인한다. | +| agy | `1.1.12` | adapter가 이 버전을 명시적으로 gate한다. 다른 버전은 재검증 전 fail closed한다. | +| Codex CLI | `0.147.0` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`과 `exec --help`를 확인한다. | + +### 3.2 Testbed + +- path: `../iop-s2` +- 현재 확인 branch: `dev` +- 현재 확인 상태: clean +- benchmark는 testbed를 read-only provenance로 취급한다. +- caller별 workspace와 session은 run tree 아래에 새로 만들며 서로 공유하지 않는다. +- testbed를 benchmark 결과로 수정하거나 결과 파일을 다시 복사하지 않는다. + +fixture checksum과 source file 목록은 manifest가 고정한다. testbed HEAD와 clean 상태는 실행일에 다시 확인한다. + +### 3.3 Dev runtime + +현재 검증된 runtime class는 macOS/ARM64 remote dev runner다. exact SSH target과 checkout은 `agent-test/dev/rules.md` 및 `agent-test/inventory-dev.yaml`을 따른다. + +benchmark 관련 runtime 역할: + +| Port | 역할 | +|---:|---| +| `18082` | Edge artifact/bootstrap HTTP | +| `18083` | managed Edge public HTTPS; Anthropic/OpenAI/Gemini caller ingress | +| `18084` | native dev-runtime Edge↔Node TCP | +| `19093` | Edge admin/config refresh | +| `19101` | Edge metrics | + +2026-08-12 read-only 확인에서 위 listener와 managed Edge process는 모두 active였고 remote checkout은 clean release 상태였다. exact commit, binary checksum, process id와 endpoint는 실행 evidence에만 기록하고 이 가이드에 고정하지 않는다. + +최근 완료된 live readiness evidence는 4 connected Nodes와 8 healthy/available providers를 확인했다. 최신 Node/provider 세부와 접속 위치는 반드시 `agent-test/inventory-dev.yaml`에서 다시 확인한다. + +이 benchmark는 compose dev stack의 Edge-Node TCP `19003`이 아니라 native dev-runtime provider pool의 `18084`를 사용한다. compose와 native profile은 포트, process와 판정 evidence가 서로 다르므로 한 실행에서 섞지 않는다. 현재 inventory가 가리키는 배포 산출물은 다음과 같다. + +| Artifact | Path | +|---|---| +| native Edge config | `build/dev-runtime/edge.yaml` | +| Edge binary | `build/dev-runtime/bin/edge` | +| macOS Node binary | `build/dev-runtime/bin/iop-node` | +| Linux ARM64 Node binary | `build/dev-runtime/bin/iop-node-linux-arm64` | +| Windows AMD64 Node binary | `build/dev-runtime/bin/iop-node-windows-amd64.exe` | + +모든 Edge/Node binary는 scored execution 전에 동일 source ref로 rebuild·redeploy·restart한다. `build/dev-runtime/**`의 runtime config와 untracked credential material은 원격 runner가 소유하며 tracked 문서나 testbed로 복사하지 않는다. + +## 4. 보호 파일과 credential 역할 + +benchmark runner의 `token/` 아래에는 다음 파일이 준비돼 있다. 파일 존재와 mode만 확인하며 내용을 출력하지 않는다. + +| Path | 역할 | 실행 시 사용 | +|---|---|---| +| `token/.iop-bench` | benchmark용 IOP principal token | preflight/run/score caller가 Edge를 인증할 때 사용 | +| `token/iop-dev-ca.pem` | managed dev Edge HTTPS CA certificate | `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`로 전달 | +| `token/.claude` | Claude provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 | +| `token/.gemini` | Gemini provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 | +| `token/.gpt` | GPT provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 | + +현재 secret source 파일은 `0600`, CA certificate는 `0644`로 확인됐다. CA certificate는 public trust material이지만 private key는 아니다. + +안전 확인: + +```bash +for benchmark_secret_file in token/.iop-bench token/.claude token/.gemini token/.gpt; do + test -f "$benchmark_secret_file" + test "$(stat -c '%a' "$benchmark_secret_file")" = 600 +done +test -f token/iop-dev-ca.pem +``` + +macOS에서 동일 검사를 수행할 때는 BSD `stat` 문법을 사용한다. 어떤 경우에도 `cat`, `echo`, shell tracing(`set -x`)으로 secret 내용을 출력하지 않는다. + +## 5. Provider와 managed credential 설정 + +현재 dev 구성은 다음 절차로 만들어졌다. + +1. Control Plane credential plane용 CA, workload certificate, at-rest keyring과 lease issuer/recipient key를 operator-owned untracked 경로에 생성했다. +2. Control Plane, Edge와 각 Node에 role/name-bound mTLS identity를 배치했다. +3. Edge public ingress를 HTTPS로 구성하고 benchmark runner에 CA certificate만 전달했다. +4. benchmark principal을 bootstrap하고 one-time token을 `token/.iop-bench`에 저장했다. +5. `token/.claude`, `token/.gemini`, `token/.gpt`의 raw provider key를 credential HTTPS request body로 직접 등록했다. command argument, YAML, tracked docs나 task evidence에는 넣지 않았다. +6. credential slot과 public route를 별도로 생성하고 principal projection에 direct route와 hybrid preset stage authorization을 연결했다. +7. Control Plane → Edge → Nodes 순서로 bounded restart하고 fresh projection, sealed lease, route revision과 no-fallback 동작을 검증했다. + +재구성이 필요하면 `docs/edge-local-dev-guide.md`의 “Managed credential plane and TLS startup”과 “Safe slot lifecycle”을 따른다. 실제 slot id, alias, revision과 lease id는 운영 상태이므로 이 문서에 복사하지 않는다. + +## 6. Caller별 연결 방식 + +| Caller | Edge surface | Child 설정 | 중요한 제한 | +|---|---|---|---| +| Claude Code | Anthropic-compatible Messages | `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`; `--bare --print --verbose --output-format stream-json --no-session-persistence --permission-mode dontAsk --tools Read,Write,Edit --allowedTools Read,Write,Edit` | task는 stdin으로 한 번 제출한다. network/shell 도구 없이 격리 workspace 파일 작업만 허용한다. model/effort는 manifest 값을 그대로 전달한다. | +| agy | Gemini-native `streamGenerateContent` | fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`; `--sandbox --output-format stream-json --model ... --print ...` | `agy 1.1.12`만 승인된다. API-key provider에 `--effort`를 전달하지 않으며 ambient user config를 읽지 않는다. | +| Codex | OpenAI-compatible Responses | fresh session `HOME`, isolated `iop_benchmark` provider override, `IOP_BENCHMARK_API_KEY`; `exec --sandbox workspace-write --json --ephemeral --ignore-user-config --strict-config` | user config를 읽지 않고 격리 workspace에만 쓸 수 있다. current adapter effort는 `xhigh`; caller binding event가 없으면 admitted config binding을 사용하고, 보고된 mismatch는 거부한다. | + +세 caller child에는 필요한 `PATH`와 CA 변수만 allowlist로 전달한다. parent의 caller/provider 설정이나 unrelated secret은 상속하지 않는다. + +agy의 public base는 adapter가 cell별로 다음처럼 route-qualified 한다. + +```text +/gemini/ +``` + +`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 benchmark transport가 아니다. + +official agy planner가 Gemini `generationConfig.responseMimeType`과 `responseSchema` 또는 `responseJsonSchema`를 보내면 Edge는 이를 기존 Chat `response_format`으로 변환한다. function declaration은 official SDK가 사용하는 `parameters`와 `parametersJsonSchema` 표기 중 정확히 하나를 허용한다. 동의어 필드가 동시에 있거나 schema가 JSON object가 아니면 provider dispatch 전에 거부한다. 구조가 유효한 agy `result.status=ERROR`는 stream parser 오류로 바꾸지 않고 caller process 실패로 기록한다. + +## 7. Benchmark 고정 설정 + +| 설정 | 값 | +|---|---| +| pipeline version | `2` | +| environment | `dev` | +| execution seed | `bench-02-c01-c09-v1` | +| repetitions | `1` | +| session policy | `fresh` | +| setup cache policy | `isolated` | +| run timeout | 300 seconds | +| idle timeout | 30 seconds | +| quiet window | 10 seconds | +| cleanup grace | 5 seconds | +| desktop viewport | `1920x1080` | +| mobile viewport | `375x812` | +| output root | `agent-test/runs/bench-02` | +| rubric | `one-shot-agent-comparison-v1` | + +execution seed로 결정되는 현재 slot 순서는 다음과 같다. 표의 C 번호 순서와 실제 실행 순서는 다르다. + +1. C02 Claude→Gemini direct +2. C05 Codex→GPT direct +3. C03 agy→Gemini direct +4. C06 Claude→Gemini hybrid +5. C08 Claude→GPT hybrid +6. C09 Codex→GPT hybrid +7. C01 Claude→Sonnet direct +8. C07 agy→Gemini hybrid +9. C04 Claude→GPT direct + +## 8. C01-C09 matrix와 route binding + +| Cell | Caller | Route kind/id | Requested model/effort | Effective stage binding | +|---|---|---|---|---| +| C01 | Claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / `max` | request=`claude-sonnet-5` max | +| C02 | Claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high | +| C03 | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high | +| C04 | Claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh | +| C05 | Codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh | +| C06 | Claude | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` | +| C07 | agy | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` | +| C08 | Claude | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` | +| C09 | Codex | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` | + +hybrid preset은 caller가 stage를 따로 호출하는 구조가 아니다. 하나의 caller request 안에서 Edge가 selector/plan/work/review/repair를 소유한다. + +## 9. Fixture와 결과 조건 + +공통 task는 fictional product “Lumen Atlas”의 responsive one-page landing page다. + +- prompt: `scripts/fixtures/agent-comparison-benchmark/prompt.md` +- copy: `scripts/fixtures/agent-comparison-benchmark/reference.txt` +- images: `aurora-grid.svg`, `orbit-rings.svg` +- 생성 파일: workspace root의 `index.html`, `styles.css`, `script.js` 정확히 세 개 +- 외부 asset, framework, package manager, build tool, analytics와 network dependency 금지 +- desktop/mobile responsive, semantic HTML, focus/contrast/accessibility 요구 +- 각 attempt는 fresh workspace/session에서 task를 한 번만 제출 + +fixture checksum은 manifest의 값이 유일한 기준이다. prompt나 asset을 변경하면 기존 run과 비교하지 말고 manifest/version/checksum을 함께 갱신하는 별도 작업으로 처리한다. + +## 10. Process environment 준비 + +다음은 value를 출력하지 않는 process-local 예시다. ``를 문서에 실제 값으로 치환하지 말고 실행 환경에서만 주입한다. + +Public live registry가 소비하는 environment contract는 다음과 같다. + +| Variable | 값/의미 | Durable evidence | +|---|---|---| +| `IOP_BENCH_CLAUDE_BASE_URL` | managed Edge HTTPS origin | raw 값 금지; endpoint digest만 허용 | +| `IOP_BENCH_AGY_BASE_URL` | managed Edge HTTPS origin; adapter가 `/gemini/`를 추가 | raw 값 금지; endpoint digest만 허용 | +| `IOP_BENCH_CODEX_BASE_URL` | managed Edge OpenAI-compatible `/v1` base | raw 값 금지; endpoint digest만 허용 | +| `IOP_BENCH_CLAUDE_SECRET_ENV` | Claude가 사용할 secret-bearing variable 이름 | variable 이름만 허용 | +| `IOP_BENCH_AGY_SECRET_ENV` | agy가 사용할 secret-bearing variable 이름 | variable 이름만 허용 | +| `IOP_BENCH_CODEX_SECRET_ENV` | Codex가 사용할 secret-bearing variable 이름 | variable 이름만 허용 | +| `IOP_BENCH_SHARED_TOKEN` | 이 가이드 예시의 secret-bearing variable | 값 기록 금지 | +| `SSL_CERT_FILE` | dev Edge HTTPS CA certificate path | repository-relative file reference만 허용 | +| `NODE_EXTRA_CA_CERTS` | Node.js caller용 동일 CA certificate path | repository-relative file reference만 허용 | +| `IOP_BENCH_CONFIG_OBSERVATION_ENV` | config JSON을 보유한 variable 이름 | variable 이름만 허용 | +| `BENCH_CONFIG` | schema v1 route/model/stage observation JSON | secret은 없지만 runtime과 일치하는 canonical digest만 evidence에 기록 | +| `PATH` | caller binary resolution | resolved executable path/version만 preflight에서 확인 | + +Child adapter가 내부적으로 만드는 값은 caller별로 격리된다. + +- Claude: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, restricted `Read,Write,Edit`, nonessential traffic/autoupdater disable flags와 CA variables +- agy: fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `LANG=C.UTF-8`, `LC_ALL=C.UTF-8`, `TZ=UTC`와 CA variables +- Codex: fresh session `HOME`, `IOP_BENCHMARK_API_KEY`, strict ephemeral provider override와 CA variables + +이 child variable은 사용자가 별도로 준비할 값이 아니다. live registry가 위 public contract에서 파생하며, parent의 같은 이름 값을 그대로 신뢰하거나 상속하지 않는다. + +```bash +set -euo pipefail + +benchmark_edge_origin="${BENCHMARK_EDGE_ORIGIN:?set BENCHMARK_EDGE_ORIGIN to the managed dev Edge HTTPS origin}" +read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench +export IOP_BENCH_SHARED_TOKEN + +export IOP_BENCH_CLAUDE_BASE_URL="$benchmark_edge_origin" +export IOP_BENCH_AGY_BASE_URL="$benchmark_edge_origin" +export IOP_BENCH_CODEX_BASE_URL="$benchmark_edge_origin/v1" + +export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN +export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN + +export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem" +export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem" + +export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG +BENCH_CONFIG="$(python3 - <<'PY' +import json + +routes = [ + {"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]}, + {"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]}, + {"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]}, + {"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]}, +] +print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":"))) +PY +)" +export BENCH_CONFIG +``` + +중요한 의미: + +- `IOP_BENCH_*_SECRET_ENV`의 값은 secret이 아니라 실제 secret을 보유한 environment variable의 이름이다. +- 세 caller는 같은 IOP principal을 사용하지만 서로 다른 protocol base를 받는다. +- `BENCH_CONFIG`는 runtime에서 독립적으로 확인한 route/stage snapshot이어야 한다. manifest를 보고 임의 합성한 값을 live readiness evidence로 사용하면 안 된다. +- endpoint와 config는 evidence에 raw 값 대신 digest identity로만 남는다. + +작업 후에는 같은 shell에서 다음 변수를 제거한다. + +```bash +unset IOP_BENCH_SHARED_TOKEN +unset IOP_BENCH_CLAUDE_BASE_URL IOP_BENCH_AGY_BASE_URL IOP_BENCH_CODEX_BASE_URL +unset IOP_BENCH_CLAUDE_SECRET_ENV IOP_BENCH_AGY_SECRET_ENV IOP_BENCH_CODEX_SECRET_ENV +unset IOP_BENCH_CONFIG_OBSERVATION_ENV BENCH_CONFIG +unset SSL_CERT_FILE NODE_EXTRA_CA_CERTS +unset BENCHMARK_EDGE_ORIGIN benchmark_edge_origin +``` + +## 11. 실행 절차 + +manifest path는 모든 명령에서 동일하게 사용한다. + +```bash +benchmark_manifest=scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json +``` + +### 11.1 Source와 deterministic verification + +```bash +python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test +python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' +python3 scripts/agent_comparison_benchmark.py validate --manifest "$benchmark_manifest" +git diff --check +``` + +필요하면 변경 범위에 따라 Go tests와 managed credential qualification도 실행한다. + +```bash +go test -count=1 ./... +credential_smoke_parent="$(mktemp -d /tmp/iop-benchmark-credential.XXXXXX)" +TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke +rmdir "$credential_smoke_parent" +``` + +### 11.2 Direct-first qualification + +동일 clean source ref를 모든 runtime binary에 배포하고 4/4 Node와 provider health를 확인한 다음, 먼저 기존 5-cell direct manifest를 사용한다. + +```bash +direct_manifest="scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json" +python3 scripts/agent_comparison_benchmark.py preflight --manifest "$direct_manifest" +python3 scripts/agent_comparison_benchmark.py run --manifest "$direct_manifest" +``` + +direct canary는 unscored이며 각 최신 slot의 `product=succeeded`, `harness=passed`, `process=exited`/exit 0/no signal, `artifact=passed`가 모두 5/5여야 한다. `running`, `interrupted`, `unresolved`는 0이어야 한다. 실패 시 hybrid 또는 9-cell scored run을 할당하지 않는다. + +### 11.3 Public nine-cell preflight + +10절의 환경을 같은 shell에 준비한 뒤 실행한다. + +```bash +python3 scripts/agent_comparison_benchmark.py preflight --manifest "$benchmark_manifest" +``` + +성공 조건: + +```text +status=ready ready=9 registration_required=0 implementation_gap=0 +``` + +preflight는 다음을 함께 확인한다. + +- caller binary/version/help +- Edge model catalog +- principal auth와 endpoint compatibility +- direct/preset route 존재 +- requested model/effort +- exact stage binding과 order +- official agy transport capability + +`registration_required` 또는 `implementation_gap`이면 즉시 중단한다. alias, model, effort, route나 caller를 대체하지 않는다. preflight-only run root는 evidence이므로 삭제하지 않는다. + +direct canary 5/5 뒤 fresh one-shot preflight가 `ready=9`인지 확인하고 멈춘다. 이 qualification 단계에서는 hybrid canary나 C01-C09 `run`을 호출하지 않는다. + +### 11.4 Scored execution + +fresh preflight와 명시적 실행 권한이 있는 현재 plan에서만 다음 명령을 한 번 호출한다. + +```bash +python3 scripts/agent_comparison_benchmark.py run --manifest "$benchmark_manifest" +``` + +- direct CLI stdout/stderr와 exit code를 보존한다. +- CLI가 출력한 canonical `run-...` id만 이후 `status`, `score`, `report`에 사용한다. +- command가 nonzero여도 같은 plan에서 `run`을 다시 호출하지 않는다. +- CLI가 run id를 출력하지 않으면 임의 id나 성공 pointer를 만들지 않는다. +- caller나 provider를 CLI 밖에서 별도로 호출해 scored result를 보충하지 않는다. + +### 11.5 Status + +```bash +python3 scripts/agent_comparison_benchmark.py status \ + --manifest "$benchmark_manifest" \ + --run-id "$benchmark_run_id" +``` + +현재 comparison execution 완료 조건: + +- controller terminal 수 `completed + timed_out + cancelled + interrupted = 9` +- `running = 0` +- `interrupted = 0` +- 각 cell/repetition에 retained terminal attempt가 존재 +- 최신 attempt에서 `product_succeeded=9`, `harness_passed=9`, `process_exited=9`, `artifact_passed=9`, `unresolved=0` + +`completed`는 controller 종료만 뜻하며 product 성공을 뜻하지 않는다. product/harness/process/artifact 실패는 scoring eligibility와 최종 비교에서 각각 별도로 표시된다. + +### 11.6 Blind scoring + +유효한 execution run에 대해서만 수행한다. + +```bash +python3 scripts/agent_comparison_benchmark.py score \ + --manifest "$benchmark_manifest" \ + --run-id "$benchmark_run_id" +``` + +evaluator는 Codex→`gpt-5.6-luna` xhigh direct route다. identity가 제거된 blind workspace만 보며 source cell identity mapping은 blind tree 밖에 유지한다. + +product, harness, acceptable process 또는 artifact gate 실패는 각각의 reason을 가진 `unscored`이고 0점으로 바꾸지 않는다. `scoring_failed`도 명시적 `--retry-scoring-failed` 권한 없이 재시도하지 않는다. + +### 11.7 Report + +```bash +python3 scripts/agent_comparison_benchmark.py report \ + --manifest "$benchmark_manifest" \ + --run-id "$benchmark_run_id" +``` + +report는 run root의 immutable evidence를 읽어 idempotent `report.md`를 만든다. 기존 report 내용과 새 projection이 다르면 덮어쓰지 않고 실패한다. + +## 12. 수집 evidence + +각 attempt는 다음 범주의 evidence를 가진다. + +| 범주 | 내용 | +|---|---| +| lifecycle | typed caller terminal, submission, first output, finish, idle, quiet와 독립 product/harness/process 결과 | +| timeline | submitted, first output, first workspace write observation/mtime, total duration | +| usage | input/output/reasoning/cache read/cache write/total tokens, model/tool calls와 duration | +| workspace | fresh session identity, fixture checksum, testbed provenance, generated file tree | +| web validation | product/harness 결과와 무관하게 모든 terminal workspace에서 생성되는 generated files, static safety, images, network, console, responsive, accessibility | +| screenshots | desktop `1920x1080`, mobile `375x812` | +| scoring | eligibility, blind allocation, rubric worksheet, score status | + +caller가 보고하지 않은 metric은 0으로 만들지 않고 `unavailable`과 reason/source를 보존한다. + +Codex current usage mapping: + +| Caller JSONL field | Canonical metric | +|---|---| +| `input_tokens` | `input_tokens` | +| `cached_input_tokens` | `cached_input_tokens` | +| `cache_write_input_tokens` | `cache_write_tokens` | +| `output_tokens` | `output_tokens` | +| `reasoning_output_tokens` | `reasoning_tokens` | +| `total_tokens` | `total_tokens` | + +누락된 `total_tokens`는 하위 category 합으로 재구성하지 않는다. + +## 13. 자동 gate와 100점 rubric + +automatic web gates는 scoring eligibility만 결정하고 점수에 포함되지 않는다. + +| Gate | +|---| +| generated files | +| static safety | +| local images | +| no external network dependency | +| console safety | +| responsive layout | +| accessibility | + +quality rubric: + +| Category | Max | +|---|---:| +| requirements fidelity | 25 | +| visual completeness | 25 | +| responsive accessibility | 15 | +| image/detail usage | 10 | +| behavior stability | 10 | +| code quality | 10 | +| self verification | 5 | +| Total | 100 | + +## 14. 실패 처리와 재개 원칙 + +| 상황 | 조치 | +|---|---| +| manifest invalid | source/manifest를 수정하고 validate부터 다시 시작한다. run을 만들지 않는다. | +| preflight not ready | blocker를 해결하고 fresh preflight한다. attempt를 할당하지 않는다. | +| caller launch 전 interruption | retained evidence를 보존한다. run tree를 직접 수정하지 않는다. | +| lifecycle/parser failure | exact retained output으로 source 원인을 수정하고 deterministic regression을 추가한다. | +| completed이지만 product/harness/process/artifact gate 실패 또는 timed_out/cancelled | 각 축 evidence로 보존한다. 암묵 retry하지 않는다. | +| state가 `running`이지만 process가 없음 | manual JSON 수정/삭제/reconcile을 하지 않는다. reviewer evidence로 남기고 승인된 새 plan에서만 다음 상태를 결정한다. | +| scoring_failed | 0점 처리하지 않는다. 명시적 retry 권한 없이는 중단한다. | +| report unavailable | run evidence를 수정하거나 report를 수작업 생성하지 않는다. | + +현재 milestone의 원칙은 old incomplete/failed run을 `resume --retry-failed`하지 않고 distinct fresh run을 만드는 것이다. 일반 CLI가 `resume`을 지원한다는 사실이 현재 benchmark에서 사용 권한을 뜻하지 않는다. + +## 15. Secret-safe 기록 규칙 + +다음 값은 tracked docs, task review, run metadata, log, metric label과 command argument에 남기지 않는다. + +- IOP principal token +- raw provider credential +- private key, at-rest key, issuer/recipient private key +- slot alias와 lease id +- credential-bearing URL +- raw prompt/response, tool input/output +- caller/provider session content + +허용되는 내용: + +- secret file의 상대 path와 존재/mode +- route/model/stage 이름 +- safe credential slot reference/revision +- hashed endpoint/config/spec identity +- redacted lifecycle 상태, duration과 usage count +- run id와 attempt identity가 필요한 controller evidence + +incident evidence를 보존하기 전에 repository와 run output에서 secret 원문이 없는지 확인한다. 의심되는 artifact는 내용을 복사하지 말고 path와 redaction failure만 보고한다. + +## 16. 실행 전 체크리스트 + +- [ ] 현재 manifest validation 통과 +- [ ] focused/full deterministic tests fresh PASS +- [ ] benchmark runner와 `../iop-s2` provenance 확인 +- [ ] Claude/agy/Codex command와 current version/help 확인 +- [ ] `token/.iop-bench`, CA와 provider provisioning source 존재/mode 확인; 내용 출력 없음 +- [ ] dev runtime source/build identity, process와 listener 확인 +- [ ] managed projection, provider slot/route와 no-legacy-fallback 확인 +- [ ] config observation이 runtime route/stage와 정확히 일치 +- [ ] public preflight `ready=9` +- [ ] 현재 plan이 exactly one scored run을 소유하고 사용자 권한이 명확함 +- [ ] old run resume/retry/state edit 계획 없음 +- [ ] run 이후 status, scoring, report의 run id 전달 경로 준비 + +## 17. 관련 구현 + +- controller/state: `scripts/agent_benchmark/attempts.py` +- manifest: `scripts/agent_benchmark/manifest.py` +- lifecycle: `scripts/agent_benchmark/lifecycle.py` +- workspace isolation: `scripts/agent_benchmark/workspace.py` +- live routing/admission: `scripts/agent_benchmark/live_iop.py` +- Claude adapter: `scripts/agent_benchmark/claude_iop.py` +- agy adapter: `scripts/agent_benchmark/agy_iop.py` +- Codex adapter: `scripts/agent_benchmark/codex_iop.py` +- measurement: `scripts/agent_benchmark/measurement.py` +- browser/web gate: `scripts/agent_benchmark/web_validation.py` +- blind scoring: `scripts/agent_benchmark/scoring.py` +- rubric: `scripts/agent_benchmark/rubric.py` +- report: `scripts/agent_benchmark/reporting.py` diff --git a/docs/edge-local-dev-guide.md b/docs/edge-local-dev-guide.md index a062cda4..d0d32c9a 100644 --- a/docs/edge-local-dev-guide.md +++ b/docs/edge-local-dev-guide.md @@ -253,6 +253,31 @@ rmdir "$credential_smoke_parent" The deterministic Messages qualification succeeds alongside Chat: the Control Plane canonicalizes built-in lowercase API-key header names (for example `x-api-key` to `X-Api-Key`) before signing the lease scope, so both managed profiles reach Node/upstream exactly once with their exact header semantics. A lease failure fails closed before dispatch and never falls back to caller auth or another slot; treat a Chat-only result or any fallback as a qualification failure. +### Official agy route smoke + +공식 `agy` 1.1.12 API-key provider는 upstream Gemini key가 아니라 관리형 IOP principal token을 사용한다. dev operator가 이미 발급한 token과 CA 파일을 보호된 `token/` 아래에 둔 경우 값을 명령행에 직접 쓰지 않고 다음처럼 읽는다. + +```bash +read -r IOP_BENCH_TOKEN < token/.iop-bench +export GEMINI_API_KEY="$IOP_BENCH_TOKEN" +export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem" +export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem" + +GOOGLE_GEMINI_BASE_URL="https://:/gemini/" \ + agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \ + --print 'Reply only with OK. Do not use tools or modify files.' + +GOOGLE_GEMINI_BASE_URL="https://:/gemini/" \ + agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \ + --print 'Inspect README.md and report only its first Markdown heading. Do not modify files.' + +unset IOP_BENCH_TOKEN GEMINI_API_KEY SSL_CERT_FILE NODE_EXTRA_CA_CERTS +``` + +`--effort`는 API-key provider 호출에 넣지 않는다. direct와 hybrid 모두 JSONL의 마지막 record가 `event=result`, 중첩 `result.status=SUCCESS` 한 건이어야 한다. hybrid는 plan/work/review가 포함되므로 direct보다 오래 걸릴 수 있으며, caller timeout을 이유로 같은 scored attempt를 재실행하지 않는다. Gemini ingress와 공식 event 구조의 상세 계약은 `agent-contract/outer/gemini-compatible-api.md`를 기준으로 한다. + +benchmark 전체 preflight에서는 Claude/agy/Codex에 같은 IOP principal을 secret environment reference로 연결하고 `IOP_BENCH_CONFIG_OBSERVATION_ENV`가 가리키는 operator-owned route/binding observation을 함께 제공한다. 사설 CA 환경은 각 isolated caller child에 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`로 전달된다. preflight가 모든 cell을 `ready`로 판정하기 전에는 scored `run`을 시작하지 않는다. + ### Incident redaction check Before retaining logs or evidence, reject any artifact containing an IOP bearer token, provider credential, slot alias, lease id, certificate private key, keyring material, recipient/issuer private key, target URL with credentials, prompt, or response body. Public metrics may contain only stable safe references such as `credential_slot_ref` and `credential_revision`; request/run/session/attempt/node ids and raw payloads are not credential-attribution labels. diff --git a/packages/go/singlerequesttemplate/template.go b/packages/go/singlerequesttemplate/template.go index a2399adc..e16ce2df 100644 --- a/packages/go/singlerequesttemplate/template.go +++ b/packages/go/singlerequesttemplate/template.go @@ -60,6 +60,12 @@ type ReviewFields struct { Summary string } +type PlanFields struct { + Goal string + Steps []string + Verification []string +} + // exactLineOffsets returns the byte offset of the first standalone line equal to // want and how many standalone lines matched. A standalone line is a maximal // "\n"-delimited segment compared byte for byte, so decorated variants such as @@ -244,50 +250,117 @@ func ParsePlan(tmpl string, rawOutput string, maxOutputBytes int) ([]byte, error if f3 == "" { vVerif = rem } else { - if !strings.HasSuffix(rem, f3) { - return nil, ErrMalformedPlan + suffix := f3 + if !strings.HasSuffix(rem, suffix) { + // Provider chat APIs commonly omit the model's final line feed. Treat + // only that last byte as optional; all other static suffix text must + // still match the configured template exactly. + if !strings.HasSuffix(f3, "\n") { + return nil, ErrMalformedPlan + } + suffix = strings.TrimSuffix(f3, "\n") + if !strings.HasSuffix(rem, suffix) { + return nil, ErrMalformedPlan + } } - vVerif = rem[:len(rem)-len(f3)] + vVerif = rem[:len(rem)-len(suffix)] } - trimmedGoal := strings.TrimSpace(vGoal) - if trimmedGoal == "" || strings.Contains(trimmedGoal, "\n") { - return nil, ErrMalformedPlan - } - - trimmedSteps := strings.TrimSpace(vSteps) - if trimmedSteps == "" { - return nil, ErrMalformedPlan - } - stepLines := strings.Split(trimmedSteps, "\n") - if len(stepLines) < 2 || len(stepLines) > 6 { - return nil, ErrMalformedPlan - } - for _, l := range stepLines { - trimmedLine := strings.TrimSpace(l) - if !strings.HasPrefix(trimmedLine, "- ") || strings.TrimSpace(trimmedLine[2:]) == "" { - return nil, ErrMalformedPlan - } - } - - trimmedVerif := strings.TrimSpace(vVerif) - if trimmedVerif == "" { - return nil, ErrMalformedPlan - } - verifLines := strings.Split(trimmedVerif, "\n") - if len(verifLines) < 1 || len(verifLines) > 3 { - return nil, ErrMalformedPlan - } - for _, l := range verifLines { - trimmedLine := strings.TrimSpace(l) - if !strings.HasPrefix(trimmedLine, "- ") || strings.TrimSpace(trimmedLine[2:]) == "" { - return nil, ErrMalformedPlan - } + if _, _, _, err := normalizePlanSections(vGoal, vSteps, vVerif); err != nil { + return nil, err } return []byte(rawOutput), nil } +func normalizePlanSections(goal, steps, verification string) (string, string, string, error) { + goal = strings.TrimSpace(goal) + steps = strings.TrimSpace(steps) + verification = strings.TrimSpace(verification) + if goal == "" || strings.ContainsAny(goal, "\r\n") || steps == "" || verification == "" { + return "", "", "", ErrMalformedPlan + } + if strings.Contains(goal, "{{") || strings.Contains(goal, "}}") || strings.Contains(steps, "{{") || strings.Contains(steps, "}}") || strings.Contains(verification, "{{") || strings.Contains(verification, "}}") { + return "", "", "", ErrMalformedPlan + } + + normalizeBullets := func(value string, minimum, maximum int) (string, error) { + lines := strings.Split(value, "\n") + if len(lines) < minimum || len(lines) > maximum { + return "", ErrMalformedPlan + } + for i, line := range lines { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "- ") || strings.TrimSpace(line[2:]) == "" { + return "", ErrMalformedPlan + } + lines[i] = line + } + return strings.Join(lines, "\n"), nil + } + + steps, err := normalizeBullets(steps, 2, 6) + if err != nil { + return "", "", "", err + } + verification, err = normalizeBullets(verification, 1, 3) + if err != nil { + return "", "", "", err + } + return goal, steps, verification, nil +} + +func normalizePlanFields(fields PlanFields) (string, string, string, error) { + goal := strings.TrimSpace(fields.Goal) + if goal == "" || strings.ContainsAny(goal, "\r\n") || strings.Contains(goal, "{{") || strings.Contains(goal, "}}") { + return "", "", "", ErrMalformedPlan + } + normalizeItems := func(items []string, minimum, maximum int) (string, error) { + if len(items) < minimum || len(items) > maximum { + return "", ErrMalformedPlan + } + lines := make([]string, len(items)) + for i, item := range items { + item = strings.TrimSpace(item) + if item == "" || strings.ContainsAny(item, "\r\n") || strings.Contains(item, "{{") || strings.Contains(item, "}}") { + return "", ErrMalformedPlan + } + lines[i] = "- " + item + } + return strings.Join(lines, "\n"), nil + } + steps, err := normalizeItems(fields.Steps, 2, 6) + if err != nil { + return "", "", "", err + } + verification, err := normalizeItems(fields.Verification, 1, 3) + if err != nil { + return "", "", "", err + } + return goal, steps, verification, nil +} + +func RenderPlan(tmpl string, fields PlanFields, maxOutputBytes int) ([]byte, error) { + if maxOutputBytes < 1 { + return nil, ErrMalformedPlan + } + if err := ValidatePlanTemplate(tmpl); err != nil { + return nil, err + } + + goal, steps, verification, err := normalizePlanFields(fields) + if err != nil { + return nil, err + } + res := strings.ReplaceAll(tmpl, "{{goal}}", goal) + res = strings.ReplaceAll(res, "{{steps}}", steps) + res = strings.ReplaceAll(res, "{{verification}}", verification) + if strings.Contains(res, "{{") || strings.Contains(res, "}}") || len(res) > maxOutputBytes { + return nil, ErrMalformedPlan + } + return []byte(res), nil +} + func RenderReview(tmpl string, fields ReviewFields, maxOutputBytes int) ([]byte, error) { if maxOutputBytes < 1 { return nil, ErrMalformedReview diff --git a/packages/go/singlerequesttemplate/template_test.go b/packages/go/singlerequesttemplate/template_test.go index a87c950c..71f4b745 100644 --- a/packages/go/singlerequesttemplate/template_test.go +++ b/packages/go/singlerequesttemplate/template_test.go @@ -290,6 +290,63 @@ Fix single-request template handling bug. maxOutputBytes: 1024, wantErr: false, }, + { + name: "provider omits final line feed", + tmpl: singlerequesttemplate.DefaultPlanTemplate, + raw: strings.TrimSuffix(validOutput, "\n"), + maxOutputBytes: 1024, + wantErr: false, + }, + { + name: "provider omits only final line feed after static suffix", + tmpl: `# Plan + +## Goal +{{goal}} + +## Steps +{{steps}} + +## Verification +{{verification}} + +END +`, + raw: `# Plan + +## Goal +Fix suffix parsing. + +## Steps +- Keep the static suffix. +- Allow the final line feed omission. + +## Verification +- Run the parser tests. + +END`, + maxOutputBytes: 1024, + wantErr: false, + }, + { + name: "provider omits static suffix text", + tmpl: `# Plan + +## Goal +{{goal}} + +## Steps +{{steps}} + +## Verification +{{verification}} + +END +`, + raw: strings.TrimSuffix(validOutput, "\n"), + maxOutputBytes: 1024, + wantErr: true, + }, { name: "boundary steps = 6, verif = 3", tmpl: singlerequesttemplate.DefaultPlanTemplate, @@ -470,6 +527,61 @@ Fix {{goal}} bug. } } +func TestRenderPlan(t *testing.T) { + fields := singlerequesttemplate.PlanFields{ + Goal: " Inspect the target. ", + Steps: []string{" Step one. ", "\tStep two. "}, + Verification: []string{" Run focused tests. "}, + } + want := "# Plan\n\n## Goal\nInspect the target.\n\n## Steps\n- Step one.\n- Step two.\n\n## Verification\n- Run focused tests.\n" + got, err := singlerequesttemplate.RenderPlan(singlerequesttemplate.DefaultPlanTemplate, fields, 1024) + if err != nil { + t.Fatal(err) + } + if string(got) != want { + t.Fatalf("RenderPlan() = %q, want %q", got, want) + } + + custom := "# Plan\n\nOperator note.\n\n## Goal\n{{goal}}\n\n## Steps\n{{steps}}\n\n## Verification\n{{verification}}\n\nEND\n" + wantCustom := "# Plan\n\nOperator note.\n\n## Goal\nInspect the target.\n\n## Steps\n- Step one.\n- Step two.\n\n## Verification\n- Run focused tests.\n\nEND\n" + got, err = singlerequesttemplate.RenderPlan(custom, fields, 1024) + if err != nil { + t.Fatal(err) + } + if string(got) != wantCustom { + t.Fatalf("custom RenderPlan() = %q, want %q", got, wantCustom) + } +} + +func TestRenderPlanRejectsMalformedFields(t *testing.T) { + valid := singlerequesttemplate.PlanFields{Goal: "Fix bug.", Steps: []string{"Step 1", "Step 2"}, Verification: []string{"Verify 1"}} + tests := []struct { + name string + fields singlerequesttemplate.PlanFields + max int + }{ + {"empty-goal", singlerequesttemplate.PlanFields{Steps: valid.Steps, Verification: valid.Verification}, 1024}, + {"multiline-goal", singlerequesttemplate.PlanFields{Goal: "Line 1\nLine 2", Steps: valid.Steps, Verification: valid.Verification}, 1024}, + {"one-step", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: []string{"Step 1"}, Verification: valid.Verification}, 1024}, + {"seven-steps", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: []string{"S1", "S2", "S3", "S4", "S5", "S6", "S7"}, Verification: valid.Verification}, 1024}, + {"empty-step", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: []string{"Step 1", ""}, Verification: valid.Verification}, 1024}, + {"multiline-step", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: []string{"Step 1\ncontinued", "Step 2"}, Verification: valid.Verification}, 1024}, + {"four-verifications", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: valid.Steps, Verification: []string{"V1", "V2", "V3", "V4"}}, 1024}, + {"empty-verification", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: valid.Steps, Verification: []string{""}}, 1024}, + {"multiline-verification", singlerequesttemplate.PlanFields{Goal: valid.Goal, Steps: valid.Steps, Verification: []string{"Verify 1\ncontinued"}}, 1024}, + {"unresolved-token", singlerequesttemplate.PlanFields{Goal: "Fix {{goal}}.", Steps: valid.Steps, Verification: valid.Verification}, 1024}, + {"zero-max", valid, 0}, + {"output-over-limit", valid, 10}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if _, err := singlerequesttemplate.RenderPlan(singlerequesttemplate.DefaultPlanTemplate, tt.fields, tt.max); err == nil { + t.Fatal("RenderPlan() succeeded, want error") + } + }) + } +} + func TestValidateReviewTemplate(t *testing.T) { tests := []struct { name string diff --git a/scripts/agent_benchmark/agy_iop.py b/scripts/agent_benchmark/agy_iop.py index 710dbcfa..dede2900 100644 --- a/scripts/agent_benchmark/agy_iop.py +++ b/scripts/agent_benchmark/agy_iop.py @@ -22,48 +22,60 @@ from scripts.agent_benchmark.connectivity import ( CallerCapability, ConnectivityResult, ConnectivityIssue, - EffectiveBinding, RequestedEffectiveBinding, classify_issues, make_result, ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, + CALLER_STATUS_FAILED, + CALLER_STATUS_SUCCEEDED, COMPLETION_EXIT_AFTER_IDLE, - SUBMISSION_STDIN_ONCE, + SUBMISSION_ARGV_TASK, + CallerEvent, + CallerTerminal, InvocationResult, InvocationSpec, LifecycleMetricError, ParsedMetric, SupervisorLocator, + TLS_CA_ENV_KEYS, + count_metric, duration_metric, env_pairs, exact_value_redactor, - is_reported_number, + inherited_tls_ca_environment, run_invocation, ) -from scripts.agent_benchmark.manifest import MatrixCell, TOKEN_RE, Timeout +from scripts.agent_benchmark.manifest import MatrixCell, Timeout from scripts.agent_benchmark.workspace import PreparedWorkspace AGY_CALLER = "agy" -AGY_KNOWN_VERSION = "1.1.11" -AGY_PROVIDER_ENV = "AGY_PROVIDER" -AGY_ENDPOINT_ENV = "AGY_OPENAI_BASE_URL" -AGY_AUTH_ENV = "AGY_OPENAI_API_KEY" -_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE) -_SAFE_EVENT_FIELDS = ("type", "subtype", "model", "effort", "route_kind", "route_id") -_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model", "--effort") -_DOCUMENTED_ENVIRONMENT = (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV) -AGY_SAFE_METRIC_LABELS = ("metric:duration_ms",) -# agy reports whole durations in milliseconds. Only these subtypes are -# converted, and a model-stage duration is marked as overlapping because it is -# reported inside the same window as the total. -AGY_DURATION_METRICS = { - "duration_ms": ("total_duration", False), - "model_duration_ms": ("model_duration", True), - "queue_duration_ms": ("queue_duration", False), +AGY_KNOWN_VERSION = "1.1.12" +AGY_ENDPOINT_ENV = "GOOGLE_GEMINI_BASE_URL" +AGY_AUTH_ENV = "GEMINI_API_KEY" +AGY_SETTINGS_RELATIVE_PATH = Path(".gemini/antigravity-cli/settings.json") +_AGY_ISOLATED_SETTINGS = { + "enableTelemetry": False, + "modelProvider": "gemini", + "toolPermission": "always-proceed", +} +AGY_MODEL_LABELS = { + "gemini-3.6-flash": "Gemini 3.6 Flash", + "gemini-hybrid": "Gemini 3.6 Flash", +} +_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE) +_SAFE_EVENT_FIELDS = ("event", "state", "step_type", "status") +_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model") +_AGY_USAGE_METRICS = { + "input_tokens": "input_tokens", + "cache_read_tokens": "cached_input_tokens", + "output_tokens": "output_tokens", + "thinking_tokens": "reasoning_tokens", + "total_tokens": "total_tokens", } -_AGY_METRIC_KEYS = frozenset({"type", "subtype", "value", "model"}) _IDENTITY_RE = re.compile(r"sha256:[0-9a-f]{64}\Z") @@ -162,7 +174,7 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili return AgyDocumentedCapabilities((), (), ()) return AgyDocumentedCapabilities( tuple(token for token in _DOCUMENTED_OPTIONS if _exact_token_present(help_output, token)), - tuple(token for token in _DOCUMENTED_ENVIRONMENT if _exact_token_present(help_output, token)), + (), ("stream-json",) if _exact_token_present(help_output, "stream-json") else (), ) @@ -170,9 +182,9 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapability: """Inspect only public, versioned help text for the closed IOP transport. - A version string is accepted only when it names the known agy release and - every required transport variable is documented. This prevents a new or - partially documented client from silently inheriting ambient provider state. + The known release is pinned because its Gemini provider environment is not + printed by ``--help``. The public options and stream format still have to + match exactly; a changed release fails closed until re-qualified. """ if not isinstance(version_output, str) or not isinstance(help_output, str): return AgyCapability(None, False, False, False, False, False, (), ()) @@ -180,11 +192,11 @@ def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapa version = matched.group(1) if matched else None known_version = version == AGY_KNOWN_VERSION documented = parse_documented_agy_capabilities(help_output) - endpoint_supported = AGY_ENDPOINT_ENV in documented.environment - auth_supported = AGY_AUTH_ENV in documented.environment + endpoint_supported = known_version + auth_supported = known_version protocol_supported = known_version and all( option in documented.options for option in _DOCUMENTED_OPTIONS - ) and AGY_PROVIDER_ENV in documented.environment + ) stream_supported = "stream-json" in documented.output_formats supported = endpoint_supported and auth_supported and protocol_supported and stream_supported if not supported: @@ -238,7 +250,14 @@ def validate_agy_iop_runtime( if not all(isinstance(value, str) and value for value in (runtime.binary, runtime.endpoint, runtime.credential)): raise AgyAdapterError("agy runtime values are unavailable") endpoint = urlsplit(runtime.endpoint) - if endpoint.scheme not in ("http", "https") or not endpoint.netloc or endpoint.query or endpoint.fragment: + route_path = f"/gemini/{cell.iop.route_id}" + if ( + endpoint.scheme != "https" + or not endpoint.netloc + or endpoint.query + or endpoint.fragment + or endpoint.path.rstrip("/") != route_path + ): raise AgyAdapterError("agy IOP endpoint is invalid") _validate_config_owner_observation(cell, observation) if observation.endpoint_identity != _runtime_identity("endpoint", runtime.endpoint): @@ -269,7 +288,7 @@ def preflight_agy_iop( issues.append(_issue("endpoint_incompatible")) if not runtime.credential: issues.append(_issue("credential_missing")) - if not cell.iop.request_model: + if cell.iop.request_model not in AGY_MODEL_LABELS: issues.append(_issue("model_missing")) if not runtime.endpoint: issues.append(_issue("endpoint_incompatible")) @@ -307,16 +326,27 @@ def build_agy_invocation( timeout: Timeout, preflight: AgyPreflightResult, ) -> InvocationSpec: - """Build one isolated stdin-only agy invocation after a ready preflight.""" + """Build one isolated official agy print invocation after a ready preflight.""" if preflight.status != "ready" or not preflight.capability.iop_transport_supported or preflight.runtime is None: raise AgyAdapterError("agy IOP transport is not proven") runtime = preflight.runtime if not runtime.binary or not Path(runtime.binary).is_file(): raise AgyAdapterError("agy binary is unavailable") - if not isinstance(prepared, PreparedWorkspace) or not prepared.workspace_dir: + if ( + not isinstance(prepared, PreparedWorkspace) + or not Path(prepared.workspace_dir).is_dir() + or not Path(prepared.session_dir).is_dir() + or not Path(prepared.attempt_root).is_dir() + ): raise AgyAdapterError("prepared workspace is unavailable") if not isinstance(task_payload, bytes) or not task_payload: raise AgyAdapterError("agy task payload is unavailable") + try: + task_text = task_payload.decode("utf-8") + except UnicodeDecodeError as exc: + raise AgyAdapterError("agy task payload must be UTF-8") from exc + + _stage_agy_provider_settings(prepared.session_dir) # The child receives a minimal environment and explicit IOP-only provider # settings. No parent agy/Gemini config or session variable is inherited. @@ -325,33 +355,55 @@ def build_agy_invocation( "LANG": "C.UTF-8", "LC_ALL": "C.UTF-8", "TZ": "UTC", - AGY_PROVIDER_ENV: "iop-openai", + "HOME": prepared.session_dir, AGY_ENDPOINT_ENV: runtime.endpoint, AGY_AUTH_ENV: runtime.credential, } + environment.update(inherited_tls_ca_environment()) return InvocationSpec( argv=( runtime.binary, - "--print", "--sandbox", "--output-format", "stream-json", - "--model", cell.iop.request_model, - "--effort", cell.iop.requested_effort, + "--model", AGY_MODEL_LABELS[cell.iop.request_model], + "--print", task_text, ), cwd=prepared.workspace_dir, env=env_pairs(environment), - env_allowlist=(AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV), - submission_mode=SUBMISSION_STDIN_ONCE, + env_allowlist=(AGY_ENDPOINT_ENV, AGY_AUTH_ENV, *TLS_CA_ENV_KEYS), + submission_mode=SUBMISSION_ARGV_TASK, completion_mode=COMPLETION_EXIT_AFTER_IDLE, timeout=timeout, evidence_dir=prepared.attempt_root, - task_payload=task_payload, control_dir=str(Path(prepared.attempt_root) / "agy-control"), ) +def _stage_agy_provider_settings(session_dir: str) -> None: + """Create the exact secret-free provider selector in one isolated HOME.""" + try: + session = Path(session_dir).resolve(strict=True) + if not session.is_dir(): + raise OSError("session is not a directory") + settings = session / AGY_SETTINGS_RELATIVE_PATH + settings.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + if settings.parent.resolve(strict=True) != session / AGY_SETTINGS_RELATIVE_PATH.parent: + raise OSError("settings directory escapes session") + payload = ( + json.dumps(_AGY_ISOLATED_SETTINGS, sort_keys=True, separators=(",", ":")) + "\n" + ).encode("utf-8") + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(settings, flags, 0o600) + with os.fdopen(descriptor, "wb") as handle: + handle.write(payload) + except (OSError, ValueError) as exc: + raise AgyAdapterError("agy isolated provider settings are unavailable") from exc + + def _safe_identifier(value: Any) -> str | None: - return value if isinstance(value, str) and TOKEN_RE.fullmatch(value) else None + return value if isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:+-]{0,63}", value) else None def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> str: @@ -362,128 +414,128 @@ def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> s return '{"event":"unparseable"}' if not isinstance(parsed, dict): return '{"event":"unparseable"}' - safe: dict[str, str] = {} - for field in _SAFE_EVENT_FIELDS: - value = _safe_identifier(parsed.get(field)) + event = _safe_identifier(parsed.get("event")) + if event is None or event in sensitive_values: + return '{"event":"unparseable"}' + safe: dict[str, str] = {"event": event} + payload = parsed.get(event) + if not isinstance(payload, dict): + payload = {} + for field in _SAFE_EVENT_FIELDS[1:]: + value = _safe_identifier(payload.get(field)) if value is not None and value not in sensitive_values: safe[field] = value - if "type" not in safe or "subtype" not in safe: - return '{"event":"unparseable"}' return json.dumps(safe, sort_keys=True, separators=(",", ":"), ensure_ascii=True) class AgyEventParser: - """Strict stream-json parser bound to exactly one requested IOP cell.""" + """Parse the official 1.1.12 stream while trusting binding only from config.""" - def __init__(self, cell: MatrixCell) -> None: + def __init__(self, cell: MatrixCell, admitted_binding: RequestedEffectiveBinding) -> None: + if not isinstance(admitted_binding, RequestedEffectiveBinding): + raise AgyAdapterError("agy admitted binding is invalid") + if admitted_binding.cell_id != cell.id or admitted_binding.caller != cell.caller: + raise AgyAdapterError("agy admitted binding mismatch") self._cell = cell - self._observed_binding: RequestedEffectiveBinding | None = None - self._binding_invalid = False + self._admitted_binding = admitted_binding + self._init_seen = False + self._result_seen = False + self._latest_usage: dict[str, Any] | None = None - def __call__(self, stream: str, raw_line: str) -> str | ParsedMetric | None: + def __call__(self, stream: str, raw_line: str) -> Any: return self.parse(stream, raw_line) - def parse(self, stream: str, raw_line: str) -> str | ParsedMetric | None: + def parse(self, stream: str, raw_line: str) -> Any: if stream != "stdout": return None try: - event = json.loads(raw_line) + item = json.loads(raw_line) except (TypeError, json.JSONDecodeError): - return "malformed" - if not isinstance(event, dict): - return "malformed" - event_type = event.get("type") - subtype = event.get("subtype") - if (event_type, subtype) == ("iop", "effective_binding"): - self._observe_effective_binding(event) + raise AgyAdapterError("malformed agy event") + if not isinstance(item, dict): + raise AgyAdapterError("malformed agy event") + event = item.get("event") + payload = item.get(event) if isinstance(event, str) else None + if not isinstance(payload, dict): + raise AgyAdapterError("malformed agy event") + if event == "init": + if self._init_seen or self._result_seen: + raise AgyAdapterError("malformed agy event") + self._init_seen = True return None - if event_type == "metric": - return self._observe_duration(event, subtype) - if event_type == "result" and subtype == "error": - # Quota/provider errors can never be interpreted as finish/idle. - return "quota_error" if event.get("reason") == "quota" else "malformed" - terminal = ( - "finish" if (event_type, subtype) == ("result", "success") - else "idle" if (event_type, subtype) == ("system", "idle") - else None - ) - if terminal is None or not self._matches_exact_binding(event): - return "malformed" - return terminal - - def _observe_duration(self, event: dict[str, Any], subtype: Any) -> str | ParsedMetric: - """Convert one allowlisted agy duration losslessly, or fail closed.""" - mapped = AGY_DURATION_METRICS.get(subtype) if isinstance(subtype, str) else None - value = event.get("value") - if ( - mapped is None - or not set(event) <= _AGY_METRIC_KEYS - or not is_reported_number(value) - or ("model" in event and event["model"] != self._cell.iop.request_model) - ): - return "malformed" - name, overlap = mapped - try: - return duration_metric( - name, value, reported_unit="ms", - model=self._cell.iop.request_model, overlap=overlap, + if event == "step_update": + if not self._init_seen or self._result_seen: + raise AgyAdapterError("malformed agy event") + usage = payload.get("usage") + if usage is not None: + if self._usage_metrics(usage) is None: + raise AgyAdapterError("malformed agy event") + self._latest_usage = usage + return None + if event != "result" or not self._init_seen or self._result_seen: + raise AgyAdapterError("malformed agy event") + self._result_seen = True + if payload.get("status") != "SUCCESS": + if payload.get("status") != "ERROR": + raise AgyAdapterError("malformed agy result status") + return ( + CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR), + CallerEvent("finish"), + CallerEvent("idle"), ) + metrics: list[ParsedMetric] = [] + usage = payload.get("usage", self._latest_usage) + if usage is not None: + parsed_usage = self._usage_metrics(usage) + if parsed_usage is None: + raise AgyAdapterError("malformed agy usage") + metrics.extend(parsed_usage) + try: + if "duration_seconds" in payload: + metrics.append(duration_metric( + "total_duration", payload["duration_seconds"], reported_unit="s", + model=self._cell.iop.request_model, + )) + if "num_turns" in payload: + metrics.append(count_metric( + "model_calls", payload["num_turns"], model=self._cell.iop.request_model, + )) except LifecycleMetricError: - return "malformed" - - def _matches_exact_binding(self, event: dict[str, Any]) -> bool: - expected = self._cell.iop - return ( - event.get("model") == expected.request_model - and event.get("effort") == expected.requested_effort - and event.get("route_kind") == expected.route_kind - and event.get("route_id") == expected.route_id + raise AgyAdapterError("malformed agy metric") + return tuple(metrics) + ( + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + CallerEvent("finish"), + CallerEvent("idle"), ) - def _observe_effective_binding(self, event: dict[str, Any]) -> None: - expected_keys = {"type", "subtype", "route_kind", "route_id", "model", "effort", "stages"} - if set(event) != expected_keys or self._observed_binding is not None: - self._binding_invalid = True - return - values = tuple(event[key] for key in ("route_kind", "route_id", "model", "effort")) - stages = event.get("stages") - if not all(isinstance(value, str) and TOKEN_RE.fullmatch(value) for value in values) or not isinstance(stages, list): - self._binding_invalid = True - return - parsed_stages: list[EffectiveBinding] = [] - for stage in stages: - if not isinstance(stage, dict) or set(stage) != {"stage", "model", "effort"}: - self._binding_invalid = True - return - if not isinstance(stage["stage"], str) or not isinstance(stage["model"], str): - self._binding_invalid = True - return - if stage["effort"] is not None and not isinstance(stage["effort"], str): - self._binding_invalid = True - return - parsed_stages.append(EffectiveBinding(stage["stage"], stage["model"], stage["effort"])) - self._observed_binding = RequestedEffectiveBinding( - self._cell.id, self._cell.caller, - self._cell.iop.route_kind, self._cell.iop.route_id, - self._cell.iop.request_model, self._cell.iop.requested_effort, - values[0], values[1], values[2], values[3], tuple(parsed_stages), - ) + def _usage_metrics(self, usage: Any) -> tuple[ParsedMetric, ...] | None: + if not isinstance(usage, dict) or not set(usage) <= set(_AGY_USAGE_METRICS): + return None + metrics: list[ParsedMetric] = [] + try: + for wire_name, metric_name in _AGY_USAGE_METRICS.items(): + if wire_name in usage: + metrics.append(count_metric( + metric_name, usage[wire_name], model=self._cell.iop.request_model, + )) + except LifecycleMetricError: + return None + return tuple(metrics) def observed_result(self, capability: AgyCapability, lifecycle: InvocationResult) -> ConnectivityResult: - """Report ready only for successful lifecycle-owned explicit evidence.""" requested = _requested_binding(self._cell) closed_gap = (_issue("stream_incompatible"),) caller_capability = CallerCapability(AGY_CALLER, capability.route_kinds, capability.efforts) if ( not isinstance(lifecycle, InvocationResult) - or not lifecycle.success - or not lifecycle.finish_then_idle_then_quiet - or self._binding_invalid - or self._observed_binding is None + or lifecycle.product.status != CALLER_STATUS_SUCCEEDED + or lifecycle.harness.status != "passed" + or not lifecycle.harness.ordered_terminal + or not self._result_seen ): return make_result(self._cell, caller_capability, requested, closed_gap) try: - return make_result(self._cell, caller_capability, self._observed_binding) + return make_result(self._cell, caller_capability, self._admitted_binding) except Exception: return make_result(self._cell, caller_capability, requested, closed_gap) @@ -511,5 +563,5 @@ def run_agy_invocation( spec, parse_event=parser, on_started=on_started, - redact=lambda line: line if line in AGY_SAFE_METRIC_LABELS else structural(exact(line)), + redact=lambda line: structural(exact(line)), ) diff --git a/scripts/agent_benchmark/agy_iop_test.py b/scripts/agent_benchmark/agy_iop_test.py index 55a2c0ba..a5920ab8 100644 --- a/scripts/agent_benchmark/agy_iop_test.py +++ b/scripts/agent_benchmark/agy_iop_test.py @@ -1,4 +1,4 @@ -"""Credential-free tests for the fail-closed agy IOP adapter.""" +"""Credential-free tests for the official agy 1.1.12 IOP adapter.""" from __future__ import annotations @@ -9,25 +9,29 @@ import tempfile import unittest from dataclasses import replace from pathlib import Path +from unittest.mock import patch from scripts.agent_benchmark.agy_iop import ( AGY_AUTH_ENV, AGY_ENDPOINT_ENV, AGY_KNOWN_VERSION, - AGY_PROVIDER_ENV, + AGY_SETTINGS_RELATIVE_PATH, AgyAdapterError, AgyEventParser, AgyRuntimeInputs, AgyRuntimeObservation, + _runtime_identity, build_agy_invocation, inspect_agy_iop_capability, preflight_agy_iop, redact_agy_event, run_agy_invocation, ) +from scripts.agent_benchmark.connectivity import EffectiveBinding, RequestedEffectiveBinding from scripts.agent_benchmark.lifecycle import ( - REASON_DUPLICATE_EVENT, REASON_MALFORMED_EVENT, + REASON_NONZERO_EXIT, + SUBMISSION_ARGV_TASK, InvocationSpec, env_pairs, ) @@ -35,29 +39,24 @@ from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCel from scripts.agent_benchmark.workspace import AttemptIdentity, PreparedWorkspace, TestbedProvenance -def _help(*, transport: bool = True) -> str: - basic = "--print --output-format stream-json --sandbox --model --effort" - return basic + (f" {AGY_PROVIDER_ENV} {AGY_ENDPOINT_ENV} {AGY_AUTH_ENV}" if transport else "") +def _help() -> str: + return "--print --output-format stream-json --sandbox --model --effort" def _cell() -> MatrixCell: return MatrixCell( "agy-direct", "agy", - IopCell("gemini-2.0-flash", "high", "direct", "agy-direct", ( - ExpectedBinding("request", "gemini-2.0-flash", "high"), + IopCell("gemini-3.6-flash", "high", "direct", "agy-direct", ( + ExpectedBinding("request", "gemini-3.6-flash", "high"), )), ) -def _iop_config_observation() -> AgyRuntimeObservation: - """Fixed evidence from the independent IOP config owner for this cell.""" - return AgyRuntimeObservation( - "agy-direct", - "direct", - "agy-direct", - "sha256:feb4c33d4e775c775bfb3c333fdb7d4f97069af31c8e824094fb13181fad53d3", - "sha256:ab1b96f33fc4a662c870f349d92c54bc8e2574028fa41b79526d4edaf6f49daa", - "sha256:" + "c" * 64, +def _binding() -> RequestedEffectiveBinding: + return RequestedEffectiveBinding( + "agy-direct", "agy", "direct", "agy-direct", "gemini-3.6-flash", "high", + "direct", "agy-direct", "gemini-3.6-flash", "high", + (EffectiveBinding("request", "gemini-3.6-flash", "high"),), ) @@ -67,316 +66,231 @@ class AgyIopTest(unittest.TestCase): self.root = Path(self.temp.name) self.workspace = self.root / "workspace" self.workspace.mkdir() - self.runtime = AgyRuntimeInputs(sys.executable, "https://private.invalid/v1", "iop_secret_123456789") + self.session = self.root / "session" + self.session.mkdir() + self.runtime = AgyRuntimeInputs( + sys.executable, "https://private.invalid/gemini/agy-direct", "iop_secret_123456789" + ) def tearDown(self) -> None: self.temp.cleanup() + def _observation(self, runtime: AgyRuntimeInputs | None = None) -> AgyRuntimeObservation: + value = runtime or self.runtime + return AgyRuntimeObservation( + "agy-direct", "direct", "agy-direct", + _runtime_identity("endpoint", value.endpoint), + _runtime_identity("credential", value.credential), + "sha256:" + "c" * 64, + ) + def _prepared(self) -> PreparedWorkspace: return PreparedWorkspace( AttemptIdentity("run", "agy-direct", 1, 1), str(self.root), str(self.workspace), - str(self.root / "session"), "fresh-session", True, "sha256:" + "0" * 64, + str(self.session), "fresh-session", True, "sha256:" + "0" * 64, "isolated", TestbedProvenance("/testbed", "main", "0" * 40, "sha256:" + "1" * 64, True), "2026-01-01T00:00:00+00:00", ) - def _preflight(self, *, runtime: AgyRuntimeInputs | None = None, help_text: str | None = None): - values = self.runtime if runtime is None else runtime + def _preflight(self, runtime: AgyRuntimeInputs | None = None): + value = runtime or self.runtime return preflight_agy_iop( - _cell(), - inspect_agy_iop_capability("agy 1.1.11", _help() if help_text is None else help_text), - values, - _iop_config_observation(), + _cell(), inspect_agy_iop_capability("agy 1.1.12", _help()), value, + self._observation(value), ) - def _run_lines(self, lines: list[str], parser: AgyEventParser, preflight): + def _run_lines(self, lines: list[str], parser: AgyEventParser, *, exit_code: int = 0): evidence = self.root / f"evidence-{len(list(self.root.glob('evidence-*')))}" evidence.mkdir() - source = "import sys; lines=" + repr(lines) + "; [print(line) for line in lines]" + source = ( + "import sys; lines=" + repr(lines) + + "; [print(line) for line in lines]; raise SystemExit(" + repr(exit_code) + ")" + ) spec = InvocationSpec( argv=(sys.executable, "-u", "-c", source), cwd=str(self.root), env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}), submission_mode="stdin_once", completion_mode="exit_after_idle", timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task", ) - return run_agy_invocation(spec, parser, preflight, lambda _: None) + return run_agy_invocation(spec, parser, self._preflight(), lambda _: None) - def test_absent_or_unknown_transport_never_constructs_launch(self) -> None: - for version, help_text, expected in ( - ("agy 1.1.11", _help(transport=False), ("endpoint_incompatible", "auth_incompatible", "protocol_incompatible")), - ("agy 9.9.9", _help(), "protocol_incompatible"), - ): - with self.subTest(version=version): - preflight = preflight_agy_iop( - _cell(), inspect_agy_iop_capability(version, help_text), self.runtime, - _iop_config_observation(), - ) - self.assertEqual(preflight.status, "implementation_gap") - expected_codes = (expected,) if isinstance(expected, str) else expected - self.assertEqual([item.code for item in preflight.issues], list(expected_codes)) - with self.assertRaises(AgyAdapterError): - build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight) - - def test_installed_public_surface_is_exact_fail_closed_gap(self) -> None: - public_help = "--print --output-format stream-json --sandbox --model --effort" - for transport_name in (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV): - self.assertNotIn(transport_name, public_help) - - capability = inspect_agy_iop_capability("1.1.11", public_help) + def test_official_public_surface_is_pinned_without_invented_environment(self) -> None: + capability = inspect_agy_iop_capability("1.1.12", _help()) self.assertEqual(capability.version, AGY_KNOWN_VERSION) - self.assertTrue(capability.stream_supported) - self.assertFalse(capability.endpoint_supported) - self.assertFalse(capability.auth_supported) - self.assertFalse(capability.protocol_supported) - self.assertFalse(capability.iop_transport_supported) + self.assertTrue(capability.iop_transport_supported) + self.assertTrue(capability.endpoint_supported) + self.assertTrue(capability.auth_supported) + self.assertFalse(inspect_agy_iop_capability("1.1.11", _help()).iop_transport_supported) + self.assertFalse(inspect_agy_iop_capability("1.1.12", _help().replace("stream-json", "json")).stream_supported) - preflight = preflight_agy_iop( - _cell(), capability, self.runtime, _iop_config_observation() - ) - self.assertEqual(preflight.status, "implementation_gap") - self.assertEqual( - [item.code for item in preflight.issues], - ["endpoint_incompatible", "auth_incompatible", "protocol_incompatible"], - ) - with self.assertRaises(AgyAdapterError): - build_agy_invocation( - _cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight - ) - - def test_non_ready_preflight_cannot_start_supplied_invocation(self) -> None: - preflight = self._preflight(help_text=_help(transport=False)) - self.assertEqual(preflight.status, "implementation_gap") - self.assertIsNotNone(preflight.runtime) - marker = self.root / "caller-launched" - evidence = self.root / "blocked-evidence" - source = "from pathlib import Path; Path(" + repr(str(marker)) + ").write_text('launched')" - spec = InvocationSpec( - argv=(sys.executable, "-u", "-c", source), cwd=str(self.root), - env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}), - submission_mode="stdin_once", completion_mode="exit_after_idle", - timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task", - ) - started: list[object] = [] - - with self.assertRaisesRegex(AgyAdapterError, "agy IOP transport is not proven"): - run_agy_invocation(spec, AgyEventParser(_cell()), preflight, started.append) - - self.assertFalse(marker.exists()) - self.assertEqual(started, []) - self.assertFalse(evidence.exists()) - - def test_registration_gaps_remain_distinct_from_implementation_gap(self) -> None: - no_credential = AgyRuntimeInputs(sys.executable, self.runtime.endpoint, "") - supported = inspect_agy_iop_capability("agy 1.1.11", _help()) - result = preflight_agy_iop( - _cell(), supported, no_credential, _iop_config_observation() - ) - self.assertEqual(result.status, "registration_required") - self.assertEqual([item.code for item in result.issues], ["credential_missing"]) - gap = preflight_agy_iop( - _cell(), inspect_agy_iop_capability("agy 1.1.11", _help(transport=False)), no_credential, - _iop_config_observation(), - ) - self.assertEqual(gap.status, "implementation_gap") - self.assertEqual([item.code for item in gap.issues], ["credential_missing", "endpoint_incompatible", "auth_incompatible", "protocol_incompatible"]) - - def test_endpoint_auth_and_protocol_gaps_are_exact(self) -> None: - cases = ( - (_help().replace(AGY_ENDPOINT_ENV, ""), "endpoint_incompatible"), - (_help().replace(AGY_AUTH_ENV, ""), "auth_incompatible"), - (_help().replace("--sandbox", ""), "protocol_incompatible"), - ) - for help_text, expected in cases: - with self.subTest(expected=expected): - outcome = self._preflight(help_text=help_text) - self.assertEqual([item.code for item in outcome.issues], [expected]) - unknown = inspect_agy_iop_capability(None, None) # type: ignore[arg-type] - self.assertFalse(unknown.iop_transport_supported) - - def test_build_is_fresh_stdin_sandbox_and_iop_only(self) -> None: - preflight = self._preflight() - spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), preflight) - self.assertEqual(spec.submission_mode, "stdin_once") - self.assertIn("--print", spec.argv) - self.assertIn("--sandbox", spec.argv) - self.assertNotIn("--resume", spec.argv) + def test_build_uses_official_gemini_api_key_transport(self) -> None: + with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}): + spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), self._preflight()) environment = dict(spec.env) - self.assertEqual(environment[AGY_PROVIDER_ENV], "iop-openai") self.assertEqual(environment[AGY_ENDPOINT_ENV], self.runtime.endpoint) self.assertEqual(environment[AGY_AUTH_ENV], self.runtime.credential) + self.assertEqual(environment["HOME"], str(self.session)) + self.assertNotIn("AGY_PROVIDER", environment) + self.assertNotIn("AGY_OPENAI_BASE_URL", environment) + self.assertNotIn("AGY_OPENAI_API_KEY", environment) + self.assertNotIn("--effort", spec.argv) + self.assertEqual(spec.argv[spec.argv.index("--model") + 1], "Gemini 3.6 Flash") + self.assertEqual(spec.argv[-2:], ("--print", "one task")) + self.assertEqual(spec.submission_mode, SUBMISSION_ARGV_TASK) + self.assertEqual(spec.task_payload, b"") + self.assertEqual(environment["SSL_CERT_FILE"], "/operator/dev-ca.pem") + self.assertEqual(environment["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem") + self.assertEqual(spec.env_allowlist, (AGY_ENDPOINT_ENV, AGY_AUTH_ENV, "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS")) + settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH + self.assertEqual(json.loads(settings_path.read_text(encoding="utf-8")), { + "enableTelemetry": False, + "modelProvider": "gemini", + "toolPermission": "always-proceed", + }) + self.assertEqual(settings_path.stat().st_mode & 0o777, 0o600) + settings_text = settings_path.read_text(encoding="utf-8") + self.assertNotIn(self.runtime.endpoint, settings_text) + self.assertNotIn(self.runtime.credential, settings_text) - def test_exact_help_tokens_and_stream_format_gate(self) -> None: - lookalike = _help().replace("--print", "--print-json").replace( - AGY_ENDPOINT_ENV, AGY_ENDPOINT_ENV + "_EXTRA" - ).replace("stream-json", "stream-jsonl") - capability = inspect_agy_iop_capability("agy 1.1.11", lookalike) - self.assertFalse(capability.iop_transport_supported) - self.assertFalse(capability.endpoint_supported) - self.assertFalse(capability.stream_supported) - missing_stream = self._preflight(help_text=_help().replace("stream-json", "")) - self.assertEqual([issue.code for issue in missing_stream.issues], ["stream_incompatible"]) + def test_build_rejects_preexisting_isolated_settings(self) -> None: + settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH + settings_path.parent.mkdir(parents=True) + settings_path.write_text('{"modelProvider":"other"}\n', encoding="utf-8") + with self.assertRaisesRegex(AgyAdapterError, "isolated provider settings"): + build_agy_invocation( + _cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), self._preflight() + ) - def test_unvalidated_runtime_cannot_launch(self) -> None: - observation = _iop_config_observation() - for mismatched in ( - replace(observation, cell_id="other-cell"), - replace(observation, route_id="other-route"), - replace(observation, endpoint_identity="sha256:" + "d" * 64), - replace(observation, config_identity="not-a-config-identity"), + def test_build_rejects_missing_prepared_workspace_or_session(self) -> None: + prepared = self._prepared() + for field in ("workspace_dir", "session_dir", "attempt_root"): + missing = prepared.__class__(**{ + **prepared.__dict__, field: str(self.root / f"missing-{field}"), + }) + with self.subTest(field=field): + with self.assertRaisesRegex(AgyAdapterError, "prepared workspace"): + build_agy_invocation( + _cell(), missing, b"task", Timeout(5, 1, 1, 1), self._preflight() + ) + + def test_build_rejects_non_utf8_task_for_print_argument(self) -> None: + with self.assertRaisesRegex(AgyAdapterError, "must be UTF-8"): + build_agy_invocation( + _cell(), self._prepared(), b"\xff", Timeout(5, 1, 1, 1), self._preflight() + ) + + def test_route_qualified_https_runtime_is_required(self) -> None: + for endpoint in ( + "http://private.invalid/gemini/agy-direct", + "https://private.invalid/v1", + "https://private.invalid/gemini/other", ): - with self.subTest(observation=mismatched): - preflight = preflight_agy_iop( - _cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), self.runtime, mismatched - ) - self.assertEqual(preflight.status, "implementation_gap") - self.assertIsNone(preflight.runtime) - with self.assertRaises(AgyAdapterError): - build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight) - - def test_arbitrary_runtime_cannot_self_issue_iop_proof(self) -> None: - arbitrary = AgyRuntimeInputs(sys.executable, "https://api.openai.com/v1", "unrelated_token_123456789") - preflight = preflight_agy_iop( - _cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), arbitrary, - _iop_config_observation(), + runtime = replace(self.runtime, endpoint=endpoint) + result = self._preflight(runtime) + self.assertEqual(result.status, "implementation_gap") + self.assertEqual([issue.code for issue in result.issues], ["endpoint_incompatible"]) + missing = replace(self.runtime, credential="") + result = preflight_agy_iop( + _cell(), inspect_agy_iop_capability("1.1.12", _help()), missing, + self._observation(self.runtime), ) - self.assertEqual(preflight.status, "implementation_gap") - self.assertIsNone(preflight.runtime) + self.assertEqual(result.status, "registration_required") + self.assertEqual([issue.code for issue in result.issues], ["credential_missing"]) + + def test_unknown_model_and_unvalidated_observation_fail_closed(self) -> None: + unsupported = replace(_cell(), iop=replace(_cell().iop, request_model="gemini-unknown")) + result = preflight_agy_iop( + unsupported, inspect_agy_iop_capability("1.1.12", _help()), self.runtime, + self._observation(), + ) + self.assertIn("model_missing", [issue.code for issue in result.issues]) + mismatched = replace(self._observation(), endpoint_identity="sha256:" + "d" * 64) + result = preflight_agy_iop( + _cell(), inspect_agy_iop_capability("1.1.12", _help()), self.runtime, mismatched, + ) + self.assertIsNone(result.runtime) with self.assertRaises(AgyAdapterError): - build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight) + build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), result) - def test_lifecycle_fixture_success_and_metric_preservation(self) -> None: - parser = AgyEventParser(_cell()) + def test_official_fixture_completes_and_preserves_metrics(self) -> None: + parser = AgyEventParser(_cell(), _binding()) fixture = Path("scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl") - result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser, self._preflight()) - self.assertTrue(result.success) - self.assertTrue(result.finish_then_idle_then_quiet) - journal = Path(result.journal_path).read_text(encoding="utf-8") - self.assertIn('"kind": "metric:total_duration"', journal) - self.assertIn('"kind": "metric:model_duration"', journal) - observed = {metric.name: metric for metric in result.metrics} - self.assertEqual(set(observed), {"total_duration", "model_duration"}) - self.assertEqual(observed["total_duration"].value, 12 * 10 ** 6) - self.assertFalse(observed["total_duration"].overlap) - # 8.5 ms is preserved exactly; agy's model stage overlaps its total. - self.assertEqual(observed["model_duration"].value, 8_500_000) - self.assertTrue(observed["model_duration"].overlap) - for metric in result.metrics: - self.assertEqual(metric.unit, "ns") - self.assertEqual(metric.clock, "caller_reported") - self.assertEqual(metric.source, "caller_output") - self.assertEqual(metric.model, "gemini-2.0-flash") - capability = inspect_agy_iop_capability("agy 1.1.11", _help()) - self.assertEqual(parser.observed_result(capability, result).status, "ready") - - def test_only_allowlisted_bound_representable_durations_are_observed(self) -> None: - parser = AgyEventParser(_cell()) - rejected = ( - {"type": "metric", "subtype": "unknown_ms", "value": 5}, - {"type": "metric", "subtype": "duration_ms", "value": "5"}, - {"type": "metric", "subtype": "duration_ms", "value": True}, - {"type": "metric", "subtype": "duration_ms", "value": -1}, - # 0.0000001 ms is 0.1 ns and cannot be represented without invention. - {"type": "metric", "subtype": "duration_ms", "value": 0.0000001}, - {"type": "metric", "subtype": "duration_ms", "value": 5, "model": "other"}, - {"type": "metric", "subtype": "duration_ms", "value": 5, "extra": 1}, + result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser) + self.assertTrue(result.product.status == "succeeded") + self.assertTrue(result.harness.ordered_terminal) + metrics = {metric.name: metric for metric in result.metrics} + self.assertEqual( + set(metrics), + {"total_duration", "model_calls", "input_tokens", "cached_input_tokens", "output_tokens", "reasoning_tokens", "total_tokens"}, ) - for event in rejected: - with self.subTest(event=event): - self.assertEqual(parser("stdout", json.dumps(event)), "malformed") - bound = parser("stdout", json.dumps({ - "type": "metric", "subtype": "queue_duration_ms", "value": 2, - "model": "gemini-2.0-flash", - })) - self.assertEqual((bound.name, bound.value, bound.overlap), ("queue_duration", 2_000_000, False)) + self.assertEqual(metrics["total_duration"].value, 12_000_000) + self.assertEqual(metrics["model_calls"].value, 1) + self.assertEqual(metrics["total_tokens"].value, 12) + capability = inspect_agy_iop_capability("1.1.12", _help()) + self.assertEqual(parser.observed_result(capability, result).binding, _binding()) - def test_unrepresentable_duration_fails_the_run_without_partial_metric(self) -> None: - parser = AgyEventParser(_cell()) - result = self._run_lines( - [json.dumps({"type": "metric", "subtype": "duration_ms", "value": 0.0000001})], - parser, self._preflight(), - ) - self.assertFalse(result.success) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) - self.assertEqual(result.metrics, ()) + def test_result_must_follow_init_be_unique_and_successful(self) -> None: + result = { + "event": "result", "result": { + "status": "SUCCESS", "duration_seconds": 0.1, "num_turns": 1, + "usage": {"input_tokens": 1}, "response": "secret content", + }, + } + for lines, reason in ( + ([json.dumps(result)], REASON_MALFORMED_EVENT), + ([json.dumps({"event": "init", "init": {}}), json.dumps(result), json.dumps(result)], REASON_MALFORMED_EVENT), + ): + parser = AgyEventParser(_cell(), _binding()) + invocation = self._run_lines(lines, parser) + self.assertFalse(invocation.product.status == "succeeded") + self.assertEqual(invocation.harness.reason, "parser_error") - def test_metric_prefix_cannot_bypass_durable_redaction(self) -> None: - parser = AgyEventParser(_cell()) - raw_lines = [f"metric:{self.runtime.endpoint}", f"metric:{self.runtime.credential}", "metric:not-json"] - result = self._run_lines(raw_lines, parser, self._preflight()) - self.assertFalse(result.success) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) - persisted = Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8") - for forbidden in (*raw_lines, self.runtime.endpoint, self.runtime.credential): - self.assertNotIn(forbidden, persisted) + parser = AgyEventParser(_cell(), _binding()) + error_result = json.dumps({ + "event": "result", "result": {"status": "ERROR", "response": "private"}, + }) + invocation = self._run_lines([ + json.dumps({"event": "init", "init": {}}), error_result, + ], parser, exit_code=1) + self.assertFalse(invocation.product.status == "succeeded") + self.assertEqual(invocation.product.status, "failed") + self.assertEqual(invocation.harness.status, "passed") + self.assertEqual(invocation.process.exit_code, 1) - def test_mismatch_duplicate_and_quota_cannot_pass(self) -> None: - event = {"type": "result", "subtype": "success", "model": "other", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"} - self.assertEqual(AgyEventParser(_cell())("stdout", json.dumps(event)), "malformed") - parser = AgyEventParser(_cell()) - finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"} - self.assertEqual(parser("stdout", json.dumps(finish)), "finish") - self.assertEqual(parser("stdout", json.dumps(finish)), "finish") - self.assertEqual(AgyEventParser(_cell())("stdout", '{"type":"result","subtype":"error","reason":"quota"}'), "quota_error") + def test_latest_step_usage_is_used_only_when_result_omits_usage(self) -> None: + parser = AgyEventParser(_cell(), _binding()) + lines = [ + json.dumps({"event": "init", "conversation_id": "c", "init": {"model": "Gemini 3.6 Flash"}}), + json.dumps({"event": "step_update", "step_update": {"state": "DONE", "step_index": 0, "step_type": "agent_response", "text_delta": "private", "usage": {"input_tokens": 2, "output_tokens": 1, "total_tokens": 3}}}), + json.dumps({"event": "result", "result": {"status": "SUCCESS", "duration_seconds": 0.2, "num_turns": 1, "response": "private"}}), + ] + invocation = self._run_lines(lines, parser) + self.assertTrue(invocation.product.status == "succeeded") + self.assertEqual({m.name: m.value for m in invocation.metrics}["total_tokens"], 3) - evidence = self.root / "duplicate-evidence" - evidence.mkdir() - source = "import json; event=" + repr(finish) + "; print(json.dumps(event)); print(json.dumps(event))" - spec = InvocationSpec( - argv=(sys.executable, "-u", "-c", source), cwd=str(self.root), - env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}), - submission_mode="stdin_once", completion_mode="exit_after_idle", - timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task", - ) - duplicate = run_agy_invocation(spec, AgyEventParser(_cell()), self._preflight(), lambda _: None) - self.assertFalse(duplicate.success) - self.assertEqual(duplicate.terminal_reason, REASON_DUPLICATE_EVENT) + def test_malformed_usage_fails_without_partial_metric(self) -> None: + parser = AgyEventParser(_cell(), _binding()) + invocation = self._run_lines([ + '{"event":"init","init":{}}', + '{"event":"result","result":{"status":"SUCCESS","usage":{"input_tokens":"1"}}}', + ], parser) + self.assertFalse(invocation.product.status == "succeeded") + self.assertEqual(invocation.harness.reason, "parser_error") + self.assertEqual(invocation.metrics, ()) - def test_structural_redaction_excludes_content_tools_endpoints_and_secrets(self) -> None: - raw = json.dumps({"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "content": "raw prompt", "tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint, "token": self.runtime.credential}) + def test_structural_redaction_excludes_response_tools_endpoint_and_secret(self) -> None: + raw = json.dumps({ + "event": "result", "result": { + "status": "SUCCESS", "response": "raw prompt", + "tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint, + "token": self.runtime.credential, + }, + }) redacted = redact_agy_event(raw, (self.runtime.endpoint, self.runtime.credential)) - self.assertEqual(redacted, '{"model":"gemini-2.0-flash","subtype":"success","type":"result"}') + self.assertEqual(redacted, '{"event":"result","status":"SUCCESS"}') for forbidden in ("raw prompt", "tool_input", self.runtime.endpoint, self.runtime.credential): self.assertNotIn(forbidden, redacted) - def test_lifecycle_rejects_quota_without_durable_leak(self) -> None: - evidence = self.root / "evidence" - evidence.mkdir() - parser = AgyEventParser(_cell()) - secret = self.runtime.credential - endpoint = self.runtime.endpoint - source = "import json; print(json.dumps(" + repr({ - "type": "result", "subtype": "error", "reason": "quota", - "content": secret, "endpoint": endpoint, - }) + "))" - spec = InvocationSpec( - argv=(sys.executable, "-u", "-c", source), cwd=str(self.root), - env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}), - submission_mode="stdin_once", completion_mode="exit_after_idle", - timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task", - ) - result = run_agy_invocation(spec, parser, self._preflight(), lambda _: None) - self.assertFalse(result.success) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) - persisted = (Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8")) - self.assertNotIn(secret, persisted) - self.assertNotIn(endpoint, persisted) - - def test_ready_requires_observed_stage_binding_and_successful_lifecycle(self) -> None: - capability = inspect_agy_iop_capability("agy 1.1.11", _help()) - finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"} - idle = {"type": "system", "subtype": "idle", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"} - binding = {"type": "iop", "subtype": "effective_binding", "route_kind": "direct", "route_id": "agy-direct", "model": "gemini-2.0-flash", "effort": "high", "stages": [{"stage": "request", "model": "gemini-2.0-flash", "effort": "high"}]} - for lines in ( - [json.dumps(finish), json.dumps(idle)], - [json.dumps(binding), json.dumps(idle), json.dumps(finish)], - [json.dumps(binding), json.dumps(finish), json.dumps(finish), json.dumps(idle)], - [json.dumps({**binding, "route_id": "other"}), json.dumps(finish), json.dumps(idle)], - ): - with self.subTest(lines=lines): - parser = AgyEventParser(_cell()) - result = self._run_lines(lines, parser, self._preflight()) - self.assertEqual(parser.observed_result(capability, result).status, "implementation_gap") - if __name__ == "__main__": unittest.main() diff --git a/scripts/agent_benchmark/attempts.py b/scripts/agent_benchmark/attempts.py index 74792308..9f4a32ea 100644 --- a/scripts/agent_benchmark/attempts.py +++ b/scripts/agent_benchmark/attempts.py @@ -35,16 +35,25 @@ from scripts.agent_benchmark.connectivity import ( ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_SUCCESS, COMPLETION_MODES, + EVENT_CALLER_TERMINAL, EVENT_FINISH, EVENT_IDLE, EVENT_QUIET, EVENT_SUBMITTED, InvocationResult, + JOURNAL_VERSION, + HARNESS_REASONS, + HARNESS_STATUSES, + PROCESS_STATUSES, + PRODUCT_REASONS, + PRODUCT_STATUSES, LifecycleRecoveryError, RECEIPT_VERSION, REASON_CONTROLLER_LOST, REASON_RECOVERED_STOP, + SOURCE_CALLER_OUTPUT, SOCKET_FILENAME, SUBMISSION_MODES, SupervisorLocator, @@ -82,8 +91,9 @@ PREFLIGHT_RE = re.compile(r"^preflight-([0-9]{6})\.json$") DIGEST_RE = re.compile(r"^sha256:[0-9a-f]{64}$") PREFLIGHT_SCHEMA_VERSION = "1" PREFLIGHT_STATUSES = ("ready", "registration_required", "implementation_gap") -TERMINAL_STATES = frozenset(("success", "failed", "timed_out", "cancelled", "interrupted")) +TERMINAL_STATES = frozenset(("completed", "timed_out", "cancelled", "interrupted")) NONTERMINAL_STATE = "running" +ATTEMPT_RESULT_VERSION = 2 MEASUREMENT_POLICY_REQUIRED_V1 = "required-v1" MEASUREMENT_POLICIES = frozenset((MEASUREMENT_POLICY_REQUIRED_V1,)) MEASUREMENT_POLICY_FILENAME = "attempt-measurement-policy.json" @@ -95,6 +105,7 @@ WEB_VALIDATION_POLICY_FILENAME = "web-validation-policy.json" WEB_VALIDATION_POLICY_RECORD = "web-validation-policy" WEB_VALIDATION_POLICY_VERSION = 1 SUCCESS_EVIDENCE_KINDS = (EVENT_SUBMITTED, EVENT_FINISH, EVENT_IDLE, EVENT_QUIET) +BOUND_EVIDENCE_KINDS = SUCCESS_EVIDENCE_KINDS + (EVENT_CALLER_TERMINAL,) CONTROL_ALIAS_PREFIX = "iop-bench-attempt-" CONTROL_DIRECTORY_NAME = "control" CONTROL_ALIAS_DIGEST_HEX_LENGTH = 24 @@ -108,6 +119,65 @@ RECEIPT_ONLY_TERMINAL_REASONS = frozenset( ) +def _outcome_records(result: InvocationResult) -> dict[str, dict[str, Any]]: + return { + "product": { + "status": result.product.status, + "reason": result.product.reason, + }, + "harness": { + "status": result.harness.status, + "reason": result.harness.reason, + "ordered_terminal": result.harness.ordered_terminal, + "cleanup_complete": result.harness.cleanup_complete, + }, + "process": { + "status": result.process.status, + "exit_code": result.process.exit_code, + "signal": result.process.signal, + }, + } + + +def _unknown_terminal( + reason: str, *, process_status: str = "not_started", + exit_code: int | None = None, signal: int | None = None, +) -> dict[str, dict[str, Any]]: + return { + "product": {"status": "unknown", "reason": "unavailable"}, + "harness": { + "status": "failed", + "reason": reason, + "ordered_terminal": False, + "cleanup_complete": True, + }, + "process": { + "status": process_status, + "exit_code": exit_code, + "signal": signal, + }, + } + + +def _terminal_reason(terminal: Mapping[str, Any]) -> str: + harness = terminal.get("harness") + return str(harness.get("reason") if isinstance(harness, Mapping) else "") + + +def _terminal_passed(terminal: Mapping[str, Any]) -> bool: + process = terminal.get("process") + return ( + isinstance(terminal.get("product"), Mapping) + and terminal["product"].get("status") == "succeeded" + and isinstance(terminal.get("harness"), Mapping) + and terminal["harness"].get("status") == "passed" + and isinstance(process, Mapping) + and process.get("status") == "exited" + and process.get("exit_code") == 0 + and process.get("signal") is None + ) + + class AttemptError(Exception): """Base error whose message is safe to present to a benchmark caller.""" @@ -363,7 +433,7 @@ def _connectivity_result_from_payload( tuple(stages), ) # Capability is deliberately not serialized. The persisted proof is - # revalidated against the exact immutable cell and its one direct route. + # revalidated against the exact immutable cell and its declared route. capability = CallerCapability( cell.caller, (cell.iop.route_kind,), (cell.iop.requested_effort,) ) @@ -387,11 +457,7 @@ def collect_preflight_observations( manifest: Manifest, adapters: Mapping[str, PreflightAdapter], ) -> dict[str, PreflightObservation]: - """Validate the full registry, then probe direct cells in manifest order. - - Execution-preset cells exercise only the local adapter capability contract in - this milestone. They never become a synthetic live-ready observation. - """ + """Validate the full registry, then probe every cell in manifest order.""" if not isinstance(adapters, Mapping): raise CapabilityUnavailable("capability-unavailable: caller-adapter") required_callers = {cell.caller for cell in manifest.matrix} @@ -405,8 +471,6 @@ def collect_preflight_observations( observations: dict[str, PreflightObservation] = {} for cell in manifest.matrix: - if cell.iop.route_kind != "direct": - continue observation = adapters[cell.caller].preflight(cell) if not isinstance(observation, PreflightObservation): raise AttemptStateError("preflight observation is invalid") @@ -512,8 +576,8 @@ class RunStore: os.close(fd) @staticmethod - def _direct_cells(manifest: Manifest) -> tuple[MatrixCell, ...]: - return tuple(cell for cell in manifest.matrix if cell.iop.route_kind == "direct") + def _preflight_cells(manifest: Manifest) -> tuple[MatrixCell, ...]: + return tuple(manifest.matrix) def _preflight_root(self, run: RunIdentity, *, create: bool) -> Path: root = Path(run.root) / "preflight" @@ -560,11 +624,11 @@ class RunStore: or not isinstance(record["results"], list) ): raise AttemptStateError("preflight record identity is invalid") - direct_cells = self._direct_cells(manifest) - if len(record["results"]) != len(direct_cells): + preflight_cells = self._preflight_cells(manifest) + if len(record["results"]) != len(preflight_cells): raise AttemptStateError("preflight result set is invalid") statuses: list[str] = [] - for cell, result_payload in zip(direct_cells, record["results"]): + for cell, result_payload in zip(preflight_cells, record["results"]): result, _, _ = _connectivity_result_from_payload(result_payload, cell) statuses.append(result.status) if record["status"] != _overall_preflight_status(iter(statuses)): @@ -629,11 +693,11 @@ class RunStore: bound_run = self.open(manifest, run.run_id) if bound_run != run: raise AttemptStateError("run identity is invalid") - direct_cells = self._direct_cells(manifest) - if set(observations) != {cell.id for cell in direct_cells}: + preflight_cells = self._preflight_cells(manifest) + if set(observations) != {cell.id for cell in preflight_cells}: raise AttemptStateError("preflight observation set is invalid") results: list[dict[str, Any]] = [] - for cell in direct_cells: + for cell in preflight_cells: observation = observations[cell.id] if not isinstance(observation, PreflightObservation): raise AttemptStateError("preflight observation is invalid") @@ -705,6 +769,7 @@ class RunStore: @staticmethod def _expected_record(run: RunIdentity, identity: AttemptIdentity, state: str) -> dict[str, Any]: return { + "attempt_result_version": ATTEMPT_RESULT_VERSION, "run_id": run.run_id, "manifest_digest": run.manifest_digest, "cell_id": identity.cell_id, @@ -765,7 +830,7 @@ class RunStore: if record["state"] in TERMINAL_STATES: lifecycle = record.get("lifecycle") expected_receipt_reason = ( - lifecycle.get("terminal_reason") + _terminal_reason(lifecycle) if isinstance(lifecycle, dict) else None ) @@ -779,10 +844,21 @@ class RunStore: expected_receipt_reason=expected_receipt_reason, measurement_policy=policy, ) - if "lifecycle" in record and (not isinstance(record["lifecycle"], dict) or set(record["lifecycle"]) != {"terminal_reason"} or not isinstance(record["lifecycle"]["terminal_reason"], str)): - raise AttemptStateError("attempt lifecycle is invalid") + if "lifecycle" in record: + self._validate_outcomes(record["lifecycle"], "attempt lifecycle") if record["state"] in TERMINAL_STATES: - self._validate_web_validation(root, run, identity, web_policy) + pre_registration_interrupted = ( + record["state"] == "interrupted" + and "locator" not in record + and record.get("lifecycle") == _unknown_terminal("interrupted") + ) + self._validate_web_validation( + root, + run, + identity, + web_policy, + allow_pre_registration_absence=pre_registration_interrupted, + ) elif (root / WEB_VALIDATION_FILENAME).exists() or ( root / WEB_VALIDATION_FILENAME ).is_symlink(): @@ -823,6 +899,33 @@ class RunStore: for attempt in self.attempts(bound_run, slot) ) + def attempt_outcomes(self, attempt: Attempt) -> dict[str, Any]: + """Return the independently validated gates for one terminal attempt.""" + run, root = self._bound_attempt(attempt) + record = self._attempt_record(root, run, attempt.identity) + if record is None or record["state"] not in TERMINAL_STATES: + raise AttemptStateError("attempt outcomes require terminal state") + lifecycle = record.get("lifecycle") + if not isinstance(lifecycle, Mapping): + raise AttemptStateError("attempt lifecycle is unavailable") + try: + artifact = load_web_validation(root, manifest=self.open_manifest_snapshot(run)).status + except WebValidationError: + # The only permitted terminal without a web record is a controller + # interruption before caller registration. It remains unresolved. + artifact = "not_run" + return { + "product": lifecycle["product"]["status"], + "harness": lifecycle["harness"]["status"], + "process": lifecycle["process"]["status"], + "artifact": artifact, + "passed": ( + record["state"] == "completed" + and _terminal_passed(lifecycle) + and artifact == "passed" + ), + } + def allocate(self, run: RunIdentity, slot: Slot) -> Attempt: """Create the exclusive, deliberately empty attempt root.""" existing = self.attempts(run, slot) @@ -880,7 +983,7 @@ class RunStore: raise AttemptStateError("attempt web validation policy is invalid") record["web_validation_policy"] = web_validation_policy if reason is not None: - record["lifecycle"] = {"terminal_reason": reason} + record["lifecycle"] = _unknown_terminal(reason) return record @staticmethod @@ -989,9 +1092,12 @@ class RunStore: def _validate_web_validation( self, root: Path, run: RunIdentity, identity: AttemptIdentity, policy: str | None, + *, allow_pre_registration_absence: bool = False, ) -> None: path = root / WEB_VALIDATION_FILENAME if not path.exists() and not path.is_symlink(): + if allow_pre_registration_absence: + return if policy == WEB_VALIDATION_POLICY_REQUIRED_V1: raise AttemptStateError("required web validation is unavailable") return @@ -1014,11 +1120,8 @@ class RunStore: _read_regular_bytes(root / MEASUREMENT_FILENAME, "measurement")).hexdigest() if record["measurement_digest"] != digest or measurement.run_id != run.run_id: raise AttemptStateError("attempt web validation measurement is invalid") - if ( - (measurement.terminal_reason == "success" and web.status == "not_run") - or (measurement.terminal_reason != "success" and web.status != "not_run") - ): - raise AttemptStateError("attempt web validation lifecycle is invalid") + if web.status == "not_run": + raise AttemptStateError("terminal workspace web validation was not run") def open_manifest_snapshot(self, run: RunIdentity) -> Manifest: """Load the exact immutable run manifest used by recovery and evidence.""" @@ -1054,6 +1157,20 @@ class RunStore: occurs before ``attempt.json`` is replaced, leaving recovery resumable. """ policy = record.get("web_validation_policy") + if ( + policy == WEB_VALIDATION_POLICY_REQUIRED_V1 + and record.get("locator") is None + and _terminal_reason(terminal) == "interrupted" + ): + path = root / WEB_VALIDATION_FILENAME + measurement = root / MEASUREMENT_FILENAME + if not ( + path.exists() + or path.is_symlink() + or measurement.exists() + or measurement.is_symlink() + ): + return if policy != WEB_VALIDATION_POLICY_REQUIRED_V1: self._validate_web_validation(root, run, identity, policy) return @@ -1167,8 +1284,20 @@ class RunStore: run, root = self._bound_attempt(attempt) record = self._attempt_record(root, run, attempt.identity, absent_ok=True) if record is None: - reason = str((result or {}).get("terminal_reason") or state) - _write_new(root / "attempt.json", _json_bytes(self._initial_record(run, attempt, state, reason=reason))) + terminal_result = result or _unknown_terminal(state) + self._validate_outcomes(terminal_result, "attempt terminal") + if (root / WEB_VALIDATION_FILENAME).exists() or ( + root / WEB_VALIDATION_FILENAME + ).is_symlink(): + self._validate_web_validation( + root, run, attempt.identity, policy=None + ) + initial = self._expected_record(run, attempt.identity, state) + initial["lifecycle"] = { + name: dict(terminal_result[name]) + for name in ("product", "harness", "process") + } + _write_new(root / "attempt.json", _json_bytes(initial)) return Attempt(attempt.identity, attempt.root, state) if record["state"] in TERMINAL_STATES: if record["state"] != state: @@ -1176,7 +1305,8 @@ class RunStore: return Attempt(attempt.identity, attempt.root, state) if record["state"] != NONTERMINAL_STATE: raise AttemptStateError("attempt transition is invalid") - terminal_result = result or {"terminal_reason": state} + terminal_result = result or _unknown_terminal(state) + self._validate_outcomes(terminal_result, "attempt terminal") self._ensure_required_web_validation( root, run, @@ -1186,7 +1316,8 @@ class RunStore: ) record["state"] = state record["lifecycle"] = { - "terminal_reason": str(terminal_result.get("terminal_reason") or state) + name: dict(terminal_result[name]) + for name in ("product", "harness", "process") } _replace(root / "attempt.json", _json_bytes(record)) return Attempt(attempt.identity, attempt.root, state) @@ -1258,7 +1389,7 @@ class RunStore: ) if actual != expected or measurement.spec_digest != expected_digest: raise AttemptStateError("attempt measurement identity is invalid") - if expected_reason is not None and measurement.terminal_reason != expected_reason: + if expected_reason is not None and measurement.harness.reason != expected_reason: raise AttemptStateError("attempt measurement terminal is invalid") if lifecycle is not None: try: @@ -1344,15 +1475,26 @@ class RunStore: @staticmethod def _state_for_reason(reason: str) -> str: - if reason == "success": - return "success" if reason == "timed_out": return "timed_out" if reason == "cancelled": return "cancelled" if reason in RECEIPT_ONLY_TERMINAL_REASONS: return "interrupted" - return "failed" + return "completed" + + @staticmethod + def _state_for_terminal(terminal: Mapping[str, Any]) -> str: + process = terminal.get("process") + status = process.get("status") if isinstance(process, Mapping) else None + if status == "timed_out": + return "timed_out" + if status == "cancelled": + return "cancelled" + reason = _terminal_reason(terminal) + if reason in RECEIPT_ONLY_TERMINAL_REASONS or reason == "interrupted": + return "interrupted" + return "completed" def _read_json_file(self, root: Path, name: str) -> dict[str, Any]: path = root / name @@ -1376,19 +1518,18 @@ class RunStore: def _validate_result_record(self, result: dict[str, Any], locator: SupervisorLocator, expected_digest: str) -> None: fields = { - "record", "success", "terminal_reason", "exit_code", "signal", "submitted", - "finish_then_idle_then_quiet", "cleanup_complete", "process_group_alive", + "record", "product", "harness", "process", "submitted", + "process_group_alive", "submission_mode", "completion_mode", "spec_digest", "locator", "started_at", "ended_at", "duration_ns", "stdout", "stderr", "events", } self._exact_fields(result, fields, "lifecycle result") - required_bools = ("success", "submitted", "finish_then_idle_then_quiet", "cleanup_complete", "process_group_alive") + self._validate_outcomes(result, "lifecycle result") + required_bools = ("submitted", "process_group_alive") if result["record"] != "result" or any(not isinstance(result[key], bool) for key in required_bools): raise AttemptStateError("lifecycle result is invalid") - if result["terminal_reason"] not in TERMINAL_REASONS or result["spec_digest"] != expected_digest: + if result["spec_digest"] != expected_digest: raise AttemptStateError("lifecycle result identity is invalid") - if not self._optional_int(result["exit_code"]) or not self._optional_int(result["signal"]): - raise AttemptStateError("lifecycle result is invalid") if result["submission_mode"] not in SUBMISSION_MODES or result["completion_mode"] not in COMPLETION_MODES: raise AttemptStateError("lifecycle result is invalid") if not all(isinstance(result[key], str) for key in ("started_at", "ended_at")) or not isinstance(result["duration_ns"], int) or isinstance(result["duration_ns"], bool) or result["duration_ns"] < 0: @@ -1406,11 +1547,52 @@ class RunStore: checked = self._exact_fields(event, {"record", "kind", "source", "stream", "monotonic_ns", "source_monotonic_ns", "observed_at", "detail"}, "lifecycle event") if checked["record"] != "event" or not all(isinstance(checked[key], str) for key in ("kind", "source", "stream", "observed_at", "detail")) or not all(isinstance(checked[key], int) and not isinstance(checked[key], bool) and checked[key] >= 0 for key in ("monotonic_ns", "source_monotonic_ns")): raise AttemptStateError("lifecycle event is invalid") - if not result["cleanup_complete"] or result["process_group_alive"] or result["success"] != (result["terminal_reason"] == "success"): + if ( + result["process_group_alive"] + or (not result["harness"]["cleanup_complete"]) + != (result["harness"]["reason"] == "cleanup_failed") + ): raise AttemptStateError("lifecycle terminal outcome is invalid") - if result["success"] and not result["finish_then_idle_then_quiet"]: + if result["harness"]["status"] == "passed" and not result["harness"]["ordered_terminal"]: raise AttemptStateError("lifecycle terminal outcome is invalid") + def _validate_outcomes(self, value: Any, label: str) -> None: + if not isinstance(value, Mapping): + raise AttemptStateError(f"{label} schema is invalid") + product = self._exact_fields( + value.get("product"), {"status", "reason"}, f"{label} product" + ) + harness = self._exact_fields( + value.get("harness"), + {"status", "reason", "ordered_terminal", "cleanup_complete"}, + f"{label} harness", + ) + process = self._exact_fields( + value.get("process"), {"status", "exit_code", "signal"}, + f"{label} process", + ) + if ( + product["status"] not in PRODUCT_STATUSES + or product["reason"] not in PRODUCT_REASONS + or (product["status"] == "succeeded") != (product["reason"] == CALLER_REASON_SUCCESS) + or (product["status"] == "failed") != (product["reason"] == "caller_error") + or harness["status"] not in HARNESS_STATUSES + or harness["reason"] not in HARNESS_REASONS + or not isinstance(harness["ordered_terminal"], bool) + or not isinstance(harness["cleanup_complete"], bool) + or (harness["status"] == "passed") != (harness["reason"] == "success") + or (not harness["cleanup_complete"]) + != (harness["reason"] == "cleanup_failed") + or (harness["status"] == "passed" and not harness["ordered_terminal"]) + or process["status"] not in PROCESS_STATUSES + or not self._optional_int(process["exit_code"]) + or not self._optional_int(process["signal"]) + or (process["status"] == "signalled" and process["signal"] is None) + or (process["status"] in {"exited", "not_started"} and process["signal"] is not None) + or (process["status"] == "not_started" and process["exit_code"] is not None) + ): + raise AttemptStateError(f"{label} is invalid") + @staticmethod def _instant(value: Any, label: str) -> _datetime.datetime: """Parse one produced ISO-8601 instant into a comparable UTC value.""" @@ -1422,11 +1604,11 @@ class RunStore: @staticmethod def _validate_terminal_events(events: list[Any]) -> dict[str, int]: - """Return the unique ordinal of every success-evidence event kind.""" + """Return the unique ordinal of every product/harness evidence event.""" positions: dict[str, int] = {} for index, event in enumerate(events): kind = event["kind"] - if kind not in SUCCESS_EVIDENCE_KINDS: + if kind not in BOUND_EVIDENCE_KINDS: continue if kind in positions: raise AttemptStateError("lifecycle events are invalid") @@ -1436,15 +1618,33 @@ class RunStore: def _validate_terminal_coherence(self, result: dict[str, Any], receipt: dict[str, Any], positions: dict[str, int]) -> None: """Bind result, ordered events and cleanup receipt to one terminal projection.""" submitted, finish, idle, quiet = (positions.get(kind) for kind in SUCCESS_EVIDENCE_KINDS) + caller_terminal = positions.get(EVENT_CALLER_TERMINAL) ordered = finish is not None and idle is not None and quiet is not None and finish < idle < quiet - if ordered != result["finish_then_idle_then_quiet"]: + if ordered != result["harness"]["ordered_terminal"]: raise AttemptStateError("lifecycle ordered evidence is invalid") - if result["exit_code"] != receipt["exit_code"] or result["signal"] != receipt["signal"]: + if result["process"]["exit_code"] != receipt["exit_code"] or result["process"]["signal"] != receipt["signal"]: raise AttemptStateError("lifecycle terminal outcome is invalid") if result["submitted"] and not receipt["caller_launched"]: raise AttemptStateError("lifecycle terminal outcome is invalid") - if result["success"] and (not result["submitted"] or submitted is None or finish is None or submitted > finish): - raise AttemptStateError("lifecycle success evidence is invalid") + if result["product"]["status"] != "unknown": + product = result["product"] + expected_detail = f"status={product['status']} reason={product['reason']}" + terminal_event = ( + None if caller_terminal is None else result["events"][caller_terminal] + ) + if ( + not result["submitted"] + or submitted is None + or finish is None + or idle is None + or caller_terminal is None + or submitted > finish + or submitted > caller_terminal + or caller_terminal > idle + or terminal_event["source"] != SOURCE_CALLER_OUTPUT + or terminal_event["detail"] != expected_detail + ): + raise AttemptStateError("lifecycle product evidence is invalid") started = self._instant(result["started_at"], "lifecycle result") ended = self._instant(result["ended_at"], "lifecycle result") completed = self._instant(receipt["completed_at"], "cleanup receipt") @@ -1454,7 +1654,7 @@ class RunStore: def _validate_receipt_record(self, receipt: dict[str, Any], locator: SupervisorLocator) -> dict[str, Any]: """Validate one cleanup receipt against the registered supervisor identity.""" self._exact_fields(receipt, RECEIPT_FIELDS, "cleanup receipt") - if receipt["receipt_version"] != RECEIPT_VERSION or receipt["supervisor_pid"] != locator.supervisor_pid or receipt["challenge_digest"] != self._public_locator(locator)["challenge_digest"] or receipt["reason"] not in TERMINAL_REASONS or not self._optional_int(receipt["exit_code"]) or not self._optional_int(receipt["signal"]) or not isinstance(receipt["caller_launched"], bool) or receipt["cleanup_complete"] is not True or receipt["process_group_alive"] is not False or not isinstance(receipt["completed_at"], str): + if receipt["receipt_version"] != RECEIPT_VERSION or receipt["supervisor_pid"] != locator.supervisor_pid or receipt["challenge_digest"] != self._public_locator(locator)["challenge_digest"] or receipt["reason"] not in TERMINAL_REASONS or not self._optional_int(receipt["exit_code"]) or not self._optional_int(receipt["signal"]) or not isinstance(receipt["caller_launched"], bool) or not isinstance(receipt["cleanup_complete"], bool) or (not receipt["cleanup_complete"]) != (receipt["reason"] == "cleanup_failed") or receipt["process_group_alive"] is not False or not isinstance(receipt["completed_at"], str): raise AttemptStateError("cleanup receipt is invalid") return receipt @@ -1542,8 +1742,9 @@ class RunStore: raise AttemptStateError("lifecycle journal is invalid") header, terminal = lines[0], lines[-1] self._exact_fields(header, {"record", "journal_version", "spec_digest", "submission_mode", "completion_mode", "started_at"}, "lifecycle journal header") - self._exact_fields(terminal, {"record", "terminal_reason", "success", "cleanup_complete", "process_group_alive", "ended_at"}, "lifecycle journal terminal") - if header["record"] != "header" or header["journal_version"] != 1 or header["spec_digest"] != expected_digest or header["submission_mode"] != result["submission_mode"] or header["completion_mode"] != result["completion_mode"] or header["started_at"] != result["started_at"] or not isinstance(header["started_at"], str) or terminal["record"] != "terminal" or terminal["terminal_reason"] != result["terminal_reason"] or terminal["success"] != result["success"] or terminal["cleanup_complete"] is not True or terminal["process_group_alive"] is not False or terminal["ended_at"] != result["ended_at"] or not isinstance(terminal["ended_at"], str): + self._exact_fields(terminal, {"record", "product", "harness", "process", "process_group_alive", "ended_at"}, "lifecycle journal terminal") + self._validate_outcomes(terminal, "lifecycle journal terminal") + if header["record"] != "header" or header["journal_version"] != JOURNAL_VERSION or header["spec_digest"] != expected_digest or header["submission_mode"] != result["submission_mode"] or header["completion_mode"] != result["completion_mode"] or header["started_at"] != result["started_at"] or not isinstance(header["started_at"], str) or terminal["record"] != "terminal" or terminal["product"] != result["product"] or terminal["harness"] != result["harness"] or terminal["process"] != result["process"] or terminal["process_group_alive"] is not False or terminal["ended_at"] != result["ended_at"] or not isinstance(terminal["ended_at"], str): raise AttemptStateError("lifecycle journal is invalid") for event in lines[1:-1]: self._exact_fields(event, {"record", "kind", "source", "stream", "monotonic_ns", "source_monotonic_ns", "observed_at", "detail"}, "lifecycle journal event") @@ -1577,7 +1778,7 @@ class RunStore: receipt_data = self._validate_receipt_record( self._read_json_file(control, "cleanup-receipt.json"), locator ) - if receipt_data["reason"] != result["terminal_reason"]: + if receipt_data["reason"] != result["harness"]["reason"]: raise AttemptStateError("cleanup receipt is invalid") events = self._validate_terminal_events(result["events"]) self._validate_terminal_coherence(result, receipt_data, events) @@ -1607,7 +1808,7 @@ class RunStore: raise AttemptStateError("lifecycle terminal is unavailable") self._validate_measurement( root, run, attempt.identity, expected_digest, - str(terminal["terminal_reason"]), + _terminal_reason(terminal), record.get("measurement_policy"), terminal, ) self._validate_web_validation( @@ -1616,7 +1817,13 @@ class RunStore: attempt.identity, record.get("web_validation_policy"), ) - fields = ("success", "terminal_reason", "exit_code", "signal", "submitted", "finish_then_idle_then_quiet", "cleanup_complete", "process_group_alive", "spec_digest", "started_at", "ended_at", "duration_ns") + if ( + _outcome_records(invocation)["product"] != terminal["product"] + or _outcome_records(invocation)["harness"] != terminal["harness"] + or _outcome_records(invocation)["process"] != terminal["process"] + ): + raise AttemptStateError("invocation result does not match durable terminal") + fields = ("submitted", "process_group_alive", "spec_digest", "started_at", "ended_at", "duration_ns") if any(getattr(invocation, field) != terminal[field] for field in fields): raise AttemptStateError("invocation result does not match durable terminal") return terminal @@ -1660,7 +1867,7 @@ class RunStore: except MeasurementError as exc: raise AttemptStateError("attempt measurement is invalid") from exc self._validate_measurement( - root, run, attempt.identity, expected_digest, result.terminal_reason, + root, run, attempt.identity, expected_digest, result.harness.reason, record.get("measurement_policy"), ) @@ -1693,7 +1900,7 @@ class RunStore: record = self._attempt_record(root, run, attempt.identity, absent_ok=True) if record is None: terminal = self.publish_terminal( - attempt, "interrupted", result={"terminal_reason": "interrupted"} + attempt, "interrupted", result=_unknown_terminal("interrupted") ) self.release_control_lease(terminal) return terminal @@ -1704,7 +1911,7 @@ class RunStore: raw_locator = record.get("locator") if raw_locator is None: terminal = self.publish_terminal( - attempt, "interrupted", result={"terminal_reason": "interrupted"} + attempt, "interrupted", result=_unknown_terminal("interrupted") ) self.release_control_lease(terminal) return terminal @@ -1718,26 +1925,34 @@ class RunStore: if terminal is not None: self._validate_measurement( root, run, attempt.identity, expected_digest, - str(terminal["terminal_reason"]), + _terminal_reason(terminal), record.get("measurement_policy"), terminal, ) self._ensure_required_web_validation( root, run, attempt.identity, record, terminal ) published = self.publish_terminal( - attempt, self._state_for_reason(terminal["terminal_reason"]), result=terminal + attempt, self._state_for_terminal(terminal), result=terminal ) self.release_control_lease(published) return published closed_receipt = self._closed_cleanup_receipt(root, locator) if closed_receipt is not None: - recovery_terminal = {"terminal_reason": closed_receipt["reason"]} + recovery_terminal = _unknown_terminal( + closed_receipt["reason"], + process_status=( + "signalled" if closed_receipt.get("signal") is not None + else "exited" + ), + exit_code=closed_receipt.get("exit_code"), + signal=closed_receipt.get("signal"), + ) self._ensure_required_web_validation( root, run, attempt.identity, record, recovery_terminal ) published = self.publish_terminal( attempt, - self._state_for_reason(recovery_terminal["terminal_reason"]), + self._state_for_terminal(recovery_terminal), result=recovery_terminal, ) self.release_control_lease(published) @@ -1759,15 +1974,21 @@ class RunStore: raise AttemptStateError("recovery is unverified") from exc if recovered_receipt is None: raise AttemptStateError("recovery is unverified") from exc - recovery_terminal = { - "terminal_reason": recovered_receipt["reason"] - } + recovery_terminal = _unknown_terminal( + recovered_receipt["reason"], + process_status=( + "signalled" if recovered_receipt.get("signal") is not None + else "exited" + ), + exit_code=recovered_receipt.get("exit_code"), + signal=recovered_receipt.get("signal"), + ) self._ensure_required_web_validation( root, run, attempt.identity, record, recovery_terminal ) terminal = self.publish_terminal( attempt, - self._state_for_reason(recovery_terminal["terminal_reason"]), + self._state_for_terminal(recovery_terminal), result=recovery_terminal, ) self.release_control_lease(terminal) @@ -1779,13 +2000,18 @@ class RunStore: self._validate_receipt_record(self._read_json_file(receipt.parent, receipt.name), locator) except AttemptStateError as exc: raise AttemptStateError("recovery cleanup is unverified") from exc - recovery_terminal = {"terminal_reason": outcome.reason} + recovery_terminal = _unknown_terminal( + outcome.reason, + process_status="signalled" if outcome.signal is not None else "exited", + exit_code=outcome.exit_code, + signal=outcome.signal, + ) self._ensure_required_web_validation( root, run, attempt.identity, record, recovery_terminal ) terminal = self.publish_terminal( attempt, - self._state_for_reason(recovery_terminal["terminal_reason"]), + self._state_for_terminal(recovery_terminal), result=recovery_terminal, ) self.release_control_lease(terminal) @@ -1807,7 +2033,7 @@ class RunStore: try: prepare(attempt) except Exception: - _write_new(root / "attempt.json", _json_bytes(self._initial_record(run, attempt, "failed", reason="preparation_failed"))) + _write_new(root / "attempt.json", _json_bytes(self._initial_record(run, attempt, "interrupted", reason="launch_failed"))) raise _write_new( root / "attempt.json", @@ -1832,7 +2058,7 @@ class RunStore: result = invoke(attempt, lambda locator, digest: self.record_locator(attempt, locator, digest)) terminal = self.validate_invocation_terminal(attempt, result) published = self.publish_terminal( - attempt, self._state_for_reason(terminal["terminal_reason"]), result=terminal + attempt, self._state_for_terminal(terminal), result=terminal ) self.release_control_lease(published) return published @@ -1843,9 +2069,31 @@ class RunStore: if bound_run != run: raise AttemptStateError("run identity is invalid") states = {name: 0 for name in sorted(TERMINAL_STATES | {NONTERMINAL_STATE})} + outcomes = { + "product": {name: 0 for name in ("succeeded", "failed", "unknown")}, + "harness": {name: 0 for name in ("passed", "failed")}, + "process": { + name: 0 for name in ( + "exited", "signalled", "timed_out", "cancelled", "not_started" + ) + }, + "artifact": { + name: 0 for name in ("passed", "failed", "blocked", "not_run") + }, + "unresolved": 0, + } for slot in self.slots(manifest): - for attempt in self.attempts(bound_run, slot): + retained = self.attempts(bound_run, slot) + for attempt in retained: states[attempt.state] += 1 + if not retained or retained[-1].state == NONTERMINAL_STATE: + outcomes["unresolved"] += 1 + continue + projection = self.attempt_outcomes(retained[-1]) + for axis in ("product", "harness", "process", "artifact"): + outcomes[axis][projection[axis]] += 1 + if not projection["passed"]: + outcomes["unresolved"] += 1 preflights = self._preflight_records(bound_run, manifest) latest = preflights[-1] if preflights else None projection = { @@ -1863,6 +2111,7 @@ class RunStore: "manifest_digest": bound_run.manifest_digest, "preflight": projection, "attempts": states, + "outcomes": outcomes, } @@ -1873,10 +2122,8 @@ def preflight_manifest( *, adapters: Mapping[str, PreflightAdapter], ) -> tuple[RunIdentity, dict[str, Any]]: - """Collect direct observations, then create one run and append one record.""" + """Collect all-cell observations, then create one run and append one record.""" observations = collect_preflight_observations(manifest, adapters) - if not observations: - raise AttemptStateError("preflight requires a direct cell") run = store.create(manifest, manifest_bytes) return run, store.record_preflight(run, manifest, observations) @@ -1946,8 +2193,6 @@ def run_slots( raise AttemptStateError("workspace preparer is invalid") observations = collect_preflight_observations(manifest, adapters) - if not observations: - raise AttemptStateError("preflight requires a direct cell") bound_run = store.open(manifest, run.run_id) if bound_run != run: raise AttemptStateError("run identity is invalid") @@ -1968,14 +2213,13 @@ def run_slots( if cell is None: raise AttemptStateError("slot cell identity is invalid") existing = store.attempts(bound_run, slot) - if existing and existing[-1].state == "success": - continue - if existing and existing[-1].state in {"failed", "timed_out", "cancelled"} and not retry_failed: - continue if existing and existing[-1].state not in TERMINAL_STATES: store.reconcile(existing[-1]) existing = store.attempts(bound_run, slot) - if existing[-1].state == "success" or (existing[-1].state in {"failed", "timed_out", "cancelled"} and not retry_failed): + if existing: + if store.attempt_outcomes(existing[-1])["passed"]: + continue + if not retry_failed: continue attempt = store.allocate(bound_run, slot) prepared: PreparedWorkspace | None = None diff --git a/scripts/agent_benchmark/attempts_test.py b/scripts/agent_benchmark/attempts_test.py index 53cfa4a6..d810ee7c 100644 --- a/scripts/agent_benchmark/attempts_test.py +++ b/scripts/agent_benchmark/attempts_test.py @@ -42,10 +42,14 @@ from scripts.agent_benchmark.connectivity import ( make_result, ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_SUCCESS, + CALLER_STATUS_SUCCEEDED, COMPLETION_EXIT_AFTER_IDLE, SUBMISSION_ARGV_TASK, InvocationResult, InvocationSpec, + CallerEvent, + CallerTerminal, LifecycleRecoveryError, REASON_CONTROLLER_LOST, REASON_RECOVERED_STOP, @@ -72,7 +76,12 @@ from scripts.agent_benchmark.web_validation import ( from scripts.agent_benchmark.workspace import AttemptIdentity, prepare_workspace -def _manifest(root: Path, repetitions: int = 1): +def _manifest( + root: Path, + repetitions: int = 1, + execution_order_seed: str | None = None, + cell_ids: tuple[str, ...] = ("a",), +): fixtures = root / "scripts/fixtures" fixtures.mkdir(parents=True, exist_ok=True) (fixtures / "prompt.md").write_text("prompt", encoding="utf-8") @@ -89,16 +98,27 @@ def _manifest(root: Path, repetitions: int = 1): "viewports": [{"id": "desktop", "width": 1, "height": 1}], "rubric_version": "landing-quality-v1", "evaluator": {"caller": "codex", "iop": {"request_model": "judge", "requested_effort": "high", "route_kind": "direct", "route_id": "judge", "expected_bindings": [{"stage": "request", "model": "judge", "effort": "high"}]}}, "output_root": "agent-test/runs/a", "fixture": fixture, "repetitions": repetitions, - "matrix": [{"id": "a", "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}}], + "matrix": [ + {"id": cell_id, "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}} + for cell_id in cell_ids + ], } + if execution_order_seed is not None: + data["execution_order_seed"] = execution_order_seed path = root / "manifest.json" raw = json.dumps(data, sort_keys=True).encode("utf-8") path.write_bytes(raw) return load_manifest(path, repo_root=root), raw, path -def _events(_: str, line: str) -> str | None: - return {"FINISH": "finish", "IDLE": "idle"}.get(line.strip()) +def _events(_: str, line: str): + return { + "FINISH": ( + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + CallerEvent("finish"), + ), + "IDLE": CallerEvent("idle"), + }.get(line.strip()) def _preflight_observation(cell, issue_code: str | None = None) -> PreflightObservation: @@ -454,6 +474,33 @@ class AttemptBase(unittest.TestCase): class AttemptStoreTest(AttemptBase): + def test_slots_follow_seeded_manifest_order_before_repetitions(self): + self.manifest, self.raw, self.manifest_path = _manifest( + self.root, + repetitions=2, + execution_order_seed="seed-a", + cell_ids=("cell-a", "cell-b", "cell-c"), + ) + matrix_order = [cell.id for cell in self.manifest.matrix] + self.assertEqual(matrix_order, ["cell-c", "cell-a", "cell-b"]) + + slots = self.store.slots(self.manifest) + self.assertEqual( + [(slot.cell_id, slot.repetition) for slot in slots], + [ + ("cell-c", 1), + ("cell-c", 2), + ("cell-a", 1), + ("cell-a", 2), + ("cell-b", 1), + ("cell-b", 2), + ], + ) + self.assertEqual( + [slot.cell_id for slot in slots if slot.repetition == 1], + matrix_order, + ) + def test_slots_and_append_only_terminals(self): self.manifest, self.raw, self.manifest_path = _manifest(self.root, repetitions=2) run = self.create_run() @@ -461,9 +508,9 @@ class AttemptStoreTest(AttemptBase): with self.store.writer(run): first = self.store.allocate(run, Slot("a", 1)) self.assertEqual(list(Path(first.root).iterdir()), []) - terminal = self.store.publish_terminal(first, "failed") + terminal = self.store.publish_terminal(first, "interrupted") with self.assertRaises(AttemptStateError): - self.store.publish_terminal(terminal, "success") + self.store.publish_terminal(terminal, "completed") second = self.store.allocate(run, Slot("a", 1)) self.assertEqual(second.identity.attempt, 2) @@ -490,7 +537,7 @@ class AttemptStoreTest(AttemptBase): run = self.create_run() with self.store.writer(run): attempt = self.store.allocate(run, Slot("a", 1)) - self.store.publish_terminal(attempt, "failed") + self.store.publish_terminal(attempt, "interrupted") record = Path(attempt.root) / "attempt.json" foreign = json.loads(record.read_text(encoding="utf-8")) foreign["run_id"] = "run-20260102T030405Z-ffffffffffff" @@ -612,16 +659,17 @@ class AttemptOrchestrationTest(AttemptBase): return prepare_workspace(manifest, attempt.root, attempt.identity, repo_root=self.root) completed = run_slots(self.store, run, self.manifest, adapters={"claude": self.adapter("success", calls)}, prepare=prepare) - self.assertEqual([item.state for item in completed], ["success"]) + self.assertEqual([item.state for item in completed], ["completed"]) self.assertEqual(calls, ["preflight", "prepare", "invoke"]) attempt_root = Path(completed[0].root) self.assertTrue((attempt_root / "prepared.json").is_file()) state = json.loads((attempt_root / "attempt.json").read_text(encoding="utf-8")) + self.assertEqual(state["attempt_result_version"], 2) alias = Path(state["locator"]["control_dir"]).parent self.assertFalse(os.path.lexists(alias)) self.assertTrue((attempt_root / "control/locator.json").is_file()) self.assertTrue((attempt_root / "control/cleanup-receipt.json").is_file()) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) record_path = attempt_root / "attempt.json" original = record_path.read_bytes() corruptions = ( @@ -703,7 +751,7 @@ class AttemptOrchestrationTest(AttemptBase): with self.assertRaisesRegex(RuntimeError, "prepare failure"): run_slots(self.store, run, self.manifest, adapters={"claude": self.adapter("success", calls)}, prepare=fail_prepare) self.assertEqual(calls, ["preflight", "prepare"]) - self.assertEqual(self.store.attempts(run, Slot("a", 1))[-1].state, "failed") + self.assertEqual(self.store.attempts(run, Slot("a", 1))[-1].state, "interrupted") def test_retry_and_skip_preserve_prior_terminal_bytes(self): self._init_testbed() @@ -788,7 +836,7 @@ class AttemptMeasurementTest(AttemptBase): def test_successful_attempt_publishes_one_bound_measurement(self): threads_before = set(threading.enumerate()) run, completed = self._run() - self.assertEqual([item.state for item in completed], ["success"]) + self.assertEqual([item.state for item in completed], ["completed"]) attempt_root = Path(completed[0].root) measurement = load_measurement(attempt_root) @@ -797,7 +845,7 @@ class AttemptMeasurementTest(AttemptBase): (run.run_id, "a", 1, 1), ) self.assertEqual(measurement.caller, "claude") - self.assertEqual(measurement.terminal_reason, "success") + self.assertEqual(measurement.harness.reason, "success") state = json.loads((attempt_root / "attempt.json").read_text(encoding="utf-8")) self.assertEqual(measurement.spec_digest, state["spec_digest"]) self.assertEqual(state["measurement_policy"], "required-v1") @@ -836,7 +884,7 @@ class AttemptMeasurementTest(AttemptBase): _run, completed = self._run(mode) self.assertEqual(completed[0].state, state) measurement = load_measurement(Path(completed[0].root)) - self.assertEqual(measurement.terminal_reason, reason) + self.assertEqual(measurement.harness.reason, reason) self.assertEqual(measurement.observations, ()) for name in ("total_duration", "input_tokens", "model_calls"): self.assertEqual(measurement.usage[name].status, "unavailable") @@ -847,7 +895,7 @@ class AttemptMeasurementTest(AttemptBase): ) def test_failed_attempt_keeps_unavailable_values(self): - self._assert_unavailable_measurement("failed", "failed", "nonzero_exit") + self._assert_unavailable_measurement("failed", "completed", "nonzero_exit") def test_timed_out_attempt_keeps_unavailable_values(self): self._assert_unavailable_measurement("timeout", "timed_out", "timed_out") @@ -874,7 +922,18 @@ class AttemptMeasurementTest(AttemptBase): cases = { "foreign-attempt": {**record, "attempt": {**record["attempt"], "cell_id": "other"}}, "foreign-digest": {**record, "spec_digest": "sha256:" + "0" * 64}, - "rewritten-terminal": {**record, "terminal_reason": "timed_out"}, + "rewritten-terminal": { + **record, + "harness": {**record["harness"], "reason": "timed_out"}, + }, + "rewritten-product": { + **record, + "product": {"status": "failed", "reason": "caller_error"}, + }, + "rewritten-process": { + **record, + "process": {**record["process"], "exit_code": 7}, + }, "invented-total": { **record, "usage": { @@ -896,7 +955,7 @@ class AttemptMeasurementTest(AttemptBase): self.store.status(run, self.manifest) self.assertEqual(before, sidecar.read_bytes()) sidecar.write_bytes(original) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) def test_marked_measurement_is_required_and_bound_to_lifecycle_events(self): run, completed = self._run() @@ -1008,7 +1067,7 @@ class AttemptMeasurementTest(AttemptBase): reject_on_status_and_reconcile() finally: AttemptRecoveryTest._restore(marker, saved_marker) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) def test_explicitly_unmarked_lower_level_attempt_remains_compatible(self): run = self.create_run() @@ -1023,7 +1082,7 @@ class AttemptMeasurementTest(AttemptBase): self.assertNotIn("measurement_policy", record) self.assertFalse((root / MEASUREMENT_POLICY_FILENAME).exists()) self.assertFalse((root / MEASUREMENT_FILENAME).exists()) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) def test_nonregular_measurement_fails_closed_without_blocking(self): run, completed = self._run() @@ -1035,7 +1094,7 @@ class AttemptMeasurementTest(AttemptBase): with self.assertRaises(AttemptStateError): self.store.attempts(run, Slot("a", 1)) AttemptRecoveryTest._restore(sidecar, saved) - self.assertEqual(self.store.attempts(run, Slot("a", 1))[-1].state, "success") + self.assertEqual(self.store.attempts(run, Slot("a", 1))[-1].state, "completed") def test_recovery_commits_only_a_valid_bound_sidecar(self): self._init_testbed() @@ -1084,13 +1143,36 @@ class AttemptMeasurementTest(AttemptBase): sidecar.write_bytes(original) with self.store.writer(run): recovered = self.store.reconcile(attempt) - self.assertEqual(recovered.state, "success") + self.assertEqual(recovered.state, "completed") self.assertEqual(load_measurement(Path(attempt.root)).caller, "claude") class AttemptWebValidationTest(AttemptBase): """Required S12 policy, lifecycle mapping, and recovery-before-terminal.""" + def test_pre_registration_interruption_needs_no_impossible_sidecars(self): + run = self.create_run() + with self.store.writer(run): + attempt = self.store.allocate(run, Slot("a", 1)) + with self.assertRaisesRegex(ControllerCrash, "before registration"): + self.store.execute_attempt( + attempt, + prepare=lambda _: None, + invoke=lambda _attempt, _started: (_ for _ in ()).throw( + ControllerCrash("before registration") + ), + require_measurement=True, + require_web_validation=True, + ) + terminal = self.store.reconcile(attempt) + self.assertEqual(terminal.state, "interrupted") + root = Path(terminal.root) + self.assertFalse((root / MEASUREMENT_FILENAME).exists()) + self.assertFalse((root / WEB_VALIDATION_FILENAME).exists()) + self.assertEqual( + self.store.status(run, self.manifest)["attempts"]["interrupted"], 1 + ) + def _run(self, mode: str = "success"): self._init_testbed() run = self.create_run() @@ -1168,11 +1250,11 @@ class AttemptWebValidationTest(AttemptBase): ) return run, attempt - def test_lifecycle_status_matrix_publishes_not_run_for_non_success(self): + def test_lifecycle_status_matrix_validates_every_terminal_workspace(self): cases = ( - ("success", "success", "failed"), - ("failed", "failed", "not_run"), - ("timeout", "timed_out", "not_run"), + ("success", "completed", "failed"), + ("failed", "completed", "failed"), + ("timeout", "timed_out", "failed"), ) for mode, terminal, web_status in cases: with self.subTest(mode=mode): @@ -1182,9 +1264,7 @@ class AttemptWebValidationTest(AttemptBase): self.assertEqual(completed[0].state, terminal) web = load_web_validation(Path(completed[0].root)) self.assertEqual(web.status, web_status) - if web_status == "not_run": - self.assertTrue(web.record["reason"].startswith("lifecycle_")) - self.assertFalse(any(item["passed"] for item in web.record["gates"])) + self.assertNotEqual(web.status, "not_run") def test_normal_terminal_requires_web_sidecar_before_commit(self): _run, attempt = self._running_required_web(return_result=True) @@ -1199,10 +1279,10 @@ class AttemptWebValidationTest(AttemptBase): before = attempt_record.read_bytes() with self.store.writer(run): recovered = self.store.reconcile(attempt) - self.assertEqual(recovered.state, "success") + self.assertEqual(recovered.state, "completed") self.assertNotEqual(attempt_record.read_bytes(), before) self.assertEqual(load_web_validation(root).status, "failed") - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) def test_recovery_browser_start_failure_publishes_blocked(self): run, attempt = self._running_required_web(generated=True) @@ -1212,7 +1292,7 @@ class AttemptWebValidationTest(AttemptBase): ): with self.store.writer(run): recovered = self.store.reconcile(attempt) - self.assertEqual(recovered.state, "success") + self.assertEqual(recovered.state, "completed") web = load_web_validation(Path(attempt.root)) self.assertEqual(web.status, "blocked") self.assertFalse(web.record["screenshots"]) @@ -1285,7 +1365,7 @@ class AttemptWebValidationTest(AttemptBase): rejected() web_path.rmdir() web_path.write_bytes(saved["web"]) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) class AttemptRecoveryTest(AttemptBase): @@ -1314,10 +1394,10 @@ class AttemptRecoveryTest(AttemptBase): self.assertTrue(alias.is_symlink()) with self.store.writer(run): recovered = self.store.reconcile(attempt) - self.assertEqual(recovered.state, "success") + self.assertEqual(recovered.state, "completed") self.assertFalse(os.path.lexists(alias)) - self.assertEqual(self.store.status(run, self.manifest)["attempts"]["success"], 1) - self.assertEqual(self.store.reconcile(recovered).state, "success") + self.assertEqual(self.store.status(run, self.manifest)["attempts"]["completed"], 1) + self.assertEqual(self.store.reconcile(recovered).state, "completed") def test_corrupt_terminal_variants_fail_closed_and_preserve_bytes(self): run, attempt = self._running_with_terminal() @@ -1518,14 +1598,29 @@ class AttemptRecoveryTest(AttemptBase): ("receipt-completed-after-end", "receipt", lambda raw: raw.__setitem__("completed_at", "2100-01-01T00:00:00+00:00")), ("receipt-completed-unparseable", "receipt", lambda raw: raw.__setitem__("completed_at", "not-a-timestamp")), ("result-submitted", "result", lambda raw: raw.__setitem__("submitted", False)), - ("result-exit-code", "result", lambda raw: raw.__setitem__("exit_code", 7)), + ( + "result-exit-code", + "result", + lambda raw: raw["process"].__setitem__("exit_code", 7), + ), ("events-cleared", "events", lambda events: []), ("events-missing-submitted", "events", _without("submitted")), + ("events-missing-caller-terminal", "events", _without("caller_terminal")), ("events-missing-finish", "events", _without("finish")), ("events-missing-idle", "events", _without("idle")), ("events-missing-quiet", "events", _without("quiet")), ("events-out-of-order", "events", _reordered), ("events-duplicate-finish", "events", _duplicated("finish")), + ( + "events-caller-terminal-detail", + "events", + lambda events: [ + {**event, "detail": "status=failed reason=caller_error"} + if event["kind"] == "caller_terminal" + else event + for event in events + ], + ), ) for name, record, mutate in cases: with self.subTest(case=name): @@ -1545,13 +1640,51 @@ class AttemptRecoveryTest(AttemptBase): with self.store.writer(run): recovered = self.store.reconcile(attempt) published = record.read_bytes() - self.assertEqual(recovered.state, "success") + self.assertEqual(recovered.state, "completed") self.assertNotEqual(running, published) with self.store.writer(run): - self.assertEqual(self.store.reconcile(recovered).state, "success") + self.assertEqual(self.store.reconcile(recovered).state, "completed") self.assertEqual(published, record.read_bytes()) self.assertEqual(len(self.store.attempts(run, Slot("a", 1))), 1) + def test_coherent_cleanup_failure_remains_a_terminal_independent_axis(self): + run, attempt = self._running_with_terminal() + root = Path(attempt.root) + result_path = root / "lifecycle-result.json" + journal_path = root / "lifecycle-journal.jsonl" + receipt_path = root / "control" / "cleanup-receipt.json" + + result = json.loads(result_path.read_text(encoding="utf-8")) + result["harness"].update({ + "status": "failed", + "reason": "cleanup_failed", + "cleanup_complete": False, + }) + result_path.write_text(json.dumps(result), encoding="utf-8") + + journal = [ + json.loads(line) + for line in journal_path.read_text(encoding="utf-8").splitlines() + ] + journal[-1]["harness"] = result["harness"] + journal_path.write_text( + "".join(json.dumps(line) + "\n" for line in journal), + encoding="utf-8", + ) + + receipt = json.loads(receipt_path.read_text(encoding="utf-8")) + receipt.update({"reason": "cleanup_failed", "cleanup_complete": False}) + receipt_path.write_text(json.dumps(receipt), encoding="utf-8") + + with self.store.writer(run): + terminal = self.store.reconcile(attempt) + projection = self.store.attempt_outcomes(terminal) + self.assertEqual(terminal.state, "completed") + self.assertEqual( + (projection["product"], projection["harness"], projection["process"]), + ("succeeded", "failed", "exited"), + ) + def test_symlink_lifecycle_evidence_fails_closed(self): run, attempt = self._running_with_terminal() journal = Path(attempt.root) / "lifecycle-journal.jsonl" @@ -1576,7 +1709,7 @@ class AttemptRecoveryTest(AttemptBase): result = self.adapter("success", [])(current, started) path = Path(current.root) / "lifecycle-result.json" raw = json.loads(path.read_text(encoding="utf-8")) - raw["success"] = False + raw["product"]["status"] = "unknown" path.write_text(json.dumps(raw), encoding="utf-8") return result @@ -1772,7 +1905,7 @@ class AttemptRecoveryTest(AttemptBase): published = json.loads(attempt_record.read_text(encoding="utf-8")) self.assertEqual(published["state"], "interrupted") self.assertEqual( - published["lifecycle"]["terminal_reason"], + published["lifecycle"]["harness"]["reason"], REASON_CONTROLLER_LOST, ) self.assertTrue(alias.is_symlink()) @@ -1799,19 +1932,19 @@ class AttemptRecoveryTest(AttemptBase): ( "terminal-record-reason", attempt_record, - lambda raw: raw["lifecycle"].__setitem__( - "terminal_reason", "success" + lambda raw: raw["lifecycle"]["harness"].__setitem__( + "reason", "success" ), ), ( "terminal-record-state-success", attempt_record, - lambda raw: raw.__setitem__("state", "success"), + lambda raw: raw.__setitem__("state", "completed"), ), ( "terminal-record-state-failed", attempt_record, - lambda raw: raw.__setitem__("state", "failed"), + lambda raw: raw.__setitem__("state", "cancelled"), ), ) for name, target, tamper in terminal_tamper_cases: @@ -1919,8 +2052,8 @@ class AttemptRecoveryTest(AttemptBase): ) ) self.assertEqual(record["state"], "interrupted") - self.assertEqual(record["lifecycle"]["terminal_reason"], REASON_RECOVERED_STOP) - self.assertEqual(result["terminal_reason"], REASON_RECOVERED_STOP) + self.assertEqual(record["lifecycle"]["harness"]["reason"], REASON_RECOVERED_STOP) + self.assertEqual(result["harness"]["reason"], REASON_RECOVERED_STOP) self.assertEqual(receipt["reason"], REASON_RECOVERED_STOP) self.assertTrue(receipt["cleanup_complete"]) self.assertFalse(receipt["process_group_alive"]) @@ -1961,7 +2094,10 @@ class AttemptCliContractTest(AttemptBase): output = io.StringIO() with mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root), contextlib.redirect_stdout(output): self.assertEqual(benchmark_cli.main(["status", "--manifest", str(self.manifest_path), "--run-id", run.run_id]), 0) - self.assertIn("'running': 0", output.getvalue()) + self.assertIn("running=0", output.getvalue()) + self.assertIn("product_succeeded=0", output.getvalue()) + self.assertIn("artifact_passed=0", output.getvalue()) + self.assertIn("unresolved=1", output.getvalue()) self.assertEqual( run_before, { diff --git a/scripts/agent_benchmark/claude_iop.py b/scripts/agent_benchmark/claude_iop.py index 0d10d4c3..3208e01b 100644 --- a/scripts/agent_benchmark/claude_iop.py +++ b/scripts/agent_benchmark/claude_iop.py @@ -23,14 +23,22 @@ from scripts.agent_benchmark.connectivity import ( make_result, ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, + CALLER_STATUS_FAILED, + CALLER_STATUS_SUCCEEDED, COMPLETION_EXIT_AFTER_IDLE, SUBMISSION_STDIN_ONCE, + CallerEvent, + CallerTerminal, InvocationSpec, LifecycleMetricError, ParsedMetric, + TLS_CA_ENV_KEYS, count_metric, duration_metric, exact_value_redactor, + inherited_tls_ca_environment, is_reported_number, ) from scripts.agent_benchmark.manifest import MatrixCell, Timeout @@ -62,6 +70,15 @@ _CLAUDE_USAGE_FIELDS = { "cache_read_input_tokens": "cached_input_tokens", "cache_creation_input_tokens": "cache_write_tokens", } +_CLAUDE_USAGE_BOOKKEEPING_FIELDS = { + "cache_creation", "inference_geo", "iterations", "output_tokens_details", + "server_tool_use", "service_tier", "speed", +} +_CLAUDE_USAGE_NESTED_COUNTS = { + "cache_creation": {"ephemeral_1h_input_tokens", "ephemeral_5m_input_tokens"}, + "output_tokens_details": {"thinking_tokens"}, + "server_tool_use": {"web_fetch_requests", "web_search_requests"}, +} class ClaudeIopError(Exception): @@ -221,6 +238,10 @@ class ClaudeStreamParser: self.claude_session_id: str | None = None self._phase = "await_init" self._assistant_messages = 0 + self._active_message_id: str | None = None + self._completed_message_ids: set[str] = set() + self._continuation_pending = False + self._api_error_seen = False def _require_bound_session(self, event: dict[str, Any]) -> None: if self.claude_session_id is None: @@ -236,27 +257,123 @@ class ClaudeStreamParser: self.claude_session_id = _required_string(event, "session_id") self._phase = "await_assistant" - def _consume_assistant(self, event: dict[str, Any]) -> str: + def _consume_assistant(self, event: dict[str, Any]) -> CallerEvent | None: if self._phase != "await_assistant": raise ClaudeIopProtocolError("duplicate or out-of-order Claude assistant") self._require_bound_session(event) message = event.get("message") - if not isinstance(message, dict) or message.get("stop_reason") != "end_turn": - raise ClaudeIopProtocolError("invalid Claude assistant terminal") + if not isinstance(message, dict): + raise ClaudeIopProtocolError("invalid Claude assistant event") + if event.get("is_api_error_message") is True: + if message.get("model") != "" or message.get("stop_reason") != "stop_sequence": + raise ClaudeIopProtocolError("invalid Claude API error event") + self._complete_active_message() + self._api_error_seen = True + self._phase = "await_error_result" + return None if _required_string(message, "model") != self.cell.iop.request_model: raise ClaudeIopProtocolError("Claude model binding mismatch") - self._assistant_messages += 1 - self._phase = "await_result" - return "finish" + stop_reason = message.get("stop_reason") + if stop_reason not in (None, "tool_use", "end_turn"): + raise ClaudeIopProtocolError("invalid Claude assistant stop reason") + message_id = message.get("id") + if message_id is not None and (not isinstance(message_id, str) or not message_id): + raise ClaudeIopProtocolError("invalid Claude assistant message id") + if self._continuation_pending: + if message_id != self._active_message_id: + self._complete_active_message() + self._continuation_pending = False + if stop_reason is None: + if message_id is None: + raise ClaudeIopProtocolError("unbound Claude assistant snapshot") + if message_id in self._completed_message_ids: + raise ClaudeIopProtocolError("duplicate Claude assistant message") + if self._active_message_id not in (None, message_id): + raise ClaudeIopProtocolError("overlapping Claude assistant messages") + self._active_message_id = message_id + return None - def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...]: - if self._phase != "await_result": + if message_id is None: + # Older fixture-shaped output did not include a message id. It is + # admissible only for the single final assistant event. + if stop_reason != "end_turn" or self._active_message_id is not None: + raise ClaudeIopProtocolError("unbound Claude assistant terminal") + self._assistant_messages += 1 + else: + if message_id in self._completed_message_ids: + raise ClaudeIopProtocolError("duplicate Claude assistant message") + if self._active_message_id not in (None, message_id): + raise ClaudeIopProtocolError("overlapping Claude assistant messages") + self._active_message_id = message_id + self._complete_active_message() + if stop_reason == "tool_use": + self._phase = "await_tool_result" + return None + self._phase = "await_result" + return CallerEvent("finish") + + def _consume_user(self, event: dict[str, Any]) -> None: + if self._phase not in ("await_assistant", "await_tool_result"): + raise ClaudeIopProtocolError("out-of-order Claude user event") + self._require_bound_session(event) + if self._active_message_id is not None: + # Claude Code may publish cumulative snapshots with the same + # assistant message id on both sides of one or more tool results. + # The next assistant snapshot (or the result terminal) determines + # whether this message continues or a new model call begins. + self._continuation_pending = True + elif self._phase != "await_tool_result": + raise ClaudeIopProtocolError("unexpected Claude user event") + self._phase = "await_assistant" + + def _complete_active_message(self) -> None: + if self._active_message_id is None: + return + if self._active_message_id in self._completed_message_ids: + raise ClaudeIopProtocolError("duplicate Claude assistant message") + self._completed_message_ids.add(self._active_message_id) + self._active_message_id = None + self._continuation_pending = False + self._assistant_messages += 1 + + def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...] | None: + if self._phase == "await_error_result": + self._require_bound_session(event) + if ( + not self._api_error_seen + or event.get("subtype") != "success" + or event.get("is_error") is not True + or event.get("terminal_reason") != "api_error" + ): + raise ClaudeIopProtocolError("invalid Claude API error terminal") + self._phase = "complete" + return ( + CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR), + CallerEvent("finish"), + CallerEvent("idle"), + ) + result_completes_active_message = False + if self._phase == "await_assistant" and self._active_message_id is not None: + if self._continuation_pending: + raise ClaudeIopProtocolError("Claude result followed an unresolved tool result") + self._complete_active_message() + result_completes_active_message = True + elif self._phase != "await_result": raise ClaudeIopProtocolError("duplicate or out-of-order Claude result") self._require_bound_session(event) - if event.get("subtype") != "success": + if event.get("subtype") != "success" or event.get("is_error") is True: raise ClaudeIopProtocolError("invalid Claude result terminal") self._phase = "complete" - return (*self._observations(event), "idle") + terminal = ( + (CallerEvent("finish"), CallerEvent("idle")) + if result_completes_active_message + else (CallerEvent("idle"),) + ) + return ( + *self._observations(event), + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + *terminal, + ) def _observations(self, event: dict[str, Any]) -> tuple[ParsedMetric, ...]: """Convert only allowlisted reported Claude values into observations.""" @@ -284,14 +401,34 @@ class ClaudeStreamParser: usage = event.get("usage") if usage is None: return [] - if not isinstance(usage, dict) or not set(usage) <= set(_CLAUDE_USAGE_FIELDS): + allowed = set(_CLAUDE_USAGE_FIELDS) | _CLAUDE_USAGE_BOOKKEEPING_FIELDS + if not isinstance(usage, dict) or not set(usage) <= allowed: raise ClaudeIopProtocolError("invalid Claude usage observation") + ClaudeStreamParser._validate_usage_bookkeeping(usage) return [ count_metric(_CLAUDE_USAGE_FIELDS[field], value, model=model) - for field, value in sorted(usage.items()) + for field, value in sorted(usage.items()) if field in _CLAUDE_USAGE_FIELDS ] - def __call__(self, stream: str, raw_line: str) -> str | tuple[Any, ...] | None: + @staticmethod + def _validate_usage_bookkeeping(usage: dict[str, Any]) -> None: + for field, nested_fields in _CLAUDE_USAGE_NESTED_COUNTS.items(): + if field not in usage: + continue + value = usage[field] + if not isinstance(value, dict) or not set(value) <= nested_fields: + raise ClaudeIopProtocolError("invalid Claude usage observation") + if any(not is_reported_number(item) for item in value.values()): + raise ClaudeIopProtocolError("invalid Claude usage observation") + if "iterations" in usage and usage["iterations"] != []: + raise ClaudeIopProtocolError("invalid Claude usage observation") + for field in ("inference_geo", "service_tier", "speed"): + if field in usage and ( + not isinstance(usage[field], str) or len(usage[field]) > 64 + ): + raise ClaudeIopProtocolError("invalid Claude usage observation") + + def __call__(self, stream: str, raw_line: str) -> Any: if stream != "stdout": return None event = _exact_object(raw_line) @@ -301,6 +438,9 @@ class ClaudeStreamParser: return None if event_type == "assistant": return self._consume_assistant(event) + if event_type == "user": + self._consume_user(event) + return None if event_type == "result": return self._consume_result(event) # Informational events are deliberately ignored only after they have @@ -371,19 +511,21 @@ class ClaudeIopAdapter: ("ANTHROPIC_API_KEY", self.api_key), ("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "1"), ("CLAUDE_CODE_DISABLE_AUTOUPDATER", "1"), - ) + ) + inherited_tls_ca_environment() return InvocationSpec( argv=( self.binary, "--bare", "--print", "--verbose", "--input-format", "text", "--output-format", "stream-json", "--model", self.cell.iop.request_model, "--effort", self.cell.iop.requested_effort, - "--no-session-persistence", "--permission-mode", "dontAsk", "--tools=", + "--no-session-persistence", "--permission-mode", "dontAsk", + "--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit", ), cwd=str(cwd), env=env, env_allowlist=( "ANTHROPIC_BASE_URL", "ANTHROPIC_API_KEY", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "CLAUDE_CODE_DISABLE_AUTOUPDATER", + *TLS_CA_ENV_KEYS, ), submission_mode=SUBMISSION_STDIN_ONCE, completion_mode=COMPLETION_EXIT_AFTER_IDLE, diff --git a/scripts/agent_benchmark/claude_iop_test.py b/scripts/agent_benchmark/claude_iop_test.py index 368e78df..d1ea5028 100644 --- a/scripts/agent_benchmark/claude_iop_test.py +++ b/scripts/agent_benchmark/claude_iop_test.py @@ -8,6 +8,7 @@ import tempfile import textwrap import unittest from pathlib import Path +from unittest.mock import patch from scripts.agent_benchmark.claude_iop import ( ClaudeIopAdapter, @@ -18,7 +19,10 @@ from scripts.agent_benchmark.claude_iop import ( parse_preflight_binding, redact_claude_event, ) -from scripts.agent_benchmark.lifecycle import REASON_PARSER_ERROR, REASON_SUCCESS, run_invocation +from scripts.agent_benchmark.lifecycle import ( + CallerEvent, CallerTerminal, ParsedMetric, + REASON_PARSER_ERROR, REASON_SUCCESS, run_invocation, +) from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCell, Timeout from scripts.agent_benchmark.workspace import ( AttemptIdentity, @@ -99,7 +103,7 @@ class ClaudeIopTest(unittest.TestCase): fixture = Path("scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl") return fixture.read_text(encoding="utf-8").splitlines() - def _run_fake(self, lines: list[str]): + def _run_fake(self, lines: list[str], *, exit_code: int = 0): self.binary.write_text(textwrap.dedent(f"""\ #!/usr/bin/env python3 import os, sys @@ -108,6 +112,7 @@ class ClaudeIopTest(unittest.TestCase): assert os.environ["ANTHROPIC_API_KEY"] == {SENTINELS[2]!r} for line in {lines!r}: print(line) + raise SystemExit({exit_code!r}) """), encoding="utf-8") evidence = self.root / f"evidence-{len(tuple(self.root.glob('evidence-*')))}" evidence.mkdir() @@ -124,18 +129,22 @@ class ClaudeIopTest(unittest.TestCase): def test_exact_iop_only_invocation_and_fresh_workspace(self) -> None: adapter = self._adapter() - spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1)) + with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}): + spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1)) self.assertEqual(spec.cwd, self.workspace.workspace_dir) self.assertEqual(spec.submission_mode, "stdin_once") self.assertEqual(spec.task_payload, SENTINELS[0].encode()) self.assertEqual(spec.argv[1:], ( "--bare", "--print", "--verbose", "--input-format", "text", "--output-format", "stream-json", "--model", "claude-sonnet", "--effort", "high", - "--no-session-persistence", "--permission-mode", "dontAsk", "--tools=", + "--no-session-persistence", "--permission-mode", "dontAsk", + "--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit", )) env = dict(spec.env) self.assertEqual(env["ANTHROPIC_BASE_URL"], SENTINELS[1]) self.assertEqual(env["ANTHROPIC_API_KEY"], SENTINELS[2]) + self.assertEqual(env["SSL_CERT_FILE"], "/operator/dev-ca.pem") + self.assertEqual(env["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem") self.assertNotIn("ANTHROPIC_AUTH_TOKEN", env) self.assertNotIn("CLAUDE_CONFIG_DIR", env) (self.root / "other").mkdir() @@ -176,8 +185,8 @@ class ClaudeIopTest(unittest.TestCase): lines = self._fixture_lines() parser = ClaudeStreamParser(self.cell, "session-fixture") parsed = [parser("stdout", line) for line in lines] - self.assertEqual(parsed[:2], [None, "finish"]) - self.assertEqual(parsed[2][-1], "idle") + self.assertEqual(parsed[:2], [None, CallerEvent("finish")]) + self.assertEqual(parsed[2][-1], CallerEvent("idle")) malformed = json.loads(lines[1]) del malformed["message"]["model"] @@ -213,7 +222,135 @@ class ClaudeIopTest(unittest.TestCase): for event in events: parser("stdout", event) parser = ClaudeStreamParser(self.cell, "session-fixture") - self.assertEqual([parser("stdout", event) for event in missing_result], [None, "finish"]) + self.assertEqual([parser("stdout", event) for event in missing_result], [None, CallerEvent("finish")]) + + def test_parser_accepts_partial_snapshots_and_tool_result_cycles(self) -> None: + init, _, result = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + partial = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-tool-1", "model": "claude-sonnet", "stop_reason": None, + "content": [{"type": "tool_use", "id": "tool-1"}]}, + }) + user = json.dumps({ + "type": "user", "session_id": "claude-session-fixture", + "message": {"content": [{"type": "tool_result", "tool_use_id": "tool-1"}]}, + }) + final = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-final", "model": "claude-sonnet", + "stop_reason": "end_turn", "content": []}, + }) + self.assertEqual( + [parser("stdout", line) for line in (init, partial, partial, user, final)], + [None, None, None, None, CallerEvent("finish")], + ) + observations = parser("stdout", result) + metrics = { + metric.name: metric.value for metric in observations + if isinstance(metric, ParsedMetric) + } + self.assertEqual(metrics["model_calls"], 2) + + def test_parser_accepts_cumulative_message_ids_and_direct_result(self) -> None: + init, _, result = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + partial = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-cumulative", "model": "claude-sonnet", + "stop_reason": None, "content": []}, + }) + user = json.dumps({ + "type": "user", "session_id": "claude-session-fixture", + "message": {"content": []}, + }) + terminal = json.loads(result) + terminal["usage"].update({ + "cache_creation": {"ephemeral_1h_input_tokens": 0, "ephemeral_5m_input_tokens": 0}, + "inference_geo": "", "iterations": [], + "output_tokens_details": {"thinking_tokens": 0}, + "server_tool_use": {"web_fetch_requests": 0, "web_search_requests": 0}, + "service_tier": "standard", "speed": "standard", + }) + parsed = [ + parser("stdout", line) + for line in (init, partial, partial, user, partial, user, user, partial) + ] + self.assertEqual(parsed, [None] * len(parsed)) + observations = parser("stdout", json.dumps(terminal)) + self.assertEqual(observations[-2:], (CallerEvent("finish"), CallerEvent("idle"))) + self.assertEqual( + {metric.name: metric.value for metric in observations if isinstance(metric, ParsedMetric)}["model_calls"], 1 + ) + + def test_parser_classifies_synthetic_api_error_without_parser_failure(self) -> None: + init, _, _ = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + synthetic = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "is_api_error_message": True, + "message": {"id": "synthetic", "model": "", + "stop_reason": "stop_sequence", "content": []}, + }) + terminal = json.dumps({ + "type": "result", "session_id": "claude-session-fixture", + "subtype": "success", "is_error": True, "terminal_reason": "api_error", + }) + self.assertEqual( + [parser("stdout", line) for line in (init, synthetic, terminal)], + [None, None, ( + CallerTerminal("failed", "caller_error"), + CallerEvent("finish"), CallerEvent("idle"), + )], + ) + + outcome, _ = self._run_fake([init, synthetic, terminal], exit_code=1) + self.assertFalse(outcome.product.status == "succeeded") + self.assertEqual(outcome.product.status, "failed") + self.assertEqual(outcome.harness.status, "passed") + + def test_lifecycle_accepts_result_direct_active_snapshot(self) -> None: + init, _, result = self._fixture_lines() + partial = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-direct", "model": "claude-sonnet", + "stop_reason": None, "content": []}, + }) + outcome, _ = self._run_fake([init, partial, result]) + self.assertTrue(outcome.product.status == "succeeded", outcome) + kinds = [event.kind for event in outcome.events] + self.assertLess(kinds.index("finish"), kinds.index("idle")) + + def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None: + init, _, result = self._fixture_lines() + parser = ClaudeStreamParser(self.cell, "session-fixture") + tool_use = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-tool-1", "model": "claude-sonnet", + "stop_reason": "tool_use", "content": []}, + }) + user = json.dumps({ + "type": "user", "session_id": "claude-session-fixture", "message": {"content": []}, + }) + final = json.dumps({ + "type": "assistant", "session_id": "claude-session-fixture", + "message": {"id": "msg-final", "model": "claude-sonnet", + "stop_reason": "end_turn", "content": []}, + }) + self.assertEqual( + [parser("stdout", line) for line in (init, tool_use, user, final)], + [None, None, None, CallerEvent("finish")], + ) + self.assertEqual( + {metric.name: metric.value for metric in parser("stdout", result) if isinstance(metric, ParsedMetric)}["model_calls"], + 2, + ) + duplicate = ClaudeStreamParser(self.cell, "session-fixture") + duplicate("stdout", init) + duplicate("stdout", tool_use) + duplicate("stdout", user) + with self.assertRaises(ClaudeIopProtocolError): + duplicate("stdout", tool_use) def test_reported_result_values_become_bound_normalized_observations(self) -> None: init, assistant, result = self._fixture_lines() @@ -221,8 +358,8 @@ class ClaudeIopTest(unittest.TestCase): parser("stdout", init) parser("stdout", assistant) parsed = parser("stdout", result) - self.assertEqual(parsed[-1], "idle") - observed = {metric.name: metric for metric in parsed[:-1]} + self.assertEqual(parsed[-1], CallerEvent("idle")) + observed = {metric.name: metric for metric in parsed if isinstance(metric, ParsedMetric)} self.assertEqual(observed["total_duration"].value, 1234 * 10 ** 6) self.assertFalse(observed["total_duration"].overlap) # The reported API duration is inside the reported total, so it is @@ -236,7 +373,9 @@ class ClaudeIopTest(unittest.TestCase): # The fixture omits cache creation, so that category stays unreported # rather than being reported as zero. self.assertNotIn("cache_write_tokens", observed) - for metric in parsed[:-1]: + for metric in parsed: + if not isinstance(metric, ParsedMetric): + continue self.assertEqual(metric.model, "claude-sonnet") self.assertEqual(metric.source, "caller_output") self.assertEqual(observed["input_tokens"].clock, "none") @@ -293,10 +432,10 @@ class ClaudeIopTest(unittest.TestCase): json.dumps(init), json.dumps(diagnostic), json.dumps(assistant), json.dumps(result), ]) result = outcome - self.assertTrue(result.success, result) - self.assertEqual(result.terminal_reason, REASON_SUCCESS) + self.assertTrue(result.product.status == "succeeded", result) + self.assertEqual(result.harness.reason, REASON_SUCCESS) self.assertTrue(result.submitted) - self.assertTrue(result.finish_then_idle_then_quiet) + self.assertTrue(result.harness.ordered_terminal) for sentinel in (*SENTINELS, *ARBITRARY_SENTINELS): self.assertNotIn(sentinel, durable) self.assertEqual( @@ -326,15 +465,15 @@ class ClaudeIopTest(unittest.TestCase): for name, lines in cases.items(): with self.subTest(name=name): outcome, durable = self._run_fake(lines) - self.assertFalse(outcome.success, outcome) + self.assertFalse(outcome.product.status == "succeeded", outcome) for sentinel in (*SENTINELS, *ARBITRARY_SENTINELS): self.assertNotIn(sentinel, durable) def test_metric_prefixed_malformed_output_is_redacted_before_durable_capture(self) -> None: metric_sentinel = "metric:" + SENTINELS[0] outcome, durable = self._run_fake([metric_sentinel]) - self.assertFalse(outcome.success, outcome) - self.assertEqual(outcome.terminal_reason, REASON_PARSER_ERROR) + self.assertFalse(outcome.product.status == "succeeded", outcome) + self.assertEqual(outcome.harness.reason, REASON_PARSER_ERROR) self.assertTrue(outcome.submitted) self.assertIn("invalid_claude_json", durable) for sentinel in (*SENTINELS, metric_sentinel, SENTINELS[0]): diff --git a/scripts/agent_benchmark/codex_iop.py b/scripts/agent_benchmark/codex_iop.py index dac87d2f..46e2b949 100644 --- a/scripts/agent_benchmark/codex_iop.py +++ b/scripts/agent_benchmark/codex_iop.py @@ -29,18 +29,26 @@ if __package__ in (None, ""): sys.path.insert(0, str(Path(__file__).resolve().parents[2])) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, + CALLER_STATUS_FAILED, + CALLER_STATUS_SUCCEEDED, COMPLETION_EXIT_AFTER_IDLE, SUBMISSION_STDIN_ONCE, + CallerEvent, + CallerTerminal, InvocationResult, InvocationSpec, LifecycleMetricError, LifecycleValidationError, ParsedMetric, SupervisorLocator, + TLS_CA_ENV_KEYS, count_metric, duration_metric, env_pairs, exact_value_redactor, + inherited_tls_ca_environment, is_reported_number, run_invocation, ) @@ -77,12 +85,16 @@ _SAFE_STRING_KEYS = frozenset({ _CODEX_USAGE_FIELDS = { "input_tokens": "input_tokens", "cached_input_tokens": "cached_input_tokens", + "cache_write_input_tokens": "cache_write_tokens", "output_tokens": "output_tokens", "reasoning_output_tokens": "reasoning_tokens", "total_tokens": "total_tokens", } _CODEX_ITEM_COMPLETED = "item.completed" _CODEX_CALL_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.:-]{0,63}$") +_CODEX_TOOL_ITEM_TYPES = frozenset({ + "command_execution", "file_change", "mcp_tool_call", "web_search", +}) class CodexIOPError(Exception): @@ -219,7 +231,7 @@ def build_codex_spec( raise CodexRuntimeError("invalid Codex executable") codex_argv: list[str] = [ - *executable_argv, "exec", "--json", "--ephemeral", "--ignore-user-config", + *executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config", "--strict-config", "--skip-git-repo-check", "-C", prepared.workspace_dir, "-m", cell.iop.request_model, ] @@ -232,8 +244,13 @@ def build_codex_spec( # import resolution in the child bridge. argv=(sys.executable, str(Path(__file__).resolve()), "--bridge", "--", *codex_argv), cwd=prepared.workspace_dir, - env=env_pairs({"PATH": runtime.path, "HOME": prepared.session_dir, SECRET_ENV_KEY: runtime.api_key}), - env_allowlist=(SECRET_ENV_KEY,), + env=env_pairs({ + "PATH": runtime.path, + "HOME": prepared.session_dir, + SECRET_ENV_KEY: runtime.api_key, + **dict(inherited_tls_ca_environment()), + }), + env_allowlist=(SECRET_ENV_KEY, *TLS_CA_ENV_KEYS), submission_mode=SUBMISSION_STDIN_ONCE, completion_mode=COMPLETION_EXIT_AFTER_IDLE, timeout=timeout, @@ -300,7 +317,7 @@ class CodexJSONLParser: ) return make_result(self._cell, codex_capability(), binding, issues) - def parse(self, stream: str, line: str) -> str | ParsedMetric | tuple[Any, ...] | None: + def parse(self, stream: str, line: str) -> Any: if stream != "stdout": return None try: @@ -313,15 +330,22 @@ class CodexJSONLParser: if record["type"] == "turn.completed": status = record.get("status") if status is not None and status not in ("completed", "success"): - raise CodexJSONLError("unsuccessful Codex terminal turn") + return ( + CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR), + CallerEvent("finish"), + ) self._turns += 1 - return (*self._turn_observations(record), "finish") + return ( + *self._turn_observations(record), + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + CallerEvent("finish"), + ) if record["type"] == _CODEX_ITEM_COMPLETED: return self._tool_interval(record) if record["type"] == _BRIDGE_IDLE_TYPE: if record != {"type": _BRIDGE_IDLE_TYPE, "adapter": _BRIDGE_ID, "nonce": self._idle_nonce, "child_exit": 0}: raise CodexJSONLError("unverified bridge idle marker") - return "idle" + return CallerEvent("idle") return None def _turn_observations(self, record: dict[str, Any]) -> tuple[ParsedMetric, ...]: @@ -346,17 +370,20 @@ class CodexJSONLParser: return tuple(observations) def _tool_interval(self, record: dict[str, Any]) -> ParsedMetric | None: - """Observe one explicitly paired tool interval; never infer a pairing.""" + """Count a completed tool and optionally observe its reported interval.""" item = record.get("item") - if not isinstance(item, dict) or "duration_ms" not in item: + if not isinstance(item, dict) or item.get("type") not in _CODEX_TOOL_ITEM_TYPES: return None call_id = item.get("id") if not isinstance(call_id, str) or _CODEX_CALL_ID_RE.fullmatch(call_id) is None: - raise CodexJSONLError("unpaired Codex tool interval") + raise CodexJSONLError("unpaired Codex tool completion") + if call_id in self._tool_calls: + raise CodexJSONLError("duplicate Codex tool completion") + if "duration_ms" not in item: + self._tool_calls.add(call_id) + return None if not is_reported_number(item["duration_ms"]): raise CodexJSONLError("invalid Codex tool interval") - if call_id in self._tool_calls: - raise CodexJSONLError("duplicate Codex tool interval") try: observation = duration_metric( "tool_duration", item["duration_ms"], reported_unit="ms", diff --git a/scripts/agent_benchmark/codex_iop_test.py b/scripts/agent_benchmark/codex_iop_test.py index a967e6dd..0282737b 100644 --- a/scripts/agent_benchmark/codex_iop_test.py +++ b/scripts/agent_benchmark/codex_iop_test.py @@ -8,6 +8,7 @@ import sys import tempfile import unittest from pathlib import Path +from unittest.mock import patch from scripts.agent_benchmark.codex_iop import ( BASE_URL_ENV_KEY, @@ -22,6 +23,8 @@ from scripts.agent_benchmark.codex_iop import ( runtime_from_environment, ) from scripts.agent_benchmark.lifecycle import ( + CallerEvent, + ParsedMetric, REASON_DUPLICATE_EVENT, REASON_NONZERO_EXIT, REASON_PARSER_ERROR, @@ -95,13 +98,14 @@ class CodexIOPTest(unittest.TestCase): return str(path) def test_exact_isolated_responses_spec_uses_one_stdin_submission(self) -> None: - spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout()) + with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}): + spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout()) argv = list(spec.argv) codex = argv[argv.index("--") + 1:] self.assertFalse((self.workspace / ".git").exists()) self.assertEqual(codex.count("--skip-git-repo-check"), 1) - self.assertEqual(codex[:12], [ - "codex", "exec", "--json", "--ephemeral", "--ignore-user-config", + self.assertEqual(codex[:14], [ + "codex", "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config", "--strict-config", "--skip-git-repo-check", "-C", str(self.workspace), "-m", "gpt-5.6-luna", "-c", ]) @@ -118,6 +122,8 @@ class CodexIOPTest(unittest.TestCase): self.assertEqual(spec.submission_mode, "stdin_once") self.assertEqual(spec.completion_mode, "exit_after_idle") self.assertEqual(dict(spec.env)[SECRET_ENV_KEY], _SECRET) + self.assertEqual(dict(spec.env)["SSL_CERT_FILE"], "/operator/dev-ca.pem") + self.assertEqual(dict(spec.env)["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem") self.assertNotIn(BASE_URL_ENV_KEY, dict(spec.env)) self.assertNotIn("OPENAI_API_KEY", dict(spec.env)) @@ -133,17 +139,18 @@ class CodexIOPTest(unittest.TestCase): parser = CodexJSONLParser(_cell(), "fixture-nonce-0001") fixture = Path("scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl") events = [parser.parse("stdout", line) for line in fixture.read_text(encoding="utf-8").splitlines()] - self.assertEqual([events[0], events[-1]], [None, "idle"]) + self.assertEqual([events[0], events[-1]], [None, CallerEvent("idle")]) tool = events[1] self.assertEqual((tool.name, tool.value, tool.call_id), ("tool_duration", 7_250_000, "call-1")) # A tool interval is reported inside the turn, so it is published as an # overlapping interval instead of a subtractable slice. self.assertTrue(tool.overlap) - self.assertEqual(events[2][-1], "finish") - turn = {metric.name: metric.value for metric in events[2][:-1]} + self.assertEqual(events[2][-1], CallerEvent("finish")) + turn = {metric.name: metric.value for metric in events[2] if isinstance(metric, ParsedMetric)} self.assertEqual(turn, { - "cached_input_tokens": 8, "input_tokens": 31, "output_tokens": 12, - "reasoning_tokens": 4, "model_calls": 1, "tool_calls": 1, + "cache_write_tokens": 6, "cached_input_tokens": 8, + "input_tokens": 31, "output_tokens": 12, "reasoning_tokens": 4, + "model_calls": 1, "tool_calls": 1, }) # The fixture omits the provider total, so it is never reconstructed # from the reported categories. @@ -161,10 +168,10 @@ class CodexIOPTest(unittest.TestCase): fake = self._fake_codex(["TASK"]) invocation = build_codex_invocation(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake)) result = run_codex_invocation(invocation, lambda _: None) - self.assertTrue(result.lifecycle.success) + self.assertTrue(result.lifecycle.product.status == "succeeded") self.assertEqual([event.kind for event in result.lifecycle.events], [ "submitted", "first_output", "metric:model_calls", "metric:tool_calls", - "finish", "idle", "exited", "quiet", + "caller_terminal", "finish", "idle", "exited", "quiet", ]) capture = result.lifecycle.stdout.text self.assertNotIn(_PROMPT.decode(), capture) @@ -175,8 +182,8 @@ class CodexIOPTest(unittest.TestCase): def test_child_failure_never_synthesizes_idle(self) -> None: fake = self._fake_codex([{"type": "turn.completed", "status": "completed"}], exit_code=7) result = run_codex_invocation(build_codex_invocation(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake)), lambda _: None) - self.assertFalse(result.lifecycle.success) - self.assertEqual(result.lifecycle.terminal_reason, REASON_NONZERO_EXIT) + self.assertFalse(result.lifecycle.product.status == "succeeded") + self.assertEqual(result.lifecycle.harness.reason, REASON_NONZERO_EXIT) self.assertNotIn("idle", [event.kind for event in result.lifecycle.events]) def test_duplicate_malformed_and_unverified_idle_fail_closed(self) -> None: @@ -192,30 +199,36 @@ class CodexIOPTest(unittest.TestCase): prepared = self._prepared().__class__(**{**self._prepared().__dict__, "attempt_root": str(evidence)}) fake = self._fake_codex(records) result = run_codex_invocation(build_codex_invocation(_cell(), prepared, self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake)), lambda _: None) - self.assertFalse(result.lifecycle.success) - self.assertEqual(result.lifecycle.terminal_reason, reason) + self.assertFalse(result.lifecycle.product.status == "succeeded") + self.assertEqual(result.lifecycle.harness.reason, reason) def test_tool_intervals_require_one_explicit_unique_pairing(self) -> None: parser = CodexJSONLParser(_cell(), "0123456789abcdef") - # An item without a reported duration carries no interval to pair. + # A non-tool item carries neither a tool count nor an interval. self.assertIsNone(parser.parse("stdout", json.dumps( {"type": "item.completed", "item": {"type": "agent_message", "text": "x"}} ))) - paired = json.dumps({"type": "item.completed", "item": {"id": "call-1", "duration_ms": 3}}) + # Real Codex command completions omit duration; they must still count. + self.assertIsNone(parser.parse("stdout", json.dumps( + {"type": "item.completed", "item": {"id": "call-0", "type": "command_execution"}} + ))) + paired = json.dumps({"type": "item.completed", "item": { + "id": "call-1", "type": "command_execution", "duration_ms": 3, + }}) self.assertEqual(parser.parse("stdout", paired).call_id, "call-1") for name, item in { - "duplicate-call": {"id": "call-1", "duration_ms": 4}, - "unpaired-duration": {"duration_ms": 4}, - "unsafe-call-id": {"id": "call 1", "duration_ms": 4}, - "string-duration": {"id": "call-2", "duration_ms": "4"}, - "negative-duration": {"id": "call-3", "duration_ms": -4}, + "duplicate-call": {"id": "call-1", "type": "command_execution", "duration_ms": 4}, + "unpaired-duration": {"type": "command_execution", "duration_ms": 4}, + "unsafe-call-id": {"id": "call 1", "type": "command_execution", "duration_ms": 4}, + "string-duration": {"id": "call-2", "type": "command_execution", "duration_ms": "4"}, + "negative-duration": {"id": "call-3", "type": "command_execution", "duration_ms": -4}, }.items(): with self.subTest(name=name): with self.assertRaises(CodexJSONLError): parser.parse("stdout", json.dumps({"type": "item.completed", "item": item})) turn = parser.parse("stdout", json.dumps({"type": "turn.completed"})) - counts = {metric.name: metric.value for metric in turn[:-1]} - self.assertEqual(counts, {"model_calls": 1, "tool_calls": 1}) + counts = {metric.name: metric.value for metric in turn if isinstance(metric, ParsedMetric)} + self.assertEqual(counts, {"model_calls": 1, "tool_calls": 2}) def test_unknown_or_fractional_turn_usage_fails_closed(self) -> None: for usage in ( diff --git a/scripts/agent_benchmark/connectivity_integration_test.py b/scripts/agent_benchmark/connectivity_integration_test.py index 7af2bdd9..96fbb5b9 100644 --- a/scripts/agent_benchmark/connectivity_integration_test.py +++ b/scripts/agent_benchmark/connectivity_integration_test.py @@ -25,6 +25,7 @@ from scripts import agent_comparison_benchmark as benchmark_cli from scripts.agent_benchmark import live_iop from scripts.agent_benchmark.browser_cdp import RenderObservation, ViewportObservation from scripts.agent_benchmark.attempts import ( + AttemptStateError, CapabilityUnavailable, PreflightObservation, RunBusyError, @@ -42,7 +43,7 @@ from scripts.agent_benchmark.connectivity import ( canonical_evidence_bytes, make_result, ) -from scripts.agent_benchmark.codex_iop import CodexInvocationResult +from scripts.agent_benchmark.codex_iop import CodexInvocation, CodexInvocationResult from scripts.agent_benchmark.manifest import ( AssetMapping, ExpectedBinding, @@ -51,6 +52,9 @@ from scripts.agent_benchmark.manifest import ( load_manifest, ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, + CALLER_STATUS_SUCCEEDED, CLOCK_HARNESS_MONOTONIC, COMPLETION_EXIT_AFTER_IDLE, METRIC_NAMES, @@ -59,10 +63,15 @@ from scripts.agent_benchmark.lifecycle import ( SUBMISSION_STDIN_ONCE, UNIT_NANOSECONDS, CaptureStream, + CallerEvent, + CallerTerminal, InvocationSpec, InvocationResult, + HarnessOutcome, LifecycleRecoveryError, ParsedMetric, + ProcessOutcome, + ProductOutcome, env_pairs, recover_invocation, run_invocation, @@ -88,6 +97,49 @@ from scripts.agent_benchmark.web_validation import ( ) +def _invocation_result( + *, + product_status: str = "succeeded", + harness_reason: str = "success", + exit_code: int = 0, + metrics: tuple[ParsedMetric, ...] = (), + spec_digest_value: str = "sha256:" + "a" * 64, + started_at: str = "2026-08-11T00:00:00+00:00", + ended_at: str = "2026-08-11T00:00:01+00:00", +) -> InvocationResult: + """Build an independent-axis lifecycle value for integration seams.""" + stream = CaptureStream("stdout", "", 0, 0, False) + product_reason = { + "succeeded": CALLER_REASON_SUCCESS, + "failed": CALLER_REASON_ERROR, + "unknown": "unavailable", + }[product_status] + harness_passed = harness_reason == "success" + return InvocationResult( + ProductOutcome(product_status, product_reason), + HarnessOutcome( + "passed" if harness_passed else "failed", + harness_reason, + harness_passed, + True, + ), + ProcessOutcome("exited", exit_code, None), + True, + False, + (), + stream, + replace(stream, stream="stderr"), + "", + "", + None, + spec_digest_value, + started_at, + ended_at, + 1, + metrics, + ) + + def _cell(cell_id: str, caller: str, model: str, effort: str) -> dict: return { "id": cell_id, @@ -200,15 +252,17 @@ if name == "claude": "result": leak}, ] elif name == "agy": - binding = {"route_kind": "direct", "route_id": route_id, - "model": option("--model"), "effort": option("--effort")} events = [ - {"type": "metric", "subtype": "duration_ms", "value": 12.5}, - dict(binding, type="iop", subtype="effective_binding", - stages=[{"stage": "request", "model": binding["model"], - "effort": binding["effort"]}]), - dict(binding, type="result", subtype="success", text=leak), - dict(binding, type="system", subtype="idle"), + {"event": "init", "conversation_id": "fixture", + "init": {"model": "Gemini 3.6 Flash", "permission_mode": "sandbox", "tools": []}}, + {"event": "step_update", "step_update": { + "state": "DONE", "step_index": 0, "step_type": "agent_response", + "text_delta": leak, + "usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}}, + {"event": "result", "result": { + "conversation_id": "fixture", "status": "SUCCESS", + "duration_seconds": 0.0125, "num_turns": 1, "response": leak, + "usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}}, ] else: events = [ @@ -217,10 +271,8 @@ else: "item": {"id": "call-1", "type": "command_execution", "duration_ms": 7.25, "output": leak}}, {"type": "turn.completed", "status": "completed", - "usage": {"input_tokens": 31, "cached_input_tokens": 8, "output_tokens": 12}, - "iop_effective_binding": { - "route_kind": "direct", "route_id": route_id, "model": option("-m"), - "effort": override("model_reasoning_effort=")}}, + "usage": {"input_tokens": 31, "cached_input_tokens": 8, + "cache_write_input_tokens": 6, "output_tokens": 12}}, ] for event in events: @@ -386,8 +438,11 @@ class FakeAdapter: return run_invocation( spec, parse_event=lambda _stream, line: { - "FINISH": "finish", - "IDLE": "idle", + "FINISH": ( + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + CallerEvent("finish"), + ), + "IDLE": CallerEvent("idle"), }.get(line.strip()), on_started=lambda locator: on_started(locator, spec_digest(spec)), ) @@ -465,6 +520,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): for caller in ("CLAUDE", "AGY", "CODEX"): environment[f"IOP_BENCH_{caller}_BASE_URL"] = "http://127.0.0.1:18083/v1" environment[f"IOP_BENCH_{caller}_SECRET_ENV"] = "BENCH_TOKEN" + environment["IOP_BENCH_AGY_BASE_URL"] = "https://127.0.0.1:18083" return environment @staticmethod @@ -493,7 +549,9 @@ class ConnectivityIntegrationTest(unittest.TestCase): attempt.identity.attempt, caller, "sha256:" + "3" * 64, - "success", + ProductOutcome("succeeded", CALLER_REASON_SUCCESS), + HarnessOutcome("passed", "success", True, True), + ProcessOutcome("exited", 0, None), timeline, usage, WorkspaceWriteObservation( @@ -624,7 +682,25 @@ class ConnectivityIntegrationTest(unittest.TestCase): build_web_validation(manifest, workspace, measurement, render), ) return self.store.publish_terminal( - attempt, "success", result={"terminal_reason": "success"} + attempt, + "completed", + result={ + "product": { + "status": "succeeded", + "reason": CALLER_REASON_SUCCESS, + }, + "harness": { + "status": "passed", + "reason": "success", + "ordered_terminal": True, + "cleanup_complete": True, + }, + "process": { + "status": "exited", + "exit_code": 0, + "signal": None, + }, + }, ) def _run_live_scoring_mutation(self, case, mutate): @@ -649,14 +725,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): def invoke(invocation, _on_started): mutate(invocation, secret, base_url) - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), - ) + lifecycle = _invocation_result() binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -691,6 +760,8 @@ class ConnectivityIntegrationTest(unittest.TestCase): environment[secret_env] = branch["secret"] environment[prefix + "SECRET_ENV"] = secret_env environment[prefix + "BASE_URL"] = f"http://{branch['endpoint']}.invalid:18083/v1" + if caller == "agy": + environment[prefix + "BASE_URL"] = f"https://{branch['endpoint']}.invalid:18083" return environment @staticmethod @@ -800,11 +871,12 @@ class ConnectivityIntegrationTest(unittest.TestCase): digests = {state["spec_digest"], result["spec_digest"], header["spec_digest"]} self.assertEqual(len(digests), 1, attempt_root) self.assertRegex(digests.pop(), r"^sha256:[0-9a-f]{64}$") - self.assertEqual(state["state"], "success") - self.assertIs(result["success"], True) - self.assertEqual(result["terminal_reason"], "success") - self.assertIs(result["finish_then_idle_then_quiet"], True) - self.assertIs(result["cleanup_complete"], True) + self.assertEqual(state["state"], "completed") + self.assertEqual(result["product"]["status"], "succeeded") + self.assertEqual(result["harness"]["status"], "passed") + self.assertIs(result["harness"]["ordered_terminal"], True) + self.assertIs(result["harness"]["cleanup_complete"], True) + self.assertEqual(result["process"]["status"], "exited") self.assertIs(result["process_group_alive"], False) control_dir = Path(state["locator"]["control_dir"]) alias = control_dir.parent @@ -824,10 +896,13 @@ class ConnectivityIntegrationTest(unittest.TestCase): "total_duration", "model_duration", "model_calls", "input_tokens", "output_tokens", "cached_input_tokens", }, - "agy": {"total_duration"}, + "agy": { + "total_duration", "model_calls", "input_tokens", "output_tokens", + "total_tokens", + }, "codex": { "model_calls", "tool_calls", "input_tokens", "output_tokens", - "cached_input_tokens", + "cached_input_tokens", "cache_write_tokens", }, } @@ -842,15 +917,19 @@ class ConnectivityIntegrationTest(unittest.TestCase): callers.add(caller) # This matrix binds the cell id to the caller name. self.assertEqual(measurement.cell_id, caller) - self.assertEqual(measurement.terminal_reason, "success") + self.assertEqual(measurement.harness.reason, "success") observed = { name for name, item in measurement.usage.items() if item.status == "observed" } self.assertEqual(observed, self._EXPECTED_TOTALS[caller], attempt_root) - # No caller reports a provider total, and none is reconstructed. - self.assertEqual(measurement.usage["total_tokens"].status, "unavailable") - self.assertIsNone(measurement.usage["total_tokens"].value) + # Official agy reports total_tokens; other callers in this fixture + # do not, and no total is reconstructed for them. + if caller == "agy": + self.assertEqual(measurement.usage["total_tokens"].value, 33) + else: + self.assertEqual(measurement.usage["total_tokens"].status, "unavailable") + self.assertIsNone(measurement.usage["total_tokens"].value) self.assertEqual(measurement.timeline["first_output_at"].status, "observed") self.assertEqual( measurement.timeline["first_write_mtime"].clock, "filesystem_mtime" @@ -910,7 +989,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): root.mkdir() matrix = [ _cell("claude", "claude", "sonnet", "max"), - _cell("agy", "agy", "gemini", "high"), + _cell("agy", "agy", "gemini-3.6-flash", "high"), _cell("codex", "codex", "gpt", "xhigh"), ] sentinels = _branch_sentinels() @@ -931,7 +1010,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): tuple(sorted(cell.iop.request_model for cell in manifest.matrix)), "sha256:" + "f" * 64, True, - "agy 1.1.11", + "agy 1.1.12", "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json", ) @@ -1006,14 +1085,15 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.assertFalse((run_roots[0] / "cells").exists()) self.assertTrue(all(adapter.invocations == [] for adapter in registry.values())) - def test_cli_mixed_manifest_never_invokes_unobserved_preset_cells(self) -> None: + def test_cli_mixed_manifest_preflights_and_invokes_every_cell(self) -> None: + self._init_testbed() manifest, _, path = _write_manifest( self.root, [ _cell("direct-ready", "claude", "claude-sonnet-5", "max"), - _preset("preset-unobserved", "claude", "claude-sonnet-5", "max"), + _preset("preset-ready", "claude", "claude-sonnet-5", "max"), ], - output_id="mixed-unobserved", + output_id="mixed-ready", ) registry = self._registry() stdout = io.StringIO() @@ -1030,8 +1110,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.assertEqual(exit_code, 69) self.assertEqual(stdout.getvalue(), "") - self.assertIn("error: benchmark execution failed", stderr.getvalue()) - self.assertIn("completed=0 unresolved=2", stderr.getvalue()) + self.assertIn("artifact_failed=2", stderr.getvalue()) run_roots = list((self.root / manifest.output_root).glob("run-*")) self.assertEqual(len(run_roots), 1) preflight = json.loads( @@ -1042,11 +1121,64 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.assertEqual(preflight["status"], "ready") self.assertEqual( [result["cell"]["id"] for result in preflight["results"]], - ["direct-ready"], + [cell.id for cell in manifest.matrix], ) - self.assertFalse((run_roots[0] / "cells").exists()) - self.assertEqual(registry["claude"].calls, ["direct-ready"]) - self.assertTrue(all(adapter.invocations == [] for adapter in registry.values())) + self.assertEqual( + registry["claude"].calls, + [cell.id for cell in manifest.matrix], + ) + self.assertEqual( + [item[0] for item in registry["claude"].invocations], + [cell.id for cell in manifest.matrix], + ) + self.assertEqual( + len(list(run_roots[0].glob("cells/*/repetition-*/attempt-*"))), + len(manifest.matrix), + ) + + def test_cli_failed_attempt_does_not_prevent_later_slot(self) -> None: + self._init_testbed() + manifest, _, path = _write_manifest( + self.root, + [ + _cell("agy-first", "agy", "gemini-3.6-flash", "high"), + _cell("codex-later", "codex", "gpt-5.6-luna", "xhigh"), + ], + output_id="terminal-failure-continues", + ) + agy = FakeAdapter("agy", ("high",)) + agy.fail_invocation = True + codex = FakeAdapter("codex", ("xhigh",)) + self.addCleanup(agy.cleanup) + self.addCleanup(codex.cleanup) + stdout = io.StringIO() + stderr = io.StringIO() + with ( + mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root), + mock.patch.object( + benchmark_cli, + "build_adapter_registry", + return_value={"agy": agy, "codex": codex}, + ), + contextlib.redirect_stdout(stdout), + contextlib.redirect_stderr(stderr), + ): + exit_code = benchmark_cli.main(["run", "--manifest", str(path)]) + + self.assertEqual(exit_code, 69) + self.assertEqual(stdout.getvalue(), "") + match = re.search(r"run_id=(run-[0-9A-Za-z-]+)", stderr.getvalue()) + self.assertIsNotNone(match) + self.assertEqual([item[0] for item in agy.invocations], ["agy-first"]) + self.assertEqual([item[0] for item in codex.invocations], ["codex-later"]) + status = self.store.status(manifest=manifest, run=self.store.open( + manifest, match.group(1) # type: ignore[union-attr] + )) + self.assertEqual(status["attempts"]["completed"], 2) + self.assertEqual(status["outcomes"]["product"]["unknown"], 1) + self.assertEqual(status["outcomes"]["product"]["succeeded"], 1) + self.assertEqual(status["outcomes"]["artifact"]["failed"], 2) + self.assertEqual(status["attempts"]["running"], 0) def test_cli_resume_retries_append_only_and_status_is_read_only(self) -> None: self._init_testbed() @@ -1110,8 +1242,10 @@ class ConnectivityIntegrationTest(unittest.TestCase): ] ) - self.assertEqual(resume_exit, 0, resume_stderr.getvalue()) - self.assertIn("ok: resume", resume_stdout.getvalue()) + self.assertEqual(resume_exit, 69) + self.assertEqual(resume_stdout.getvalue(), "") + self.assertIn("product_succeeded=1", resume_stderr.getvalue()) + self.assertIn("artifact_failed=1", resume_stderr.getvalue()) self.assertEqual( old_bytes, { @@ -1122,7 +1256,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.assertTrue( next(run_root.glob("cells/*/repetition-*/attempt-000002/attempt.json")) .read_text(encoding="utf-8") - .find('"state":"success"') + .find('"state":"completed"') >= 0 ) self.assertEqual(len(list((run_root / "preflight").glob("*.json"))), 2) @@ -1143,7 +1277,8 @@ class ConnectivityIntegrationTest(unittest.TestCase): ["status", "--manifest", str(path), "--run-id", run_id] ) self.assertEqual(status_exit, 0, status_stderr.getvalue()) - self.assertIn("'success': 1", status_stdout.getvalue()) + self.assertIn("product_succeeded=1", status_stdout.getvalue()) + self.assertIn("artifact_failed=1", status_stdout.getvalue()) self.assertEqual( before_status, { @@ -1163,8 +1298,8 @@ class ConnectivityIntegrationTest(unittest.TestCase): ) self.assertFalse(output_root.exists()) - def test_generic_preset_cells_are_local_contract_only(self) -> None: - generic, _, _ = _write_manifest( + def test_preset_only_public_preflight_appends_exact_results(self) -> None: + generic, _, path = _write_manifest( self.root, [ _preset("claude-generic", "claude", "claude-sonnet-5", "high"), @@ -1174,31 +1309,6 @@ class ConnectivityIntegrationTest(unittest.TestCase): output_id="generic", ) registry = self._registry() - observations = collect_preflight_observations(generic, registry) - self.assertEqual(observations, {}) - self.assertTrue(all(adapter.calls == [] for adapter in registry.values())) - - def test_generic_preset_only_public_preflight_fails_closed_without_run_state(self) -> None: - generic, raw, path = _write_manifest( - self.root, - [ - _preset("claude-generic", "claude", "claude-sonnet-5", "high"), - _preset("agy-generic", "agy", "gemini-3.6-flash", "high"), - _preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"), - ], - output_id="generic", - ) - registry = self._registry() - output_root = self.root / generic.output_root - with self.assertRaises(Exception) as ctx: - preflight_manifest( - self.store, generic, raw, adapters=registry - ) - self.assertIn("preflight requires a direct cell", str(ctx.exception)) - self.assertFalse(output_root.exists()) - self.assertTrue(all(adapter.calls == [] for adapter in registry.values())) - - sentinel = "private_endpoint_and_token_must_not_appear" stdout = io.StringIO() stderr = io.StringIO() with ( @@ -1208,10 +1318,103 @@ class ConnectivityIntegrationTest(unittest.TestCase): contextlib.redirect_stderr(stderr), ): exit_code = benchmark_cli.main(["preflight", "--manifest", str(path)]) + + self.assertEqual(exit_code, 0, stderr.getvalue()) + self.assertIn("status=ready", stdout.getvalue()) + self.assertEqual(stderr.getvalue(), "") + run_roots = list((self.root / generic.output_root).glob("run-*")) + self.assertEqual(len(run_roots), 1) + record = json.loads( + (run_roots[0] / "preflight/preflight-000001.json").read_text( + encoding="ascii" + ) + ) + self.assertEqual( + [result["cell"]["id"] for result in record["results"]], + [cell.id for cell in generic.matrix], + ) + self.assertFalse((run_roots[0] / "cells").exists()) + self.assertEqual( + {caller: adapter.calls for caller, adapter in registry.items()}, + { + "claude": ["claude-generic"], + "agy": ["agy-generic"], + "codex": ["codex-generic"], + }, + ) + + def test_preset_blocker_appends_without_attempt_allocation(self) -> None: + generic, _, path = _write_manifest( + self.root, + [ + _preset("claude-generic", "claude", "claude-sonnet-5", "high"), + _preset("agy-generic", "agy", "gemini-3.6-flash", "high"), + _preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"), + ], + output_id="generic-blocked", + ) + registry = self._registry( + {"agy-generic": ("credential_missing",)} + ) + stdout = io.StringIO() + stderr = io.StringIO() + with ( + mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root), + mock.patch.object(benchmark_cli, "build_adapter_registry", return_value=registry), + contextlib.redirect_stdout(stdout), + contextlib.redirect_stderr(stderr), + ): + exit_code = benchmark_cli.main(["run", "--manifest", str(path)]) self.assertEqual(exit_code, 69) self.assertEqual(stdout.getvalue(), "") - self.assertNotIn(sentinel, stderr.getvalue()) - self.assertFalse(output_root.exists()) + self.assertIn("error: preflight blocked", stderr.getvalue()) + run_roots = list((self.root / generic.output_root).glob("run-*")) + self.assertEqual(len(run_roots), 1) + record = json.loads( + (run_roots[0] / "preflight/preflight-000001.json").read_text( + encoding="ascii" + ) + ) + self.assertEqual(record["status"], "registration_required") + self.assertEqual( + [result["cell"]["id"] for result in record["results"]], + [cell.id for cell in generic.matrix], + ) + self.assertFalse((run_roots[0] / "cells").exists()) + self.assertTrue(all(adapter.invocations == [] for adapter in registry.values())) + + def test_all_cell_preflight_rejects_missing_extra_and_reordered_results(self) -> None: + manifest, raw, _ = _write_manifest( + self.root, + [ + _cell("direct", "claude", "claude-sonnet-5", "max"), + _preset("preset", "claude", "claude-sonnet-5", "max"), + ], + output_id="all-cell-corruption", + ) + run, record = preflight_manifest( + self.store, manifest, raw, adapters=self._registry() + ) + path = Path(run.root) / "preflight/preflight-000001.json" + original = path.read_bytes() + mutations = { + "missing": record["results"][:-1], + "extra": [*record["results"], record["results"][0]], + "reordered": list(reversed(record["results"])), + } + for label, results in mutations.items(): + with self.subTest(label=label): + mutated = {**record, "results": results} + path.write_bytes( + json.dumps( + mutated, sort_keys=True, separators=(",", ":") + ).encode("ascii") + + b"\n" + ) + with self.assertRaises(AttemptStateError): + self.store.preflights(run, manifest) + path.write_bytes(original) + self.assertEqual(self.store.preflights(run, manifest), (record,)) def test_live_registry_dereferences_secret_names_without_persisting_values(self) -> None: sentinel = "live-token-must-not-persist" @@ -1223,7 +1426,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "c" * 64, True, - "agy 1.1.11", + "agy 1.1.12", "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json", ) @@ -1294,23 +1497,14 @@ class ConnectivityIntegrationTest(unittest.TestCase): ) captured = {} + caller_binding = {"value": None} + def invoke(invocation, _on_started): captured["spec"] = invocation.spec - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), + lifecycle = _invocation_result() + return CodexInvocationResult( + lifecycle, caller_binding["value"] ) - binding = ( - evaluator.iop.route_kind, - evaluator.iop.route_id, - evaluator.iop.request_model, - evaluator.iop.requested_effort, - ) - return CodexInvocationResult(lifecycle, binding) adapter = live_iop.build_live_scoring_adapter( environment, @@ -1348,11 +1542,49 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.manifest.timeout, lambda *_args: None, ) - self.assertTrue(result.success) - self.assertEqual(result.effective_binding[0], route_kind) + self.assertEqual( + (result.product, result.harness, result.process), + ("succeeded", "passed", "exited"), + ) + expected_binding = ( + evaluator.iop.route_kind, + evaluator.iop.route_id, + evaluator.iop.request_model, + evaluator.iop.requested_effort, + ) + self.assertEqual(result.effective_binding, expected_binding) finalized = adapter.finalize_evidence(blind) self.assertTrue(finalized.safe) + caller_binding["value"] = ( + evaluator.iop.route_kind, + "contradictory-route", + evaluator.iop.request_model, + evaluator.iop.requested_effort, + ) + mismatch_root = self.root / "runs" / f"blind-{route_kind}-mismatch" + for name in ("input", "session", "output"): + (mismatch_root / name).mkdir(parents=True, exist_ok=True) + mismatch_blind = BlindWorkspace( + f"blind-{route_kind}-mismatch", str(mismatch_root), + str(mismatch_root / "input"), str(mismatch_root / "session"), + str(mismatch_root / "output"), "sha256:" + "e" * 64, + "sha256:" + "f" * 64, + ) + mismatch = adapter.invoke( + evaluator, + mismatch_blind, + prompt, + self.manifest.timeout, + lambda *_args: None, + ) + self.assertEqual(mismatch.harness, "failed") + self.assertEqual(mismatch.reason, "binding_mismatch") + self.assertEqual( + mismatch.effective_binding, caller_binding["value"] + ) + self.assertTrue(adapter.finalize_evidence(mismatch_blind).safe) + spec = captured["spec"] visible = "\n".join( (*spec.argv, spec.cwd, *(value for pair in spec.env for value in pair)) @@ -1399,14 +1631,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): (evidence / "lifecycle-result.json").write_text( '{"record":"result"}\n', encoding="utf-8" ) - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), - ) + lifecycle = _invocation_result() binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -1442,7 +1667,10 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.manifest.timeout, lambda *_args: None, ) - self.assertTrue(result.success) + self.assertEqual( + (result.product, result.harness, result.process), + ("succeeded", "passed", "exited"), + ) spec = captured["spec"] self.assertIsInstance(spec, InvocationSpec) assert isinstance(spec, InvocationSpec) @@ -1513,14 +1741,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): (output / "diagnostic.txt").write_text( f"{base_url}\n{secret}\n", encoding="utf-8" ) - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), - ) + lifecycle = _invocation_result() binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -1558,7 +1779,10 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.manifest.timeout, lambda *_args: None, ) - self.assertTrue(result.success) + self.assertEqual( + (result.product, result.harness, result.process), + ("succeeded", "passed", "exited"), + ) finalized = adapter.finalize_evidence(blind) self.assertEqual( (finalized.safe, finalized.reason), @@ -1723,14 +1947,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): (sensitive_dir / "runtime.txt").write_text( secret + "\n" + base_url, encoding="utf-8" ) - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), - ) + lifecycle = _invocation_result() binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -1823,14 +2040,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): if not first: self.assertTrue(workers) self.assertFalse(workers[0].is_alive()) - stream = CaptureStream("stdout", "", 0, 0, False) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (), - ) + lifecycle = _invocation_result() binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -1935,7 +2145,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): workers[0].join(5) self.assertFalse(workers[0].is_alive()) self.assertEqual(len(worker_results), 1) - self.assertTrue(worker_results[0].cleanup_complete) + self.assertTrue(worker_results[0].harness.cleanup_complete) self.assertFalse(worker_results[0].process_group_alive) receipt = json.loads( ( @@ -1953,7 +2163,10 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.manifest.timeout, lambda *_args: None, ) - self.assertTrue(retry.success) + self.assertEqual( + (retry.product, retry.harness, retry.process), + ("succeeded", "passed", "exited"), + ) self.assertTrue(adapter.finalize_evidence(blind).safe) def test_catalog_only_never_creates_ready_binding(self) -> None: @@ -2107,7 +2320,6 @@ class ConnectivityIntegrationTest(unittest.TestCase): ) def invoke(_invocation, _on_started): - stream = CaptureStream("stdout", "", 0, 0, False) metric = ParsedMetric( "model_duration", 1, @@ -2126,13 +2338,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): metric_model, "call-1", ) - lifecycle = InvocationResult( - True, "success", 0, None, True, True, True, False, - (), stream, replace(stream, stream="stderr"), "", "", None, - "sha256:" + "a" * 64, - "2026-08-11T00:00:00+00:00", - "2026-08-11T00:00:01+00:00", 1, (metric,), - ) + lifecycle = _invocation_result(metrics=(metric,)) binding = ( evaluator.iop.route_kind, evaluator.iop.route_id, @@ -2172,14 +2378,19 @@ class ConnectivityIntegrationTest(unittest.TestCase): finally: adapter.finalize_evidence(blind) - self.assertTrue(exercise(direct_manifest, "judge", "direct").success) - self.assertTrue( - exercise(preset_manifest, "judge-work", "preset-work").success + self.assertEqual( + exercise(direct_manifest, "judge", "direct").product, + "succeeded", ) - self.assertTrue( + self.assertEqual( + exercise(preset_manifest, "judge-work", "preset-work").product, + "succeeded", + ) + self.assertEqual( exercise( preset_manifest, "judge-plan", "preset-unqualified", metric_stage="" - ).success + ).product, + "succeeded", ) for stage, model in (("plan", "judge-work"), ("work", "judge-plan")): with self.subTest(stage=stage, model=model): @@ -2289,7 +2500,7 @@ class ConnectivityIntegrationTest(unittest.TestCase): ] no_model = {**environment, "BENCH_CONFIG": json.dumps({"schema_version": "1", "routes": no_model_routes})} unsupported = replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy"), iop=replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy").iop, requested_effort="max")) - observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.11", "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json") + observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.12", "--print --output-format --sandbox --model --effort stream-json") checks = ( (live_iop.build_live_adapter_registry(no_route, observer=observed)["claude"].preflight(claude).result, "route_missing"), (live_iop.build_live_adapter_registry(no_model, observer=observed)["claude"].preflight(claude).result, "model_missing"), @@ -2310,12 +2521,12 @@ class ConnectivityIntegrationTest(unittest.TestCase): (("effort_unsupported", ISSUE_RESUME_CODES["effort_unsupported"]),), ) - def test_live_invocation_rejects_missing_or_mismatched_caller_binding(self) -> None: + def test_live_invocation_uses_config_owned_codex_binding_and_rejects_mismatch(self) -> None: environment = self._live_environment() observed = lambda _runtime: live_iop._Observation( tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), - "sha256:" + "2" * 64, True, "agy 1.1.11", - "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json", + "sha256:" + "2" * 64, True, "agy 1.1.12", + "--print --output-format --sandbox --model --effort stream-json", ) registry = live_iop.build_live_adapter_registry( environment, observer=observed, binary_resolver=lambda _name: "/bin/true" @@ -2332,11 +2543,6 @@ class ConnectivityIntegrationTest(unittest.TestCase): lambda *_args: None, live_iop._DEFAULT_INVOKERS.codex, ) - codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined] - live_iop._DEFAULT_INVOKERS.claude, - live_iop._DEFAULT_INVOKERS.agy, - lambda *_args: type("Mismatch", (), {"effective_binding": None, "lifecycle": None})(), - ) with mock.patch.object(live_iop, "build_agy_invocation", return_value=object()): with self.assertRaises(live_iop.LiveIopError) as raised: agy_adapter.invoke( @@ -2344,7 +2550,65 @@ class ConnectivityIntegrationTest(unittest.TestCase): self.manifest.timeout, lambda *_args: None, ) self.assertEqual(raised.exception.issue_code, "stream_incompatible") - with mock.patch.object(live_iop, "build_codex_invocation", return_value=type("Invocation", (), {"spec": object()})()): + + failed_lifecycle = _invocation_result( + product_status="unknown", + harness_reason="nonzero_exit", + exit_code=1, + spec_digest_value="sha256:" + "9" * 64, + ) + agy_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined] + live_iop._DEFAULT_INVOKERS.claude, + lambda *_args: failed_lifecycle, + live_iop._DEFAULT_INVOKERS.codex, + ) + with ( + mock.patch.object(live_iop, "build_agy_invocation", return_value=object()), + mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]), + ): + failed = agy_adapter.invoke( + agy, object(), object(), "/tmp/control", b"task", + self.manifest.timeout, lambda *_args: None, + ) + self.assertIs(failed, failed_lifecycle) + + lifecycle = _invocation_result() + invocation = CodexInvocation( + InvocationSpec( + argv=("codex",), cwd="/tmp", env=(), + submission_mode=SUBMISSION_STDIN_ONCE, + completion_mode=COMPLETION_EXIT_AFTER_IDLE, + timeout=self.manifest.timeout, evidence_dir="/tmp", + ), + None, # type: ignore[arg-type] + lambda line: line, + ) + observed_binding = {"value": None} + + def invoke_codex(*_args): + return CodexInvocationResult(lifecycle, observed_binding["value"]) + + codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined] + live_iop._DEFAULT_INVOKERS.claude, + live_iop._DEFAULT_INVOKERS.agy, + invoke_codex, + ) + with ( + mock.patch.object(live_iop, "build_codex_invocation", return_value=invocation), + mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]), + ): + result = codex_adapter.invoke( + codex, object(), object(), "/tmp/control", b"task", + self.manifest.timeout, lambda *_args: None, + ) + self.assertIs(result, lifecycle) + + observed_binding["value"] = ( + codex.iop.route_kind, + "contradictory-route", + codex.iop.request_model, + codex.iop.requested_effort, + ) with self.assertRaises(live_iop.LiveIopError) as raised: codex_adapter.invoke( codex, object(), object(), "/tmp/control", b"task", @@ -2352,6 +2616,127 @@ class ConnectivityIntegrationTest(unittest.TestCase): ) self.assertEqual(raised.exception.issue_code, "stream_incompatible") + def test_live_agy_multiple_cells_consume_their_own_preflight_state(self) -> None: + manifest, _, _ = _write_manifest( + self.root, + [ + _cell("agy-direct", "agy", "gemini-3.6-flash", "high"), + _preset("agy-preset", "agy", "gemini-hybrid", "high"), + ], + output_id="agy-cell-state", + ) + environment = self._live_environment(manifest=manifest) + observed_models = tuple( + sorted(cell.iop.request_model for cell in manifest.matrix) + ) + ready_observation = lambda _runtime: live_iop._Observation( + observed_models, + "sha256:" + "2" * 64, + True, + "agy 1.1.12", + "--print --output-format --sandbox --model --effort stream-json", + ) + adapter = live_iop.build_live_adapter_registry( + environment, + observer=ready_observation, + binary_resolver=lambda _name: "/bin/true", + )["agy"] + cells = {cell.id: cell for cell in manifest.matrix} + for cell in manifest.matrix: + self.assertEqual(adapter.preflight(cell).result.status, "ready") + + preflights = dict(adapter._agy_preflights) # type: ignore[attr-defined] + self.assertEqual(set(preflights), set(cells)) + self.assertIsNot(preflights["agy-direct"], preflights["agy-preset"]) + self.assertEqual( + preflights["agy-direct"].runtime.observation.cell_id, + "agy-direct", + ) + self.assertEqual( + preflights["agy-preset"].runtime.observation.cell_id, + "agy-preset", + ) + + lifecycle = _invocation_result( + started_at="2026-08-12T00:00:00+00:00", + ended_at="2026-08-12T00:00:01+00:00", + ) + built: list[tuple[str, object]] = [] + invoked: list[tuple[str, object]] = [] + + def build(cell, _prepared, _task, _timeout, preflight): + built.append((cell.id, preflight)) + return object() + + def invoke(_spec, parser, preflight, _on_started): + invoked.append((parser._cell.id, preflight)) + return lifecycle + + def observed_result(parser, capability, _result): + cell = parser._cell + admitted = adapter._admitted_bindings[cell.id] # type: ignore[attr-defined] + caller_capability = CallerCapability( + "agy", capability.route_kinds, capability.efforts + ) + return make_result(cell, caller_capability, admitted) + + adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined] + live_iop._DEFAULT_INVOKERS.claude, + invoke, + live_iop._DEFAULT_INVOKERS.codex, + ) + with ( + mock.patch.object(live_iop, "build_agy_invocation", side_effect=build), + mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]), + mock.patch.object( + live_iop.AgyEventParser, + "observed_result", + new=observed_result, + ), + ): + for cell_id in ("agy-preset", "agy-direct"): + cell = cells[cell_id] + result = adapter.invoke( + cell, + object(), + object(), + "/tmp/control", + b"task", + manifest.timeout, + lambda *_args: None, + ) + self.assertIs(result, lifecycle) + + self.assertEqual( + built, + [ + ("agy-preset", preflights["agy-preset"]), + ("agy-direct", preflights["agy-direct"]), + ], + ) + self.assertEqual(invoked, built) + + adapter._observer = lambda _runtime: live_iop._Observation( # type: ignore[attr-defined] + observed_models, + "sha256:" + "2" * 64, + False, + ) + self.assertEqual( + adapter.preflight(cells["agy-direct"]).result.status, + "implementation_gap", + ) + with self.assertRaises(live_iop.LiveIopError) as raised: + adapter.invoke( + cells["agy-direct"], + object(), + object(), + "/tmp/control", + b"task", + manifest.timeout, + lambda *_args: None, + ) + self.assertEqual(raised.exception.issue_code, "stream_incompatible") + def test_cli_missing_live_input_fails_closed_without_secret_or_attempt(self) -> None: sentinel = "missing-input-token" stdout = io.StringIO() diff --git a/scripts/agent_benchmark/lifecycle.py b/scripts/agent_benchmark/lifecycle.py index 33d880c5..ef6f44a8 100644 --- a/scripts/agent_benchmark/lifecycle.py +++ b/scripts/agent_benchmark/lifecycle.py @@ -18,6 +18,7 @@ from __future__ import annotations import ctypes import datetime +import errno import hashlib import hmac import json @@ -28,6 +29,7 @@ import secrets import shutil import signal import socket +import stat import subprocess import sys import tempfile @@ -36,7 +38,7 @@ import time from dataclasses import dataclass from decimal import Decimal, InvalidOperation from pathlib import Path -from typing import Any, Callable, Optional +from typing import Any, Callable, Mapping, Optional from scripts.agent_benchmark.manifest import Timeout @@ -59,6 +61,7 @@ EVENT_IDLE = "idle" EVENT_QUIET = "quiet" EVENT_EXITED = "exited" EVENT_TERMINAL = "terminal" +EVENT_CALLER_TERMINAL = "caller_terminal" PARSER_TERMINAL_KINDS = (EVENT_FINISH, EVENT_IDLE) METRIC_PREFIX = "metric:" @@ -120,6 +123,7 @@ REASON_CONTROLLER_LOST = "controller_lost" REASON_RECOVERED_STOP = "recovered_stop" REASON_CLEANUP_FAILED = "cleanup_failed" REASON_SUPERVISOR_ERROR = "supervisor_error" +REASON_INTERRUPTED = "interrupted" TERMINAL_REASONS = ( REASON_SUCCESS, REASON_START_CALLBACK_FAILED, @@ -137,6 +141,43 @@ TERMINAL_REASONS = ( REASON_RECOVERED_STOP, REASON_CLEANUP_FAILED, REASON_SUPERVISOR_ERROR, + REASON_INTERRUPTED, +) + +CALLER_STATUS_SUCCEEDED = "succeeded" +CALLER_STATUS_FAILED = "failed" +CALLER_STATUSES = (CALLER_STATUS_SUCCEEDED, CALLER_STATUS_FAILED) +CALLER_REASON_SUCCESS = "caller_success" +CALLER_REASON_ERROR = "caller_error" +CALLER_REASONS = (CALLER_REASON_SUCCESS, CALLER_REASON_ERROR) + +PRODUCT_STATUS_SUCCEEDED = "succeeded" +PRODUCT_STATUS_FAILED = "failed" +PRODUCT_STATUS_UNKNOWN = "unknown" +PRODUCT_STATUSES = ( + PRODUCT_STATUS_SUCCEEDED, PRODUCT_STATUS_FAILED, PRODUCT_STATUS_UNKNOWN, +) +PRODUCT_REASON_UNAVAILABLE = "unavailable" +PRODUCT_REASONS = ( + CALLER_REASON_SUCCESS, CALLER_REASON_ERROR, PRODUCT_REASON_UNAVAILABLE, +) + +HARNESS_STATUS_PASSED = "passed" +HARNESS_STATUS_FAILED = "failed" +HARNESS_STATUSES = (HARNESS_STATUS_PASSED, HARNESS_STATUS_FAILED) +HARNESS_REASONS = TERMINAL_REASONS + +PROCESS_STATUS_EXITED = "exited" +PROCESS_STATUS_SIGNALLED = "signalled" +PROCESS_STATUS_TIMED_OUT = "timed_out" +PROCESS_STATUS_CANCELLED = "cancelled" +PROCESS_STATUS_NOT_STARTED = "not_started" +PROCESS_STATUSES = ( + PROCESS_STATUS_EXITED, + PROCESS_STATUS_SIGNALLED, + PROCESS_STATUS_TIMED_OUT, + PROCESS_STATUS_CANCELLED, + PROCESS_STATUS_NOT_STARTED, ) FAULT_NONE = "" @@ -148,6 +189,7 @@ DEFAULT_ENV_ALLOWLIST = ( "USER", "LOGNAME", "SHELL", "PWD", "PYTHONPATH", "PYTHONHASHSEED", "NO_COLOR", "CI", ) +TLS_CA_ENV_KEYS = ("SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS") ENV_KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$") METRIC_KIND_RE = re.compile(r"^metric:[a-z0-9][a-z0-9_.+-]{0,63}$") @@ -162,7 +204,7 @@ SUPERVISOR_ERR_FILENAME = "supervisor.err" REDACTED = "[redacted]" RECEIPT_VERSION = 1 -JOURNAL_VERSION = 1 +JOURNAL_VERSION = 2 MAX_TASK_PAYLOAD_BYTES = 1 << 20 MAX_CAPTURE_BYTES_LIMIT = 1 << 24 @@ -276,6 +318,36 @@ class ParsedMetric: overlap: bool = False +@dataclass(frozen=True) +class CallerEvent: + """One closed caller lifecycle observation.""" + + kind: str + + def __post_init__(self) -> None: + if self.kind not in PARSER_TERMINAL_KINDS: + raise LifecycleValidationError("caller event kind is invalid") + + +@dataclass(frozen=True) +class CallerTerminal: + """The caller-declared product outcome, independent of its process exit.""" + + status: str + reason: str + + def __post_init__(self) -> None: + expected = { + CALLER_STATUS_SUCCEEDED: CALLER_REASON_SUCCESS, + CALLER_STATUS_FAILED: CALLER_REASON_ERROR, + } + if self.status not in CALLER_STATUSES or self.reason != expected[self.status]: + raise LifecycleValidationError("caller terminal is invalid") + + +CallerObservation = CallerEvent | CallerTerminal | ParsedMetric + + @dataclass(frozen=True) class CaptureStream: stream: str @@ -307,16 +379,73 @@ class TerminalOutcome: @dataclass(frozen=True) -class InvocationResult: - """Terminal projection of one invocation, including typed observations.""" +class ProductOutcome: + status: str + reason: str - success: bool - terminal_reason: str + def __post_init__(self) -> None: + expected = { + PRODUCT_STATUS_SUCCEEDED: CALLER_REASON_SUCCESS, + PRODUCT_STATUS_FAILED: CALLER_REASON_ERROR, + PRODUCT_STATUS_UNKNOWN: PRODUCT_REASON_UNAVAILABLE, + } + if self.status not in PRODUCT_STATUSES or self.reason != expected[self.status]: + raise LifecycleValidationError("product outcome is invalid") + + +@dataclass(frozen=True) +class HarnessOutcome: + status: str + reason: str + ordered_terminal: bool + cleanup_complete: bool + + def __post_init__(self) -> None: + if ( + self.status not in HARNESS_STATUSES + or self.reason not in HARNESS_REASONS + or not isinstance(self.ordered_terminal, bool) + or not isinstance(self.cleanup_complete, bool) + or (self.status == HARNESS_STATUS_PASSED) != (self.reason == REASON_SUCCESS) + or (not self.cleanup_complete) != (self.reason == REASON_CLEANUP_FAILED) + or (self.status == HARNESS_STATUS_PASSED and not self.ordered_terminal) + ): + raise LifecycleValidationError("harness outcome is invalid") + + +@dataclass(frozen=True) +class ProcessOutcome: + status: str exit_code: Optional[int] signal: Optional[int] + + def __post_init__(self) -> None: + if self.status not in PROCESS_STATUSES: + raise LifecycleValidationError("process outcome is invalid") + if self.exit_code is not None and ( + not isinstance(self.exit_code, int) or isinstance(self.exit_code, bool) + ): + raise LifecycleValidationError("process exit code is invalid") + if self.signal is not None and ( + not isinstance(self.signal, int) or isinstance(self.signal, bool) + ): + raise LifecycleValidationError("process signal is invalid") + if self.status == PROCESS_STATUS_SIGNALLED and self.signal is None: + raise LifecycleValidationError("signalled process requires a signal") + if self.status in (PROCESS_STATUS_EXITED, PROCESS_STATUS_NOT_STARTED) and self.signal is not None: + raise LifecycleValidationError("process signal contradicts its status") + if self.status == PROCESS_STATUS_NOT_STARTED and self.exit_code is not None: + raise LifecycleValidationError("not-started process has an exit code") + + +@dataclass(frozen=True) +class InvocationResult: + """Independent product, harness, and process outcomes for one invocation.""" + + product: ProductOutcome + harness: HarnessOutcome + process: ProcessOutcome submitted: bool - finish_then_idle_then_quiet: bool - cleanup_complete: bool process_group_alive: bool events: tuple[LifecycleEvent, ...] stdout: CaptureStream @@ -681,6 +810,18 @@ def env_pairs(mapping: dict[str, str]) -> tuple[tuple[str, str], ...]: return tuple(sorted((str(k), str(v)) for k, v in mapping.items())) +def inherited_tls_ca_environment( + environment: Mapping[str, str] | None = None, +) -> tuple[tuple[str, str], ...]: + """Freeze only the two standard public CA bundle settings for a child.""" + source = os.environ if environment is None else environment + return tuple( + (key, value) + for key in TLS_CA_ENV_KEYS + if isinstance((value := source.get(key)), str) and value + ) + + # --------------------------------------------------------------------------- # Frame transport # --------------------------------------------------------------------------- @@ -759,11 +900,47 @@ class _Supervisor: socket_path = self.control_dir / SOCKET_FILENAME self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) previous_umask = os.umask(0o177) + original_dir_fd = -1 + control_dir_fd = -1 try: - self.sock.bind(str(socket_path)) + # Some shared/container filesystems reject bind(2) when the socket + # pathname traverses the short attempt symlink, even though the + # resolved directory supports Unix sockets. Resolve the directory + # once with fchdir and bind the basename; the published locator + # remains the authenticated short absolute alias. + directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) + original_dir_fd = os.open(".", directory_flags) + control_dir_fd = os.open(self.control_dir, directory_flags) + os.fchdir(control_dir_fd) + self.sock.bind(SOCKET_FILENAME) + try: + os.chmod(SOCKET_FILENAME, 0o600) + except OSError as exc: + # A small class of shared filesystems permits Unix sockets but + # rejects chmod on the socket inode. The containing directory + # is still an exclusive 0700 security boundary. + directory = os.stat(".") + unsupported = {errno.EINVAL} + if hasattr(errno, "ENOTSUP"): + unsupported.add(errno.ENOTSUP) + if ( + exc.errno not in unsupported + or stat.S_IMODE(directory.st_mode) != 0o700 + or directory.st_uid != os.geteuid() + ): + raise + else: + socket_mode = stat.S_IMODE(os.lstat(SOCKET_FILENAME).st_mode) + if socket_mode & 0o077: + raise PermissionError("control socket permissions are too broad") finally: + if original_dir_fd >= 0: + os.fchdir(original_dir_fd) + if control_dir_fd >= 0: + os.close(control_dir_fd) + if original_dir_fd >= 0: + os.close(original_dir_fd) os.umask(previous_umask) - os.chmod(socket_path, 0o600) self.sock.listen(4) locator = { "supervisor_pid": os.getpid(), @@ -914,15 +1091,24 @@ class _Supervisor: if self.child is not None: grace = int(self.spec.get("cleanup_grace_seconds", 5)) self._terminate_group(grace) + io_complete = self._join_io_threads() + descendants_reaped = self._reap_owned_descendants() + if self.child is not None: + # The exit watcher publishes the controller's exited frame. + # Join it before freezing the receipt so both durable views + # observe the same authoritative child return code. + self.child.poll() code = self.child.returncode if code is not None: exit_code = code if code >= 0 else None signal_num = -code if code < 0 else None group_alive = _group_alive(self.pgid) - io_complete = self._join_io_threads() - descendants_reaped = self._reap_owned_descendants() outcome = { - "reason": reason, + "reason": ( + reason + if not group_alive and io_complete and descendants_reaped + else REASON_CLEANUP_FAILED + ), "exit_code": exit_code, "signal": signal_num, "caller_launched": self.child is not None, @@ -1259,11 +1445,13 @@ class _Invocation: self.submitted = False self.finish_at: Optional[float] = None self.idle_at: Optional[float] = None + self.caller_terminal: Optional[CallerTerminal] = None self.last_output_at: Optional[float] = None self.quiet = False self.exited = False self.exit_code: Optional[int] = None self.signal: Optional[int] = None + self.process_status_hint: Optional[str] = None self.run_deadline = 0.0 self.control_dir: Optional[Path] = None self.owns_control_dir = False @@ -1502,7 +1690,7 @@ class _Invocation: def _apply_parsed( self, parsed: Any, stream: str, frame: dict[str, Any], redacted: str ) -> None: - """Apply one parser result: a terminal string, observations, or both.""" + """Apply one closed caller observation or a bounded tuple of them.""" if parsed is None: return items = parsed if isinstance(parsed, tuple) else (parsed,) @@ -1523,27 +1711,29 @@ class _Invocation: if isinstance(item, ParsedMetric): self._record_metric(item, stream, frame) return - if not isinstance(item, str) or not item: - self.reason = self.reason or REASON_MALFORMED_EVENT + if isinstance(item, CallerEvent): + self._apply_terminal_evidence(item.kind, stream, frame, redacted) return - if item.startswith(METRIC_PREFIX): - self._record_metric_label(item, stream, frame, redacted) + if isinstance(item, CallerTerminal): + self._apply_caller_terminal(item, stream, frame) return - if item not in PARSER_TERMINAL_KINDS: - self.reason = self.reason or REASON_MALFORMED_EVENT - return - self._apply_terminal_evidence(item, stream, frame, redacted) + self.reason = self.reason or REASON_MALFORMED_EVENT - def _record_metric_label( - self, parsed: str, stream: str, frame: dict[str, Any], redacted: str + def _apply_caller_terminal( + self, terminal: CallerTerminal, stream: str, frame: dict[str, Any] ) -> None: - """Record one untyped caller metric label with no numeric payload.""" - metric_kind = self._validate_metric_kind(parsed) - if metric_kind is None: - self.reason = self.reason or REASON_MALFORMED_EVENT - elif self.metric_events < MAX_METRIC_EVENTS: - self.metric_events += 1 - self._add_event(metric_kind, SOURCE_CALLER_OUTPUT, stream, frame, redacted) + if self.caller_terminal is not None: + self.reason = self.reason or REASON_DUPLICATE_EVENT + return + self.caller_terminal = terminal + self._add_event( + EVENT_CALLER_TERMINAL, + SOURCE_CALLER_OUTPUT, + stream, + frame, + f"status={terminal.status} reason={terminal.reason}", + safe=True, + ) def _record_metric( self, metric: ParsedMetric, stream: str, frame: dict[str, Any] @@ -1613,9 +1803,11 @@ class _Invocation: def _check_deadlines(self) -> None: now = time.monotonic() if _is_cancelled(self.cancellation): + self.process_status_hint = PROCESS_STATUS_CANCELLED self.reason = REASON_CANCELLED return if now >= self.run_deadline: + self.process_status_hint = PROCESS_STATUS_TIMED_OUT self.reason = REASON_TIMED_OUT return if ( @@ -1632,6 +1824,11 @@ class _Invocation: and len(self.stream_eof) == len(self.captures) and not self.quiet ): + if self.caller_terminal is not None and self.idle_at is not None: + # A caller-declared product failure commonly exits non-zero. + # Preserve that code on the process axis while still allowing + # the ordered terminal stream to reach quiet. + return if self.exit_code != 0: self.reason = REASON_NONZERO_EXIT elif self.idle_at is None: @@ -1646,10 +1843,18 @@ class _Invocation: self.quiet = True self._add_event(EVENT_QUIET, SOURCE_HARNESS, "", {"ns": time.monotonic_ns()}, "") if self.spec.completion_mode == COMPLETION_STOP_AFTER_IDLE: - self.reason = REASON_SUCCESS + self.reason = ( + REASON_SUCCESS + if self.caller_terminal is not None + else REASON_MISSING_IDLE + ) return if self.exited: - self.reason = REASON_SUCCESS if self.exit_code == 0 else REASON_NONZERO_EXIT + self.reason = ( + REASON_SUCCESS + if self.caller_terminal is not None + else (REASON_NONZERO_EXIT if self.exit_code != 0 else REASON_MISSING_IDLE) + ) # -- terminal handshake ------------------------------------------------ @@ -1758,16 +1963,48 @@ class _Invocation: cleanup_complete = bool(outcome.get("cleanup_complete")) group_alive = bool(outcome.get("process_group_alive")) ordered = bool(self.finish_at is not None and self.idle_at is not None and self.quiet) - success = reason == REASON_SUCCESS and cleanup_complete and not group_alive and ordered + harness_passed = ( + reason == REASON_SUCCESS + and cleanup_complete + and not group_alive + and ordered + and self.caller_terminal is not None + ) + product_evidence_valid = ( + self.caller_terminal is not None + and self.finish_at is not None + and self.idle_at is not None + and reason in {REASON_SUCCESS, REASON_CLEANUP_FAILED} + ) + if not product_evidence_valid: + product = ProductOutcome(PRODUCT_STATUS_UNKNOWN, PRODUCT_REASON_UNAVAILABLE) + elif self.caller_terminal.status == CALLER_STATUS_SUCCEEDED: + product = ProductOutcome(PRODUCT_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS) + else: + product = ProductOutcome(PRODUCT_STATUS_FAILED, CALLER_REASON_ERROR) + exit_code = self.exit_code if self.exit_code is not None else outcome.get("exit_code") + process_signal = self.signal if self.signal is not None else outcome.get("signal") + if self.process_status_hint is not None: + process_status = self.process_status_hint + elif not self.submitted: + process_status = PROCESS_STATUS_NOT_STARTED + exit_code = None + process_signal = None + elif process_signal is not None: + process_status = PROCESS_STATUS_SIGNALLED + else: + process_status = PROCESS_STATUS_EXITED evidence_dir = Path(self.spec.evidence_dir) result = InvocationResult( - success=success, - terminal_reason=reason, - exit_code=self.exit_code if self.exit_code is not None else outcome.get("exit_code"), - signal=self.signal if self.signal is not None else outcome.get("signal"), + product=product, + harness=HarnessOutcome( + HARNESS_STATUS_PASSED if harness_passed else HARNESS_STATUS_FAILED, + REASON_SUCCESS if harness_passed else reason, + ordered, + cleanup_complete, + ), + process=ProcessOutcome(process_status, exit_code, process_signal), submitted=self.submitted, - finish_then_idle_then_quiet=ordered, - cleanup_complete=cleanup_complete, process_group_alive=group_alive, events=tuple(self.events), stdout=self.captures["stdout"].freeze(), @@ -1844,13 +2081,10 @@ def _event_record(event: LifecycleEvent) -> dict[str, Any]: def _result_record(result: InvocationResult, spec: InvocationSpec) -> dict[str, Any]: return { "record": "result", - "success": result.success, - "terminal_reason": result.terminal_reason, - "exit_code": result.exit_code, - "signal": result.signal, + "product": _product_record(result.product), + "harness": _harness_record(result.harness), + "process": _process_record(result.process), "submitted": result.submitted, - "finish_then_idle_then_quiet": result.finish_then_idle_then_quiet, - "cleanup_complete": result.cleanup_complete, "process_group_alive": result.process_group_alive, "submission_mode": spec.submission_mode, "completion_mode": spec.completion_mode, @@ -1865,6 +2099,27 @@ def _result_record(result: InvocationResult, spec: InvocationSpec) -> dict[str, } +def _product_record(outcome: ProductOutcome) -> dict[str, Any]: + return {"status": outcome.status, "reason": outcome.reason} + + +def _harness_record(outcome: HarnessOutcome) -> dict[str, Any]: + return { + "status": outcome.status, + "reason": outcome.reason, + "ordered_terminal": outcome.ordered_terminal, + "cleanup_complete": outcome.cleanup_complete, + } + + +def _process_record(outcome: ProcessOutcome) -> dict[str, Any]: + return { + "status": outcome.status, + "exit_code": outcome.exit_code, + "signal": outcome.signal, + } + + def _capture_record(capture: CaptureStream) -> dict[str, Any]: return { "stream": capture.stream, @@ -1887,9 +2142,9 @@ def _publish_evidence(result: InvocationResult, spec: InvocationSpec) -> None: } terminal = { "record": "terminal", - "terminal_reason": result.terminal_reason, - "success": result.success, - "cleanup_complete": result.cleanup_complete, + "product": _product_record(result.product), + "harness": _harness_record(result.harness), + "process": _process_record(result.process), "process_group_alive": result.process_group_alive, "ended_at": result.ended_at, } @@ -2045,9 +2300,9 @@ def run_invocation( Args: spec: Frozen invocation specification. parse_event: Adapter parser mapping ``(stream, line)`` to ``None``, - ``"finish"``, ``"idle"``, a ``"metric:"`` label, a typed - ``ParsedMetric``, or a bounded tuple of those items when one caller - line carries observations and terminal evidence together. + a typed ``CallerEvent``, ``CallerTerminal``, ``ParsedMetric``, or a + bounded tuple of those items when one caller line carries multiple + observations. on_started: Required durable locator commit callback. redact: Optional adapter redactor for exact secret values. cancellation: Optional cancellation token, event or predicate. diff --git a/scripts/agent_benchmark/lifecycle_test.py b/scripts/agent_benchmark/lifecycle_test.py index 9b1f74c1..63b8d6b1 100644 --- a/scripts/agent_benchmark/lifecycle_test.py +++ b/scripts/agent_benchmark/lifecycle_test.py @@ -16,6 +16,10 @@ from dataclasses import replace from pathlib import Path from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, + CALLER_STATUS_FAILED, + CALLER_STATUS_SUCCEEDED, COMPLETION_EXIT_AFTER_IDLE, COMPLETION_STOP_AFTER_IDLE, REASON_CANCELLED, @@ -26,6 +30,7 @@ from scripts.agent_benchmark.lifecycle import ( REASON_MISSING_IDLE, REASON_NONZERO_EXIT, REASON_OUT_OF_ORDER_EVENT, + REASON_PARSER_ERROR, REASON_READER_ERROR, REASON_RECOVERED_STOP, REASON_START_CALLBACK_FAILED, @@ -34,6 +39,9 @@ from scripts.agent_benchmark.lifecycle import ( SUBMISSION_ARGV_TASK, SUBMISSION_STDIN_ONCE, CancellationToken, + CallerEvent, + CallerTerminal, + HarnessOutcome, InvocationSpec, LifecycleError, LifecycleRecoveryError, @@ -51,8 +59,14 @@ from scripts.agent_benchmark.lifecycle import ( from scripts.agent_benchmark.manifest import Timeout -def _events(_: str, line: str) -> str | None: - return {"FINISH": "finish", "IDLE": "idle"}.get(line) +def _events(_: str, line: str): + return { + "FINISH": ( + CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS), + CallerEvent("finish"), + ), + "IDLE": CallerEvent("idle"), + }.get(line) class LifecycleTest(unittest.TestCase): @@ -162,12 +176,13 @@ class LifecycleTest(unittest.TestCase): def test_exit_after_idle_publishes_ordered_atomic_evidence(self) -> None: result = self._run(self._spec("print('FINISH'); print('IDLE')")) - self.assertTrue(result.success) - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.product.status == "succeeded") + self.assertTrue(result.harness.cleanup_complete) self.assertFalse(result.process_group_alive) - self.assertTrue(result.finish_then_idle_then_quiet) + self.assertTrue(result.harness.ordered_terminal) self.assertEqual([event.kind for event in result.events], [ - "submitted", "first_output", "finish", "idle", "exited", "quiet", + "submitted", "first_output", "caller_terminal", "finish", "idle", + "exited", "quiet", ]) self.assertTrue(Path(result.journal_path).is_file()) published = json.loads(Path(result.result_path).read_text(encoding="utf-8")) @@ -182,7 +197,7 @@ class LifecycleTest(unittest.TestCase): payload=b"single task payload", )) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") self.assertIn("single task payload", result.stdout.text) self.assertEqual(sum(event.kind == "submitted" for event in result.events), 1) @@ -196,10 +211,10 @@ class LifecycleTest(unittest.TestCase): )) self.assertLess(time.monotonic() - started, 6) - self.assertEqual(result.terminal_reason, REASON_TIMED_OUT) + self.assertEqual(result.harness.reason, REASON_TIMED_OUT) self.assertFalse(result.submitted) self.assertEqual(sum(event.kind == "submitted" for event in result.events), 0) - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.harness.cleanup_complete) self.assertFalse(result.process_group_alive) def test_unterminated_final_idle_is_consumed_before_terminal(self) -> None: @@ -215,7 +230,7 @@ class LifecycleTest(unittest.TestCase): evidence_dir=str(evidence), )) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") kinds = [event.kind for event in result.events] self.assertLess(kinds.index("finish"), kinds.index("idle")) self.assertLess(kinds.index("idle"), kinds.index("quiet")) @@ -233,15 +248,15 @@ class LifecycleTest(unittest.TestCase): completion_mode=COMPLETION_STOP_AFTER_IDLE, )) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") self.assertLess(time.monotonic() - started, 8) - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.harness.cleanup_complete) self.assertFalse(result.process_group_alive) def test_caller_output_cannot_synthesize_submission(self) -> None: result = self._run(self._spec("print('submitted'); print('FINISH'); print('IDLE')")) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") self.assertEqual(sum(event.kind == "submitted" for event in result.events), 1) self.assertEqual(result.events[0].source, "harness") @@ -257,9 +272,9 @@ class LifecycleTest(unittest.TestCase): evidence.mkdir() spec = replace(self._spec(source), evidence_dir=str(evidence)) result = self._run(spec) - self.assertFalse(result.success) - self.assertEqual(result.terminal_reason, reason) - self.assertTrue(result.cleanup_complete) + self.assertFalse(result.product.status == "succeeded") + self.assertEqual(result.harness.reason, reason) + self.assertTrue(result.harness.cleanup_complete) def test_malformed_parser_and_nonzero_exit_fail_closed(self) -> None: malformed = run_invocation( @@ -267,8 +282,8 @@ class LifecycleTest(unittest.TestCase): parse_event=lambda _stream, _line: "submitted", on_started=lambda _: None, ) - self.assertEqual(malformed.terminal_reason, REASON_MALFORMED_EVENT) - self.assertTrue(malformed.cleanup_complete) + self.assertEqual(malformed.harness.reason, REASON_MALFORMED_EVENT) + self.assertTrue(malformed.harness.cleanup_complete) evidence = self.root / "nonzero" evidence.mkdir() @@ -276,13 +291,111 @@ class LifecycleTest(unittest.TestCase): self._spec("print('FINISH'); print('IDLE'); raise SystemExit(7)"), evidence_dir=str(evidence), )) - self.assertEqual(failed.terminal_reason, REASON_NONZERO_EXIT) - self.assertTrue(failed.cleanup_complete) + self.assertEqual(failed.product.status, "succeeded") + self.assertEqual(failed.harness.status, "passed") + self.assertEqual(failed.process.exit_code, 7) + self.assertTrue(failed.harness.cleanup_complete) + + def test_product_error_can_have_clean_harness_and_process(self) -> None: + def parse_error(_stream: str, line: str): + return { + "ERROR": ( + CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR), + CallerEvent("finish"), + ), + "IDLE": CallerEvent("idle"), + }.get(line) + + result = run_invocation( + self._spec("print('ERROR'); print('IDLE')"), + parse_event=parse_error, + on_started=lambda _: None, + ) + + self.assertEqual((result.product.status, result.product.reason), ( + "failed", CALLER_REASON_ERROR, + )) + self.assertEqual((result.harness.status, result.harness.reason), ( + "passed", "success", + )) + self.assertEqual((result.process.status, result.process.exit_code), ( + "exited", 0, + )) + + with self.assertRaises(LifecycleValidationError): + HarnessOutcome("failed", "success", True, True) + with self.assertRaises(LifecycleValidationError): + HarnessOutcome("passed", "success", False, True) + + def test_parser_failure_leaves_product_unknown(self) -> None: + def broken_parser(_stream: str, _line: str): + raise ValueError("synthetic parser failure") + + result = run_invocation( + self._spec("print('BROKEN')"), + parse_event=broken_parser, + on_started=lambda _: None, + ) + + self.assertEqual((result.product.status, result.product.reason), ( + "unknown", "unavailable", + )) + self.assertEqual((result.harness.status, result.harness.reason), ( + "failed", REASON_PARSER_ERROR, + )) + + def test_timeout_cancel_and_cleanup_do_not_fabricate_product(self) -> None: + timeout = self._run(self._spec("import time; time.sleep(30)", run_seconds=1)) + + token = CancellationToken() + timer = threading.Timer(0.2, token.cancel) + timer.start() + try: + cancel_root = self.root / "independent-cancel" + cancel_root.mkdir() + cancelled = self._run( + replace( + self._spec("import time; time.sleep(30)"), + evidence_dir=str(cancel_root), + ), + cancellation=token, + ) + finally: + timer.cancel() + + cleanup_root = self.root / "independent-cleanup" + cleanup_root.mkdir() + cleanup_control = self.root / "independent-cleanup-control" + + def collide_receipt(locator: SupervisorLocator) -> None: + (Path(locator.control_dir) / "cleanup-receipt.json").write_bytes( + b"collision" + ) + + cleanup = run_invocation( + replace( + self._spec("print('FINISH'); print('IDLE')"), + evidence_dir=str(cleanup_root), + control_dir=str(cleanup_control), + ), + parse_event=_events, + on_started=collide_receipt, + ) + + for result, product_status, process_status, reason in ( + (timeout, "unknown", "timed_out", REASON_TIMED_OUT), + (cancelled, "unknown", "cancelled", REASON_CANCELLED), + (cleanup, "succeeded", "exited", REASON_CLEANUP_FAILED), + ): + with self.subTest(reason=reason): + self.assertEqual(result.product.status, product_status) + self.assertEqual(result.harness.reason, reason) + self.assertEqual(result.process.status, process_status) def test_timeout_cancel_and_reader_error_all_cleanup(self) -> None: timeout = self._run(self._spec("import time; time.sleep(30)", run_seconds=1)) - self.assertEqual(timeout.terminal_reason, REASON_TIMED_OUT) - self.assertTrue(timeout.cleanup_complete) + self.assertEqual(timeout.harness.reason, REASON_TIMED_OUT) + self.assertTrue(timeout.harness.cleanup_complete) token = CancellationToken() timer = threading.Timer(0.2, token.cancel) @@ -296,8 +409,8 @@ class LifecycleTest(unittest.TestCase): ) finally: timer.cancel() - self.assertEqual(cancelled.terminal_reason, REASON_CANCELLED) - self.assertTrue(cancelled.cleanup_complete) + self.assertEqual(cancelled.harness.reason, REASON_CANCELLED) + self.assertTrue(cancelled.harness.cleanup_complete) evidence = self.root / "reader" evidence.mkdir() @@ -305,8 +418,8 @@ class LifecycleTest(unittest.TestCase): self._spec("print('FINISH'); print('IDLE')", fault_injection="reader_error"), evidence_dir=str(evidence), )) - self.assertEqual(reader_error.terminal_reason, REASON_READER_ERROR) - self.assertTrue(reader_error.cleanup_complete) + self.assertEqual(reader_error.harness.reason, REASON_READER_ERROR) + self.assertTrue(reader_error.harness.cleanup_complete) def test_redaction_and_capture_bounds_apply_before_publication(self) -> None: secret = "EXACT_SECRET_123456789" @@ -409,22 +522,22 @@ class LifecycleTest(unittest.TestCase): parse_event=_events, on_started=collide_receipt, ) - self.assertEqual(receipt_result.terminal_reason, REASON_CLEANUP_FAILED) - self.assertFalse(receipt_result.success) - self.assertFalse(receipt_result.cleanup_complete) + self.assertEqual(receipt_result.harness.reason, REASON_CLEANUP_FAILED) + self.assertEqual(receipt_result.product.status, "succeeded") + self.assertFalse(receipt_result.harness.cleanup_complete) self.assertFalse(receipt_result.process_group_alive) self.assertEqual( (receipt_control / "cleanup-receipt.json").read_bytes(), receipt_sentinel ) published = json.loads(Path(receipt_result.result_path).read_text(encoding="utf-8")) - self.assertEqual(published["terminal_reason"], REASON_CLEANUP_FAILED) + self.assertEqual(published["harness"]["reason"], REASON_CLEANUP_FAILED) journal = [ json.loads(line) for line in Path(receipt_result.journal_path).read_text(encoding="utf-8").splitlines() ] terminals = [record for record in journal if record.get("record") == "terminal"] self.assertEqual(len(terminals), 1) - self.assertEqual(terminals[0]["terminal_reason"], REASON_CLEANUP_FAILED) + self.assertEqual(terminals[0]["harness"]["reason"], REASON_CLEANUP_FAILED) def test_metric_kind_cannot_leak_secret(self) -> None: cases = ( @@ -452,7 +565,7 @@ class LifecycleTest(unittest.TestCase): on_started=lambda _: None, redact=redactor, ) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) + self.assertEqual(result.harness.reason, REASON_MALFORMED_EVENT) self.assertFalse(any(event.kind == f"metric:{metric_name}" for event in result.events)) persisted = "\n".join( path.read_text(encoding="utf-8") @@ -476,8 +589,9 @@ class LifecycleTest(unittest.TestCase): parse_event=parse_valid, on_started=lambda _: None, ) - self.assertTrue(valid.success) - self.assertIn("metric:duration_ms", [event.kind for event in valid.events]) + self.assertEqual(valid.product.status, "unknown") + self.assertEqual(valid.harness.reason, REASON_MALFORMED_EVENT) + self.assertNotIn("metric:duration_ms", [event.kind for event in valid.events]) def test_first_output_is_recorded_once_before_terminal_evidence(self) -> None: source = ( @@ -486,7 +600,7 @@ class LifecycleTest(unittest.TestCase): ) result = self._run(self._spec(source)) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") kinds = [event.kind for event in result.events] self.assertEqual(kinds.count("first_output"), 1) self.assertLess(kinds.index("submitted"), kinds.index("first_output")) @@ -499,7 +613,7 @@ class LifecycleTest(unittest.TestCase): def test_silent_caller_records_no_first_output(self) -> None: result = self._run(self._spec("import time; time.sleep(30)", run_seconds=1)) - self.assertEqual(result.terminal_reason, REASON_TIMED_OUT) + self.assertEqual(result.harness.reason, REASON_TIMED_OUT) self.assertNotIn("first_output", [event.kind for event in result.events]) def test_typed_observations_are_published_with_terminal_evidence(self) -> None: @@ -516,7 +630,7 @@ class LifecycleTest(unittest.TestCase): parse_event=parse_metric, on_started=lambda _: None, ) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") self.assertEqual( [(metric.name, metric.value) for metric in result.metrics], [("total_duration", 12_500_000), ("input_tokens", 11)], @@ -554,7 +668,7 @@ class LifecycleTest(unittest.TestCase): ), on_started=lambda _: None, ) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) + self.assertEqual(result.harness.reason, REASON_MALFORMED_EVENT) self.assertEqual(result.metrics, ()) def test_metric_event_overflow_is_malformed_after_retaining_the_bound(self) -> None: @@ -569,7 +683,7 @@ class LifecycleTest(unittest.TestCase): parse_event=parse_metric, on_started=lambda _: None, ) - self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT) + self.assertEqual(result.harness.reason, REASON_MALFORMED_EVENT) self.assertEqual(len(result.metrics), MAX_METRIC_EVENTS) def test_adapter_redactor_cannot_corrupt_a_validated_observation(self) -> None: @@ -588,7 +702,7 @@ class LifecycleTest(unittest.TestCase): # rewrite a detail built from already validated closed fields. redact=lambda _line: "[structural]", ) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") detail = next( event.detail for event in result.events if event.kind == "metric:output_tokens" ) @@ -605,10 +719,10 @@ class LifecycleTest(unittest.TestCase): on_started=lambda _locator: (_ for _ in ()).throw(RuntimeError("durable write failed")), ) - self.assertEqual(result.terminal_reason, REASON_START_CALLBACK_FAILED) + self.assertEqual(result.harness.reason, REASON_START_CALLBACK_FAILED) self.assertFalse(result.submitted) self.assertFalse(marker.exists()) - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.harness.cleanup_complete) def test_forged_live_locator_refuses_recovery(self) -> None: checked: list[SupervisorLocator] = [] @@ -625,7 +739,27 @@ class LifecycleTest(unittest.TestCase): on_started=verify, ) self.assertTrue(checked) - self.assertTrue(result.success) + self.assertTrue(result.product.status == "succeeded") + + def test_control_socket_bind_supports_short_symlink_alias(self) -> None: + with tempfile.TemporaryDirectory( + dir=Path.cwd(), prefix=".lifecycle-symlink-target-" + ) as target_name, tempfile.TemporaryDirectory( + dir=tempfile.gettempdir(), prefix="iop-life-alias-parent-" + ) as alias_parent_name: + target = Path(target_name) + alias = Path(alias_parent_name) / "attempt" + alias.symlink_to(target.resolve(), target_is_directory=True) + evidence = target / "evidence" + evidence.mkdir() + result = self._run(replace( + self._spec("print('FINISH'); print('IDLE')"), + evidence_dir=str(evidence), + control_dir=str(alias / "control"), + )) + self.assertTrue(result.product.status == "succeeded") + self.assertTrue((target / "control" / "locator.json").is_file()) + self.assertFalse((target / "control" / "control.sock").exists()) def test_owned_descendant_ignoring_term_is_killed_and_reaped(self) -> None: descendant_pid_path = self.root / "descendant.pid" @@ -661,8 +795,8 @@ class LifecycleTest(unittest.TestCase): deadline = time.monotonic() + 2 while proc_path.exists() and time.monotonic() < deadline: time.sleep(.02) - self.assertTrue(result.success) - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.product.status == "succeeded") + self.assertTrue(result.harness.cleanup_complete) self.assertFalse(result.process_group_alive) self.assertFalse(proc_path.exists()) receipt = json.loads( @@ -809,13 +943,15 @@ class LifecycleTest(unittest.TestCase): for line in Path(result.journal_path).read_text(encoding="utf-8").splitlines() ] terminals = [record for record in journal if record.get("record") == "terminal"] - self.assertEqual(result.terminal_reason, expected) + self.assertEqual(result.harness.reason, expected) self.assertEqual(receipt["reason"], expected) - self.assertEqual(published["terminal_reason"], expected) + self.assertEqual(published["harness"]["reason"], expected) + self.assertEqual(receipt["exit_code"], published["process"]["exit_code"]) + self.assertEqual(receipt["signal"], published["process"]["signal"]) self.assertEqual(len(terminals), 1) - self.assertEqual(terminals[0]["terminal_reason"], expected) + self.assertEqual(terminals[0]["harness"]["reason"], expected) self.assertEqual(journal[-1]["record"], "terminal") - self.assertTrue(result.cleanup_complete) + self.assertTrue(result.harness.cleanup_complete) self.assertFalse(result.process_group_alive) def test_controller_eof_routes_supervisor_through_cleanup(self) -> None: diff --git a/scripts/agent_benchmark/live_iop.py b/scripts/agent_benchmark/live_iop.py index 80ffe7aa..a15aa89a 100644 --- a/scripts/agent_benchmark/live_iop.py +++ b/scripts/agent_benchmark/live_iop.py @@ -27,6 +27,7 @@ from urllib.request import Request, urlopen from scripts.agent_benchmark.agy_iop import ( AGY_CALLER, AgyEventParser, + AgyPreflightResult, AgyRuntimeInputs, AgyRuntimeObservation, build_agy_invocation, @@ -158,6 +159,9 @@ def _base_url(value: str) -> str: def _models_url(base_url: str) -> str: parsed = urlsplit(base_url) path = parsed.path.rstrip("/") + marker = "/gemini/" + if marker in path: + path = path.split(marker, 1)[0] if path.endswith("/v1"): path += "/models" else: @@ -165,6 +169,18 @@ def _models_url(base_url: str) -> str: return urlunsplit((parsed.scheme, parsed.netloc, path, "", "")) +def _agy_route_base(base_url: str, route_id: str) -> str: + parsed = urlsplit(base_url) + path = parsed.path.rstrip("/") + marker = "/gemini/" + if marker in path: + path = path.split(marker, 1)[0] + if path.endswith("/v1"): + path = path[:-3] + path = path.rstrip("/") + f"/gemini/{route_id}" + return urlunsplit((parsed.scheme, parsed.netloc, path, "", "")) + + def _command(argv: tuple[str, ...]) -> str: try: completed = subprocess.run( @@ -855,12 +871,15 @@ class _LiveAdapter: self._observer = observer self._binary_resolver = binary_resolver self._invokers = invokers - self._agy_preflight: Any = None + self._agy_preflights: dict[str, AgyPreflightResult] = {} self._admitted_bindings: dict[str, RequestedEffectiveBinding] = {} def preflight(self, cell: MatrixCell) -> PreflightObservation: if cell.caller != self.caller: raise LiveIopError("protocol_incompatible") + self._admitted_bindings.pop(cell.id, None) + if self.caller == AGY_CALLER: + self._agy_preflights.pop(cell.id, None) resolution = self._runtime_resolution runtime = resolution.runtime if runtime is None: @@ -874,31 +893,38 @@ class _LiveAdapter: if not observed.caller_ready: result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible")) return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity) + binding, issues = _binding_from_config(cell, self.capability, runtime.config) + if not issues and binding.effective_model not in observed.catalog_models: + binding, issues = _requested(cell), _issues("model_missing") + if issues: + result = make_result(cell, self.capability, binding, issues) + return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity) + agy_preflight: AgyPreflightResult | None = None if self.caller == AGY_CALLER: capability = inspect_agy_iop_capability(observed.agy_version, observed.agy_help) + agy_endpoint = _agy_route_base(runtime.base_url, cell.iop.route_id) agy_observation = AgyRuntimeObservation( cell.id, cell.iop.route_kind, cell.iop.route_id, - _agy_runtime_identity("endpoint", runtime.base_url), + _agy_runtime_identity("endpoint", agy_endpoint), _agy_runtime_identity("credential", runtime.secret), runtime.config.identity, ) try: - self._agy_preflight = preflight_agy_iop( + agy_preflight = preflight_agy_iop( cell, capability, - AgyRuntimeInputs(self._binary_resolver("agy"), runtime.base_url, runtime.secret), + AgyRuntimeInputs(self._binary_resolver("agy"), agy_endpoint, runtime.secret), agy_observation, ) except Exception: result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible")) return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity) - if self._agy_preflight.issues: - result = make_result(cell, self.capability, _requested(cell), self._agy_preflight.issues) + if agy_preflight.issues: + result = make_result(cell, self.capability, _requested(cell), agy_preflight.issues) return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity) - binding, issues = _binding_from_config(cell, self.capability, runtime.config) - if not issues and binding.effective_model not in observed.catalog_models: - binding, issues = _requested(cell), _issues("model_missing") result = make_result(cell, self.capability, binding, issues) if result.status == "ready": self._admitted_bindings[cell.id] = result.binding + if agy_preflight is not None: + self._agy_preflights[cell.id] = agy_preflight return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity) def invoke( @@ -933,7 +959,8 @@ class _LiveAdapter: admitted, ) if self.caller == AGY_CALLER: - if self._agy_preflight is None: + agy_preflight = self._agy_preflights.get(cell.id) + if agy_preflight is None or agy_preflight.binding != _requested(cell): raise LiveIopError("stream_incompatible") spec = _bind_live_spec( cell, @@ -941,12 +968,17 @@ class _LiveAdapter: attempt, control_dir, build_agy_invocation( - cell, prepared, task_payload, timeout, self._agy_preflight + cell, prepared, task_payload, timeout, agy_preflight ), ) - parser = AgyEventParser(cell) - result = self._invokers.agy(spec, parser, self._agy_preflight, lambda locator: on_started(locator, spec_digest(spec))) - observed = parser.observed_result(self._agy_preflight.capability, result) + parser = AgyEventParser(cell, admitted) + result = self._invokers.agy(spec, parser, agy_preflight, lambda locator: on_started(locator, spec_digest(spec))) + if ( + result.product.status != "succeeded" + or result.harness.status != "passed" + ): + return _bound_observations(result, admitted) + observed = parser.observed_result(agy_preflight.capability, result) if observed.status != "ready" or observed.binding != admitted: raise LiveIopError("stream_incompatible") return _bound_observations(result, admitted) @@ -964,7 +996,10 @@ class _LiveAdapter: ) result = self._invokers.codex(invocation, lambda locator: on_started(locator, spec_digest(invocation.spec))) expected = (admitted.effective_route_kind, admitted.effective_route_id, admitted.effective_model, admitted.effective_effort) - if result.effective_binding != expected: + if ( + result.effective_binding is not None + and result.effective_binding != expected + ): raise LiveIopError("stream_incompatible") return _bound_observations(result.lifecycle, admitted) raise LiveIopError("protocol_incompatible") @@ -1093,15 +1128,28 @@ class _LiveScoringAdapter: admitted.effective_model, admitted.effective_effort, ) - if result.effective_binding != expected: + if ( + result.effective_binding is not None + and result.effective_binding != expected + ): return ScoringInvocationResult( - False, "binding_mismatch", result.effective_binding + result.lifecycle.product.status, + "failed", + result.lifecycle.process.status, + result.lifecycle.process.exit_code, + result.lifecycle.process.signal, + "binding_mismatch", + result.effective_binding, ) lifecycle = _bound_observations(result.lifecycle, admitted) return ScoringInvocationResult( - lifecycle.success, - lifecycle.terminal_reason, - result.effective_binding, + lifecycle.product.status, + lifecycle.harness.status, + lifecycle.process.status, + lifecycle.process.exit_code, + lifecycle.process.signal, + lifecycle.harness.reason, + expected, ) def finalize_evidence( diff --git a/scripts/agent_benchmark/manifest.py b/scripts/agent_benchmark/manifest.py index c8ce10da..19eb8655 100644 --- a/scripts/agent_benchmark/manifest.py +++ b/scripts/agent_benchmark/manifest.py @@ -28,6 +28,8 @@ SESSION_POLICY = "fresh" SETUP_CACHE_POLICY = "isolated" DEFAULT_REPETITIONS = 1 RUBRIC_VERSION = "landing-quality-v1" +ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1" +RUBRIC_VERSIONS = (RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION) CALLER_ENUM = ("claude", "agy", "codex") ROUTE_KIND_ENUM = ("direct", "execution_preset") @@ -124,6 +126,7 @@ class Manifest: fixture: Fixture matrix: tuple[MatrixCell, ...] digest: str + execution_order_seed: str | None = None # --------------------------------------------------------------------------- @@ -190,7 +193,7 @@ def _require_enum(value: Any, field_name: str, allowed: tuple[str, ...]) -> str: def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str: s = _require_str(value, field_name) - if not pattern.match(s): + if pattern.fullmatch(s) is None: raise ManifestValidationError( f"field '{field_name}' does not match required pattern" ) @@ -310,7 +313,7 @@ def digest_manifest_and_resolved_inputs(manifest: Manifest) -> str: def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]: """Convert a Manifest to a plain dict for canonical JSON serialization.""" - return { + canonical = { "pipeline_version": manifest.pipeline_version, "environment": manifest.environment, "testbed": manifest.testbed, @@ -377,6 +380,11 @@ def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]: for c in manifest.matrix ], } + # Pipeline-v2 manifests without an explicit seed keep their legacy + # canonical bytes and digest. An explicit seed is part of the contract. + if manifest.execution_order_seed is not None: + canonical["execution_order_seed"] = manifest.execution_order_seed + return canonical # --------------------------------------------------------------------------- @@ -608,7 +616,19 @@ def _validate_evaluator(data: dict[str, Any]) -> Evaluator: return Evaluator(caller=caller, iop=_validate_iop_cell(obj["iop"])) -def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]: +def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]: + rank = hashlib.sha256( + b"iop-benchmark-order-v1\0" + + seed.encode("ascii") + + b"\0" + + cell.id.encode("ascii") + ).digest() + return rank, cell.id + + +def _validate_matrix( + data: list[Any], execution_order_seed: str | None +) -> tuple[MatrixCell, ...]: arr = _require_array(data, "matrix") if len(arr) < 1: raise ManifestValidationError("matrix must have at least 1 item") @@ -622,8 +642,13 @@ def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]: seen_ids.add(cell.id) cells.append(cell) - # Sort cells by id for canonical ordering - cells.sort(key=lambda c: c.id) + # No seed preserves the pipeline-v2 cell-id order. An explicit seed is + # stable across JSON input permutations and platforms. + cells.sort( + key=(lambda cell: cell.id) + if execution_order_seed is None + else (lambda cell: _execution_order_key(execution_order_seed, cell)) + ) return tuple(cells) @@ -684,7 +709,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani "session_policy", "setup_cache_policy", "timeout", "viewports", "rubric_version", "evaluator", "output_root", } - optional_top = {"repetitions"} + optional_top = {"repetitions", "execution_order_seed"} declared_keys = set(raw.keys()) required_present = expected_top.issubset(declared_keys) extra = declared_keys - expected_top - optional_top @@ -701,6 +726,12 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani if "repetitions" in raw: repetitions = _require_positive_int(raw["repetitions"], "repetitions") + execution_order_seed: str | None = None + if "execution_order_seed" in raw: + execution_order_seed = _require_pattern( + raw["execution_order_seed"], "execution_order_seed", CELL_ID_RE + ) + session_policy = _require_enum(raw["session_policy"], "session_policy", (SESSION_POLICY,)) setup_cache_policy = _require_enum( raw["setup_cache_policy"], "setup_cache_policy", (SETUP_CACHE_POLICY,) @@ -708,13 +739,13 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani timeout = _validate_timeout(raw["timeout"]) viewports = _validate_viewports(raw["viewports"]) rubric_version = _require_enum( - raw["rubric_version"], "rubric_version", (RUBRIC_VERSION,) + raw["rubric_version"], "rubric_version", RUBRIC_VERSIONS ) evaluator = _validate_evaluator(raw["evaluator"]) output_root = _validate_output_root(raw["output_root"], repo_root) fixture = _validate_fixture(raw["fixture"], repo_root) - matrix = _validate_matrix(raw["matrix"]) + matrix = _validate_matrix(raw["matrix"], execution_order_seed) # Compute manifest digest during load dummy_manifest = Manifest( @@ -732,6 +763,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani fixture=fixture, matrix=matrix, digest="", + execution_order_seed=execution_order_seed, ) computed_digest = digest_manifest_and_resolved_inputs(dummy_manifest) @@ -750,6 +782,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani fixture=fixture, matrix=matrix, digest=computed_digest, + execution_order_seed=execution_order_seed, ) diff --git a/scripts/agent_benchmark/manifest_test.py b/scripts/agent_benchmark/manifest_test.py index 28c23599..9a2fbd2c 100644 --- a/scripts/agent_benchmark/manifest_test.py +++ b/scripts/agent_benchmark/manifest_test.py @@ -14,6 +14,7 @@ import hashlib import inspect import json import os +import re import shutil import struct import subprocess @@ -38,11 +39,15 @@ from scripts.agent_benchmark.manifest import ( ManifestPathError, ManifestValidationError, MatrixCell, + ONE_SHOT_RUBRIC_VERSION, + RUBRIC_VERSION, + RUBRIC_VERSIONS, Timeout, Viewport, digest_manifest_and_resolved_inputs, digest_workspace_inputs, load_manifest, + _manifest_to_dict, validate_manifest_bytes, ) @@ -159,6 +164,230 @@ def _load_tmp_manifest(path: Path) -> Manifest: return load_manifest(path, repo_root=_REPO_ROOT) +class ManifestValidationTest(unittest.TestCase): + """Exact tracked-manifest regressions used by benchmark readiness packets.""" + + def test_iop_one_shot_manifest_locks_benchmark_readiness(self): + """The bench-02 manifest locks identical inputs and the complete C01-C09 matrix.""" + manifest_path = ( + _REPO_ROOT + / "scripts" + / "fixtures" + / "agent-comparison-benchmark-iop-one-shot.json" + ) + raw = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT) + + self.assertEqual( + { + key: raw[key] + for key in ( + "pipeline_version", + "environment", + "testbed", + "execution_order_seed", + "repetitions", + "session_policy", + "setup_cache_policy", + "timeout", + "viewports", + "rubric_version", + "evaluator", + "output_root", + ) + }, + { + "pipeline_version": "2", + "environment": "dev", + "testbed": "../iop-s2", + "execution_order_seed": "bench-02-c01-c09-v1", + "repetitions": 1, + "session_policy": "fresh", + "setup_cache_policy": "isolated", + "timeout": { + "run_seconds": 300, + "idle_seconds": 30, + "quiet_seconds": 10, + "cleanup_grace_seconds": 5, + }, + "viewports": [ + {"id": "desktop_1080", "width": 1920, "height": 1080}, + {"id": "mobile_375", "width": 375, "height": 812}, + ], + "rubric_version": ONE_SHOT_RUBRIC_VERSION, + "evaluator": { + "caller": "codex", + "iop": { + "request_model": "gpt-5.6-luna", + "requested_effort": "xhigh", + "route_kind": "direct", + "route_id": "gpt-5.6-luna", + "expected_bindings": [ + { + "stage": "request", + "model": "gpt-5.6-luna", + "effort": "xhigh", + } + ], + }, + }, + "output_root": "agent-test/runs/bench-02", + }, + ) + + expected_fixture = { + "version": "landing-v1", + "prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md", + "assets": [ + { + "source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", + "workspace_path": "brief/reference.txt", + }, + { + "source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", + "workspace_path": "assets/aurora-grid.svg", + }, + { + "source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", + "workspace_path": "assets/orbit-rings.svg", + }, + ], + "checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98", + } + self.assertEqual(raw["fixture"], expected_fixture) + self.assertEqual( + _compute_fixture_checksum(_REPO_ROOT, raw["fixture"]), + expected_fixture["checksum"], + ) + self.assertEqual( + [ + asset["workspace_path"] + for asset in raw["fixture"]["assets"] + if asset["source"].endswith(".svg") + ], + ["assets/aurora-grid.svg", "assets/orbit-rings.svg"], + ) + self.assertEqual( + manifest.viewports, + ( + Viewport(id="desktop_1080", width=1920, height=1080), + Viewport(id="mobile_375", width=375, height=812), + ), + ) + self.assertEqual( + manifest.timeout, + Timeout( + run_seconds=300, + idle_seconds=30, + quiet_seconds=10, + cleanup_grace_seconds=5, + ), + ) + + def direct(cell_id: str, caller: str, model: str, effort: str) -> dict: + return { + "id": cell_id, + "caller": caller, + "iop": { + "request_model": model, + "requested_effort": effort, + "route_kind": "direct", + "route_id": model, + "expected_bindings": [ + {"stage": "request", "model": model, "effort": effort} + ], + }, + } + + def hybrid( + cell_id: str, + caller: str, + route_id: str, + request_effort: str, + cloud_model: str, + ) -> dict: + return { + "id": cell_id, + "caller": caller, + "iop": { + "request_model": route_id, + "requested_effort": request_effort, + "route_kind": "execution_preset", + "route_id": route_id, + "expected_bindings": [ + {"stage": "selector", "model": cloud_model, "effort": "high"}, + {"stage": "plan", "model": cloud_model, "effort": "high"}, + {"stage": "work", "model": "ornith-fast"}, + {"stage": "review", "model": cloud_model, "effort": "high"}, + {"stage": "repair", "model": cloud_model, "effort": "high"}, + ], + }, + } + + expected_cells = [ + direct("c01-claude-sonnet-direct", "claude", "claude-sonnet-5", "max"), + direct("c02-claude-gemini-direct", "claude", "gemini-3.6-flash", "high"), + direct("c03-agy-gemini-direct", "agy", "gemini-3.6-flash", "high"), + direct("c04-claude-gpt-direct", "claude", "gpt-5.6-luna", "xhigh"), + direct("c05-codex-gpt-direct", "codex", "gpt-5.6-luna", "xhigh"), + hybrid( + "c06-claude-gemini-hybrid", + "claude", + "gemini-hybrid", + "high", + "gemini-3.6-flash", + ), + hybrid( + "c07-agy-gemini-hybrid", + "agy", + "gemini-hybrid", + "high", + "gemini-3.6-flash", + ), + hybrid( + "c08-claude-gpt-hybrid", + "claude", + "gpt-hybrid", + "xhigh", + "gpt-5.6-terra", + ), + hybrid( + "c09-codex-gpt-hybrid", + "codex", + "gpt-hybrid", + "xhigh", + "gpt-5.6-terra", + ), + ] + self.assertEqual(raw["matrix"], expected_cells) + + expected_order = [ + "c02-claude-gemini-direct", + "c05-codex-gpt-direct", + "c03-agy-gemini-direct", + "c06-claude-gemini-hybrid", + "c08-claude-gpt-hybrid", + "c09-codex-gpt-hybrid", + "c01-claude-sonnet-direct", + "c07-agy-gemini-hybrid", + "c04-claude-gpt-direct", + ] + expected_by_id = {cell["id"]: cell for cell in expected_cells} + self.assertEqual( + [cell.id for cell in manifest.matrix], + expected_order, + ) + self.assertEqual( + _manifest_to_dict(manifest)["matrix"], + [expected_by_id[cell_id] for cell_id in expected_order], + ) + self.assertEqual(manifest.execution_order_seed, "bench-02-c01-c09-v1") + self.assertEqual(manifest.rubric_version, ONE_SHOT_RUBRIC_VERSION) + self.assertEqual(manifest.fixture.checksum, expected_fixture["checksum"]) + self.assertEqual(manifest.digest, digest_manifest_and_resolved_inputs(manifest)) + self.assertRegex(manifest.digest, r"^sha256:[0-9a-f]{64}$") + + class TestLoadManifestValid(unittest.TestCase): """Valid manifest loading tests.""" @@ -374,6 +603,90 @@ class TestMatrixExtension(unittest.TestCase): class TestDeterministicOrdering(unittest.TestCase): """Deterministic cell and binding ordering tests.""" + @staticmethod + def _direct_cells(*cell_ids: str) -> list[dict]: + return [ + { + "id": cell_id, + "caller": "claude", + "iop": { + "request_model": "claude-sonnet-4-20250514", + "requested_effort": "high", + "route_kind": "direct", + "route_id": "claude-direct", + "expected_bindings": [ + { + "stage": "request", + "model": "claude-sonnet-4-20250514", + } + ], + }, + } + for cell_id in cell_ids + ] + + def test_execution_order_seed_is_canonical_and_permutation_stable(self): + """An explicit seed produces one repeatable order for every input permutation.""" + with tempfile.TemporaryDirectory() as tmp: + tmp_dir = Path(tmp) + cells = self._direct_cells("cell-a", "cell-b", "cell-c") + seeded = _make_minimal_manifest_dict( + execution_order_seed="seed-a", matrix=cells + ) + permuted = _make_minimal_manifest_dict( + execution_order_seed="seed-a", matrix=list(reversed(cells)) + ) + different_seed = _make_minimal_manifest_dict( + execution_order_seed="seed-b", matrix=cells + ) + + m1 = _load_tmp_manifest( + _write_tmp_manifest(tmp_dir, seeded, "seeded.json") + ) + m2 = _load_tmp_manifest( + _write_tmp_manifest(tmp_dir, permuted, "permuted.json") + ) + m3 = _load_tmp_manifest( + _write_tmp_manifest(tmp_dir, different_seed, "different.json") + ) + + self.assertEqual(m1.execution_order_seed, "seed-a") + self.assertEqual( + _manifest_to_dict(m1)["execution_order_seed"], + "seed-a", + ) + self.assertEqual( + [cell.id for cell in m1.matrix], + ["cell-c", "cell-a", "cell-b"], + ) + self.assertEqual(m1.matrix, m2.matrix) + self.assertEqual(m1.digest, m2.digest) + self.assertEqual( + [cell.id for cell in m3.matrix], + ["cell-a", "cell-c", "cell-b"], + ) + self.assertNotEqual(m1.matrix, m3.matrix) + self.assertNotEqual(m1.digest, m3.digest) + + def test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract(self): + """Omitting the seed keeps id order and excludes a synthetic default from canonical JSON.""" + with tempfile.TemporaryDirectory() as tmp: + tmp_dir = Path(tmp) + data = _make_minimal_manifest_dict( + matrix=self._direct_cells("cell-c", "cell-a", "cell-b") + ) + manifest = _load_tmp_manifest(_write_tmp_manifest(tmp_dir, data)) + + self.assertIsNone(manifest.execution_order_seed) + self.assertEqual( + [cell.id for cell in manifest.matrix], + ["cell-a", "cell-b", "cell-c"], + ) + self.assertNotIn("execution_order_seed", _manifest_to_dict(manifest)) + self.assertEqual( + manifest.digest, digest_manifest_and_resolved_inputs(manifest) + ) + def test_cells_sorted_by_id(self): """Cells are sorted by id regardless of input order.""" with tempfile.TemporaryDirectory() as tmp: @@ -951,7 +1264,7 @@ class TestEnumsAndBounds(unittest.TestCase): _load_tmp_manifest(path) def test_invalid_rubric_version_rejected(self): - """Invalid rubric_version pattern is rejected.""" + """A rubric version outside the closed catalog is rejected.""" with tempfile.TemporaryDirectory() as tmp: tmp_dir = Path(tmp) d = _make_minimal_manifest_dict(rubric_version="Invalid Version!") @@ -1313,6 +1626,44 @@ class TestSchemaLoaderParity(unittest.TestCase): m = load_manifest(example_path) self.assertEqual(m.pipeline_version, "2") self.assertEqual(m.testbed, "../iop-s2") + self.assertEqual(m.rubric_version, RUBRIC_VERSION) + + def test_schema_and_loader_accept_exact_rubric_version_catalog(self): + """The schema and loader accept both immutable rubric versions only.""" + schema_path = ( + _REPO_ROOT + / "scripts" + / "fixtures" + / "agent-comparison-benchmark-manifest.schema.json" + ) + schema = json.loads(schema_path.read_text(encoding="utf-8")) + rubric_schema = schema["properties"]["rubric_version"] + self.assertEqual(tuple(rubric_schema["enum"]), RUBRIC_VERSIONS) + self.assertEqual( + RUBRIC_VERSIONS, + (RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION), + ) + + with tempfile.TemporaryDirectory() as tmp: + tmp_dir = Path(tmp) + for index, version in enumerate(RUBRIC_VERSIONS): + with self.subTest(version=version): + path = _write_tmp_manifest( + tmp_dir, + _make_minimal_manifest_dict(rubric_version=version), + f"rubric-{index}.json", + ) + self.assertEqual(_load_tmp_manifest(path).rubric_version, version) + + unknown = "one-shot-agent-comparison-v2" + self.assertNotIn(unknown, rubric_schema["enum"]) + path = _write_tmp_manifest( + tmp_dir, + _make_minimal_manifest_dict(rubric_version=unknown), + "rubric-unknown.json", + ) + with self.assertRaises(ManifestValidationError): + _load_tmp_manifest(path) def test_tracked_fixtures_separate_generic_contract_from_direct_preflight(self): fixtures = _REPO_ROOT / "scripts" / "fixtures" @@ -1377,6 +1728,52 @@ class TestSchemaLoaderParity(unittest.TestCase): with self.assertRaises(ManifestValidationError): _load_tmp_manifest(p) + def test_schema_and_loader_reject_same_invalid_execution_order_seeds(self): + """The tracked schema and loader enforce the same bounded seed token.""" + schema_path = ( + _REPO_ROOT + / "scripts" + / "fixtures" + / "agent-comparison-benchmark-manifest.schema.json" + ) + schema = json.loads(schema_path.read_text(encoding="utf-8")) + seed_schema = schema["properties"]["execution_order_seed"] + self.assertEqual(seed_schema["type"], "string") + self.assertEqual(seed_schema["pattern"], "^[a-z0-9][a-z0-9_-]{0,63}$") + + boundary_seed = "a" * 64 + self.assertIsNotNone(re.fullmatch(seed_schema["pattern"], boundary_seed)) + with tempfile.TemporaryDirectory() as tmp: + tmp_dir = Path(tmp) + data = _make_minimal_manifest_dict( + execution_order_seed=boundary_seed + ) + path = _write_tmp_manifest(tmp_dir, data, "boundary-seed.json") + self.assertEqual( + _load_tmp_manifest(path).execution_order_seed, + boundary_seed, + ) + + invalid_seeds: tuple[object, ...] = ( + "", + "Uppercase", + "contains.dot", + "seed-a\n", + "a" * 65, + True, + None, + ) + pattern = re.compile(seed_schema["pattern"]) + for index, seed in enumerate(invalid_seeds): + schema_accepts = isinstance(seed, str) and pattern.fullmatch(seed) is not None + self.assertFalse(schema_accepts) + with tempfile.TemporaryDirectory() as tmp: + tmp_dir = Path(tmp) + data = _make_minimal_manifest_dict(execution_order_seed=seed) + path = _write_tmp_manifest(tmp_dir, data, f"invalid-seed-{index}.json") + with self.assertRaises(ManifestValidationError): + _load_tmp_manifest(path) + def test_preset_with_request_stage_rejected(self): """Execution preset cell with extra request stage raises ManifestValidationError.""" with tempfile.TemporaryDirectory() as tmp: diff --git a/scripts/agent_benchmark/measurement.py b/scripts/agent_benchmark/measurement.py index 35ffe94b..8f119ff2 100644 --- a/scripts/agent_benchmark/measurement.py +++ b/scripts/agent_benchmark/measurement.py @@ -40,15 +40,19 @@ from scripts.agent_benchmark.lifecycle import ( SOURCE_WORKSPACE_POLL, UNIT_NANOSECONDS, InvocationResult, + HarnessOutcome, LifecycleMetricError, + LifecycleValidationError, ParsedMetric, + ProcessOutcome, + ProductOutcome, metric_record, publish_bytes_no_replace, validate_metric, ) MEASUREMENT_FILENAME = "attempt-measurement.json" -MEASUREMENT_VERSION = 1 +MEASUREMENT_VERSION = 2 MEASUREMENT_RECORD = "attempt_measurement" STATUS_OBSERVED = "observed" @@ -133,7 +137,9 @@ class AttemptMeasurement: attempt: int caller: str spec_digest: str - terminal_reason: str + product: ProductOutcome + harness: HarnessOutcome + process: ProcessOutcome timeline: dict[str, Observation] usage: dict[str, Observation] observer: WorkspaceWriteObservation @@ -490,7 +496,7 @@ def build_measurement( for number in (repetition, attempt) ): raise MeasurementError("measurement identity is invalid") - if not _is_digest(result.spec_digest) or not result.terminal_reason: + if not _is_digest(result.spec_digest): raise MeasurementError("measurement invocation identity is invalid") metrics = tuple(result.metrics) if len(metrics) > MAX_OBSERVATION_RECORDS: @@ -514,7 +520,9 @@ def build_measurement( attempt=attempt, caller=caller, spec_digest=result.spec_digest, - terminal_reason=result.terminal_reason, + product=result.product, + harness=result.harness, + process=result.process, timeline=_timeline(result, observation), usage=_usage(metrics), observer=observation, @@ -538,7 +546,21 @@ def measurement_record(measurement: AttemptMeasurement) -> dict[str, Any]: }, "caller": measurement.caller, "spec_digest": measurement.spec_digest, - "terminal_reason": measurement.terminal_reason, + "product": { + "status": measurement.product.status, + "reason": measurement.product.reason, + }, + "harness": { + "status": measurement.harness.status, + "reason": measurement.harness.reason, + "ordered_terminal": measurement.harness.ordered_terminal, + "cleanup_complete": measurement.harness.cleanup_complete, + }, + "process": { + "status": measurement.process.status, + "exit_code": measurement.process.exit_code, + "signal": measurement.process.signal, + }, "timeline": { name: observation_record(measurement.timeline[name]) for name in TIMELINE_NAMES @@ -682,7 +704,8 @@ def load_measurement(attempt_root: str | Path) -> AttemptMeasurement: raise MeasurementError("measurement is not canonical JSON") from exc fields = { "record", "measurement_version", "attempt", "caller", "spec_digest", - "terminal_reason", "timeline", "usage", "observer", "observations", + "product", "harness", "process", "timeline", "usage", "observer", + "observations", } if not isinstance(record, dict) or set(record) != fields: raise MeasurementError("measurement schema is invalid") @@ -691,11 +714,16 @@ def load_measurement(attempt_root: str | Path) -> AttemptMeasurement: or record["measurement_version"] != MEASUREMENT_VERSION or not isinstance(record["caller"], str) or not record["caller"] or not isinstance(record["spec_digest"], str) or not _is_digest(record["spec_digest"]) - or not isinstance(record["terminal_reason"], str) or not record["terminal_reason"] or not isinstance(record["observations"], list) ): raise MeasurementError("measurement identity is invalid") run_id, cell_id, repetition, attempt = _identity_from_record(record["attempt"]) + try: + product = ProductOutcome(**record["product"]) + harness = HarnessOutcome(**record["harness"]) + process = ProcessOutcome(**record["process"]) + except (TypeError, LifecycleValidationError, ValueError) as exc: + raise MeasurementError("measurement outcomes are invalid") from exc measurement = AttemptMeasurement( run_id=run_id, cell_id=cell_id, @@ -703,7 +731,9 @@ def load_measurement(attempt_root: str | Path) -> AttemptMeasurement: attempt=attempt, caller=record["caller"], spec_digest=record["spec_digest"], - terminal_reason=record["terminal_reason"], + product=product, + harness=harness, + process=process, timeline=_observation_map(record["timeline"], TIMELINE_NAMES, "timeline"), usage=_observation_map(record["usage"], METRIC_NAMES, "usage"), observer=_observer_from_record(record["observer"]), @@ -757,6 +787,25 @@ def validate_measurement_lifecycle_binding( """ if not isinstance(lifecycle, Mapping): raise MeasurementError("lifecycle evidence is invalid") + expected_outcomes = { + "product": { + "status": measurement.product.status, + "reason": measurement.product.reason, + }, + "harness": { + "status": measurement.harness.status, + "reason": measurement.harness.reason, + "ordered_terminal": measurement.harness.ordered_terminal, + "cleanup_complete": measurement.harness.cleanup_complete, + }, + "process": { + "status": measurement.process.status, + "exit_code": measurement.process.exit_code, + "signal": measurement.process.signal, + }, + } + if any(lifecycle.get(name) != value for name, value in expected_outcomes.items()): + raise MeasurementError("measurement outcomes do not match lifecycle evidence") events = lifecycle.get("events") if not isinstance(events, list): raise MeasurementError("lifecycle events are invalid") diff --git a/scripts/agent_benchmark/measurement_test.py b/scripts/agent_benchmark/measurement_test.py index 7f93f492..f3485cfa 100644 --- a/scripts/agent_benchmark/measurement_test.py +++ b/scripts/agent_benchmark/measurement_test.py @@ -26,6 +26,9 @@ from scripts.agent_benchmark.lifecycle import ( SOURCE_WORKSPACE_POLL, UNIT_NANOSECONDS, CaptureStream, + ProductOutcome, + HarnessOutcome, + ProcessOutcome, InvocationResult, LifecycleMetricError, LifecycleEvent, @@ -80,13 +83,18 @@ def _result( _event(METRIC_PREFIX + metric.name, 3_000, metric.source) for metric in metrics ) return InvocationResult( - success=terminal_reason == "success", - terminal_reason=terminal_reason, - exit_code=0, - signal=None, + product=ProductOutcome( + "succeeded" if terminal_reason == "success" else "unknown", + "caller_success" if terminal_reason == "success" else "unavailable", + ), + harness=HarnessOutcome( + "passed" if terminal_reason == "success" else "failed", + terminal_reason, + True, + True, + ), + process=ProcessOutcome("exited", 0, None), submitted=True, - finish_then_idle_then_quiet=True, - cleanup_complete=True, process_group_alive=False, events=events + published, stdout=_capture("stdout"), @@ -284,7 +292,7 @@ class MeasurementRecordTest(unittest.TestCase): for override in ( {"run_id": ""}, {"cell_id": None}, {"caller": ""}, {"repetition": 0}, {"attempt": True}, - {"result": _result(terminal_reason="")}, + {"result": object()}, {"observation": None}, ): with self.subTest(override=tuple(override)): @@ -572,7 +580,7 @@ class MeasurementSidecarTest(unittest.TestCase): record = measurement_record(self.measurement) cases = { "unknown-field": {**record, "extra": 1}, - "wrong-version": {**record, "measurement_version": 2}, + "wrong-version": {**record, "measurement_version": 1}, "forged-digest": {**record, "spec_digest": "sha256:not-a-digest"}, "zeroed-unavailable": { **record, diff --git a/scripts/agent_benchmark/reporting.py b/scripts/agent_benchmark/reporting.py index 407ddd8e..650e6e88 100644 --- a/scripts/agent_benchmark/reporting.py +++ b/scripts/agent_benchmark/reporting.py @@ -424,17 +424,20 @@ def render_report(projection: ReportProjection) -> bytes: else: lines.append("| — | unavailable | 0 |") - lines.extend(("", "## Attempt outcomes", "", "| cell | repetition | attempt | execution | terminal | web | scoring | total | rank |", "|---|---:|---:|---|---|---|---|---:|---:|")) + lines.extend(("", "## Attempt outcomes", "", "| cell | repetition | attempt | controller | product | harness | process | artifact | scoring | total | rank |", "|---|---:|---:|---|---|---|---|---|---|---:|---:|")) if not projection.attempts: - lines.append("| — | — | — | blocked | unavailable | unavailable | unavailable | — | — |") + lines.append("| — | — | — | blocked | unavailable | unavailable | unavailable | unavailable | unavailable | — | — |") for item in projection.attempts: - terminal = "unavailable" if item.measurement is None else item.measurement.terminal_reason - web = "unavailable" if item.web is None else item.web.status + product = "unavailable" if item.measurement is None else item.measurement.product.status + harness = "unavailable" if item.measurement is None else item.measurement.harness.status + process = "unavailable" if item.measurement is None else item.measurement.process.status + artifact = "unavailable" if item.web is None else item.web.status total = "—" if item.score.total is None else str(item.score.total) rank = "—" if item.score.rank is None else str(item.score.rank) lines.append( f"| {_markdown(item.cell.id)} | {item.attempt.identity.repetition} | {item.attempt.identity.attempt} | " - f"{_markdown(item.attempt.state)} | {_markdown(terminal)} | {_markdown(web)} | " + f"{_markdown(item.attempt.state)} | {_markdown(product)} | {_markdown(harness)} | " + f"{_markdown(process)} | {_markdown(artifact)} | " f"{_markdown(item.score.status)} | {total} | {rank} |" ) raw_paths.update(item.raw_paths) diff --git a/scripts/agent_benchmark/rubric.py b/scripts/agent_benchmark/rubric.py index 1891d836..5288b26d 100644 --- a/scripts/agent_benchmark/rubric.py +++ b/scripts/agent_benchmark/rubric.py @@ -1,4 +1,4 @@ -"""Strict D12 landing-page worksheet contract. +"""Strict versioned benchmark worksheet contract. Automatic web gates establish scoring eligibility; they are intentionally not represented in this 100-point worksheet and can never contribute points. @@ -11,9 +11,13 @@ import os import stat from dataclasses import dataclass from pathlib import Path -from typing import Any +from types import MappingProxyType +from typing import Any, Mapping -from scripts.agent_benchmark.manifest import RUBRIC_VERSION +from scripts.agent_benchmark.manifest import ( + ONE_SHOT_RUBRIC_VERSION, + RUBRIC_VERSION, +) MAX_WORKSHEET_BYTES = 64 * 1024 @@ -25,12 +29,39 @@ RUBRIC_CATEGORIES = ( ("typography_readability", 15), ("polish_consistency", 15), ) +ONE_SHOT_RUBRIC_CATEGORIES = ( + ("requirements_fidelity", 25), + ("visual_completeness", 25), + ("responsive_accessibility", 15), + ("image_detail_usage", 10), + ("behavior_stability", 10), + ("code_quality", 10), + ("self_verification", 5), +) +RUBRIC_CATEGORIES_BY_VERSION: Mapping[str, tuple[tuple[str, int], ...]] = ( + MappingProxyType( + { + RUBRIC_VERSION: RUBRIC_CATEGORIES, + ONE_SHOT_RUBRIC_VERSION: ONE_SHOT_RUBRIC_CATEGORIES, + } + ) +) class RubricError(Exception): """A worksheet is missing, malformed, non-canonical, or out of bounds.""" +def rubric_categories(version: str) -> tuple[tuple[str, int], ...]: + """Return the immutable ordered category table for a known rubric version.""" + if not isinstance(version, str): + raise RubricError("rubric version is invalid") + try: + return RUBRIC_CATEGORIES_BY_VERSION[version] + except KeyError as exc: + raise RubricError("rubric version is invalid") from exc + + @dataclass(frozen=True) class CategoryScore: id: str @@ -75,22 +106,28 @@ def canonical_worksheet_bytes(worksheet: Worksheet) -> bytes: ) -def validate_worksheet(value: Any) -> Worksheet: +def validate_worksheet( + value: Any, expected_version: str | None = None +) -> Worksheet: if not isinstance(value, dict) or set(value) != { "rubric_version", "categories", "total", }: raise RubricError("worksheet schema is invalid") - if value["rubric_version"] != RUBRIC_VERSION: + rubric_version = value["rubric_version"] + expected_categories = rubric_categories(rubric_version) + if expected_version is not None: + rubric_categories(expected_version) + if expected_version is not None and rubric_version != expected_version: raise RubricError("worksheet rubric version is invalid") raw_categories = value["categories"] if not isinstance(raw_categories, list) or len(raw_categories) != len( - RUBRIC_CATEGORIES + expected_categories ): raise RubricError("worksheet categories are invalid") categories: list[CategoryScore] = [] for raw, (expected_id, expected_max) in zip( - raw_categories, RUBRIC_CATEGORIES + raw_categories, expected_categories ): if not isinstance(raw, dict) or set(raw) != { "id", "max_score", "score", "evidence", @@ -123,10 +160,12 @@ def validate_worksheet(value: Any) -> Worksheet: or not 0 <= total <= 100 ): raise RubricError("worksheet total is invalid") - return Worksheet(RUBRIC_VERSION, tuple(categories), total) + return Worksheet(rubric_version, tuple(categories), total) -def load_worksheet(path: str | Path) -> Worksheet: +def load_worksheet( + path: str | Path, expected_version: str | None = None +) -> Worksheet: target = Path(path) flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK if hasattr(os, "O_NOFOLLOW"): @@ -155,4 +194,4 @@ def load_worksheet(path: str | Path) -> Worksheet: value = json.loads(bytes(raw).decode("utf-8")) except (UnicodeDecodeError, json.JSONDecodeError) as exc: raise RubricError("worksheet JSON is invalid") from exc - return validate_worksheet(value) + return validate_worksheet(value, expected_version=expected_version) diff --git a/scripts/agent_benchmark/rubric_test.py b/scripts/agent_benchmark/rubric_test.py index 37f689c6..70399843 100644 --- a/scripts/agent_benchmark/rubric_test.py +++ b/scripts/agent_benchmark/rubric_test.py @@ -1,21 +1,29 @@ from __future__ import annotations +import hashlib import json import os import tempfile import unittest from pathlib import Path +from scripts.agent_benchmark.manifest import ( + ONE_SHOT_RUBRIC_VERSION, + RUBRIC_VERSION, +) from scripts.agent_benchmark.rubric import ( + ONE_SHOT_RUBRIC_CATEGORIES, RUBRIC_CATEGORIES, + RUBRIC_CATEGORIES_BY_VERSION, RubricError, canonical_worksheet_bytes, load_worksheet, + rubric_categories, validate_worksheet, ) -def _worksheet() -> dict: +def _worksheet(rubric_version: str = RUBRIC_VERSION) -> dict: categories = [ { "id": ident, @@ -23,10 +31,10 @@ def _worksheet() -> dict: "score": maximum, "evidence": f"Evidence for {ident}.", } - for ident, maximum in RUBRIC_CATEGORIES + for ident, maximum in rubric_categories(rubric_version) ] return { - "rubric_version": "landing-quality-v1", + "rubric_version": rubric_version, "categories": categories, "total": 100, } @@ -44,6 +52,62 @@ class RubricTest(unittest.TestCase): validate_worksheet(json.loads(canonical_worksheet_bytes(worksheet))), worksheet, ) + self.assertEqual( + hashlib.sha256(canonical_worksheet_bytes(worksheet)).hexdigest(), + "233727170226ab30409657007751c13616a16c83ca75700e15a2c8dce56a96ee", + ) + + def test_one_shot_rubric_exact_categories_and_total_are_accepted(self): + expected = ( + ("requirements_fidelity", 25), + ("visual_completeness", 25), + ("responsive_accessibility", 15), + ("image_detail_usage", 10), + ("behavior_stability", 10), + ("code_quality", 10), + ("self_verification", 5), + ) + self.assertEqual(ONE_SHOT_RUBRIC_CATEGORIES, expected) + self.assertEqual(rubric_categories(ONE_SHOT_RUBRIC_VERSION), expected) + self.assertEqual(tuple(RUBRIC_CATEGORIES_BY_VERSION), ( + RUBRIC_VERSION, + ONE_SHOT_RUBRIC_VERSION, + )) + self.assertEqual(sum(maximum for _, maximum in RUBRIC_CATEGORIES), 100) + self.assertEqual(sum(maximum for _, maximum in expected), 100) + + worksheet = validate_worksheet( + _worksheet(ONE_SHOT_RUBRIC_VERSION), + expected_version=ONE_SHOT_RUBRIC_VERSION, + ) + self.assertEqual(worksheet.rubric_version, ONE_SHOT_RUBRIC_VERSION) + self.assertEqual(worksheet.total, 100) + self.assertEqual( + [(item.id, item.max_score) for item in worksheet.categories], + list(expected), + ) + + def test_unknown_and_cross_version_worksheets_are_rejected(self): + with self.assertRaises(RubricError): + rubric_categories("unknown-rubric-v1") + with self.assertRaises(RubricError): + validate_worksheet(_worksheet(), expected_version=ONE_SHOT_RUBRIC_VERSION) + with self.assertRaises(RubricError): + validate_worksheet( + _worksheet(ONE_SHOT_RUBRIC_VERSION), + expected_version=RUBRIC_VERSION, + ) + unknown = _worksheet() + unknown["rubric_version"] = "unknown-rubric-v1" + with self.assertRaises(RubricError): + validate_worksheet(unknown) + + def test_cross_version_worksheet_is_rejected(self): + with tempfile.TemporaryDirectory() as temp: + path = Path(temp) / "worksheet.json" + path.write_text(json.dumps(_worksheet()), encoding="utf-8") + with self.assertRaises(RubricError): + load_worksheet(path, expected_version=ONE_SHOT_RUBRIC_VERSION) def test_missing_extra_reordered_and_out_of_range_values_fail(self): cases = [] diff --git a/scripts/agent_benchmark/scoring.py b/scripts/agent_benchmark/scoring.py index 990693b0..9d63b83a 100644 --- a/scripts/agent_benchmark/scoring.py +++ b/scripts/agent_benchmark/scoring.py @@ -2,8 +2,8 @@ The original cell identity remains under ``cells/`` and in a run-owned mapping that is never copied into the evaluator tree. A scorer receives only one -opaque ``blind/`` directory, anonymous input bytes, the fixed D12 rubric, -and a fresh session/output pair. +opaque ``blind/`` directory, anonymous input bytes, the manifest-selected +rubric, and a fresh session/output pair. """ from __future__ import annotations @@ -34,6 +34,7 @@ from scripts.agent_benchmark.connectivity import ( ) from scripts.agent_benchmark.manifest import Manifest, MatrixCell, Timeout from scripts.agent_benchmark.lifecycle import ( + JOURNAL_VERSION, LifecycleRecoveryError, REASON_CONTROLLER_LOST, REASON_RECOVERED_STOP, @@ -42,11 +43,11 @@ from scripts.agent_benchmark.lifecycle import ( recover_invocation, ) from scripts.agent_benchmark.rubric import ( - RUBRIC_CATEGORIES, RubricError, Worksheet, canonical_worksheet_bytes, load_worksheet, + rubric_categories, ) from scripts.agent_benchmark.web_validation import ( GENERATED_FILES, @@ -57,7 +58,7 @@ from scripts.agent_benchmark.web_validation import ( ) -SCORING_VERSION = 1 +SCORING_VERSION = 2 SCORE_RE = re.compile(r"^score-([0-9]{6})$") BLIND_ID_RE = re.compile(r"^blind-[0-9a-f]{32}$") DIGEST_RE = re.compile(r"^sha256:[0-9a-f]{64}$") @@ -70,6 +71,9 @@ INPUT_FILENAME = "input.json" RESULT_FILENAME = "result.json" RUNNER_FILENAME = "runner.json" SCORING_STATUSES = ("scored", "unscored", "scoring_failed", "blocked") +SCORING_INVOCATION_REASONS = TERMINAL_REASONS + ( + "binding_mismatch", "evaluator_failed", +) _POST_CLEANUP_TIMEOUT_SECONDS = 2.0 _POST_CLEANUP_QUIET_SECONDS = 0.2 _POST_CLEANUP_POLL_SECONDS = 0.01 @@ -92,10 +96,46 @@ class BlindWorkspace: @dataclass(frozen=True) class ScoringInvocationResult: - success: bool - terminal_reason: str + product: str + harness: str + process: str + process_exit_code: int | None + process_signal: int | None + reason: str effective_binding: tuple[str, str, str, str] | None + def __post_init__(self) -> None: + if ( + self.product not in {"succeeded", "failed", "unknown"} + or self.harness not in {"passed", "failed"} + or self.process not in { + "exited", "signalled", "timed_out", "cancelled", "not_started" + } + or ( + self.process_exit_code is not None + and ( + not isinstance(self.process_exit_code, int) + or isinstance(self.process_exit_code, bool) + ) + ) + or ( + self.process_signal is not None + and ( + not isinstance(self.process_signal, int) + or isinstance(self.process_signal, bool) + ) + ) + or (self.process == "signalled" and self.process_signal is None) + or ( + self.process in {"exited", "not_started"} + and self.process_signal is not None + ) + or (self.process == "not_started" and self.process_exit_code is not None) + or self.reason not in SCORING_INVOCATION_REASONS + or (self.harness == "passed") != (self.reason == "success") + ): + raise ScoringError("scoring invocation result is invalid") + @dataclass(frozen=True) class ScoringEvidenceFinalization: @@ -693,7 +733,9 @@ def _validate_cleanup_receipt( or receipt["reason"] not in TERMINAL_REASONS or (expected_reason is not None and receipt["reason"] != expected_reason) or not isinstance(receipt["caller_launched"], bool) - or receipt["cleanup_complete"] is not True + or not isinstance(receipt["cleanup_complete"], bool) + or (not receipt["cleanup_complete"]) + != (receipt["reason"] == "cleanup_failed") or receipt["process_group_alive"] is not False or not isinstance(receipt["completed_at"], str) ): @@ -717,22 +759,66 @@ def _validate_lifecycle_binding( if not path.exists() and not path.is_symlink(): return None value = _load_json(path, "evaluator lifecycle") + product = value.get("product") + harness = value.get("harness") + process = value.get("process") if ( value.get("record") != "result" or value.get("spec_digest") != invocation_digest or value.get("locator") != _locator_public(locator) - or value.get("terminal_reason") not in TERMINAL_REASONS - or value.get("cleanup_complete") is not True + or not isinstance(product, dict) + or not isinstance(harness, dict) + or not isinstance(process, dict) + or set(product) != {"status", "reason"} + or product.get("status") not in {"succeeded", "failed", "unknown"} + or product.get("reason") + != { + "succeeded": "caller_success", + "failed": "caller_error", + "unknown": "unavailable", + }.get(product.get("status")) + or set(harness) + != {"status", "reason", "ordered_terminal", "cleanup_complete"} + or harness.get("status") not in {"passed", "failed"} + or harness.get("reason") not in TERMINAL_REASONS + or (harness.get("status") == "passed") + != (harness.get("reason") == "success") + or not isinstance(harness.get("ordered_terminal"), bool) + or (harness.get("status") == "passed") + and not harness.get("ordered_terminal") + or not isinstance(harness.get("cleanup_complete"), bool) + or (not harness.get("cleanup_complete")) + != (harness.get("reason") == "cleanup_failed") + or set(process) != {"status", "exit_code", "signal"} + or process.get("status") + not in {"exited", "signalled", "timed_out", "cancelled", "not_started"} + or any( + value is not None + and (not isinstance(value, int) or isinstance(value, bool)) + for value in (process.get("exit_code"), process.get("signal")) + ) + or (process.get("status") == "signalled" and process.get("signal") is None) + or ( + process.get("status") in {"exited", "not_started"} + and process.get("signal") is not None + ) + or ( + process.get("status") == "not_started" + and process.get("exit_code") is not None + ) or value.get("process_group_alive") is not False - or value.get("success") - is not (value.get("terminal_reason") == "success") ): raise ScoringError("evaluator lifecycle binding is invalid") - _validate_cleanup_receipt( + receipt, _ = _validate_cleanup_receipt( locator, - expected_reason=str(value["terminal_reason"]), + expected_reason=str(harness["reason"]), control_target=control_target, ) + if ( + receipt["exit_code"] != process["exit_code"] + or receipt["signal"] != process["signal"] + ): + raise ScoringError("evaluator lifecycle binding is invalid") journal = blind_root / "output" / "lifecycle-journal.jsonl" try: lines = _read_regular( @@ -745,11 +831,13 @@ def _validate_lifecycle_binding( if ( not isinstance(header, dict) or header.get("record") != "header" + or header.get("journal_version") != JOURNAL_VERSION or header.get("spec_digest") != invocation_digest or not isinstance(terminal, dict) or terminal.get("record") != "terminal" - or terminal.get("terminal_reason") != value["terminal_reason"] - or terminal.get("cleanup_complete") is not True + or terminal.get("product") != product + or terminal.get("harness") != harness + or terminal.get("process") != process or terminal.get("process_group_alive") is not False ): raise ScoringError("evaluator lifecycle journal is invalid") @@ -942,13 +1030,29 @@ def _release_runner_alias(runner: Mapping[str, Any]) -> None: def _eligibility(manifest: Manifest, attempt: Attempt) -> tuple[bool, tuple[str, ...]]: if attempt.state not in TERMINAL_STATES: return False, ("lifecycle_running",) - if attempt.state != "success": - return False, (f"lifecycle_{attempt.state}",) + try: + execution = _load_json(Path(attempt.root) / "attempt.json", "execution attempt") + except ScoringError: + raise + lifecycle = execution.get("lifecycle") + if not isinstance(lifecycle, dict): + raise ScoringError("execution eligibility evidence is invalid") + reasons: list[str] = [] + product = lifecycle.get("product") + harness = lifecycle.get("harness") + process = lifecycle.get("process") + if not isinstance(product, dict) or product.get("status") != "succeeded": + reasons.append("product_" + str((product or {}).get("status", "unknown"))) + if not isinstance(harness, dict) or harness.get("status") != "passed": + reasons.append("harness_" + str((harness or {}).get("status", "failed"))) + if not isinstance(process, dict) or process.get("status") != "exited": + reasons.append("process_" + str((process or {}).get("status", "not_started"))) + elif process.get("exit_code") != 0 or process.get("signal") is not None: + reasons.append("process_nonzero_exit") try: web = load_web_validation(attempt.root, manifest=manifest) except WebValidationError as exc: raise ScoringError("web eligibility evidence is invalid") from exc - reasons: list[str] = [] if web.status != "passed": reasons.append(f"web_{web.status}") if web.record["reason"]: @@ -1204,9 +1308,10 @@ def _materialize_blind( ) -def _prompt(blind: BlindWorkspace) -> bytes: +def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes: categories = ", ".join( - f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES + f"{ident} ({maximum})" + for ident, maximum in rubric_categories(rubric_version) ) value = ( "Evaluate the anonymous landing page using only the files under input/. " @@ -1216,7 +1321,7 @@ def _prompt(blind: BlindWorkspace) -> bytes: "Write exactly one JSON object to output/worksheet.json with fields " "rubric_version, categories, and total. Each category must have exactly " "id, max_score, score, and non-empty evidence. Use rubric_version " - "landing-quality-v1, integer scores within each maximum, and total equal " + f"{rubric_version}, integer scores within each maximum, and total equal " "to their sum. Do not add automatic gate points or any extra field." ) return value.encode("utf-8") @@ -1654,7 +1759,8 @@ def _result_status( raise ScoringError("scored result is invalid") try: worksheet = load_worksheet( - Path(run.root) / allocation["blind_path"] / "output" / "worksheet.json" + Path(run.root) / allocation["blind_path"] / "output" / "worksheet.json", + expected_version=manifest.rubric_version, ) except RubricError as exc: raise ScoringError("scored worksheet is invalid") from exc @@ -1886,7 +1992,7 @@ def _score_one( ) return "scoring_failed" - prompt = _prompt(blind) + prompt = _prompt(blind, manifest.rubric_version) identities = _identity_values(manifest, attempt) if _contains_identity(prompt, identities): _publish_current_failure( @@ -1964,11 +2070,30 @@ def _score_one( lifecycle, runner_digest, receipt_digest, _ = _evidence_digests( score_root, blind_root, allocation, run, attempt ) + lifecycle_record = ( + None + if lifecycle is None + else _load_json( + blind_root / "output" / "lifecycle-result.json", + "evaluator lifecycle", + ) + ) if ( - not invocation.success - or invocation.terminal_reason != "success" + invocation.product != "succeeded" + or invocation.harness != "passed" + or invocation.process != "exited" + or invocation.process_exit_code != 0 + or invocation.process_signal is not None + or invocation.reason != "success" or invocation.effective_binding != expected_binding - or lifecycle is None + or lifecycle_record is None + or lifecycle_record["product"]["status"] != invocation.product + or lifecycle_record["harness"]["status"] != invocation.harness + or lifecycle_record["process"]["status"] != invocation.process + or lifecycle_record["process"]["exit_code"] + != invocation.process_exit_code + or lifecycle_record["process"]["signal"] != invocation.process_signal + or lifecycle_record["harness"]["reason"] != invocation.reason or runner_digest is None or receipt_digest is None ): @@ -1984,7 +2109,10 @@ def _score_one( ) return "scoring_failed" try: - worksheet = load_worksheet(blind_root / "output" / "worksheet.json") + worksheet = load_worksheet( + blind_root / "output" / "worksheet.json", + expected_version=manifest.rubric_version, + ) post_tree = _blind_tree_digest(blind_root) _publish_success( score_root, diff --git a/scripts/agent_benchmark/scoring_test.py b/scripts/agent_benchmark/scoring_test.py index 4e3b68fd..ab0cfe18 100644 --- a/scripts/agent_benchmark/scoring_test.py +++ b/scripts/agent_benchmark/scoring_test.py @@ -29,6 +29,8 @@ from scripts.agent_benchmark.connectivity import ( make_result, ) from scripts.agent_benchmark.lifecycle import ( + CALLER_REASON_ERROR, + CALLER_REASON_SUCCESS, COMPLETION_EXIT_AFTER_IDLE, CLOCK_HARNESS_MONOTONIC, METRIC_NAMES, @@ -37,13 +39,23 @@ from scripts.agent_benchmark.lifecycle import ( SUBMISSION_STDIN_ONCE, UNIT_NANOSECONDS, InvocationSpec, + JOURNAL_VERSION, + HarnessOutcome, + ProcessOutcome, + ProductOutcome, SupervisorLocator, env_pairs, recover_invocation, run_invocation, spec_digest, ) -from scripts.agent_benchmark.manifest import AssetMapping, digest_workspace_inputs, load_manifest +from scripts.agent_benchmark.manifest import ( + AssetMapping, + ONE_SHOT_RUBRIC_VERSION, + RUBRIC_VERSION, + digest_workspace_inputs, + load_manifest, +) from scripts.agent_benchmark.measurement import ( AttemptMeasurement, REASON_NOT_OBSERVED, @@ -53,7 +65,7 @@ from scripts.agent_benchmark.measurement import ( publish_measurement, unavailable, ) -from scripts.agent_benchmark.rubric import RUBRIC_CATEGORIES +from scripts.agent_benchmark.rubric import rubric_categories from scripts.agent_benchmark import scoring as scoring_module from scripts.agent_benchmark.scoring import ( BlindWorkspace, @@ -73,9 +85,11 @@ def _digest(data: bytes) -> str: return "sha256:" + hashlib.sha256(data).hexdigest() -def _worksheet(total_delta: int = 0) -> dict: +def _worksheet( + total_delta: int = 0, rubric_version: str = RUBRIC_VERSION +) -> dict: categories = [] - for index, (ident, maximum) in enumerate(RUBRIC_CATEGORIES): + for index, (ident, maximum) in enumerate(rubric_categories(rubric_version)): score = maximum - (1 if index == 0 else 0) categories.append( { @@ -86,7 +100,7 @@ def _worksheet(total_delta: int = 0) -> dict: } ) return { - "rubric_version": "landing-quality-v1", + "rubric_version": rubric_version, "categories": categories, "total": sum(item["score"] for item in categories) + total_delta, } @@ -103,12 +117,14 @@ class FakeScoringAdapter: blocked: bool = False, modes: list[str] | None = None, sensitive_value: str = "", + rubric_version: str = RUBRIC_VERSION, ): self.blocked = blocked self.modes = list(modes or ["success"]) self.preflights = 0 self.invocations: list[tuple[BlindWorkspace, bytes]] = [] self.sensitive_value = sensitive_value + self.rubric_version = rubric_version self.last_mode = "" def preflight(self, cell): @@ -178,6 +194,7 @@ class FakeScoringAdapter: invocation_digest = "sha256:" + "4" * 64 on_started(locator, invocation_digest) terminal_reason = "nonzero_exit" if mode == "raise" else "success" + exit_code = 7 if mode == "process_nonzero" else (1 if mode == "raise" else 0) receipt = { "receipt_version": 1, "supervisor_pid": locator.supervisor_pid, @@ -185,7 +202,7 @@ class FakeScoringAdapter: locator.challenge.encode("utf-8") ).hexdigest(), "reason": terminal_reason, - "exit_code": 1 if mode == "raise" else 0, + "exit_code": exit_code, "signal": None, "caller_launched": True, "cleanup_complete": True, @@ -201,11 +218,27 @@ class FakeScoringAdapter: if key != "challenge" } public_locator["challenge_digest"] = receipt["challenge_digest"] + product = ( + {"status": "unknown", "reason": "unavailable"} + if mode == "raise" + else {"status": "succeeded", "reason": CALLER_REASON_SUCCESS} + ) + harness = { + "status": "failed" if mode == "raise" else "passed", + "reason": terminal_reason, + "ordered_terminal": mode != "raise", + "cleanup_complete": True, + } + process = { + "status": "exited", + "exit_code": exit_code, + "signal": None, + } lifecycle = { "record": "result", - "success": terminal_reason == "success", - "terminal_reason": terminal_reason, - "cleanup_complete": True, + "product": product, + "harness": harness, + "process": process, "process_group_alive": False, "spec_digest": invocation_digest, "locator": public_locator, @@ -221,14 +254,19 @@ class FakeScoringAdapter: ) journal = ( json.dumps( - {"record": "header", "spec_digest": invocation_digest} + { + "record": "header", + "journal_version": JOURNAL_VERSION, + "spec_digest": invocation_digest, + } ) + "\n" + json.dumps( { "record": "terminal", - "terminal_reason": terminal_reason, - "cleanup_complete": True, + "product": product, + "harness": harness, + "process": process, "process_group_alive": False, } ) @@ -253,7 +291,7 @@ class FakeScoringAdapter: if mode == "malformed": (output / "worksheet.json").write_text("{}", encoding="utf-8") else: - worksheet = _worksheet() + worksheet = _worksheet(rubric_version=self.rubric_version) if mode == "secret": worksheet["categories"][0]["evidence"] = self.sensitive_value (output / "worksheet.json").write_text( @@ -269,7 +307,16 @@ class FakeScoringAdapter: ) if mode == "binding": binding = (binding[0], "substituted", binding[2], binding[3]) - return ScoringInvocationResult(mode not in {"failed", "binding"}, "success" if mode not in {"failed", "binding"} else "failed", binding) + passed = mode not in {"failed", "binding"} + return ScoringInvocationResult( + "succeeded" if passed else "failed", + "passed" if passed else "failed", + "exited", + 7 if mode == "process_nonzero" else 0, + None, + "success" if passed else "evaluator_failed", + binding, + ) def finalize_evidence(self, blind): leaked = False @@ -418,6 +465,7 @@ class ScoringTest(unittest.TestCase): name: unavailable(REASON_NOT_REPORTED, SOURCE_HARNESS) for name in METRIC_NAMES } + succeeded = terminal_reason == "success" return AttemptMeasurement( attempt.identity.run_id, attempt.identity.cell_id, @@ -425,7 +473,12 @@ class ScoringTest(unittest.TestCase): attempt.identity.attempt, "claude", "sha256:" + "3" * 64, - terminal_reason, + ProductOutcome( + "succeeded" if succeeded else "failed", + CALLER_REASON_SUCCESS if succeeded else CALLER_REASON_ERROR, + ), + HarnessOutcome("passed", "success", True, True), + ProcessOutcome("exited", 0 if succeeded else 1, None), timeline, usage, WorkspaceWriteObservation( @@ -516,7 +569,7 @@ class ScoringTest(unittest.TestCase): (workspace / "script.js").write_text( "document.body.dataset.ready='1';", encoding="utf-8" ) - terminal_reason = "success" if state == "success" else state + terminal_reason = "success" if state == "success" else "caller_error" measurement = self._measurement(attempt, terminal_reason) publish_measurement(attempt.root, measurement) if state == "success" and rendered: @@ -540,8 +593,28 @@ class ScoringTest(unittest.TestCase): self.manifest, workspace, measurement, render ) publish_web_validation(attempt.root, web) + succeeded = state == "success" + lifecycle = { + "product": { + "status": "succeeded" if succeeded else "failed", + "reason": ( + CALLER_REASON_SUCCESS if succeeded else CALLER_REASON_ERROR + ), + }, + "harness": { + "status": "passed", + "reason": "success", + "ordered_terminal": True, + "cleanup_complete": True, + }, + "process": { + "status": "exited", + "exit_code": 0 if succeeded else 1, + "signal": None, + }, + } terminal = self.store.publish_terminal( - attempt, state, result={"terminal_reason": terminal_reason} + attempt, "completed", result=lifecycle ) return terminal @@ -558,6 +631,22 @@ class ScoringTest(unittest.TestCase): self.assertEqual(adapter.preflights, 1) self.assertEqual(len(adapter.invocations), 1) blind, prompt = adapter.invocations[0] + self.assertEqual( + prompt, + ( + "Evaluate the anonymous landing page using only the files under input/. " + "Inspect index.html, styles.css, script.js, the two local images, and all " + "screenshots. Do not use network access or infer the producing system. " + "Use exactly these categories: task_fidelity (25), visual_hierarchy " + "(25), responsive_composition (20), typography_readability (15), " + "polish_consistency (15). Write exactly one JSON object to " + "output/worksheet.json with fields rubric_version, categories, and " + "total. Each category must have exactly id, max_score, score, and " + "non-empty evidence. Use rubric_version landing-quality-v1, integer " + "scores within each maximum, and total equal to their sum. Do not add " + "automatic gate points or any extra field." + ).encode("utf-8"), + ) self.assertNotIn("cell-sentinel", blind.root) self.assertNotIn("source-route", blind.root) self.assertNotIn(str(Path(attempt.root).resolve()), blind.root) @@ -601,6 +690,88 @@ class ScoringTest(unittest.TestCase): self.assertEqual(len(adapter.invocations), 1) self.assertEqual(before, {path: path.read_bytes() for path in Path(attempt.root).rglob("*") if path.is_file()}) + def test_scoring_invocation_result_rejects_open_or_contradictory_reasons(self): + with self.assertRaises(ScoringError): + ScoringInvocationResult( + "succeeded", "passed", "exited", 0, None, + "arbitrary_reason", None, + ) + with self.assertRaises(ScoringError): + ScoringInvocationResult( + "succeeded", "passed", "exited", 0, None, + "evaluator_failed", None, + ) + + def test_manifest_selected_rubric_drives_prompt_and_worksheet_validation(self): + raw = json.loads(self.manifest_path.read_text(encoding="utf-8")) + raw["rubric_version"] = ONE_SHOT_RUBRIC_VERSION + raw["output_root"] = "agent-test/runs/new-rubric-bench" + path = self.root / "new-rubric.json" + path.write_text(json.dumps(raw), encoding="utf-8") + self.manifest_path = path + self.manifest = load_manifest(path, repo_root=self.root) + + self.run = self.store.create(self.manifest, path.read_bytes()) + attempt = self._attempt() + selected = FakeScoringAdapter(rubric_version=ONE_SHOT_RUBRIC_VERSION) + summary = score_run( + self.store, self.run, self.manifest, adapter=selected + ) + self.assertEqual((summary.scored, summary.scoring_failed), (1, 0)) + self.assertEqual(len(selected.invocations), 1) + _, prompt = selected.invocations[0] + self.assertIn( + b"Use exactly these categories: requirements_fidelity (25), " + b"visual_completeness (25), responsive_accessibility (15), " + b"image_detail_usage (10), behavior_stability (10), code_quality (10), " + b"self_verification (5).", + prompt, + ) + self.assertIn( + b"Use rubric_version one-shot-agent-comparison-v1, integer scores", + prompt, + ) + result_path = ( + Path(attempt.root) / "scoring" / "score-000001" / "result.json" + ) + result = json.loads(result_path.read_text(encoding="ascii")) + self.assertEqual(result["status"], "scored") + self.assertEqual( + result["worksheet"]["rubric_version"], ONE_SHOT_RUBRIC_VERSION + ) + self.assertEqual( + [item["id"] for item in result["worksheet"]["categories"]], + [ident for ident, _ in rubric_categories(ONE_SHOT_RUBRIC_VERSION)], + ) + + retained = score_run( + self.store, + self.run, + self.manifest, + adapter=FakeScoringAdapter(rubric_version=ONE_SHOT_RUBRIC_VERSION), + ) + self.assertEqual(retained.scored, 1) + + self.run = self.store.create(self.manifest, path.read_bytes()) + mismatched_attempt = self._attempt() + legacy = FakeScoringAdapter() + rejected = score_run( + self.store, self.run, self.manifest, adapter=legacy + ) + self.assertEqual((rejected.scored, rejected.scoring_failed), (0, 1)) + self.assertEqual(len(legacy.invocations), 1) + mismatch_result = json.loads( + ( + Path(mismatched_attempt.root) + / "scoring" + / "score-000001" + / "result.json" + ).read_text(encoding="ascii") + ) + self.assertEqual(mismatch_result["status"], "scoring_failed") + self.assertEqual(mismatch_result["reason"], "invalid_worksheet") + self.assertNotIn("worksheet", mismatch_result) + def test_ineligible_attempt_is_unscored_without_preflight_or_zero(self): attempt = self._attempt("failed") adapter = FakeScoringAdapter() @@ -611,7 +782,20 @@ class ScoringTest(unittest.TestCase): before = path.read_bytes() record = json.loads(before) self.assertEqual(record["status"], "unscored") - self.assertEqual(record["reasons"], ["lifecycle_failed"]) + self.assertEqual( + record["reasons"], + [ + "product_failed", + "process_nonzero_exit", + "web_failed", + "web_reason_render_not_run", + "gate_images", + "gate_network", + "gate_console", + "gate_responsive", + "gate_accessibility", + ], + ) self.assertFalse(set(record) & {"score", "total", "worksheet"}) score_run(self.store, self.run, self.manifest, adapter=adapter) self.assertEqual(path.read_bytes(), before) @@ -707,6 +891,23 @@ class ScoringTest(unittest.TestCase): self.assertEqual(result["status"], "scoring_failed") self.assertNotIn("worksheet", result) + self.run = self.store.create(self.manifest, self.manifest_path.read_bytes()) + process_attempt = self._attempt() + process = FakeScoringAdapter(modes=["process_nonzero"]) + summary = score_run(self.store, self.run, self.manifest, adapter=process) + self.assertEqual(summary.scoring_failed, 1) + result = json.loads( + ( + Path(process_attempt.root) + / "scoring" + / "score-000001" + / "result.json" + ).read_text() + ) + self.assertEqual((result["status"], result["reason"]), ( + "scoring_failed", "evaluator_failed", + )) + # A separate run proves an identity sentinel in retained page bytes is # rejected before the evaluator is invoked. self.run = self.store.create(self.manifest, self.manifest_path.read_bytes()) @@ -1036,23 +1237,38 @@ class ScoringTest(unittest.TestCase): key: value for key, value in locator_payload.items() if key != "challenge" } public_locator["challenge_digest"] = receipt["challenge_digest"] + product = {"status": "succeeded", "reason": CALLER_REASON_SUCCESS} + harness = { + "status": "passed", + "reason": "success", + "ordered_terminal": True, + "cleanup_complete": True, + } + process = {"status": "exited", "exit_code": 0, "signal": None} lifecycle = { "record": "result", - "success": True, - "terminal_reason": "success", - "cleanup_complete": True, + "product": product, + "harness": harness, + "process": process, "process_group_alive": False, "spec_digest": invocation_digest, "locator": public_locator, } journal = ( - json.dumps({"record": "header", "spec_digest": invocation_digest}) + json.dumps( + { + "record": "header", + "journal_version": JOURNAL_VERSION, + "spec_digest": invocation_digest, + } + ) + "\n" + json.dumps( { "record": "terminal", - "terminal_reason": "success", - "cleanup_complete": True, + "product": product, + "harness": harness, + "process": process, "process_group_alive": False, } ) @@ -1495,7 +1711,7 @@ class ScoringTest(unittest.TestCase): self.assertFalse(adapter.worker.is_alive()) # type: ignore[union-attr] self.assertIsNone(adapter.worker_error) self.assertIsNotNone(adapter.worker_result) - self.assertTrue(adapter.worker_result.cleanup_complete) + self.assertTrue(adapter.worker_result.harness.cleanup_complete) self.assertFalse(adapter.worker_result.process_group_alive) receipt = json.loads( ( diff --git a/scripts/agent_benchmark/skill_contract_test.py b/scripts/agent_benchmark/skill_contract_test.py index 4482ab6b..dfe20300 100644 --- a/scripts/agent_benchmark/skill_contract_test.py +++ b/scripts/agent_benchmark/skill_contract_test.py @@ -248,13 +248,13 @@ class BenchmarkSkillContractTest(unittest.TestCase): """Assert report is a supported operation alongside run/resume.""" procedure = self._get_section(skill_text, "Procedure") self.assertNotIn("capability-unavailable: caller-adapter", procedure) - self.assertIn("append a fresh preflight before attempt allocation", procedure) + self.assertIn("append a fresh all-cell preflight before attempt allocation", procedure) self.assertIn("invoke each eligible cell exactly once", procedure) # report must be delegated, not gated self.assertIn("report", procedure) def _assert_preflight_contract(self, skill_text: str) -> None: - """Require direct-only append semantics and fail-closed blocker language.""" + """Require exact all-cell append semantics and fail-closed blockers.""" procedure = self._get_section(skill_text, "Procedure") validation = self._get_section(skill_text, "Validation") prohibitions = self._get_section(skill_text, "Prohibitions") @@ -262,16 +262,31 @@ class BenchmarkSkillContractTest(unittest.TestCase): "python3 scripts/agent_comparison_benchmark.py preflight --manifest ", procedure, ) - self.assertIn("records only direct-cell observations", procedure) - self.assertIn("Generic preset cells are local contract validation only", skill_text) + self.assertIn( + "records one fresh live observation for every immutable matrix cell", + procedure, + ) + self.assertIn("including direct and execution-preset routes", procedure) + self.assertIn("in canonical matrix order", procedure) self.assertIn("registration_required", procedure) self.assertIn("implementation_gap", procedure) self.assertIn("Never bypass the blocker", procedure) self.assertIn("substitute a route/model/effort", procedure) - self.assertIn("Preflight evidence is append-only, direct-only", validation) + self.assertIn( + "Preflight evidence is append-only, covers every immutable matrix cell in canonical order", + validation, + ) self.assertIn("created no scored attempt", validation) self.assertIn("Do not bypass a preflight blocker", prohibitions) self.assertIn("Do not claim execution-preset fixture validation as live readiness", prohibitions) + for obsolete in ( + "records only direct-cell observations", + "Generic preset cells are local contract validation only", + "Preflight evidence is append-only, direct-only", + "Direct preflight never allocates a scored attempt", + "fresh direct preflight", + ): + self.assertNotIn(obsolete, skill_text) def _assert_scoring_contract(self, skill_text: str) -> None: procedure = self._get_section(skill_text, "Procedure") @@ -284,7 +299,13 @@ class BenchmarkSkillContractTest(unittest.TestCase): self.assertIn("immutable `unscored`", procedure) self.assertIn("without invoking the evaluator or assigning zero", procedure) self.assertIn("manifest-bound fresh Codex evaluator session", procedure) + self.assertIn("exact immutable manifest-selected rubric", procedure) + self.assertIn("closed supported catalog", procedure) + self.assertIn("`landing-quality-v1`", procedure) + self.assertIn("`one-shot-agent-comparison-v1`", procedure) + self.assertIn("no substitute rubric or reinterpretation is permitted", procedure) self.assertIn("allocates a new score id and preserves every prior byte", procedure) + self.assertIn("failed product, harness, process, or required artifact gate", procedure) self.assertIn("`scoring_failed` used no fallback", validation) self.assertIn("Do not retry scoring implicitly", prohibitions) self.assertIn("convert `unscored`/`scoring_failed` to zero", prohibitions) @@ -319,6 +340,11 @@ class BenchmarkSkillContractTest(unittest.TestCase): self._assert_preflight_contract(skill_text) self._assert_scoring_contract(skill_text) self._assert_no_secret_operational_language(skill_text) + self.assertIn("product_succeeded=", skill_text) + self.assertIn("harness_passed=", skill_text) + self.assertIn("process_exited=", skill_text) + self.assertIn("artifact_passed=", skill_text) + self.assertIn("five-cell direct manifest", skill_text) # ------------------------------------------------------------------ # Template / frontmatter invariants @@ -441,7 +467,7 @@ class BenchmarkSkillContractTest(unittest.TestCase): """Run/resume must document execution rather than a capability gate.""" skill_text = _SKILL_FILE.read_text(encoding="utf-8") self.assertNotIn("capability-unavailable: caller-adapter", skill_text) - self.assertIn("append a fresh preflight before attempt allocation", skill_text) + self.assertIn("append a fresh all-cell preflight before attempt allocation", skill_text) def test_cli_help_exits_zero(self) -> None: """Skill must not expose a public prepare operation across all steps and sections.""" @@ -783,11 +809,11 @@ class BenchmarkSkillContractTest(unittest.TestCase): with self.assertRaises(AssertionError): self._assert_full_skill_contract(mutated) - def test_mutation_claims_preset_live_readiness(self) -> None: + def test_mutation_restores_direct_only_preflight(self) -> None: base = self._skill_base_text() mutated = base.replace( - "Generic preset cells are local contract validation only.", - "Generic preset cells are live readiness evidence.", + "records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order", + "records only direct-cell observations", ) self.assertNotEqual(mutated, base, "mutation fixture did not apply") with self.assertRaises(AssertionError): @@ -803,6 +829,17 @@ class BenchmarkSkillContractTest(unittest.TestCase): with self.assertRaises(AssertionError): self._assert_full_skill_contract(mutated) + def test_mutation_restores_fixed_legacy_scoring_rubric(self) -> None: + """A fixed legacy-only worksheet must fail manifest-selected scoring.""" + base = self._skill_base_text() + mutated = base.replace( + "the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted", + "the exact `landing-quality-v1` worksheet", + ) + self.assertNotEqual(mutated, base, "mutation fixture did not apply") + with self.assertRaises(AssertionError): + self._assert_full_skill_contract(mutated) + def test_mutation_allows_blocker_attempt_allocation(self) -> None: base = self._skill_base_text() mutated = base.replace( diff --git a/scripts/agent_benchmark/web_validation.py b/scripts/agent_benchmark/web_validation.py index 6a8a7810..fe3447c4 100644 --- a/scripts/agent_benchmark/web_validation.py +++ b/scripts/agent_benchmark/web_validation.py @@ -31,7 +31,7 @@ from scripts.agent_benchmark.measurement import ( ) WEB_VALIDATION_FILENAME = "web-validation.json" -WEB_VALIDATION_VERSION = 1 +WEB_VALIDATION_VERSION = 2 WEB_STATUSES = ("passed", "failed", "blocked", "not_run") WEB_GATES = ( "generated_files", @@ -488,20 +488,12 @@ def build_web_validation( snapshot = _workspace_snapshot(workspace, manifest) generated_gate = _generated_gate(snapshot) static_gate = _static_gate(workspace, generated_gate, manifest) - terminal_reason = _reason_token( - getattr(measurement, "terminal_reason", "success"), "invalid_lifecycle" - ) - browser = {"status": "not_observed", "product": "", "origin": ""} requests: list[dict[str, Any]] = [] console: list[dict[str, Any]] = [] viewports: list[dict[str, Any]] = [] reason = "" - if terminal_reason != "success": - status = "not_run" - reason = f"lifecycle_{terminal_reason}" - gates = _not_observed_gates(reason, "lifecycle") - elif blocked: + if blocked: status = "blocked" reason = _reason_token(blocked, "browser_failure") gates = _not_observed_gates(reason, "browser") @@ -571,8 +563,6 @@ def validate_web_attempt( browser_binary: str = "chromium", ) -> WebValidation: workspace = Path(prepared.workspace_dir) - if getattr(measurement, "terminal_reason", "") != "success": - return build_web_validation(manifest, prepared, measurement, None) render = None blocked = "" generated_ready = all( diff --git a/scripts/agent_benchmark/web_validation_test.py b/scripts/agent_benchmark/web_validation_test.py index c25e35ab..e041ea06 100644 --- a/scripts/agent_benchmark/web_validation_test.py +++ b/scripts/agent_benchmark/web_validation_test.py @@ -87,7 +87,6 @@ class WebValidationTest(unittest.TestCase): cell_id="cell", repetition=1, attempt=1, - terminal_reason=reason, ) def _view(self, ident: str, width: int, *, suffix: str = "") -> ViewportObservation: @@ -294,15 +293,19 @@ class WebValidationTest(unittest.TestCase): gates = {item["id"]: item for item in record.record["gates"]} self.assertFalse(gates[gate]["passed"]) - def test_lifecycle_non_success_is_not_run_without_browser(self): + def test_lifecycle_non_success_still_validates_workspace(self): for reason in ("nonzero_exit", "timed_out", "cancelled", "controller_lost"): with self.subTest(reason=reason): record = build_web_validation( self._manifest(), self.workspace, self._measurement(reason), None ) - self.assertEqual(record.status, "not_run") - self.assertEqual(record.record["reason"], f"lifecycle_{reason}") - self.assertFalse(any(item["passed"] for item in record.record["gates"])) + self.assertEqual(record.status, "failed") + self.assertEqual(record.record["reason"], "render_not_run") + gates = {item["id"]: item for item in record.record["gates"]} + self.assertTrue(gates["generated_files"]["passed"]) + self.assertTrue(gates["static_safety"]["passed"]) + self.assertFalse(gates["images"]["passed"]) + self.assertFalse(gates["responsive"]["passed"]) def test_browser_discovery_or_start_failure_is_blocked(self): prepared = SimpleNamespace( diff --git a/scripts/agent_comparison_benchmark.py b/scripts/agent_comparison_benchmark.py index 84533e2d..0d8eb4d4 100644 --- a/scripts/agent_comparison_benchmark.py +++ b/scripts/agent_comparison_benchmark.py @@ -11,7 +11,7 @@ Usage: python3 scripts/agent_comparison_benchmark.py score --manifest PATH --run-id RUN_ID Exits: - 0 - manifest is valid or every direct preflight cell is ready + 0 - manifest is valid or every matrix preflight cell is ready 64 - usage error (missing args, bad flags) 69 - validation/state failed or preflight is blocked """ @@ -126,7 +126,24 @@ def _cmd_state(args: argparse.Namespace) -> int: else: run = store.open(manifest, args.run_id, raw) if args.command == "status": - print("ok: " + str(store.status(run, manifest)["attempts"])) + status = store.status(run, manifest) + attempts = status["attempts"] + outcomes = status["outcomes"] + attempt_summary = " ".join( + f"{state}={attempts[state]}" + for state in ( + "completed", "timed_out", "cancelled", "interrupted", "running" + ) + ) + axes = " ".join( + f"{axis}_{name}={outcomes[axis][name]}" + for axis in ("product", "harness", "process", "artifact") + for name in outcomes[axis] + ) + print( + f"ok: status run_id={run.run_id} " + f"unresolved={outcomes['unresolved']} {attempt_summary} {axes}" + ) return EXIT_VALID completed = run_slots( @@ -158,17 +175,24 @@ def _cmd_state(args: argparse.Namespace) -> int: attempt_summary = " ".join( f"{state}={attempts[state]}" for state in ( - "success", "failed", "timed_out", "cancelled", "interrupted", "running" + "completed", "timed_out", "cancelled", "interrupted", "running" ) ) - unresolved = 0 - for slot in store.slots(manifest): - retained = store.attempts(run, slot) - if not retained or retained[-1].state != "success": - unresolved += 1 + outcomes = status["outcomes"] + unresolved = outcomes["unresolved"] + axes = " ".join( + f"{prefix}_{name}={counts[name]}" + for prefix, counts in ( + ("product", outcomes["product"]), + ("harness", outcomes["harness"]), + ("process", outcomes["process"]), + ("artifact", outcomes["artifact"]), + ) + for name in counts + ) summary = ( - f"run_id={run.run_id} completed={len(completed)} " - f"unresolved={unresolved} {attempt_summary}" + f"run_id={run.run_id} executed={len(completed)} " + f"unresolved={unresolved} {attempt_summary} {axes}" ) if unresolved: print("error: benchmark execution failed " + summary, file=sys.stderr) diff --git a/scripts/e2e-credential-slot-smoke.sh b/scripts/e2e-credential-slot-smoke.sh index 8144b95b..429bf615 100755 --- a/scripts/e2e-credential-slot-smoke.sh +++ b/scripts/e2e-credential-slot-smoke.sh @@ -798,11 +798,11 @@ create_slot_route() { } wait_route_projection() { - local route_id="$1" + local public_route_id="$1" local deadline=$((SECONDS + 20)) while true; do if curl_timeout "$HTTP_PROBE_TIMEOUT" -fsS "${EDGE_CURL[@]}" "$EDGE_BASE_URL/v1/models" -H "Authorization: Bearer $IOP_TOKEN" >"$TMP_DIR/models.json" 2>/dev/null \ - && jq -e --arg id "$route_id" '.data | any(.[]; .id == $id)' "$TMP_DIR/models.json" >/dev/null; then + && jq -e --arg id "$public_route_id" '.data | any(.[]; .id == $id)' "$TMP_DIR/models.json" >/dev/null; then rm -f "$TMP_DIR/models.json" return 0 fi @@ -940,8 +940,8 @@ run_deterministic() { create_slot_route CHAT "$chat_profile" "$chat_vendor" "$chat_kind" "$chat_slot_alias" "$chat_route_alias" "$chat_provider" "$upstream_model" "$TMP_DIR/chat.secret" create_slot_route MESSAGES "$messages_profile" "$messages_vendor" "$messages_kind" "$messages_slot_alias" "$messages_route_alias" "$messages_provider" "$upstream_model" "$TMP_DIR/messages.secret" - wait_route_projection "$CHAT_ROUTE_ID" - wait_route_projection "$MESSAGES_ROUTE_ID" + wait_route_projection "$chat_route_alias" + wait_route_projection "$messages_route_alias" request_chat "$chat_route_alias" "$TMP_DIR/chat.response.json" request_messages "$messages_route_alias" "$TMP_DIR/messages.response.json" @@ -1064,7 +1064,7 @@ run_live() { write_runtime_configs "$LIVE_PROFILE" "$LIVE_PROFILE" "$LIVE_MODEL" "$provider_id" "credential-live-unselected" "$endpoint" start_managed_stack create_slot_route LIVE "$LIVE_PROFILE" "$vendor" "$kind" "$slot_alias" "$route_alias" "$provider_id" "$LIVE_MODEL" "$TMP_DIR/live.secret" - wait_route_projection "$LIVE_ROUTE_ID" + wait_route_projection "$route_alias" local body live_code body="$(jq -cn --arg model "$route_alias" --arg effort "$LIVE_REASONING_EFFORT" --argjson cap "$LIVE_MAX_COMPLETION_TOKENS" \ diff --git a/scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json b/scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json new file mode 100644 index 00000000..81b5d5d8 --- /dev/null +++ b/scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json @@ -0,0 +1,178 @@ +{ + "pipeline_version": "2", + "environment": "dev", + "testbed": "../iop-s2", + "execution_order_seed": "bench-02-c01-c09-v1", + "repetitions": 1, + "session_policy": "fresh", + "setup_cache_policy": "isolated", + "timeout": { + "run_seconds": 300, + "idle_seconds": 30, + "quiet_seconds": 10, + "cleanup_grace_seconds": 5 + }, + "viewports": [ + {"id": "desktop_1080", "width": 1920, "height": 1080}, + {"id": "mobile_375", "width": 375, "height": 812} + ], + "rubric_version": "one-shot-agent-comparison-v1", + "evaluator": { + "caller": "codex", + "iop": { + "request_model": "gpt-5.6-luna", + "requested_effort": "xhigh", + "route_kind": "direct", + "route_id": "gpt-5.6-luna", + "expected_bindings": [ + {"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"} + ] + } + }, + "output_root": "agent-test/runs/bench-02", + "fixture": { + "version": "landing-v1", + "prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md", + "assets": [ + {"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"}, + {"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"}, + {"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"} + ], + "checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98" + }, + "matrix": [ + { + "id": "c01-claude-sonnet-direct", + "caller": "claude", + "iop": { + "request_model": "claude-sonnet-5", + "requested_effort": "max", + "route_kind": "direct", + "route_id": "claude-sonnet-5", + "expected_bindings": [ + {"stage": "request", "model": "claude-sonnet-5", "effort": "max"} + ] + } + }, + { + "id": "c02-claude-gemini-direct", + "caller": "claude", + "iop": { + "request_model": "gemini-3.6-flash", + "requested_effort": "high", + "route_kind": "direct", + "route_id": "gemini-3.6-flash", + "expected_bindings": [ + {"stage": "request", "model": "gemini-3.6-flash", "effort": "high"} + ] + } + }, + { + "id": "c03-agy-gemini-direct", + "caller": "agy", + "iop": { + "request_model": "gemini-3.6-flash", + "requested_effort": "high", + "route_kind": "direct", + "route_id": "gemini-3.6-flash", + "expected_bindings": [ + {"stage": "request", "model": "gemini-3.6-flash", "effort": "high"} + ] + } + }, + { + "id": "c04-claude-gpt-direct", + "caller": "claude", + "iop": { + "request_model": "gpt-5.6-luna", + "requested_effort": "xhigh", + "route_kind": "direct", + "route_id": "gpt-5.6-luna", + "expected_bindings": [ + {"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"} + ] + } + }, + { + "id": "c05-codex-gpt-direct", + "caller": "codex", + "iop": { + "request_model": "gpt-5.6-luna", + "requested_effort": "xhigh", + "route_kind": "direct", + "route_id": "gpt-5.6-luna", + "expected_bindings": [ + {"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"} + ] + } + }, + { + "id": "c06-claude-gemini-hybrid", + "caller": "claude", + "iop": { + "request_model": "gemini-hybrid", + "requested_effort": "high", + "route_kind": "execution_preset", + "route_id": "gemini-hybrid", + "expected_bindings": [ + {"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "work", "model": "ornith-fast"}, + {"stage": "review", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"} + ] + } + }, + { + "id": "c07-agy-gemini-hybrid", + "caller": "agy", + "iop": { + "request_model": "gemini-hybrid", + "requested_effort": "high", + "route_kind": "execution_preset", + "route_id": "gemini-hybrid", + "expected_bindings": [ + {"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "work", "model": "ornith-fast"}, + {"stage": "review", "model": "gemini-3.6-flash", "effort": "high"}, + {"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"} + ] + } + }, + { + "id": "c08-claude-gpt-hybrid", + "caller": "claude", + "iop": { + "request_model": "gpt-hybrid", + "requested_effort": "xhigh", + "route_kind": "execution_preset", + "route_id": "gpt-hybrid", + "expected_bindings": [ + {"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "work", "model": "ornith-fast"}, + {"stage": "review", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"} + ] + } + }, + { + "id": "c09-codex-gpt-hybrid", + "caller": "codex", + "iop": { + "request_model": "gpt-hybrid", + "requested_effort": "xhigh", + "route_kind": "execution_preset", + "route_id": "gpt-hybrid", + "expected_bindings": [ + {"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "work", "model": "ornith-fast"}, + {"stage": "review", "model": "gpt-5.6-terra", "effort": "high"}, + {"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"} + ] + } + } + ] +} diff --git a/scripts/fixtures/agent-comparison-benchmark-manifest.schema.json b/scripts/fixtures/agent-comparison-benchmark-manifest.schema.json index 0d89ce54..1cfa2740 100644 --- a/scripts/fixtures/agent-comparison-benchmark-manifest.schema.json +++ b/scripts/fixtures/agent-comparison-benchmark-manifest.schema.json @@ -28,11 +28,17 @@ "minimum": 1, "default": 1 }, + "execution_order_seed": { + "type": "string", + "pattern": "^[a-z0-9][a-z0-9_-]{0,63}$" + }, "session_policy": { "const": "fresh" }, "setup_cache_policy": { "const": "isolated" }, "timeout": { "$ref": "#/$defs/timeout" }, "viewports": { "$ref": "#/$defs/viewports" }, - "rubric_version": { "const": "landing-quality-v1" }, + "rubric_version": { + "enum": ["landing-quality-v1", "one-shot-agent-comparison-v1"] + }, "evaluator": { "$ref": "#/$defs/evaluator" }, "output_root": { "$ref": "#/$defs/output_root" }, "fixture": { "$ref": "#/$defs/fixture" }, diff --git a/scripts/fixtures/agent-comparison-benchmark-report.expected.md b/scripts/fixtures/agent-comparison-benchmark-report.expected.md index e3f319b8..c2b0c11e 100644 --- a/scripts/fixtures/agent-comparison-benchmark-report.expected.md +++ b/scripts/fixtures/agent-comparison-benchmark-report.expected.md @@ -27,13 +27,13 @@ ## Attempt outcomes -| cell | repetition | attempt | execution | terminal | web | scoring | total | rank | -|---|---:|---:|---|---|---|---|---:|---:| -| cell-sentinel | 1 | 1 | success | success | passed | scored | 99 | 1 | -| cell-sentinel | 1 | 2 | success | success | passed | scored | 99 | 1 | -| cell-sentinel | 1 | 3 | failed | failed | not_run | unscored | — | — | -| cell-sentinel | 1 | 4 | success | success | passed | scoring_failed | — | — | -| cell-sentinel | 1 | 5 | success | success | passed | blocked | — | — | +| cell | repetition | attempt | controller | product | harness | process | artifact | scoring | total | rank | +|---|---:|---:|---|---|---|---|---|---|---:|---:| +| cell-sentinel | 1 | 1 | completed | succeeded | passed | exited | passed | scored | 99 | 1 | +| cell-sentinel | 1 | 2 | completed | succeeded | passed | exited | passed | scored | 99 | 1 | +| cell-sentinel | 1 | 3 | completed | failed | passed | exited | failed | unscored | — | — | +| cell-sentinel | 1 | 4 | completed | succeeded | passed | exited | passed | scoring_failed | — | — | +| cell-sentinel | 1 | 5 | completed | succeeded | passed | exited | passed | blocked | — | — | ## Quality score breakdown @@ -66,7 +66,7 @@ |---|---|---|---|---|---| | cell-sentinel/r1/a1 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded | | cell-sentinel/r1/a2 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded | -| cell-sentinel/r1/a3 | generated_files=fail, static_safety=fail, images=fail, network=fail, console=fail, responsive=fail, accessibility=fail | unavailable | — | unavailable | lifecycle_failed | +| cell-sentinel/r1/a3 | generated_files=pass, static_safety=pass, images=fail, network=fail, console=fail, responsive=fail, accessibility=fail | unavailable | — | unavailable | product_failed, process_nonzero_exit, web_failed, web_reason_render_not_run, gate_images, gate_network, gate_console, gate_responsive, gate_accessibility | | cell-sentinel/r1/a4 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | invalid_worksheet | | cell-sentinel/r1/a5 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | — | unavailable | evaluator_preflight_blocked | diff --git a/scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl b/scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl index 459635b0..477c4b25 100644 --- a/scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl +++ b/scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl @@ -1,5 +1,3 @@ -{"type":"metric","subtype":"duration_ms","value":12} -{"type":"metric","subtype":"model_duration_ms","value":8.5} -{"type":"iop","subtype":"effective_binding","route_kind":"direct","route_id":"agy-direct","model":"gemini-2.0-flash","effort":"high","stages":[{"stage":"request","model":"gemini-2.0-flash","effort":"high"}]} -{"type":"result","subtype":"success","model":"gemini-2.0-flash","effort":"high","route_kind":"direct","route_id":"agy-direct","content":"[redacted]"} -{"type":"system","subtype":"idle","model":"gemini-2.0-flash","effort":"high","route_kind":"direct","route_id":"agy-direct","tool_input":"[redacted]"} +{"conversation_id":"fixture-conversation","event":"init","init":{"cwd":"/workspace","model":"Gemini 3.6 Flash","permission_mode":"sandbox","tools":[]}} +{"event":"step_update","step_update":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"state":"DONE","step_index":0,"step_type":"agent_response","text_delta":"fixture content must be redacted","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}} +{"event":"result","result":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"num_turns":1,"response":"fixture content must be redacted","status":"SUCCESS","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}} diff --git a/scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl b/scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl index 115d137e..ef0de3a4 100644 --- a/scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl +++ b/scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl @@ -1,4 +1,4 @@ {"type":"thread.started","thread_id":"public-fixture"} {"type":"item.completed","item":{"id":"call-1","type":"command_execution","duration_ms":7.25}} -{"type":"turn.completed","status":"completed","usage":{"input_tokens":31,"cached_input_tokens":8,"output_tokens":12,"reasoning_output_tokens":4},"item":{"content":"public fixture content"}} +{"type":"turn.completed","status":"completed","usage":{"input_tokens":31,"cached_input_tokens":8,"cache_write_input_tokens":6,"output_tokens":12,"reasoning_output_tokens":4},"item":{"content":"public fixture content"}} {"type":"adapter.idle","adapter":"codex_iop","nonce":"fixture-nonce-0001","child_exit":0}