fix(benchmark): 공식 caller 종료 계약을 보정한다

2026-08-12 17:22:27 +09:00 · 2026-08-12 17:22:27 +09:00 · 30ac305e18
commit 30ac305e18
parent 731482ee80
16 changed files with 441 additions and 76 deletions
--- a/agent-contract/outer/gemini-compatible-api.md
+++ b/agent-contract/outer/gemini-compatible-api.md
@ -60,11 +60,11 @@ x-goog-api-key: <IOP principal token>

 - `contents[]`: `role`, `parts[].text`, `parts[].functionCall`, `parts[].functionResponse`, optional opaque `thoughtSignature`
 - `systemInstruction`: official `agy`의 `role: user`와 `parts[].text`
- `generationConfig`: `candidateCount`, `maxOutputTokens`, `stopSequences`, `temperature`, `topK`, `topP`, `thinkingConfig.includeThoughts`, `thinkingConfig.thinkingBudget`
- `tools[].functionDeclarations[]`: `name`, `description`, `parametersJsonSchema`
+- `generationConfig`: `candidateCount`, `maxOutputTokens`, `stopSequences`, `temperature`, `topK`, `topP`, `thinkingConfig.includeThoughts`, `thinkingConfig.thinkingBudget`, `responseMimeType`, 그리고 상호 배타적인 `responseSchema`/`responseJsonSchema`
+- `tools[].functionDeclarations[]`: `name`, `description`, 상호 배타적인 `parameters`/`parametersJsonSchema`, optional 상호 배타적인 `response`/`responseJsonSchema`
 - `toolConfig.functionCallingConfig.mode`

-Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message, tool schema와 output cap으로 변환한다. `thinkingConfig`는 Gemini OpenAI-compatible upstream이 요구하는 `extra_body.google.thinking_config`로 보존한다. Gemini 3.6에서 폐기된 `temperature`, `topP`, `topK`는 형식과 범위만 검증하고 Chat upstream에는 전달하지 않는다. 함수명·JSON argument·opaque thought signature는 caller turn 사이에 의미를 바꾸지 않는다. 지원하지 않는 content part, duplicate member, 잘못된 role, malformed function payload 또는 둘 이상의 candidate 요청은 provider dispatch 전에 `400 INVALID_ARGUMENT`으로 거부한다.
+Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message, tool schema와 output cap으로 변환한다. `thinkingConfig`는 Gemini OpenAI-compatible upstream이 요구하는 `extra_body.google.thinking_config`로 보존한다. `responseMimeType=application/json`은 Chat `response_format`으로 변환하며 schema가 있으면 고정 이름의 strict `json_schema`, 없으면 `json_object`를 사용한다. `text/plain`은 schema를 허용하지 않는다. Gemini 3.6에서 폐기된 `temperature`, `topP`, `topK`는 형식과 범위만 검증하고 Chat upstream에는 전달하지 않는다. 함수명·JSON argument·opaque thought signature는 caller turn 사이에 의미를 바꾸지 않는다. 동의어 schema 필드가 동시에 존재하거나 schema가 JSON object가 아니면 거부한다. 지원하지 않는 content part, duplicate member, 잘못된 role, malformed function payload 또는 둘 이상의 candidate 요청도 provider dispatch 전에 `400 INVALID_ARGUMENT`으로 거부한다.

 ## SSE 응답

@ -83,7 +83,7 @@ Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message,
 - step: `{"event":"step_update","step_update":{"state":...,"step_type":...,"usage":{...}}}`
 - terminal: `{"event":"result","result":{"status":"SUCCESS","duration_seconds":...,"num_turns":...,"usage":{...}}}`

-benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 terminal로 인정한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다.
+benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 success terminal로 인정한다. 구조가 유효한 `result.status=ERROR`는 stream incompatibility가 아니므로 success terminal을 만들지 않고 caller process의 non-zero exit를 lifecycle failure authority로 보존한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다.

 ## 오류

@ -105,3 +105,5 @@ HTTP commit 전 오류는 다음 Gemini envelope 한 건으로 반환한다.
 - live proof: official `agy --output-format stream-json` through the dev Edge route-specific base URL, with direct and execution-preset effective binding evidence

 2026-08-12 dev 검증에서 normal/boundary/auth/tool/SSE 회귀 테스트와 공식 `agy` 1.1.12 direct·hybrid 실호출이 통과해 이 계약을 active로 전환했다.
+
+2026-08-12 benchmark 실측 보정에서 official agy planner의 structured-output schema field를 Chat `response_format`으로 변환하고, 구조가 유효한 ERROR result를 parser failure와 분리했다.
--- a/agent-spec/testing/agent-comparison-benchmark.md
+++ b/agent-spec/testing/agent-comparison-benchmark.md
@ -105,6 +105,8 @@ flowchart LR
 - manifest의 matrix cell은 stable id, caller, IOP route kind, requested/effective model과 effort를 가진다. unsupported alias나 effort는 다른 값으로 대체하지 않고 fail-closed한다.
 - run state는 `agent-test/runs/<output-id>/<run-id>/` 아래에 격리되며 manifest digest가 다른 상태를 재개하지 않는다.
 - preflight는 scored attempt가 아니며, 실행 중 실패·timeout·cancel과 scoring 실패는 기존 attempt를 수정하지 않고 보존한다.
+- Claude result가 마지막 active assistant snapshot을 직접 완성하면 adapter가 `finish`와 `idle`을 함께 투영하고, assistant가 이미 finish를 냈으면 result는 `idle`만 투영한다. synthetic API error와 agy ERROR result는 parser malformed로 바꾸지 않고 process terminal에 실패 판정을 맡긴다.
+- lifecycle supervisor는 exit watcher와 출력 reader를 join한 뒤 하나의 child return code를 동결해 lifecycle result와 cleanup receipt가 동일한 exit/signal을 갖게 한다. 불일치 evidence는 resume에서 fail-closed한다.
 - raw credential과 private endpoint는 tracked manifest, event, log, screenshot과 report에 기록하지 않는다.
 - report는 run state의 canonical evidence에서 생성되며 성공하지 않은 결과를 0점으로 변환하거나 동점에 임의 순위를 부여하지 않는다.

@ -125,4 +127,5 @@ flowchart LR

 ## 변경 기록

+- 2026-08-12: official Claude result-direct/API-error 및 agy ERROR terminal을 lifecycle 계약에 맞게 분리하고, timeout cleanup result/receipt가 같은 child exit snapshot을 사용하도록 동기화했다.
 - 2026-08-12: `[bench-01]` 종료 감사에서 확인한 421개 benchmark test, manifest/CLI 계약과 구현 evidence를 기준으로 생성했다.
--- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md
+++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md
@ -1,34 +1,46 @@
-<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=2 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
+<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=3 tag=REVIEW_REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->

-# Code Review Reference - REVIEW_TEST
+# Code Review Reference - REVIEW_REVIEW_TEST

 ## Overview

 date=2026-08-12
-task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=2, tag=REVIEW_TEST
+task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=3, tag=REVIEW_REVIEW_TEST
+
+## Archive Evidence Snapshot
+
+- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log`
+- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log`
+- Verdict: `FAIL`; Required R1 Claude result-direct/API-error terminal projection, Required R2 agy top-level official JSONL, Required R3 timeout result/receipt exit-code race.
+- Retained run evidence: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 `timed_out` with lifecycle exit `143` and receipt exit `null`; C05/C09 successful. Public resume appended a ready=9 preflight but correctly rejected the inconsistent terminal and did not retry a failed cell.

 ## Implementation Item Completion

 | Item | Status |
 |---|---|
-| REVIEW_TEST-1 Claude parser | [ ] |
-| REVIEW_TEST-2 agy settings | [ ] |
-| REVIEW_TEST-3 OpenAI Chat bridge | [ ] |
-| REVIEW_TEST-4 hybrid classification | [ ] |
-| REVIEW_TEST-5 release deploy | [ ] |
-| REVIEW_TEST-6 public smokes/preflight | [ ] |
-| REVIEW_TEST-7 scored run/report | [ ] |
+| REVIEW_REVIEW_TEST-1 Claude terminal projection | [x] |
+| REVIEW_REVIEW_TEST-2 agy request/error compatibility | [x] |
+| REVIEW_REVIEW_TEST-3 lifecycle exit coherence | [x] |
+| REVIEW_REVIEW_TEST-4 complete local verification | [ ] |
+| REVIEW_REVIEW_TEST-5 clean release deployment | [ ] |
+| REVIEW_REVIEW_TEST-6 public smokes/preflight | [ ] |
+| REVIEW_REVIEW_TEST-7 scored run/report | [ ] |

 ## Implementation Evidence

-_Fill with sanitized exact commands, identities, run ids, and terminal/web summaries._
+- `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.lifecycle_test` passed 55 tests after R1-R3 implementation.
+- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` passed 437 tests.
+- `go test -count=1 ./apps/edge/internal/openai` passed with official agy structured-output request coverage.
+- Sequential `go test -count=1` passed every package returned by `go list ./apps/control-plane/... ./apps/edge/... ./apps/node/... ./packages/go/...`, excluding only the declared `agenttask` boundary.
+- Sanitized replay of retained C02/C06/C08 streams now projects C02 as `finish,idle` and both synthetic API-error cells as no success terminal without parser failure.

 ## Review-Only Checklist

- [ ] Verify no credential value, ambient caller config, direct provider call, manifest change, or hidden retry was introduced.
- [ ] Verify the immutable failed run is untouched and exactly one new scored run exists.
- [ ] Verify all R1-R4 regressions, clean deployment identities, ready=9, success=9, web gates, and Markdown report.
- [ ] Append final verdict and routing signals.
+- [ ] Verify every R1-R3 regression exercises the exact production lifecycle path and remains fail-closed for unknown data.
+- [ ] Verify no credential value, ambient caller config, direct provider call, manifest change, hidden retry, or provider reselection was introduced.
+- [ ] Verify the prior run is immutable, exactly one plan-3 scored run exists, and all nine attempts plus web gates succeed.
+- [ ] Verify clean build/deployment identities, 4/4 node and 8/8 provider health, ready=9, and report pointer consistency.
+- [ ] Append final verdict and routing signals, then complete the mandated archive/next-state action.

 ## Section Ownership

--- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md
+++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G10.md
@ -1,56 +1,98 @@
-<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=2 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
+<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=3 tag=REVIEW_REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->

-# Caller/provider compatibility fixes and comparison rerun
+# Official caller terminal compatibility and evidence coherence

 ## For the Implementing Agent

-Resolve R1-R4 without changing benchmark inputs, credentials, route aliases, scoring, or the immutable failed run. Use only the public benchmark CLI for live execution. After local qualification, rebuild and deploy the clean release, run one non-scored hybrid smoke, require ready=9, and execute exactly one new scored run.
+Implement R1-R3 exactly as selected below. Do not alter benchmark inputs, route aliases, credentials, scoring, or retained run evidence. Use only the public benchmark CLI for live caller/provider execution. Run deterministic local qualification first, then clean-build and deploy the release, run non-scored direct/hybrid smokes, require ready=9, and execute exactly one fresh scored run. Fill the implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with sanitized actual evidence and leave both active files in place for review.

 ## Background

-The clean release and ready=9 preflight succeeded. The first scored run `run-20260812T055534Z-402e1f1d5b30` completed nine cells but exposed official caller stream variants, missing isolated agy provider settings, an OpenAI Chat bridge field mismatch, and hybrid stage failures. The evidence is valid and must not be retried or rewritten.
+Plan 2 fixed the first live compatibility set and deployed clean release `04f7c39372ae526a51c5583926aefaf4bdc76394`. Public preflight was ready=9, but the single scored run `run-20260812T074548Z-d15500c16009` exposed three narrower boundary defects: Claude result terminal sequencing, official agy planner request compatibility plus error-terminal classification, and a supervisor exit-watcher/receipt race. The run is immutable and must not be retried or rewritten.
+
+## Archive Evidence Snapshot
+
+- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log`
+- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log`
+- Verdict: `FAIL`; Required R1 Claude result-direct/API-error terminal projection, Required R2 agy top-level official JSONL, Required R3 timeout result/receipt exit-code race.
+- Retained run evidence: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 `timed_out` with lifecycle exit `143` and receipt exit `null`; C05/C09 successful. Public resume appended a ready=9 preflight but correctly rejected the inconsistent terminal and did not retry a failed cell.

 ## Finding Resolution Map

-| ID | Root Cause | Selected Fix | Acceptance |
-|---|---|---|---|
-| R1 | Claude snapshots/result grammar is narrower than observed official output | fix active-id/result/usage parsing and replay tests | real-stream replays plus adapter suite pass |
-| R2 | isolated agy HOME lacks required direct Gemini provider selector | stage exact secret-free per-session settings | invocation tests and official noninteractive smoke pass |
-| R3 | Anthropic bridge uses generic `max_tokens` for OpenAI Chat; synthetic API errors become parser errors | profile-aware token field and Claude API-error classification | bridge/parser regressions plus direct GPT smoke pass |
-| R4 | hybrid failures lack sufficient deterministic regression qualification | exact fail-closed classification tests and live non-scored route smokes; no hidden retry | Gemini/GPT hybrid smoke succeeds before scored run |
+| ID | Reviewer Evidence | Root Cause | Selected Fix | Mode | Acceptance |
+|---|---|---|---|---|---|
+| R1 | C02 emitted `idle` without `finish`; C06/C08 valid API-error terminals returned an empty tuple | Claude result handling does not distinguish result-direct completion and violates the no-observation parser contract | emit `finish,idle` for direct completion, `idle` after prior finish, and `None` for a valid API-error terminal; add lifecycle-backed tests | direct-fix | Claude adapter tests and retained-shape replay pass without out-of-order/malformed classification |
+| R2 | C03's structurally redacted evidence initially showed only top-level keys, but the retained raw stream and caller log prove the payload is nested and the model request first failed with Edge `400 INVALID_ARGUMENT: request body is invalid` | Edge's pinned Gemini request decoder lacks the official agy planner's structured-output request fields; the adapter then maps the caller's nested `result.status=ERROR` to `malformed_event` instead of preserving the upstream/process failure | extend the Gemini bridge's exact structured-output request contract and tests; treat a valid agy ERROR result as non-success terminal evidence without parser failure | direct-fix | Gemini bridge structured-output regression and agy error lifecycle regression pass; non-scored official agy direct smoke succeeds |
+| R3 | C01 result exit `143` differs from authenticated receipt exit `null` | supervisor snapshots returncode before joining the concurrent exit watcher | join watcher/IO, refresh the authoritative child returncode, then write the one receipt; add deterministic race/recovery regression | direct-fix | timeout/recovery test proves result and receipt exit/signal equality and public resume can reconcile |

-## Scope
+## Modified Files Summary

- `scripts/agent_benchmark/claude_iop.py` and tests
- `scripts/agent_benchmark/agy_iop.py` and tests
- `apps/edge/internal/openai/anthropic_bridge.go` and tests
- existing single-request provider/plan/review tests only when needed for R4 classification proof
- active review and id pointer files
-
-Do not modify central common rules/skills, tokens, manifest/task/rubric, provider credentials, route aliases, or archived run evidence.
+- `scripts/agent_benchmark/claude_iop.py`
+- `scripts/agent_benchmark/claude_iop_test.py`
+- `scripts/agent_benchmark/agy_iop.py`
+- `scripts/agent_benchmark/agy_iop_test.py`
+- `scripts/agent_benchmark/lifecycle.py`
+- `scripts/agent_benchmark/lifecycle_test.py` and/or the existing attempts recovery test module containing the closest deterministic oracle
+- `apps/edge/internal/openai/gemini_types.go`
+- `apps/edge/internal/openai/gemini_handler.go`
+- `apps/edge/internal/openai/gemini_handler_test.go`
+- this active review plus existing run/preflight/report pointer files after successful fresh execution

 ## Implementation Checklist

- [ ] [REVIEW_TEST-1] Implement R1 Claude stream compatibility and replay regressions.
- [ ] [REVIEW_TEST-2] Implement R2 isolated agy provider settings and invocation regressions.
- [ ] [REVIEW_TEST-3] Implement R3 OpenAI Chat bridge mapping and synthetic API-error classification.
- [ ] [REVIEW_TEST-4] Add R4 single-request failure classification regressions without retries/reselection.
- [ ] [REVIEW_TEST-5] Run local full verification, commit/push, merge clean release, rebuild/deploy all runtime binaries, and verify four-node health.
- [ ] [REVIEW_TEST-6] Run fresh public non-scored direct/hybrid smokes and ready=9 preflight.
- [ ] [REVIEW_TEST-7] Execute exactly one new scored run, record the new id, and generate the human-readable Markdown report.
- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized evidence.
+- [x] [REVIEW_REVIEW_TEST-1] Implement R1 Claude terminal projection and lifecycle-backed regressions.
+- [x] [REVIEW_REVIEW_TEST-2] Implement R2 official agy structured-output request compatibility and error-terminal classification regressions.
+- [x] [REVIEW_REVIEW_TEST-3] Implement R3 authoritative supervisor exit snapshot and timeout/recovery regression.
+- [ ] [REVIEW_REVIEW_TEST-4] Run the complete Python benchmark suite and relevant Go/runtime regressions from a clean tree.
+- [ ] [REVIEW_REVIEW_TEST-5] Commit/push, merge the clean release branch, rebuild/deploy Edge and every Node, and verify 4/4 nodes plus 8/8 healthy providers.
+- [ ] [REVIEW_REVIEW_TEST-6] Run public non-scored direct and hybrid smoke coverage and a fresh ready=9 preflight.
+- [ ] [REVIEW_REVIEW_TEST-7] Execute exactly one fresh scored run, require success=9 and all web gates, update existing id pointers, and generate the human-readable Markdown report.
+- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized commands, release identities, deployment health, run id, terminal summary, and report path.

 ## Verification

-1. Python benchmark adapter unit/integration suite passes.
-2. Edge OpenAI package and dev-runtime full Go suites pass.
-3. Clean release build identities match on Edge and all Nodes; Control Plane shows 4/4 connected and 8/8 healthy.
-4. Direct GPT and both hybrid public smokes succeed, then public preflight reports ready=9.
-5. Exactly one new scored run reports success=9 and all web gates pass; public report command writes Markdown.
+1. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` passes.
+2. Targeted Claude/agy/lifecycle/attempt recovery regressions pass on the benchmark runner's supported Python.
+3. Clean release validation/build tests pass and all deployed binaries report the same clean VCS revision.
+4. Control Plane reports four connected nodes and eight available healthy providers.
+5. Public non-scored direct/hybrid smokes succeed, then public preflight reports ready=9.
+6. Exactly one plan-3 scored run reports success=9 with all required web validations, and the public report command writes Markdown.
+
+## Constraints and Exclusions
+
+- Never print, persist, or pass provider source keys to callers. Read the managed benchmark principal only from `token/.iop-bench` into a process-local variable.
+- Do not mutate `agent-test/runs/bench-02/run-20260812T074548Z-d15500c16009` or use `--retry-failed` on it.
+- Do not change manifest/task/rubric/checksums, route models/efforts/stages, retry policy, or failure classification to manufacture success.
+- Do not add a retry, provider reselection, alternate route, or direct provider invocation.
+- Parser compatibility must remain exact to the pinned official caller version and fail closed on unknown structures.
+
+## Analysis
+
+### Outcome and Acceptance
+
+The task is complete only when deterministic regressions prove the three boundary corrections, the clean release is deployed across the managed dev runtime, public readiness is 9/9, and one fresh immutable scored run is 9/9 with a generated Markdown report.
+
+### Scope and Ownership
+
+The caller adapters own their exact JSONL interpretation and redaction. The Gemini ingress owns the exact official agy request-to-Chat conversion. The lifecycle supervisor owns the authoritative process exit projection shared by its result and receipt. Deployment and benchmark commands only consume these contracts; they do not reinterpret them.
+
+### Split Judgment
+
+Keep one plan because the scored-run acceptance depends on one indivisible evidence invariant: every caller terminal must project through the shared lifecycle into a coherent immutable attempt before the same release can be accepted. Splitting would require multiple scored reruns or accept an invalid intermediate evidence state.
+
+### Routing
+
+- finalizer=`finalize-task-policy.sh`, mode=`pair`
+- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10`
+- risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product`
+- `review_rework_count=3`; `evidence_integrity_failure=true`
+
+## Dependencies and Execution Order
+
+R1-R3 are independent code fixes but must all pass local regressions before one clean release is built. Deployment must finish before non-scored smokes; smokes and ready=9 must pass before the only fresh scored run.

 ## Final Routing

 - finalizer=`finalize-task-policy.sh pair`
 - build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10`
- review_rework_count=2; evidence_integrity_failure=false
- risks=`temporal_state,boundary_contract,variant_product,stream_protocol`
+- catalog routes=`worker/cloud/G10`, `review/cloud/G10`
--- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log
+++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/code_review_cloud_G10_2.log
@ -0,0 +1,69 @@
+<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=2 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
+
+# Code Review Reference - REVIEW_TEST
+
+## Overview
+
+date=2026-08-12
+task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=2, tag=REVIEW_TEST
+
+## Implementation Item Completion
+
+| Item | Status |
+|---|---|
+| REVIEW_TEST-1 Claude parser | [x] |
+| REVIEW_TEST-2 agy settings | [x] |
+| REVIEW_TEST-3 OpenAI Chat bridge | [x] |
+| REVIEW_TEST-4 hybrid classification | [x] |
+| REVIEW_TEST-5 release deploy | [x] |
+| REVIEW_TEST-6 public smokes/preflight | [x] |
+| REVIEW_TEST-7 scored run/report | [ ] |
+
+## Implementation Evidence
+
+- Feature commit `731482ee` passed the 436-test Python adapter suite, the Edge OpenAI package, and the sequential dev-runtime Go suites before release merge.
+- Release `04f7c39372ae526a51c5583926aefaf4bdc76394` was clean-built and deployed to Edge and all four Nodes. Control Plane reported `nodes=4`, `connected=4`, `providers=8`, `available_healthy=8`.
+- Public preflight `run-20260812T074527Z-b3b7f8bbdde4` reported `ready=9`, `registration_required=0`, `implementation_gap=0`.
+- The one plan-2 scored run is `run-20260812T074548Z-d15500c16009`. It retained two successes, four parser/lifecycle failures, and a C01 timeout whose terminal publication could not be reconciled. A public `resume` without `--retry-failed` appended preflight sequence 3 with `ready=9` but correctly rejected the internally inconsistent terminal evidence.
+- Exact retained outcomes: C02 `out_of_order_event`; C03/C06/C08 `malformed_event`; C01 lifecycle `timed_out` with result exit `143` versus cleanup-receipt exit `null`; C05/C09 `success`.
+
+## Review-Only Checklist
+
+- [x] Verify no credential value, ambient caller config, direct provider call, manifest change, or hidden retry was introduced.
+- [x] Verify the immutable failed run is untouched and exactly one new scored run exists.
+- [ ] Verify all R1-R4 regressions, clean deployment identities, ready=9, success=9, web gates, and Markdown report.
+- [x] Append final verdict and routing signals.
+
+## Section Ownership
+
+Implementation status/evidence is implementation-owned. Review checklist/verdict is review-only.
+
+## Code Review Result
+
+### Overall Verdict
+
+FAIL
+
+### Findings
+
+- Required R1 — Claude terminal projection remains inconsistent with the lifecycle contract.
+  - Evidence: retained C02 emitted metrics followed by `idle` without `finish`, producing `out_of_order_event`; retained C06/C08 returned an empty parser tuple for synthetic API-error terminals, which the lifecycle correctly classified as `malformed_event`.
+  - Root Cause: `ClaudeStreamParser._consume_result` cannot distinguish a result that completes an active assistant snapshot from one already preceded by an assistant `finish`, and uses `()` where the parser contract requires `None` for no observation.
+  - Selected Fix: in `scripts/agent_benchmark/claude_iop.py`, emit `finish,idle` only for result-direct completion, retain `idle` after an already emitted finish, and return `None` for a valid API-error terminal. Add lifecycle-backed regressions for all three sequences.
+- Required R2 — agy 1.1.12 official `stream-json` is rejected before the model call can complete.
+  - Evidence: retained C03 begins with top-level `{"event":"init"}`, then top-level `step_update/state/step_type` events and a `result/status`; the parser requires a nested object at `item[event]` and terminates on the first line.
+  - Root Cause: `AgyEventParser.parse` and `redact_agy_event` implement a nested payload grammar that does not match the pinned caller's observed top-level stream grammar.
+  - Selected Fix: in `scripts/agent_benchmark/agy_iop.py`, validate the exact top-level init/step/result fields emitted by 1.1.12, keep content structurally redacted, preserve fail-closed status/usage handling, and add official-shape lifecycle regressions.
+- Required R3 — timeout cleanup evidence can be durably self-contradictory.
+  - Evidence: C01's immutable `lifecycle-result.json` records `terminal_reason=timed_out`, `exit_code=143`, while its authenticated cleanup receipt records the same terminal with `exit_code=null`; public resume therefore returns `benchmark state is unavailable`.
+  - Root Cause: `_Supervisor.finish` snapshots `child.returncode` before joining the concurrent exit watcher, then writes the receipt; the controller can subsequently consume a later exited frame and publish a different result exit value.
+  - Selected Fix: in `scripts/agent_benchmark/lifecycle.py`, join the watcher before taking the authoritative child exit snapshot for the receipt, and add a deterministic delayed-watcher timeout/recovery regression proving result/receipt equality.
+
+### Routing Signals
+
+- `review_rework_count=3`
+- `evidence_integrity_failure=true`
+
+### Next Step
+
+Archive this failed pair and materialize the prepared `REVIEW_REVIEW_TEST` follow-up pair. Do not retry or rewrite `run-20260812T074548Z-d15500c16009`.
--- a/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log
+++ b/agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/plan_cloud_G10_2.log
@ -0,0 +1,56 @@
+<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=2 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
+
+# Caller/provider compatibility fixes and comparison rerun
+
+## For the Implementing Agent
+
+Resolve R1-R4 without changing benchmark inputs, credentials, route aliases, scoring, or the immutable failed run. Use only the public benchmark CLI for live execution. After local qualification, rebuild and deploy the clean release, run one non-scored hybrid smoke, require ready=9, and execute exactly one new scored run.
+
+## Background
+
+The clean release and ready=9 preflight succeeded. The first scored run `run-20260812T055534Z-402e1f1d5b30` completed nine cells but exposed official caller stream variants, missing isolated agy provider settings, an OpenAI Chat bridge field mismatch, and hybrid stage failures. The evidence is valid and must not be retried or rewritten.
+
+## Finding Resolution Map
+
+| ID | Root Cause | Selected Fix | Acceptance |
+|---|---|---|---|
+| R1 | Claude snapshots/result grammar is narrower than observed official output | fix active-id/result/usage parsing and replay tests | real-stream replays plus adapter suite pass |
+| R2 | isolated agy HOME lacks required direct Gemini provider selector | stage exact secret-free per-session settings | invocation tests and official noninteractive smoke pass |
+| R3 | Anthropic bridge uses generic `max_tokens` for OpenAI Chat; synthetic API errors become parser errors | profile-aware token field and Claude API-error classification | bridge/parser regressions plus direct GPT smoke pass |
+| R4 | hybrid failures lack sufficient deterministic regression qualification | exact fail-closed classification tests and live non-scored route smokes; no hidden retry | Gemini/GPT hybrid smoke succeeds before scored run |
+
+## Scope
+
+- `scripts/agent_benchmark/claude_iop.py` and tests
+- `scripts/agent_benchmark/agy_iop.py` and tests
+- `apps/edge/internal/openai/anthropic_bridge.go` and tests
+- existing single-request provider/plan/review tests only when needed for R4 classification proof
+- active review and id pointer files
+
+Do not modify central common rules/skills, tokens, manifest/task/rubric, provider credentials, route aliases, or archived run evidence.
+
+## Implementation Checklist
+
+- [ ] [REVIEW_TEST-1] Implement R1 Claude stream compatibility and replay regressions.
+- [ ] [REVIEW_TEST-2] Implement R2 isolated agy provider settings and invocation regressions.
+- [ ] [REVIEW_TEST-3] Implement R3 OpenAI Chat bridge mapping and synthetic API-error classification.
+- [ ] [REVIEW_TEST-4] Add R4 single-request failure classification regressions without retries/reselection.
+- [ ] [REVIEW_TEST-5] Run local full verification, commit/push, merge clean release, rebuild/deploy all runtime binaries, and verify four-node health.
+- [ ] [REVIEW_TEST-6] Run fresh public non-scored direct/hybrid smokes and ready=9 preflight.
+- [ ] [REVIEW_TEST-7] Execute exactly one new scored run, record the new id, and generate the human-readable Markdown report.
+- [ ] Fill `CODE_REVIEW-cloud-G10.md` with exact sanitized evidence.
+
+## Verification
+
+1. Python benchmark adapter unit/integration suite passes.
+2. Edge OpenAI package and dev-runtime full Go suites pass.
+3. Clean release build identities match on Edge and all Nodes; Control Plane shows 4/4 connected and 8/8 healthy.
+4. Direct GPT and both hybrid public smokes succeed, then public preflight reports ready=9.
+5. Exactly one new scored run reports success=9 and all web gates pass; public report command writes Markdown.
+
+## Final Routing
+
+- finalizer=`finalize-task-policy.sh pair`
+- build=`grade-boundary/cloud/G10`; review=`official-review/cloud/G10`
+- review_rework_count=2; evidence_integrity_failure=false
+- risks=`temporal_state,boundary_contract,variant_product,stream_protocol`
--- a/apps/edge/internal/openai/gemini_handler.go
+++ b/apps/edge/internal/openai/gemini_handler.go
@ -170,6 +170,33 @@ func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) {
 				}
 			}
 		}
+		responseSchema, err := geminiExclusiveSchema(config.ResponseSchema, config.ResponseJSONSchema)
+		if err != nil {
+			return nil, fmt.Errorf("response schema is invalid")
+		}
+		switch config.ResponseMIMEType {
+		case "":
+			if responseSchema != nil {
+				return nil, fmt.Errorf("response MIME type is required")
+			}
+		case "text/plain":
+			if responseSchema != nil {
+				return nil, fmt.Errorf("text response schema is invalid")
+			}
+		case "application/json":
+			if responseSchema == nil {
+				chat["response_format"] = map[string]any{"type": "json_object"}
+			} else {
+				chat["response_format"] = map[string]any{
+					"type": "json_schema",
+					"json_schema": map[string]any{
+						"name": "agy_response", "strict": true, "schema": responseSchema,
+					},
+				}
+			}
+		default:
+			return nil, fmt.Errorf("response MIME type is invalid")
+		}
 	}
 	if len(req.Tools) > 0 {
 		tools := make([]map[string]any, 0)
@ -178,13 +205,20 @@ func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) {
 				return nil, fmt.Errorf("functionDeclarations are required")
 			}
 			for _, declaration := range group.FunctionDeclarations {
-				if !geminiPathToken.MatchString(declaration.Name) || len(declaration.ParametersJSONSchema) == 0 {
+				if !geminiPathToken.MatchString(declaration.Name) {
 					return nil, fmt.Errorf("function declaration is invalid")
 				}
-				var schema map[string]any
-				if json.Unmarshal(declaration.ParametersJSONSchema, &schema) != nil {
+				schema, err := geminiExclusiveSchema(
+					declaration.Parameters, declaration.ParametersJSONSchema,
+				)
+				if err != nil || schema == nil {
 					return nil, fmt.Errorf("function schema is invalid")
 				}
+				if _, err := geminiExclusiveSchema(
+					declaration.Response, declaration.ResponseJSONSchema,
+				); err != nil {
+					return nil, fmt.Errorf("function response schema is invalid")
+				}
 				function := map[string]any{"name": declaration.Name, "parameters": schema}
 				if declaration.Description != "" {
 					function["description"] = declaration.Description
@ -209,6 +243,59 @@ func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) {
 	return json.Marshal(chat)
 }

+func geminiExclusiveSchema(first, second json.RawMessage) (map[string]any, error) {
+	if len(first) > 0 && len(second) > 0 {
+		return nil, fmt.Errorf("schema alternatives conflict")
+	}
+	raw := first
+	if len(raw) == 0 {
+		raw = second
+	}
+	if len(raw) == 0 {
+		return nil, nil
+	}
+	var schema map[string]any
+	if err := json.Unmarshal(raw, &schema); err != nil || schema == nil {
+		return nil, fmt.Errorf("schema is not an object")
+	}
+	if err := normalizeGeminiSchemaTypes(schema); err != nil {
+		return nil, err
+	}
+	return schema, nil
+}
+
+func normalizeGeminiSchemaTypes(value any) error {
+	switch typed := value.(type) {
+	case map[string]any:
+		for key, item := range typed {
+			if key == "type" {
+				name, ok := item.(string)
+				if !ok {
+					return fmt.Errorf("schema type is invalid")
+				}
+				normalized := strings.ToLower(name)
+				switch normalized {
+				case "null", "boolean", "object", "array", "number", "integer", "string":
+					typed[key] = normalized
+				default:
+					return fmt.Errorf("schema type is invalid")
+				}
+				continue
+			}
+			if err := normalizeGeminiSchemaTypes(item); err != nil {
+				return err
+			}
+		}
+	case []any:
+		for _, item := range typed {
+			if err := normalizeGeminiSchemaTypes(item); err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
 func geminiTextOnly(content geminiContent) (string, error) {
 	// Gemini represents systemInstruction as Content and official agy 1.1.12
 	// labels that Content with the API-native "user" role.
--- a/apps/edge/internal/openai/gemini_handler_test.go
+++ b/apps/edge/internal/openai/gemini_handler_test.go
@ -140,6 +140,49 @@ func TestGeminiRequestBridgeUsesProviderNativeThinkingEnvelope(t *testing.T) {
 	}
 }

+func TestGeminiRequestBridgeAcceptsOfficialAgyPlannerStructuredOutput(t *testing.T) {
+	converted, err := prepareGeminiChatBridge([]byte(`{
+	  "contents":[{"role":"user","parts":[{"text":"plan"}]}],
+	  "generationConfig":{
+	    "responseMimeType":"application/json",
+	    "responseJsonSchema":{"type":"OBJECT","properties":{"steps":{"type":"ARRAY"}},"required":["steps"]}
+	  },
+	  "tools":[{"functionDeclarations":[{
+	    "name":"read_file","parameters":{"type":"OBJECT","properties":{"path":{"type":"STRING"}}},
+	    "responseJsonSchema":{"type":"object"}
+	  }]}]
+	}`), "gemini-3.6-flash")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var body map[string]any
+	if json.Unmarshal(converted, &body) != nil {
+		t.Fatal("converted body is invalid")
+	}
+	format := body["response_format"].(map[string]any)
+	jsonSchema := format["json_schema"].(map[string]any)
+	schema := jsonSchema["schema"].(map[string]any)
+	if format["type"] != "json_schema" || jsonSchema["name"] != "agy_response" || jsonSchema["strict"] != true || schema["type"] != "object" {
+		t.Fatalf("structured output mismatch: %+v", format)
+	}
+	tools := body["tools"].([]any)
+	function := tools[0].(map[string]any)["function"].(map[string]any)
+	if function["name"] != "read_file" {
+		t.Fatalf("function conversion mismatch: %+v", function)
+	}
+}
+
+func TestGeminiRequestBridgeRejectsConflictingStructuredSchemas(t *testing.T) {
+	for _, body := range []string{
+		`{"contents":[{"role":"user","parts":[{"text":"x"}]}],"generationConfig":{"responseMimeType":"application/json","responseSchema":{"type":"object"},"responseJsonSchema":{"type":"object"}}}`,
+		`{"contents":[{"role":"user","parts":[{"text":"x"}]}],"tools":[{"functionDeclarations":[{"name":"f","parameters":{"type":"object"},"parametersJsonSchema":{"type":"object"}}]}]}`,
+	} {
+		if _, err := prepareGeminiChatBridge([]byte(body), "gemini-3.6-flash"); err == nil {
+			t.Fatal("conflicting schema alternatives must fail")
+		}
+	}
+}
+
 func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) {
 	w := httptest.NewRecorder()
 	stream := newGeminiBridgeStream(w, "gemini-3.6-flash")
--- a/apps/edge/internal/openai/gemini_types.go
+++ b/apps/edge/internal/openai/gemini_types.go
@ -39,13 +39,16 @@ type geminiFunctionResponse struct {
 }

 type geminiGenerationConfig struct {
-	CandidateCount  *int                  `json:"candidateCount,omitempty"`
-	MaxOutputTokens *int                  `json:"maxOutputTokens,omitempty"`
-	StopSequences   []string              `json:"stopSequences,omitempty"`
-	Temperature     *float64              `json:"temperature,omitempty"`
-	TopK            *int                  `json:"topK,omitempty"`
-	TopP            *float64              `json:"topP,omitempty"`
-	ThinkingConfig  *geminiThinkingConfig `json:"thinkingConfig,omitempty"`
+	CandidateCount     *int                  `json:"candidateCount,omitempty"`
+	MaxOutputTokens    *int                  `json:"maxOutputTokens,omitempty"`
+	StopSequences      []string              `json:"stopSequences,omitempty"`
+	Temperature        *float64              `json:"temperature,omitempty"`
+	TopK               *int                  `json:"topK,omitempty"`
+	TopP               *float64              `json:"topP,omitempty"`
+	ThinkingConfig     *geminiThinkingConfig `json:"thinkingConfig,omitempty"`
+	ResponseMIMEType   string                `json:"responseMimeType,omitempty"`
+	ResponseSchema     json.RawMessage       `json:"responseSchema,omitempty"`
+	ResponseJSONSchema json.RawMessage       `json:"responseJsonSchema,omitempty"`
 }

 type geminiThinkingConfig struct {
@ -60,7 +63,10 @@ type geminiTool struct {
 type geminiFunctionDeclaration struct {
 	Name                 string          `json:"name"`
 	Description          string          `json:"description,omitempty"`
-	ParametersJSONSchema json.RawMessage `json:"parametersJsonSchema"`
+	Parameters           json.RawMessage `json:"parameters,omitempty"`
+	ParametersJSONSchema json.RawMessage `json:"parametersJsonSchema,omitempty"`
+	Response             json.RawMessage `json:"response,omitempty"`
+	ResponseJSONSchema   json.RawMessage `json:"responseJsonSchema,omitempty"`
 }

 type geminiToolConfig struct {
--- a/docs/agent-comparison-benchmark-dev-guide.md
+++ b/docs/agent-comparison-benchmark-dev-guide.md
@ -187,6 +187,8 @@ agy의 public base는 adapter가 cell별로 다음처럼 route-qualified 한다.

 `GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 benchmark transport가 아니다.

+official agy planner가 Gemini `generationConfig.responseMimeType`과 `responseSchema` 또는 `responseJsonSchema`를 보내면 Edge는 이를 기존 Chat `response_format`으로 변환한다. function declaration은 official SDK가 사용하는 `parameters`와 `parametersJsonSchema` 표기 중 정확히 하나를 허용한다. 동의어 필드가 동시에 있거나 schema가 JSON object가 아니면 provider dispatch 전에 거부한다. 구조가 유효한 agy `result.status=ERROR`는 stream parser 오류로 바꾸지 않고 caller process 실패로 기록한다.
+
 ## 7. Benchmark 고정 설정

 | 설정 | 값 |
--- a/scripts/agent_benchmark/agy_iop.py
+++ b/scripts/agent_benchmark/agy_iop.py
@ -470,7 +470,10 @@ class AgyEventParser:
            return "malformed"
        self._result_seen = True
        if payload.get("status") != "SUCCESS":
-            return "malformed"
+            # A structurally valid caller error is not a stream mismatch. It
+            # emits no success terminal and lets the process exit (normally
+            # non-zero) remain the lifecycle authority.
+            return None
        metrics: list[ParsedMetric] = []
        usage = payload.get("usage", self._latest_usage)
        if usage is not None:
--- a/scripts/agent_benchmark/agy_iop_test.py
+++ b/scripts/agent_benchmark/agy_iop_test.py
@ -30,6 +30,7 @@ from scripts.agent_benchmark.agy_iop import (
 from scripts.agent_benchmark.connectivity import EffectiveBinding, RequestedEffectiveBinding
 from scripts.agent_benchmark.lifecycle import (
    REASON_MALFORMED_EVENT,
+    REASON_NONZERO_EXIT,
    SUBMISSION_ARGV_TASK,
    InvocationSpec,
    env_pairs,
@ -98,10 +99,13 @@ class AgyIopTest(unittest.TestCase):
            self._observation(value),
        )

-    def _run_lines(self, lines: list[str], parser: AgyEventParser):
+    def _run_lines(self, lines: list[str], parser: AgyEventParser, *, exit_code: int = 0):
        evidence = self.root / f"evidence-{len(list(self.root.glob('evidence-*')))}"
        evidence.mkdir()
-        source = "lines=" + repr(lines) + "; [print(line) for line in lines]"
+        source = (
+            "import sys; lines=" + repr(lines)
+            + "; [print(line) for line in lines]; raise SystemExit(" + repr(exit_code) + ")"
+        )
        spec = InvocationSpec(
            argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
            env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
@ -235,13 +239,22 @@ class AgyIopTest(unittest.TestCase):
        for lines, reason in (
            ([json.dumps(result)], REASON_MALFORMED_EVENT),
            ([json.dumps({"event": "init", "init": {}}), json.dumps(result), json.dumps(result)], REASON_MALFORMED_EVENT),
-            ([json.dumps({"event": "init", "init": {}}), json.dumps({**result, "result": {**result["result"], "status": "ERROR"}})], REASON_MALFORMED_EVENT),
        ):
            parser = AgyEventParser(_cell(), _binding())
            invocation = self._run_lines(lines, parser)
            self.assertFalse(invocation.success)
            self.assertEqual(invocation.terminal_reason, reason)

+        parser = AgyEventParser(_cell(), _binding())
+        error_result = json.dumps({
+            "event": "result", "result": {"status": "ERROR", "response": "private"},
+        })
+        invocation = self._run_lines([
+            json.dumps({"event": "init", "init": {}}), error_result,
+        ], parser, exit_code=1)
+        self.assertFalse(invocation.success)
+        self.assertEqual(invocation.terminal_reason, REASON_NONZERO_EXIT)
+
    def test_latest_step_usage_is_used_only_when_result_omits_usage(self) -> None:
        parser = AgyEventParser(_cell(), _binding())
        lines = [
--- a/scripts/agent_benchmark/claude_iop.py
+++ b/scripts/agent_benchmark/claude_iop.py
@ -330,7 +330,7 @@ class ClaudeStreamParser:
        self._continuation_pending = False
        self._assistant_messages += 1

-    def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...]:
+    def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...] | None:
        if self._phase == "await_error_result":
            self._require_bound_session(event)
            if (
@ -344,18 +344,21 @@ class ClaudeStreamParser:
            # Do not emit finish/idle for an upstream API failure. The caller's
            # non-zero exit is the lifecycle terminal; an unexpected zero exit
            # still fails closed as missing terminal evidence.
-            return ()
+            return None
+        result_completes_active_message = False
        if self._phase == "await_assistant" and self._active_message_id is not None:
            if self._continuation_pending:
                raise ClaudeIopProtocolError("Claude result followed an unresolved tool result")
            self._complete_active_message()
+            result_completes_active_message = True
        elif self._phase != "await_result":
            raise ClaudeIopProtocolError("duplicate or out-of-order Claude result")
        self._require_bound_session(event)
        if event.get("subtype") != "success" or event.get("is_error") is True:
            raise ClaudeIopProtocolError("invalid Claude result terminal")
        self._phase = "complete"
-        return (*self._observations(event), "idle")
+        terminal = ("finish", "idle") if result_completes_active_message else ("idle",)
+        return (*self._observations(event), *terminal)

    def _observations(self, event: dict[str, Any]) -> tuple[ParsedMetric, ...]:
        """Convert only allowlisted reported Claude values into observations."""
--- a/scripts/agent_benchmark/claude_iop_test.py
+++ b/scripts/agent_benchmark/claude_iop_test.py
@ -100,7 +100,7 @@ class ClaudeIopTest(unittest.TestCase):
        fixture = Path("scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl")
        return fixture.read_text(encoding="utf-8").splitlines()

-    def _run_fake(self, lines: list[str]):
+    def _run_fake(self, lines: list[str], *, exit_code: int = 0):
        self.binary.write_text(textwrap.dedent(f"""\
            #!/usr/bin/env python3
            import os, sys
@ -109,6 +109,7 @@ class ClaudeIopTest(unittest.TestCase):
            assert os.environ["ANTHROPIC_API_KEY"] == {SENTINELS[2]!r}
            for line in {lines!r}:
                print(line)
+            raise SystemExit({exit_code!r})
        """), encoding="utf-8")
        evidence = self.root / f"evidence-{len(tuple(self.root.glob('evidence-*')))}"
        evidence.mkdir()
@ -271,9 +272,9 @@ class ClaudeIopTest(unittest.TestCase):
        ]
        self.assertEqual(parsed, [None] * len(parsed))
        observations = parser("stdout", json.dumps(terminal))
-        self.assertEqual(observations[-1], "idle")
+        self.assertEqual(observations[-2:], ("finish", "idle"))
        self.assertEqual(
-            {metric.name: metric.value for metric in observations[:-1]}["model_calls"], 1
+            {metric.name: metric.value for metric in observations[:-2]}["model_calls"], 1
        )

    def test_parser_classifies_synthetic_api_error_without_parser_failure(self) -> None:
@ -291,9 +292,25 @@ class ClaudeIopTest(unittest.TestCase):
        })
        self.assertEqual(
            [parser("stdout", line) for line in (init, synthetic, terminal)],
-            [None, None, ()],
+            [None, None, None],
        )

+        outcome, _ = self._run_fake([init, synthetic, terminal], exit_code=1)
+        self.assertFalse(outcome.success)
+        self.assertNotIn(outcome.terminal_reason, ("malformed_event", "parser_error"))
+
+    def test_lifecycle_accepts_result_direct_active_snapshot(self) -> None:
+        init, _, result = self._fixture_lines()
+        partial = json.dumps({
+            "type": "assistant", "session_id": "claude-session-fixture",
+            "message": {"id": "msg-direct", "model": "claude-sonnet",
+                        "stop_reason": None, "content": []},
+        })
+        outcome, _ = self._run_fake([init, partial, result])
+        self.assertTrue(outcome.success, outcome)
+        kinds = [event.kind for event in outcome.events]
+        self.assertLess(kinds.index("finish"), kinds.index("idle"))
+
    def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None:
        init, _, result = self._fixture_lines()
        parser = ClaudeStreamParser(self.cell, "session-fixture")
--- a/scripts/agent_benchmark/lifecycle.py
+++ b/scripts/agent_benchmark/lifecycle.py
@ -965,13 +965,18 @@ class _Supervisor:
            if self.child is not None:
                grace = int(self.spec.get("cleanup_grace_seconds", 5))
                self._terminate_group(grace)
+            io_complete = self._join_io_threads()
+            descendants_reaped = self._reap_owned_descendants()
+            if self.child is not None:
+                # The exit watcher publishes the controller's exited frame.
+                # Join it before freezing the receipt so both durable views
+                # observe the same authoritative child return code.
+                self.child.poll()
                code = self.child.returncode
                if code is not None:
                    exit_code = code if code >= 0 else None
                    signal_num = -code if code < 0 else None
                group_alive = _group_alive(self.pgid)
-            io_complete = self._join_io_threads()
-            descendants_reaped = self._reap_owned_descendants()
            outcome = {
                "reason": reason,
                "exit_code": exit_code,
--- a/scripts/agent_benchmark/lifecycle_test.py
+++ b/scripts/agent_benchmark/lifecycle_test.py
@ -832,6 +832,8 @@ class LifecycleTest(unittest.TestCase):
                self.assertEqual(result.terminal_reason, expected)
                self.assertEqual(receipt["reason"], expected)
                self.assertEqual(published["terminal_reason"], expected)
+                self.assertEqual(receipt["exit_code"], published["exit_code"])
+                self.assertEqual(receipt["signal"], published["signal"])
                self.assertEqual(len(terminals), 1)
                self.assertEqual(terminals[0]["terminal_reason"], expected)
                self.assertEqual(journal[-1]["record"], "terminal")