toki/iop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(benchmark): 원샷 비교 실행 실패를 해소한다

Browse source 
호출기별 격리 쓰기 계약과 Gemini ingress, 단일 요청 stage 처리를 맞춰 실제 9-cell 비교가 생성물을 남길 수 있게 한다.
This commit is contained in:
toki 2026-08-12 14:37:17 +09:00
parent 8dcf2a3246
commit 39fa1da55b
70 changed files with 6267 additions and 546 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -37,5 +37,6 @@ Start with `agent-contract/index.md` for protocol and runtime contracts, and `ag
Operator and client setup guides:
- [Edge Local Quickstart](docs/edge-local-dev-guide.md)
- [Agent Comparison Benchmark Dev Guide](docs/agent-comparison-benchmark-dev-guide.md)
- [dev OpenCode Settings Guide](docs/dev-opencode-settings-guide.md)
- [dev-corp Pi Settings Guide](docs/dev-corp-pi-settings-guide.md)

1
agent-contract/index.md
View file

@ -14,6 +14,7 @@
|----|-----------|-----------|------|
| `iop.openai-compatible-api` | OpenAI-compatible API, Responses API, Chat Completions, legacy Completions, error envelope/SSE terminal error, `model` route, managed projection principal auth and slot-route binding, managed-versus-legacy provider credential selection, model-driven passthrough/normalized routing, provider-pool admission/unavailable error, safe credential-slot attribution, standard metadata, and provider-native extension fields such as `chat_template_kwargs` | `apps/edge/internal/openai/*`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/service/provider_tunnel.go`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/openai-compatible-api.md` |
| `iop.anthropic-compatible-api` | Anthropic Messages API, count_tokens, models list, bearer or `X-Api-Key` principal auth, active managed projection auth and slot-route binding, `anthropic-version` routing, native Anthropic tunnel, Chat bridge, provider-pool-only admission, profile capability checks, managed-versus-legacy provider credentials, marked-preset single-request admission with Edge-owned internal Plan/Review template customization that leaves caller I/O unchanged, and current no-OpenAI-metric status | `apps/edge/internal/openai/anthropic_handler.go`, `apps/edge/internal/openai/anthropic_native.go`, `apps/edge/internal/openai/anthropic_bridge.go`, `apps/edge/internal/openai/anthropic_stream.go`, `apps/edge/internal/openai/anthropic_types.go`, `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/openai/provider_tunnel.go`, `apps/edge/internal/openai/provider_model_rewrite.go`, `apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_plan_stage.go`, `apps/edge/internal/openai/single_request_review_stage.go`, `packages/go/singlerequesttemplate/template.go`, `packages/go/config/protocol_profile.go` | `agent-contract/outer/anthropic-compatible-api.md` |
| `iop.gemini-compatible-api` | Gemini Developer API `streamGenerateContent`, route-qualified Gemini-native ingress, `x-goog-api-key` principal auth, `GOOGLE_GEMINI_BASE_URL`, official agy 1.1.12 API-key transport, Gemini function calls/thought signatures/SSE, and direct-versus-execution-preset binding | `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/live_iop.py` | `agent-contract/outer/gemini-compatible-api.md` |
| `iop.a2a-json-rpc-api` | A2A JSON-RPC API, `message/send`, `tasks/get`, `tasks/cancel`, A2A task state, agent card, `a2a.bearer_token`, Edge A2A input surface | `apps/edge/internal/input/a2a/*`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/a2a-json-rpc-api.md` |
## Inner Contracts

107
agent-contract/outer/gemini-compatible-api.md Normal file
View file

@ -0,0 +1,107 @@
# Gemini-Compatible API Contract
## 계약 메타
- id: `iop.gemini-compatible-api`
- boundary: `outer`
- status: active
- 원본 경로:
- `apps/edge/internal/openai/routes.go`
- `apps/edge/internal/openai/principal.go`
- `apps/edge/internal/openai/chat_handler.go`
- `scripts/agent_benchmark/agy_iop.py`
- `scripts/agent_benchmark/live_iop.py`
- external caller surface: official Antigravity CLI `agy` 1.1.12 Gemini API-key provider
## 읽는 조건
- Gemini Developer API `streamGenerateContent`, `x-goog-api-key`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `agy` API-key provider, Gemini-native tool/function call, 또는 Gemini-native SSE ingress를 구현·검증할 때 읽는다.
- IOP execution preset을 Gemini-native caller에 노출하거나 `agy` benchmark transport를 변경할 때 읽는다.
## 범위
IOP Edge가 외부 Gemini-native caller에게 제공하는 초기 호환 표면은 다음 streaming endpoint다.
```http
POST /gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse
Content-Type: application/json
x-goog-api-key: <IOP principal token>
```
- `{route-id}`는 Edge가 인증된 principal에 대해 해석할 direct route 또는 virtual execution-preset id다. URL path segment 하나의 canonical token이어야 한다.
- `{caller-model}`은 caller가 선택한 Gemini 모델 id이며 관측·호환성 검증 대상이다. provider/credential 또는 execution preset 선택 권한은 갖지 않는다.
- 현재 `agy` 호환 표면은 `alt=sse`인 `streamGenerateContent`만 지원한다. `generateContent`, batch, files, cached content, tuning API는 비범위다.
- direct route와 marked single-request preset은 모두 기존 Edge route resolution, managed admission, provider-pool, preset coordinator를 사용한다. 별도 Gemini 전용 우회 dispatch를 만들지 않는다.
## Caller 설정
공식 `agy` 1.1.12 API-key provider는 다음 값으로 실행한다.
- `~/.gemini/antigravity-cli/settings.json`: `modelProvider`는 `gemini`다.
- `GEMINI_API_KEY`: upstream provider key가 아니라 IOP principal token이다.
- `GOOGLE_GEMINI_BASE_URL`: `https://<edge>/gemini/{route-id}`다.
- 사설 dev CA를 사용하는 경우 caller child에는 표준 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`만 명시적으로 전달한다.
- `--model`: 공식 CLI가 인식하는 Gemini 모델 label을 사용한다. benchmark의 Gemini 3.6 Flash 호출은 `Gemini 3.6 Flash`다.
- `--effort`는 API-key provider 호출에 전달하지 않는다. 요청된 high effort는 인증된 IOP route/preset의 effective binding으로 검증한다.
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 계약의 transport가 아니다.
## Auth 및 credential 경계
- `x-goog-api-key`는 Gemini ingress에서 IOP caller 인증 헤더다. Edge는 이를 SHA-256 projection match에만 사용하고 raw 값을 log, metric, response, task evidence에 남기지 않는다.
- `Authorization: Bearer`를 함께 보내면 두 token은 constant-time 비교로 같아야 한다. 다르거나 malformed이면 provider dispatch 전에 Gemini 오류 envelope로 `401`을 반환한다.
- managed mode에서 provider credential은 projected slot과 sealed lease에서만 온다. inbound `x-goog-api-key`를 upstream `Authorization` 또는 upstream `x-goog-api-key`로 전달하지 않는다.
- legacy mode에서도 inbound `x-goog-api-key`를 provider credential로 재사용하지 않는다. provider auth가 별도로 필요하면 기존 명시적 legacy provider-auth 계약만 적용한다.
- marked single-request preset은 managed projection과 고정 stage authorization 없이는 fail closed한다.
## 요청 변환
초기 호환 범위는 official `agy` 1.1.12가 보내는 다음 top-level field다.
- `contents[]`: `role`, `parts[].text`, `parts[].functionCall`, `parts[].functionResponse`, optional opaque `thoughtSignature`
- `systemInstruction`: official `agy`의 `role: user`와 `parts[].text`
- `generationConfig`: `candidateCount`, `maxOutputTokens`, `stopSequences`, `temperature`, `topK`, `topP`, `thinkingConfig.includeThoughts`, `thinkingConfig.thinkingBudget`
- `tools[].functionDeclarations[]`: `name`, `description`, `parametersJsonSchema`
- `toolConfig.functionCallingConfig.mode`
Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message, tool schema와 output cap으로 변환한다. `thinkingConfig`는 Gemini OpenAI-compatible upstream이 요구하는 `extra_body.google.thinking_config`로 보존한다. Gemini 3.6에서 폐기된 `temperature`, `topP`, `topK`는 형식과 범위만 검증하고 Chat upstream에는 전달하지 않는다. 함수명·JSON argument·opaque thought signature는 caller turn 사이에 의미를 바꾸지 않는다. 지원하지 않는 content part, duplicate member, 잘못된 role, malformed function payload 또는 둘 이상의 candidate 요청은 provider dispatch 전에 `400 INVALID_ARGUMENT`으로 거부한다.
## SSE 응답
- 성공 stream은 `Content-Type: text/event-stream`과 `data: <Gemini GenerateContentResponse JSON>` frame을 사용한다.
- text delta는 `candidates[0].content.parts[].text`, reasoning delta는 `thought=true`인 part, 완성된 tool call은 `functionCall` part로 투영한다.
- tool-call argument fragment는 Edge가 bounded buffer에서 완성된 JSON object로 검증한 뒤 한 번만 공개한다. malformed·oversize argument는 성공 tool call로 내보내지 않는다.
- OpenAI `stop`, `length`, `tool_calls` terminal은 Gemini `finishReason`의 `STOP`, `MAX_TOKENS`, `STOP`으로 닫는다. stream 종료 뒤 별도 합성 `system idle` event를 만들지 않는다.
- provider-reported usage가 있으면 `usageMetadata.promptTokenCount`, `candidatesTokenCount`, `thoughtsTokenCount`, `cachedContentTokenCount`, `totalTokenCount`의 존재하는 값만 투영한다. 누락 값을 0으로 발명하지 않는다.
- caller disconnect는 기존 request cancellation 경계를 사용하며 이후 frame을 쓰지 않는다.
## 공식 agy stream-json lifecycle
`agy` 1.1.12의 각 JSONL record는 `event` discriminator와 같은 이름의 중첩 payload를 사용한다.
- init: `{"event":"init","init":{...}}`
- step: `{"event":"step_update","step_update":{"state":...,"step_type":...,"usage":{...}}}`
- terminal: `{"event":"result","result":{"status":"SUCCESS","duration_seconds":...,"num_turns":...,"usage":{...}}}`
benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 terminal로 인정한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다.
## 오류
HTTP commit 전 오류는 다음 Gemini envelope 한 건으로 반환한다.
```json
{"error":{"code":400,"message":"request is invalid","status":"INVALID_ARGUMENT"}}
```
- 인증 실패는 `401 UNAUTHENTICATED`, route/요청 검증 실패는 `400 INVALID_ARGUMENT`, runtime/provider 실패는 `502 UNAVAILABLE`의 고정된 caller-safe message를 사용한다.
- stream commit 뒤 오류는 Gemini `error` payload 한 건으로 끝내며 raw provider body, endpoint, route binding, credential/slot/lease id, prompt, tool argument/result를 포함하지 않는다.
## 변경 시 확인할 코드와 테스트
- route/auth: `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`
- Gemini request/SSE bridge: `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`
- Edge regression: `apps/edge/internal/openai/gemini_handler_test.go`, existing Chat/preset/auth tests
- caller adapter: `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/agy_iop_test.py`, `scripts/agent_benchmark/live_iop.py`
- live proof: official `agy --output-format stream-json` through the dev Edge route-specific base URL, with direct and execution-preset effective binding evidence
2026-08-12 dev 검증에서 normal/boundary/auth/tool/SSE 회귀 테스트와 공식 `agy` 1.1.12 direct·hybrid 실호출이 통과해 이 계약을 active로 전환했다.

6
agent-contract/outer/openai-compatible-api.md
View file

@ -54,8 +54,10 @@ Edge 설정에 `openai.principal_tokens[]`가 설정된 경우, caller는 기존
In managed mode, OpenAI-compatible routes authenticate `Authorization: Bearer <IOP token>` by hashing the token and matching the projected digest. Static principal mappings and the legacy bearer are prohibited by configuration and never act as fallbacks. Unknown or removed digests, malformed headers, and expired snapshots return `401 unauthorized` before model lookup or dispatch. Expiry never returns the process to legacy behavior.
When managed mode is active, model discovery (`GET /v1/models`) lists only active
projected `route_id`s for the authenticated principal. Request model resolution binds
the request strictly to one projected route's `slot_id`, `profile_id`, and `upstream_model`.
projected public route identities for the authenticated principal: `route_alias` when
it is non-empty, otherwise `route_id`. Request model resolution accepts that same
public identity and binds the request strictly to one projected route's `slot_id`,
`profile_id`, and `upstream_model`.
Unknown, inactive, or cross-principal routes never fall back to legacy `model_routes`,
global catalog, or single-target default.

6
agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md
View file

@ -60,7 +60,9 @@
실행 전에 공정한 fixture와 실제 IOP route/credential 상태를 고정한다.
- [x] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
- [ ] [route-readiness] dev `../iop-s2`에서 Claude Code·agy·Codex의 IOP 인증, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 preflight를 통과했는지 확인한다.
- [x] [agy-iop-compatibility] 공식 `agy 1.1.12`의 Gemini API-key provider가 route별 `GOOGLE_GEMINI_BASE_URL`과 `GEMINI_API_KEY=<IOP principal token>`으로 Edge의 Gemini-native `streamGenerateContent` ingress를 호출하고, Gemini request/tool/SSE를 기존 direct·execution-preset 실행에 연결하며 실제 `stream-json` lifecycle/usage를 benchmark adapter가 수집하도록 구현한다. `--effort`와 비공식 custom model에 의존하지 않고 high effort는 IOP effective binding으로 검증한다.
- [x] [managed-credential-dev] dev Control Plane·Edge·Node에 CA-signed mTLS, Edge HTTPS, at-rest/issuer/recipient key material, principal projection과 Gemini/Claude/GPT slot-route를 operator-owned secret 경로로 구성한다. legacy static credential source를 제거하고 marked hybrid preset이 같은 principal의 고정 stage route를 managed lease로 실행하는지 secret-safe smoke로 확인한다.
- [x] [route-readiness] dev `../iop-s2`에서 Claude Code·Codex의 호환 ingress와 공식 agy의 Gemini API-key transport가 각각 IOP 인증 경계를 통과하고, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 실제 caller preflight를 통과했는지 확인한다. 존재하지 않는 caller 환경 변수나 합성 event fixture는 live 호환 근거로 인정하지 않는다.
- [x] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
### Epic: [comparison-runs] 9개 원샷 실행
@ -103,7 +105,7 @@
- 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2`
- 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다.
- 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다.
- 현재 차단: `route-readiness`는 clean `../iop-s2` dev HEAD의 `iop-edge` artifact가 현재 HEAD에서 빌드되지 않아 artifact freshness gate에서 fail-closed했다. testbed artifact를 현재 HEAD로 다시 빌드한 뒤 caller/version/help와 C01-C09 public preflight 전체를 재실행한다.
- 현재 차단: readiness는 완료됐고 all-cell preflight는 `ready=9`였다. 승인된 C01-C09 `run`은 2026-08-12에 한 번 호출됐으나 첫 slot의 caller launch 전 control socket 등록이 workspace filesystem의 symlink-path `bind(2)` `EINVAL`로 중단됐다. caller는 실행되지 않았고 dangling attempt는 harness reconcile로 `interrupted=1`, `running=0`이 됐다. socket 등록과 pre-registration reconcile 결함은 회귀 테스트와 함께 수정했지만 repetitions=1·no-retry 정책 때문에 같은 scored run을 재실행하거나 resume하지 않는다. C01-C09를 다시 시작하려면 기존 실패를 덮지 않는 새 실행 승인과 SDD/manifest run 정책 결정이 필요하다.
- 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md)
- 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md)
- 확인 필요: 없음

14
agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md
View file

@ -24,6 +24,8 @@
- [x] [D08] token은 input/output/reasoning/cached/total과 source를 model/stage별로 기록하고 미제공 값을 exact로 추정하지 않는다.
- [x] [D09] 결과 identity를 가린 뒤 동일 100점 rubric으로 Codex가 채점하고 자동 검증과 수동 점수를 분리한다.
- [x] [D10] scored failure는 보존하고 재실행은 새 attempt로 기록하며 성공 결과만 골라 대표하지 않는다.
- [x] [D11] 공식 `agy 1.1.12`는 Gemini API-key provider의 route별 `GOOGLE_GEMINI_BASE_URL`을 IOP Edge로 지정하고 `GEMINI_API_KEY`에는 upstream key가 아닌 IOP principal token을 넣는다. `--effort`와 비공식 custom model은 사용하지 않고 high effort는 IOP effective binding으로 검증한다.
- [x] [D12] marked hybrid preset은 dev managed credential plane의 fresh projection, 고정 stage authorization과 sealed provider lease가 준비된 뒤에만 실행하며 legacy credential fallback을 허용하지 않는다.
## 문제 / 비목표
@ -44,7 +46,7 @@
| Fixture | versioned prompt, 이미지 2장과 vanilla workspace checksum | 모든 cell의 동일 입력 기준 |
| Evidence | `agent-test/runs/<run-id>/` | attempt별 timeline, usage, validation, screenshot와 score |
| Report | `agent-test/dev/iop-one-shot-agent-comparison-<date>.md` | 현재 프로젝트의 사람이 읽는 비교 결과 |
| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 |
| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 |
| Config Contract | [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) | model route, preset, protocol profile과 credential 경계 |
| User Decision | D01-D10 | 2026-08-06 확정 방향, 추가 사용자 결정 없음 |
@ -53,7 +55,7 @@
| 상태 | 진입 조건 | 다음 상태 | 근거 |
|------|-----------|-----------|------|
| `blocked` | `[route-02]` smoke 또는 `[bench-01]` 완료 전 | `preflighting`, 종료 | active Milestone 상태와 pipeline evidence |
| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | redacted preflight matrix |
| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | official caller transport, managed projection/lease와 redacted preflight matrix |
| `ready` | fixture와 C01-C09 immutable manifest 확정 | `running`, `cancelled` | manifest/fixture/rubric digest |
| `running` | seed 순서에 따라 각 cell에 사용자 작업 1회 제출 | `validating`, `failed`, `timed_out`, `cancelled` | cell/attempt event timeline |
| `validating` | cell finish/complete 후 idle 확정 | `scoring`, `failed` | workspace, build/render/test evidence |
@ -75,13 +77,14 @@ State invariant:
## Interface Contract
- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md)
- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md)
- 입력:
- `fixture`: 동일 이미지 2장, one-page 요구사항, vanilla HTML/CSS/JS initial workspace와 checksum이다.
- `cells`: C01-C09의 caller, IOP route/preset, expected model/stage와 effort binding이다.
- `repetitions=1`, `session_policy=fresh`, `setup_cache_policy`: 초기 scored attempt 수, conversation/resume 격리와 공통 setup/cache 기준이다.
- `environment=dev`, `testbed=../iop-s2`: 실제 IOP runtime 선택이다.
- `completion`: caller별 finish/complete event와 idle 판정 규칙이다.
- `agy`: `modelProvider=gemini`, route별 `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY=<IOP principal token>`과 official `stream-json` event다. API-key provider가 지원하지 않는 `--effort`는 전달하지 않는다.
- 측정 출력:
- timestamp: submitted, first output, first file write, model/stage start/end, tool start/end, finish, idle의 monotonic 값과 observation source다. overlap과 unavailable을 명시한다.
- usage: call count, input/output/reasoning/cached/total token과 source다.
@ -111,6 +114,8 @@ State invariant:
| S10 | `quality-scoring` | identity가 제거된 9개 결과 | Codex rubric 평가 | 항목별 점수/근거와 총점이 자동 gate와 분리되어 기록된다. |
| S11 | `performance-usage` | 모든 attempt timeline/usage | 비교 집계 | 첫 output·첫 write·model/tool/queue/total 시간의 clock/source·overlap, 호출 수와 token/source가 cell·stage별 표가 된다. |
| S12 | `benchmark-report` | S01-S11 evidence | 보고서 생성 | 조건·버전·9개 결과·속도·token·품질·실패·한계와 raw evidence 링크가 Markdown에 남는다. |
| S13 | `agy-iop-compatibility` | official `agy 1.1.12`와 dev Edge | direct·hybrid route별 Gemini base URL로 실제 API-key 호출 | 두 호출 모두 `x-goog-api-key` IOP principal auth, Gemini-native request/tool/SSE, official `stream-json` finish/exit와 config-owned effective binding evidence를 남기고 upstream key 직접 호출이나 합성 event에 의존하지 않는다. |
| S14 | `managed-credential-dev` | dev Control Plane·Edge·Node와 operator-owned security material | managed credential profile로 재기동하고 slot/route를 등록 | CA-signed mTLS·Edge HTTPS·fresh projection·sealed lease가 확인되고 legacy credential source나 cross-route fallback 없이 direct와 marked preset stage가 실행된다. |
## Evidence Map
@ -128,6 +133,8 @@ State invariant:
| S10 | blind mapping 분리와 Codex rubric worksheet | `agent-task/m-iop-one-shot-agent-model-comparison/quality-scoring/` | `quality-scoring` 100-point evidence |
| S11 | cell/stage별 normalized timeline, calls와 token-source table | `agent-task/m-iop-one-shot-agent-model-comparison/performance-usage/` | `performance-usage` speed/token evidence |
| S12 | `agent-test/dev/` Markdown과 raw run links | `agent-task/m-iop-one-shot-agent-model-comparison/benchmark-report/` | `benchmark-report` complete comparison evidence |
| S13 | official agy request-shape capture, Edge Gemini bridge tests, direct·hybrid live preflight와 sanitized lifecycle/usage | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `agy-iop-compatibility` official 1.1.12 IOP transport evidence |
| S14 | dev config check, TLS/workload identity, projection generation, slot-route/lease attribution과 post-revoke no-fallback smoke | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `managed-credential-dev` secure composition and hybrid admission evidence |
공통 완료 검증은 C01-C09 모두가 success/failure/blocked 중 하나의 terminal evidence를 가지고, 성공 결과의 자동 gate·screenshot·blind score와 모든 attempt의 timing/usage source가 보고서에 연결되는지 확인한다. 필수 credential/model이 없으면 raw secret을 요구하거나 기록하지 않고 운영 절차로 등록을 요청한다.
@ -145,6 +152,7 @@ State invariant:
## 사용자 리뷰 이력
- 2026-08-06: 사용자가 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 비교군, Claude Code·agy·Codex caller, finish/idle 원샷, 초기 1회, dev `../iop-s2`, 동일 정적 웹 fixture와 시간·token·Codex 품질 평가를 확정했다.
- 2026-08-12: 공식 `agy 1.1.12` API-key provider의 실제 Gemini-native 요청과 `stream-json` event를 확인했고, 사용자의 provider 직접 설정 지시에 따라 upstream key와 IOP principal token을 분리하며 dev managed credential plane까지 구성하는 D11-D12를 기술 보강했다.
## 작업 컨텍스트

200
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G06_5.log Normal file
View file

@ -0,0 +1,200 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=5 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - REVIEW_TEST
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, run public benchmark `run|resume`, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=5, tag=REVIEW_TEST
## Archive Evidence Snapshot
- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다.
- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다.
- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다.
- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다.
- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다.
## For the Review Agent
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
Review completion means the following steps are finished:
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_5.log` and `PLAN-local-G06.md` → `plan_local_G06_5.log`.
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
---
## Implementation Item Completion
| Item | Status |
|---|---|
| FIX-1: Current Codex usage schema | [x] |
| FIX-2: Config-owned effective binding consumption | [x] |
## Implementation Checklist
- [x] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals.
- [x] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G06_5.log`.
- [x] Archive active `PLAN-*-G??.md` to `plan_local_G06_5.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove empty active parent or verify it was kept due to remaining siblings/files.
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
## Deviations from Plan
없음. public benchmark `run|resume|score`, caller/provider 직접 호출, run tree 수정 없이 plan의 다섯 source/test/fixture 파일만 수정했다.
## Key Design Decisions
- Codex의 `cache_write_input_tokens`는 caller가 명시적으로 보고한 값만 canonical `cache_write_tokens`로 보존하고 `total_tokens`는 합성하지 않는다.
- independently validated config observation을 execution/scoring의 canonical binding으로 사용한다. optional caller binding은 absent일 수 있지만, 보고된 non-`None` 값은 admitted tuple과 정확히 일치해야 한다.
- scoring result에는 admitted tuple을 기록해 downstream이 caller observation 유무와 무관하게 검증된 effective binding을 소비하도록 했다.
## Reviewer Checkpoints
- Verify `_CODEX_USAGE_FIELDS` accepts only the current explicit keys and maps `cache_write_input_tokens` to `cache_write_tokens` without deriving `total_tokens`.
- Verify the tracked Codex fixture no longer relies on a fabricated `iop_effective_binding` event for production execution correctness.
- Verify execution and scoring both accept `None` caller observation only after ready config admission and still reject any contradictory non-`None` observation.
- Verify no attempt recovery, run tree, provider, remote runtime, or unrelated caller behavior changed.
- Verify no public `run|resume|score` or direct caller/provider command was executed in this follow-up.
## Verification Results
### Focused parser and integration coverage
Command:
```bash
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
```
Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection.
Actual stdout/stderr:
```text
.........................................
----------------------------------------------------------------------
Ran 41 tests in 14.500s
OK
```
### Full benchmark suite
Command:
```bash
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
```
Expected: all benchmark tests pass with no skipped required test or network/provider call.
Actual stdout/stderr:
```text
.............................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 429 tests in 115.180s
OK
```
### Manifest and repository diff
Command:
```bash
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Expected: `ok: manifest is valid`; `git diff --check` has no output.
Actual stdout/stderr:
```text
ok: manifest is valid
```
`git diff --check` stdout/stderr: `(none)`
Reviewer fresh rerun:
```text
focused: Ran 41 tests in 14.154s — OK
full: Ran 429 tests in 116.096s — OK
manifest: ok: manifest is valid
git diff --check: (none)
```
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.
## Section Ownership
| Section | Owner | Note |
|---|---|---|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns applicable commands and records fresh output |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
- Overall Verdict: FAIL
- Dimension Assessment:
- Correctness: Pass — R1 current usage parsing and R2 config-owned execution/scoring binding now match production behavior while preserving mismatch rejection.
- Completeness: Fail — no replacement C01-C09 scored execution exists after the retained plan=4 failure.
- Test coverage: Pass — production-shaped usage, absent observation success, non-None mismatch failure, execution, and scoring paths have meaningful regression coverage.
- API contract: Pass — optional caller observation and canonical config admission now have one consistent ownership contract.
- Code quality: Pass — changes are bounded to the selected parser/consumer/test/fixture paths with no debug or dead code.
- Implementation deviation: Pass — implementation followed R1/R2 exactly and did not invoke a caller/provider or mutate retained run state.
- Verification trust: Pass — reviewer fresh reruns match implementation evidence: focused 41/OK, full 429/OK, manifest valid, diff check clean.
- Spec conformance: Fail — S04-S08 still require nine retained C01-C09 terminal attempts from an authorized scored execution.
- Findings:
- Required R3 — Authorized replacement scored execution is still required.
- Evidence: source-fix reviewer verification passes, but read-only public status for `run-20260812T040619Z-00a5e6664764` remains `failed=1, running=1`; `run_id.log` is absent and no replacement execution was invoked by plan=5. The benchmark skill requires stopping after a retained execution failure until the user explicitly authorizes another stateful execution.
- Root Cause: the single scored run authorized for plan=4 was consumed before R1/R2 were known; append-only policy forbids rewriting that attempt, and plan=5 intentionally repaired only deterministic source/test boundaries.
- Selected Fix: after explicit user authorization, create a new execution plan that rebinds the protected local token/CA references, requires fresh public preflight `ready=9`, invokes `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once, stores only its CLI-emitted run id, and accepts only nine retained terminal attempts with zero `running|interrupted`. Preserve every older run and do not call direct callers/providers or edit state.
- Routing Signals: review_rework_count=2 evidence_integrity_failure=false
- Next Step: USER_REVIEW external-execution — obtain explicit authorization for exactly one new public C01-C09 scored run after the retained failure; then re-enter the plan skill for this exact task path.

0
agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_1.log
View file

265
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_2.log Normal file
View file

@ -0,0 +1,265 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=2 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - C01-C09 원샷 비교 실행
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=2, tag=TEST
## Archive Evidence Snapshot
- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` proves task-protocol dependency completion while preserving a fail-closed live-readiness blocker; it does not mark Milestone `route-readiness` complete.
- Generation 0/1 are preserved as `plan_cloud_G09_0.log`, `code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`, `code_review_cloud_G10_1.log`, all without an official verdict. Active plan=2 replaces their semantic defects; do not execute archived commands.
## For the Review Agent
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. Never repeat the one-time provider execution in Final Verification step 3. If implementation is present, repair missing or stale repeatable verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating a follow-up plan.
Review completion means the following steps are finished:
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-{review_lane}-{review_grade}.md` and `PLAN-{build_lane}-{build_grade}.md` to the next canonical log indices.
3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/{task_name}/`. If WARN/FAIL, write the next filesystem state required by the code-review skill.
4. If PASS, preserve the first-line `milestone-task` metadata and report it for runtime aggregation. Roadmap evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
---
## Implementation Item Completion
| Item | Status |
|------|--------|
| TEST-1: Immutable C01-C09 comparison run | [ ] |
## Implementation Checklist
- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [ ] Run applicable required repeatable verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
- [ ] Archive active `CODE_REVIEW-*-G??.md` and `PLAN-*-G??.md` to the next canonical log indices.
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` from the canonical template and leave no active `.md` files.
- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/{task_name}/` and update this checklist at the final archive path.
- [ ] If PASS, preserve and report the five exact `milestone-task` ids for runtime aggregation without modifying roadmap or directly calling `update-roadmap`.
- [ ] If PASS for split work, remove an empty active parent or verify it remains because siblings/files exist.
- [ ] If WARN/FAIL, write the next filesystem state matching the verdict and do not write `complete.log`.
## Deviations from Plan
_Record any deviations from the plan and the rationale here._
## Key Design Decisions
_Record key design decisions here._
## Reviewer Checkpoints
- Confirm active PLAN/review headers both use plan=2 and the exact five-task union: `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`.
- Confirm active index `06+05`, exact archived predecessor `05+04/.../complete.log`, and the preserved readiness blocker; do not infer that `route-readiness` is complete.
- Confirm S04-S08 map C01-C09 to those five tasks and source/spec/contract/SDD/Milestone remained outside the write set.
- Confirm the implementer did not directly invoke a caller/provider, inspect secret values, duplicate live-preflight policy, rebuild/mutate `../iop-s2`, or invoke `resume`, retry, score, evaluator, or report.
- Confirm step 2 proved fixed host/testbed/artifact identity and command presence before any public `run` call.
- Confirm step 3 was invoked at most once, the pointer came only from one unique CLI-emitted id, that id did not preexist, and its exact run directory exists. Ignore unrelated concurrent run directories.
- Confirm public status validates exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted`. Retained failure is valid execution evidence; zero/partial/invalid/unidentified state is not.
- Confirm CLI exit/output, state classification and checklist agree, no secret/raw private endpoint/config payload was recorded, and `../iop-s2` stayed clean.
- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid`.
## Verification Results
> The implementing agent must replace each placeholder with exact command results and verbatim stdout/stderr. If a fixed gate or execution state blocks, retain unchecked completion boxes and record the exact public error and resume condition. The review agent reruns only sections marked repeatable; the one-time provider execution is evaluated from preserved evidence and public `status`.
### Static manifest and full benchmark suite — repeatable
Command:
```bash
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
```
Expected: manifest valid and all discovered benchmark tests pass.
Actual stdout/stderr:
_Fill with verbatim output._
### Fixed host/testbed/artifact gate — repeatable, no caller/provider invocation
Command:
```bash
set -euo pipefail
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
test -x "$binary" || blocked "$binary must be executable"
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
done
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
import subprocess, sys
expected = sys.argv[1]
for binary in sys.argv[2:]:
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
build = {}
for line in output.splitlines():
fields = line.strip().split("\t", 1)
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
key, value = fields[1].split("=", 1)
build[key] = value
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
print("ok: Edge/Node build identities match the clean testbed HEAD")
PY
```
Expected: all checks exit 0. Any blocker stops before the scored command.
Actual stdout/stderr:
_Fill with verbatim output._
### C01-C09 run — implementation-only, do not repeat in review
Command:
```bash
set -euo pipefail
task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
run_root=agent-test/runs/bench-02
test ! -e "$task_dir/run_id.log"
before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
trap 'rm -f "$before" "$out" "$err"' EXIT
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
set +e
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
bench_exit=$?
set -e
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
printf '%s\n' 'stderr:'; cat "$err"
emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)"
emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)"
if test "$emitted_count" -ne 1; then
printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count"
exit 69
fi
bench_run_id="$(printf '%s\n' "$emitted_ids")"
if grep -Fxq "$bench_run_id" "$before"; then
printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id"
exit 69
fi
test -d "$run_root/$bench_run_id"
printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log"
set +e
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)"
status_exit=$?
set -e
printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output"
if test "$status_exit" -ne 0; then
printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id"
exit 69
fi
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
import ast, os
text = os.environ["STATUS_OUTPUT"]
assert text.startswith("ok: "), text
states = ast.literal_eval(text[4:])
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
bench_exit = int(os.environ["BENCH_EXIT"])
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
assert bench_exit == (0 if states["success"] == 9 else 69)
print(f"classification=execution_complete states={states}")
raise SystemExit(0)
if sum(states.values()) == 0 and bench_exit == 69:
print(f"classification=blocked_preflight states={states}")
else:
print(f"classification=blocked_partial states={states}")
raise SystemExit(69)
PY
```
Expected: one CLI-emitted new run id and `classification=execution_complete`. Exit 69 is accepted only when all nine non-interrupted terminal attempts are retained.
Actual stdout/stderr:
_Fill with verbatim output, including command, exit code, both streams, status and classification._
### Recorded run status and worktrees — repeatable, no provider invocation
Command:
```bash
set -euo pipefail
pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
test "$(wc -l <"$pointer")" -eq 1
bench_run_id="$(tr -d '\n' <"$pointer")"
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
printf '%s\n' "$status_output"
STATUS_OUTPUT="$status_output" python3 - <<'PY'
import ast, os
text = os.environ["STATUS_OUTPUT"]
assert text.startswith("ok: "), text
states = ast.literal_eval(text[4:])
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
print(f"ok: nine retained terminal attempts states={states}")
PY
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
git diff --check
```
Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed and no whitespace errors.
Actual stdout/stderr:
_Fill with verbatim output._
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.
## Section Ownership
| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
| Code Review Result | Review agent appends | Not included in stub |

206
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log Normal file
View file

@ -0,0 +1,206 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=3 tag=API milestone-task=agy-iop-compatibility,managed-credential-dev,route-readiness,claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - official agy IOP 호환, managed dev 배포와 C01-C09 실행
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Finalization (`Code Review Result`, archive, `complete.log`) is review-agent-only.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=3, tag=API
## Archive Evidence Snapshot
- Generation 0/1/2 are preserved in the same task directory and have no official verdict. Active plan=3 replaces the invalid synthetic agy/legacy credential premise while preserving the immutable one-run rule.
- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` satisfies the index dependency but does not remove the current live-readiness blocker.
## For the Review Agent
Compare every implementation item with source and fresh repeatable verification. Never repeat the one-time C01-C09 provider run. Append verdict and routing signals, archive the active pair, and create `complete.log` only on PASS.
## Implementation Item Completion
| Item | Status |
|------|--------|
| API-1: Gemini-native Edge ingress | [x] |
| API-2: official agy 1.1.12 adapter | [x] |
| API-3: managed dev deployment and live readiness | [x] |
| API-4: immutable C01-C09 run | [ ] |
## Implementation Checklist
- [x] API-1: Implement and test the route-qualified Gemini request/auth/SSE bridge through existing route and preset admission.
- [x] API-2: Use official agy 1.1.12 Gemini API-key transport and real stream-json lifecycle with config-owned binding evidence.
- [x] API-3: Deploy fresh artifacts and a complete managed credential/TLS composition to dev, proving direct/hybrid readiness without legacy fallback.
- [ ] API-4: Invoke the public C01-C09 run exactly once after readiness and retain exactly nine non-interrupted terminal attempts.
- [x] Fill all implementation-owned sections below with actual notes and verification output.
## Review-Only Checklist
- [ ] Append one verdict and verified `review_rework_count` / `evidence_integrity_failure` signals.
- [ ] Run all applicable repeatable verification and compare it with implementation evidence.
- [ ] Confirm no inbound principal token can become provider authorization and no secret appears in tracked evidence.
- [ ] Confirm official agy argv/env/events and config-owned effective binding match the contract.
- [ ] Confirm dev has CA-signed TLS, projection, slot-route authorization and no legacy credential fallback.
- [ ] Confirm the scored run occurred at most once and has exactly nine retained terminal attempts.
- [ ] Archive the active pair and write `complete.log` only for PASS.
## Deviations from Plan
- 공식 `agy` 1.1.12의 실제 JSONL은 flat `status`/`usage`가 아니라 `event`와 같은 이름의 `init`/`step_update`/`result` 중첩 payload를 사용했다. adapter, fixture와 integration seam을 실제 출력 구조로 교정했다.
- `/tmp`는 실행 불가라 deterministic credential smoke의 임시 실행 경로를 ignored `build/test-tmp`로 옮겼다. 이 과정에서 smoke가 내부 route id를 기다리던 오래된 판정을 찾아, 계약대로 public route alias를 기다리도록 수정했다.
- 공식 agy hybrid와 raw Chat hybrid에서 plan output의 마지막 LF 생략, work/review tool path와 review JSON strictness 문제가 드러나 parser의 최종 LF 경계, strict tool schema/한 번의 bounded correction, review response schema를 보완했다.
- 공개 `run`은 all-cell preflight 뒤 정확히 한 번 호출했지만 첫 caller launch 전에 shared filesystem이 symlink 경유 Unix socket bind를 `EINVAL`로 거부했다. 상대 basename bind와 0700 parent fallback, pre-registration reconcile을 구현하고 회귀 테스트를 추가했다. one-run 원칙에 따라 `run`/`resume`은 다시 호출하지 않았다.
## Key Design Decisions
- Gemini ingress는 별도 provider 우회 경로를 만들지 않고 인증된 내부 Chat request로 변환해 기존 route/preset admission, managed credential projection과 stage authorization을 그대로 사용한다.
- inbound `x-goog-api-key`는 IOP principal token으로만 사용하고 provider credential은 slot/lease projection에서만 가져온다. caller token, provider token과 private key는 tracked evidence에 기록하지 않는다.
- official caller child에는 최소 환경을 유지하되 private dev CA가 필요한 경우 표준 `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS` 두 값만 명시적으로 전달한다.
- pre-registration interruption은 invocation identity가 존재하지 않으므로 불가능한 measurement/web sidecar를 발명하지 않는다. locator 또는 일부 sidecar가 존재하는 경우에는 기존 엄격 검증을 유지한다.
## Reviewer Checkpoints
- Confirm `/gemini/{route}/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse` is authenticated using the IOP principal, while provider credentials still come only from managed projection.
- Confirm malformed/conflicting auth and unsupported Gemini bodies fail before dispatch, and stream cancellation/error/tool/usage paths remain bounded and caller-safe.
- Confirm agy uses `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`, official model label and no `--effort` or invented `AGY_*` variables/events.
- Confirm remote config and live evidence prove managed credentials, HTTPS/mTLS, exact direct/hybrid routes and no static provider credential source.
- Confirm C01-C09 is invoked only after all-cell preflight; no retry/resume/manual result editing is allowed.
## Verification Results
### Focused local suites — repeatable
Command:
```bash
go test -count=1 ./apps/edge/internal/openai
python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test
```
Actual stdout/stderr:
```text
ok iop/apps/edge/internal/openai 8.326s
ok iop/apps/edge/internal/service 8.252s
ok iop/packages/go/singlerequesttemplate 0.006s
Ran 62 tests in 19.224s
OK
```
### Full local suites and contract hygiene — repeatable
Command:
```bash
go test -p 1 -count=1 ./...
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Actual stdout/stderr:
```text
go test -p 1 -count=1 ./...: PASS (all packages)
Ran 429 tests in 115.981s
OK
ok: manifest is valid
git diff --check: PASS
```
첫 full Go 병렬 검증에서는 unrelated transport duplicate-registration test 한 건이 45초 timeout이었으나 단독 0.01초, package 4.766초에 통과했고 부하 없이 반복한 전체 `-p 1` run은 모두 통과했다.
### Managed credential deterministic qualification — repeatable
Command:
```bash
credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)"
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
rmdir "$credential_smoke_parent"
```
Actual stdout/stderr:
```text
mode=deterministic
profiles=seulgi_chat,seulgi_messages
same_model_two_slot=true
exact_auth.chat=2
exact_auth.messages=1
ciphertext_only=true
tls_negative_matrix=passed
post_revoke.counters_unchanged=true
post_revoke.no_fallback=true
result=success
```
실제 slot/route 식별자는 secret-safe evidence 요구에 따라 생략했다.
### Remote managed dev and live caller readiness — repeatable except provisioning
Record sanitized source/artifact/config identity, runtime health, official agy direct/hybrid smoke and public preflight output. Do not record secrets, raw request content, private keys, slot aliases or lease ids.
Actual stdout/stderr:
```text
edge_sha256=c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc
edge_pid=172
ports=18082,18083,18084,19093,19101 all_up
nodes=4 connected=4 providers=8 provider_states=available/healthy
nodes=mac-codex-node,gx10-vllm-node,onexplayer-lemonade-node,rtx5090-lemonade-node
raw gemini-hybrid smoke #1: HTTP 200, finish=stop, errors=0, done=true
raw gemini-hybrid smoke #2: HTTP 200, finish=stop, errors=0, done=true
official agy direct: exit=0, result.status=SUCCESS, stderr_bytes=0
official agy hybrid: exit=0, result.status=SUCCESS, workspace_unchanged=true, stderr_bytes=0
ok: preflight status=ready ready=9 registration_required=0 implementation_gap=0
```
### C01-C09 run — implementation-only, never repeat in review
Record the single public `run` command exit/output, its one CLI-emitted run id, and public `status` classification.
Actual stdout/stderr:
```text
$ python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
error: benchmark state is unavailable
exit=69
```
CLI-emitted run id는 없었으므로 `run_id.log`는 생성하지 않았다. Controller가 만든 state는 첫 slot에서 `caller_launched=false`, `terminal_reason=supervisor_error`, cleanup 완료였다. 원인은 symlink path Unix socket `bind(2)`의 `EINVAL`이며 수정 후 symlink control socket 및 pre-registration reconcile 회귀 테스트가 통과했다. 같은 run을 재시도하거나 resume하지 않았다.
### Recorded run status and worktrees — repeatable
Record public status for the stored pointer, nine terminal attempts, zero running/interrupted, and `git diff --check`.
Actual stdout/stderr:
```text
controller_state=run-20260812T031657Z-0a24376c2414
status before reconcile: running=1, all other terminal counts=0
status after harness reconcile: interrupted=1, running=0, success=0, failed=0, timed_out=0, cancelled=0
caller_launched=false
live_supervisor=false
control_alias_count=0
run_id.log=absent
testbed=../iop-s2 branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
git diff --check: PASS
```
API-4 acceptance인 nine terminal attempts/zero interrupted는 충족하지 못했다. resume/retry 없이 새 실행 정책 승인이 있어야 C01-C09를 다시 시작할 수 있다.
---
> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section and check only completed items. Leave review-only sections unchanged.
## Code Review Result
_Review-agent only._

267
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log Normal file
View file

@ -0,0 +1,267 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=4 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - TEST
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
> The task is NOT complete until every implementation-owned section below is filled in.
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
> Execute the plan's selected scope and state boundary as written. Do not choose another owner, narrow/expand the write boundary, or replace the run with an alternate caller path.
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
> Follow the ownership table at the bottom of this file for which sections you own.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=4, tag=TEST
## Archive Evidence Snapshot
- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다.
- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다.
- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다.
- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다.
## For the Review Agent
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
Compare implementation of each item against the active Plan and immutable run state. Repeat only local tests, manifest validation, secret-safe preflight and public `status`; never repeat `run`, invoke `resume`, use `--retry-failed`, call a caller/provider directly, or edit a run tree. Review completion means the following steps are finished:
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_4.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_4.log`.
3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; if WARN/FAIL, write the next state required by the code-review skill.
4. Preserve first-line `milestone-task` metadata for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
---
## Implementation Item Completion
| Item | Status |
|------|---------|
| TEST-1: Protected runtime binding and fresh readiness | [x] |
| TEST-2: One replacement C01-C09 execution run | [ ] |
## Implementation Checklist
- [x] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output.
- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
> Implementing agents must not modify or check this section.
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
- [x] Run applicable repeatable verification and record fresh output; do not repeat the execution run.
- [x] Confirm prior interrupted/preflight-only run roots remain append-only and no stored pointer was written for the non-emitted execution id.
- [x] Confirm public status is `failed=1, running=1` rather than the required nine terminal attempts and classify the gap as Required.
- [x] Confirm no secret value, raw prompt/response, private key, slot alias or lease id appears in tracked evidence.
- [x] Archive active `CODE_REVIEW-cloud-G10.md` to `code_review_cloud_G10_4.log`.
- [x] Archive active `PLAN-cloud-G10.md` to `plan_cloud_G10_4.log`.
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
- [ ] If PASS, write `complete.log` and leave no active `.md` files.
- [ ] If PASS, move the active task directory to the archive path and update this checklist at the final archive path.
- [ ] If PASS, preserve and report `milestone-task` metadata without modifying roadmap directly.
- [x] If WARN/FAIL, write the next filesystem state matching the code-review verdict and do not write `complete.log`.
## Deviations from Plan
- Planning 중 보호된 runtime reference가 없는 셸에서 공개 preflight를 호출해 `implementation_gap=9`가 발생했다. attempt는 생성되지 않았으며 `token/.iop-bench`, private dev CA와 배포 config observation을 process-local environment에 연결한 뒤 `ready=9`를 확인했다.
- 최초 execution wrapper는 임시파일 cleanup 구문이 host safety policy에서 process 시작 전에 거부됐다. public benchmark CLI는 호출되지 않았고 active Plan/Review의 command를 direct CLI output 방식으로 교정한 뒤 실행했다.
- 교정 후 public `run`은 정확히 한 번 호출됐다. 첫 read-only status는 `failed=1,running=1`이었고 CLI는 이후 `error: benchmark state is unavailable`, exit 69로 종료했다. CLI-emitted run id가 없고 최종 public status도 `failed=1,running=1`이라 TEST-2 acceptance를 충족하지 못했다.
- observed controller directory는 `run-20260812T040619Z-00a5e6664764`지만 CLI-emitted canonical id가 아니므로 `run_id.log`를 생성하지 않았다. 해당 run을 resume/retry/reconcile하거나 run tree를 편집하지 않았다.
## Key Design Decisions
- preflight/run/status는 모두 public deterministic benchmark CLI만 사용했다. caller/provider 직접 호출이나 route/model/effort 대체는 하지 않았다.
- retained failure와 dangling status를 성공으로 재분류하지 않는다. 실행 process가 없는 것을 read-only로 확인했지만 내부 `RunStore`나 reconcile API로 상태를 바꾸지 않는다.
- TEST-2는 exact blocker 상태로 남긴다. 재개 조건은 official review가 `benchmark state is unavailable`의 root cause와 one selected fix를 확정하고, append-only evidence를 보존하는 follow-up plan을 생성하는 것이다.
## Reviewer Checkpoints
- Confirm the no-reference `implementation_gap=9` preflight and the corrected `ready=9` preflight allocated no scored attempts.
- Confirm TEST-2 uses one new public `run` identity and does not resume or modify `run-20260812T031657Z-0a24376c2414`.
- Confirm each C01-C09 cell has one retained accepted terminal attempt under the new run, with fresh workspace/session identity and IOP route binding.
- Confirm `run_id.log` contains exactly the CLI-emitted new execution run id and public `status` matches the recorded evidence.
- Never use execution success alone to complete the third Epic; validation/scoring/report remain separately planned S09-S12 work.
## Verification Results
### Fresh local contract checks — repeatable
Command:
```bash
python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output.
Actual stdout/stderr:
```text
.....................................................................
----------------------------------------------------------------------
Ran 69 tests in 59.949s
OK
ok: manifest is valid
```
`git diff --check` stdout/stderr: `(none)`
### Protected all-cell preflight — repeatable, no scored attempt
Command:
```bash
set -euo pipefail
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
export IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083
export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083
export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
BENCH_CONFIG="$(python3 - <<'PY'
import json
routes = [
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
]
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
PY
)"
export BENCH_CONFIG
for tool in python3 claude agy codex git; do command -v "$tool"; done
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`.
Actual stdout/stderr:
```text
/bin/python3
/config/.npm-global/bin/claude
/config/.local/bin/agy
/config/.npm-global/bin/codex
/bin/git
ok: preflight run_id=run-20260812T040619Z-1ed70fffdbda status=ready ready=9 registration_required=0 implementation_gap=0
```
### C01-C09 execution — implementation-only, never repeat in review
Command:
```bash
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Expected: the CLI emits exactly one canonical execution id. Record its exit code and stdout/stderr verbatim; public status is checked separately. A retained failure is preserved and never retried.
Actual stdout/stderr:
```text
command: run
exit_code: 69
stdout:
benchmark_exit_code=69
stderr:
error: benchmark state is unavailable
cli_emitted_run_id=(none)
```
Intermediate public status while the command was active:
```text
ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}
```
### Stored execution status — repeatable, provider-free
Command:
```bash
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id"
git diff --check
```
Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error.
Actual stdout/stderr:
The planned stored-pointer command was not run because `run_id.log` was correctly absent. Read-only blocker checks were:
```text
$ python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id run-20260812T040619Z-00a5e6664764
ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}
$ protected path/process check
run_id.log=absent
live_execution_process=false
```
`git diff --check` stdout/stderr: `(none)`
---
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
> If anything is blank, go back and fill it in before saving this file.
> Leave review-agent-only sections unchanged.
## Section Ownership
| Section | Owner | Note |
|---------|-------|------|
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly |
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns only repeatable commands and never the execution run |
| Code Review Result | Review agent appends | Not included in stub |
## Code Review Result
- Overall Verdict: FAIL
- Dimension Assessment:
- Correctness: Fail — production Codex output is rejected and the public run aborts with one dead attempt retained as `running`.
- Completeness: Fail — TEST-2 requires nine non-interrupted terminal attempts, but public status is `failed=1, running=1` after only two allocated cells.
- Test coverage: Fail — the fixture and integration seam omit a production usage key and synthesize a caller binding event that the real Codex CLI does not emit.
- API contract: Fail — parser and consumers disagree about both the current Codex JSONL usage schema and the documented optional caller binding observation.
- Code quality: Pass — no unrelated debug output, TODO, or formatting defect was found in the reviewed scope.
- Implementation deviation: Pass — the implementation preserved the failed run, did not write `run_id.log`, and did not resume, retry, or edit state.
- Verification trust: Pass — recorded exit/status evidence matches fresh read-only status, process inspection, lifecycle replay, and retained attempt files.
- Spec conformance: Fail — S04-S08 still lack the required C01-C09 terminal execution evidence.
- Findings:
- Required R1 — Current Codex usage JSONL is rejected.
- Evidence: replaying the retained C05 `lifecycle-result.json` stdout through `CodexJSONLParser` fails on line 26 with `CodexJSONLError: invalid Codex usage observation`; the `turn.completed.usage` keys are `cache_write_input_tokens`, `cached_input_tokens`, `input_tokens`, `output_tokens`, and `reasoning_output_tokens`. `scripts/agent_benchmark/codex_iop.py:79-85,334-345` does not admit `cache_write_input_tokens`. Fresh `python3 -m unittest scripts.agent_benchmark.codex_iop_test` still reports 9 tests/OK, proving the current test set misses the production shape.
- Root Cause: the Codex adapter's exact usage-key mapping predates the caller's `cache_write_input_tokens` field, while the tracked fixture and parser unit test still use the older subset.
- Selected Fix: add `cache_write_input_tokens -> cache_write_tokens` to `_CODEX_USAGE_FIELDS`; update `scripts/agent_benchmark/codex_iop_test.py` and `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` with the retained production-shaped usage object and assertions that the value is preserved without reconstructing totals. Acceptance commands: `python3 -m unittest scripts.agent_benchmark.codex_iop_test` and the focused lifecycle/integration suite in the follow-up plan.
- Required R2 — Config-owned binding admission is contradicted by caller-owned enforcement.
- Evidence: `CodexJSONLParser.effective_binding` is explicitly optional and `test_bridge_proves_finish_then_idle_after_child_exit` asserts `None`, but `scripts/agent_benchmark/live_iop.py:992-995` rejects every live invocation unless the caller emits an exact `iop_effective_binding`. The retained real Codex stdout contains no such synthetic event. The live scoring consumer at `scripts/agent_benchmark/live_iop.py:1123-1131` applies the same impossible requirement. `scripts/agent_benchmark/connectivity_integration_test.py:217-227` currently injects the synthetic event and `:2403-2436` asserts that absence is an error, hiding the production mismatch.
- Root Cause: preflight moved effective route/model/stage ownership to the independently validated config observation, but execution and scoring consumers retained the earlier caller-event equality gate and their seams continued fabricating that event.
- Selected Fix: in `_LiveAdapter.invoke` and `_LiveScoringAdapter.invoke`, use the admitted config binding as the canonical result; treat an optional caller observation as an additional exact-match check only when present, and still reject any non-`None` mismatch. Remove the synthetic Codex binding from the integration fixture and add execution plus scoring assertions that absent observation succeeds from admitted config while mismatches fail. Acceptance commands: the focused `codex_iop_test` and named connectivity integration tests, followed by the full Python benchmark suite and `git diff --check`.
- Routing Signals: review_rework_count=1 evidence_integrity_failure=false
- Next Step: Run the mandatory plan-skill WARN/FAIL follow-up for Required R1 and R2; after deterministic fixes pass, a new scored execution remains authorization-gated because this retained run must not be retried implicitly.

101
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_6.log Normal file
View file

@ -0,0 +1,101 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=6 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - TEST
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=6, tag=TEST
## Archive Evidence Snapshot
- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained execution defect evidence.
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 fixed with 41/429 tests passing.
- `user_review_0.log`: continuing execution authorization resolved.
## Implementation Item Completion
| Item | Status |
|---|---|
| TEST-1 deterministic regression | [x] |
| TEST-2 protected ready=9 preflight | [x] |
| TEST-3 new C01-C09 public run | [ ] |
## Implementation Checklist
- [x] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh.
- [x] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`.
- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output.
## Review-Only Checklist
- [ ] Append verdict and routing signals.
- [ ] Run repeatable verification without repeating scored execution.
- [ ] Archive active pair to `plan_cloud_G10_6.log` and `code_review_cloud_G10_6.log`.
- [ ] If PASS, write complete.log and archive task; if non-PASS, materialize the required next state.
## Deviations from Plan
TEST-3의 public `run`은 계획대로 정확히 한 번 호출했으나 exit 69로 종료됐다. CLI가 canonical id를 출력하지 않아 `run_id.log`를 쓰지 않았고, 새 run은 read-only blocker evidence로만 식별했다. resume/retry/direct caller/provider/state edit는 수행하지 않았다.
## Key Design Decisions
Codex C05 success가 종결된 뒤 C03 agy의 ordinary nonzero lifecycle에서 controller exception이 발생한 사실을 보존했다. retained run은 재개하지 않고, ordinary lifecycle failure를 terminal attempt로 처리하는 source fix를 후속 plan에서 적용한다.
## Verification Results
### TEST-1 deterministic regression
```text
focused: Ran 41 tests in 15.084s — OK
full: Ran 429 tests in 118.691s — OK
manifest: ok: manifest is valid
git diff --check: (none)
```
### TEST-2 protected preflight
```text
ok: preflight run_id=run-20260812T043640Z-91c437c5c7cd status=ready ready=9 registration_required=0 implementation_gap=0
```
### TEST-3 one public run and status
```text
command: run
exit_code: 69
stdout: (none)
stderr: error: benchmark state is unavailable
cli_emitted_run_id: (none)
identified_new_run: run-20260812T043704Z-041320764a11
public_status: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 1, 'timed_out': 0}
attempts: c02=failed, c05=success, c03=running
```
C03 retained lifecycle is `success=false`, `terminal_reason=nonzero_exit`, `exit_code=1`, `cleanup_complete=true`, `process_group_alive=false`, but attempt-measurement/web-validation are absent and `attempt.json` remains `running`.
## Section Ownership
- Implementation owns completion status, deviations, decisions, and verification output.
- Review owns verdict, archive, complete.log, next state, and roadmap completion event.
## Code Review Result
- Overall Verdict: FAIL
- Dimension Assessment:
- Correctness: Fail — an ordinary agy nonzero lifecycle is converted to an adapter exception and leaves a dead attempt running.
- Completeness: Fail — only three attempts were allocated and C01-C09 terminal evidence is incomplete.
- Test coverage: Fail — current live integration coverage does not assert that an agy failed lifecycle is returned for terminal publication and that later slots continue.
- API contract: Fail — execution adapter treats caller execution failure as connectivity-contract failure after preflight was already admitted.
- Code quality: Pass — retained evidence and prior Codex fix are consistent and bounded.
- Implementation deviation: Pass — exactly one run was called and no forbidden recovery path was used.
- Verification trust: Pass — public status and retained lifecycle/sidecar evidence agree.
- Spec conformance: Fail — S04-S08 still lack nine terminal attempts.
- Findings:
- Required R4 — agy ordinary failed lifecycle aborts the whole run instead of terminalizing the attempt.
- Evidence: C03 `lifecycle-result.json` is a clean `nonzero_exit` with process cleanup complete; `live_iop.py` calls `parser.observed_result`, receives non-ready because lifecycle success is false, then raises `LiveIopError`. This occurs before `run_slots.invoke_bound` publishes measurement/web validation, so `attempt.json` stays `running` and the CLI returns unavailable state. C05 immediately before it is `success`, proving the earlier Codex fix works.
- Root Cause: `_LiveAdapter.invoke` conflates post-admission execution outcome with preflight binding verification. It applies the successful-stream `observed_result` gate even when lifecycle already has a valid terminal failure.
- Selected Fix: in the agy branch, return `_bound_observations(result, admitted)` immediately for any `result.success == false`; apply `parser.observed_result` exact binding checks only to successful lifecycle results. Add credential-free integration coverage for failed agy lifecycle terminal publication and later-slot continuation, then run full benchmark tests, fresh ready=9 preflight, and one new public run under the user's continuing authorization.
- Routing Signals: review_rework_count=3 evidence_integrity_failure=false
- Next Step: Run the mandatory plan-skill FAIL follow-up for R4, including one authorized new public run after the deterministic fix; preserve this run without resume/retry/edit.

97
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_7.log Normal file
View file

@ -0,0 +1,97 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - REVIEW_TEST
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=7
## Archive Evidence Snapshot
- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: R4 evidence and retained run.
- `user_review_0.log`: continuing execution authorization.
## Implementation Item Completion
| Item | Status |
|---|---|
| FIX-1 agy failure terminalization | [x] |
| FIX-2 regression coverage | [x] |
| TEST-1 deterministic verification | [x] |
| TEST-2 ready=9 and one new run | [x] |
## Implementation Checklist
- [x] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams.
- [x] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots.
- [x] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check.
- [x] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted.
- [x] Fill implementation evidence.
## Review-Only Checklist
- [x] Append PASS verdict/routing signals, run repeatable verification, archive to `_7.log`, write `complete.log`, move the task directory, and preserve milestone completion metadata.
## Deviations from Plan
없음. 새 public run은 정확히 한 번 호출했으며 exit 69은 retained failed cells 때문이었다. CLI-emitted canonical id만 `run_id.log`에 기록했고 old run resume/retry/state edit 또는 direct caller/provider 호출은 없었다.
## Key Design Decisions
- agy lifecycle 자체가 실패하면 connectivity binding을 재판정하지 않고 admitted binding에 대해 typed observation만 검증해 RunStore가 정상 terminal publication을 수행하게 했다.
- 성공 lifecycle에만 `observed_result` exact config-binding gate를 유지해 contract mismatch는 계속 fail-closed한다.
- 개별 failed 결과를 재시도하지 않고 D10 retained evidence로 수용했다.
## Verification Results
### Source/tests
```text
focused: Ran 75 tests in 51.600s — OK
full: Ran 430 tests in 122.446s — OK
manifest: ok: manifest is valid
git diff --check: (none)
```
### Protected preflight/run/status
```text
preflight: ok: preflight run_id=run-20260812T044743Z-40dd65f85a35 status=ready ready=9 registration_required=0 implementation_gap=0
run exit_code: 69
run stderr: error: benchmark execution failed run_id=run-20260812T044800Z-412e05fc80df completed=9 unresolved=7 success=2 failed=7 timed_out=0 cancelled=0 interrupted=0 running=0
status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0}
attempt_count: 9
all_terminal: True
all_sidecars: True
```
## Section Ownership
Implementation owns status/evidence; review owns verdict/archive/complete/next state.
## Code Review Result
- Overall Verdict: PASS
- Dimension Assessment:
- Correctness: Pass — agy failed lifecycles now terminalize and later slots continue; successful binding checks remain exact.
- Completeness: Pass — C01-C09 all have retained terminal attempts, required measurement/web sidecars, and zero running/interrupted.
- Test coverage: Pass — focused 75 and full 430 tests include adapter failure return and later-slot continuation.
- API contract: Pass — post-admission execution failure is distinct from successful-stream binding validation.
- Code quality: Pass — the source change is a bounded branch guard with focused regression coverage.
- Implementation deviation: Pass — one fresh run, no resume/retry/substitution/state edit/direct caller/provider.
- Verification trust: Pass — reviewer fresh tests, manifest, diff, pointer format, and public status all agree.
- Spec conformance: Pass — S04-S08/D06/D10 accept the complete nine-cell success/failure terminal matrix with preserved failures.
- Findings: None
- Routing Signals: review_rework_count=3 evidence_integrity_failure=false
- Next Step: PASS — archive the pair, write complete.log, move the task directory, and emit milestone completion metadata for runtime aggregation.
### Reviewer Fresh Verification
```text
focused: Ran 75 tests in 45.609s — OK
full: Ran 430 tests in 117.951s — OK
manifest: ok: manifest is valid
status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0}
git diff --check: (none)
```

47
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log Normal file
View file

@ -0,0 +1,47 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Complete - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
## 완료 일시
2026-08-12
## 요약
8개 plan/review revision과 지속 실행 승인 1건을 거쳐 C01-C09의 fresh one-shot terminal evidence를 완성했고 최종 verdict는 PASS다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|---|---|---|---|
| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | 초기 실행 준비 revision. |
| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | readiness 정합화 revision. |
| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | runtime/provider 준비 revision. |
| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, managed dev, all-cell readiness와 pre-launch controller fix. |
| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Codex current usage와 config-owned binding 결함 발견. |
| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | Codex 결함 수정 및 실행 지속 승인 대기. |
| `plan_cloud_G10_6.log` | `code_review_cloud_G10_6.log` | FAIL | Codex success 확인 후 agy failed-lifecycle terminalization 결함 발견. |
| `plan_cloud_G10_7.log` | `code_review_cloud_G10_7.log` | PASS | R4 수정, 9개 terminal attempt 및 sidecar evidence 완성. |
## 구현/정리 내용
- Gemini-native Edge ingress, official agy transport, managed credential dev binding과 all-cell readiness를 정합화했다.
- Codex `cache_write_input_tokens`를 canonical metric으로 수용하고 config-owned binding을 execution/scoring에서 일관되게 적용했다.
- agy ordinary failed lifecycle을 terminal attempt로 보존해 이후 슬롯 실행이 계속되도록 수정했다.
- canonical run `run-20260812T044800Z-412e05fc80df`에 C01-C09 9개 attempt, `success=2`, `failed=7`, `running=0`, `interrupted=0`과 모든 measurement/web-validation sidecar를 보존했다.
## 최종 검증
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test` - PASS; reviewer fresh 75 tests/OK.
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; reviewer fresh 430 tests/OK.
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; manifest valid.
- public `status --run-id run-20260812T044800Z-412e05fc80df` - PASS; 9 terminal, running/interrupted 0.
- `git diff --check` - PASS; output 없음.
## 잔여 Nit
- 없음
## 후속 작업
- 다음 Epic의 objective validation, blind scoring, aggregation/reporting은 별도 task로 진행한다.

0
agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_0.log → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_0.log
View file

0
agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/PLAN-cloud-G09.md → agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_1.log
View file

248
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G09_2.log Normal file
View file

@ -0,0 +1,248 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=2 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - C01-C09 원샷 비교 실행
## For the Implementing Agent
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 검증 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다.
## Background
승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행한다. checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`의 generation 0 의도인 “fresh preflight 뒤 public run 한 번, retry/resume 없음, 한 run의 아홉 cell 보존”을 유지한다. Generation 1의 caller 직접 호출·preflight 정책 중복과 global run-directory 차집합 의존을 제거한다. Caller capability, environment, live route readiness는 public `run`의 fresh all-cell preflight만 판정하며, CLI가 방출한 run id만 canonical pointer로 채택한다.
외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아닌 하나의 indivisible execution slice다. Scored failure도 SDD D10에 따라 보존하므로 아홉 cell이 모두 non-interrupted terminal이면 이 Epic의 실행 evidence는 완결된다.
## Archive Evidence Snapshot
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 exact `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 blocker를 보존한 readiness 절차의 완료이지 Milestone `route-readiness` 완료 선언이 아니다.
- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 current-HEAD Edge/Node artifact를 요구한다. 이 dependency는 TEST-1에서 fail-closed로 다시 확인한다.
- Generation 0은 `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`, generation 1은 `plan_cloud_G09_1.log` / `code_review_cloud_G10_1.log`로 공식 verdict 없이 보존됐다. Generation 0의 atomic intent는 유지하고 success-only acceptance를 바로잡았으며, generation 1의 직접 caller probe와 concurrent run 오인 가능성을 이 generation이 대체한다. 구현자는 이 plan과 active review만 실행하며 archive를 다시 읽지 않는다.
## Analysis
### Files Read
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
- Runner/fixture: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/workspace.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
- Rules/skills: `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
- Exact prior evidence: predecessor `complete.log`와 직접 연결된 plan/review log, 그리고 current task의 generation 0/1 pair
### SDD Criteria and Task Union
- SDD 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다.
- 이 pair의 Milestone task union은 정확히 `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다. S04는 C01, S05는 C02-C03, S06은 C04-C05, S07은 C06-C07, S08은 C08-C09에 대응한다. 다른 Milestone task는 소유하지 않는다.
- 각 cell은 event/timing/usage/workspace evidence를 보존하고 hybrid cell은 stage/terminal evidence도 보존해야 한다.
- D10과 state invariant는 scored failure를 보존하고 성공 결과만 선별하지 않도록 요구한다. 따라서 preflight 차단이나 1~8개 partial attempt는 미완료지만, fresh run에서 C01-C09가 각각 정확히 한 번 제출되어 9개 모두 `success|failed|timed_out|cancelled`이면 실행 Epic evidence는 완결된다. `resume --retry-failed`는 호출하지 않는다.
- S09-S12의 score/evaluator/report evidence는 이 Epic 범위가 아니다.
### Verification Context
- 별도 handoff는 없었다. repository-native CLI, manifest, SDD, living spec/contracts, current source와 exact predecessor evidence에서 실행 계약을 재구성했다.
- 현재 repository HEAD는 checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`다. 제품 source는 직전 `e86113f0ae3faecf8dfe715990e4755cdc42bedf` 이후 변경되지 않았고 checkpoint에는 task pair만 추가됐다.
- 2026-08-12 read-only audit에서 `../iop-s2`는 clean `dev`의 fixed HEAD이고 `iop-edge` metadata는 그 exact HEAD/clean과 일치하지만, `iop-node` metadata는 revision `9b2f...`, `vcs.modified=true`로 stale다. 현재는 step 2가 fail-closed해야 하며 scored `run`을 호출하면 안 된다.
- Generic dev runner 경로보다 SDD/immutable manifest의 task-specific local sibling `../iop-s2`가 이 benchmark의 고정 testbed 계약을 우선한다. Testbed source/config는 read-only다.
- Caller/provider를 public CLI 밖에서 실행하지 않는다. `command -v`는 설치 여부만 확인하고 caller version/help, secret dereference, endpoint/process/config observation과 live connectivity는 public `run`의 fresh all-cell preflight가 단독 소유한다.
- Secret 값과 raw private endpoint/config payload를 출력하지 않는다. Dynamic state는 CLI가 검증한 `agent-test/runs/bench-02/<emitted-run-id>/` 아래에만 쓰고, `run_id.log`는 그 exact state를 가리키는 task-local pointer다.
### Test Coverage Gaps
- Source behavior change는 없다. Repository full benchmark suite는 manifest, run allocation, single writer, fresh preflight, terminal state, retry 보존, caller registry와 skill command contract를 검증한다.
- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, workspace/web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 C01-C09 run 대상이다.
- 새 test는 추가하지 않는다. Fresh full benchmark suite와 actual one-shot run/public status가 acceptance oracle이다.
### Symbol References
None. Rename/remove/change하는 symbol이 없다.
### Split Judgment
- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가진다.
- 다섯 task id는 fixed seed, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유하므로 하나의 plan으로 실행한다.
- 디렉터리 predecessor index `05`는 archived `05+04_readiness_preflight/complete.log`로 task-protocol상 충족한다. 보존된 readiness blocker는 TEST-1의 fail-closed precondition이며 아직 해소되지 않았다.
### Scope and Write Ownership
- Source/spec/contract/SDD/Milestone는 변경하지 않는다. Caller/model/effort/preset 대체, extra repetition, manual run-state edit, `resume`, retry, score, evaluator, report, testbed rebuild/source/config mutation은 제외한다.
- Implementer-owned write: task-local `run_id.log`와 active `CODE_REVIEW-cloud-G10.md`의 implementation-owned evidence/checklist sections.
- CLI-owned dynamic write: public `run`이 방출한 exact `agent-test/runs/bench-02/<run-id>/**`. Implementer는 이 tree를 직접 편집하지 않는다.
- Review-owned write/archive는 review verdict 이후 절차이며 이 implementation slice가 실행하지 않는다.
### Final Routing
- `evaluation_mode=isolated-reassessment`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh`, mode=`pair`, status=`routed`.
- Build closures는 모두 true, capability gap은 none, grade scores=`1/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G09`, filename=`PLAN-cloud-G09.md`, catalog route=`worker/cloud/G09`.
- Review closures는 모두 true, capability gap은 none, grade scores=`2/2/2/2/2`, route basis=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog route=`review/cloud/G10`.
- `large_indivisible_context=false`; matched risks=`temporal_state,concurrent_consistency,boundary_contract,variant_product` (4); `review_rework_count=0`; `evidence_integrity_failure=false`. Grade boundary가 build cloud route를 결정한다.
## Dependencies and Execution Order
1. Active index `06+05`의 predecessor는 archived index `05+04`이고 exact `complete.log`가 task-protocol dependency를 충족한다.
2. Step 2가 host/testbed/artifact identity와 tool presence를 모두 통과해야 step 3을 시작할 수 있다. 현재 stale `iop-node`가 남아 있으면 stop하고 TEST-1을 unchecked로 유지한다.
3. Step 3의 public `run`은 caller/environment/live readiness를 fresh all-cell preflight로 판정하고, gate 통과 시 C01-C09를 동일 run에서 각각 한 번 제출한다. 이 명령은 이 plan에서 정확히 한 번만 실행한다.
4. CLI가 한 canonical run id를 방출하고 exact run root가 새로 생긴 경우에만 pointer를 쓴다. Id가 없거나 ambiguous하면 global directory 차집합으로 추측하지 않고 `blocked_unidentified`로 중단하며 이 plan에서 rerun하지 않는다.
5. Public status가 아홉 non-interrupted terminal attempt를 증명한 뒤에만 TEST-1을 완료로 표시한다.
## Implementation Checklist
- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [TEST-1] Immutable C01-C09 comparison run
#### Problem
Generation 1 directly invoked `agy --version/--help`, dereferenced benchmark environment references, and duplicated policy already owned by the public all-cell preflight. It also inferred identity from “exactly one new global run directory,” which can reject the correct non-repeatable run when another actor concurrently creates an unrelated run. These defects violate the benchmark skill boundary and weaken exactly-once evidence.
#### Solution
Use a pre-run gate only for task-specific host/testbed/artifact identity and command presence. Let the public `run` exclusively invoke callers and assess capability, secrets, endpoints, processes, config observation and connectivity. Invoke it once, preserve both streams and exit code, and extract exactly one unique `run-...` id only from CLI output. Confirm that id was absent before the command and its exact directory now exists; unrelated concurrent run roots are ignored.
Write `run_id.log` only after those checks. Use public read-only `status` on that exact id. Zero attempts, partial attempts, `running`, `interrupted`, invalid state, or an unidentified run leave TEST-1 unchecked with an exact resume condition. Exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted` complete this execution slice. Exit 0 must mean nine successes; retained non-success terminal results require exit 69. Never invoke `run` or `resume` again under this plan.
#### Modified Files and Checklist
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one CLI-emitted canonical `run-...` id; never infer, fabricate, or replace it.
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim command output, classification, deviations and blocker/resume evidence without secrets.
- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02/<run-id>/**` or anything under `../iop-s2`.
#### Test Strategy
No test file is added because there is no source/API change. Fresh repository benchmark tests verify the harness. The actual one-shot run and public status supply the external acceptance evidence unavailable to mocks.
#### Verification
Run Final Verification steps 1 and 2. Only if step 2 passes, run step 3 exactly once. Mark TEST-1 complete only when step 3 prints `classification=execution_complete`; otherwise retain unchecked items and exact blocker/resume evidence. The review agent reruns steps 1, 2 and 4 only, never step 3.
## Modified Files Summary
| Path | Owner | Item |
|------|-------|------|
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | Implementer | TEST-1 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | Implementer evidence fields | TEST-1 |
| `agent-test/runs/bench-02/<CLI-emitted-run-id>/**` | Public benchmark CLI | TEST-1 |
## Final Verification
1. Validate the immutable manifest and run the full repository benchmark suite. Expected: manifest valid and all discovered benchmark tests pass.
```bash
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
```
2. Prove only the fixed host/testbed/artifact and command-presence assumptions. This command does not invoke a caller/provider or reproduce public live-preflight policy. Expected: every check succeeds. If any check fails, stop before step 3.
```bash
set -euo pipefail
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
test -x "$binary" || blocked "$binary must be executable"
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
done
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
import subprocess, sys
expected = sys.argv[1]
for binary in sys.argv[2:]:
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
build = {}
for line in output.splitlines():
fields = line.strip().split("\t", 1)
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
key, value = fields[1].split("=", 1)
build[key] = value
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
print("ok: Edge/Node build identities match the clean testbed HEAD")
PY
```
3. Execute the nine cells exactly once. This block is implementation-only and non-repeatable. Expected task completion is `classification=execution_complete`; the benchmark exit remains 0 for all-success or 69 for retained failures.
```bash
set -euo pipefail
task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
run_root=agent-test/runs/bench-02
test ! -e "$task_dir/run_id.log"
before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
trap 'rm -f "$before" "$out" "$err"' EXIT
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
set +e
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
bench_exit=$?
set -e
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
printf '%s\n' 'stderr:'; cat "$err"
emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)"
emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)"
if test "$emitted_count" -ne 1; then
printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count"
exit 69
fi
bench_run_id="$(printf '%s\n' "$emitted_ids")"
if grep -Fxq "$bench_run_id" "$before"; then
printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id"
exit 69
fi
test -d "$run_root/$bench_run_id"
printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log"
set +e
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)"
status_exit=$?
set -e
printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output"
if test "$status_exit" -ne 0; then
printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id"
exit 69
fi
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
import ast, os
text = os.environ["STATUS_OUTPUT"]
assert text.startswith("ok: "), text
states = ast.literal_eval(text[4:])
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
bench_exit = int(os.environ["BENCH_EXIT"])
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
assert bench_exit == (0 if states["success"] == 9 else 69)
print(f"classification=execution_complete states={states}")
raise SystemExit(0)
if sum(states.values()) == 0 and bench_exit == 69:
print(f"classification=blocked_preflight states={states}")
else:
print(f"classification=blocked_partial states={states}")
raise SystemExit(69)
PY
```
4. Re-read the recorded run without invoking a provider and check both worktrees. Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed, no whitespace errors.
```bash
set -euo pipefail
pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
test "$(wc -l <"$pointer")" -eq 1
bench_run_id="$(tr -d '\n' <"$pointer")"
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
printf '%s\n' "$status_output"
STATUS_OUTPUT="$status_output" python3 - <<'PY'
import ast, os
text = os.environ["STATUS_OUTPUT"]
assert text.startswith("ok: "), text
states = ast.literal_eval(text[4:])
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
print(f"ok: nine retained terminal attempts states={states}")
PY
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
git diff --check
```

294
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log Normal file
View file

@ -0,0 +1,294 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=3 tag=API milestone-task=agy-iop-compatibility,managed-credential-dev,route-readiness,claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - official agy IOP 호환, managed dev 배포와 C01-C09 실행
## For the Implementing Agent
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 설계와 검증을 그대로 구현하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행한 명령/출력과 재개 조건만 evidence 필드에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다.
## Background
기존 active plan은 이미 준비됐다고 가정하고 C01-C09만 실행하도록 했으나, official `agy 1.1.12`의 실제 Gemini API-key transport와 dev managed credential runtime이 그 전제와 맞지 않았다. 실제 CLI는 Gemini-native `streamGenerateContent`, `x-goog-api-key`, `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`과 `event=init|step_update|result` stream JSON을 사용하며 API-key mode에서 `--effort`를 거부한다. Edge에는 이 ingress가 없고 dev는 legacy credential mode라 marked hybrid preset admission도 불가능하다. 이 plan은 caller→Edge→preset/provider→official lifecycle 전체가 한 번의 실제 preflight에서 닫힌 뒤에만 immutable 9-cell run을 시작한다.
## Archive Evidence Snapshot
- `plan_cloud_G09_0.log`/`code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`/`code_review_cloud_G10_1.log`, `plan_cloud_G09_2.log`/`code_review_cloud_G10_2.log`는 공식 verdict 없이 종료된 이전 실행 계획이다. 그중 plan=2의 one-run 원칙은 API-4에 유지하지만 “source/contract/dev 변경 없음” 전제는 폐기한다.
- predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 index `05` 의존성을 충족하지만 live blocker를 완료로 바꾸지 않는다.
- remote dev checkout은 `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이며 runtime config는 `build/dev-runtime/edge.yaml`, `control-plane.yaml`, `node-*.yaml`이다. public Edge는 현재 legacy HTTP `:18083`이다.
## Analysis
### Files Read
- Rules: `agent-ops/rules/project/domain/control-plane/rules.md`, `agent-ops/rules/project/domain/edge/rules.md`, `agent-ops/rules/project/domain/node/rules.md`, `agent-ops/rules/project/domain/platform-common/rules.md`, `agent-test/local/rules.md`, `agent-test/dev/rules.md`
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- Contracts/docs: `agent-contract/index.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`, `docs/edge-local-dev-guide.md`
- Edge: `apps/edge/internal/openai/routes.go`, `principal.go`, `server.go`, `route_resolution.go`, `dispatch_context.go`, `chat_handler.go`, `provider_tunnel.go`, `anthropic_bridge.go`, `single_request_preset_binding.go`, `apps/edge/internal/service/provider_tunnel.go`
- Config/Node/Control Plane: `packages/go/config/protocol_profile.go`, `edge_types.go`, `node_types.go`, `validate.go`, `credential_plane_config_test.go`, `apps/node/internal/node/tunnel_handler.go`, `apps/control-plane/cmd/control-plane/main.go`, `credential_commands.go`, `credential_http_handlers.go`, `secure_delivery_integration_test.go`, `configs/control-plane.yaml`, `scripts/e2e-credential-slot-smoke.sh`
- Benchmark: `scripts/agent_benchmark/agy_iop.py`, `agy_iop_test.py`, `live_iop.py`, `manifest.py`, `lifecycle.py`, `scripts/agent_comparison_benchmark.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
- Active/prior evidence: current task plan=0/1/2 logs and active pair, exact predecessor `complete.log`
### SDD Criteria
- SDD는 `[승인됨]`, 잠금 `해제`다. 이 pair의 first-line task ids는 `agy-iop-compatibility`, `managed-credential-dev`, `route-readiness`, `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다.
- S13은 official agy 1.1.12의 route별 Gemini base URL, IOP principal auth, Gemini request/tool/SSE와 실제 stream-json lifecycle을 요구하므로 API-1/API-2를 정한다.
- S14는 CA-signed mTLS, Edge HTTPS, projection, slot-route와 sealed lease/no-fallback evidence를 요구하므로 API-3을 정한다.
- S02는 모든 caller의 auth/route/effort/terminal live preflight를 요구하고, S04-S08은 C01-C09 one-submission evidence를 요구하므로 API-3 통과 후에만 API-4를 실행한다.
### Verification Context
- handoff는 없으며 repository source, contracts, official CLI 실행과 remote read-only preflight를 직접 확인했다.
- official `agy 1.1.12` known model은 `Gemini 3.6 Flash`다. API-key call은 `POST /v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse`, `x-goog-api-key`와 `contents/generationConfig/systemInstruction/toolConfig/tools`를 보냈다. `GOOGLE_GEMINI_BASE_URL` invalid endpoint가 network failure를 만들었고 `GEMINI_BASE_URL`은 override가 아니었다.
- official stream JSON은 `event=init`, `event=step_update`와 `event=result`; result는 `status`, `duration_seconds`, `num_turns`, `usage(input_tokens/output_tokens/thinking_tokens/cache_read_tokens/total_tokens)`를 가진다. 합성 `iop effective_binding`, metric, `system idle` event는 없다.
- remote dev는 clean release checkout과 `build/dev-runtime` config/artifacts를 갖지만 `credential_plane`/TLS material이 없고 legacy static provider credential이 남아 있다. 모든 raw provider token은 workspace `token/.gemini`, `.claude`, `.gpt`에서만 읽어 credential HTTPS stdin으로 등록하며 출력·명령 인자·tracked file에 넣지 않는다.
- external target: benchmark host Linux/AArch64의 current checkout, Mac dev Edge/Control Plane/local Node, inventory의 GX10/OneXPlayer/RTX5090 Nodes. Source sync→fresh build→config check→bounded restart→health/identity→direct/hybrid smoke 순으로 진행한다.
### Test Coverage Gaps
- Gemini-native ingress/auth/body conversion/SSE projection은 기존 test가 없다. normal text, tool, usage, invalid path/body, conflicting auth, route denial, stream error/cancel regression을 새 Go test로 작성한다.
- agy adapter test는 fake 1.1.11 help와 합성 events라 실제 1.1.12를 가리지 못한다. official help/event fixtures와 exact invocation env/argv tests로 교체하고 live registry integration test를 갱신한다.
- managed credential code 자체는 deterministic full-cycle smoke가 있다. gap은 실제 dev security material, slot/route와 multi-node rollout이므로 config check와 live secret-safe smoke가 필요하다.
- C01-C09 결과는 unit test로 대체할 수 없다. API-4의 public run/status가 유일한 scored evidence다.
### Symbol References
삭제·rename symbol은 없다. 새 Gemini handler/bridge symbol은 `routes.go` 한 곳에서 등록되고, benchmark 상수 변경 call site는 `agy_iop.py`, `agy_iop_test.py`, `live_iop.py`와 connectivity integration tests다.
### Split Judgment
- API translator, official event parser, managed route authorization과 actual caller smoke는 각각 독립 test가 있지만, 이 task의 correctness 조건은 `GEMINI_API_KEY`가 upstream key가 아닌 projected IOP token이고 route-specific base가 direct/hybrid binding으로 연결된다는 하나의 cross-boundary invariant다.
- API-4는 그 invariant가 live에서 닫힌 후 exactly once여야 하므로 별도 활성 sibling으로 이동하면 기존 fixed index와 non-repeatable state coordination을 분리할 수 없다. 하나의 순차 plan으로 유지한다.
- index `05` predecessor는 archived `complete.log`로 충족됐다.
### Scope Rationale
- Gemini non-stream/batch/files/cache/tuning API, arbitrary third-party Gemini SDK, unofficial agy custom model, new protocol-profile driver, pricing과 report/scoring task는 제외한다.
- 기존 OpenAI/Anthropic caller schema, preset stage semantics와 provider OpenAI-compatible upstream은 변경하지 않는다. Gemini ingress는 이를 재사용하는 outer translation만 소유한다.
- remote dev config와 operator secret material은 repository 밖 external state다. raw token/cert private key/slot alias/lease id는 plan/review/run evidence에 쓰지 않는다.
- scored run 전 preflight가 실패하면 API-4는 실행하지 않는다. 한 번 시작한 public run은 retry/resume하지 않는다.
### Final Routing
- `evaluation_mode=first-pass`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`.
- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`.
- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`.
## Dependencies and Execution Order
1. API-1 Go tests와 API-2 Python tests/full local suite가 통과해야 API-3 remote build/deploy를 시작한다.
2. API-3는 current source의 fresh artifacts, managed config validation, all runtime health, official agy direct/hybrid smoke와 all-cell public preflight가 모두 통과해야 완료다.
3. API-3 완료 전 API-4 public `run`을 호출하지 않는다. API-4는 exactly once이며 CLI-emitted run id만 보존한다.
## Implementation Checklist
- [x] API-1: Implement the route-qualified Gemini `streamGenerateContent` ingress, `x-goog-api-key` principal authentication, bounded Gemini↔Chat/tool/SSE translation, caller-safe errors and full Edge regression tests without bypassing existing route/preset admission.
- [x] API-2: Replace the invented agy 1.1.11/OpenAI env/event contract with the official agy 1.1.12 Gemini API-key invocation and stream-json parser, config-owned effective binding evidence, exact redaction and updated unit/integration fixtures.
- [x] API-3: Build and deploy fresh artifacts to dev, provision the complete operator-owned managed credential/TLS composition and projected provider routes, then pass deterministic credential qualification plus official agy direct/hybrid and all-caller live preflight without secret leakage or legacy fallback.
- [ ] API-4: Invoke the immutable public C01-C09 run exactly once only after API-3, preserve its CLI-emitted canonical run id and all nine non-interrupted terminal attempts, and never retry, resume, substitute, or manually edit run state.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [API-1] Gemini-native Edge ingress
#### Problem
`apps/edge/internal/openai/routes.go:11-19` registers only OpenAI, Anthropic and Ollama surfaces. `principal.go:164-183` understands bearer and Anthropic `X-Api-Key` but not official agy `x-goog-api-key`. Sending agy's native body into `chat_handler.go:55-190` is impossible without a schema and streaming translation, so current official calls either bypass IOP or fail before route/preset admission.
#### Solution
Register `/gemini/` under the shared auth wrapper. Parse only `/gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse`, freeze the bounded body, validate the contract fields, and translate to a synthetic internal streaming Chat request whose `model` is `{route-id}`. Call the existing Chat handler directly with the already-authenticated context and a streaming response writer that maps Chat SSE text/reasoning/tool/usage/terminal/error frames to Gemini SSE. Accept `x-goog-api-key` as a surface-specific IOP token, require equality with bearer if both exist, and never forward it as provider auth.
Before:
```go
// apps/edge/internal/openai/routes.go:11-19
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
s.registerAnthropicRoutes(mux)
```
After:
```go
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent))
s.registerAnthropicRoutes(mux)
```
#### Modified Files and Checklist
- [x] `apps/edge/internal/openai/routes.go`: register Gemini route and Gemini-native auth/error selection.
- [x] `apps/edge/internal/openai/principal.go`: add constant-time `x-goog-api-key` principal extraction/conflict checks.
- [x] `apps/edge/internal/openai/gemini_types.go`: define bounded request/response/error DTOs for the contracted subset.
- [x] `apps/edge/internal/openai/gemini_handler.go`: validate path/query/body, construct internal Chat request and preserve cancellation.
- [x] `apps/edge/internal/openai/gemini_bridge.go`: incrementally translate Chat SSE to Gemini SSE with bounded tool assembly and usage mapping.
- [x] `apps/edge/internal/openai/gemini_handler_test.go`: cover normal, boundary, auth, route, tool, usage, error and cancellation behavior.
#### Test Strategy
Write table-driven tests using existing mock run/provider/preset services. Assert zero dispatch for malformed path/body/conflicting key, direct and preset route identity, no inbound key in provider headers/log body, streaming text/tool/usage shape, one terminal and cancellation propagation.
#### Verification
Run `gofmt -w` on changed Go files, then `go test -count=1 ./apps/edge/internal/openai`. Expected: all package tests pass fresh.
### [API-2] official agy 1.1.12 benchmark adapter
#### Problem
`scripts/agent_benchmark/agy_iop.py:48-66` hard-codes 1.1.11, nonexistent `AGY_*` OpenAI transport and synthetic duration events. `build_agy_invocation` adds unsupported `--effort`, while `AgyEventParser` waits for events official agy never emits. `live_iop.py:881-908` therefore classifies the installed valid CLI as an implementation gap.
#### Solution
Use `GEMINI_API_KEY` and `GOOGLE_GEMINI_BASE_URL`; map `gemini-3.6-flash` to official label `Gemini 3.6 Flash`; omit `--effort`. Version-gate 1.1.12 and its real help options. Parse `init`, `step_update`, `result`; preserve result/step usage as caller-reported count metrics and duration seconds as a duration metric, derive finish+idle from one successful result plus process exit/quiescence, and classify non-success as failure. Effective route/stage binding comes only from the independently hashed config observation validated before launch, never a fabricated caller event. Structural redaction persists only event/state/step type/status and numeric metrics.
#### Modified Files and Checklist
- [x] `scripts/agent_benchmark/agy_iop.py`: replace constants, invocation, capability and official event parser/redactor.
- [x] `scripts/agent_benchmark/agy_iop_test.py`: replace fake transport/synthetic event tests with captured official 1.1.12 shapes and boundary cases.
- [x] `scripts/agent_benchmark/live_iop.py`: derive catalog root from route-qualified Gemini base and bind config-owned effective route before agy invocation.
- [x] `scripts/agent_benchmark/connectivity_integration_test.py`: update live agy seams and exact admitted binding assertions.
- [x] `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`: replace synthetic fixture with content-free official event shapes.
#### Test Strategy
Tests assert exact env allowlist/argv, no `--effort`, known-model rejection, 1.1.12 help/version gate, official success/error/result order, usage/duration units, malformed/duplicate result rejection, config-binding mismatch and durable secret/content redaction. No unit test contacts a provider.
#### Verification
Run `python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test`. Expected: all tests pass.
### [API-3] Managed dev deployment and live readiness
#### Problem
Remote `build/dev-runtime/edge.yaml` serves legacy HTTP and contains static provider auth. Marked presets require managed principal projection/stage authorization, and the new Gemini ingress contract requires HTTPS plus `x-goog-api-key` as an IOP token. Source compatibility alone cannot prove C03/C07 or hybrid readiness.
#### Solution
First run the repository deterministic credential-slot smoke. Build current Control Plane/Edge/Node artifacts for each target and stage them on the clean remote release checkout. Under remote `build/dev-runtime/.secrets/credential-plane/` generate one CA, role/name-bound CP/Edge/Node certs, Edge HTTP cert for `toki-labs.com`, at-rest keyring, issuer Ed25519 and per-node X25519 recipient keys with restrictive modes. Create managed candidate configs that remove all legacy bearer/principal/provider header/env values, enable CP credential HTTPS/edge-wire mTLS, Edge CP/Node mTLS and public HTTPS, and Node lease crypto. Bootstrap one benchmark principal into local protected `token/.iop-bench`, register `.gemini`, `.claude`, `.gpt` through credential HTTPS stdin as slots/routes for the canonical direct and stage models, then bounded-restart CP→Edge→Nodes. Verify fresh projection, advertised route ids, safe slot revision/lease attribution, direct agy and hybrid agy through route-qualified bases, Claude/Codex compatibility and the public all-cell preflight. Update the Gemini contract status to active and the human dev guide only after these pass.
#### Modified Files and Checklist
- [x] `agent-contract/outer/gemini-compatible-api.md`: change planned→active and record final implemented constraints only after live proof.
- [x] `agent-contract/index.md`: keep source pointers and trigger terms aligned with implemented files.
- [x] `docs/edge-local-dev-guide.md`: add secret-safe route-qualified official agy and managed dev verification commands without token values.
- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record sanitized deterministic/remote/live outputs and exact external config paths.
#### Test Strategy
Reuse `make test-credential-slot-smoke` for deterministic security lifecycle. Remote checks use each binary's `config check` where available, TLS health, authenticated model discovery, official agy direct/hybrid minimal workspace calls and benchmark `preflight`; raw provider responses and credentials are not retained.
#### Verification
Run the Final Verification local suite, deterministic credential smoke, remote clean/source/artifact/config checks, then official agy direct/hybrid and `agent_comparison_benchmark.py preflight`. Expected: all cells ready, no legacy source accepted, and sanitized evidence only.
### [API-4] Immutable C01-C09 run
#### Problem
The benchmark must retain one attempt per cell and scored failures, but invoking it before API-3 or rerunning after partial output would violate the approved manifest and D10.
#### Solution
After API-3, invoke public `run` exactly once. Extract exactly one CLI-emitted new run id, write only that id to `run_id.log`, and use public `status` to require exactly nine `success|failed|timed_out|cancelled` with zero running/interrupted. Never call `resume`, retry failed cells, directly invoke scored callers, or edit the run tree.
#### Modified Files and Checklist
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store one CLI-emitted canonical run id after the one-time command.
- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim sanitized run/status output and classification.
#### Test Strategy
No additional test file. The immutable manifest validation, full benchmark suite, fresh all-cell preflight, one-time public run and read-only status are the acceptance oracle.
#### Verification
Run Final Verification step 5 exactly once. Mark API-4 complete only for nine retained non-interrupted terminal attempts; otherwise record blocker and do not rerun.
## Modified Files Summary
| Path | Item |
|------|------|
| `apps/edge/internal/openai/routes.go` | API-1 |
| `apps/edge/internal/openai/principal.go` | API-1 |
| `apps/edge/internal/openai/gemini_types.go` | API-1 |
| `apps/edge/internal/openai/gemini_handler.go` | API-1 |
| `apps/edge/internal/openai/gemini_bridge.go` | API-1 |
| `apps/edge/internal/openai/gemini_handler_test.go` | API-1 |
| `scripts/agent_benchmark/agy_iop.py` | API-2 |
| `scripts/agent_benchmark/agy_iop_test.py` | API-2 |
| `scripts/agent_benchmark/claude_iop.py` | API-2 |
| `scripts/agent_benchmark/claude_iop_test.py` | API-2 |
| `scripts/agent_benchmark/codex_iop.py` | API-2 |
| `scripts/agent_benchmark/codex_iop_test.py` | API-2 |
| `scripts/agent_benchmark/lifecycle.py` | API-2, API-4 recovery |
| `scripts/agent_benchmark/lifecycle_test.py` | API-2, API-4 recovery |
| `scripts/agent_benchmark/attempts.py` | API-4 recovery |
| `scripts/agent_benchmark/attempts_test.py` | API-4 recovery |
| `scripts/agent_benchmark/live_iop.py` | API-2 |
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-2 |
| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-2 |
| `scripts/e2e-credential-slot-smoke.sh` | API-3 |
| `agent-contract/outer/openai-compatible-api.md` | API-3 |
| `agent-contract/outer/gemini-compatible-api.md` | API-3 |
| `agent-contract/index.md` | API-3 |
| `docs/edge-local-dev-guide.md` | API-3 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | API-4 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | API-1, API-2, API-3, API-4 evidence |
## Final Verification
1. Format and run fresh focused suites:
```bash
gofmt -w apps/edge/internal/openai/routes.go apps/edge/internal/openai/principal.go apps/edge/internal/openai/gemini_types.go apps/edge/internal/openai/gemini_handler.go apps/edge/internal/openai/gemini_bridge.go apps/edge/internal/openai/gemini_handler_test.go
go test -count=1 ./apps/edge/internal/openai
python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test
```
Expected: all focused tests pass.
2. Run the full local suites and contract hygiene:
```bash
go test -p 1 -count=1 ./...
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Expected: all packages/tests pass, manifest is valid, no whitespace error.
3. Run deterministic managed credential qualification outside the repo temp tree:
```bash
credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)"
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
rmdir "$credential_smoke_parent"
```
Expected: CA/mTLS, two-slot, rotation, revoke/no-fallback and Messages/Chat qualification pass.
4. After fresh dev deploy, run the sanitized live readiness commands documented in `docs/edge-local-dev-guide.md`: official agy direct and hybrid minimal calls through `https://toki-labs.com:18083/gemini/<route-id>`, authenticated model discovery, and:
```bash
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Expected: C01-C09 all report ready with official caller version, managed route/stage binding and no secret output.
5. Implementation-only, exactly once after step 4: run the immutable public benchmark and preserve its CLI-emitted id.
```bash
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Then write the emitted `run-...` id to `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` and run:
```bash
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
```
Expected: exactly nine retained non-interrupted terminal attempts. Do not rerun step 5.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

203
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log Normal file
View file

@ -0,0 +1,203 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=4 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - pre-scored interruption 이후 C01-C09 replacement 실행
## For the Implementing Agent
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 명령과 상태 경계를 그대로 실행하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다.
## Background
plan=3의 제품 코드, managed dev 배포와 caller readiness는 완료됐지만 최초 public `run`은 caller launch 전에 control socket `EINVAL`로 중단됐다. 그 결함과 pre-registration reconcile은 수정·회귀 검증됐고, SDD D10은 실패 evidence를 보존한 새 attempt를 허용한다. 이 revision은 이전 interruption을 덮어쓰지 않고 새 public run identity에서 C01-C09를 실행하는 작업만 소유한다.
## Archive Evidence Snapshot
- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다.
- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다.
- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다.
- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다.
## Analysis
### Files Read
- Rules/skills: `agent-ops/rules/project/rules.md`, `agent-ops/rules/common/rules-roadmap.md`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-ops/skills/common/router.md`, `agent-ops/skills/common/plan/SKILL.md`, `agent-ops/skills/common/finalize-task-routing/SKILL.md`, `agent-ops/skills/common/update-test/SKILL.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
- Test rules: `agent-test/local/rules.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md`
- Roadmap/SDD: `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- Benchmark/runtime: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/codex_iop.py`, `docs/edge-local-dev-guide.md`
- Active evidence: `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log`, `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log`
### SDD Criteria
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, 상태 `[승인됨]`, 잠금 `해제`, 사용자 리뷰 없음.
- `milestone-task`는 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`다.
- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다. Evidence Map은 caller별 event/timing/usage/workspace와 hybrid stage/terminal evidence를 요구한다.
- D06은 cell별 repetitions=1과 fresh session을, D10은 실패 보존과 새 attempt 기록을 요구한다. 따라서 TEST-1은 S02 readiness를 재확인하고 TEST-2는 S04-S08의 새 run evidence를 append-only로 생성한다.
### Verification Context
- Environment: dev. Rules state는 usable이며 matching profile은 testing/edge/node smoke다. 파일 수정 없는 `resolve-context`로 local current checkout, remote dev runtime과 public benchmark CLI를 판정했다.
- Local runner: Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, HEAD `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`, 현재 task 구현 변경 때문에 dirty다. `../iop-s2`는 branch `dev`, HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean이다.
- Remote runtime: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, Darwin/arm64, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이다. `build/dev-runtime/bin/edge` hash는 위 snapshot과 일치하고 PID 172, ports 18082/18083/18084/19093/19101은 모두 up이다.
- CLI: `scripts/agent_comparison_benchmark.py`는 validate/preflight/run/resume/status/score/report를 제공하고 manifest validate는 `ok: manifest is valid`다. caller binary/version/help와 model catalog는 fresh public preflight가 실제로 확인했다.
- Runtime references: principal은 보호된 `token/.iop-bench`, private dev CA는 `token/iop-dev-ca.pem`에서 같은 process environment로만 읽는다. caller base/secret reference와 config observation 값만 child allowlist에 전달하며 값은 출력·tracked 파일·run metadata에 기록하지 않는다.
- Read-only preflight: 무설정 호출은 `implementation_gap=9`로 fail-closed했고 attempt를 만들지 않았다. reference 주입 뒤 `ok: preflight run_id=run-20260812T035838Z-4c4a056c63b1 status=ready ready=9 registration_required=0 implementation_gap=0`으로 통과했다.
- Constraints: benchmark CLI 밖 caller/provider 호출, route/model/effort 대체, old run resume, `--retry-failed`, run tree 편집, testbed 쓰기, raw secret 출력은 금지한다. 새 execution run이 retained failure로 끝나면 status만 기록하고 재실행하지 않는다.
- Gaps: 없음. Confidence는 high이며 test rule 유지보수는 필요 없다.
### Test Coverage Gaps
- 새 제품 동작이나 source 변경은 없다. control socket symlink와 pre-registration reconcile 회귀는 plan=3에서 lifecycle/attempt tests 및 전체 Python suite로 검증됐다.
- 남은 gap은 실제 C01-C09 결과뿐이며 unit test로 대체할 수 없다. public preflight/run/status가 acceptance oracle이다.
### Symbol References
- 없음. symbol rename, remove, dependency 변경이 없다.
### Split Judgment
- 새 run identity의 preflight, attempt allocation과 terminal status는 하나의 append-only state invariant다. 별도 sibling으로 나누면 runtime identity와 one-submission evidence가 분리되므로 같은 task path의 단일 plan으로 유지한다.
### Scope Rationale
- Edge/Node/Control Plane, caller adapter, manifest, SDD, contract와 dev config를 수정하지 않는다. 현재 ready runtime을 소비하는 검증 실행만 범위다.
- objective validation, blind scoring, performance aggregation과 report는 세 번째 Epic S09-S12의 후속 plan으로 남긴다.
- old interrupted run resume/retry/reconcile, 실패 cell 재시도, provider 직접 호출은 제외한다.
### Final Routing
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`.
- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`.
- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`.
## Dependencies and Execution Order
1. TEST-1의 manifest validation과 fresh all-cell preflight가 `ready=9`여야 TEST-2를 시작한다.
2. TEST-2는 public `run`을 한 번만 호출한다. CLI가 만든 새 canonical run id를 기록한 뒤에는 read-only `status`만 사용한다.
## Implementation Checklist
- [ ] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output.
- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [TEST-1] Protected runtime binding and fresh readiness
#### Problem
`scripts/agent_benchmark/live_iop.py:1-5,259-266` requires caller base/secret references and an operator-owned config observation. A fresh shell without those references produced preflight-only `implementation_gap=9`, even though the deployed runtime itself remained healthy. Starting scored work before a closed `ready=9` violates S02 and the benchmark skill stop condition.
#### Solution
Read the existing principal and CA only inside the execution process, bind Claude/agy/Codex to their deployed HTTPS surfaces, and provide the already-verified route/preset observation. Run manifest validation, the focused lifecycle regression, caller version/help checks and public preflight. No repository source or configuration is changed.
#### Modified Files and Checklist
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record exact secret-safe validation/preflight output.
#### Test Strategy
No new test file is required because there is no behavior change. Run fresh lifecycle/attempt regressions and the public preflight; cached output is not accepted.
#### Verification
Run the Final Verification readiness block. Expected: manifest valid, focused tests pass, caller commands resolve, environment reference booleans are true, and preflight reports `ready=9` with no registration or implementation gap.
### [TEST-2] One replacement C01-C09 execution run
#### Problem
`code_review_cloud_G10_3.log:159-197` records that the former execution command ended before caller launch and retained no C01-C09 result. S04-S08 therefore remain incomplete even though the infrastructure fault is fixed and readiness is green.
#### Solution
With TEST-1 references still bound, call the public `run` exactly once to create a distinct execution run. Preserve all older run roots, store only the CLI-emitted canonical execution id in `run_id.log`, and use public `status` to require nine `success|failed|timed_out|cancelled`, zero `interrupted`, and zero `running`. A retained failed/timed-out/cancelled cell remains evidence and is not retried.
#### Modified Files and Checklist
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store exactly one CLI-emitted execution run id.
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record the verbatim run output, exit code, canonical id and public status.
#### Test Strategy
No new test file is required. The deterministic CLI's append-only run/status contract is the only valid live acceptance path; callers are never invoked separately.
#### Verification
Run the Final Verification execution block once. Expected: exactly one new execution run id and nine retained accepted terminal attempts with zero interrupted/running. Do not invoke `run` or `resume` again in implementation or review.
## Modified Files Summary
| Path | Item |
|------|------|
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2 |
## Final Verification
1. Run fresh local contract checks:
```bash
python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output.
2. In one protected shell, bind runtime references and run fresh all-cell preflight. Read token values only into variables; never print them:
```bash
set -euo pipefail
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
export IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083
export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083
export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
BENCH_CONFIG="$(python3 - <<'PY'
import json
routes = [
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
]
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
PY
)"
export BENCH_CONFIG
for tool in python3 claude agy codex git; do command -v "$tool"; done
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`. If it does not, stop before TEST-2.
3. In the same protected shell, invoke the execution command exactly once and preserve its direct CLI output. A nonzero exit is acceptable only when public status later proves nine retained accepted terminal attempts:
```bash
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
Expected: the CLI emits one canonical execution id. Record the command exit code and stdout/stderr verbatim, then write that exact emitted id to `run_id.log` and use step 4 for status. Never rerun this block.
4. Verify the stored pointer without invoking a provider:
```bash
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id"
git diff --check
```
Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

86
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_6.log Normal file
View file

@ -0,0 +1,86 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=6 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - 지속 승인 기반 C01-C09 fresh scored execution
## For the Implementing Agent
`CODE_REVIEW-*-G??.md` 구현 섹션을 채우는 것이 필수 마지막 단계다. protected local references를 process environment에서만 읽고, public benchmark CLI를 통해 preflight와 새 `run`을 수행한다. caller/provider 직접 호출, old run resume, `--retry-failed`, run tree 편집, secret 출력은 금지한다. 새 run이 retained failure로 끝나더라도 사용자의 지속 승인은 이후 새 plan 진행 권한으로 유지되지만, 이 plan 안에서는 `run`을 한 번만 호출하고 status evidence를 기록한 뒤 멈춘다.
## Background
plan=5에서 Codex production usage parser와 config-owned execution/scoring binding 결함을 수정했고 focused 41 tests와 전체 benchmark 429 tests가 통과했다. 이후 `USER_REVIEW.md`는 사용자가 “이후는 계속 승인 상태이니 이어서 진행”이라고 명시해 external-execution 조건이 해소됐으며 `user_review_0.log`로 보존됐다. 이 revision은 모든 기존 run을 보존한 distinct new run identity에서 C01-C09를 실행한다.
## Archive Evidence Snapshot
- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained run `run-20260812T040619Z-00a5e6664764`의 parser/binding failure와 FAIL evidence.
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 source fix, focused 41/OK, full 429/OK와 새 execution authorization gate.
- `user_review_0.log`: 사용자 지속 승인으로 exact new public run 실행 조건이 RESOLVED 됐다.
- 이전 run은 어떤 방식으로도 resume/retry/reconcile/edit하지 않는다. 새 execution CLI가 출력한 id만 `run_id.log`에 기록한다.
## Analysis
### SDD Criteria
- Approved/unlocked SDD의 S04-S08은 C01-C09 one-submission, fresh session, caller lifecycle/usage/workspace 및 hybrid terminal evidence를 요구한다.
- D06은 repetitions=1, D10은 실패 evidence 보존과 새 attempt identity를 요구한다.
- milestone-task는 기존 다섯 id를 그대로 유지한다.
### Verification Context
- Local source regression: focused 41/OK, full 429/OK, manifest valid, diff check clean.
- Dev runtime: prior fresh public preflight가 `ready=9`; existing protected principal과 dev CA는 `token/.iop-bench`, `token/iop-dev-ca.pem`에 있다.
- Execution environment binds Claude/agy/Codex base URLs and independently verified config observation in one process only. Raw values are never printed or persisted in tracked task evidence.
- Constraints: deterministic public CLI only; no direct caller/provider, no route/model/effort substitution, no old run resume/retry, no run tree mutation.
- User authorization is continuing for subsequent necessary new executions. Each individual plan still calls public `run` at most once and preserves failure before another plan.
### Test Coverage Gaps
- Deterministic source tests are closed. Remaining gap is real C01-C09 scored evidence, which only public `run/status` can produce.
### Split Judgment
All-cell preflight, immutable run creation, nine attempt allocations and terminal status form one append-only execution invariant. Splitting would separate readiness and the exact authorized run identity, so one plan is retained.
### Scope Rationale
- No product/runtime/config/source change is planned.
- This plan owns only fresh verification, protected preflight, one new public run, stored pointer and status evidence.
- score/report and objective validation remain later Epic work.
### Final Routing
- isolated reassessment; closures true; capability gap none.
- Build scores `2/2/2/2/2`, risks all five, rework=2, integrity=false; route `grade-boundary`, cloud G10, `PLAN-cloud-G10.md`.
- Review scores `2/2/2/2/2`; `official-review`, cloud G10, `CODE_REVIEW-cloud-G10.md`.
## Dependencies and Execution Order
1. Fresh deterministic tests and manifest validation pass.
2. Protected public preflight reports exactly `ready=9` and zero gaps.
3. Call public `run` exactly once.
4. Store only a CLI-emitted canonical run id; use read-only status to verify nine accepted terminal attempts, `running=0`, `interrupted=0`.
## Implementation Checklist
- [ ] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh.
- [ ] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`.
- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output.
## Modified Files Summary
| Path | Item |
|---|---|
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-3 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 evidence |
## Final Verification
1. `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
3. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` and `git diff --check`
4. In one secret-safe process, read `token/.iop-bench`, bind the existing dev endpoints/CA/config observation, and call public `preflight`. Require `ready=9 registration_required=0 implementation_gap=0`.
5. In that protected process, call `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once. Record verbatim stdout/stderr and exit code. Write `run_id.log` only if the CLI emits one canonical id.
6. If an id was emitted, call public `status`. Completion requires nine retained terminal attempts, zero running/interrupted. If no id was emitted, identify the newly created run root read-only for blocker evidence but do not fabricate `run_id.log`.
After completing, fill CODE_REVIEW implementation-owned sections and keep the active pair for review.

79
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_7.log Normal file
View file

@ -0,0 +1,79 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - agy failed lifecycle terminalization 및 fresh C01-C09 run
## For the Implementing Agent
R4의 선택된 수정만 구현하고 credential-free tests를 통과한 뒤 protected preflight와 public `run`을 각 계획대로 수행한다. 원인 재조사, old run resume, retry-failed, caller/provider 직접 호출, state tree 편집, secret 출력은 금지한다. 새 run은 이 plan에서 정확히 한 번 호출한다.
## Background
plan=6 new run `run-20260812T043704Z-041320764a11`은 C05 Codex를 success로 종결해 R1/R2 fix를 검증했지만, C03 agy의 valid `nonzero_exit`를 live adapter가 `stream_incompatible` 예외로 바꿔 attempt를 running으로 남겼다. 사용자의 이후 실행 지속 승인은 유효하다. 기존 run은 보존하고 R4 fix 이후 distinct new run을 수행한다.
## Archive Evidence Snapshot
- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: ready=9, one public run, C02 failed/C05 success/C03 running과 R4 FAIL evidence.
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: Codex R1/R2 fixes and 429/OK.
- `user_review_0.log`: subsequent execution remains authorized.
## Finding Resolution Map
| Finding | Evidence | Root Cause | Selected Fix | Mode | Acceptance |
|---|---|---|---|---|---|
| Required R4 | C03 lifecycle is clean nonzero_exit/cleanup complete, but measurement/web absent and attempt running; agy branch raises after `observed_result` | post-admission execution failure is incorrectly reclassified as connectivity failure | return bound failed lifecycle directly; apply observed binding gate only to successful lifecycle; add integration test proving failure terminalization/continuation | direct-fix | focused integration, full 429+, ready=9, one new run with nine terminal attempts |
## Analysis
### SDD Criteria
S04-S08/D06/D10 and the five milestone-task ids remain unchanged. Failed cells are accepted terminal evidence; `running|interrupted` are not.
### Verification Context
- Local/dev context and protected `token/.iop-bench`, `token/iop-dev-ca.pem` bindings are unchanged.
- Previous fresh ready=9 and Codex success prove runtime and R1/R2.
- New gap is deterministic in `_LiveAdapter.invoke` and can be tested without provider access.
- Continuing authorization permits a new run after the fix; each plan still invokes run once.
### Test Coverage Gaps
No test proves agy non-success lifecycle is returned to RunStore and later slots continue. Add the smallest live-registry/run-slots regression using invoker seams and existing production sidecar assertions.
### Split Judgment
R4 terminalization and its verification in a new nine-cell run are one correctness boundary; separating them would leave S04-S08 unverified.
### Scope Rationale
Modify only `live_iop.py`, its integration test, active evidence and `run_id.log`. Do not change agy transport/parser, dev runtime, generic recovery, other callers, scoring/report.
### Final Routing
isolated reassessment, closures true, scores 2/2/2/2/2, five loop risks, rework=3, integrity=false; build grade-boundary cloud G10, review official cloud G10.
## Implementation Checklist
- [ ] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams.
- [ ] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots.
- [ ] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check.
- [ ] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted.
- [ ] Fill CODE_REVIEW implementation-owned evidence.
## Modified Files Summary
| Path | Item |
|---|---|
| `scripts/agent_benchmark/live_iop.py` | FIX-1 |
| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | evidence |
## Final Verification
1. `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test`
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
3. manifest validate and `git diff --check`.
4. Protected public preflight must report ready=9.
5. Public `run` exactly once. Store only CLI-emitted id and use status. PASS requires nine terminal, running=0, interrupted=0.
After completing, fill active CODE_REVIEW evidence and stop for review.

178
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_local_G06_5.log Normal file
View file

@ -0,0 +1,178 @@
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=5 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Plan - Codex production stream 및 config-owned binding 정합화
## For the Implementing Agent
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 R1/R2의 선택된 수정만 구현하고 검증 결과를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 원인을 재조사하거나 다른 해결책을 선택하지 않는다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, public benchmark `run|resume`, provider 직접 호출, verdict·archive·`complete.log` 작성은 하지 않는다.
## Background
plan=4의 새 public run `run-20260812T040619Z-00a5e6664764`은 C02를 failed로 종결한 뒤 C05 Codex 프로세스가 실제로 끝났음에도 attempt를 `running`으로 남기고 exit 69로 중단됐다. 리뷰가 retained lifecycle stdout을 동일 parser에 재생해 current Codex usage key 누락과 config-owned binding/caller-owned enforcement 모순을 확정했다. 이 revision은 그 두 source defect와 회귀 테스트만 수정한다. 실패 run은 append-only evidence로 보존하며 재개·retry·state 편집·새 scored 실행을 하지 않는다.
## Archive Evidence Snapshot
- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다.
- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다.
- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다.
- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다.
- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다.
## Finding Resolution Map
| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands |
|---|---|---|---|---|---|---|
| Required R1 | C05 retained stdout replay가 `cache_write_input_tokens`에서 `CodexJSONLError`; 기존 9 parser tests는 모두 통과해 coverage gap 확인 | `_CODEX_USAGE_FIELDS`와 tracked fixture가 current caller usage schema보다 오래됨 | `cache_write_input_tokens -> cache_write_tokens`를 exact mapping에 추가하고 unit fixture/assertion을 production shape로 갱신하며 total은 합성하지 않음 | direct-fix | 실제 `turn.completed.usage`가 parser terminal success 및 typed cache-write metric으로 보존됨 | `python3 -m unittest scripts.agent_benchmark.codex_iop_test`; focused integration/full benchmark tests |
| Required R2 | parser는 absent binding을 허용하지만 `_LiveAdapter.invoke`와 `_LiveScoringAdapter.invoke`는 absent를 mismatch로 거부; integration fake가 synthetic event 주입 | config observation으로 admission ownership을 이전한 뒤 두 consumer와 fixture가 구 caller-event gate를 유지함 | admitted config binding을 canonical result로 사용하고 optional non-None caller observation만 exact mismatch 검사; execution/scoring 테스트에서 absent PASS와 mismatch FAIL 검증 | direct-fix | real Codex의 absent observation이 admitted binding으로 진행되고 실제 mismatch만 fail-closed함 | named connectivity integration tests; full benchmark tests; `git diff --check` |
## Analysis
### Files Read
- Reviewer evidence: archived plan/review pair 위 두 exact 경로와 retained run의 C05 `attempt.json`, `lifecycle-result.json`, `cleanup-receipt.json`.
- Source/tests: `scripts/agent_benchmark/codex_iop.py`, `codex_iop_test.py`, `live_iop.py`, `connectivity_integration_test.py`, `lifecycle.py`, `attempts.py`, `scoring.py`, tracked Codex JSONL fixture.
- Rules/skills: project/testing rules, local test rules, `code-review`, `plan`, `finalize-task-routing`, project benchmark skill.
- Roadmap/SDD: active comparison Milestone와 approved/unlocked SDD S04-S08/D06/D10.
### SDD Criteria
- `milestone-task`는 기존과 동일한 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`를 유지한다.
- S04-S08은 C01-C09 retained terminal evidence를 요구한다. 이 source-fix revision은 실패한 실행 경계를 복구하지만 그 scored evidence 자체를 대체하지 않는다.
- D10에 따라 failed/incomplete run을 수정하거나 재사용하지 않는다. 이후 새 scored 실행은 별도 명시적 authorization과 새 plan에서만 수행한다.
### Verification Context
- Environment는 local source/test다. provider, remote dev, protected token이 필요하지 않다.
- Reviewer가 production retained output으로 exact parser failure를 재현했고 current test가 gap을 놓치는 것도 확인했다.
- 변경 후 credential-free unit/integration/full Python suite와 manifest validation을 fresh 실행한다. cached output은 허용하지 않는다.
- Constraints: archived/run evidence 수정 금지, public `run|resume` 금지, caller/provider 직접 호출 금지, optional observation mismatch의 fail-closed 성질 유지.
- Confidence는 high이고 verification context gap은 없다.
### Test Coverage Gaps
- parser test/fixture에 `cache_write_input_tokens`가 없다.
- live integration fake는 실제 Codex가 내보내지 않는 `iop_effective_binding`을 합성한다.
- execution consumer의 absent-observation PASS와 scoring consumer의 absent-observation PASS/non-None mismatch FAIL 조합이 없다.
### Symbol References
- `_CODEX_USAGE_FIELDS`: exact accepted caller usage schema.
- `CodexJSONLParser._turn_observations`: typed metric conversion and unknown-key fail-closed boundary.
- `_LiveAdapter.invoke`: execution-time config admission consumer.
- `_LiveScoringAdapter.invoke`: scoring-time config admission consumer.
- `CodexInvocationResult.effective_binding`: optional caller observation; canonical config binding과 동일한 소유물이 아니다.
### Split Judgment
R1 parser terminal success와 R2 admitted binding 소비는 하나의 real Codex invocation이 lifecycle result로 반환되는 compact boundary다. 분리하면 parser가 성공한 직후 동일 caller-binding 모순으로 다시 중단되므로 한 plan에서 함께 수정한다. predecessor index 05는 기존 archived completion evidence로 이미 충족되며 새 dependency는 없다.
### Scope Rationale
- Codex parser, live execution/scoring consumer, 해당 tests/fixture만 수정한다.
- generic attempt recovery, CLI exception policy, Claude의 실제 tool-use failure, agy/Edge/provider/runtime config는 변경하지 않는다.
- scored result 생성·재개·채점·report와 roadmap 완료 갱신은 제외한다.
### Final Routing
- `evaluation_mode=isolated-reassessment`, finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
- Build closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06, risks=`boundary_contract,structured_interpretation`, `large_indivisible_context=false`, `review_rework_count=1`, `evidence_integrity_failure=false`; route=`local-fit`, lane=`local`, filename=`PLAN-local-G06.md`, catalog=`worker/local/G06`.
- Review closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06; route=`official-review`, lane=`cloud`, filename=`CODE_REVIEW-cloud-G06.md`, catalog=`review/cloud/G06`.
## Dependencies and Execution Order
1. FIX-1이 production usage shape를 terminal success로 parse해야 한다.
2. FIX-2가 그 lifecycle result를 optional caller observation 없이 admitted config binding으로 실행·채점 소비하되 non-None mismatch는 거부해야 한다.
3. focused tests가 통과한 뒤 전체 Python benchmark suite와 manifest validation을 실행한다.
## Implementation Checklist
- [ ] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals.
- [ ] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [FIX-1] Current Codex usage schema
#### Problem
`CodexJSONLParser._turn_observations` validates usage keys as an exact subset of `_CODEX_USAGE_FIELDS`. The production `turn.completed` adds `cache_write_input_tokens`; omission converts an otherwise completed invocation into `parser_error`.
#### Solution
Add only the observed key with canonical metric name `cache_write_tokens`. Update the unit event and tracked stream fixture to include the retained production-shaped five fields, assert every exact reported count, and keep `total_tokens` unavailable when the caller omits it.
#### Modified Files and Checklist
- [ ] `scripts/agent_benchmark/codex_iop.py`: extend exact usage mapping.
- [ ] `scripts/agent_benchmark/codex_iop_test.py`: assert production-shaped cache-write parsing and no derived total.
- [ ] `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`: align deterministic fixture with current caller output.
#### Test Strategy
The parser unit must fail before the mapping and pass after it. Existing unknown/fractional/negative-key/value cases continue proving fail-closed behavior.
#### Verification
Run the focused parser suite in Final Verification step 1.
### [FIX-2] Config-owned effective binding consumption
#### Problem
Preflight admits exact route/model/stage from independently validated config. Real Codex reports no custom binding event, but execution/scoring consumers currently require one, making all real successful Codex invocations impossible.
#### Solution
For execution, allow `result.effective_binding is None`, reject only a present value different from admitted, then retain the lifecycle observations already checked against admitted models/stages. For scoring, apply the same optional mismatch check and return the expected admitted tuple as the canonical scoring result. Remove the synthetic event from the integration fake; assert absent observation succeeds and present mismatch fails in both consumer paths.
#### Modified Files and Checklist
- [ ] `scripts/agent_benchmark/live_iop.py`: align execution and scoring consumers with config-owned admission.
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove synthetic Codex event and cover absent/mismatch execution and scoring behavior.
#### Test Strategy
Use credential-free invoker seams returning real-shaped `CodexInvocationResult`. Verify absent optional observation reaches success using admitted binding, while a contradictory non-None tuple remains fail-closed.
#### Verification
Run the focused integration suite in Final Verification step 1, then the full suite in step 2.
## Modified Files Summary
| Path | Item |
|---|---|
| `scripts/agent_benchmark/codex_iop.py` | FIX-1 |
| `scripts/agent_benchmark/codex_iop_test.py` | FIX-1 |
| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | FIX-1 |
| `scripts/agent_benchmark/live_iop.py` | FIX-2 |
| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 |
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G06.md` | FIX-1, FIX-2 evidence |
## Final Verification
1. Run focused parser and integration coverage:
```bash
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
```
Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection.
2. Run the fresh full benchmark suite:
```bash
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
```
Expected: all benchmark tests pass with no skipped required test or network/provider call.
3. Validate manifest and repository diff:
```bash
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
git diff --check
```
Expected: `ok: manifest is valid`; `git diff --check` has no output. Do not run public `run`, `resume`, `score`, caller, or provider commands in this follow-up.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

1
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log Normal file
View file

@ -0,0 +1 @@
run-20260812T044800Z-412e05fc80df

54
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/user_review_0.log Normal file
View file

@ -0,0 +1,54 @@
# User Review Required - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
## Requested At
2026-08-12
## Status
RESOLVED
## Reason
- Type: external-execution
- Target: `scripts/agent_comparison_benchmark.py run` against `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` using the existing protected `token/.iop-bench` and `token/iop-dev-ca.pem` dev binding
- Current review number: 5
- Final verdict: FAIL
- Summary: R1/R2 source defects are fixed and all deterministic tests pass, but the one scored run previously authorized was retained as `failed=1,running=1`. Benchmark policy requires explicit user authorization before another stateful scored execution.
## Loop History
| Plan | Review | Verdict | Note |
|---|---|---|---|
| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | Initial execution preparation; no official verdict. |
| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | Readiness revision; no official verdict. |
| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | Runtime/provider readiness revision; no official verdict. |
| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, official agy, managed dev and caller readiness completed; first controller run interrupted before scored caller launch. |
| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Replacement run exposed current Codex usage parser and config-binding consumer defects. |
| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | R1/R2 fixed; focused 41 and full 429 tests pass, but a new scored execution remains authorization-gated. |
## Blocking Evidence
- Problem: S04-S08 still lack one C01-C09 run containing exactly nine retained terminal attempts with zero `running|interrupted`.
- Current archived plan: `plan_local_G06_5.log`
- Current archived review: `code_review_cloud_G06_5.log`
- Verification command: `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test && python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' && python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json && git diff --check`
- Actual output: focused `Ran 41 tests in 14.154s — OK`; full `Ran 429 tests in 116.096s — OK`; `ok: manifest is valid`; diff check `(none)`. Existing run status remains `{'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}`.
- Blocking rationale: the declared CLI and protected runtime binding are available, but another scored run repeats stateful/cost-bearing model execution after a retained failure. The benchmark skill forbids implicit retry or replacement execution without explicit user authorization.
## Required User Action
- [x] Explicitly authorize exactly one new public C01-C09 scored `run` after the retained failure. Resolved by the user's instruction that subsequent execution remains continuously authorized; no token value or provider key handoff is needed.
## Resume Condition
- The user explicitly states that one new public C01-C09 scored run is authorized. The next plan must preserve all old run roots, use fresh public preflight `ready=9`, call public `run` once, and never call `resume`, `--retry-failed`, direct callers/providers, or edit run state.
## Next Execution Hint
- Re-enter the `plan` skill for `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; archive this file as `user_review_0.log`, create a fresh routed execution pair, consume `token/.iop-bench` and `token/iop-dev-ca.pem` only inside the protected process environment, and use the deterministic public CLI.
## Closure Rules
- If the recorded user action and evidence resolve this stop as complete/PASS, update `USER_REVIEW.md` to the resolved state, write `complete.log` from `agent-ops/skills/common/code-review/templates/complete-log-template.md`, and move the task directory to the archive.
- If new implementation is required, the `plan` skill archives `USER_REVIEW.md` as `user_review_N.log` before writing a new `PLAN-*-G??.md` / `CODE_REVIEW-*-G??.md` pair.

148
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/code_review_cloud_G07_0.log Normal file
View file

@ -0,0 +1,148 @@
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - TEST
> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run verification, keep active files in place, and report ready for review. Finalization, archive moves, and `complete.log` are review-only.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts, plan=0, tag=TEST
## Archive Evidence Snapshot
- Predecessor terminal run: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`.
- Failed-result report: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md` (`success=2`, `failed=7`, no required workspace files).
## Implementation Item Completion
| Item | Status |
|---|---|
| TEST-1 Claude write tools/parser | [x] |
| TEST-2 Codex sandbox/metrics | [x] |
| TEST-3 agy HOME | [x] |
| TEST-4 guide/full tests | [x] |
## Implementation Checklist
- [x] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage.
- [x] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage.
- [x] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage.
- [x] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests.
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
- [x] Append PASS/WARN/FAIL and routing signals after fresh review verification.
- [x] Verify finding classifications and dimension assessment.
- [x] Archive active plan/review, write `complete.log` on PASS, and move the task directory to the dated archive.
- [x] Preserve milestone-task metadata and keep the active parent because child 08 remains.
## Deviations from Plan
없음. 계획에 명시된 파일과 동작 경계 안에서 구현했다.
## Key Design Decisions
- Claude는 shell/network를 열지 않고 `Read,Write,Edit`만 허용하며 같은 목록만 사전 승인한다.
- Claude partial snapshot은 message id로 중복 제거하고 user tool-result 경계에서 미완료 tool-use turn을 확정한다.
- Codex tool completion count와 선택적 duration metric을 분리하고, 검증 실패 전에는 count 상태를 바꾸지 않는다.
- agy는 prepared session directory를 HOME으로 사용하며 workspace/session/attempt 경로가 모두 존재해야 실행한다.
## Reviewer Checkpoints
- Claude has only Read/Write/Edit and accepts actual tool-use stream lifecycle without accepting contradictory terminals.
- Codex always uses workspace-write and tool counts are not coupled to optional duration.
- agy receives the fresh prepared HOME and no ambient user config.
- No secrets or task contents enter captures.
## Verification Results
### Focused adapter tests
Command: `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test`
```text
.................................
----------------------------------------------------------------------
Ran 33 tests in 6.742s
OK
```
exit status: 0
### Complete benchmark tests
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
```text
.................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 433 tests in 119.205s
OK
```
exit status: 0
### Contract search
Command: `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md`
```text
scripts/agent_benchmark/agy_iop.py:344: "HOME": prepared.session_dir,
scripts/agent_benchmark/claude_iop.py:433: "--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
scripts/agent_benchmark/codex_iop.py:228: *executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
scripts/agent_benchmark/codex_iop.py:243: "HOME": prepared.session_dir,
docs/agent-comparison-benchmark-dev-guide.md:176:| Claude Code | Anthropic-compatible Messages | ... `--tools Read,Write,Edit --allowedTools Read,Write,Edit` ... |
docs/agent-comparison-benchmark-dev-guide.md:177:| agy | Gemini-native `streamGenerateContent` | fresh session `HOME` ... |
docs/agent-comparison-benchmark-dev-guide.md:178:| Codex | OpenAI-compatible Responses | fresh session `HOME` ... `exec --sandbox workspace-write` ... |
```
exit status: 0
## Section Ownership
Implementation item status, checklist, deviations, decisions, and verification output are implementation-owned. Review checklist and verdict are review-only.
## Reviewer Fresh Verification
```text
$ python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test
.................................
----------------------------------------------------------------------
Ran 33 tests in 6.763s
OK
$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
.................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 433 tests in 118.282s
OK
$ git diff --check -- <plan write boundary>
(no output)
```
All commands exited 0.
## Code Review Result
- Verdict: PASS
- Required: 0
- Suggested: 0
- Nit: 0
- Dimension Assessment:
- Correctness: Pass — restricted capabilities and parser state transitions are explicit and fail closed on contradictions.
- Completeness: Pass — all four plan items and documentation are implemented.
- Test coverage: Pass — launch argv, partial/tool-use streams, durationless Codex tools, and missing agy paths have regressions.
- API contract: Pass — existing builder/parser call sites remain compatible; documented CLI contracts match argv.
- Code quality: Pass — no debug output, placeholder, or ambient-secret inheritance added.
- Plan deviation: Pass — no out-of-scope implementation change.
- Verification trust: Pass — reviewer reran 33 focused and 433 complete tests plus diff check.
- Findings: 없음.
- Routing Signals: review_rework_count=0, evidence_integrity_failure=false.
- Next Step: PASS finalization and dependent child 08 execution.

38
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log Normal file
View file

@ -0,0 +1,38 @@
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Complete - m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts
## 완료 일시
2026-08-12
## 요약
Claude/Codex/agy의 격리 workspace 쓰기 계약과 parser/metric 회귀를 1회 루프 PASS로 정리했다.
## 루프 이력
| Plan | Review | Verdict | 메모 |
|------|--------|---------|------|
| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | PASS | 33 focused/433 full tests와 diff check 통과 |
## 구현/정리 내용
- Claude에 `Read,Write,Edit`만 허용하고 partial/tool-use/user-result stream lifecycle을 처리한다.
- Codex에 `workspace-write` sandbox를 지정하고 duration 없는 완료 tool도 정확히 집계한다.
- agy에 prepared session HOME을 제공하고 모든 prepared 경로 존재를 검증한다.
- dev guide의 세 caller 실행 계약과 격리 경계를 현재 구현에 맞췄다.
## 최종 검증
- `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` - PASS; 33 tests.
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 433 tests.
- `git diff --check -- <plan write boundary>` - PASS; no output.
## 잔여 Nit
- 없음
## 후속 작업
- `08+07_comparison_rerun`에서 managed dev runtime과 fresh 9-cell execution을 검증한다.

163
agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/plan_local_G07_0.log Normal file
View file

@ -0,0 +1,163 @@
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Caller workspace-write contract remediation
## For the Implementing Agent
Implement every item, run the specified verification, and fill the implementation-owned sections of `CODE_REVIEW-cloud-G07.md` with actual output. Keep both active files in place and report ready for review. If blocked, record only the exact blocker, attempted commands/output, and resume condition. Do not ask the user, create control-plane stop files, archive logs, or write `complete.log`; finalization is code-review-skill only.
## Background
The first terminal 9-cell smoke run ended with seven failed cells and no generated workspace files. The adapters launched real callers, but Claude had all tools disabled and rejected normal stream snapshots, Codex did not grant workspace-write sandbox access, and agy omitted the isolated HOME required for its runtime configuration.
## Archive Evidence Snapshot
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log` (PASS for terminal matrix execution).
- Canonical failed-result evidence: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md`; success=2, failed=7, all web gates failed because required files were absent.
- The predecessor review accepted terminal-evidence collection, not successful benchmark artifacts. This plan repairs the concrete caller launch/parser contracts exposed by that run.
## Analysis
### Files Read
- `scripts/agent_benchmark/claude_iop.py`
- `scripts/agent_benchmark/claude_iop_test.py`
- `scripts/agent_benchmark/codex_iop.py`
- `scripts/agent_benchmark/codex_iop_test.py`
- `scripts/agent_benchmark/agy_iop.py`
- `scripts/agent_benchmark/agy_iop_test.py`
- `scripts/agent_benchmark/lifecycle.py`
- `scripts/agent_benchmark/lifecycle_test.py`
- `docs/agent-comparison-benchmark-dev-guide.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
- `agent-spec/testing/agent-comparison-benchmark.md`
- `agent-test/local/rules.md`
- `agent-test/local/testing-smoke.md`
- `agent-ops/rules/project/domain/testing/rules.md`
### SDD Criteria
The approved SDD is `READY`; this plan contributes milestone-task ids `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`. S04-S08 and Evidence Map rows require terminal evidence plus generated workspace results for C01-C09. The checklist therefore repairs all three caller write paths and requires deterministic adapter regression tests before any new scored run.
### Verification Context
No separate handoff was supplied. Repository source, archived predecessor `complete.log`, its canonical run report, and official caller CLI documentation were used. The old run is immutable evidence; no direct caller/provider invocation is permitted. Confidence is high for the launch-contract defects and medium for complete external recovery until the dependent live run is executed.
### Test Coverage Gaps
- Claude tests assert the empty `--tools=` launch and only a one-shot `end_turn`; they do not cover partial snapshots or tool-use cycles.
- Codex tests do not require `--sandbox workspace-write` and count a tool only when duration is reported.
- agy tests do not require isolated HOME or prepared-session directory availability.
### Symbol References
No public symbols are renamed or removed. Existing call sites of the modified builders/parsers remain unchanged.
### Split Judgment
This is child 07 of a split remediation. It has a stable contract: every caller receives the minimum isolated filesystem capability and its parser accepts the caller's documented event lifecycle. PASS is the focused/full Python test suite. Predecessor 06 is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Child 08 depends on this PASS and owns runtime recovery/scored execution.
### Scope Rationale
Do not change IOP routing, provider credentials, benchmark task content, scoring, roadmap state, or archived runs. This packet is limited to caller launch/parser contracts, regression tests, and the human dev guide.
### Final Routing
- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair`
- build closures: all true; scores `2/1/1/2/1` => G07; base/route `local-fit`; filename `PLAN-local-G07.md`
- review closures: all true; scores `2/1/1/2/1` => G07; route `official-review`; filename `CODE_REVIEW-cloud-G07.md`
- large_indivisible_context=false; positive risks=`boundary_contract,structured_interpretation,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent.
## Implementation Checklist
- [ ] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage.
- [ ] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage.
- [ ] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage.
- [ ] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [TEST-1] Claude write-tool and stream lifecycle
**Problem:** `scripts/agent_benchmark/claude_iop.py:220-238` rejects assistant snapshots whose `stop_reason` is null/tool-use, and `:351-356` passes `--tools=` while the task requires file creation.
**Solution:** Launch Claude with exact `Read,Write,Edit` tools, pre-approve only those tools, retain `dontAsk`, and make the parser deduplicate message ids while accepting partial snapshots, tool-use completion, user tool results, then one final `end_turn` and successful result.
**Modified Files and Checklist:**
- [ ] `scripts/agent_benchmark/claude_iop.py`: restricted tool argv and multi-message lifecycle.
- [ ] `scripts/agent_benchmark/claude_iop_test.py`: exact argv and partial/tool-use regression cases.
**Test Strategy:** Add normal, duplicate-terminal, session/model mismatch, and final result ordering assertions to the existing credential-free unit tests.
**Verification:** `python3 -m unittest scripts.agent_benchmark.claude_iop_test` exits 0.
### [TEST-2] Codex workspace sandbox and tool metrics
**Problem:** `scripts/agent_benchmark/codex_iop.py:221-224` starts ephemeral Codex without a writable sandbox. `:330-351` discards real completed command items when `duration_ms` is absent, reporting zero tool calls.
**Solution:** Add `--sandbox workspace-write` to the isolated exec argv. Count a completed tool item once by safe id/type regardless of duration; publish a duration metric only when the caller explicitly reports one.
**Modified Files and Checklist:**
- [ ] `scripts/agent_benchmark/codex_iop.py`: sandbox argv and independent completed-tool count.
- [ ] `scripts/agent_benchmark/codex_iop_test.py`: argv and durationless command regressions.
**Test Strategy:** Extend the existing fixture/unit coverage for exact sandbox arguments, durationless tool completion, uniqueness, and explicit-duration preservation.
**Verification:** `python3 -m unittest scripts.agent_benchmark.codex_iop_test` exits 0.
### [TEST-3] agy isolated HOME
**Problem:** `scripts/agent_benchmark/agy_iop.py` constructs a minimal environment without HOME even though agy stores runtime configuration beneath the user's home, causing startup to exit before JSONL output.
**Solution:** Require both prepared workspace and session directories and pass `HOME=prepared.session_dir` without treating it as a secret-bearing allowlist key.
**Modified Files and Checklist:**
- [ ] `scripts/agent_benchmark/agy_iop.py`: prepared-path validation and HOME environment.
- [ ] `scripts/agent_benchmark/agy_iop_test.py`: create session fixture and assert HOME/path rejection.
**Test Strategy:** Extend the builder tests; no network or live caller execution.
**Verification:** `python3 -m unittest scripts.agent_benchmark.agy_iop_test` exits 0.
### [TEST-4] Contract guide and full local tests
**Problem:** `docs/agent-comparison-benchmark-dev-guide.md` still documents the broken launch contracts.
**Solution:** Document the minimum write-capability arguments, isolated HOME, and failure classification so future setup does not reintroduce the smoke-only state.
**Modified Files and Checklist:**
- [ ] `docs/agent-comparison-benchmark-dev-guide.md`: align caller launch and troubleshooting guidance.
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md`: record actual implementation evidence.
**Test Strategy:** Run the complete benchmark module suite fresh; documentation is checked by exact searches in review.
**Verification:** `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` exits 0.
## Dependencies and Execution Order
Predecessor `06+05_comparison_runs` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Complete TEST-1 through TEST-4 before child `08+07_comparison_rerun` starts.
## Modified Files Summary
| File | Item |
|---|---|
| `scripts/agent_benchmark/claude_iop.py` | TEST-1 |
| `scripts/agent_benchmark/claude_iop_test.py` | TEST-1 |
| `scripts/agent_benchmark/codex_iop.py` | TEST-2 |
| `scripts/agent_benchmark/codex_iop_test.py` | TEST-2 |
| `scripts/agent_benchmark/agy_iop.py` | TEST-3 |
| `scripts/agent_benchmark/agy_iop_test.py` | TEST-3 |
| `docs/agent-comparison-benchmark-dev-guide.md` | TEST-4 |
| `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md` | TEST-4 |
## Final Verification
1. `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` — exits 0.
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` — exits 0 with fresh execution; cached output is not accepted.
3. `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md` — shows only the intended restricted/write-enabled contracts.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

73
agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md Normal file
View file

@ -0,0 +1,73 @@
<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Code Review Reference - TEST
> **[IMPLEMENTING AGENT — READ FIRST]** Start only after child 07 PASS. Fill implementation evidence, leave active files in place, and report ready for review. Finalization is review-only.
## Overview
date=2026-08-12
task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=0, tag=TEST
## Archive Evidence Snapshot
- Preserve `run-20260812T044800Z-412e05fc80df` and `run-20260812T050756Z-18293db0b83f`; do not reuse them.
- Dependency: archived PASS `07+06_caller_write_contracts/complete.log`.
## Implementation Item Completion
| Item | Status |
|---|---|
| TEST-1 runtime recovery | [ ] |
| TEST-2 ready=9 preflight | [ ] |
| TEST-3 one scored run | [ ] |
## Implementation Checklist
- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source.
- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9.
- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
## Review-Only Checklist
- [ ] Append verdict/routing signals after fresh verification.
- [ ] Verify no direct caller/provider invocation or scored retry occurred.
- [ ] Archive active files and write/move `complete.log` only on PASS.
- [ ] Preserve milestone-task metadata.
## Deviations from Plan
_Record actual deviations or state none._
## Key Design Decisions
_Record actual decisions._
## Reviewer Checkpoints
- Runtime recovery used the existing managed deployment and did not expose secrets.
- Preflight is ready=9 before run.
- Exactly one new scored run id exists, with 9 success and all web gates passing.
## Verification Results
### Dependency and runtime recovery
_Paste exact commands/output with secret values omitted by command construction._
### Public preflight
Command: `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
_Paste actual stdout/stderr and exit status._
### Public scored run
Command: `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
_Paste actual stdout/stderr and exit status._
## Section Ownership
Implementation status/checklist/evidence is implementation-owned. Review checklist and verdict are review-only.

137
agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/PLAN-cloud-G09.md Normal file
View file

@ -0,0 +1,137 @@
<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
# Dev runtime recovery and scored comparison rerun
## For the Implementing Agent
Start only after child 07 has archived PASS evidence. Restore the existing managed dev runtime, use only the public benchmark CLI, store emitted ids in the exact pointer files, and fill `CODE_REVIEW-cloud-G10.md`. Do not invoke callers/providers directly, reuse a scored run id, overwrite evidence, ask the user, archive logs, or write `complete.log`.
## Background
The adapter fixes are not proven until the managed dev endpoint is available and a fresh preflight reports all nine cells ready. A new append-only scored run must then prove every caller can create the required website artifacts.
## Archive Evidence Snapshot
- Old terminal run remains at `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df` and must not be reused or overwritten.
- A later read-only preflight created `run-20260812T050756Z-18293db0b83f` with ready=0 because the current shell had no runtime endpoint variables and no local managed listener.
- Child 07 must supply the corrected caller contracts before this plan begins.
## Analysis
### Files Read
- `scripts/agent_comparison_benchmark.py`
- `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
- `docs/agent-comparison-benchmark-dev-guide.md`
- `agent-test/local/rules.md`
- `agent-test/local/testing-smoke.md`
- `agent-ops/rules/project/domain/testing/rules.md`
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
### SDD Criteria
SDD is `READY`; milestone-task ids are all five comparison tasks. S04-S08 and Evidence Map rows require C01-C09 event/timing/usage/workspace evidence, so PASS requires ready=9 followed by exactly one fresh scored run whose nine cells pass terminal and web gates.
### Verification Context
Declared runner is this local checkout; managed dev runtime inventory resolves to SSH `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, branch `dev`, CP status port 18001 and edge ports 18082-18084/19093. Current local preflight is blocked by unavailable endpoint environment. Before mutation, inspect remote branch/HEAD/dirty state, process/service ownership, config, listeners, and health without exposing secret values. Restore the existing deployed runtime/config rather than changing IOP source. The public CLI is the only allowed benchmark/provider execution surface.
### External Verification Preflight
- Runner/workdir: `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, dirty feature worktree; caller binaries are installed for Linux aarch64.
- Remote: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`; exact current HEAD/dirty/service state must be captured before restart.
- Credentials/CA: protected files under `token/`; never print values. Export only in the benchmark command process according to the dev guide.
- Resume gate: remote health succeeds and public CLI preflight emits `ready=9`, `registration_required=0`, `implementation_gap=0`.
### Test Coverage Gaps
Local tests cannot prove external endpoint health, caller installation, TLS, or generated files. This plan closes those gaps with preflight and one fresh scored run.
### Symbol References
None; this packet changes no production symbols.
### Split Judgment
This is dependent child 08. Its stable contract is external full-cycle evidence and its PASS oracle is a fresh all-success run. Directory dependency `+07` requires archived `07+06_caller_write_contracts/complete.log`; implementation must not start while it is missing.
### Scope Rationale
Do not deploy new IOP source, alter routes/models/task content, invoke direct callers/providers, retry a scored run, score subjective quality, or modify roadmap state. If the new run exposes a new code defect, preserve evidence and return it to review for a concrete follow-up.
### Final Routing
- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair`
- build closures all true; scores `1/2/2/2/2` => G09; base/route `grade-boundary`; filename `PLAN-cloud-G09.md`
- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; filename `CODE_REVIEW-cloud-G10.md`
- large_indivisible_context=false; positive risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent.
## Implementation Checklist
- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source.
- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9.
- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results.
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
### [TEST-1] Runtime recovery
**Problem:** The current public preflight reports ready=0 because the declared managed dev endpoint/config is unavailable to the runner.
**Solution:** Verify the child-07 completion log, inspect remote git/runtime state read-only, then restart/rebuild only the existing deployed managed services needed to restore health. Bind local command environment from protected `token/` files without logging values.
**Modified Files and Checklist:**
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record remote preflight/recovery evidence.
**Test Strategy:** Use remote health/listener and public CLI preflight only; no direct provider call.
**Verification:** Remote health succeeds and the public preflight gate can execute.
### [TEST-2] Fresh ready=9 preflight
**Problem:** A scored run is forbidden until every matrix cell has proven connectivity/effective binding.
**Solution:** Execute the manifest preflight once runtime inputs are present and store only the emitted id in `preflight_id.log`.
**Modified Files and Checklist:**
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact emitted preflight id.
**Test Strategy:** Assert summary `ready=9 registration_required=0 implementation_gap=0`.
**Verification:** `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0.
### [TEST-3] One scored 9-cell rerun
**Problem:** The old run proves process termination but not successful benchmark artifacts.
**Solution:** Execute exactly one new public `run`, store its id, and inspect immutable result/workspace validation evidence. Do not retry that id.
**Modified Files and Checklist:**
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact emitted scored run id.
**Test Strategy:** Require 9 success, 0 failed/interrupted/running and all required HTML/CSS/JS workspace gates PASS.
**Verification:** `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0 and emitted run evidence satisfies the stated counts.
## Dependencies and Execution Order
`07+06_caller_write_contracts` must first produce an archived `complete.log`. Then perform TEST-1, TEST-2, and TEST-3 in order. No dependency beyond index 07 is implied.
## Modified Files Summary
| File | Item |
|---|---|
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 |
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | TEST-2 |
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | TEST-3 |
## Final Verification
1. `test -f agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log` — exits 0.
2. Public preflight command above — exits 0 with ready=9.
3. Public run command above — executes exactly once and exits 0 with nine passing terminal/web results.
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.

10
apps/edge/internal/openai/chat_decode.go
View file

@ -16,7 +16,7 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest)
}
for key := range raw {
switch key {
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning":
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body":
default:
return fmt.Errorf("%s is not supported for /v1/chat/completions", key)
}
@ -40,6 +40,9 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest)
if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) {
return fmt.Errorf("top_p must be between 0 and 1")
}
if req.TopK != nil && *req.TopK <= 0 {
return fmt.Errorf("top_k must be greater than zero")
}
if err := validateThinkControl(req); err != nil {
return err
}
@ -83,7 +86,7 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe
}
for key := range raw {
switch key {
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning":
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body":
default:
// Unknown fields are tolerated for provider-pool passthrough.
}
@ -108,6 +111,9 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe
if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) {
return fmt.Errorf("top_p must be between 0 and 1")
}
if req.TopK != nil && *req.TopK <= 0 {
return fmt.Errorf("top_k must be greater than zero")
}
if err := validateThinkControl(req); err != nil {
return err
}

14
apps/edge/internal/openai/chat_handler.go
View file

@ -102,6 +102,20 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "invalid_request_error", "messages are required")
return
}
if dispatch.SingleRequest != nil {
capability, ok := s.service.(singleRequestService)
if !ok {
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
return
}
recordSingleRequestIngress()
if req.Stream {
s.handleChatSingleRequestStream(w, r, capability, dispatch, rawBody)
} else {
s.handleChatSingleRequest(w, r, capability, dispatch, rawBody)
}
return
}
outputPolicy := s.resolveOutputPolicy(basePrompt)
// The mutable model-catalog generation policy is applied here, at the single
// ingress point, before req is frozen into the dispatch context. Every stage

3
apps/edge/internal/openai/chat_types.go
View file

@ -14,6 +14,7 @@ type chatCompletionRequest struct {
MaxCompletionTokens *int `json:"max_completion_tokens,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
TopK *int `json:"top_k,omitempty"`
PresencePenalty *float64 `json:"presence_penalty,omitempty"`
FrequencyPenalty *float64 `json:"frequency_penalty,omitempty"`
Seed *int `json:"seed,omitempty"`
@ -28,6 +29,7 @@ type chatCompletionRequest struct {
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
ThinkingTokenBudget *int `json:"thinking_token_budget,omitempty"`
IncludeReasoning *bool `json:"include_reasoning,omitempty"`
ExtraBody any `json:"extra_body,omitempty"`
}
type chatMessage struct {
@ -76,6 +78,7 @@ func (req chatCompletionRequest) runInput(prompt string, messages []chatMessage,
}
setOptionFloat(options, "temperature", req.Temperature)
setOptionFloat(options, "top_p", req.TopP)
setOptionInt(options, "top_k", req.TopK)
setOptionFloat(options, "presence_penalty", req.PresencePenalty)
setOptionFloat(options, "frequency_penalty", req.FrequencyPenalty)
setOptionInt(options, "seed", req.Seed)

310
apps/edge/internal/openai/gemini_bridge.go Normal file
View file

@ -0,0 +1,310 @@
package openai
import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"sort"
"strings"
)
type geminiBridgeResponseWriter struct {
target http.ResponseWriter
header http.Header
status int
committed bool
buffer bytes.Buffer
stream *geminiBridgeStream
}
func newGeminiBridgeResponseWriter(target http.ResponseWriter, model string) *geminiBridgeResponseWriter {
return &geminiBridgeResponseWriter{
target: target, header: make(http.Header), stream: newGeminiBridgeStream(target, model),
}
}
func (w *geminiBridgeResponseWriter) Header() http.Header { return w.header }
func (w *geminiBridgeResponseWriter) WriteHeader(status int) {
if w.status == 0 {
w.status = status
}
}
func (w *geminiBridgeResponseWriter) Write(payload []byte) (int, error) {
if w.status == 0 {
w.status = http.StatusOK
}
if w.status >= http.StatusBadRequest {
return w.buffer.Write(payload)
}
w.commit()
if err := w.stream.Feed(payload); err != nil {
_ = w.stream.Error("upstream stream could not be translated")
return 0, err
}
return len(payload), nil
}
func (w *geminiBridgeResponseWriter) Flush() {
if w.status == 0 {
w.status = http.StatusOK
}
if w.status < http.StatusBadRequest {
w.commit()
}
if flusher, ok := w.target.(http.Flusher); ok {
flusher.Flush()
}
}
func (w *geminiBridgeResponseWriter) commit() {
if w.committed {
return
}
w.target.Header().Set("Content-Type", "text/event-stream")
w.target.Header().Set("Cache-Control", "no-cache")
w.target.Header().Del("Content-Length")
w.target.WriteHeader(http.StatusOK)
w.committed = true
}
func (w *geminiBridgeResponseWriter) Finish() {
if w.status == 0 {
writeGeminiError(w.target, http.StatusBadGateway, "UNAVAILABLE", "runtime request failed")
return
}
if w.status >= http.StatusBadRequest {
status, code := geminiHTTPError(w.status)
writeGeminiError(w.target, status, code, geminiSafeErrorMessage(status))
return
}
w.commit()
if err := w.stream.Finish(); err != nil {
_ = w.stream.Error("upstream stream could not be translated")
}
if flusher, ok := w.target.(http.Flusher); ok {
flusher.Flush()
}
}
func geminiHTTPError(status int) (int, string) {
switch {
case status == http.StatusUnauthorized || status == http.StatusForbidden:
return http.StatusUnauthorized, "UNAUTHENTICATED"
case status >= 400 && status < 500:
return http.StatusBadRequest, "INVALID_ARGUMENT"
default:
return http.StatusBadGateway, "UNAVAILABLE"
}
}
func geminiSafeErrorMessage(status int) string {
if status == http.StatusUnauthorized {
return "authentication failed"
}
if status == http.StatusBadRequest {
return "request is invalid"
}
return "runtime request failed"
}
type geminiBridgeToolState struct {
name string
thoughtSignature string
arguments strings.Builder
}
type geminiBridgeStream struct {
w http.ResponseWriter
model string
pendingSSE []byte
tools map[int]*geminiBridgeToolState
finish string
usage map[string]int
stopped bool
errored bool
}
func newGeminiBridgeStream(w http.ResponseWriter, model string) *geminiBridgeStream {
return &geminiBridgeStream{w: w, model: model, tools: make(map[int]*geminiBridgeToolState), usage: make(map[string]int)}
}
func (s *geminiBridgeStream) Feed(chunk []byte) error {
if s.stopped {
return nil
}
s.pendingSSE = append(s.pendingSSE, chunk...)
s.pendingSSE = bytes.ReplaceAll(s.pendingSSE, []byte("\r\n"), []byte("\n"))
for {
index := bytes.Index(s.pendingSSE, []byte("\n\n"))
if index < 0 {
return nil
}
event := append([]byte(nil), s.pendingSSE[:index]...)
s.pendingSSE = s.pendingSSE[index+2:]
if err := s.consumeSSEEvent(event); err != nil {
return err
}
if s.stopped {
s.pendingSSE = nil
return nil
}
}
}
func (s *geminiBridgeStream) consumeSSEEvent(event []byte) error {
var lines [][]byte
for _, line := range bytes.Split(event, []byte("\n")) {
line = bytes.TrimSpace(line)
if bytes.HasPrefix(line, []byte("data:")) {
lines = append(lines, bytes.TrimSpace(bytes.TrimPrefix(line, []byte("data:"))))
}
}
if len(lines) == 0 {
return nil
}
payload := bytes.Join(lines, []byte("\n"))
if bytes.Equal(payload, []byte("[DONE]")) {
return s.Finish()
}
var chunk geminiChatStreamChunk
if err := json.Unmarshal(payload, &chunk); err != nil {
return fmt.Errorf("decode Chat SSE: %w", err)
}
if chunk.Error != nil {
return s.Error("upstream request failed")
}
if chunk.Usage != nil {
setGeminiUsage(s.usage, "promptTokenCount", chunk.Usage.PromptTokens)
setGeminiUsage(s.usage, "candidatesTokenCount", chunk.Usage.CompletionTokens)
setGeminiUsage(s.usage, "totalTokenCount", chunk.Usage.TotalTokens)
if chunk.Usage.PromptTokensDetails != nil {
setGeminiUsage(s.usage, "cachedContentTokenCount", chunk.Usage.PromptTokensDetails.CachedTokens)
}
if chunk.Usage.CompletionTokensDetails != nil {
setGeminiUsage(s.usage, "thoughtsTokenCount", chunk.Usage.CompletionTokensDetails.ReasoningTokens)
}
}
for _, choice := range chunk.Choices {
reasoning := choice.Delta.ReasoningContent
if reasoning == "" {
reasoning = choice.Delta.Reasoning
}
if reasoning != "" {
if err := s.emitParts([]any{map[string]any{"text": reasoning, "thought": true}}, ""); err != nil {
return err
}
}
if choice.Delta.Content != "" {
if err := s.emitParts([]any{map[string]any{"text": choice.Delta.Content}}, ""); err != nil {
return err
}
}
for _, delta := range choice.Delta.ToolCalls {
state := s.tools[delta.Index]
if state == nil {
state = &geminiBridgeToolState{}
s.tools[delta.Index] = state
}
if delta.Function.Name != "" {
state.name = delta.Function.Name
}
if delta.ExtraContent.Google != nil && delta.ExtraContent.Google.ThoughtSignature != "" {
state.thoughtSignature = delta.ExtraContent.Google.ThoughtSignature
}
if state.arguments.Len()+len(delta.Function.Arguments) > geminiToolArgumentLimit {
return fmt.Errorf("tool arguments exceed limit")
}
state.arguments.WriteString(delta.Function.Arguments)
}
if choice.FinishReason != nil {
s.finish = *choice.FinishReason
}
}
return nil
}
func setGeminiUsage(target map[string]int, key string, value *int) {
if value != nil {
target[key] = *value
}
}
func (s *geminiBridgeStream) emitParts(parts []any, finish string) error {
candidate := map[string]any{"content": map[string]any{"role": "model", "parts": parts}}
if finish != "" {
candidate["finishReason"] = finish
}
payload := map[string]any{"candidates": []any{candidate}, "modelVersion": s.model}
if len(s.usage) > 0 {
payload["usageMetadata"] = s.usage
}
return writeGeminiSSE(s.w, payload)
}
func (s *geminiBridgeStream) emitTools() error {
if len(s.tools) == 0 {
return nil
}
indices := make([]int, 0, len(s.tools))
for index := range s.tools {
indices = append(indices, index)
}
sort.Ints(indices)
parts := make([]any, 0, len(indices))
for _, index := range indices {
state := s.tools[index]
if !geminiPathToken.MatchString(state.name) {
return fmt.Errorf("tool name is invalid")
}
var args map[string]any
if json.Unmarshal([]byte(state.arguments.String()), &args) != nil {
return fmt.Errorf("tool arguments are invalid")
}
part := map[string]any{"functionCall": map[string]any{"name": state.name, "args": args}}
if state.thoughtSignature != "" {
part["thoughtSignature"] = state.thoughtSignature
}
parts = append(parts, part)
}
return s.emitParts(parts, "")
}
func (s *geminiBridgeStream) Finish() error {
if s.stopped {
return nil
}
if len(bytes.TrimSpace(s.pendingSSE)) > 0 {
return fmt.Errorf("truncated Chat SSE")
}
if err := s.emitTools(); err != nil {
return err
}
finish := "STOP"
if s.finish == "length" {
finish = "MAX_TOKENS"
}
s.stopped = true
return s.emitParts([]any{}, finish)
}
func (s *geminiBridgeStream) Error(message string) error {
if s.errored || s.stopped {
return nil
}
s.errored, s.stopped = true, true
return writeGeminiSSE(s.w, geminiErrorResponse{Error: geminiErrorBody{
Code: http.StatusBadGateway, Message: message, Status: "UNAVAILABLE",
}})
}
func writeGeminiSSE(w http.ResponseWriter, value any) error {
payload, err := json.Marshal(value)
if err != nil {
return err
}
_, err = fmt.Fprintf(w, "data: %s\n\n", payload)
return err
}

375
apps/edge/internal/openai/gemini_handler.go Normal file
View file

@ -0,0 +1,375 @@
package openai
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"regexp"
"strings"
)
var geminiPathToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
func isGeminiRequest(r *http.Request) bool {
return r != nil && strings.HasPrefix(r.URL.Path, geminiPathPrefix)
}
func writeGeminiError(w http.ResponseWriter, status int, code, message string) {
writeJSON(w, status, geminiErrorResponse{Error: geminiErrorBody{
Code: status, Message: message, Status: code,
}})
}
func (s *Server) handleGeminiStreamGenerateContent(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
writeGeminiError(w, http.StatusMethodNotAllowed, "INVALID_ARGUMENT", "method not allowed")
return
}
routeID, callerModel, err := parseGeminiStreamPath(r)
if err != nil {
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request path is invalid")
return
}
defer r.Body.Close()
body, err := readOpenAIIngressBody(w, r, s.maxIngressSnapshotBytes())
if err != nil {
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid")
return
}
chatBody, err := prepareGeminiChatBridge(body, routeID)
if err != nil {
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid")
return
}
internal := r.Clone(r.Context())
internal.URL.Path = "/v1/chat/completions"
internal.URL.RawPath = ""
internal.URL.RawQuery = ""
internal.RequestURI = "/v1/chat/completions"
internal.Body = io.NopCloser(bytes.NewReader(chatBody))
internal.ContentLength = int64(len(chatBody))
internal.Header = r.Header.Clone()
internal.Header.Del("Authorization")
internal.Header.Del("X-Goog-Api-Key")
internal.Header.Set("Content-Type", "application/json")
bridge := newGeminiBridgeResponseWriter(w, callerModel)
s.handleChatCompletions(bridge, internal)
bridge.Finish()
}
func parseGeminiStreamPath(r *http.Request) (string, string, error) {
if r == nil || r.URL == nil {
return "", "", fmt.Errorf("missing URL")
}
parts := strings.Split(strings.TrimPrefix(r.URL.Path, geminiPathPrefix), "/")
if len(parts) != 4 || parts[1] != "v1beta" || parts[2] != "models" {
return "", "", fmt.Errorf("unexpected path")
}
const suffix = ":streamGenerateContent"
if !strings.HasSuffix(parts[3], suffix) {
return "", "", fmt.Errorf("unexpected method")
}
routeID := parts[0]
callerModel := strings.TrimSuffix(parts[3], suffix)
if !geminiPathToken.MatchString(routeID) || !geminiPathToken.MatchString(callerModel) {
return "", "", fmt.Errorf("invalid path token")
}
query := r.URL.Query()
if len(query) != 1 || len(query["alt"]) != 1 || query.Get("alt") != "sse" {
return "", "", fmt.Errorf("alt=sse is required")
}
return routeID, callerModel, nil
}
func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) {
if err := validateJSONMembers(body); err != nil {
return nil, err
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.DisallowUnknownFields()
var req geminiRequest
if err := decoder.Decode(&req); err != nil {
return nil, err
}
if err := requireGeminiJSONEOF(decoder); err != nil {
return nil, err
}
if len(req.Contents) == 0 {
return nil, fmt.Errorf("contents are required")
}
chat := map[string]any{
"model": routeID, "stream": true,
"stream_options": map[string]any{"include_usage": true},
}
messages := make([]map[string]any, 0, len(req.Contents)+1)
if req.SystemInstruction != nil {
text, err := geminiTextOnly(*req.SystemInstruction)
if err != nil || strings.TrimSpace(text) == "" {
return nil, fmt.Errorf("systemInstruction is invalid")
}
messages = append(messages, map[string]any{"role": "system", "content": text})
}
pendingCalls := make(map[string][]string)
for contentIndex, content := range req.Contents {
converted, err := geminiContentToChat(content, contentIndex, pendingCalls)
if err != nil {
return nil, err
}
messages = append(messages, converted...)
}
chat["messages"] = messages
if config := req.GenerationConfig; config != nil {
if config.CandidateCount != nil && *config.CandidateCount != 1 {
return nil, fmt.Errorf("candidateCount must be one")
}
if config.MaxOutputTokens != nil {
if *config.MaxOutputTokens <= 0 {
return nil, fmt.Errorf("maxOutputTokens must be positive")
}
chat["max_tokens"] = *config.MaxOutputTokens
}
if len(config.StopSequences) > 0 {
chat["stop"] = config.StopSequences
}
if config.Temperature != nil {
if *config.Temperature < 0 || *config.Temperature > 2 {
return nil, fmt.Errorf("temperature is invalid")
}
}
if config.TopK != nil {
if *config.TopK <= 0 {
return nil, fmt.Errorf("topK is invalid")
}
}
if config.TopP != nil {
if *config.TopP < 0 || *config.TopP > 1 {
return nil, fmt.Errorf("topP is invalid")
}
}
if thinking := config.ThinkingConfig; thinking != nil {
googleThinking := make(map[string]any)
if thinking.IncludeThoughts != nil {
googleThinking["include_thoughts"] = *thinking.IncludeThoughts
}
if thinking.ThinkingBudget != nil {
if *thinking.ThinkingBudget < -1 {
return nil, fmt.Errorf("thinkingBudget is invalid")
}
googleThinking["thinking_budget"] = *thinking.ThinkingBudget
}
if len(googleThinking) > 0 {
// Gemini's OpenAI-compatible endpoint accepts native options only
// below extra_body.google; generic think/include_reasoning fields
// are rejected by that endpoint.
chat["extra_body"] = map[string]any{
"google": map[string]any{"thinking_config": googleThinking},
}
}
}
}
if len(req.Tools) > 0 {
tools := make([]map[string]any, 0)
for _, group := range req.Tools {
if len(group.FunctionDeclarations) == 0 {
return nil, fmt.Errorf("functionDeclarations are required")
}
for _, declaration := range group.FunctionDeclarations {
if !geminiPathToken.MatchString(declaration.Name) || len(declaration.ParametersJSONSchema) == 0 {
return nil, fmt.Errorf("function declaration is invalid")
}
var schema map[string]any
if json.Unmarshal(declaration.ParametersJSONSchema, &schema) != nil {
return nil, fmt.Errorf("function schema is invalid")
}
function := map[string]any{"name": declaration.Name, "parameters": schema}
if declaration.Description != "" {
function["description"] = declaration.Description
}
tools = append(tools, map[string]any{"type": "function", "function": function})
}
}
chat["tools"] = tools
}
if req.ToolConfig != nil && req.ToolConfig.FunctionCallingConfig != nil {
switch strings.ToUpper(req.ToolConfig.FunctionCallingConfig.Mode) {
case "", "AUTO":
chat["tool_choice"] = "auto"
case "ANY":
chat["tool_choice"] = "required"
case "NONE":
chat["tool_choice"] = "none"
default:
return nil, fmt.Errorf("function calling mode is invalid")
}
}
return json.Marshal(chat)
}
func geminiTextOnly(content geminiContent) (string, error) {
// Gemini represents systemInstruction as Content and official agy 1.1.12
// labels that Content with the API-native "user" role.
if content.Role != "" && content.Role != "user" {
return "", fmt.Errorf("role is invalid")
}
texts := make([]string, 0, len(content.Parts))
for _, part := range content.Parts {
if part.Text == nil || part.FunctionCall != nil || part.FunctionResponse != nil || part.Thought || part.ThoughtSignature != "" {
return "", fmt.Errorf("only text is supported")
}
texts = append(texts, *part.Text)
}
return strings.Join(texts, "\n"), nil
}
func geminiContentToChat(content geminiContent, contentIndex int, pending map[string][]string) ([]map[string]any, error) {
role := strings.ToLower(strings.TrimSpace(content.Role))
if role != "user" && role != "model" {
return nil, fmt.Errorf("content role is invalid")
}
if len(content.Parts) == 0 {
return nil, fmt.Errorf("content parts are required")
}
var texts []string
var reasoning []string
var toolCalls []any
var toolMessages []map[string]any
for partIndex, part := range content.Parts {
set := 0
if part.Text != nil {
set++
}
if part.FunctionCall != nil {
set++
}
if part.FunctionResponse != nil {
set++
}
if set != 1 {
return nil, fmt.Errorf("content part is invalid")
}
if part.Text != nil {
if role == "model" && part.Thought {
reasoning = append(reasoning, *part.Text)
} else if part.Thought {
return nil, fmt.Errorf("user thought is invalid")
} else {
texts = append(texts, *part.Text)
}
continue
}
if part.FunctionCall != nil {
if role != "model" || !geminiPathToken.MatchString(part.FunctionCall.Name) {
return nil, fmt.Errorf("functionCall is invalid")
}
var args map[string]any
if json.Unmarshal(part.FunctionCall.Args, &args) != nil {
return nil, fmt.Errorf("functionCall args are invalid")
}
callID := fmt.Sprintf("gemini_call_%d_%d", contentIndex, partIndex)
pending[part.FunctionCall.Name] = append(pending[part.FunctionCall.Name], callID)
call := map[string]any{"id": callID, "type": "function", "function": map[string]any{"name": part.FunctionCall.Name, "arguments": string(part.FunctionCall.Args)}}
if part.ThoughtSignature != "" {
call["extra_content"] = openAIChatThoughtSignature(part.ThoughtSignature)
}
toolCalls = append(toolCalls, call)
continue
}
response := part.FunctionResponse
if role != "user" || !geminiPathToken.MatchString(response.Name) {
return nil, fmt.Errorf("functionResponse is invalid")
}
ids := pending[response.Name]
if len(ids) == 0 || !json.Valid(response.Response) {
return nil, fmt.Errorf("functionResponse has no matching call")
}
callID := ids[0]
pending[response.Name] = ids[1:]
toolMessages = append(toolMessages, map[string]any{"role": "tool", "tool_call_id": callID, "tool_name": response.Name, "content": string(response.Response)})
}
if role == "model" {
message := map[string]any{"role": "assistant", "content": strings.Join(texts, "\n")}
if len(reasoning) > 0 {
message["reasoning_content"] = strings.Join(reasoning, "")
}
if len(toolCalls) > 0 {
message["tool_calls"] = toolCalls
}
return []map[string]any{message}, nil
}
if len(texts) > 0 {
toolMessages = append(toolMessages, map[string]any{"role": "user", "content": strings.Join(texts, "\n")})
}
if len(toolMessages) == 0 {
return nil, fmt.Errorf("user content is empty")
}
return toolMessages, nil
}
func validateJSONMembers(raw []byte) error {
dec := json.NewDecoder(bytes.NewReader(raw))
if err := validateGeminiJSONValue(dec); err != nil {
return err
}
return requireGeminiJSONEOF(dec)
}
func validateGeminiJSONValue(dec *json.Decoder) error {
token, err := dec.Token()
if err != nil {
return err
}
delim, ok := token.(json.Delim)
if !ok {
return nil
}
switch delim {
case '{':
seen := map[string]struct{}{}
for dec.More() {
keyToken, err := dec.Token()
if err != nil {
return err
}
key, ok := keyToken.(string)
if !ok {
return fmt.Errorf("object key is invalid")
}
if _, duplicate := seen[key]; duplicate {
return fmt.Errorf("duplicate member")
}
seen[key] = struct{}{}
if err := validateGeminiJSONValue(dec); err != nil {
return err
}
}
closeToken, err := dec.Token()
if err != nil || closeToken != json.Delim('}') {
return fmt.Errorf("object is invalid")
}
case '[':
for dec.More() {
if err := validateGeminiJSONValue(dec); err != nil {
return err
}
}
closeToken, err := dec.Token()
if err != nil || closeToken != json.Delim(']') {
return fmt.Errorf("array is invalid")
}
default:
return fmt.Errorf("JSON delimiter is invalid")
}
return nil
}
func requireGeminiJSONEOF(dec *json.Decoder) error {
var extra any
if err := dec.Decode(&extra); err != io.EOF {
return fmt.Errorf("trailing JSON value")
}
return nil
}

173
apps/edge/internal/openai/gemini_handler_test.go Normal file
View file

@ -0,0 +1,173 @@
package openai
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"iop/packages/go/config"
iop "iop/proto/gen/iop"
)
func TestGeminiIngressAuthenticatesAndStreamsThroughChatRoute(t *testing.T) {
fake := &fakeRunService{events: bufferedRunEvents(
&iop.RunEvent{Type: "delta", Delta: "hello"},
&iop.RunEvent{Type: "complete", Usage: &iop.Usage{InputTokens: 3, OutputTokens: 2}, Metadata: map[string]string{"finish_reason": "length"}},
)}
srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama", Target: "provider-model"}, fake, nil)
body := `{
"systemInstruction":{"role":"user","parts":[{"text":"be concise"}]},
"contents":[{"role":"user","parts":[{"text":"say hello"}]}],
"generationConfig":{"candidateCount":1,"maxOutputTokens":16,"temperature":0.2,"topK":8,"topP":0.9,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}}
}`
req := httptest.NewRequest(http.MethodPost, "/gemini/gemini-direct/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", strings.NewReader(body))
req.Header.Set("X-Goog-Api-Key", "iop-principal")
w := httptest.NewRecorder()
srv.routes().ServeHTTP(w, req)
if w.Code != http.StatusOK || w.Header().Get("Content-Type") != "text/event-stream" {
t.Fatalf("status/header: %d %q body=%s", w.Code, w.Header().Get("Content-Type"), w.Body.String())
}
if fake.req.ModelGroupKey != "gemini-direct" || fake.req.Target != "provider-model" {
t.Fatalf("route bypassed Chat admission: %+v", fake.req)
}
if !strings.Contains(fake.req.Prompt, "system: be concise") || !strings.Contains(fake.req.Prompt, "user: say hello") {
t.Fatalf("prompt conversion mismatch: %q", fake.req.Prompt)
}
options := fake.req.Input["options"].(map[string]any)
if options["max_tokens"] != 16 {
t.Fatalf("generation config mismatch: input=%+v", fake.req.Input)
}
if _, exists := options["top_k"]; exists {
t.Fatalf("deprecated Gemini sampling option reached Chat upstream: input=%+v", fake.req.Input)
}
response := w.Body.String()
for _, want := range []string{`"text":"hello"`, `"finishReason":"MAX_TOKENS"`} {
if !strings.Contains(response, want) {
t.Fatalf("missing %s in %s", want, response)
}
}
if strings.Contains(response, "iop-principal") {
t.Fatal("principal token leaked to response")
}
}
func TestGeminiIngressRejectsAuthenticationAndShapeBeforeDispatch(t *testing.T) {
base := `{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}`
for _, tc := range []struct {
name, path, body, bearer, key string
status int
}{
{"missing key", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "", "", 401},
{"conflicting auth", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "Bearer other", "iop-principal", 401},
{"wrong query", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent", base, "", "iop-principal", 400},
{"duplicate member", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[],"contents":[]}`, "", "iop-principal", 400},
{"two candidates", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"candidateCount":2}}`, "", "iop-principal", 400},
{"invalid thinking budget", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"thinkingConfig":{"thinkingBudget":-2}}}`, "", "iop-principal", 400},
} {
t.Run(tc.name, func(t *testing.T) {
fake := &fakeRunService{}
srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama"}, fake, nil)
req := httptest.NewRequest(http.MethodPost, tc.path, strings.NewReader(tc.body))
if tc.bearer != "" {
req.Header.Set("Authorization", tc.bearer)
}
if tc.key != "" {
req.Header.Set("X-Goog-Api-Key", tc.key)
}
w := httptest.NewRecorder()
srv.routes().ServeHTTP(w, req)
if w.Code != tc.status {
t.Fatalf("status=%d body=%s", w.Code, w.Body.String())
}
var failure geminiErrorResponse
if json.Unmarshal(w.Body.Bytes(), &failure) != nil || failure.Error.Code != tc.status {
t.Fatalf("not a Gemini error: %s", w.Body.String())
}
if fake.req.ModelGroupKey != "" {
t.Fatalf("unexpected dispatch: %+v", fake.req)
}
})
}
}
func TestGeminiRequestBridgePreservesToolsAndThoughtSignature(t *testing.T) {
body := []byte(`{
"contents":[
{"role":"model","parts":[{"text":"thinking","thought":true},{"functionCall":{"name":"lookup","args":{"q":"x"}},"thoughtSignature":"opaque"}]},
{"role":"user","parts":[{"functionResponse":{"name":"lookup","response":{"value":1}}}]},
{"role":"user","parts":[{"text":"continue"}]}
],
"tools":[{"functionDeclarations":[{"name":"lookup","description":"find","parametersJsonSchema":{"type":"object"}}]}],
"toolConfig":{"functionCallingConfig":{"mode":"ANY"}}
}`)
converted, err := prepareGeminiChatBridge(body, "preset-gemini-hybrid")
if err != nil {
t.Fatal(err)
}
text := string(converted)
for _, want := range []string{`"model":"preset-gemini-hybrid"`, `"reasoning_content":"thinking"`, `"thought_signature":"opaque"`, `"tool_call_id":"gemini_call_0_1"`, `"tool_choice":"required"`} {
if !strings.Contains(text, want) {
t.Fatalf("missing %s in %s", want, text)
}
}
}
func TestGeminiRequestBridgeUsesProviderNativeThinkingEnvelope(t *testing.T) {
converted, err := prepareGeminiChatBridge([]byte(`{
"contents":[{"role":"user","parts":[{"text":"hello"}]}],
"generationConfig":{"temperature":1,"topK":50,"topP":1,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}}
}`), "gemini-3.6-flash")
if err != nil {
t.Fatal(err)
}
var body map[string]any
if json.Unmarshal(converted, &body) != nil {
t.Fatal("converted body is invalid")
}
for _, forbidden := range []string{"temperature", "top_k", "top_p", "think", "include_reasoning", "thinking_token_budget"} {
if _, exists := body[forbidden]; exists {
t.Fatalf("unsupported field %q in converted body: %s", forbidden, converted)
}
}
extra := body["extra_body"].(map[string]any)
google := extra["google"].(map[string]any)
thinking := google["thinking_config"].(map[string]any)
if thinking["include_thoughts"] != true || thinking["thinking_budget"] != float64(-1) {
t.Fatalf("thinking envelope mismatch: %+v", thinking)
}
}
func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) {
w := httptest.NewRecorder()
stream := newGeminiBridgeStream(w, "gemini-3.6-flash")
input := "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\"}}]}}]}\n\n" +
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"x\\\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\n" +
"data: [DONE]\n\n"
if err := stream.Feed([]byte(input)); err != nil {
t.Fatal(err)
}
if strings.Count(w.Body.String(), `"functionCall"`) != 1 || !strings.Contains(w.Body.String(), `"finishReason":"STOP"`) {
t.Fatalf("tool terminal mismatch: %s", w.Body.String())
}
usageWriter := httptest.NewRecorder()
usageStream := newGeminiBridgeStream(usageWriter, "m")
if err := usageStream.Feed([]byte("data: {\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":2,\"total_tokens\":5,\"prompt_tokens_details\":{\"cached_tokens\":1},\"completion_tokens_details\":{\"reasoning_tokens\":1}}}\n\ndata: [DONE]\n\n")); err != nil {
t.Fatal(err)
}
for _, want := range []string{`"promptTokenCount":3`, `"candidatesTokenCount":2`, `"cachedContentTokenCount":1`, `"thoughtsTokenCount":1`, `"totalTokenCount":5`} {
if !strings.Contains(usageWriter.Body.String(), want) {
t.Fatalf("missing usage %s in %s", want, usageWriter.Body.String())
}
}
oversize := newGeminiBridgeStream(httptest.NewRecorder(), "m")
state := &geminiBridgeToolState{name: "lookup"}
state.arguments.WriteString(strings.Repeat("x", geminiToolArgumentLimit))
oversize.tools[0] = state
chunk := `data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"x"}}]}}]}` + "\n\n"
if err := oversize.Feed([]byte(chunk)); err == nil {
t.Fatal("oversize tool arguments must fail")
}
}

117
apps/edge/internal/openai/gemini_types.go Normal file
View file

@ -0,0 +1,117 @@
package openai
import "encoding/json"
const (
geminiPathPrefix = "/gemini/"
geminiToolArgumentLimit = 1 << 20
)
type geminiRequest struct {
Contents []geminiContent `json:"contents"`
SystemInstruction *geminiContent `json:"systemInstruction,omitempty"`
GenerationConfig *geminiGenerationConfig `json:"generationConfig,omitempty"`
Tools []geminiTool `json:"tools,omitempty"`
ToolConfig *geminiToolConfig `json:"toolConfig,omitempty"`
}
type geminiContent struct {
Role string `json:"role,omitempty"`
Parts []geminiPart `json:"parts"`
}
type geminiPart struct {
Text *string `json:"text,omitempty"`
Thought bool `json:"thought,omitempty"`
ThoughtSignature string `json:"thoughtSignature,omitempty"`
FunctionCall *geminiFunctionCall `json:"functionCall,omitempty"`
FunctionResponse *geminiFunctionResponse `json:"functionResponse,omitempty"`
}
type geminiFunctionCall struct {
Name string `json:"name"`
Args json.RawMessage `json:"args"`
}
type geminiFunctionResponse struct {
Name string `json:"name"`
Response json.RawMessage `json:"response"`
}
type geminiGenerationConfig struct {
CandidateCount *int `json:"candidateCount,omitempty"`
MaxOutputTokens *int `json:"maxOutputTokens,omitempty"`
StopSequences []string `json:"stopSequences,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
TopK *int `json:"topK,omitempty"`
TopP *float64 `json:"topP,omitempty"`
ThinkingConfig *geminiThinkingConfig `json:"thinkingConfig,omitempty"`
}
type geminiThinkingConfig struct {
IncludeThoughts *bool `json:"includeThoughts,omitempty"`
ThinkingBudget *int `json:"thinkingBudget,omitempty"`
}
type geminiTool struct {
FunctionDeclarations []geminiFunctionDeclaration `json:"functionDeclarations"`
}
type geminiFunctionDeclaration struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
ParametersJSONSchema json.RawMessage `json:"parametersJsonSchema"`
}
type geminiToolConfig struct {
FunctionCallingConfig *geminiFunctionCallingConfig `json:"functionCallingConfig,omitempty"`
}
type geminiFunctionCallingConfig struct {
Mode string `json:"mode"`
}
type geminiErrorResponse struct {
Error geminiErrorBody `json:"error"`
}
type geminiErrorBody struct {
Code int `json:"code"`
Message string `json:"message"`
Status string `json:"status"`
}
type geminiChatStreamChunk struct {
Choices []struct {
Delta struct {
Content string `json:"content"`
ReasoningContent string `json:"reasoning_content"`
Reasoning string `json:"reasoning"`
ToolCalls []struct {
Index int `json:"index"`
ID string `json:"id"`
ExtraContent openAIChatToolExtraContent `json:"extra_content,omitempty"`
Function struct {
Name string `json:"name"`
Arguments string `json:"arguments"`
} `json:"function"`
} `json:"tool_calls"`
} `json:"delta"`
FinishReason *string `json:"finish_reason"`
} `json:"choices"`
Usage *struct {
PromptTokens *int `json:"prompt_tokens,omitempty"`
CompletionTokens *int `json:"completion_tokens,omitempty"`
TotalTokens *int `json:"total_tokens,omitempty"`
PromptTokensDetails *struct {
CachedTokens *int `json:"cached_tokens,omitempty"`
} `json:"prompt_tokens_details,omitempty"`
CompletionTokensDetails *struct {
ReasoningTokens *int `json:"reasoning_tokens,omitempty"`
} `json:"completion_tokens_details,omitempty"`
} `json:"usage,omitempty"`
Error *struct {
Type string `json:"type"`
Message string `json:"message"`
} `json:"error,omitempty"`
}

2
apps/edge/internal/openai/principal.go
View file

@ -167,6 +167,8 @@ func principalTokenFromRequest(r *http.Request) (string, bool) {
apiKey := ""
if isAnthropicRequest(r) {
apiKey = strings.TrimSpace(r.Header.Get("X-Api-Key"))
} else if isGeminiRequest(r) {
apiKey = strings.TrimSpace(r.Header.Get("X-Goog-Api-Key"))
}
if authorization != "" && bearer == "" {
return "", false

21
apps/edge/internal/openai/principal_routes.go
View file

@ -54,7 +54,10 @@ func (s *Server) advertisedModelsForPrincipal(ctx context.Context) ([]advertised
}
for _, r := range routes {
id := strings.TrimSpace(r.RouteID)
id := strings.TrimSpace(r.RouteAlias)
if id == "" {
id = strings.TrimSpace(r.RouteID)
}
addModel(id, id)
}
@ -241,8 +244,10 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M
if selector == "" {
return managedCatalogBinding{}, ErrRouteNotFound
}
routeAlias := strings.TrimSpace(route.RouteAlias)
explicit := !strings.EqualFold(selector, "default")
var matches []managedCatalogBinding
var aliasMatches []managedCatalogBinding
for _, entry := range catalog {
group := strings.TrimSpace(entry.ID)
if group == "" {
@ -260,9 +265,23 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M
binding.ProviderID = strings.TrimSpace(providerID)
}
matches = append(matches, binding)
if routeAlias != "" && routeAlias == group {
aliasMatches = append(aliasMatches, binding)
}
break
}
}
// A principal-owned public alias that exactly names a compatible catalog
// group is an explicit disambiguator. This is required when two catalog
// groups intentionally expose the same provider/upstream pair (for example
// a general model and a preset-pinned fast alias). It never bypasses the
// resource selector or upstream-model checks above.
if len(aliasMatches) == 1 {
return aliasMatches[0], nil
}
if len(aliasMatches) > 1 {
return managedCatalogBinding{}, ErrRouteNotFound
}
if len(matches) != 1 {
return managedCatalogBinding{}, ErrRouteNotFound
}

20
apps/edge/internal/openai/principal_routes_test.go
View file

@ -72,7 +72,7 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) {
"token-p1": "principal-1",
"token-p2": "principal-2",
}, map[string]authprojection.Route{
"r1": {RouteID: "route-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
"r1": {RouteID: "route-p1", RouteAlias: "model-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
"r2": {RouteID: "route-p2", PrincipalRef: "principal-2", CredentialSlotRef: "slot-2", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
})
if err := cache.Apply(proj); err != nil {
@ -96,8 +96,8 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) {
if err := json.Unmarshal(wP1.Body.Bytes(), &respP1); err != nil {
t.Fatal(err)
}
if len(respP1.Data) != 1 || respP1.Data[0].ID != "route-p1" {
t.Fatalf("P1 models: expected [route-p1], got %+v", respP1.Data)
if len(respP1.Data) != 1 || respP1.Data[0].ID != "model-p1" {
t.Fatalf("P1 models: expected public alias [model-p1], got %+v", respP1.Data)
}
// Call for P2
@ -123,7 +123,7 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) {
proj := makeTestProjection(1, now, time.Hour, map[string]string{
"token-p1": "principal-1",
}, map[string]authprojection.Route{
"r1": {RouteID: "claude-route-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"},
"r1": {RouteID: "claude-route-1", RouteAlias: "claude-public-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"},
})
if err := cache.Apply(proj); err != nil {
t.Fatal(err)
@ -148,8 +148,8 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) {
if err := json.Unmarshal(w.Body.Bytes(), &anthropicResp); err != nil {
t.Fatal(err)
}
if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-route-1" {
t.Fatalf("expected [claude-route-1], got %+v", anthropicResp.Data)
if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-public-1" {
t.Fatalf("expected public alias [claude-public-1], got %+v", anthropicResp.Data)
}
}
@ -337,9 +337,15 @@ func TestManagedCatalogBindingExplicitSelectorAndNoFallback(t *testing.T) {
t.Fatalf("missing selector err=%v", err)
}
route.ResourceSelector = "default"
if _, err := resolveManagedCatalogBinding(route, append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}})); !errors.Is(err, ErrRouteNotFound) {
ambiguous := append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}})
if _, err := resolveManagedCatalogBinding(route, ambiguous); !errors.Is(err, ErrRouteNotFound) {
t.Fatalf("ambiguous catalog err=%v", err)
}
route.RouteAlias = "another-group"
binding, err = resolveManagedCatalogBinding(route, ambiguous)
if err != nil || binding.ModelGroupKey != "another-group" || binding.ProviderID != "" {
t.Fatalf("alias-disambiguated binding=%+v err=%v", binding, err)
}
}
func TestMetadataSpoofIsOverwrittenByManagedBinding(t *testing.T) {

6
apps/edge/internal/openai/request_identity_ingress.go
View file

@ -386,6 +386,12 @@ func (s *Server) compilePresetArtifactBinding(dispatch routeDispatch, protocol s
preset = found
}
}
// Marked single-request presets use the operator-owned internal workspace
// capability compiled into dispatch.SingleRequest. They intentionally reject
// legacy caller workspace_tools, so artifact binding must not run here.
if preset.SingleRequest != nil || dispatch.SingleRequest != nil {
return nil, false, nil
}
if !isModeAllowed(preset, modeLight) {
return nil, false, nil
}

9
apps/edge/internal/openai/routes.go
View file

@ -14,6 +14,7 @@ func (s *Server) routes() *http.ServeMux {
mux.HandleFunc("/v1/models", s.withAuth(s.handleModels))
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
mux.HandleFunc("/v1/responses", s.withAuth(s.handleResponses))
mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent))
s.registerAnthropicRoutes(mux)
mux.HandleFunc("/api/", s.withAuth(s.handleOllamaAPI))
return mux
@ -54,6 +55,10 @@ func (s *Server) writeCallerProviderCredentialRejection(w http.ResponseWriter, r
writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", message)
return
}
if isGeminiRequest(r) {
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", message)
return
}
writeError(w, http.StatusBadRequest, "invalid_request_error", message)
}
@ -63,6 +68,10 @@ func (s *Server) writeAuthenticationFailure(w http.ResponseWriter, r *http.Reque
writeAnthropicError(w, http.StatusUnauthorized, "authentication_error", "authentication failed")
return
}
if isGeminiRequest(r) {
writeGeminiError(w, http.StatusUnauthorized, "UNAUTHENTICATED", "authentication failed")
return
}
writeError(w, http.StatusUnauthorized, "unauthorized", "unauthorized")
}

366
apps/edge/internal/openai/single_request_chat.go Normal file
View file

@ -0,0 +1,366 @@
package openai
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"sync"
"time"
edgeservice "iop/apps/edge/internal/service"
)
type singleRequestChatPolicy struct {
status int
finishReason string
errorType string
message string
silent bool
error bool
}
func singleRequestChatTerminalPolicy(disposition edgeservice.SingleRequestTerminalDisposition) singleRequestChatPolicy {
if disposition.Kind == "" {
disposition.Kind = edgeservice.SingleRequestTerminalEndTurn
}
if disposition.Validate() != nil {
disposition = edgeservice.SingleRequestTerminalDisposition{
Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider,
}
}
switch disposition.Kind {
case edgeservice.SingleRequestTerminalEndTurn:
return singleRequestChatPolicy{status: http.StatusOK, finishReason: "stop"}
case edgeservice.SingleRequestTerminalLength:
return singleRequestChatPolicy{status: http.StatusOK, finishReason: "length"}
case edgeservice.SingleRequestTerminalCancelled:
return singleRequestChatPolicy{silent: true}
case edgeservice.SingleRequestTerminalError:
switch disposition.ErrorClass {
case edgeservice.SingleRequestTerminalErrorValidation, edgeservice.SingleRequestTerminalErrorContext:
return singleRequestChatPolicy{status: http.StatusBadRequest, errorType: "invalid_request_error", message: "single-request execution was rejected", error: true}
case edgeservice.SingleRequestTerminalErrorTimeout:
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution timed out", error: true}
default:
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true}
}
default:
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true}
}
}
func (s *Server) handleChatSingleRequestStream(
w http.ResponseWriter,
r *http.Request,
capability singleRequestService,
dispatch routeDispatch,
body []byte,
) {
requestID, err := newLogicalRequestRandomID()
if err != nil {
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
return
}
requestID = "req_" + requestID
stream, err := newSingleRequestChatStream(w, requestID, dispatch.SingleRequest.PublicModel)
if err != nil {
writeError(w, http.StatusInternalServerError, "run_error", "single-request streaming is unavailable")
return
}
execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{
RequestID: requestID,
Binding: dispatch.SingleRequest.Clone(),
Prompt: string(append([]byte(nil), body...)),
})
if err != nil || execution == nil {
if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) {
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
return
}
writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started")
return
}
defer execution.Cancel()
_ = pumpSingleRequestChatStream(r.Context(), execution, stream, newWallClockSingleRequestAnthropicTicker)
}
func (s *Server) handleChatSingleRequest(
w http.ResponseWriter,
r *http.Request,
capability singleRequestService,
dispatch routeDispatch,
body []byte,
) {
requestID, err := newLogicalRequestRandomID()
if err != nil {
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
return
}
requestID = "req_" + requestID
execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{
RequestID: requestID,
Binding: dispatch.SingleRequest.Clone(),
Prompt: string(append([]byte(nil), body...)),
})
if err != nil || execution == nil {
if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) {
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
return
}
writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started")
return
}
defer execution.Cancel()
for {
select {
case <-r.Context().Done():
execution.Cancel()
return
case progress, ok := <-execution.Progress():
if !ok {
if r.Context().Err() == nil && execution.State() != edgeservice.SingleRequestStateCancelled {
writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed")
}
return
}
switch progress.Stage {
case edgeservice.SingleRequestStateFinalizing:
if progress.Result == nil {
_ = execution.AcknowledgeTerminal(false)
writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed")
return
}
writeErr := writeChatSingleRequestTerminal(w, requestID, dispatch.SingleRequest.PublicModel, *progress.Result)
_ = execution.AcknowledgeTerminal(writeErr == nil)
return
case edgeservice.SingleRequestStateFailed:
policy := singleRequestChatTerminalPolicy(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}))
writeError(w, policy.status, policy.errorType, policy.message)
return
case edgeservice.SingleRequestStateCancelled:
return
}
}
}
}
func writeChatSingleRequestTerminal(w http.ResponseWriter, requestID, model string, result edgeservice.SingleRequestResult) error {
policy := singleRequestChatTerminalPolicy(result.Terminal)
if policy.silent || policy.error || policy.finishReason == "" {
return errors.New("single-request result has no Chat terminal")
}
content := result.Output
if policy.finishReason == "length" {
content = ""
}
response := chatCompletionResponse{
ID: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), Object: "chat.completion",
Created: time.Now().Unix(), Model: model,
Choices: []chatCompletionChoice{{Index: 0, Message: chatMessage{Role: "assistant", Content: content}, FinishReason: policy.finishReason}},
}
encoded, err := json.Marshal(response)
if err != nil {
return err
}
encoded = append(encoded, '\n')
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(policy.status)
n, err := w.Write(encoded)
if err != nil {
return err
}
if n != len(encoded) {
return io.ErrShortWrite
}
return nil
}
type singleRequestChatStream struct {
mu sync.Mutex
w http.ResponseWriter
flusher http.Flusher
id string
model string
created int64
started bool
terminal bool
terminalErr error
}
func newSingleRequestChatStream(w http.ResponseWriter, requestID, model string) (*singleRequestChatStream, error) {
flusher, ok := w.(http.Flusher)
if !ok || strings.TrimSpace(requestID) == "" || strings.TrimSpace(model) == "" {
return nil, errors.New("single-request Chat stream is unavailable")
}
return &singleRequestChatStream{w: w, flusher: flusher, id: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), model: model, created: time.Now().Unix()}, nil
}
func (s *singleRequestChatStream) startLocked() error {
if s.started {
return nil
}
s.w.Header().Set("Content-Type", "text/event-stream")
s.w.Header().Set("Cache-Control", "no-cache")
s.w.WriteHeader(http.StatusOK)
s.flusher.Flush()
s.started = true
return nil
}
func (s *singleRequestChatStream) Start() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.terminal {
return s.terminalErr
}
return s.startLocked()
}
func (s *singleRequestChatStream) Ping() error {
s.mu.Lock()
defer s.mu.Unlock()
if s.terminal {
return s.terminalErr
}
if err := s.startLocked(); err != nil {
return err
}
_, err := io.WriteString(s.w, ": ping\n\n")
if err == nil {
s.flusher.Flush()
}
return err
}
func (s *singleRequestChatStream) writeSSELocked(value any) error {
payload, err := json.Marshal(value)
if err != nil {
return err
}
if _, err = fmt.Fprintf(s.w, "data: %s\n\n", payload); err != nil {
return err
}
s.flusher.Flush()
return nil
}
func (s *singleRequestChatStream) Final(result edgeservice.SingleRequestResult) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.terminal {
return s.terminalErr
}
if err := s.startLocked(); err != nil {
return err
}
policy := singleRequestChatTerminalPolicy(result.Terminal)
if policy.silent || policy.error || policy.finishReason == "" {
return errors.New("single-request result has no Chat stream terminal")
}
s.terminal = true
if policy.finishReason != "length" && result.Output != "" {
if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{Content: result.Output}}}}); err != nil {
s.terminalErr = err
return err
}
}
if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{}, FinishReason: policy.finishReason}}}); err != nil {
s.terminalErr = err
return err
}
if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil {
s.terminalErr = err
return err
}
s.flusher.Flush()
return nil
}
func (s *singleRequestChatStream) TerminalError(disposition edgeservice.SingleRequestTerminalDisposition) error {
s.mu.Lock()
defer s.mu.Unlock()
if s.terminal {
return s.terminalErr
}
policy := singleRequestChatTerminalPolicy(disposition)
if policy.silent {
s.terminal = true
return nil
}
if !policy.error {
policy = singleRequestChatTerminalPolicy(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
}
if err := s.startLocked(); err != nil {
return err
}
s.terminal = true
if err := s.writeSSELocked(errorResponse{Error: errorBody{Type: policy.errorType, Message: policy.message}}); err != nil {
s.terminalErr = err
return err
}
if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil {
s.terminalErr = err
return err
}
s.flusher.Flush()
return nil
}
func pumpSingleRequestChatStream(ctx context.Context, execution edgeservice.SingleRequestExecution, stream *singleRequestChatStream, tickerFactory singleRequestAnthropicTickerFactory) error {
if execution == nil || stream == nil || tickerFactory == nil {
return errors.New("single-request Chat stream is unavailable")
}
if err := ctx.Err(); err != nil {
execution.Cancel()
return err
}
if err := stream.Start(); err != nil {
execution.Cancel()
return err
}
ticker := tickerFactory()
if ticker == nil {
execution.Cancel()
return errors.New("single-request Chat stream is unavailable")
}
defer ticker.Stop()
for {
select {
case <-ctx.Done():
execution.Cancel()
return ctx.Err()
case <-ticker.Ticks():
if err := stream.Ping(); err != nil {
execution.Cancel()
return err
}
case progress, ok := <-execution.Progress():
if !ok {
if ctx.Err() != nil {
return ctx.Err()
}
if execution.State() == edgeservice.SingleRequestStateCompleted {
return nil
}
return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
}
switch progress.Stage {
case edgeservice.SingleRequestStateFinalizing:
if progress.Result == nil {
err := stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
return errors.Join(err, execution.AcknowledgeTerminal(false))
}
err := stream.Final(*progress.Result)
return errors.Join(err, execution.AcknowledgeTerminal(err == nil))
case edgeservice.SingleRequestStateFailed:
return stream.TerminalError(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}))
case edgeservice.SingleRequestStateCancelled:
return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalCancelled})
}
}
}
}

96
apps/edge/internal/openai/single_request_chat_test.go Normal file
View file

@ -0,0 +1,96 @@
package openai
import (
"context"
"errors"
"net/http/httptest"
"strings"
"testing"
edgeservice "iop/apps/edge/internal/service"
)
func TestSingleRequestChatStreamPumpProjectsOnlyFinalResult(t *testing.T) {
execution := startSingleRequestAnthropicTestExecution(t, func(
_ context.Context,
req edgeservice.SingleRequestRequest,
ctrl edgeservice.SingleRequestController,
) error {
for index, stage := range []edgeservice.SingleRequestState{
edgeservice.SingleRequestStatePlanning,
edgeservice.SingleRequestStateWorking,
edgeservice.SingleRequestStateReviewing,
edgeservice.SingleRequestStateFinalizing,
} {
envelope := edgeservice.SingleRequestEnvelope{
RequestID: req.RequestID, Sequence: uint64(index + 1), Stage: stage,
}
if stage == edgeservice.SingleRequestStateFinalizing {
envelope.Result = &edgeservice.SingleRequestResult{Output: "safe final"}
}
if err := ctrl.SubmitEnvelope(envelope); err != nil {
return err
}
}
return nil
})
w := httptest.NewRecorder()
stream, err := newSingleRequestChatStream(w, "req_chat", "virtual-model")
if err != nil {
t.Fatal(err)
}
ticker := newManualSingleRequestAnthropicTicker()
if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil {
t.Fatal(err)
}
body := w.Body.String()
if !strings.Contains(body, `"content":"safe final"`) || !strings.Contains(body, `"finish_reason":"stop"`) || strings.Count(body, "data: [DONE]") != 1 {
t.Fatalf("unexpected Chat stream: %s", body)
}
for _, private := range []string{"Planning the requested work", "Executing the requested work", "Reviewing the completed work"} {
if strings.Contains(body, private) {
t.Fatalf("internal progress leaked: %s", body)
}
}
}
func TestSingleRequestChatStreamSanitizesExecutorFailure(t *testing.T) {
execution := startSingleRequestAnthropicTestExecution(t, func(
_ context.Context,
req edgeservice.SingleRequestRequest,
ctrl edgeservice.SingleRequestController,
) error {
if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: 1, Stage: edgeservice.SingleRequestStatePlanning}); err != nil {
return err
}
return errors.New("PRIVATE_EXECUTOR_FAILURE")
})
w := httptest.NewRecorder()
stream, err := newSingleRequestChatStream(w, "req_chat_error", "virtual-model")
if err != nil {
t.Fatal(err)
}
ticker := newManualSingleRequestAnthropicTicker()
if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil {
t.Fatal(err)
}
body := w.Body.String()
if strings.Contains(body, "PRIVATE_EXECUTOR_FAILURE") || !strings.Contains(body, "single-request execution failed") || strings.Count(body, "data: [DONE]") != 1 {
t.Fatalf("unexpected sanitized Chat error: %s", body)
}
}
func TestMarkedSingleRequestSkipsLegacyWorkspaceToolBinding(t *testing.T) {
dispatch := routeDispatch{
Preset: validSingleRequestPreset(),
SingleRequest: newSingleRequestAnthropicTestBinding(t),
}
binding, pinArtifact, err := (&Server{}).compilePresetArtifactBinding(
dispatch,
"openai",
[]byte(`{"model":"virtual-model","messages":[{"role":"user","content":"task"}]}`),
)
if err != nil || binding != nil || pinArtifact {
t.Fatalf("marked single request entered legacy workspace binding: binding=%+v pin=%t err=%v", binding, pinArtifact, err)
}
}

6
apps/edge/internal/openai/single_request_provider_stage.go
View file

@ -185,8 +185,10 @@ type singleRequestProviderOutputSchema struct {
}
type singleRequestProviderOutputProperty struct {
Type string `json:"type"`
Description string `json:"description"`
Type string `json:"type"`
Description string `json:"description,omitempty"`
Enum []string `json:"enum,omitempty"`
MinLength int `json:"minLength,omitempty"`
}
// buildSingleRequestChatBody owns all request authority. Stage options are

49
apps/edge/internal/openai/single_request_review_stage.go
View file

@ -15,7 +15,7 @@ import (
)
const (
singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
singleRequestReviewStageID = "review"
)
@ -110,7 +110,9 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
{Role: "user", Content: "Task:\n" + strings.TrimSpace(req.Task) + "\n\nPLAN:\n" + string(plan) + "\n\nWORK COMPLETION:\n" + strings.TrimSpace(req.Work.Completion) + "\n\nWORK VERIFICATION:\n" + strings.TrimSpace(req.Work.Verification)},
}
repairRequired := false
for attempts := 0; attempts <= req.Limits.MaxToolIterations; attempts++ {
invalidToolCorrections := 0
toolAttempts := 0
for {
response, err := s.submit(ctx, req, messages, tools, repairRequired)
if err != nil {
return nil, quality.reclassify(err, errSingleRequestReviewStage)
@ -132,7 +134,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
}
return result, nil
}
if response.call == nil || attempts == req.Limits.MaxToolIterations {
if response.call == nil || toolAttempts >= req.Limits.MaxToolIterations {
if response.call != nil {
return nil, quality.budget(errSingleRequestReviewStage)
}
@ -142,6 +144,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
if err != nil {
return nil, quality.malformed(errSingleRequestReviewStage)
}
arguments = normalizeSingleRequestProviderToolArguments(response.call.Function.Name, arguments)
isInspection := response.call.Function.Name == edgeservice.InternalWorkspaceToolRead || response.call.Function.Name == edgeservice.InternalWorkspaceToolList
if !isInspection && !isSingleRequestReviewRepairTool(response.call.Function.Name) {
return nil, quality.malformed(errSingleRequestReviewStage)
@ -168,12 +171,24 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
}
}
key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestReviewStageID, toolCallID: response.call.ID}
call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments}
if err := edgeservice.ValidateInternalWorkspaceToolCall(call); err != nil {
if invalidToolCorrections >= 1 {
return nil, quality.malformed(errSingleRequestReviewStage)
}
invalidToolCorrections++
messages = append(messages,
chatMessage{Role: "assistant", ToolCalls: []any{response.call.asChatToolCall()}},
chatMessage{Role: "tool", ToolCallID: response.call.ID, ToolName: response.call.Function.Name, Content: singleRequestWorkToolResultContent(edgeservice.InternalWorkspaceToolResult{Status: "invalid", ErrorCode: "invalid_request"}, req.Limits.MaxOutputBytes)},
)
continue
}
toolAttempts++
resultCh, err := s.bridge.register(key)
if err != nil {
return nil, quality.internalTool(errSingleRequestReviewStage)
}
sequence++
call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments}
if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: sequence, Stage: edgeservice.SingleRequestStateInternalTool, SavedStage: stage, ToolCall: call}); err != nil {
s.bridge.unregister(key)
return nil, quality.serviceFailure(ctx, err, errSingleRequestReviewStage)
@ -249,7 +264,7 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any
if repairRequired {
toolChoice = "required"
}
body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "stream": false}
body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "response_format": singleRequestReviewResponseFormat(), "stream": false}
for key, value := range options {
folded := strings.ToLower(key)
if isSingleRequestReviewReservedOption(folded) && key != folded {
@ -269,13 +284,35 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any
func isSingleRequestReviewReservedOption(key string) bool {
switch key {
case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "stream", "credential", "credential_binding", "reasoning_effort":
case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "response_format", "stream", "credential", "credential_binding", "reasoning_effort":
return true
default:
return false
}
}
func singleRequestReviewResponseFormat() *singleRequestProviderResponseFormat {
return &singleRequestProviderResponseFormat{
Type: "json_schema",
JSONSchema: singleRequestProviderResponseJSONSchema{
Name: "single_request_review",
Strict: true,
Schema: singleRequestProviderOutputSchema{
Type: "object",
Properties: map[string]singleRequestProviderOutputProperty{
"decision": {Type: "string", Enum: []string{"pass"}},
"output": {Type: "string", MinLength: 1},
"checks": {Type: "string", MinLength: 1},
"verification": {Type: "string", MinLength: 1},
"summary": {Type: "string", MinLength: 1},
},
Required: []string{"decision", "output", "checks", "verification", "summary"},
AdditionalProperties: false,
},
},
}
}
type singleRequestReviewProviderEnvelope struct {
ID string `json:"id"`
Object string `json:"object"`

121
apps/edge/internal/openai/single_request_review_stage_test.go
View file

@ -383,6 +383,52 @@ func TestSingleRequestReviewStageInspectionAndRepairRemainInLegalStates(t *testi
})
}
func TestSingleRequestReviewStageBoundedInvalidToolCorrection(t *testing.T) {
t.Run("one malformed provider call is corrected before coordinator admission", func(t *testing.T) {
bridge := newSingleRequestWorkToolBridge()
ctrl := newReviewController(t, bridge)
var bodies [][]byte
responses := [][]byte{
reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
reviewToolBody("inspect-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":"result.txt"}`),
reviewPassBody("Approved.", "Corrected inspection passed."),
}
stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge)
if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); err != nil {
t.Fatal(err)
}
if len(bodies) != 3 || !strings.Contains(string(bodies[1]), "invalid_request") {
t.Fatalf("provider correction bodies=%d", len(bodies))
}
want := []edgeservice.SingleRequestState{edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateInternalTool, edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateFinalizing}
if len(ctrl.envelopes) != len(want) {
t.Fatalf("envelopes=%+v", ctrl.envelopes)
}
for index, state := range want {
if ctrl.envelopes[index].Stage != state {
t.Fatalf("envelopes=%+v", ctrl.envelopes)
}
}
})
t.Run("second malformed provider call fails closed", func(t *testing.T) {
bridge := newSingleRequestWorkToolBridge()
ctrl := newReviewController(t, bridge)
var bodies [][]byte
responses := [][]byte{
reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
reviewToolBody("invalid-2", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
}
stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge)
if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); !errors.Is(err, errSingleRequestReviewStage) {
t.Fatalf("err=%v, want review stage failure", err)
}
if len(bodies) != 2 || len(ctrl.envelopes) != 1 || ctrl.envelopes[0].Stage != edgeservice.SingleRequestStateReviewing || bridge.pendingCount() != 0 {
t.Fatalf("bodies=%d envelopes=%+v pending=%d", len(bodies), ctrl.envelopes, bridge.pendingCount())
}
})
}
func TestSingleRequestReviewStageFailsClosed(t *testing.T) {
for _, raw := range []string{
`{"decision":"pass","output":"x","checks":"c","verification":"v","summary":"y","extra":1}`,
@ -471,11 +517,41 @@ func TestSingleRequestReviewBodyRejectsOptionAliases(t *testing.T) {
if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "low"}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) {
t.Fatalf("err=%v", err)
}
if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "high", "Response_Format": map[string]any{"type": "caller"}}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) {
t.Fatalf("response format alias err=%v", err)
}
if _, _, err := renderSingleRequestReview(singlerequesttemplate.DefaultReviewTemplate, singleRequestReviewDecision{Decision: "pass", Output: strings.Repeat("x", 10), Checks: "c", Verification: "v", Summary: "summary"}, 9); !errors.Is(err, errSingleRequestReviewStage) {
t.Fatalf("err=%v", err)
}
}
func TestSingleRequestReviewResponseFormatIsClosed(t *testing.T) {
format := singleRequestReviewResponseFormat()
if format.Type != "json_schema" || format.JSONSchema.Name != "single_request_review" || !format.JSONSchema.Strict {
t.Fatalf("response format=%+v", format)
}
schema := format.JSONSchema.Schema
if schema.Type != "object" || schema.AdditionalProperties || len(schema.Properties) != 5 || len(schema.Required) != 5 {
t.Fatalf("review schema=%+v", schema)
}
if decision := schema.Properties["decision"]; !reflect.DeepEqual(decision.Enum, []string{"pass"}) {
t.Fatalf("decision schema=%+v", decision)
}
for _, name := range []string{"output", "checks", "verification", "summary"} {
if schema.Properties[name].MinLength != 1 {
t.Fatalf("property %q schema=%+v", name, schema.Properties[name])
}
}
}
func TestSingleRequestReviewPromptRequiresCanonicalWorkspaceRelativePaths(t *testing.T) {
for _, requirement := range []string{"canonical", "workspace-relative", "README.md", "./README.md", "absolute path", "parent traversal"} {
if !strings.Contains(singleRequestReviewPrompt, requirement) {
t.Fatalf("review prompt does not describe %q path requirement", requirement)
}
}
}
// customReviewStageTemplate is an operator-authored effective Review template
// that differs from the built-in default, so a rendered artifact cannot pass by
// accidentally falling back.
@ -645,7 +721,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"messages": [
{
"role": "system",
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
},
{
"role": "user",
@ -664,7 +740,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -681,7 +759,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -698,7 +778,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path", "content"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
},
"content": {
"type": "string"
@ -718,7 +800,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -764,7 +848,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"messages": [
{
"role": "system",
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
},
{
"role": "user",
@ -808,7 +892,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -825,7 +911,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -842,7 +930,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path", "content"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
},
"content": {
"type": "string"
@ -862,7 +952,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
"required": ["relative_path"],
"properties": {
"relative_path": {
"type": "string"
"type": "string",
"minLength": 1,
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
}
}
}
@ -900,6 +992,15 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
}
var res map[string]any
_ = json.Unmarshal([]byte(raw), &res)
for _, rawTool := range res["tools"].([]any) {
tool := rawTool.(map[string]any)
function := tool["function"].(map[string]any)
function["strict"] = true
}
encodedFormat, _ := json.Marshal(singleRequestReviewResponseFormat())
var expectedFormat any
_ = json.Unmarshal(encodedFormat, &expectedFormat)
res["response_format"] = expectedFormat
return res
}

37
apps/edge/internal/openai/single_request_work_stage.go
View file

@ -16,8 +16,9 @@ import (
)
const (
singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification."
singleRequestWorkStageID = "work"
singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification. Every relative_path argument and every workspace path mentioned in the completion or verification must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal."
singleRequestWorkStageID = "work"
singleRequestCanonicalRelativePathDescription = "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
)
var errSingleRequestWorkStage = errors.New("single-request work stage: failed")
@ -240,6 +241,7 @@ func (s *singleRequestWorkStage) run(ctx context.Context, req singleRequestWorkS
if err != nil {
return nil, quality.malformed(errSingleRequestWorkStage)
}
arguments = normalizeSingleRequestProviderToolArguments(call.Function.Name, arguments)
key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestWorkStageID, toolCallID: call.ID}
resultCh, err := s.bridge.register(key)
if err != nil {
@ -283,14 +285,15 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding
return false
}
tools := make([]any, 0, len(workspace.OperationIDs))
path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}}}
pathProperty := map[string]any{"type": "string", "minLength": 1, "description": singleRequestCanonicalRelativePathDescription}
path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": pathProperty}}
for _, pair := range []struct{ operation, name string }{{"read", edgeservice.InternalWorkspaceToolRead}, {"list", edgeservice.InternalWorkspaceToolList}, {"write", edgeservice.InternalWorkspaceToolWrite}, {"delete", edgeservice.InternalWorkspaceToolDelete}} {
if !has(pair.operation) {
continue
}
parameters := path
if pair.operation == "write" {
parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}, "content": map[string]any{"type": "string"}}}
parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": pathProperty, "content": map[string]any{"type": "string"}}}
}
tools = append(tools, singleRequestWorkToolSchema(pair.name, parameters))
}
@ -316,7 +319,7 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding
}
func singleRequestWorkToolSchema(name string, parameters map[string]any) map[string]any {
return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "parameters": parameters}}
return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "strict": true, "parameters": parameters}}
}
type singleRequestWorkProviderResponse struct {
@ -514,6 +517,30 @@ func decodeSingleRequestWorkToolArguments(arguments string) (json.RawMessage, er
return append(json.RawMessage(nil), bytes.TrimSpace(raw)...), nil
}
// normalizeSingleRequestProviderToolArguments canonicalizes the one
// containment-safe provider convention that differs from the internal tool
// contract: an empty workspace_list path means the workspace root. All other
// operations and non-canonical paths remain unchanged and are rejected by the
// service-owned decoder.
func normalizeSingleRequestProviderToolArguments(name string, arguments json.RawMessage) json.RawMessage {
if name != edgeservice.InternalWorkspaceToolList {
return arguments
}
var pathArguments struct {
RelativePath *string `json:"relative_path"`
}
decoder := json.NewDecoder(bytes.NewReader(arguments))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&pathArguments); err != nil || pathArguments.RelativePath == nil || *pathArguments.RelativePath != "" {
return arguments
}
var trailing any
if err := decoder.Decode(&trailing); err != io.EOF {
return arguments
}
return json.RawMessage(`{"relative_path":"."}`)
}
func decodeSingleRequestWorkProviderResponse(body []byte, maximum int) (*singleRequestWorkProviderResponse, error) {
if len(body) == 0 || len(body) > maximum || validateSingleRequestJSON(body) != nil {
return nil, errSingleRequestWorkStage

54
apps/edge/internal/openai/single_request_work_stage_test.go
View file

@ -877,6 +877,29 @@ func TestSingleRequestWorkStageRejectsMalformedResponsesAndOptions(t *testing.T)
}
}
func TestNormalizeSingleRequestProviderToolArguments(t *testing.T) {
tests := []struct {
name string
tool string
arguments string
want string
}{
{name: "empty list root becomes canonical dot", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":""}`, want: `{"relative_path":"."}`},
{name: "empty read remains invalid for service", tool: edgeservice.InternalWorkspaceToolRead, arguments: `{"relative_path":""}`, want: `{"relative_path":""}`},
{name: "dot-prefixed list remains invalid for service", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"./"}`, want: `{"relative_path":"./"}`},
{name: "canonical list path is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"src"}`, want: `{"relative_path":"src"}`},
{name: "unknown list field is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"","extra":true}`, want: `{"relative_path":"","extra":true}`},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
got := normalizeSingleRequestProviderToolArguments(test.tool, json.RawMessage(test.arguments))
if string(got) != test.want {
t.Fatalf("normalized arguments=%s, want %s", got, test.want)
}
})
}
}
func TestSingleRequestWorkStageRejectsReservedOptionAliases(t *testing.T) {
messageSets := map[string][]chatMessage{
"initial": {{Role: "user", Content: "immutable task"}},
@ -977,6 +1000,37 @@ func TestSingleRequestWorkStageProjectsClosedEnvironmentSchema(t *testing.T) {
}
}
func TestSingleRequestWorkspaceToolSchemasDescribeCanonicalRelativePaths(t *testing.T) {
tools, err := singleRequestWorkTools(workBinding(t).Workspace)
if err != nil {
t.Fatal(err)
}
pathTools := 0
for _, raw := range tools {
tool, _ := raw.(map[string]any)
function, _ := tool["function"].(map[string]any)
if function["strict"] != true {
t.Fatalf("tool %q strict=%v, want true", function["name"], function["strict"])
}
parameters, _ := function["parameters"].(map[string]any)
properties, _ := parameters["properties"].(map[string]any)
pathProperty, hasPath := properties["relative_path"].(map[string]any)
if !hasPath {
continue
}
pathTools++
if pathProperty["description"] != singleRequestCanonicalRelativePathDescription {
t.Fatalf("tool %q relative_path description=%v", function["name"], pathProperty["description"])
}
if pathProperty["minLength"] != 1 {
t.Fatalf("tool %q relative_path minLength=%v, want 1", function["name"], pathProperty["minLength"])
}
}
if pathTools == 0 {
t.Fatal("workspace tools did not expose a relative_path schema")
}
}
func TestSingleRequestWorkToolBridgeCorrelatesAndCleansUp(t *testing.T) {
b := newSingleRequestWorkToolBridge()
keys := []singleRequestWorkToolKey{{"request-a", "working", "one"}, {"request-b", "working", "two"}}

9
apps/edge/internal/service/single_request_tool_types.go
View file

@ -81,6 +81,15 @@ func (r InternalWorkspaceToolResult) Clone() InternalWorkspaceToolResult {
return r
}
// ValidateInternalWorkspaceToolCall applies the service-owned closed decoder
// without executing the call. Private provider stages use it to reject or
// boundedly correct malformed model output before the coordinator enters an
// internal-tool state.
func ValidateInternalWorkspaceToolCall(call *InternalWorkspaceToolCall) error {
_, err := decodeInternalWorkspaceToolCall(call.Clone())
return err
}
// SingleRequestToolContinuation is optional. An executor that emits an
// internal workspace call must implement it so the coordinator can deliver the
// correlated Node result without involving an HTTP caller.

12
apps/edge/internal/service/single_request_tool_types_test.go
View file

@ -53,6 +53,18 @@ func TestInternalWorkspaceToolDecodeClosedOperations(t *testing.T) {
}
}
func TestValidateInternalWorkspaceToolCall(t *testing.T) {
if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolList, `{"relative_path":"."}`)); err != nil {
t.Fatalf("canonical call rejected: %v", err)
}
if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolRead, `{"relative_path":""}`)); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) {
t.Fatalf("empty read error=%v, want invalid call", err)
}
if err := ValidateInternalWorkspaceToolCall(nil); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) {
t.Fatalf("nil call error=%v, want invalid call", err)
}
}
func TestInternalWorkspaceToolDecodeRejectsMalformed(t *testing.T) {
const rawSentinel = "RAW-ARGUMENT-SENTINEL-DO-NOT-LEAK"
tests := map[string]*InternalWorkspaceToolCall{

563
docs/agent-comparison-benchmark-dev-guide.md Normal file
View file

@ -0,0 +1,563 @@
# Agent Comparison Benchmark Dev Guide
이 문서는 IOP one-shot agent/model comparison benchmark의 현재 구현, dev 환경 구성, caller 연결 방식, managed credential 경계, 실행 절차와 장애 대응 기준을 한곳에 모은 운영 가이드다.
raw token, provider API key, private key, slot alias, lease id와 개인 endpoint는 이 문서에 기록하지 않는다. 실제 host, checkout, Node/provider endpoint와 최신 process 상태는 아래 source of truth에서 확인한다.
- benchmark manifest: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
- public CLI: `scripts/agent_comparison_benchmark.py`
- dev environment: `agent-test/dev/rules.md`
- machine-readable dev inventory: `agent-test/inventory-dev.yaml`
- Edge/Node verification: `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md`
- API contracts: `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`
- benchmark SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
이 문서는 위 계약과 manifest를 설명할 뿐 덮어쓰지 않는다. 값이 다르면 manifest, contract, inventory, environment rule 순서로 최신 상태를 확인한다.
## 1. 현재 상태
2026-08-12 기준 상태는 다음과 같다.
| 영역 | 상태 |
|---|---|
| Gemini-native Edge ingress | 구현 완료. route-qualified `streamGenerateContent`, `x-goog-api-key` IOP principal auth, request/tool/SSE 변환을 지원한다. |
| official agy adapter | 구현 완료. `agy 1.1.12`, Gemini API-key provider, real `init/step_update/result` JSONL을 사용한다. |
| managed dev credential runtime | 구성 완료. Control Plane projection, credential slot/route, sealed lease, Edge HTTPS, CP↔Edge/Edge↔Node mTLS를 사용한다. |
| caller readiness | 최근 완료된 readiness evidence에서 C01-C09 `ready=9`를 확인했다. 실제 실행일에는 fresh preflight가 다시 필요하다. |
| controller recovery | control socket symlink와 caller launch 전 interruption 회귀 수정 및 테스트가 완료됐다. |
| Codex production JSONL | `cache_write_input_tokens` 수용과 config-owned effective binding 정합화가 완료됐다. |
| deterministic tests | focused 41 tests, 전체 benchmark 429 tests가 통과했다. 숫자는 당시 snapshot이며 현재 suite 결과는 fresh 실행으로 판단한다. |
| C01-C09 scored execution | 아직 완료되지 않았다. 과거 incomplete/failed run은 append-only evidence로 보존하며 정상 결과로 간주하지 않는다. |
| blind scoring/report | 유효한 9-cell execution이 생긴 뒤 수행하는 후속 단계다. |
과거 실행 실패는 새 run을 정당화하는 완료 evidence가 아니다. 이전 run tree를 수정하거나 old run을 암묵적으로 `resume`/retry하지 않는다. 새 scored execution은 그 실행을 명시적으로 소유하는 현재 plan과 사용자 권한이 있을 때만 한 번 수행한다. 이 문서 자체는 지속적 실행 승인을 부여하지 않는다.
## 2. 시스템 구성
```text
Benchmark runner
├─ Claude Code ── Anthropic Messages ───────────────┐
├─ agy ───────── Gemini streamGenerateContent ─────┤
└─ Codex ─────── OpenAI Responses ─────────────────┤
v
IOP Edge HTTPS
│
┌───────────────────────────┴───────────────────────────┐
│ │
direct model route execution preset
│ selector/plan/work/review/repair
└───────────────────────────┬───────────────────────────┘
v
managed credential lease
Control Plane projection + Node sealed lease
│
v
Node-owned provider
```
핵심 보안 경계는 다음과 같다.
- benchmark caller는 하나의 IOP principal token으로 Edge만 인증한다.
- caller token은 upstream provider credential이 아니다.
- provider credential은 Control Plane에 등록된 slot과 Node 대상 sealed lease에서만 온다.
- Edge는 provider credential을 복호화하지 않고 caller가 보낸 token을 upstream auth로 재사용하지 않는다.
- managed mode는 legacy static principal/provider credential과 혼용하지 않는다.
- config observation은 secret이 아닌 route/model/stage binding의 독립 증거다.
## 3. 환경 프로필
### 3.1 Benchmark runner
현재 검증된 runner class는 Linux/AArch64다. 명령은 repository root에서 실행한다.
필수 command:
```bash
command -v python3
command -v git
command -v claude
command -v agy
command -v codex
```
2026-08-12 확인 snapshot:
| Tool | 확인된 버전 | 정책 |
|---|---|---|
| Claude Code | `2.1.228` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`/`--help`를 확인한다. |
| agy | `1.1.12` | adapter가 이 버전을 명시적으로 gate한다. 다른 버전은 재검증 전 fail closed한다. |
| Codex CLI | `0.147.0` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`과 `exec --help`를 확인한다. |
### 3.2 Testbed
- path: `../iop-s2`
- 현재 확인 branch: `dev`
- 현재 확인 상태: clean
- benchmark는 testbed를 read-only provenance로 취급한다.
- caller별 workspace와 session은 run tree 아래에 새로 만들며 서로 공유하지 않는다.
- testbed를 benchmark 결과로 수정하거나 결과 파일을 다시 복사하지 않는다.
fixture checksum과 source file 목록은 manifest가 고정한다. testbed HEAD와 clean 상태는 실행일에 다시 확인한다.
### 3.3 Dev runtime
현재 검증된 runtime class는 macOS/ARM64 remote dev runner다. exact SSH target과 checkout은 `agent-test/dev/rules.md` 및 `agent-test/inventory-dev.yaml`을 따른다.
benchmark 관련 runtime 역할:
| Port | 역할 |
|---:|---|
| `18082` | Edge artifact/bootstrap HTTP |
| `18083` | managed Edge public HTTPS; Anthropic/OpenAI/Gemini caller ingress |
| `18084` | native dev-runtime Edge↔Node TCP |
| `19093` | Edge admin/config refresh |
| `19101` | Edge metrics |
2026-08-12 read-only 확인에서 위 listener와 managed Edge process는 모두 active였고 remote checkout은 clean release 상태였다. exact commit, binary checksum, process id와 endpoint는 실행 evidence에만 기록하고 이 가이드에 고정하지 않는다.
최근 완료된 live readiness evidence는 4 connected Nodes와 8 healthy/available providers를 확인했다. 최신 Node/provider 세부와 접속 위치는 반드시 `agent-test/inventory-dev.yaml`에서 다시 확인한다.
이 benchmark는 compose dev stack의 Edge-Node TCP `19003`이 아니라 native dev-runtime provider pool의 `18084`를 사용한다. compose와 native profile은 포트, process와 판정 evidence가 서로 다르므로 한 실행에서 섞지 않는다. 현재 inventory가 가리키는 배포 산출물은 다음과 같다.
| Artifact | Path |
|---|---|
| native Edge config | `build/dev-runtime/edge.yaml` |
| Edge binary | `build/dev-runtime/bin/edge` |
| macOS Node binary | `build/dev-runtime/bin/iop-node` |
| Linux ARM64 Node binary | `build/dev-runtime/bin/iop-node-linux-arm64` |
| Windows AMD64 Node binary | `build/dev-runtime/bin/iop-node-windows-amd64.exe` |
모든 Edge/Node binary는 scored execution 전에 동일 source ref로 rebuild·redeploy·restart한다. `build/dev-runtime/**`의 runtime config와 untracked credential material은 원격 runner가 소유하며 tracked 문서나 testbed로 복사하지 않는다.
## 4. 보호 파일과 credential 역할
benchmark runner의 `token/` 아래에는 다음 파일이 준비돼 있다. 파일 존재와 mode만 확인하며 내용을 출력하지 않는다.
| Path | 역할 | 실행 시 사용 |
|---|---|---|
| `token/.iop-bench` | benchmark용 IOP principal token | preflight/run/score caller가 Edge를 인증할 때 사용 |
| `token/iop-dev-ca.pem` | managed dev Edge HTTPS CA certificate | `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`로 전달 |
| `token/.claude` | Claude provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
| `token/.gemini` | Gemini provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
| `token/.gpt` | GPT provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
현재 secret source 파일은 `0600`, CA certificate는 `0644`로 확인됐다. CA certificate는 public trust material이지만 private key는 아니다.
안전 확인:
```bash
for benchmark_secret_file in token/.iop-bench token/.claude token/.gemini token/.gpt; do
test -f "$benchmark_secret_file"
test "$(stat -c '%a' "$benchmark_secret_file")" = 600
done
test -f token/iop-dev-ca.pem
```
macOS에서 동일 검사를 수행할 때는 BSD `stat` 문법을 사용한다. 어떤 경우에도 `cat`, `echo`, shell tracing(`set -x`)으로 secret 내용을 출력하지 않는다.
## 5. Provider와 managed credential 설정
현재 dev 구성은 다음 절차로 만들어졌다.
1. Control Plane credential plane용 CA, workload certificate, at-rest keyring과 lease issuer/recipient key를 operator-owned untracked 경로에 생성했다.
2. Control Plane, Edge와 각 Node에 role/name-bound mTLS identity를 배치했다.
3. Edge public ingress를 HTTPS로 구성하고 benchmark runner에 CA certificate만 전달했다.
4. benchmark principal을 bootstrap하고 one-time token을 `token/.iop-bench`에 저장했다.
5. `token/.claude`, `token/.gemini`, `token/.gpt`의 raw provider key를 credential HTTPS request body로 직접 등록했다. command argument, YAML, tracked docs나 task evidence에는 넣지 않았다.
6. credential slot과 public route를 별도로 생성하고 principal projection에 direct route와 hybrid preset stage authorization을 연결했다.
7. Control Plane → Edge → Nodes 순서로 bounded restart하고 fresh projection, sealed lease, route revision과 no-fallback 동작을 검증했다.
재구성이 필요하면 `docs/edge-local-dev-guide.md`의 “Managed credential plane and TLS startup”과 “Safe slot lifecycle”을 따른다. 실제 slot id, alias, revision과 lease id는 운영 상태이므로 이 문서에 복사하지 않는다.
## 6. Caller별 연결 방식
| Caller | Edge surface | Child 설정 | 중요한 제한 |
|---|---|---|---|
| Claude Code | Anthropic-compatible Messages | `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`; `--bare --print --verbose --output-format stream-json --no-session-persistence --permission-mode dontAsk --tools Read,Write,Edit --allowedTools Read,Write,Edit` | task는 stdin으로 한 번 제출한다. network/shell 도구 없이 격리 workspace 파일 작업만 허용한다. model/effort는 manifest 값을 그대로 전달한다. |
| agy | Gemini-native `streamGenerateContent` | fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`; `--sandbox --output-format stream-json --model ... --print ...` | `agy 1.1.12`만 승인된다. API-key provider에 `--effort`를 전달하지 않으며 ambient user config를 읽지 않는다. |
| Codex | OpenAI-compatible Responses | fresh session `HOME`, isolated `iop_benchmark` provider override, `IOP_BENCHMARK_API_KEY`; `exec --sandbox workspace-write --json --ephemeral --ignore-user-config --strict-config` | user config를 읽지 않고 격리 workspace에만 쓸 수 있다. current adapter effort는 `xhigh`; caller binding event가 없으면 admitted config binding을 사용하고, 보고된 mismatch는 거부한다. |
세 caller child에는 필요한 `PATH`와 CA 변수만 allowlist로 전달한다. parent의 caller/provider 설정이나 unrelated secret은 상속하지 않는다.
agy의 public base는 adapter가 cell별로 다음처럼 route-qualified 한다.
```text
<edge-origin>/gemini/<route-id>
```
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 benchmark transport가 아니다.
## 7. Benchmark 고정 설정
| 설정 | 값 |
|---|---|
| pipeline version | `2` |
| environment | `dev` |
| execution seed | `bench-02-c01-c09-v1` |
| repetitions | `1` |
| session policy | `fresh` |
| setup cache policy | `isolated` |
| run timeout | 300 seconds |
| idle timeout | 30 seconds |
| quiet window | 10 seconds |
| cleanup grace | 5 seconds |
| desktop viewport | `1920x1080` |
| mobile viewport | `375x812` |
| output root | `agent-test/runs/bench-02` |
| rubric | `one-shot-agent-comparison-v1` |
execution seed로 결정되는 현재 slot 순서는 다음과 같다. 표의 C 번호 순서와 실제 실행 순서는 다르다.
1. C02 Claude→Gemini direct
2. C05 Codex→GPT direct
3. C03 agy→Gemini direct
4. C06 Claude→Gemini hybrid
5. C08 Claude→GPT hybrid
6. C09 Codex→GPT hybrid
7. C01 Claude→Sonnet direct
8. C07 agy→Gemini hybrid
9. C04 Claude→GPT direct
## 8. C01-C09 matrix와 route binding
| Cell | Caller | Route kind/id | Requested model/effort | Effective stage binding |
|---|---|---|---|---|
| C01 | Claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / `max` | request=`claude-sonnet-5` max |
| C02 | Claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
| C03 | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
| C04 | Claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
| C05 | Codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
| C06 | Claude | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
| C07 | agy | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
| C08 | Claude | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
| C09 | Codex | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
hybrid preset은 caller가 stage를 따로 호출하는 구조가 아니다. 하나의 caller request 안에서 Edge가 selector/plan/work/review/repair를 소유한다.
## 9. Fixture와 결과 조건
공통 task는 fictional product “Lumen Atlas”의 responsive one-page landing page다.
- prompt: `scripts/fixtures/agent-comparison-benchmark/prompt.md`
- copy: `scripts/fixtures/agent-comparison-benchmark/reference.txt`
- images: `aurora-grid.svg`, `orbit-rings.svg`
- 생성 파일: workspace root의 `index.html`, `styles.css`, `script.js` 정확히 세 개
- 외부 asset, framework, package manager, build tool, analytics와 network dependency 금지
- desktop/mobile responsive, semantic HTML, focus/contrast/accessibility 요구
- 각 attempt는 fresh workspace/session에서 task를 한 번만 제출
fixture checksum은 manifest의 값이 유일한 기준이다. prompt나 asset을 변경하면 기존 run과 비교하지 말고 manifest/version/checksum을 함께 갱신하는 별도 작업으로 처리한다.
## 10. Process environment 준비
다음은 value를 출력하지 않는 process-local 예시다. `<edge-host>`를 문서에 실제 값으로 치환하지 말고 실행 환경에서만 주입한다.
Public live registry가 소비하는 environment contract는 다음과 같다.
| Variable | 값/의미 | Durable evidence |
|---|---|---|
| `IOP_BENCH_CLAUDE_BASE_URL` | managed Edge HTTPS origin | raw 값 금지; endpoint digest만 허용 |
| `IOP_BENCH_AGY_BASE_URL` | managed Edge HTTPS origin; adapter가 `/gemini/<route-id>`를 추가 | raw 값 금지; endpoint digest만 허용 |
| `IOP_BENCH_CODEX_BASE_URL` | managed Edge OpenAI-compatible `/v1` base | raw 값 금지; endpoint digest만 허용 |
| `IOP_BENCH_CLAUDE_SECRET_ENV` | Claude가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
| `IOP_BENCH_AGY_SECRET_ENV` | agy가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
| `IOP_BENCH_CODEX_SECRET_ENV` | Codex가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
| `IOP_BENCH_SHARED_TOKEN` | 이 가이드 예시의 secret-bearing variable | 값 기록 금지 |
| `SSL_CERT_FILE` | dev Edge HTTPS CA certificate path | repository-relative file reference만 허용 |
| `NODE_EXTRA_CA_CERTS` | Node.js caller용 동일 CA certificate path | repository-relative file reference만 허용 |
| `IOP_BENCH_CONFIG_OBSERVATION_ENV` | config JSON을 보유한 variable 이름 | variable 이름만 허용 |
| `BENCH_CONFIG` | schema v1 route/model/stage observation JSON | secret은 없지만 runtime과 일치하는 canonical digest만 evidence에 기록 |
| `PATH` | caller binary resolution | resolved executable path/version만 preflight에서 확인 |
Child adapter가 내부적으로 만드는 값은 caller별로 격리된다.
- Claude: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, restricted `Read,Write,Edit`, nonessential traffic/autoupdater disable flags와 CA variables
- agy: fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `LANG=C.UTF-8`, `LC_ALL=C.UTF-8`, `TZ=UTC`와 CA variables
- Codex: fresh session `HOME`, `IOP_BENCHMARK_API_KEY`, strict ephemeral provider override와 CA variables
이 child variable은 사용자가 별도로 준비할 값이 아니다. live registry가 위 public contract에서 파생하며, parent의 같은 이름 값을 그대로 신뢰하거나 상속하지 않는다.
```bash
set -euo pipefail
benchmark_edge_origin="${BENCHMARK_EDGE_ORIGIN:?set BENCHMARK_EDGE_ORIGIN to the managed dev Edge HTTPS origin}"
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
export IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CLAUDE_BASE_URL="$benchmark_edge_origin"
export IOP_BENCH_AGY_BASE_URL="$benchmark_edge_origin"
export IOP_BENCH_CODEX_BASE_URL="$benchmark_edge_origin/v1"
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
BENCH_CONFIG="$(python3 - <<'PY'
import json
routes = [
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
]
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
PY
)"
export BENCH_CONFIG
```
중요한 의미:
- `IOP_BENCH_*_SECRET_ENV`의 값은 secret이 아니라 실제 secret을 보유한 environment variable의 이름이다.
- 세 caller는 같은 IOP principal을 사용하지만 서로 다른 protocol base를 받는다.
- `BENCH_CONFIG`는 runtime에서 독립적으로 확인한 route/stage snapshot이어야 한다. manifest를 보고 임의 합성한 값을 live readiness evidence로 사용하면 안 된다.
- endpoint와 config는 evidence에 raw 값 대신 digest identity로만 남는다.
작업 후에는 같은 shell에서 다음 변수를 제거한다.
```bash
unset IOP_BENCH_SHARED_TOKEN
unset IOP_BENCH_CLAUDE_BASE_URL IOP_BENCH_AGY_BASE_URL IOP_BENCH_CODEX_BASE_URL
unset IOP_BENCH_CLAUDE_SECRET_ENV IOP_BENCH_AGY_SECRET_ENV IOP_BENCH_CODEX_SECRET_ENV
unset IOP_BENCH_CONFIG_OBSERVATION_ENV BENCH_CONFIG
unset SSL_CERT_FILE NODE_EXTRA_CA_CERTS
unset BENCHMARK_EDGE_ORIGIN benchmark_edge_origin
```
## 11. 실행 절차
manifest path는 모든 명령에서 동일하게 사용한다.
```bash
benchmark_manifest=scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
```
### 11.1 Source와 deterministic verification
```bash
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
python3 scripts/agent_comparison_benchmark.py validate --manifest "$benchmark_manifest"
git diff --check
```
필요하면 변경 범위에 따라 Go tests와 managed credential qualification도 실행한다.
```bash
go test -count=1 ./...
credential_smoke_parent="$(mktemp -d /tmp/iop-benchmark-credential.XXXXXX)"
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
rmdir "$credential_smoke_parent"
```
### 11.2 Public preflight
10절의 환경을 같은 shell에 준비한 뒤 실행한다.
```bash
python3 scripts/agent_comparison_benchmark.py preflight --manifest "$benchmark_manifest"
```
성공 조건:
```text
status=ready ready=9 registration_required=0 implementation_gap=0
```
preflight는 다음을 함께 확인한다.
- caller binary/version/help
- Edge model catalog
- principal auth와 endpoint compatibility
- direct/preset route 존재
- requested model/effort
- exact stage binding과 order
- official agy transport capability
`registration_required` 또는 `implementation_gap`이면 즉시 중단한다. alias, model, effort, route나 caller를 대체하지 않는다. preflight-only run root는 evidence이므로 삭제하지 않는다.
### 11.3 Scored execution
fresh preflight와 명시적 실행 권한이 있는 현재 plan에서만 다음 명령을 한 번 호출한다.
```bash
python3 scripts/agent_comparison_benchmark.py run --manifest "$benchmark_manifest"
```
- direct CLI stdout/stderr와 exit code를 보존한다.
- CLI가 출력한 canonical `run-...` id만 이후 `status`, `score`, `report`에 사용한다.
- command가 nonzero여도 같은 plan에서 `run`을 다시 호출하지 않는다.
- CLI가 run id를 출력하지 않으면 임의 id나 성공 pointer를 만들지 않는다.
- caller나 provider를 CLI 밖에서 별도로 호출해 scored result를 보충하지 않는다.
### 11.4 Status
```bash
python3 scripts/agent_comparison_benchmark.py status \
--manifest "$benchmark_manifest" \
--run-id "$benchmark_run_id"
```
현재 comparison execution 완료 조건:
- `success + failed + timed_out + cancelled = 9`
- `running = 0`
- `interrupted = 0`
- 각 cell/repetition에 retained terminal attempt가 존재
실패/timed-out/cancelled는 terminal evidence지만 성공 결과가 아니므로 scoring eligibility와 최종 비교에서 별도로 표시된다.
### 11.5 Blind scoring
유효한 execution run에 대해서만 수행한다.
```bash
python3 scripts/agent_comparison_benchmark.py score \
--manifest "$benchmark_manifest" \
--run-id "$benchmark_run_id"
```
evaluator는 Codex→`gpt-5.6-luna` xhigh direct route다. identity가 제거된 blind workspace만 보며 source cell identity mapping은 blind tree 밖에 유지한다.
automatic gate 실패나 lifecycle failure는 `unscored`이고 0점으로 바꾸지 않는다. `scoring_failed`도 명시적 `--retry-scoring-failed` 권한 없이 재시도하지 않는다.
### 11.6 Report
```bash
python3 scripts/agent_comparison_benchmark.py report \
--manifest "$benchmark_manifest" \
--run-id "$benchmark_run_id"
```
report는 run root의 immutable evidence를 읽어 idempotent `report.md`를 만든다. 기존 report 내용과 새 projection이 다르면 덮어쓰지 않고 실패한다.
## 12. 수집 evidence
각 attempt는 다음 범주의 evidence를 가진다.
| 범주 | 내용 |
|---|---|
| lifecycle | submission, first output, finish, idle, exit, quiet, cleanup와 terminal reason |
| timeline | submitted, first output, first workspace write observation/mtime, total duration |
| usage | input/output/reasoning/cache read/cache write/total tokens, model/tool calls와 duration |
| workspace | fresh session identity, fixture checksum, testbed provenance, generated file tree |
| web validation | generated files, static safety, images, network, console, responsive, accessibility |
| screenshots | desktop `1920x1080`, mobile `375x812` |
| scoring | eligibility, blind allocation, rubric worksheet, score status |
caller가 보고하지 않은 metric은 0으로 만들지 않고 `unavailable`과 reason/source를 보존한다.
Codex current usage mapping:
| Caller JSONL field | Canonical metric |
|---|---|
| `input_tokens` | `input_tokens` |
| `cached_input_tokens` | `cached_input_tokens` |
| `cache_write_input_tokens` | `cache_write_tokens` |
| `output_tokens` | `output_tokens` |
| `reasoning_output_tokens` | `reasoning_tokens` |
| `total_tokens` | `total_tokens` |
누락된 `total_tokens`는 하위 category 합으로 재구성하지 않는다.
## 13. 자동 gate와 100점 rubric
automatic web gates는 scoring eligibility만 결정하고 점수에 포함되지 않는다.
| Gate |
|---|
| generated files |
| static safety |
| local images |
| no external network dependency |
| console safety |
| responsive layout |
| accessibility |
quality rubric:
| Category | Max |
|---|---:|
| requirements fidelity | 25 |
| visual completeness | 25 |
| responsive accessibility | 15 |
| image/detail usage | 10 |
| behavior stability | 10 |
| code quality | 10 |
| self verification | 5 |
| Total | 100 |
## 14. 실패 처리와 재개 원칙
| 상황 | 조치 |
|---|---|
| manifest invalid | source/manifest를 수정하고 validate부터 다시 시작한다. run을 만들지 않는다. |
| preflight not ready | blocker를 해결하고 fresh preflight한다. attempt를 할당하지 않는다. |
| caller launch 전 interruption | retained evidence를 보존한다. run tree를 직접 수정하지 않는다. |
| lifecycle/parser failure | exact retained output으로 source 원인을 수정하고 deterministic regression을 추가한다. |
| terminal failed/timed_out/cancelled | evidence로 보존한다. 암묵 retry하지 않는다. |
| state가 `running`이지만 process가 없음 | manual JSON 수정/삭제/reconcile을 하지 않는다. reviewer evidence로 남기고 승인된 새 plan에서만 다음 상태를 결정한다. |
| scoring_failed | 0점 처리하지 않는다. 명시적 retry 권한 없이는 중단한다. |
| report unavailable | run evidence를 수정하거나 report를 수작업 생성하지 않는다. |
현재 milestone의 원칙은 old incomplete/failed run을 `resume --retry-failed`하지 않고 distinct fresh run을 만드는 것이다. 일반 CLI가 `resume`을 지원한다는 사실이 현재 benchmark에서 사용 권한을 뜻하지 않는다.
## 15. Secret-safe 기록 규칙
다음 값은 tracked docs, task review, run metadata, log, metric label과 command argument에 남기지 않는다.
- IOP principal token
- raw provider credential
- private key, at-rest key, issuer/recipient private key
- slot alias와 lease id
- credential-bearing URL
- raw prompt/response, tool input/output
- caller/provider session content
허용되는 내용:
- secret file의 상대 path와 존재/mode
- route/model/stage 이름
- safe credential slot reference/revision
- hashed endpoint/config/spec identity
- redacted lifecycle 상태, duration과 usage count
- run id와 attempt identity가 필요한 controller evidence
incident evidence를 보존하기 전에 repository와 run output에서 secret 원문이 없는지 확인한다. 의심되는 artifact는 내용을 복사하지 말고 path와 redaction failure만 보고한다.
## 16. 실행 전 체크리스트
- [ ] 현재 manifest validation 통과
- [ ] focused/full deterministic tests fresh PASS
- [ ] benchmark runner와 `../iop-s2` provenance 확인
- [ ] Claude/agy/Codex command와 current version/help 확인
- [ ] `token/.iop-bench`, CA와 provider provisioning source 존재/mode 확인; 내용 출력 없음
- [ ] dev runtime source/build identity, process와 listener 확인
- [ ] managed projection, provider slot/route와 no-legacy-fallback 확인
- [ ] config observation이 runtime route/stage와 정확히 일치
- [ ] public preflight `ready=9`
- [ ] 현재 plan이 exactly one scored run을 소유하고 사용자 권한이 명확함
- [ ] old run resume/retry/state edit 계획 없음
- [ ] run 이후 status, scoring, report의 run id 전달 경로 준비
## 17. 관련 구현
- controller/state: `scripts/agent_benchmark/attempts.py`
- manifest: `scripts/agent_benchmark/manifest.py`
- lifecycle: `scripts/agent_benchmark/lifecycle.py`
- workspace isolation: `scripts/agent_benchmark/workspace.py`
- live routing/admission: `scripts/agent_benchmark/live_iop.py`
- Claude adapter: `scripts/agent_benchmark/claude_iop.py`
- agy adapter: `scripts/agent_benchmark/agy_iop.py`
- Codex adapter: `scripts/agent_benchmark/codex_iop.py`
- measurement: `scripts/agent_benchmark/measurement.py`
- browser/web gate: `scripts/agent_benchmark/web_validation.py`
- blind scoring: `scripts/agent_benchmark/scoring.py`
- rubric: `scripts/agent_benchmark/rubric.py`
- report: `scripts/agent_benchmark/reporting.py`

25
docs/edge-local-dev-guide.md
View file

@ -253,6 +253,31 @@ rmdir "$credential_smoke_parent"
The deterministic Messages qualification succeeds alongside Chat: the Control Plane canonicalizes built-in lowercase API-key header names (for example `x-api-key` to `X-Api-Key`) before signing the lease scope, so both managed profiles reach Node/upstream exactly once with their exact header semantics. A lease failure fails closed before dispatch and never falls back to caller auth or another slot; treat a Chat-only result or any fallback as a qualification failure.
### Official agy route smoke
공식 `agy` 1.1.12 API-key provider는 upstream Gemini key가 아니라 관리형 IOP principal token을 사용한다. dev operator가 이미 발급한 token과 CA 파일을 보호된 `token/` 아래에 둔 경우 값을 명령행에 직접 쓰지 않고 다음처럼 읽는다.
```bash
read -r IOP_BENCH_TOKEN < token/.iop-bench
export GEMINI_API_KEY="$IOP_BENCH_TOKEN"
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
GOOGLE_GEMINI_BASE_URL="https://<edge-host>:<https-port>/gemini/<direct-route-id>" \
agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \
--print 'Reply only with OK. Do not use tools or modify files.'
GOOGLE_GEMINI_BASE_URL="https://<edge-host>:<https-port>/gemini/<hybrid-preset-id>" \
agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \
--print 'Inspect README.md and report only its first Markdown heading. Do not modify files.'
unset IOP_BENCH_TOKEN GEMINI_API_KEY SSL_CERT_FILE NODE_EXTRA_CA_CERTS
```
`--effort`는 API-key provider 호출에 넣지 않는다. direct와 hybrid 모두 JSONL의 마지막 record가 `event=result`, 중첩 `result.status=SUCCESS` 한 건이어야 한다. hybrid는 plan/work/review가 포함되므로 direct보다 오래 걸릴 수 있으며, caller timeout을 이유로 같은 scored attempt를 재실행하지 않는다. Gemini ingress와 공식 event 구조의 상세 계약은 `agent-contract/outer/gemini-compatible-api.md`를 기준으로 한다.
benchmark 전체 preflight에서는 Claude/agy/Codex에 같은 IOP principal을 secret environment reference로 연결하고 `IOP_BENCH_CONFIG_OBSERVATION_ENV`가 가리키는 operator-owned route/binding observation을 함께 제공한다. 사설 CA 환경은 각 isolated caller child에 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`로 전달된다. preflight가 모든 cell을 `ready`로 판정하기 전에는 scored `run`을 시작하지 않는다.
### Incident redaction check
Before retaining logs or evidence, reject any artifact containing an IOP bearer token, provider credential, slot alias, lease id, certificate private key, keyring material, recipient/issuer private key, target URL with credentials, prompt, or response body. Public metrics may contain only stable safe references such as `credential_slot_ref` and `credential_revision`; request/run/session/attempt/node ids and raw payloads are not credential-attribution labels.

16
packages/go/singlerequesttemplate/template.go
View file

@ -244,10 +244,20 @@ func ParsePlan(tmpl string, rawOutput string, maxOutputBytes int) ([]byte, error
if f3 == "" {
vVerif = rem
} else {
if !strings.HasSuffix(rem, f3) {
return nil, ErrMalformedPlan
suffix := f3
if !strings.HasSuffix(rem, suffix) {
// Provider chat APIs commonly omit the model's final line feed. Treat
// only that last byte as optional; all other static suffix text must
// still match the configured template exactly.
if !strings.HasSuffix(f3, "\n") {
return nil, ErrMalformedPlan
}
suffix = strings.TrimSuffix(f3, "\n")
if !strings.HasSuffix(rem, suffix) {
return nil, ErrMalformedPlan
}
}
vVerif = rem[:len(rem)-len(f3)]
vVerif = rem[:len(rem)-len(suffix)]
}
trimmedGoal := strings.TrimSpace(vGoal)

57
packages/go/singlerequesttemplate/template_test.go
View file

@ -290,6 +290,63 @@ Fix single-request template handling bug.
maxOutputBytes: 1024,
wantErr: false,
},
{
name: "provider omits final line feed",
tmpl: singlerequesttemplate.DefaultPlanTemplate,
raw: strings.TrimSuffix(validOutput, "\n"),
maxOutputBytes: 1024,
wantErr: false,
},
{
name: "provider omits only final line feed after static suffix",
tmpl: `# Plan
## Goal
{{goal}}
## Steps
{{steps}}
## Verification
{{verification}}
END
`,
raw: `# Plan
## Goal
Fix suffix parsing.
## Steps
- Keep the static suffix.
- Allow the final line feed omission.
## Verification
- Run the parser tests.
END`,
maxOutputBytes: 1024,
wantErr: false,
},
{
name: "provider omits static suffix text",
tmpl: `# Plan
## Goal
{{goal}}
## Steps
{{steps}}
## Verification
{{verification}}
END
`,
raw: strings.TrimSuffix(validOutput, "\n"),
maxOutputBytes: 1024,
wantErr: true,
},
{
name: "boundary steps = 6, verif = 3",
tmpl: singlerequesttemplate.DefaultPlanTemplate,

258
scripts/agent_benchmark/agy_iop.py
View file

@ -22,48 +22,48 @@ from scripts.agent_benchmark.connectivity import (
CallerCapability,
ConnectivityResult,
ConnectivityIssue,
EffectiveBinding,
RequestedEffectiveBinding,
classify_issues,
make_result,
)
from scripts.agent_benchmark.lifecycle import (
COMPLETION_EXIT_AFTER_IDLE,
SUBMISSION_STDIN_ONCE,
SUBMISSION_ARGV_TASK,
InvocationResult,
InvocationSpec,
LifecycleMetricError,
ParsedMetric,
SupervisorLocator,
TLS_CA_ENV_KEYS,
count_metric,
duration_metric,
env_pairs,
exact_value_redactor,
is_reported_number,
inherited_tls_ca_environment,
run_invocation,
)
from scripts.agent_benchmark.manifest import MatrixCell, TOKEN_RE, Timeout
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
from scripts.agent_benchmark.workspace import PreparedWorkspace
AGY_CALLER = "agy"
AGY_KNOWN_VERSION = "1.1.11"
AGY_PROVIDER_ENV = "AGY_PROVIDER"
AGY_ENDPOINT_ENV = "AGY_OPENAI_BASE_URL"
AGY_AUTH_ENV = "AGY_OPENAI_API_KEY"
_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE)
_SAFE_EVENT_FIELDS = ("type", "subtype", "model", "effort", "route_kind", "route_id")
_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model", "--effort")
_DOCUMENTED_ENVIRONMENT = (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV)
AGY_SAFE_METRIC_LABELS = ("metric:duration_ms",)
# agy reports whole durations in milliseconds. Only these subtypes are
# converted, and a model-stage duration is marked as overlapping because it is
# reported inside the same window as the total.
AGY_DURATION_METRICS = {
"duration_ms": ("total_duration", False),
"model_duration_ms": ("model_duration", True),
"queue_duration_ms": ("queue_duration", False),
AGY_KNOWN_VERSION = "1.1.12"
AGY_ENDPOINT_ENV = "GOOGLE_GEMINI_BASE_URL"
AGY_AUTH_ENV = "GEMINI_API_KEY"
AGY_MODEL_LABELS = {
"gemini-3.6-flash": "Gemini 3.6 Flash",
"gemini-hybrid": "Gemini 3.6 Flash",
}
_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE)
_SAFE_EVENT_FIELDS = ("event", "state", "step_type", "status")
_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model")
_AGY_USAGE_METRICS = {
"input_tokens": "input_tokens",
"cache_read_tokens": "cached_input_tokens",
"output_tokens": "output_tokens",
"thinking_tokens": "reasoning_tokens",
"total_tokens": "total_tokens",
}
_AGY_METRIC_KEYS = frozenset({"type", "subtype", "value", "model"})
_IDENTITY_RE = re.compile(r"sha256:[0-9a-f]{64}\Z")
@ -162,7 +162,7 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili
return AgyDocumentedCapabilities((), (), ())
return AgyDocumentedCapabilities(
tuple(token for token in _DOCUMENTED_OPTIONS if _exact_token_present(help_output, token)),
tuple(token for token in _DOCUMENTED_ENVIRONMENT if _exact_token_present(help_output, token)),
(),
("stream-json",) if _exact_token_present(help_output, "stream-json") else (),
)
@ -170,9 +170,9 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili
def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapability:
"""Inspect only public, versioned help text for the closed IOP transport.
A version string is accepted only when it names the known agy release and
every required transport variable is documented. This prevents a new or
partially documented client from silently inheriting ambient provider state.
The known release is pinned because its Gemini provider environment is not
printed by ``--help``. The public options and stream format still have to
match exactly; a changed release fails closed until re-qualified.
"""
if not isinstance(version_output, str) or not isinstance(help_output, str):
return AgyCapability(None, False, False, False, False, False, (), ())
@ -180,11 +180,11 @@ def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapa
version = matched.group(1) if matched else None
known_version = version == AGY_KNOWN_VERSION
documented = parse_documented_agy_capabilities(help_output)
endpoint_supported = AGY_ENDPOINT_ENV in documented.environment
auth_supported = AGY_AUTH_ENV in documented.environment
endpoint_supported = known_version
auth_supported = known_version
protocol_supported = known_version and all(
option in documented.options for option in _DOCUMENTED_OPTIONS
) and AGY_PROVIDER_ENV in documented.environment
)
stream_supported = "stream-json" in documented.output_formats
supported = endpoint_supported and auth_supported and protocol_supported and stream_supported
if not supported:
@ -238,7 +238,14 @@ def validate_agy_iop_runtime(
if not all(isinstance(value, str) and value for value in (runtime.binary, runtime.endpoint, runtime.credential)):
raise AgyAdapterError("agy runtime values are unavailable")
endpoint = urlsplit(runtime.endpoint)
if endpoint.scheme not in ("http", "https") or not endpoint.netloc or endpoint.query or endpoint.fragment:
route_path = f"/gemini/{cell.iop.route_id}"
if (
endpoint.scheme != "https"
or not endpoint.netloc
or endpoint.query
or endpoint.fragment
or endpoint.path.rstrip("/") != route_path
):
raise AgyAdapterError("agy IOP endpoint is invalid")
_validate_config_owner_observation(cell, observation)
if observation.endpoint_identity != _runtime_identity("endpoint", runtime.endpoint):
@ -269,7 +276,7 @@ def preflight_agy_iop(
issues.append(_issue("endpoint_incompatible"))
if not runtime.credential:
issues.append(_issue("credential_missing"))
if not cell.iop.request_model:
if cell.iop.request_model not in AGY_MODEL_LABELS:
issues.append(_issue("model_missing"))
if not runtime.endpoint:
issues.append(_issue("endpoint_incompatible"))
@ -307,16 +314,25 @@ def build_agy_invocation(
timeout: Timeout,
preflight: AgyPreflightResult,
) -> InvocationSpec:
"""Build one isolated stdin-only agy invocation after a ready preflight."""
"""Build one isolated official agy print invocation after a ready preflight."""
if preflight.status != "ready" or not preflight.capability.iop_transport_supported or preflight.runtime is None:
raise AgyAdapterError("agy IOP transport is not proven")
runtime = preflight.runtime
if not runtime.binary or not Path(runtime.binary).is_file():
raise AgyAdapterError("agy binary is unavailable")
if not isinstance(prepared, PreparedWorkspace) or not prepared.workspace_dir:
if (
not isinstance(prepared, PreparedWorkspace)
or not Path(prepared.workspace_dir).is_dir()
or not Path(prepared.session_dir).is_dir()
or not Path(prepared.attempt_root).is_dir()
):
raise AgyAdapterError("prepared workspace is unavailable")
if not isinstance(task_payload, bytes) or not task_payload:
raise AgyAdapterError("agy task payload is unavailable")
try:
task_text = task_payload.decode("utf-8")
except UnicodeDecodeError as exc:
raise AgyAdapterError("agy task payload must be UTF-8") from exc
# The child receives a minimal environment and explicit IOP-only provider
# settings. No parent agy/Gemini config or session variable is inherited.
@ -325,33 +341,32 @@ def build_agy_invocation(
"LANG": "C.UTF-8",
"LC_ALL": "C.UTF-8",
"TZ": "UTC",
AGY_PROVIDER_ENV: "iop-openai",
"HOME": prepared.session_dir,
AGY_ENDPOINT_ENV: runtime.endpoint,
AGY_AUTH_ENV: runtime.credential,
}
environment.update(inherited_tls_ca_environment())
return InvocationSpec(
argv=(
runtime.binary,
"--print",
"--sandbox",
"--output-format", "stream-json",
"--model", cell.iop.request_model,
"--effort", cell.iop.requested_effort,
"--model", AGY_MODEL_LABELS[cell.iop.request_model],
"--print", task_text,
),
cwd=prepared.workspace_dir,
env=env_pairs(environment),
env_allowlist=(AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV),
submission_mode=SUBMISSION_STDIN_ONCE,
env_allowlist=(AGY_ENDPOINT_ENV, AGY_AUTH_ENV, *TLS_CA_ENV_KEYS),
submission_mode=SUBMISSION_ARGV_TASK,
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
timeout=timeout,
evidence_dir=prepared.attempt_root,
task_payload=task_payload,
control_dir=str(Path(prepared.attempt_root) / "agy-control"),
)
def _safe_identifier(value: Any) -> str | None:
return value if isinstance(value, str) and TOKEN_RE.fullmatch(value) else None
return value if isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:+-]{0,63}", value) else None
def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> str:
@ -362,115 +377,105 @@ def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> s
return '{"event":"unparseable"}'
if not isinstance(parsed, dict):
return '{"event":"unparseable"}'
safe: dict[str, str] = {}
for field in _SAFE_EVENT_FIELDS:
value = _safe_identifier(parsed.get(field))
event = _safe_identifier(parsed.get("event"))
if event is None or event in sensitive_values:
return '{"event":"unparseable"}'
safe: dict[str, str] = {"event": event}
payload = parsed.get(event)
if not isinstance(payload, dict):
payload = {}
for field in _SAFE_EVENT_FIELDS[1:]:
value = _safe_identifier(payload.get(field))
if value is not None and value not in sensitive_values:
safe[field] = value
if "type" not in safe or "subtype" not in safe:
return '{"event":"unparseable"}'
return json.dumps(safe, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
class AgyEventParser:
"""Strict stream-json parser bound to exactly one requested IOP cell."""
"""Parse the official 1.1.12 stream while trusting binding only from config."""
def __init__(self, cell: MatrixCell) -> None:
def __init__(self, cell: MatrixCell, admitted_binding: RequestedEffectiveBinding) -> None:
if not isinstance(admitted_binding, RequestedEffectiveBinding):
raise AgyAdapterError("agy admitted binding is invalid")
if admitted_binding.cell_id != cell.id or admitted_binding.caller != cell.caller:
raise AgyAdapterError("agy admitted binding mismatch")
self._cell = cell
self._observed_binding: RequestedEffectiveBinding | None = None
self._binding_invalid = False
self._admitted_binding = admitted_binding
self._init_seen = False
self._result_seen = False
self._latest_usage: dict[str, Any] | None = None
def __call__(self, stream: str, raw_line: str) -> str | ParsedMetric | None:
def __call__(self, stream: str, raw_line: str) -> str | ParsedMetric | tuple[Any, ...] | None:
return self.parse(stream, raw_line)
def parse(self, stream: str, raw_line: str) -> str | ParsedMetric | None:
def parse(self, stream: str, raw_line: str) -> str | ParsedMetric | tuple[Any, ...] | None:
if stream != "stdout":
return None
try:
event = json.loads(raw_line)
item = json.loads(raw_line)
except (TypeError, json.JSONDecodeError):
return "malformed"
if not isinstance(event, dict):
if not isinstance(item, dict):
return "malformed"
event_type = event.get("type")
subtype = event.get("subtype")
if (event_type, subtype) == ("iop", "effective_binding"):
self._observe_effective_binding(event)
event = item.get("event")
payload = item.get(event) if isinstance(event, str) else None
if not isinstance(payload, dict):
return "malformed"
if event == "init":
if self._init_seen or self._result_seen:
return "malformed"
self._init_seen = True
return None
if event_type == "metric":
return self._observe_duration(event, subtype)
if event_type == "result" and subtype == "error":
# Quota/provider errors can never be interpreted as finish/idle.
return "quota_error" if event.get("reason") == "quota" else "malformed"
terminal = (
"finish" if (event_type, subtype) == ("result", "success")
else "idle" if (event_type, subtype) == ("system", "idle")
else None
)
if terminal is None or not self._matches_exact_binding(event):
if event == "step_update":
if not self._init_seen or self._result_seen:
return "malformed"
usage = payload.get("usage")
if usage is not None:
if self._usage_metrics(usage) is None:
return "malformed"
self._latest_usage = usage
return None
if event != "result" or not self._init_seen or self._result_seen:
return "malformed"
return terminal
def _observe_duration(self, event: dict[str, Any], subtype: Any) -> str | ParsedMetric:
"""Convert one allowlisted agy duration losslessly, or fail closed."""
mapped = AGY_DURATION_METRICS.get(subtype) if isinstance(subtype, str) else None
value = event.get("value")
if (
mapped is None
or not set(event) <= _AGY_METRIC_KEYS
or not is_reported_number(value)
or ("model" in event and event["model"] != self._cell.iop.request_model)
):
self._result_seen = True
if payload.get("status") != "SUCCESS":
return "malformed"
name, overlap = mapped
metrics: list[ParsedMetric] = []
usage = payload.get("usage", self._latest_usage)
if usage is not None:
parsed_usage = self._usage_metrics(usage)
if parsed_usage is None:
return "malformed"
metrics.extend(parsed_usage)
try:
return duration_metric(
name, value, reported_unit="ms",
model=self._cell.iop.request_model, overlap=overlap,
)
if "duration_seconds" in payload:
metrics.append(duration_metric(
"total_duration", payload["duration_seconds"], reported_unit="s",
model=self._cell.iop.request_model,
))
if "num_turns" in payload:
metrics.append(count_metric(
"model_calls", payload["num_turns"], model=self._cell.iop.request_model,
))
except LifecycleMetricError:
return "malformed"
return tuple(metrics) + ("finish", "idle")
def _matches_exact_binding(self, event: dict[str, Any]) -> bool:
expected = self._cell.iop
return (
event.get("model") == expected.request_model
and event.get("effort") == expected.requested_effort
and event.get("route_kind") == expected.route_kind
and event.get("route_id") == expected.route_id
)
def _observe_effective_binding(self, event: dict[str, Any]) -> None:
expected_keys = {"type", "subtype", "route_kind", "route_id", "model", "effort", "stages"}
if set(event) != expected_keys or self._observed_binding is not None:
self._binding_invalid = True
return
values = tuple(event[key] for key in ("route_kind", "route_id", "model", "effort"))
stages = event.get("stages")
if not all(isinstance(value, str) and TOKEN_RE.fullmatch(value) for value in values) or not isinstance(stages, list):
self._binding_invalid = True
return
parsed_stages: list[EffectiveBinding] = []
for stage in stages:
if not isinstance(stage, dict) or set(stage) != {"stage", "model", "effort"}:
self._binding_invalid = True
return
if not isinstance(stage["stage"], str) or not isinstance(stage["model"], str):
self._binding_invalid = True
return
if stage["effort"] is not None and not isinstance(stage["effort"], str):
self._binding_invalid = True
return
parsed_stages.append(EffectiveBinding(stage["stage"], stage["model"], stage["effort"]))
self._observed_binding = RequestedEffectiveBinding(
self._cell.id, self._cell.caller,
self._cell.iop.route_kind, self._cell.iop.route_id,
self._cell.iop.request_model, self._cell.iop.requested_effort,
values[0], values[1], values[2], values[3], tuple(parsed_stages),
)
def _usage_metrics(self, usage: Any) -> tuple[ParsedMetric, ...] | None:
if not isinstance(usage, dict) or not set(usage) <= set(_AGY_USAGE_METRICS):
return None
metrics: list[ParsedMetric] = []
try:
for wire_name, metric_name in _AGY_USAGE_METRICS.items():
if wire_name in usage:
metrics.append(count_metric(
metric_name, usage[wire_name], model=self._cell.iop.request_model,
))
except LifecycleMetricError:
return None
return tuple(metrics)
def observed_result(self, capability: AgyCapability, lifecycle: InvocationResult) -> ConnectivityResult:
"""Report ready only for successful lifecycle-owned explicit evidence."""
requested = _requested_binding(self._cell)
closed_gap = (_issue("stream_incompatible"),)
caller_capability = CallerCapability(AGY_CALLER, capability.route_kinds, capability.efforts)
@ -478,12 +483,11 @@ class AgyEventParser:
not isinstance(lifecycle, InvocationResult)
or not lifecycle.success
or not lifecycle.finish_then_idle_then_quiet
or self._binding_invalid
or self._observed_binding is None
or not self._result_seen
):
return make_result(self._cell, caller_capability, requested, closed_gap)
try:
return make_result(self._cell, caller_capability, self._observed_binding)
return make_result(self._cell, caller_capability, self._admitted_binding)
except Exception:
return make_result(self._cell, caller_capability, requested, closed_gap)
@ -511,5 +515,5 @@ def run_agy_invocation(
spec,
parse_event=parser,
on_started=on_started,
redact=lambda line: line if line in AGY_SAFE_METRIC_LABELS else structural(exact(line)),
redact=lambda line: structural(exact(line)),
)

443
scripts/agent_benchmark/agy_iop_test.py
View file

@ -1,4 +1,4 @@
"""Credential-free tests for the fail-closed agy IOP adapter."""
"""Credential-free tests for the official agy 1.1.12 IOP adapter."""
from __future__ import annotations
@ -9,25 +9,27 @@ import tempfile
import unittest
from dataclasses import replace
from pathlib import Path
from unittest.mock import patch
from scripts.agent_benchmark.agy_iop import (
AGY_AUTH_ENV,
AGY_ENDPOINT_ENV,
AGY_KNOWN_VERSION,
AGY_PROVIDER_ENV,
AgyAdapterError,
AgyEventParser,
AgyRuntimeInputs,
AgyRuntimeObservation,
_runtime_identity,
build_agy_invocation,
inspect_agy_iop_capability,
preflight_agy_iop,
redact_agy_event,
run_agy_invocation,
)
from scripts.agent_benchmark.connectivity import EffectiveBinding, RequestedEffectiveBinding
from scripts.agent_benchmark.lifecycle import (
REASON_DUPLICATE_EVENT,
REASON_MALFORMED_EVENT,
SUBMISSION_ARGV_TASK,
InvocationSpec,
env_pairs,
)
@ -35,29 +37,24 @@ from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCel
from scripts.agent_benchmark.workspace import AttemptIdentity, PreparedWorkspace, TestbedProvenance
def _help(*, transport: bool = True) -> str:
basic = "--print --output-format stream-json --sandbox --model --effort"
return basic + (f" {AGY_PROVIDER_ENV} {AGY_ENDPOINT_ENV} {AGY_AUTH_ENV}" if transport else "")
def _help() -> str:
return "--print --output-format stream-json --sandbox --model --effort"
def _cell() -> MatrixCell:
return MatrixCell(
"agy-direct", "agy",
IopCell("gemini-2.0-flash", "high", "direct", "agy-direct", (
ExpectedBinding("request", "gemini-2.0-flash", "high"),
IopCell("gemini-3.6-flash", "high", "direct", "agy-direct", (
ExpectedBinding("request", "gemini-3.6-flash", "high"),
)),
)
def _iop_config_observation() -> AgyRuntimeObservation:
"""Fixed evidence from the independent IOP config owner for this cell."""
return AgyRuntimeObservation(
"agy-direct",
"direct",
"agy-direct",
"sha256:feb4c33d4e775c775bfb3c333fdb7d4f97069af31c8e824094fb13181fad53d3",
"sha256:ab1b96f33fc4a662c870f349d92c54bc8e2574028fa41b79526d4edaf6f49daa",
"sha256:" + "c" * 64,
def _binding() -> RequestedEffectiveBinding:
return RequestedEffectiveBinding(
"agy-direct", "agy", "direct", "agy-direct", "gemini-3.6-flash", "high",
"direct", "agy-direct", "gemini-3.6-flash", "high",
(EffectiveBinding("request", "gemini-3.6-flash", "high"),),
)
@ -67,316 +64,198 @@ class AgyIopTest(unittest.TestCase):
self.root = Path(self.temp.name)
self.workspace = self.root / "workspace"
self.workspace.mkdir()
self.runtime = AgyRuntimeInputs(sys.executable, "https://private.invalid/v1", "iop_secret_123456789")
self.session = self.root / "session"
self.session.mkdir()
self.runtime = AgyRuntimeInputs(
sys.executable, "https://private.invalid/gemini/agy-direct", "iop_secret_123456789"
)
def tearDown(self) -> None:
self.temp.cleanup()
def _observation(self, runtime: AgyRuntimeInputs | None = None) -> AgyRuntimeObservation:
value = runtime or self.runtime
return AgyRuntimeObservation(
"agy-direct", "direct", "agy-direct",
_runtime_identity("endpoint", value.endpoint),
_runtime_identity("credential", value.credential),
"sha256:" + "c" * 64,
)
def _prepared(self) -> PreparedWorkspace:
return PreparedWorkspace(
AttemptIdentity("run", "agy-direct", 1, 1), str(self.root), str(self.workspace),
str(self.root / "session"), "fresh-session", True, "sha256:" + "0" * 64,
str(self.session), "fresh-session", True, "sha256:" + "0" * 64,
"isolated", TestbedProvenance("/testbed", "main", "0" * 40, "sha256:" + "1" * 64, True),
"2026-01-01T00:00:00+00:00",
)
def _preflight(self, *, runtime: AgyRuntimeInputs | None = None, help_text: str | None = None):
values = self.runtime if runtime is None else runtime
def _preflight(self, runtime: AgyRuntimeInputs | None = None):
value = runtime or self.runtime
return preflight_agy_iop(
_cell(),
inspect_agy_iop_capability("agy 1.1.11", _help() if help_text is None else help_text),
values,
_iop_config_observation(),
_cell(), inspect_agy_iop_capability("agy 1.1.12", _help()), value,
self._observation(value),
)
def _run_lines(self, lines: list[str], parser: AgyEventParser, preflight):
def _run_lines(self, lines: list[str], parser: AgyEventParser):
evidence = self.root / f"evidence-{len(list(self.root.glob('evidence-*')))}"
evidence.mkdir()
source = "import sys; lines=" + repr(lines) + "; [print(line) for line in lines]"
source = "lines=" + repr(lines) + "; [print(line) for line in lines]"
spec = InvocationSpec(
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
submission_mode="stdin_once", completion_mode="exit_after_idle",
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
)
return run_agy_invocation(spec, parser, preflight, lambda _: None)
return run_agy_invocation(spec, parser, self._preflight(), lambda _: None)
def test_absent_or_unknown_transport_never_constructs_launch(self) -> None:
for version, help_text, expected in (
("agy 1.1.11", _help(transport=False), ("endpoint_incompatible", "auth_incompatible", "protocol_incompatible")),
("agy 9.9.9", _help(), "protocol_incompatible"),
):
with self.subTest(version=version):
preflight = preflight_agy_iop(
_cell(), inspect_agy_iop_capability(version, help_text), self.runtime,
_iop_config_observation(),
)
self.assertEqual(preflight.status, "implementation_gap")
expected_codes = (expected,) if isinstance(expected, str) else expected
self.assertEqual([item.code for item in preflight.issues], list(expected_codes))
with self.assertRaises(AgyAdapterError):
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
def test_installed_public_surface_is_exact_fail_closed_gap(self) -> None:
public_help = "--print --output-format stream-json --sandbox --model --effort"
for transport_name in (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV):
self.assertNotIn(transport_name, public_help)
capability = inspect_agy_iop_capability("1.1.11", public_help)
def test_official_public_surface_is_pinned_without_invented_environment(self) -> None:
capability = inspect_agy_iop_capability("1.1.12", _help())
self.assertEqual(capability.version, AGY_KNOWN_VERSION)
self.assertTrue(capability.stream_supported)
self.assertFalse(capability.endpoint_supported)
self.assertFalse(capability.auth_supported)
self.assertFalse(capability.protocol_supported)
self.assertFalse(capability.iop_transport_supported)
self.assertTrue(capability.iop_transport_supported)
self.assertTrue(capability.endpoint_supported)
self.assertTrue(capability.auth_supported)
self.assertFalse(inspect_agy_iop_capability("1.1.11", _help()).iop_transport_supported)
self.assertFalse(inspect_agy_iop_capability("1.1.12", _help().replace("stream-json", "json")).stream_supported)
preflight = preflight_agy_iop(
_cell(), capability, self.runtime, _iop_config_observation()
)
self.assertEqual(preflight.status, "implementation_gap")
self.assertEqual(
[item.code for item in preflight.issues],
["endpoint_incompatible", "auth_incompatible", "protocol_incompatible"],
)
with self.assertRaises(AgyAdapterError):
build_agy_invocation(
_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight
)
def test_non_ready_preflight_cannot_start_supplied_invocation(self) -> None:
preflight = self._preflight(help_text=_help(transport=False))
self.assertEqual(preflight.status, "implementation_gap")
self.assertIsNotNone(preflight.runtime)
marker = self.root / "caller-launched"
evidence = self.root / "blocked-evidence"
source = "from pathlib import Path; Path(" + repr(str(marker)) + ").write_text('launched')"
spec = InvocationSpec(
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
submission_mode="stdin_once", completion_mode="exit_after_idle",
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
)
started: list[object] = []
with self.assertRaisesRegex(AgyAdapterError, "agy IOP transport is not proven"):
run_agy_invocation(spec, AgyEventParser(_cell()), preflight, started.append)
self.assertFalse(marker.exists())
self.assertEqual(started, [])
self.assertFalse(evidence.exists())
def test_registration_gaps_remain_distinct_from_implementation_gap(self) -> None:
no_credential = AgyRuntimeInputs(sys.executable, self.runtime.endpoint, "")
supported = inspect_agy_iop_capability("agy 1.1.11", _help())
result = preflight_agy_iop(
_cell(), supported, no_credential, _iop_config_observation()
)
self.assertEqual(result.status, "registration_required")
self.assertEqual([item.code for item in result.issues], ["credential_missing"])
gap = preflight_agy_iop(
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help(transport=False)), no_credential,
_iop_config_observation(),
)
self.assertEqual(gap.status, "implementation_gap")
self.assertEqual([item.code for item in gap.issues], ["credential_missing", "endpoint_incompatible", "auth_incompatible", "protocol_incompatible"])
def test_endpoint_auth_and_protocol_gaps_are_exact(self) -> None:
cases = (
(_help().replace(AGY_ENDPOINT_ENV, ""), "endpoint_incompatible"),
(_help().replace(AGY_AUTH_ENV, ""), "auth_incompatible"),
(_help().replace("--sandbox", ""), "protocol_incompatible"),
)
for help_text, expected in cases:
with self.subTest(expected=expected):
outcome = self._preflight(help_text=help_text)
self.assertEqual([item.code for item in outcome.issues], [expected])
unknown = inspect_agy_iop_capability(None, None) # type: ignore[arg-type]
self.assertFalse(unknown.iop_transport_supported)
def test_build_is_fresh_stdin_sandbox_and_iop_only(self) -> None:
preflight = self._preflight()
spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), preflight)
self.assertEqual(spec.submission_mode, "stdin_once")
self.assertIn("--print", spec.argv)
self.assertIn("--sandbox", spec.argv)
self.assertNotIn("--resume", spec.argv)
def test_build_uses_official_gemini_api_key_transport(self) -> None:
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), self._preflight())
environment = dict(spec.env)
self.assertEqual(environment[AGY_PROVIDER_ENV], "iop-openai")
self.assertEqual(environment[AGY_ENDPOINT_ENV], self.runtime.endpoint)
self.assertEqual(environment[AGY_AUTH_ENV], self.runtime.credential)
self.assertEqual(environment["HOME"], str(self.session))
self.assertNotIn("AGY_PROVIDER", environment)
self.assertNotIn("AGY_OPENAI_BASE_URL", environment)
self.assertNotIn("AGY_OPENAI_API_KEY", environment)
self.assertNotIn("--effort", spec.argv)
self.assertEqual(spec.argv[spec.argv.index("--model") + 1], "Gemini 3.6 Flash")
self.assertEqual(spec.argv[-2:], ("--print", "one task"))
self.assertEqual(spec.submission_mode, SUBMISSION_ARGV_TASK)
self.assertEqual(spec.task_payload, b"")
self.assertEqual(environment["SSL_CERT_FILE"], "/operator/dev-ca.pem")
self.assertEqual(environment["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
self.assertEqual(spec.env_allowlist, (AGY_ENDPOINT_ENV, AGY_AUTH_ENV, "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS"))
def test_exact_help_tokens_and_stream_format_gate(self) -> None:
lookalike = _help().replace("--print", "--print-json").replace(
AGY_ENDPOINT_ENV, AGY_ENDPOINT_ENV + "_EXTRA"
).replace("stream-json", "stream-jsonl")
capability = inspect_agy_iop_capability("agy 1.1.11", lookalike)
self.assertFalse(capability.iop_transport_supported)
self.assertFalse(capability.endpoint_supported)
self.assertFalse(capability.stream_supported)
missing_stream = self._preflight(help_text=_help().replace("stream-json", ""))
self.assertEqual([issue.code for issue in missing_stream.issues], ["stream_incompatible"])
def test_build_rejects_missing_prepared_workspace_or_session(self) -> None:
prepared = self._prepared()
for field in ("workspace_dir", "session_dir", "attempt_root"):
missing = prepared.__class__(**{
**prepared.__dict__, field: str(self.root / f"missing-{field}"),
})
with self.subTest(field=field):
with self.assertRaisesRegex(AgyAdapterError, "prepared workspace"):
build_agy_invocation(
_cell(), missing, b"task", Timeout(5, 1, 1, 1), self._preflight()
)
def test_unvalidated_runtime_cannot_launch(self) -> None:
observation = _iop_config_observation()
for mismatched in (
replace(observation, cell_id="other-cell"),
replace(observation, route_id="other-route"),
replace(observation, endpoint_identity="sha256:" + "d" * 64),
replace(observation, config_identity="not-a-config-identity"),
def test_build_rejects_non_utf8_task_for_print_argument(self) -> None:
with self.assertRaisesRegex(AgyAdapterError, "must be UTF-8"):
build_agy_invocation(
_cell(), self._prepared(), b"\xff", Timeout(5, 1, 1, 1), self._preflight()
)
def test_route_qualified_https_runtime_is_required(self) -> None:
for endpoint in (
"http://private.invalid/gemini/agy-direct",
"https://private.invalid/v1",
"https://private.invalid/gemini/other",
):
with self.subTest(observation=mismatched):
preflight = preflight_agy_iop(
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), self.runtime, mismatched
)
self.assertEqual(preflight.status, "implementation_gap")
self.assertIsNone(preflight.runtime)
with self.assertRaises(AgyAdapterError):
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
def test_arbitrary_runtime_cannot_self_issue_iop_proof(self) -> None:
arbitrary = AgyRuntimeInputs(sys.executable, "https://api.openai.com/v1", "unrelated_token_123456789")
preflight = preflight_agy_iop(
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), arbitrary,
_iop_config_observation(),
runtime = replace(self.runtime, endpoint=endpoint)
result = self._preflight(runtime)
self.assertEqual(result.status, "implementation_gap")
self.assertEqual([issue.code for issue in result.issues], ["endpoint_incompatible"])
missing = replace(self.runtime, credential="")
result = preflight_agy_iop(
_cell(), inspect_agy_iop_capability("1.1.12", _help()), missing,
self._observation(self.runtime),
)
self.assertEqual(preflight.status, "implementation_gap")
self.assertIsNone(preflight.runtime)
with self.assertRaises(AgyAdapterError):
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
self.assertEqual(result.status, "registration_required")
self.assertEqual([issue.code for issue in result.issues], ["credential_missing"])
def test_lifecycle_fixture_success_and_metric_preservation(self) -> None:
parser = AgyEventParser(_cell())
def test_unknown_model_and_unvalidated_observation_fail_closed(self) -> None:
unsupported = replace(_cell(), iop=replace(_cell().iop, request_model="gemini-unknown"))
result = preflight_agy_iop(
unsupported, inspect_agy_iop_capability("1.1.12", _help()), self.runtime,
self._observation(),
)
self.assertIn("model_missing", [issue.code for issue in result.issues])
mismatched = replace(self._observation(), endpoint_identity="sha256:" + "d" * 64)
result = preflight_agy_iop(
_cell(), inspect_agy_iop_capability("1.1.12", _help()), self.runtime, mismatched,
)
self.assertIsNone(result.runtime)
with self.assertRaises(AgyAdapterError):
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), result)
def test_official_fixture_completes_and_preserves_metrics(self) -> None:
parser = AgyEventParser(_cell(), _binding())
fixture = Path("scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl")
result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser, self._preflight())
result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser)
self.assertTrue(result.success)
self.assertTrue(result.finish_then_idle_then_quiet)
journal = Path(result.journal_path).read_text(encoding="utf-8")
self.assertIn('"kind": "metric:total_duration"', journal)
self.assertIn('"kind": "metric:model_duration"', journal)
observed = {metric.name: metric for metric in result.metrics}
self.assertEqual(set(observed), {"total_duration", "model_duration"})
self.assertEqual(observed["total_duration"].value, 12 * 10 ** 6)
self.assertFalse(observed["total_duration"].overlap)
# 8.5 ms is preserved exactly; agy's model stage overlaps its total.
self.assertEqual(observed["model_duration"].value, 8_500_000)
self.assertTrue(observed["model_duration"].overlap)
for metric in result.metrics:
self.assertEqual(metric.unit, "ns")
self.assertEqual(metric.clock, "caller_reported")
self.assertEqual(metric.source, "caller_output")
self.assertEqual(metric.model, "gemini-2.0-flash")
capability = inspect_agy_iop_capability("agy 1.1.11", _help())
self.assertEqual(parser.observed_result(capability, result).status, "ready")
def test_only_allowlisted_bound_representable_durations_are_observed(self) -> None:
parser = AgyEventParser(_cell())
rejected = (
{"type": "metric", "subtype": "unknown_ms", "value": 5},
{"type": "metric", "subtype": "duration_ms", "value": "5"},
{"type": "metric", "subtype": "duration_ms", "value": True},
{"type": "metric", "subtype": "duration_ms", "value": -1},
# 0.0000001 ms is 0.1 ns and cannot be represented without invention.
{"type": "metric", "subtype": "duration_ms", "value": 0.0000001},
{"type": "metric", "subtype": "duration_ms", "value": 5, "model": "other"},
{"type": "metric", "subtype": "duration_ms", "value": 5, "extra": 1},
metrics = {metric.name: metric for metric in result.metrics}
self.assertEqual(
set(metrics),
{"total_duration", "model_calls", "input_tokens", "cached_input_tokens", "output_tokens", "reasoning_tokens", "total_tokens"},
)
for event in rejected:
with self.subTest(event=event):
self.assertEqual(parser("stdout", json.dumps(event)), "malformed")
bound = parser("stdout", json.dumps({
"type": "metric", "subtype": "queue_duration_ms", "value": 2,
"model": "gemini-2.0-flash",
}))
self.assertEqual((bound.name, bound.value, bound.overlap), ("queue_duration", 2_000_000, False))
self.assertEqual(metrics["total_duration"].value, 12_000_000)
self.assertEqual(metrics["model_calls"].value, 1)
self.assertEqual(metrics["total_tokens"].value, 12)
capability = inspect_agy_iop_capability("1.1.12", _help())
self.assertEqual(parser.observed_result(capability, result).binding, _binding())
def test_unrepresentable_duration_fails_the_run_without_partial_metric(self) -> None:
parser = AgyEventParser(_cell())
result = self._run_lines(
[json.dumps({"type": "metric", "subtype": "duration_ms", "value": 0.0000001})],
parser, self._preflight(),
)
self.assertFalse(result.success)
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
self.assertEqual(result.metrics, ())
def test_result_must_follow_init_be_unique_and_successful(self) -> None:
result = {
"event": "result", "result": {
"status": "SUCCESS", "duration_seconds": 0.1, "num_turns": 1,
"usage": {"input_tokens": 1}, "response": "secret content",
},
}
for lines, reason in (
([json.dumps(result)], REASON_MALFORMED_EVENT),
([json.dumps({"event": "init", "init": {}}), json.dumps(result), json.dumps(result)], REASON_MALFORMED_EVENT),
([json.dumps({"event": "init", "init": {}}), json.dumps({**result, "result": {**result["result"], "status": "ERROR"}})], REASON_MALFORMED_EVENT),
):
parser = AgyEventParser(_cell(), _binding())
invocation = self._run_lines(lines, parser)
self.assertFalse(invocation.success)
self.assertEqual(invocation.terminal_reason, reason)
def test_metric_prefix_cannot_bypass_durable_redaction(self) -> None:
parser = AgyEventParser(_cell())
raw_lines = [f"metric:{self.runtime.endpoint}", f"metric:{self.runtime.credential}", "metric:not-json"]
result = self._run_lines(raw_lines, parser, self._preflight())
self.assertFalse(result.success)
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
persisted = Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8")
for forbidden in (*raw_lines, self.runtime.endpoint, self.runtime.credential):
self.assertNotIn(forbidden, persisted)
def test_latest_step_usage_is_used_only_when_result_omits_usage(self) -> None:
parser = AgyEventParser(_cell(), _binding())
lines = [
json.dumps({"event": "init", "conversation_id": "c", "init": {"model": "Gemini 3.6 Flash"}}),
json.dumps({"event": "step_update", "step_update": {"state": "DONE", "step_index": 0, "step_type": "agent_response", "text_delta": "private", "usage": {"input_tokens": 2, "output_tokens": 1, "total_tokens": 3}}}),
json.dumps({"event": "result", "result": {"status": "SUCCESS", "duration_seconds": 0.2, "num_turns": 1, "response": "private"}}),
]
invocation = self._run_lines(lines, parser)
self.assertTrue(invocation.success)
self.assertEqual({m.name: m.value for m in invocation.metrics}["total_tokens"], 3)
def test_mismatch_duplicate_and_quota_cannot_pass(self) -> None:
event = {"type": "result", "subtype": "success", "model": "other", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
self.assertEqual(AgyEventParser(_cell())("stdout", json.dumps(event)), "malformed")
parser = AgyEventParser(_cell())
finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
self.assertEqual(parser("stdout", json.dumps(finish)), "finish")
self.assertEqual(parser("stdout", json.dumps(finish)), "finish")
self.assertEqual(AgyEventParser(_cell())("stdout", '{"type":"result","subtype":"error","reason":"quota"}'), "quota_error")
def test_malformed_usage_fails_without_partial_metric(self) -> None:
parser = AgyEventParser(_cell(), _binding())
invocation = self._run_lines([
'{"event":"init","init":{}}',
'{"event":"result","result":{"status":"SUCCESS","usage":{"input_tokens":"1"}}}',
], parser)
self.assertFalse(invocation.success)
self.assertEqual(invocation.terminal_reason, REASON_MALFORMED_EVENT)
self.assertEqual(invocation.metrics, ())
evidence = self.root / "duplicate-evidence"
evidence.mkdir()
source = "import json; event=" + repr(finish) + "; print(json.dumps(event)); print(json.dumps(event))"
spec = InvocationSpec(
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
submission_mode="stdin_once", completion_mode="exit_after_idle",
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
)
duplicate = run_agy_invocation(spec, AgyEventParser(_cell()), self._preflight(), lambda _: None)
self.assertFalse(duplicate.success)
self.assertEqual(duplicate.terminal_reason, REASON_DUPLICATE_EVENT)
def test_structural_redaction_excludes_content_tools_endpoints_and_secrets(self) -> None:
raw = json.dumps({"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "content": "raw prompt", "tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint, "token": self.runtime.credential})
def test_structural_redaction_excludes_response_tools_endpoint_and_secret(self) -> None:
raw = json.dumps({
"event": "result", "result": {
"status": "SUCCESS", "response": "raw prompt",
"tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint,
"token": self.runtime.credential,
},
})
redacted = redact_agy_event(raw, (self.runtime.endpoint, self.runtime.credential))
self.assertEqual(redacted, '{"model":"gemini-2.0-flash","subtype":"success","type":"result"}')
self.assertEqual(redacted, '{"event":"result","status":"SUCCESS"}')
for forbidden in ("raw prompt", "tool_input", self.runtime.endpoint, self.runtime.credential):
self.assertNotIn(forbidden, redacted)
def test_lifecycle_rejects_quota_without_durable_leak(self) -> None:
evidence = self.root / "evidence"
evidence.mkdir()
parser = AgyEventParser(_cell())
secret = self.runtime.credential
endpoint = self.runtime.endpoint
source = "import json; print(json.dumps(" + repr({
"type": "result", "subtype": "error", "reason": "quota",
"content": secret, "endpoint": endpoint,
}) + "))"
spec = InvocationSpec(
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
submission_mode="stdin_once", completion_mode="exit_after_idle",
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
)
result = run_agy_invocation(spec, parser, self._preflight(), lambda _: None)
self.assertFalse(result.success)
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
persisted = (Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8"))
self.assertNotIn(secret, persisted)
self.assertNotIn(endpoint, persisted)
def test_ready_requires_observed_stage_binding_and_successful_lifecycle(self) -> None:
capability = inspect_agy_iop_capability("agy 1.1.11", _help())
finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
idle = {"type": "system", "subtype": "idle", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
binding = {"type": "iop", "subtype": "effective_binding", "route_kind": "direct", "route_id": "agy-direct", "model": "gemini-2.0-flash", "effort": "high", "stages": [{"stage": "request", "model": "gemini-2.0-flash", "effort": "high"}]}
for lines in (
[json.dumps(finish), json.dumps(idle)],
[json.dumps(binding), json.dumps(idle), json.dumps(finish)],
[json.dumps(binding), json.dumps(finish), json.dumps(finish), json.dumps(idle)],
[json.dumps({**binding, "route_id": "other"}), json.dumps(finish), json.dumps(idle)],
):
with self.subTest(lines=lines):
parser = AgyEventParser(_cell())
result = self._run_lines(lines, parser, self._preflight())
self.assertEqual(parser.observed_result(capability, result).status, "implementation_gap")
if __name__ == "__main__":
unittest.main()

30
scripts/agent_benchmark/attempts.py
View file

@ -776,7 +776,18 @@ class RunStore:
if "lifecycle" in record and (not isinstance(record["lifecycle"], dict) or set(record["lifecycle"]) != {"terminal_reason"} or not isinstance(record["lifecycle"]["terminal_reason"], str)):
raise AttemptStateError("attempt lifecycle is invalid")
if record["state"] in TERMINAL_STATES:
self._validate_web_validation(root, run, identity, web_policy)
pre_registration_interrupted = (
record["state"] == "interrupted"
and "locator" not in record
and record.get("lifecycle") == {"terminal_reason": "interrupted"}
)
self._validate_web_validation(
root,
run,
identity,
web_policy,
allow_pre_registration_absence=pre_registration_interrupted,
)
elif (root / WEB_VALIDATION_FILENAME).exists() or (
root / WEB_VALIDATION_FILENAME
).is_symlink():
@ -983,9 +994,12 @@ class RunStore:
def _validate_web_validation(
self, root: Path, run: RunIdentity, identity: AttemptIdentity,
policy: str | None,
*, allow_pre_registration_absence: bool = False,
) -> None:
path = root / WEB_VALIDATION_FILENAME
if not path.exists() and not path.is_symlink():
if allow_pre_registration_absence:
return
if policy == WEB_VALIDATION_POLICY_REQUIRED_V1:
raise AttemptStateError("required web validation is unavailable")
return
@ -1048,6 +1062,20 @@ class RunStore:
occurs before ``attempt.json`` is replaced, leaving recovery resumable.
"""
policy = record.get("web_validation_policy")
if (
policy == WEB_VALIDATION_POLICY_REQUIRED_V1
and record.get("locator") is None
and terminal.get("terminal_reason") == "interrupted"
):
path = root / WEB_VALIDATION_FILENAME
measurement = root / MEASUREMENT_FILENAME
if not (
path.exists()
or path.is_symlink()
or measurement.exists()
or measurement.is_symlink()
):
return
if policy != WEB_VALIDATION_POLICY_REQUIRED_V1:
self._validate_web_validation(root, run, identity, policy)
return

23
scripts/agent_benchmark/attempts_test.py
View file

@ -1128,6 +1128,29 @@ class AttemptMeasurementTest(AttemptBase):
class AttemptWebValidationTest(AttemptBase):
"""Required S12 policy, lifecycle mapping, and recovery-before-terminal."""
def test_pre_registration_interruption_needs_no_impossible_sidecars(self):
run = self.create_run()
with self.store.writer(run):
attempt = self.store.allocate(run, Slot("a", 1))
with self.assertRaisesRegex(ControllerCrash, "before registration"):
self.store.execute_attempt(
attempt,
prepare=lambda _: None,
invoke=lambda _attempt, _started: (_ for _ in ()).throw(
ControllerCrash("before registration")
),
require_measurement=True,
require_web_validation=True,
)
terminal = self.store.reconcile(attempt)
self.assertEqual(terminal.state, "interrupted")
root = Path(terminal.root)
self.assertFalse((root / MEASUREMENT_FILENAME).exists())
self.assertFalse((root / WEB_VALIDATION_FILENAME).exists())
self.assertEqual(
self.store.status(run, self.manifest)["attempts"]["interrupted"], 1
)
def _run(self, mode: str = "success"):
self._init_testbed()
run = self.create_run()

62
scripts/agent_benchmark/claude_iop.py
View file

@ -28,9 +28,11 @@ from scripts.agent_benchmark.lifecycle import (
InvocationSpec,
LifecycleMetricError,
ParsedMetric,
TLS_CA_ENV_KEYS,
count_metric,
duration_metric,
exact_value_redactor,
inherited_tls_ca_environment,
is_reported_number,
)
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
@ -221,6 +223,8 @@ class ClaudeStreamParser:
self.claude_session_id: str | None = None
self._phase = "await_init"
self._assistant_messages = 0
self._active_message_id: str | None = None
self._completed_message_ids: set[str] = set()
def _require_bound_session(self, event: dict[str, Any]) -> None:
if self.claude_session_id is None:
@ -241,14 +245,59 @@ class ClaudeStreamParser:
raise ClaudeIopProtocolError("duplicate or out-of-order Claude assistant")
self._require_bound_session(event)
message = event.get("message")
if not isinstance(message, dict) or message.get("stop_reason") != "end_turn":
raise ClaudeIopProtocolError("invalid Claude assistant terminal")
if not isinstance(message, dict):
raise ClaudeIopProtocolError("invalid Claude assistant event")
if _required_string(message, "model") != self.cell.iop.request_model:
raise ClaudeIopProtocolError("Claude model binding mismatch")
stop_reason = message.get("stop_reason")
if stop_reason not in (None, "tool_use", "end_turn"):
raise ClaudeIopProtocolError("invalid Claude assistant stop reason")
message_id = message.get("id")
if message_id is not None and (not isinstance(message_id, str) or not message_id):
raise ClaudeIopProtocolError("invalid Claude assistant message id")
if stop_reason is None:
if message_id is None:
raise ClaudeIopProtocolError("unbound Claude assistant snapshot")
if message_id in self._completed_message_ids:
raise ClaudeIopProtocolError("duplicate Claude assistant message")
if self._active_message_id not in (None, message_id):
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
self._active_message_id = message_id
return None
if message_id is None:
# Older fixture-shaped output did not include a message id. It is
# admissible only for the single final assistant event.
if stop_reason != "end_turn" or self._active_message_id is not None:
raise ClaudeIopProtocolError("unbound Claude assistant terminal")
else:
if message_id in self._completed_message_ids:
raise ClaudeIopProtocolError("duplicate Claude assistant message")
if self._active_message_id not in (None, message_id):
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
self._completed_message_ids.add(message_id)
self._active_message_id = None
self._assistant_messages += 1
if stop_reason == "tool_use":
self._phase = "await_tool_result"
return None
self._phase = "await_result"
return "finish"
def _consume_user(self, event: dict[str, Any]) -> None:
if self._phase not in ("await_assistant", "await_tool_result"):
raise ClaudeIopProtocolError("out-of-order Claude user event")
self._require_bound_session(event)
if self._active_message_id is not None:
if self._active_message_id in self._completed_message_ids:
raise ClaudeIopProtocolError("duplicate Claude assistant message")
self._completed_message_ids.add(self._active_message_id)
self._active_message_id = None
self._assistant_messages += 1
elif self._phase != "await_tool_result":
raise ClaudeIopProtocolError("unexpected Claude user event")
self._phase = "await_assistant"
def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...]:
if self._phase != "await_result":
raise ClaudeIopProtocolError("duplicate or out-of-order Claude result")
@ -301,6 +350,9 @@ class ClaudeStreamParser:
return None
if event_type == "assistant":
return self._consume_assistant(event)
if event_type == "user":
self._consume_user(event)
return None
if event_type == "result":
return self._consume_result(event)
# Informational events are deliberately ignored only after they have
@ -371,19 +423,21 @@ class ClaudeIopAdapter:
("ANTHROPIC_API_KEY", self.api_key),
("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "1"),
("CLAUDE_CODE_DISABLE_AUTOUPDATER", "1"),
)
) + inherited_tls_ca_environment()
return InvocationSpec(
argv=(
self.binary, "--bare", "--print", "--verbose",
"--input-format", "text", "--output-format", "stream-json",
"--model", self.cell.iop.request_model, "--effort", self.cell.iop.requested_effort,
"--no-session-persistence", "--permission-mode", "dontAsk", "--tools=",
"--no-session-persistence", "--permission-mode", "dontAsk",
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
),
cwd=str(cwd),
env=env,
env_allowlist=(
"ANTHROPIC_BASE_URL", "ANTHROPIC_API_KEY",
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "CLAUDE_CODE_DISABLE_AUTOUPDATER",
*TLS_CA_ENV_KEYS,
),
submission_mode=SUBMISSION_STDIN_ONCE,
completion_mode=COMPLETION_EXIT_AFTER_IDLE,

65
scripts/agent_benchmark/claude_iop_test.py
View file

@ -8,6 +8,7 @@ import tempfile
import textwrap
import unittest
from pathlib import Path
from unittest.mock import patch
from scripts.agent_benchmark.claude_iop import (
ClaudeIopAdapter,
@ -124,18 +125,22 @@ class ClaudeIopTest(unittest.TestCase):
def test_exact_iop_only_invocation_and_fresh_workspace(self) -> None:
adapter = self._adapter()
spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1))
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1))
self.assertEqual(spec.cwd, self.workspace.workspace_dir)
self.assertEqual(spec.submission_mode, "stdin_once")
self.assertEqual(spec.task_payload, SENTINELS[0].encode())
self.assertEqual(spec.argv[1:], (
"--bare", "--print", "--verbose", "--input-format", "text",
"--output-format", "stream-json", "--model", "claude-sonnet", "--effort", "high",
"--no-session-persistence", "--permission-mode", "dontAsk", "--tools=",
"--no-session-persistence", "--permission-mode", "dontAsk",
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
))
env = dict(spec.env)
self.assertEqual(env["ANTHROPIC_BASE_URL"], SENTINELS[1])
self.assertEqual(env["ANTHROPIC_API_KEY"], SENTINELS[2])
self.assertEqual(env["SSL_CERT_FILE"], "/operator/dev-ca.pem")
self.assertEqual(env["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
self.assertNotIn("ANTHROPIC_AUTH_TOKEN", env)
self.assertNotIn("CLAUDE_CONFIG_DIR", env)
(self.root / "other").mkdir()
@ -215,6 +220,62 @@ class ClaudeIopTest(unittest.TestCase):
parser = ClaudeStreamParser(self.cell, "session-fixture")
self.assertEqual([parser("stdout", event) for event in missing_result], [None, "finish"])
def test_parser_accepts_partial_snapshots_and_tool_result_cycles(self) -> None:
init, _, result = self._fixture_lines()
parser = ClaudeStreamParser(self.cell, "session-fixture")
partial = json.dumps({
"type": "assistant", "session_id": "claude-session-fixture",
"message": {"id": "msg-tool-1", "model": "claude-sonnet", "stop_reason": None,
"content": [{"type": "tool_use", "id": "tool-1"}]},
})
user = json.dumps({
"type": "user", "session_id": "claude-session-fixture",
"message": {"content": [{"type": "tool_result", "tool_use_id": "tool-1"}]},
})
final = json.dumps({
"type": "assistant", "session_id": "claude-session-fixture",
"message": {"id": "msg-final", "model": "claude-sonnet",
"stop_reason": "end_turn", "content": []},
})
self.assertEqual(
[parser("stdout", line) for line in (init, partial, partial, user, final)],
[None, None, None, None, "finish"],
)
observations = parser("stdout", result)
metrics = {metric.name: metric.value for metric in observations[:-1]}
self.assertEqual(metrics["model_calls"], 2)
def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None:
init, _, result = self._fixture_lines()
parser = ClaudeStreamParser(self.cell, "session-fixture")
tool_use = json.dumps({
"type": "assistant", "session_id": "claude-session-fixture",
"message": {"id": "msg-tool-1", "model": "claude-sonnet",
"stop_reason": "tool_use", "content": []},
})
user = json.dumps({
"type": "user", "session_id": "claude-session-fixture", "message": {"content": []},
})
final = json.dumps({
"type": "assistant", "session_id": "claude-session-fixture",
"message": {"id": "msg-final", "model": "claude-sonnet",
"stop_reason": "end_turn", "content": []},
})
self.assertEqual(
[parser("stdout", line) for line in (init, tool_use, user, final)],
[None, None, None, "finish"],
)
self.assertEqual(
{metric.name: metric.value for metric in parser("stdout", result)[:-1]}["model_calls"],
2,
)
duplicate = ClaudeStreamParser(self.cell, "session-fixture")
duplicate("stdout", init)
duplicate("stdout", tool_use)
duplicate("stdout", user)
with self.assertRaises(ClaudeIopProtocolError):
duplicate("stdout", tool_use)
def test_reported_result_values_become_bound_normalized_observations(self) -> None:
init, assistant, result = self._fixture_lines()
parser = ClaudeStreamParser(self.cell, "session-fixture")

30
scripts/agent_benchmark/codex_iop.py
View file

@ -37,10 +37,12 @@ from scripts.agent_benchmark.lifecycle import (
LifecycleValidationError,
ParsedMetric,
SupervisorLocator,
TLS_CA_ENV_KEYS,
count_metric,
duration_metric,
env_pairs,
exact_value_redactor,
inherited_tls_ca_environment,
is_reported_number,
run_invocation,
)
@ -77,12 +79,16 @@ _SAFE_STRING_KEYS = frozenset({
_CODEX_USAGE_FIELDS = {
"input_tokens": "input_tokens",
"cached_input_tokens": "cached_input_tokens",
"cache_write_input_tokens": "cache_write_tokens",
"output_tokens": "output_tokens",
"reasoning_output_tokens": "reasoning_tokens",
"total_tokens": "total_tokens",
}
_CODEX_ITEM_COMPLETED = "item.completed"
_CODEX_CALL_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.:-]{0,63}$")
_CODEX_TOOL_ITEM_TYPES = frozenset({
"command_execution", "file_change", "mcp_tool_call", "web_search",
})
class CodexIOPError(Exception):
@ -219,7 +225,7 @@ def build_codex_spec(
raise CodexRuntimeError("invalid Codex executable")
codex_argv: list[str] = [
*executable_argv, "exec", "--json", "--ephemeral", "--ignore-user-config",
*executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
"--strict-config", "--skip-git-repo-check", "-C", prepared.workspace_dir,
"-m", cell.iop.request_model,
]
@ -232,8 +238,13 @@ def build_codex_spec(
# import resolution in the child bridge.
argv=(sys.executable, str(Path(__file__).resolve()), "--bridge", "--", *codex_argv),
cwd=prepared.workspace_dir,
env=env_pairs({"PATH": runtime.path, "HOME": prepared.session_dir, SECRET_ENV_KEY: runtime.api_key}),
env_allowlist=(SECRET_ENV_KEY,),
env=env_pairs({
"PATH": runtime.path,
"HOME": prepared.session_dir,
SECRET_ENV_KEY: runtime.api_key,
**dict(inherited_tls_ca_environment()),
}),
env_allowlist=(SECRET_ENV_KEY, *TLS_CA_ENV_KEYS),
submission_mode=SUBMISSION_STDIN_ONCE,
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
timeout=timeout,
@ -346,17 +357,20 @@ class CodexJSONLParser:
return tuple(observations)
def _tool_interval(self, record: dict[str, Any]) -> ParsedMetric | None:
"""Observe one explicitly paired tool interval; never infer a pairing."""
"""Count a completed tool and optionally observe its reported interval."""
item = record.get("item")
if not isinstance(item, dict) or "duration_ms" not in item:
if not isinstance(item, dict) or item.get("type") not in _CODEX_TOOL_ITEM_TYPES:
return None
call_id = item.get("id")
if not isinstance(call_id, str) or _CODEX_CALL_ID_RE.fullmatch(call_id) is None:
raise CodexJSONLError("unpaired Codex tool interval")
raise CodexJSONLError("unpaired Codex tool completion")
if call_id in self._tool_calls:
raise CodexJSONLError("duplicate Codex tool completion")
if "duration_ms" not in item:
self._tool_calls.add(call_id)
return None
if not is_reported_number(item["duration_ms"]):
raise CodexJSONLError("invalid Codex tool interval")
if call_id in self._tool_calls:
raise CodexJSONLError("duplicate Codex tool interval")
try:
observation = duration_metric(
"tool_duration", item["duration_ms"], reported_unit="ms",

37
scripts/agent_benchmark/codex_iop_test.py
View file

@ -8,6 +8,7 @@ import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from scripts.agent_benchmark.codex_iop import (
BASE_URL_ENV_KEY,
@ -95,13 +96,14 @@ class CodexIOPTest(unittest.TestCase):
return str(path)
def test_exact_isolated_responses_spec_uses_one_stdin_submission(self) -> None:
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
argv = list(spec.argv)
codex = argv[argv.index("--") + 1:]
self.assertFalse((self.workspace / ".git").exists())
self.assertEqual(codex.count("--skip-git-repo-check"), 1)
self.assertEqual(codex[:12], [
"codex", "exec", "--json", "--ephemeral", "--ignore-user-config",
self.assertEqual(codex[:14], [
"codex", "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
"--strict-config", "--skip-git-repo-check", "-C", str(self.workspace),
"-m", "gpt-5.6-luna", "-c",
])
@ -118,6 +120,8 @@ class CodexIOPTest(unittest.TestCase):
self.assertEqual(spec.submission_mode, "stdin_once")
self.assertEqual(spec.completion_mode, "exit_after_idle")
self.assertEqual(dict(spec.env)[SECRET_ENV_KEY], _SECRET)
self.assertEqual(dict(spec.env)["SSL_CERT_FILE"], "/operator/dev-ca.pem")
self.assertEqual(dict(spec.env)["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
self.assertNotIn(BASE_URL_ENV_KEY, dict(spec.env))
self.assertNotIn("OPENAI_API_KEY", dict(spec.env))
@ -142,8 +146,9 @@ class CodexIOPTest(unittest.TestCase):
self.assertEqual(events[2][-1], "finish")
turn = {metric.name: metric.value for metric in events[2][:-1]}
self.assertEqual(turn, {
"cached_input_tokens": 8, "input_tokens": 31, "output_tokens": 12,
"reasoning_tokens": 4, "model_calls": 1, "tool_calls": 1,
"cache_write_tokens": 6, "cached_input_tokens": 8,
"input_tokens": 31, "output_tokens": 12, "reasoning_tokens": 4,
"model_calls": 1, "tool_calls": 1,
})
# The fixture omits the provider total, so it is never reconstructed
# from the reported categories.
@ -197,25 +202,31 @@ class CodexIOPTest(unittest.TestCase):
def test_tool_intervals_require_one_explicit_unique_pairing(self) -> None:
parser = CodexJSONLParser(_cell(), "0123456789abcdef")
# An item without a reported duration carries no interval to pair.
# A non-tool item carries neither a tool count nor an interval.
self.assertIsNone(parser.parse("stdout", json.dumps(
{"type": "item.completed", "item": {"type": "agent_message", "text": "x"}}
)))
paired = json.dumps({"type": "item.completed", "item": {"id": "call-1", "duration_ms": 3}})
# Real Codex command completions omit duration; they must still count.
self.assertIsNone(parser.parse("stdout", json.dumps(
{"type": "item.completed", "item": {"id": "call-0", "type": "command_execution"}}
)))
paired = json.dumps({"type": "item.completed", "item": {
"id": "call-1", "type": "command_execution", "duration_ms": 3,
}})
self.assertEqual(parser.parse("stdout", paired).call_id, "call-1")
for name, item in {
"duplicate-call": {"id": "call-1", "duration_ms": 4},
"unpaired-duration": {"duration_ms": 4},
"unsafe-call-id": {"id": "call 1", "duration_ms": 4},
"string-duration": {"id": "call-2", "duration_ms": "4"},
"negative-duration": {"id": "call-3", "duration_ms": -4},
"duplicate-call": {"id": "call-1", "type": "command_execution", "duration_ms": 4},
"unpaired-duration": {"type": "command_execution", "duration_ms": 4},
"unsafe-call-id": {"id": "call 1", "type": "command_execution", "duration_ms": 4},
"string-duration": {"id": "call-2", "type": "command_execution", "duration_ms": "4"},
"negative-duration": {"id": "call-3", "type": "command_execution", "duration_ms": -4},
}.items():
with self.subTest(name=name):
with self.assertRaises(CodexJSONLError):
parser.parse("stdout", json.dumps({"type": "item.completed", "item": item}))
turn = parser.parse("stdout", json.dumps({"type": "turn.completed"}))
counts = {metric.name: metric.value for metric in turn[:-1]}
self.assertEqual(counts, {"model_calls": 1, "tool_calls": 1})
self.assertEqual(counts, {"model_calls": 1, "tool_calls": 2})
def test_unknown_or_fractional_turn_usage_fails_closed(self) -> None:
for usage in (

232
scripts/agent_benchmark/connectivity_integration_test.py
View file

@ -43,7 +43,7 @@ from scripts.agent_benchmark.connectivity import (
canonical_evidence_bytes,
make_result,
)
from scripts.agent_benchmark.codex_iop import CodexInvocationResult
from scripts.agent_benchmark.codex_iop import CodexInvocation, CodexInvocationResult
from scripts.agent_benchmark.manifest import (
AssetMapping,
ExpectedBinding,
@ -201,15 +201,17 @@ if name == "claude":
"result": leak},
]
elif name == "agy":
binding = {"route_kind": "direct", "route_id": route_id,
"model": option("--model"), "effort": option("--effort")}
events = [
{"type": "metric", "subtype": "duration_ms", "value": 12.5},
dict(binding, type="iop", subtype="effective_binding",
stages=[{"stage": "request", "model": binding["model"],
"effort": binding["effort"]}]),
dict(binding, type="result", subtype="success", text=leak),
dict(binding, type="system", subtype="idle"),
{"event": "init", "conversation_id": "fixture",
"init": {"model": "Gemini 3.6 Flash", "permission_mode": "sandbox", "tools": []}},
{"event": "step_update", "step_update": {
"state": "DONE", "step_index": 0, "step_type": "agent_response",
"text_delta": leak,
"usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}},
{"event": "result", "result": {
"conversation_id": "fixture", "status": "SUCCESS",
"duration_seconds": 0.0125, "num_turns": 1, "response": leak,
"usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}},
]
else:
events = [
@ -218,10 +220,8 @@ else:
"item": {"id": "call-1", "type": "command_execution",
"duration_ms": 7.25, "output": leak}},
{"type": "turn.completed", "status": "completed",
"usage": {"input_tokens": 31, "cached_input_tokens": 8, "output_tokens": 12},
"iop_effective_binding": {
"route_kind": "direct", "route_id": route_id, "model": option("-m"),
"effort": override("model_reasoning_effort=")}},
"usage": {"input_tokens": 31, "cached_input_tokens": 8,
"cache_write_input_tokens": 6, "output_tokens": 12}},
]
for event in events:
@ -466,6 +466,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
for caller in ("CLAUDE", "AGY", "CODEX"):
environment[f"IOP_BENCH_{caller}_BASE_URL"] = "http://127.0.0.1:18083/v1"
environment[f"IOP_BENCH_{caller}_SECRET_ENV"] = "BENCH_TOKEN"
environment["IOP_BENCH_AGY_BASE_URL"] = "https://127.0.0.1:18083"
return environment
@staticmethod
@ -692,6 +693,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
environment[secret_env] = branch["secret"]
environment[prefix + "SECRET_ENV"] = secret_env
environment[prefix + "BASE_URL"] = f"http://{branch['endpoint']}.invalid:18083/v1"
if caller == "agy":
environment[prefix + "BASE_URL"] = f"https://{branch['endpoint']}.invalid:18083"
return environment
@staticmethod
@ -825,10 +828,13 @@ class ConnectivityIntegrationTest(unittest.TestCase):
"total_duration", "model_duration", "model_calls", "input_tokens",
"output_tokens", "cached_input_tokens",
},
"agy": {"total_duration"},
"agy": {
"total_duration", "model_calls", "input_tokens", "output_tokens",
"total_tokens",
},
"codex": {
"model_calls", "tool_calls", "input_tokens", "output_tokens",
"cached_input_tokens",
"cached_input_tokens", "cache_write_tokens",
},
}
@ -849,9 +855,13 @@ class ConnectivityIntegrationTest(unittest.TestCase):
if item.status == "observed"
}
self.assertEqual(observed, self._EXPECTED_TOTALS[caller], attempt_root)
# No caller reports a provider total, and none is reconstructed.
self.assertEqual(measurement.usage["total_tokens"].status, "unavailable")
self.assertIsNone(measurement.usage["total_tokens"].value)
# Official agy reports total_tokens; other callers in this fixture
# do not, and no total is reconstructed for them.
if caller == "agy":
self.assertEqual(measurement.usage["total_tokens"].value, 33)
else:
self.assertEqual(measurement.usage["total_tokens"].status, "unavailable")
self.assertIsNone(measurement.usage["total_tokens"].value)
self.assertEqual(measurement.timeline["first_output_at"].status, "observed")
self.assertEqual(
measurement.timeline["first_write_mtime"].clock, "filesystem_mtime"
@ -911,7 +921,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
root.mkdir()
matrix = [
_cell("claude", "claude", "sonnet", "max"),
_cell("agy", "agy", "gemini", "high"),
_cell("agy", "agy", "gemini-3.6-flash", "high"),
_cell("codex", "codex", "gpt", "xhigh"),
]
sentinels = _branch_sentinels()
@ -932,7 +942,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
tuple(sorted(cell.iop.request_model for cell in manifest.matrix)),
"sha256:" + "f" * 64,
True,
"agy 1.1.11",
"agy 1.1.12",
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
)
@ -1058,6 +1068,48 @@ class ConnectivityIntegrationTest(unittest.TestCase):
len(manifest.matrix),
)
def test_cli_failed_attempt_does_not_prevent_later_slot(self) -> None:
self._init_testbed()
manifest, _, path = _write_manifest(
self.root,
[
_cell("agy-first", "agy", "gemini-3.6-flash", "high"),
_cell("codex-later", "codex", "gpt-5.6-luna", "xhigh"),
],
output_id="terminal-failure-continues",
)
agy = FakeAdapter("agy", ("high",))
agy.fail_invocation = True
codex = FakeAdapter("codex", ("xhigh",))
self.addCleanup(agy.cleanup)
self.addCleanup(codex.cleanup)
stdout = io.StringIO()
stderr = io.StringIO()
with (
mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root),
mock.patch.object(
benchmark_cli,
"build_adapter_registry",
return_value={"agy": agy, "codex": codex},
),
contextlib.redirect_stdout(stdout),
contextlib.redirect_stderr(stderr),
):
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
self.assertEqual(exit_code, 69)
self.assertEqual(stdout.getvalue(), "")
match = re.search(r"run_id=(run-[0-9A-Za-z-]+)", stderr.getvalue())
self.assertIsNotNone(match)
self.assertEqual([item[0] for item in agy.invocations], ["agy-first"])
self.assertEqual([item[0] for item in codex.invocations], ["codex-later"])
status = self.store.status(manifest=manifest, run=self.store.open(
manifest, match.group(1) # type: ignore[union-attr]
))
self.assertEqual(status["attempts"]["failed"], 1)
self.assertEqual(status["attempts"]["success"], 1)
self.assertEqual(status["attempts"]["running"], 0)
def test_cli_resume_retries_append_only_and_status_is_read_only(self) -> None:
self._init_testbed()
manifest, _, path = _write_manifest(
@ -1301,7 +1353,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)),
"sha256:" + "c" * 64,
True,
"agy 1.1.11",
"agy 1.1.12",
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
)
@ -1372,6 +1424,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
)
captured = {}
caller_binding = {"value": None}
def invoke(invocation, _on_started):
captured["spec"] = invocation.spec
stream = CaptureStream("stdout", "", 0, 0, False)
@ -1382,13 +1436,9 @@ class ConnectivityIntegrationTest(unittest.TestCase):
"2026-08-11T00:00:00+00:00",
"2026-08-11T00:00:01+00:00", 1, (),
)
binding = (
evaluator.iop.route_kind,
evaluator.iop.route_id,
evaluator.iop.request_model,
evaluator.iop.requested_effort,
return CodexInvocationResult(
lifecycle, caller_binding["value"]
)
return CodexInvocationResult(lifecycle, binding)
adapter = live_iop.build_live_scoring_adapter(
environment,
@ -1427,10 +1477,45 @@ class ConnectivityIntegrationTest(unittest.TestCase):
lambda *_args: None,
)
self.assertTrue(result.success)
self.assertEqual(result.effective_binding[0], route_kind)
expected_binding = (
evaluator.iop.route_kind,
evaluator.iop.route_id,
evaluator.iop.request_model,
evaluator.iop.requested_effort,
)
self.assertEqual(result.effective_binding, expected_binding)
finalized = adapter.finalize_evidence(blind)
self.assertTrue(finalized.safe)
caller_binding["value"] = (
evaluator.iop.route_kind,
"contradictory-route",
evaluator.iop.request_model,
evaluator.iop.requested_effort,
)
mismatch_root = self.root / "runs" / f"blind-{route_kind}-mismatch"
for name in ("input", "session", "output"):
(mismatch_root / name).mkdir(parents=True, exist_ok=True)
mismatch_blind = BlindWorkspace(
f"blind-{route_kind}-mismatch", str(mismatch_root),
str(mismatch_root / "input"), str(mismatch_root / "session"),
str(mismatch_root / "output"), "sha256:" + "e" * 64,
"sha256:" + "f" * 64,
)
mismatch = adapter.invoke(
evaluator,
mismatch_blind,
prompt,
self.manifest.timeout,
lambda *_args: None,
)
self.assertFalse(mismatch.success)
self.assertEqual(mismatch.terminal_reason, "binding_mismatch")
self.assertEqual(
mismatch.effective_binding, caller_binding["value"]
)
self.assertTrue(adapter.finalize_evidence(mismatch_blind).safe)
spec = captured["spec"]
visible = "\n".join(
(*spec.argv, spec.cwd, *(value for pair in spec.env for value in pair))
@ -2367,7 +2452,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
]
no_model = {**environment, "BENCH_CONFIG": json.dumps({"schema_version": "1", "routes": no_model_routes})}
unsupported = replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy"), iop=replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy").iop, requested_effort="max"))
observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.11", "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json")
observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.12", "--print --output-format --sandbox --model --effort stream-json")
checks = (
(live_iop.build_live_adapter_registry(no_route, observer=observed)["claude"].preflight(claude).result, "route_missing"),
(live_iop.build_live_adapter_registry(no_model, observer=observed)["claude"].preflight(claude).result, "model_missing"),
@ -2388,12 +2473,12 @@ class ConnectivityIntegrationTest(unittest.TestCase):
(("effort_unsupported", ISSUE_RESUME_CODES["effort_unsupported"]),),
)
def test_live_invocation_rejects_missing_or_mismatched_caller_binding(self) -> None:
def test_live_invocation_uses_config_owned_codex_binding_and_rejects_mismatch(self) -> None:
environment = self._live_environment()
observed = lambda _runtime: live_iop._Observation(
tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)),
"sha256:" + "2" * 64, True, "agy 1.1.11",
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
"sha256:" + "2" * 64, True, "agy 1.1.12",
"--print --output-format --sandbox --model --effort stream-json",
)
registry = live_iop.build_live_adapter_registry(
environment, observer=observed, binary_resolver=lambda _name: "/bin/true"
@ -2410,11 +2495,6 @@ class ConnectivityIntegrationTest(unittest.TestCase):
lambda *_args: None,
live_iop._DEFAULT_INVOKERS.codex,
)
codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
live_iop._DEFAULT_INVOKERS.claude,
live_iop._DEFAULT_INVOKERS.agy,
lambda *_args: type("Mismatch", (), {"effective_binding": None, "lifecycle": None})(),
)
with mock.patch.object(live_iop, "build_agy_invocation", return_value=object()):
with self.assertRaises(live_iop.LiveIopError) as raised:
agy_adapter.invoke(
@ -2422,7 +2502,74 @@ class ConnectivityIntegrationTest(unittest.TestCase):
self.manifest.timeout, lambda *_args: None,
)
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
with mock.patch.object(live_iop, "build_codex_invocation", return_value=type("Invocation", (), {"spec": object()})()):
failed_stream = CaptureStream("stdout", "", 0, 0, False)
failed_lifecycle = InvocationResult(
False, "nonzero_exit", 1, None, True, False, True, False,
(), failed_stream, replace(failed_stream, stream="stderr"), "", "", None,
"sha256:" + "9" * 64,
"2026-08-11T00:00:00+00:00",
"2026-08-11T00:00:01+00:00", 1, (),
)
agy_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
live_iop._DEFAULT_INVOKERS.claude,
lambda *_args: failed_lifecycle,
live_iop._DEFAULT_INVOKERS.codex,
)
with (
mock.patch.object(live_iop, "build_agy_invocation", return_value=object()),
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
):
failed = agy_adapter.invoke(
agy, object(), object(), "/tmp/control", b"task",
self.manifest.timeout, lambda *_args: None,
)
self.assertIs(failed, failed_lifecycle)
stream = CaptureStream("stdout", "", 0, 0, False)
lifecycle = InvocationResult(
True, "success", 0, None, True, True, True, False,
(), stream, replace(stream, stream="stderr"), "", "", None,
"sha256:" + "a" * 64,
"2026-08-11T00:00:00+00:00",
"2026-08-11T00:00:01+00:00", 1, (),
)
invocation = CodexInvocation(
InvocationSpec(
argv=("codex",), cwd="/tmp", env=(),
submission_mode=SUBMISSION_STDIN_ONCE,
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
timeout=self.manifest.timeout, evidence_dir="/tmp",
),
None, # type: ignore[arg-type]
lambda line: line,
)
observed_binding = {"value": None}
def invoke_codex(*_args):
return CodexInvocationResult(lifecycle, observed_binding["value"])
codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
live_iop._DEFAULT_INVOKERS.claude,
live_iop._DEFAULT_INVOKERS.agy,
invoke_codex,
)
with (
mock.patch.object(live_iop, "build_codex_invocation", return_value=invocation),
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
):
result = codex_adapter.invoke(
codex, object(), object(), "/tmp/control", b"task",
self.manifest.timeout, lambda *_args: None,
)
self.assertIs(result, lifecycle)
observed_binding["value"] = (
codex.iop.route_kind,
"contradictory-route",
codex.iop.request_model,
codex.iop.requested_effort,
)
with self.assertRaises(live_iop.LiveIopError) as raised:
codex_adapter.invoke(
codex, object(), object(), "/tmp/control", b"task",
@ -2434,8 +2581,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
manifest, _, _ = _write_manifest(
self.root,
[
_cell("agy-direct", "agy", "gemini-direct", "high"),
_preset("agy-preset", "agy", "gemini-preset", "high"),
_cell("agy-direct", "agy", "gemini-3.6-flash", "high"),
_preset("agy-preset", "agy", "gemini-hybrid", "high"),
],
output_id="agy-cell-state",
)
@ -2447,9 +2594,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
observed_models,
"sha256:" + "2" * 64,
True,
"agy 1.1.11",
"--print --output-format --sandbox --model --effort "
"AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
"agy 1.1.12",
"--print --output-format --sandbox --model --effort stream-json",
)
adapter = live_iop.build_live_adapter_registry(
environment,

57
scripts/agent_benchmark/lifecycle.py
View file

@ -18,6 +18,7 @@ from __future__ import annotations
import ctypes
import datetime
import errno
import hashlib
import hmac
import json
@ -28,6 +29,7 @@ import secrets
import shutil
import signal
import socket
import stat
import subprocess
import sys
import tempfile
@ -36,7 +38,7 @@ import time
from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
from pathlib import Path
from typing import Any, Callable, Optional
from typing import Any, Callable, Mapping, Optional
from scripts.agent_benchmark.manifest import Timeout
@ -148,6 +150,7 @@ DEFAULT_ENV_ALLOWLIST = (
"USER", "LOGNAME", "SHELL", "PWD", "PYTHONPATH", "PYTHONHASHSEED",
"NO_COLOR", "CI",
)
TLS_CA_ENV_KEYS = ("SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS")
ENV_KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$")
METRIC_KIND_RE = re.compile(r"^metric:[a-z0-9][a-z0-9_.+-]{0,63}$")
@ -681,6 +684,18 @@ def env_pairs(mapping: dict[str, str]) -> tuple[tuple[str, str], ...]:
return tuple(sorted((str(k), str(v)) for k, v in mapping.items()))
def inherited_tls_ca_environment(
environment: Mapping[str, str] | None = None,
) -> tuple[tuple[str, str], ...]:
"""Freeze only the two standard public CA bundle settings for a child."""
source = os.environ if environment is None else environment
return tuple(
(key, value)
for key in TLS_CA_ENV_KEYS
if isinstance((value := source.get(key)), str) and value
)
# ---------------------------------------------------------------------------
# Frame transport
# ---------------------------------------------------------------------------
@ -759,11 +774,47 @@ class _Supervisor:
socket_path = self.control_dir / SOCKET_FILENAME
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
previous_umask = os.umask(0o177)
original_dir_fd = -1
control_dir_fd = -1
try:
self.sock.bind(str(socket_path))
# Some shared/container filesystems reject bind(2) when the socket
# pathname traverses the short attempt symlink, even though the
# resolved directory supports Unix sockets. Resolve the directory
# once with fchdir and bind the basename; the published locator
# remains the authenticated short absolute alias.
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
original_dir_fd = os.open(".", directory_flags)
control_dir_fd = os.open(self.control_dir, directory_flags)
os.fchdir(control_dir_fd)
self.sock.bind(SOCKET_FILENAME)
try:
os.chmod(SOCKET_FILENAME, 0o600)
except OSError as exc:
# A small class of shared filesystems permits Unix sockets but
# rejects chmod on the socket inode. The containing directory
# is still an exclusive 0700 security boundary.
directory = os.stat(".")
unsupported = {errno.EINVAL}
if hasattr(errno, "ENOTSUP"):
unsupported.add(errno.ENOTSUP)
if (
exc.errno not in unsupported
or stat.S_IMODE(directory.st_mode) != 0o700
or directory.st_uid != os.geteuid()
):
raise
else:
socket_mode = stat.S_IMODE(os.lstat(SOCKET_FILENAME).st_mode)
if socket_mode & 0o077:
raise PermissionError("control socket permissions are too broad")
finally:
if original_dir_fd >= 0:
os.fchdir(original_dir_fd)
if control_dir_fd >= 0:
os.close(control_dir_fd)
if original_dir_fd >= 0:
os.close(original_dir_fd)
os.umask(previous_umask)
os.chmod(socket_path, 0o600)
self.sock.listen(4)
locator = {
"supervisor_pid": os.getpid(),

20
scripts/agent_benchmark/lifecycle_test.py
View file

@ -627,6 +627,26 @@ class LifecycleTest(unittest.TestCase):
self.assertTrue(checked)
self.assertTrue(result.success)
def test_control_socket_bind_supports_short_symlink_alias(self) -> None:
with tempfile.TemporaryDirectory(
dir=Path.cwd(), prefix=".lifecycle-symlink-target-"
) as target_name, tempfile.TemporaryDirectory(
dir=tempfile.gettempdir(), prefix="iop-life-alias-parent-"
) as alias_parent_name:
target = Path(target_name)
alias = Path(alias_parent_name) / "attempt"
alias.symlink_to(target.resolve(), target_is_directory=True)
evidence = target / "evidence"
evidence.mkdir()
result = self._run(replace(
self._spec("print('FINISH'); print('IDLE')"),
evidence_dir=str(evidence),
control_dir=str(alias / "control"),
))
self.assertTrue(result.success)
self.assertTrue((target / "control" / "locator.json").is_file())
self.assertFalse((target / "control" / "control.sock").exists())
def test_owned_descendant_ignoring_term_is_killed_and_reaped(self) -> None:
descendant_pid_path = self.root / "descendant.pid"
descendant_source = (

45
scripts/agent_benchmark/live_iop.py
View file

@ -159,6 +159,9 @@ def _base_url(value: str) -> str:
def _models_url(base_url: str) -> str:
parsed = urlsplit(base_url)
path = parsed.path.rstrip("/")
marker = "/gemini/"
if marker in path:
path = path.split(marker, 1)[0]
if path.endswith("/v1"):
path += "/models"
else:
@ -166,6 +169,18 @@ def _models_url(base_url: str) -> str:
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
def _agy_route_base(base_url: str, route_id: str) -> str:
parsed = urlsplit(base_url)
path = parsed.path.rstrip("/")
marker = "/gemini/"
if marker in path:
path = path.split(marker, 1)[0]
if path.endswith("/v1"):
path = path[:-3]
path = path.rstrip("/") + f"/gemini/{route_id}"
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
def _command(argv: tuple[str, ...]) -> str:
try:
completed = subprocess.run(
@ -878,18 +893,25 @@ class _LiveAdapter:
if not observed.caller_ready:
result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible"))
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
binding, issues = _binding_from_config(cell, self.capability, runtime.config)
if not issues and binding.effective_model not in observed.catalog_models:
binding, issues = _requested(cell), _issues("model_missing")
if issues:
result = make_result(cell, self.capability, binding, issues)
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
agy_preflight: AgyPreflightResult | None = None
if self.caller == AGY_CALLER:
capability = inspect_agy_iop_capability(observed.agy_version, observed.agy_help)
agy_endpoint = _agy_route_base(runtime.base_url, cell.iop.route_id)
agy_observation = AgyRuntimeObservation(
cell.id, cell.iop.route_kind, cell.iop.route_id,
_agy_runtime_identity("endpoint", runtime.base_url),
_agy_runtime_identity("endpoint", agy_endpoint),
_agy_runtime_identity("credential", runtime.secret), runtime.config.identity,
)
try:
agy_preflight = preflight_agy_iop(
cell, capability,
AgyRuntimeInputs(self._binary_resolver("agy"), runtime.base_url, runtime.secret),
AgyRuntimeInputs(self._binary_resolver("agy"), agy_endpoint, runtime.secret),
agy_observation,
)
except Exception:
@ -898,9 +920,6 @@ class _LiveAdapter:
if agy_preflight.issues:
result = make_result(cell, self.capability, _requested(cell), agy_preflight.issues)
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
binding, issues = _binding_from_config(cell, self.capability, runtime.config)
if not issues and binding.effective_model not in observed.catalog_models:
binding, issues = _requested(cell), _issues("model_missing")
result = make_result(cell, self.capability, binding, issues)
if result.status == "ready":
self._admitted_bindings[cell.id] = result.binding
@ -952,8 +971,10 @@ class _LiveAdapter:
cell, prepared, task_payload, timeout, agy_preflight
),
)
parser = AgyEventParser(cell)
parser = AgyEventParser(cell, admitted)
result = self._invokers.agy(spec, parser, agy_preflight, lambda locator: on_started(locator, spec_digest(spec)))
if not result.success:
return _bound_observations(result, admitted)
observed = parser.observed_result(agy_preflight.capability, result)
if observed.status != "ready" or observed.binding != admitted:
raise LiveIopError("stream_incompatible")
@ -972,7 +993,10 @@ class _LiveAdapter:
)
result = self._invokers.codex(invocation, lambda locator: on_started(locator, spec_digest(invocation.spec)))
expected = (admitted.effective_route_kind, admitted.effective_route_id, admitted.effective_model, admitted.effective_effort)
if result.effective_binding != expected:
if (
result.effective_binding is not None
and result.effective_binding != expected
):
raise LiveIopError("stream_incompatible")
return _bound_observations(result.lifecycle, admitted)
raise LiveIopError("protocol_incompatible")
@ -1101,7 +1125,10 @@ class _LiveScoringAdapter:
admitted.effective_model,
admitted.effective_effort,
)
if result.effective_binding != expected:
if (
result.effective_binding is not None
and result.effective_binding != expected
):
return ScoringInvocationResult(
False, "binding_mismatch", result.effective_binding
)
@ -1109,7 +1136,7 @@ class _LiveScoringAdapter:
return ScoringInvocationResult(
lifecycle.success,
lifecycle.terminal_reason,
result.effective_binding,
expected,
)
def finalize_evidence(

10
scripts/e2e-credential-slot-smoke.sh
View file

@ -798,11 +798,11 @@ create_slot_route() {
}
wait_route_projection() {
local route_id="$1"
local public_route_id="$1"
local deadline=$((SECONDS + 20))
while true; do
if curl_timeout "$HTTP_PROBE_TIMEOUT" -fsS "${EDGE_CURL[@]}" "$EDGE_BASE_URL/v1/models" -H "Authorization: Bearer $IOP_TOKEN" >"$TMP_DIR/models.json" 2>/dev/null \
&& jq -e --arg id "$route_id" '.data | any(.[]; .id == $id)' "$TMP_DIR/models.json" >/dev/null; then
&& jq -e --arg id "$public_route_id" '.data | any(.[]; .id == $id)' "$TMP_DIR/models.json" >/dev/null; then
rm -f "$TMP_DIR/models.json"
return 0
fi
@ -940,8 +940,8 @@ run_deterministic() {
create_slot_route CHAT "$chat_profile" "$chat_vendor" "$chat_kind" "$chat_slot_alias" "$chat_route_alias" "$chat_provider" "$upstream_model" "$TMP_DIR/chat.secret"
create_slot_route MESSAGES "$messages_profile" "$messages_vendor" "$messages_kind" "$messages_slot_alias" "$messages_route_alias" "$messages_provider" "$upstream_model" "$TMP_DIR/messages.secret"
wait_route_projection "$CHAT_ROUTE_ID"
wait_route_projection "$MESSAGES_ROUTE_ID"
wait_route_projection "$chat_route_alias"
wait_route_projection "$messages_route_alias"
request_chat "$chat_route_alias" "$TMP_DIR/chat.response.json"
request_messages "$messages_route_alias" "$TMP_DIR/messages.response.json"
@ -1064,7 +1064,7 @@ run_live() {
write_runtime_configs "$LIVE_PROFILE" "$LIVE_PROFILE" "$LIVE_MODEL" "$provider_id" "credential-live-unselected" "$endpoint"
start_managed_stack
create_slot_route LIVE "$LIVE_PROFILE" "$vendor" "$kind" "$slot_alias" "$route_alias" "$provider_id" "$LIVE_MODEL" "$TMP_DIR/live.secret"
wait_route_projection "$LIVE_ROUTE_ID"
wait_route_projection "$route_alias"
local body live_code
body="$(jq -cn --arg model "$route_alias" --arg effort "$LIVE_REASONING_EFFORT" --argjson cap "$LIVE_MAX_COMPLETION_TOKENS" \

8
scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl
View file

@ -1,5 +1,3 @@
{"type":"metric","subtype":"duration_ms","value":12}
{"type":"metric","subtype":"model_duration_ms","value":8.5}
{"type":"iop","subtype":"effective_binding","route_kind":"direct","route_id":"agy-direct","model":"gemini-2.0-flash","effort":"high","stages":[{"stage":"request","model":"gemini-2.0-flash","effort":"high"}]}
{"type":"result","subtype":"success","model":"gemini-2.0-flash","effort":"high","route_kind":"direct","route_id":"agy-direct","content":"[redacted]"}
{"type":"system","subtype":"idle","model":"gemini-2.0-flash","effort":"high","route_kind":"direct","route_id":"agy-direct","tool_input":"[redacted]"}
{"conversation_id":"fixture-conversation","event":"init","init":{"cwd":"/workspace","model":"Gemini 3.6 Flash","permission_mode":"sandbox","tools":[]}}
{"event":"step_update","step_update":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"state":"DONE","step_index":0,"step_type":"agent_response","text_delta":"fixture content must be redacted","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}}
{"event":"result","result":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"num_turns":1,"response":"fixture content must be redacted","status":"SUCCESS","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}}

2
scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl
View file

@ -1,4 +1,4 @@
{"type":"thread.started","thread_id":"public-fixture"}
{"type":"item.completed","item":{"id":"call-1","type":"command_execution","duration_ms":7.25}}
{"type":"turn.completed","status":"completed","usage":{"input_tokens":31,"cached_input_tokens":8,"output_tokens":12,"reasoning_output_tokens":4},"item":{"content":"public fixture content"}}
{"type":"turn.completed","status":"completed","usage":{"input_tokens":31,"cached_input_tokens":8,"cache_write_input_tokens":6,"output_tokens":12,"reasoning_output_tokens":4},"item":{"content":"public fixture content"}}
{"type":"adapter.idle","adapter":"codex_iop","nonce":"fixture-nonce-0001","child_exit":0}