Merge branch 'feature/iop-one-shot-agent-model-comparison' into release/dev-936
This commit is contained in:
commit
b244b6e224
110 changed files with 14119 additions and 672 deletions
|
|
@ -37,5 +37,6 @@ Start with `agent-contract/index.md` for protocol and runtime contracts, and `ag
|
|||
Operator and client setup guides:
|
||||
|
||||
- [Edge Local Quickstart](docs/edge-local-dev-guide.md)
|
||||
- [Agent Comparison Benchmark Dev Guide](docs/agent-comparison-benchmark-dev-guide.md)
|
||||
- [dev OpenCode Settings Guide](docs/dev-opencode-settings-guide.md)
|
||||
- [dev-corp Pi Settings Guide](docs/dev-corp-pi-settings-guide.md)
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@
|
|||
|----|-----------|-----------|------|
|
||||
| `iop.openai-compatible-api` | OpenAI-compatible API, Responses API, Chat Completions, legacy Completions, error envelope/SSE terminal error, `model` route, managed projection principal auth and slot-route binding, managed-versus-legacy provider credential selection, model-driven passthrough/normalized routing, provider-pool admission/unavailable error, safe credential-slot attribution, standard metadata, and provider-native extension fields such as `chat_template_kwargs` | `apps/edge/internal/openai/*`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/service/provider_tunnel.go`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/openai-compatible-api.md` |
|
||||
| `iop.anthropic-compatible-api` | Anthropic Messages API, count_tokens, models list, bearer or `X-Api-Key` principal auth, active managed projection auth and slot-route binding, `anthropic-version` routing, native Anthropic tunnel, Chat bridge, provider-pool-only admission, profile capability checks, managed-versus-legacy provider credentials, marked-preset single-request admission with Edge-owned internal Plan/Review template customization that leaves caller I/O unchanged, and current no-OpenAI-metric status | `apps/edge/internal/openai/anthropic_handler.go`, `apps/edge/internal/openai/anthropic_native.go`, `apps/edge/internal/openai/anthropic_bridge.go`, `apps/edge/internal/openai/anthropic_stream.go`, `apps/edge/internal/openai/anthropic_types.go`, `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/openai/provider_tunnel.go`, `apps/edge/internal/openai/provider_model_rewrite.go`, `apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_plan_stage.go`, `apps/edge/internal/openai/single_request_review_stage.go`, `packages/go/singlerequesttemplate/template.go`, `packages/go/config/protocol_profile.go` | `agent-contract/outer/anthropic-compatible-api.md` |
|
||||
| `iop.gemini-compatible-api` | Gemini Developer API `streamGenerateContent`, route-qualified Gemini-native ingress, `x-goog-api-key` principal auth, `GOOGLE_GEMINI_BASE_URL`, official agy 1.1.12 API-key transport, Gemini function calls/thought signatures/SSE, and direct-versus-execution-preset binding | `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/live_iop.py` | `agent-contract/outer/gemini-compatible-api.md` |
|
||||
| `iop.a2a-json-rpc-api` | A2A JSON-RPC API, `message/send`, `tasks/get`, `tasks/cancel`, A2A task state, agent card, `a2a.bearer_token`, Edge A2A input surface | `apps/edge/internal/input/a2a/*`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/a2a-json-rpc-api.md` |
|
||||
|
||||
## Inner Contracts
|
||||
|
|
|
|||
107
agent-contract/outer/gemini-compatible-api.md
Normal file
107
agent-contract/outer/gemini-compatible-api.md
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
# Gemini-Compatible API Contract
|
||||
|
||||
## 계약 메타
|
||||
|
||||
- id: `iop.gemini-compatible-api`
|
||||
- boundary: `outer`
|
||||
- status: active
|
||||
- 원본 경로:
|
||||
- `apps/edge/internal/openai/routes.go`
|
||||
- `apps/edge/internal/openai/principal.go`
|
||||
- `apps/edge/internal/openai/chat_handler.go`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- external caller surface: official Antigravity CLI `agy` 1.1.12 Gemini API-key provider
|
||||
|
||||
## 읽는 조건
|
||||
|
||||
- Gemini Developer API `streamGenerateContent`, `x-goog-api-key`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `agy` API-key provider, Gemini-native tool/function call, 또는 Gemini-native SSE ingress를 구현·검증할 때 읽는다.
|
||||
- IOP execution preset을 Gemini-native caller에 노출하거나 `agy` benchmark transport를 변경할 때 읽는다.
|
||||
|
||||
## 범위
|
||||
|
||||
IOP Edge가 외부 Gemini-native caller에게 제공하는 초기 호환 표면은 다음 streaming endpoint다.
|
||||
|
||||
```http
|
||||
POST /gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse
|
||||
Content-Type: application/json
|
||||
x-goog-api-key: <IOP principal token>
|
||||
```
|
||||
|
||||
- `{route-id}`는 Edge가 인증된 principal에 대해 해석할 direct route 또는 virtual execution-preset id다. URL path segment 하나의 canonical token이어야 한다.
|
||||
- `{caller-model}`은 caller가 선택한 Gemini 모델 id이며 관측·호환성 검증 대상이다. provider/credential 또는 execution preset 선택 권한은 갖지 않는다.
|
||||
- 현재 `agy` 호환 표면은 `alt=sse`인 `streamGenerateContent`만 지원한다. `generateContent`, batch, files, cached content, tuning API는 비범위다.
|
||||
- direct route와 marked single-request preset은 모두 기존 Edge route resolution, managed admission, provider-pool, preset coordinator를 사용한다. 별도 Gemini 전용 우회 dispatch를 만들지 않는다.
|
||||
|
||||
## Caller 설정
|
||||
|
||||
공식 `agy` 1.1.12 API-key provider는 다음 값으로 실행한다.
|
||||
|
||||
- `~/.gemini/antigravity-cli/settings.json`: `modelProvider`는 `gemini`다.
|
||||
- `GEMINI_API_KEY`: upstream provider key가 아니라 IOP principal token이다.
|
||||
- `GOOGLE_GEMINI_BASE_URL`: `https://<edge>/gemini/{route-id}`다.
|
||||
- 사설 dev CA를 사용하는 경우 caller child에는 표준 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`만 명시적으로 전달한다.
|
||||
- `--model`: 공식 CLI가 인식하는 Gemini 모델 label을 사용한다. benchmark의 Gemini 3.6 Flash 호출은 `Gemini 3.6 Flash`다.
|
||||
- `--effort`는 API-key provider 호출에 전달하지 않는다. 요청된 high effort는 인증된 IOP route/preset의 effective binding으로 검증한다.
|
||||
|
||||
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 계약의 transport가 아니다.
|
||||
|
||||
## Auth 및 credential 경계
|
||||
|
||||
- `x-goog-api-key`는 Gemini ingress에서 IOP caller 인증 헤더다. Edge는 이를 SHA-256 projection match에만 사용하고 raw 값을 log, metric, response, task evidence에 남기지 않는다.
|
||||
- `Authorization: Bearer`를 함께 보내면 두 token은 constant-time 비교로 같아야 한다. 다르거나 malformed이면 provider dispatch 전에 Gemini 오류 envelope로 `401`을 반환한다.
|
||||
- managed mode에서 provider credential은 projected slot과 sealed lease에서만 온다. inbound `x-goog-api-key`를 upstream `Authorization` 또는 upstream `x-goog-api-key`로 전달하지 않는다.
|
||||
- legacy mode에서도 inbound `x-goog-api-key`를 provider credential로 재사용하지 않는다. provider auth가 별도로 필요하면 기존 명시적 legacy provider-auth 계약만 적용한다.
|
||||
- marked single-request preset은 managed projection과 고정 stage authorization 없이는 fail closed한다.
|
||||
|
||||
## 요청 변환
|
||||
|
||||
초기 호환 범위는 official `agy` 1.1.12가 보내는 다음 top-level field다.
|
||||
|
||||
- `contents[]`: `role`, `parts[].text`, `parts[].functionCall`, `parts[].functionResponse`, optional opaque `thoughtSignature`
|
||||
- `systemInstruction`: official `agy`의 `role: user`와 `parts[].text`
|
||||
- `generationConfig`: `candidateCount`, `maxOutputTokens`, `stopSequences`, `temperature`, `topK`, `topP`, `thinkingConfig.includeThoughts`, `thinkingConfig.thinkingBudget`
|
||||
- `tools[].functionDeclarations[]`: `name`, `description`, `parametersJsonSchema`
|
||||
- `toolConfig.functionCallingConfig.mode`
|
||||
|
||||
Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message, tool schema와 output cap으로 변환한다. `thinkingConfig`는 Gemini OpenAI-compatible upstream이 요구하는 `extra_body.google.thinking_config`로 보존한다. Gemini 3.6에서 폐기된 `temperature`, `topP`, `topK`는 형식과 범위만 검증하고 Chat upstream에는 전달하지 않는다. 함수명·JSON argument·opaque thought signature는 caller turn 사이에 의미를 바꾸지 않는다. 지원하지 않는 content part, duplicate member, 잘못된 role, malformed function payload 또는 둘 이상의 candidate 요청은 provider dispatch 전에 `400 INVALID_ARGUMENT`으로 거부한다.
|
||||
|
||||
## SSE 응답
|
||||
|
||||
- 성공 stream은 `Content-Type: text/event-stream`과 `data: <Gemini GenerateContentResponse JSON>` frame을 사용한다.
|
||||
- text delta는 `candidates[0].content.parts[].text`, reasoning delta는 `thought=true`인 part, 완성된 tool call은 `functionCall` part로 투영한다.
|
||||
- tool-call argument fragment는 Edge가 bounded buffer에서 완성된 JSON object로 검증한 뒤 한 번만 공개한다. malformed·oversize argument는 성공 tool call로 내보내지 않는다.
|
||||
- OpenAI `stop`, `length`, `tool_calls` terminal은 Gemini `finishReason`의 `STOP`, `MAX_TOKENS`, `STOP`으로 닫는다. stream 종료 뒤 별도 합성 `system idle` event를 만들지 않는다.
|
||||
- provider-reported usage가 있으면 `usageMetadata.promptTokenCount`, `candidatesTokenCount`, `thoughtsTokenCount`, `cachedContentTokenCount`, `totalTokenCount`의 존재하는 값만 투영한다. 누락 값을 0으로 발명하지 않는다.
|
||||
- caller disconnect는 기존 request cancellation 경계를 사용하며 이후 frame을 쓰지 않는다.
|
||||
|
||||
## 공식 agy stream-json lifecycle
|
||||
|
||||
`agy` 1.1.12의 각 JSONL record는 `event` discriminator와 같은 이름의 중첩 payload를 사용한다.
|
||||
|
||||
- init: `{"event":"init","init":{...}}`
|
||||
- step: `{"event":"step_update","step_update":{"state":...,"step_type":...,"usage":{...}}}`
|
||||
- terminal: `{"event":"result","result":{"status":"SUCCESS","duration_seconds":...,"num_turns":...,"usage":{...}}}`
|
||||
|
||||
benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 terminal로 인정한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다.
|
||||
|
||||
## 오류
|
||||
|
||||
HTTP commit 전 오류는 다음 Gemini envelope 한 건으로 반환한다.
|
||||
|
||||
```json
|
||||
{"error":{"code":400,"message":"request is invalid","status":"INVALID_ARGUMENT"}}
|
||||
```
|
||||
|
||||
- 인증 실패는 `401 UNAUTHENTICATED`, route/요청 검증 실패는 `400 INVALID_ARGUMENT`, runtime/provider 실패는 `502 UNAVAILABLE`의 고정된 caller-safe message를 사용한다.
|
||||
- stream commit 뒤 오류는 Gemini `error` payload 한 건으로 끝내며 raw provider body, endpoint, route binding, credential/slot/lease id, prompt, tool argument/result를 포함하지 않는다.
|
||||
|
||||
## 변경 시 확인할 코드와 테스트
|
||||
|
||||
- route/auth: `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`
|
||||
- Gemini request/SSE bridge: `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`
|
||||
- Edge regression: `apps/edge/internal/openai/gemini_handler_test.go`, existing Chat/preset/auth tests
|
||||
- caller adapter: `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/agy_iop_test.py`, `scripts/agent_benchmark/live_iop.py`
|
||||
- live proof: official `agy --output-format stream-json` through the dev Edge route-specific base URL, with direct and execution-preset effective binding evidence
|
||||
|
||||
2026-08-12 dev 검증에서 normal/boundary/auth/tool/SSE 회귀 테스트와 공식 `agy` 1.1.12 direct·hybrid 실호출이 통과해 이 계약을 active로 전환했다.
|
||||
|
|
@ -54,8 +54,10 @@ Edge 설정에 `openai.principal_tokens[]`가 설정된 경우, caller는 기존
|
|||
In managed mode, OpenAI-compatible routes authenticate `Authorization: Bearer <IOP token>` by hashing the token and matching the projected digest. Static principal mappings and the legacy bearer are prohibited by configuration and never act as fallbacks. Unknown or removed digests, malformed headers, and expired snapshots return `401 unauthorized` before model lookup or dispatch. Expiry never returns the process to legacy behavior.
|
||||
|
||||
When managed mode is active, model discovery (`GET /v1/models`) lists only active
|
||||
projected `route_id`s for the authenticated principal. Request model resolution binds
|
||||
the request strictly to one projected route's `slot_id`, `profile_id`, and `upstream_model`.
|
||||
projected public route identities for the authenticated principal: `route_alias` when
|
||||
it is non-empty, otherwise `route_id`. Request model resolution accepts that same
|
||||
public identity and binds the request strictly to one projected route's `slot_id`,
|
||||
`profile_id`, and `upstream_model`.
|
||||
Unknown, inactive, or cross-principal routes never fall back to legacy `model_routes`,
|
||||
global catalog, or single-target default.
|
||||
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
|
|||
## When to use
|
||||
|
||||
- User requests benchmark manifest validation: `validate`, `validate manifest`, `manifest 검증`
|
||||
- User requests direct-route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검`
|
||||
- User requests route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검`
|
||||
- User requests benchmark execution: `run`, `run benchmark`, `벤치마크 실행`, `시작해`
|
||||
- User requests benchmark resume: `resume`, `resume benchmark`, `재개`, `계속해`
|
||||
- User requests benchmark status: `status`, `status benchmark`, `상태 확인`, `어디까지 왔어`
|
||||
|
|
@ -55,14 +55,14 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
|
|||
|
||||
4. **Delegate preflight to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py preflight --manifest <manifest-path>`
|
||||
- The CLI validates generic preset cells locally and records only direct-cell observations in one append-only run record.
|
||||
- The CLI records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order in one append-only run record.
|
||||
- On exit 0, report the exact closed `ready` summary from stdout.
|
||||
- On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat a generic preset contract as live readiness.
|
||||
- On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat local manifest validation as live readiness.
|
||||
|
||||
5. **Delegate run to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py run --manifest <manifest-path>`
|
||||
- On missing or invalid manifest, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before creating execution state.
|
||||
- The CLI creates one run and uses its single writer to append a fresh preflight before attempt allocation.
|
||||
- The CLI creates one run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
|
||||
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69, allocates no attempt, and preserves the run id for a later resume.
|
||||
- On `ready`, it binds the exact caller, cell, fresh workspace, session, and attempt identity, then must invoke each eligible cell exactly once with the fixture task.
|
||||
- Exit 0 only when every retained attempt is successful; otherwise report the exact closed execution failure summary from stderr with exit 69.
|
||||
|
|
@ -70,7 +70,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
|
|||
6. **Delegate resume to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py resume --manifest <manifest-path> --run-id <run-id> [--retry-failed]`
|
||||
- On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before changing the run.
|
||||
- The CLI opens the exact immutable run and uses its single writer to append a fresh preflight before attempt allocation.
|
||||
- The CLI opens the exact immutable run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
|
||||
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69 and allocates no attempt.
|
||||
- On `ready`, it reconciles interrupted state, skips successful slots, preserves prior attempt bytes, and allocates a new attempt only for eligible work. `--retry-failed` admits a new attempt for failed, timed-out, or cancelled slots.
|
||||
- It must invoke each eligible cell exactly once with a new workspace and session identity.
|
||||
|
|
@ -84,7 +84,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
|
|||
8. **Delegate score to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py score --manifest <manifest-path> --run-id <run-id> [--retry-scoring-failed]`
|
||||
- The CLI classifies lifecycle or required web-gate failures as immutable `unscored`, without invoking the evaluator or assigning zero.
|
||||
- Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact `landing-quality-v1` worksheet.
|
||||
- Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted.
|
||||
- A prior `scored` result is terminal. A prior `scoring_failed` result is retried only with `--retry-scoring-failed`, which allocates a new score id and preserves every prior byte.
|
||||
- On exit 0, report the exact closed `scored`, `unscored`, `scoring_failed`, and `blocked` counts from stdout.
|
||||
- On exit 69, report the exact closed counts or unavailable-state line from stderr. Never substitute evaluator route/model/effort, fabricate a worksheet, or turn failure into zero.
|
||||
|
|
@ -97,7 +97,7 @@ Route agent comparison benchmark requests to the deterministic CLI while enforci
|
|||
|
||||
- [ ] The CLI command was executed and the exit code matches the documented contract.
|
||||
- [ ] The reported stdout/stderr matches the CLI output exactly.
|
||||
- [ ] Preflight evidence is append-only, direct-only, and its output uses only closed status/count fields.
|
||||
- [ ] Preflight evidence is append-only, covers every immutable matrix cell in canonical order, and uses only closed status/count fields.
|
||||
- [ ] A preflight blocker created no scored attempt and was not bypassed.
|
||||
- [ ] An ineligible execution attempt became `unscored` without an evaluator invocation or a zero score.
|
||||
- [ ] Each eligible score id used one opaque blind workspace and one fresh evaluator session; retry preserved prior bytes and used a new id.
|
||||
|
|
@ -169,8 +169,8 @@ stderr: <verbatim closed preflight or execution failure summary>
|
|||
- Durable run/attempt state and preflight evidence are persisted only under the validated run root (`agent-test/runs/<output-id>/<run-id>/`).
|
||||
- Caller sessions, output workspaces, and caches are fresh and isolated for every cell, repetition, and attempt; session or cache state is never shared within a run or across runs.
|
||||
- Read-only testbed/fixture inputs (such as `../iop-s2`) are not copied back or mutated; no writes occur outside the validated run root.
|
||||
- Direct preflight never allocates a scored attempt. Generic preset cells are local contract validation only.
|
||||
- Run/resume append a fresh direct preflight under the run writer before any attempt allocation; a blocker allocates no attempt.
|
||||
- Preflight never allocates a scored attempt. Every immutable matrix cell requires its own fresh live observation.
|
||||
- Run/resume append a fresh all-cell preflight under the run writer before any attempt allocation; a blocker allocates no attempt.
|
||||
- Ready execution binds one exact cell and immutable attempt identity to one fresh workspace/session and one task submission.
|
||||
- Scoring copies only anonymous generated files, two local images, and screenshots into an opaque run-owned blind tree; the identity mapping remains outside that tree.
|
||||
- Scoring records `unscored`, `scored`, and `scoring_failed` append-only, and a retry always allocates a fresh score id/session.
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실
|
|||
- 경로: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](../../archive/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md)
|
||||
- 요약: 모델·caller·prompt·반복 횟수를 manifest로 바꾸고 Claude Code, agy, Codex의 IOP 연결부터 finish/idle, 시간·token·웹 검증·익명 채점·Markdown 보고까지 같은 pipeline으로 재현한다.
|
||||
|
||||
- [계획] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
- [진행중] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
- 경로: [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](milestones/iop-one-shot-agent-model-comparison.md)
|
||||
- 요약: route-02 정식 smoke와 benchmark pipeline 준비 뒤 dev `../iop-s2`에서 동일 정적 웹 fixture로 9개 IOP 경유 단독·하이브리드 caller 조합을 각각 한 번 실행해 속도·token·품질을 비교한다.
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
|
||||
## 상태
|
||||
|
||||
[계획]
|
||||
[진행중]
|
||||
|
||||
## 구현 잠금
|
||||
|
||||
|
|
@ -59,9 +59,11 @@
|
|||
|
||||
실행 전에 공정한 fixture와 실제 IOP route/credential 상태를 고정한다.
|
||||
|
||||
- [ ] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
|
||||
- [ ] [route-readiness] dev `../iop-s2`에서 Claude Code·agy·Codex의 IOP 인증, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 preflight를 통과했는지 확인한다.
|
||||
- [ ] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
|
||||
- [x] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
|
||||
- [x] [agy-iop-compatibility] 공식 `agy 1.1.12`의 Gemini API-key provider가 route별 `GOOGLE_GEMINI_BASE_URL`과 `GEMINI_API_KEY=<IOP principal token>`으로 Edge의 Gemini-native `streamGenerateContent` ingress를 호출하고, Gemini request/tool/SSE를 기존 direct·execution-preset 실행에 연결하며 실제 `stream-json` lifecycle/usage를 benchmark adapter가 수집하도록 구현한다. `--effort`와 비공식 custom model에 의존하지 않고 high effort는 IOP effective binding으로 검증한다.
|
||||
- [x] [managed-credential-dev] dev Control Plane·Edge·Node에 CA-signed mTLS, Edge HTTPS, at-rest/issuer/recipient key material, principal projection과 Gemini/Claude/GPT slot-route를 operator-owned secret 경로로 구성한다. legacy static credential source를 제거하고 marked hybrid preset이 같은 principal의 고정 stage route를 managed lease로 실행하는지 secret-safe smoke로 확인한다.
|
||||
- [x] [route-readiness] dev `../iop-s2`에서 Claude Code·Codex의 호환 ingress와 공식 agy의 Gemini API-key transport가 각각 IOP 인증 경계를 통과하고, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 실제 caller preflight를 통과했는지 확인한다. 존재하지 않는 caller 환경 변수나 합성 event fixture는 live 호환 근거로 인정하지 않는다.
|
||||
- [x] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
|
||||
|
||||
### Epic: [comparison-runs] 9개 원샷 실행
|
||||
|
||||
|
|
@ -103,6 +105,7 @@
|
|||
- 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2`
|
||||
- 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다.
|
||||
- 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다.
|
||||
- 현재 차단: readiness는 완료됐고 all-cell preflight는 `ready=9`였다. 승인된 C01-C09 `run`은 2026-08-12에 한 번 호출됐으나 첫 slot의 caller launch 전 control socket 등록이 workspace filesystem의 symlink-path `bind(2)` `EINVAL`로 중단됐다. caller는 실행되지 않았고 dangling attempt는 harness reconcile로 `interrupted=1`, `running=0`이 됐다. socket 등록과 pre-registration reconcile 결함은 회귀 테스트와 함께 수정했지만 repetitions=1·no-retry 정책 때문에 같은 scored run을 재실행하거나 resume하지 않는다. C01-C09를 다시 시작하려면 기존 실패를 덮지 않는 새 실행 승인과 SDD/manifest run 정책 결정이 필요하다.
|
||||
- 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md)
|
||||
- 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md)
|
||||
- 확인 필요: 없음
|
||||
|
|
|
|||
|
|
@ -24,6 +24,8 @@
|
|||
- [x] [D08] token은 input/output/reasoning/cached/total과 source를 model/stage별로 기록하고 미제공 값을 exact로 추정하지 않는다.
|
||||
- [x] [D09] 결과 identity를 가린 뒤 동일 100점 rubric으로 Codex가 채점하고 자동 검증과 수동 점수를 분리한다.
|
||||
- [x] [D10] scored failure는 보존하고 재실행은 새 attempt로 기록하며 성공 결과만 골라 대표하지 않는다.
|
||||
- [x] [D11] 공식 `agy 1.1.12`는 Gemini API-key provider의 route별 `GOOGLE_GEMINI_BASE_URL`을 IOP Edge로 지정하고 `GEMINI_API_KEY`에는 upstream key가 아닌 IOP principal token을 넣는다. `--effort`와 비공식 custom model은 사용하지 않고 high effort는 IOP effective binding으로 검증한다.
|
||||
- [x] [D12] marked hybrid preset은 dev managed credential plane의 fresh projection, 고정 stage authorization과 sealed provider lease가 준비된 뒤에만 실행하며 legacy credential fallback을 허용하지 않는다.
|
||||
|
||||
## 문제 / 비목표
|
||||
|
||||
|
|
@ -44,7 +46,7 @@
|
|||
| Fixture | versioned prompt, 이미지 2장과 vanilla workspace checksum | 모든 cell의 동일 입력 기준 |
|
||||
| Evidence | `agent-test/runs/<run-id>/` | attempt별 timeline, usage, validation, screenshot와 score |
|
||||
| Report | `agent-test/dev/iop-one-shot-agent-comparison-<date>.md` | 현재 프로젝트의 사람이 읽는 비교 결과 |
|
||||
| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 |
|
||||
| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 |
|
||||
| Config Contract | [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) | model route, preset, protocol profile과 credential 경계 |
|
||||
| User Decision | D01-D10 | 2026-08-06 확정 방향, 추가 사용자 결정 없음 |
|
||||
|
||||
|
|
@ -53,7 +55,7 @@
|
|||
| 상태 | 진입 조건 | 다음 상태 | 근거 |
|
||||
|------|-----------|-----------|------|
|
||||
| `blocked` | `[route-02]` smoke 또는 `[bench-01]` 완료 전 | `preflighting`, 종료 | active Milestone 상태와 pipeline evidence |
|
||||
| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | redacted preflight matrix |
|
||||
| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | official caller transport, managed projection/lease와 redacted preflight matrix |
|
||||
| `ready` | fixture와 C01-C09 immutable manifest 확정 | `running`, `cancelled` | manifest/fixture/rubric digest |
|
||||
| `running` | seed 순서에 따라 각 cell에 사용자 작업 1회 제출 | `validating`, `failed`, `timed_out`, `cancelled` | cell/attempt event timeline |
|
||||
| `validating` | cell finish/complete 후 idle 확정 | `scoring`, `failed` | workspace, build/render/test evidence |
|
||||
|
|
@ -75,13 +77,14 @@ State invariant:
|
|||
|
||||
## Interface Contract
|
||||
|
||||
- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md)
|
||||
- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md)
|
||||
- 입력:
|
||||
- `fixture`: 동일 이미지 2장, one-page 요구사항, vanilla HTML/CSS/JS initial workspace와 checksum이다.
|
||||
- `cells`: C01-C09의 caller, IOP route/preset, expected model/stage와 effort binding이다.
|
||||
- `repetitions=1`, `session_policy=fresh`, `setup_cache_policy`: 초기 scored attempt 수, conversation/resume 격리와 공통 setup/cache 기준이다.
|
||||
- `environment=dev`, `testbed=../iop-s2`: 실제 IOP runtime 선택이다.
|
||||
- `completion`: caller별 finish/complete event와 idle 판정 규칙이다.
|
||||
- `agy`: `modelProvider=gemini`, route별 `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY=<IOP principal token>`과 official `stream-json` event다. API-key provider가 지원하지 않는 `--effort`는 전달하지 않는다.
|
||||
- 측정 출력:
|
||||
- timestamp: submitted, first output, first file write, model/stage start/end, tool start/end, finish, idle의 monotonic 값과 observation source다. overlap과 unavailable을 명시한다.
|
||||
- usage: call count, input/output/reasoning/cached/total token과 source다.
|
||||
|
|
@ -111,6 +114,8 @@ State invariant:
|
|||
| S10 | `quality-scoring` | identity가 제거된 9개 결과 | Codex rubric 평가 | 항목별 점수/근거와 총점이 자동 gate와 분리되어 기록된다. |
|
||||
| S11 | `performance-usage` | 모든 attempt timeline/usage | 비교 집계 | 첫 output·첫 write·model/tool/queue/total 시간의 clock/source·overlap, 호출 수와 token/source가 cell·stage별 표가 된다. |
|
||||
| S12 | `benchmark-report` | S01-S11 evidence | 보고서 생성 | 조건·버전·9개 결과·속도·token·품질·실패·한계와 raw evidence 링크가 Markdown에 남는다. |
|
||||
| S13 | `agy-iop-compatibility` | official `agy 1.1.12`와 dev Edge | direct·hybrid route별 Gemini base URL로 실제 API-key 호출 | 두 호출 모두 `x-goog-api-key` IOP principal auth, Gemini-native request/tool/SSE, official `stream-json` finish/exit와 config-owned effective binding evidence를 남기고 upstream key 직접 호출이나 합성 event에 의존하지 않는다. |
|
||||
| S14 | `managed-credential-dev` | dev Control Plane·Edge·Node와 operator-owned security material | managed credential profile로 재기동하고 slot/route를 등록 | CA-signed mTLS·Edge HTTPS·fresh projection·sealed lease가 확인되고 legacy credential source나 cross-route fallback 없이 direct와 marked preset stage가 실행된다. |
|
||||
|
||||
## Evidence Map
|
||||
|
||||
|
|
@ -128,6 +133,8 @@ State invariant:
|
|||
| S10 | blind mapping 분리와 Codex rubric worksheet | `agent-task/m-iop-one-shot-agent-model-comparison/quality-scoring/` | `quality-scoring` 100-point evidence |
|
||||
| S11 | cell/stage별 normalized timeline, calls와 token-source table | `agent-task/m-iop-one-shot-agent-model-comparison/performance-usage/` | `performance-usage` speed/token evidence |
|
||||
| S12 | `agent-test/dev/` Markdown과 raw run links | `agent-task/m-iop-one-shot-agent-model-comparison/benchmark-report/` | `benchmark-report` complete comparison evidence |
|
||||
| S13 | official agy request-shape capture, Edge Gemini bridge tests, direct·hybrid live preflight와 sanitized lifecycle/usage | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `agy-iop-compatibility` official 1.1.12 IOP transport evidence |
|
||||
| S14 | dev config check, TLS/workload identity, projection generation, slot-route/lease attribution과 post-revoke no-fallback smoke | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `managed-credential-dev` secure composition and hybrid admission evidence |
|
||||
|
||||
공통 완료 검증은 C01-C09 모두가 success/failure/blocked 중 하나의 terminal evidence를 가지고, 성공 결과의 자동 gate·screenshot·blind score와 모든 attempt의 timing/usage source가 보고서에 연결되는지 확인한다. 필수 credential/model이 없으면 raw secret을 요구하거나 기록하지 않고 운영 절차로 등록을 요청한다.
|
||||
|
||||
|
|
@ -145,6 +152,7 @@ State invariant:
|
|||
## 사용자 리뷰 이력
|
||||
|
||||
- 2026-08-06: 사용자가 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 비교군, Claude Code·agy·Codex caller, finish/idle 원샷, 초기 1회, dev `../iop-s2`, 동일 정적 웹 fixture와 시간·token·Codex 품질 평가를 확정했다.
|
||||
- 2026-08-12: 공식 `agy 1.1.12` API-key provider의 실제 Gemini-native 요청과 `stream-json` event를 확인했고, 사용자의 provider 직접 설정 지시에 따라 upstream key와 IOP principal token을 분리하며 dev managed credential plane까지 구성하는 D11-D12를 기술 보강했다.
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,260 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
|
||||
|
||||
# Code Review Reference - REVIEW_API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=2, tag=REVIEW_API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`.
|
||||
- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0.
|
||||
- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean.
|
||||
- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged.
|
||||
- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G05.md` → `code_review_cloud_G05_2.log` and `PLAN-cloud-G05.md` → `plan_cloud_G05_2.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REVIEW_API-1 Require complete token matches | [x] |
|
||||
| REVIEW_API-2 Re-run the benchmark regression boundary | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior.
|
||||
- [x] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G05_2.log`.
|
||||
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G05_2.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path.
|
||||
- [x] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
없음. PLAN이 선택한 두 지점만 수정했고 명령 교체나 검증 생략 없이 지정된 검증을 그대로 실행했다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- 공유 token helper의 의미를 별도 seed 전용 validator 없이 `Pattern.fullmatch`로 일관되게 강화했다.
|
||||
- 기존 schema pattern, seeded rank, canonical digest/omission 처리, slot allocation과 fixture는 변경하지 않았다.
|
||||
- terminal newline seed는 기존 schema/loader parity corpus에 추가해 양쪽 경계가 같은 값을 거부하도록 고정했다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm `_require_pattern` uses a complete-string match and rejects `"seed-a\n"` while valid 64-character seeds remain accepted.
|
||||
- Confirm the existing schema/loader parity corpus includes the terminal-newline value and asserts `ManifestValidationError`.
|
||||
- Confirm the schema, seeded rank, canonical digest/omission behavior, and `RunStore.slots` remain unchanged.
|
||||
- Confirm fresh focused and combined regression commands pass without invoking external providers.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### REVIEW_API-1 Focused Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
exit_code: 0
|
||||
stdout: (none)
|
||||
stderr:
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.005s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### REVIEW_API-1 Manifest Regression
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
exit_code: 0
|
||||
stdout: (none)
|
||||
stderr:
|
||||
...............................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 111 tests in 1.224s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### REVIEW_API-2 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
manifest validation loop:
|
||||
exit_code: 0
|
||||
stdout:
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
stderr: (none)
|
||||
|
||||
combined unittest:
|
||||
exit_code: 0
|
||||
stdout: (none)
|
||||
stderr:
|
||||
........................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 184 tests in 46.261s
|
||||
|
||||
OK
|
||||
|
||||
git diff --check:
|
||||
exit_code: 0
|
||||
stdout: (none)
|
||||
stderr: (none)
|
||||
```
|
||||
|
||||
### Reviewer Fresh Verification (2026-08-12)
|
||||
|
||||
The reviewer reran the required commands from `/config/workspace/iop-s0` without invoking external providers.
|
||||
|
||||
```text
|
||||
focused parity test:
|
||||
exit_code: 0
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.008s
|
||||
|
||||
OK
|
||||
|
||||
full manifest suite:
|
||||
exit_code: 0
|
||||
...............................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 111 tests in 1.381s
|
||||
|
||||
OK
|
||||
|
||||
manifest validation loop:
|
||||
exit_code: 0
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
|
||||
combined unittest:
|
||||
exit_code: 0
|
||||
........................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 184 tests in 50.511s
|
||||
|
||||
OK
|
||||
|
||||
git diff --check:
|
||||
exit_code: 0
|
||||
stdout: (none)
|
||||
stderr: (none)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
### Overall Verdict
|
||||
|
||||
PASS
|
||||
|
||||
### Dimension Assessment
|
||||
|
||||
| Dimension | Result | Evidence |
|
||||
|-----------|--------|----------|
|
||||
| Correctness | Pass | `_require_pattern` now requires `Pattern.fullmatch`, so the terminal-newline seed is rejected before ordering or digest processing. |
|
||||
| Completeness | Pass | Both REVIEW_API items are implemented and the selected R1 fix is present without schema, ordering, digest, slot, or fixture changes in this follow-up. |
|
||||
| Test coverage | Pass | The existing schema/loader parity corpus now includes `"seed-a\n"`; the focused test, 111 manifest tests, and 184 combined tests pass freshly. |
|
||||
| API contract | Pass | Loader behavior now agrees with the tracked seed pattern's full-string interpretation while valid shipped manifests remain accepted. |
|
||||
| Code quality | Pass | The fix is localized to the shared token helper and reuses the existing parity test without duplicate validation logic. |
|
||||
| Implementation deviation | Pass | The implementation follows the reviewer-selected files, symbols, behavior, and verification commands with no scope expansion. |
|
||||
| Verification trust | Pass | Fresh reviewer output matches the implementation-owned evidence and `git diff --check` is clean. |
|
||||
| Spec conformance | Pass | The reusable seed grammar remains consistent with SDD S03 and `milestone-task=matrix-lock`; concrete C01-C09 manifest evidence remains outside this packet. |
|
||||
|
||||
### Findings
|
||||
|
||||
None. Required 0, Suggested 0, Nit 0.
|
||||
|
||||
### Routing Signals
|
||||
|
||||
- `review_rework_count=1`
|
||||
- `evidence_integrity_failure=false`
|
||||
|
||||
### Next Step
|
||||
|
||||
Write `complete.log`, archive this PASS task with `milestone-task=matrix-lock` metadata preserved, and emit the runtime completion event without modifying the roadmap.
|
||||
|
|
@ -0,0 +1,131 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=0 tag=API milestone-task=matrix-lock -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-local-G08.md` → `plan_local_G08_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Add one canonical seeded-order contract | [ ] |
|
||||
| API-2 Bind slot allocation to canonical matrix order | [ ] |
|
||||
| API-3 Preserve the complete benchmark baseline | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
|
||||
- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
|
||||
- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_0.log`.
|
||||
- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm omission preserves legacy id ordering and does not inject a default into canonical bytes.
|
||||
- Confirm explicit seeds use the documented domain-separated rank and stable cell-id tie-breaker.
|
||||
- Confirm the JSON schema and Python loader reject the same invalid seed shapes.
|
||||
- Confirm the slot allocator has no independent sorting rule.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### API-1 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-2 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-3 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,243 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=1 tag=API milestone-task=matrix-lock -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found one stale dependency statement: it named a nonexistent integration child and incorrectly implied that child depended directly on both indices 01 and 02.
|
||||
- Replan keeps the implementation contract unchanged and corrects the DAG to `01 -> 03+01`, `02` independent, and `(02,03) -> 04+02,03`. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-local-G08.md` → `plan_local_G08_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Add one canonical seeded-order contract | [x] |
|
||||
| API-2 Bind slot allocation to canonical matrix order | [x] |
|
||||
| API-3 Preserve the complete benchmark baseline | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
|
||||
- [x] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
|
||||
- [x] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
|
||||
- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None. The implementation followed the selected owner, files, algorithm, and verification commands. Production `RunStore.slots` remains unchanged.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Added `execution_order_seed: str | None = None` after `Manifest.digest`, so existing constructors remain compatible.
|
||||
- Reused the cell-id token grammar for loader and schema validation. The 64-character boundary is accepted; empty, oversized, non-token, and non-string values fail closed.
|
||||
- Applied the exact `iop-benchmark-order-v1\0<seed>\0<cell-id>` SHA-256 rank with cell id as the collision tie-breaker.
|
||||
- Included the seed in canonical serialization only when explicitly present. Omission retains pipeline-v2 lexical cell ordering and legacy canonical bytes.
|
||||
- Kept slot allocation single-sourced from `Manifest.matrix`; the new regression proves repetitions remain adjacent within each seeded cell.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm omission preserves legacy id ordering and does not inject a default into canonical bytes.
|
||||
- Confirm explicit seeds use the documented domain-separated rank and stable cell-id tie-breaker.
|
||||
- Confirm the JSON schema and Python loader reject the same invalid seed shapes.
|
||||
- Confirm the slot allocator has no independent sorting rule.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### API-1 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
...............................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 111 tests in 1.071s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### API-2 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.001s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### API-3 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
.......................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 184 tests in 46.125s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
`git diff --check` produced no output and exited 0.
|
||||
|
||||
### Reviewer Fail-Closed Reproducer
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
import json
|
||||
import re
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest_test import (
|
||||
_load_tmp_manifest,
|
||||
_make_minimal_manifest_dict,
|
||||
_write_tmp_manifest,
|
||||
)
|
||||
seed = "seed-a\n"
|
||||
schema = json.loads(Path("scripts/fixtures/agent-comparison-benchmark-manifest.schema.json").read_text(encoding="utf-8"))
|
||||
pattern = schema["properties"]["execution_order_seed"]["pattern"]
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
manifest = _load_tmp_manifest(
|
||||
_write_tmp_manifest(
|
||||
Path(tmp),
|
||||
_make_minimal_manifest_dict(execution_order_seed=seed),
|
||||
)
|
||||
)
|
||||
print(f"schema_fullmatch={re.fullmatch(pattern, seed) is not None}")
|
||||
print(f"loader_accepted={manifest.execution_order_seed!r}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
schema_fullmatch=False
|
||||
loader_accepted='seed-a\n'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
### Overall Verdict
|
||||
|
||||
FAIL
|
||||
|
||||
### Dimension Assessment
|
||||
|
||||
| Dimension | Result | Evidence |
|
||||
|-----------|--------|----------|
|
||||
| Correctness | Fail | A trailing-newline seed bypasses the intended bounded token grammar and reaches canonical ordering/digest code. |
|
||||
| Completeness | Fail | API-1 requires invalid non-token seeds to fail closed, but that boundary remains open. |
|
||||
| Test coverage | Fail | The invalid-seed corpus omits the Python `$`/trailing-newline boundary and therefore passes while the defect remains. |
|
||||
| API contract | Fail | The tracked schema pattern rejects the reproduced seed under the parity test's full-string interpretation, while the loader accepts it. |
|
||||
| Code quality | Pass | The seeded ordering and conditional serialization are localized and readable. |
|
||||
| Implementation deviation | Pass | The implementation otherwise follows the planned owner, algorithm, files, and slot-allocation boundary. |
|
||||
| Verification trust | Fail | Fresh focused and full suites pass, but the implementation claim that non-token seed values fail closed is contradicted by the reviewer reproducer. |
|
||||
| Spec conformance | Pass | The explicit seed, deterministic order, digest binding, and slot consumption otherwise conform to SDD S03's reusable order contract. |
|
||||
|
||||
### Findings
|
||||
|
||||
- **Required R1 — The loader accepts an `execution_order_seed` ending in `\n`.**
|
||||
- **Evidence:** The fresh reviewer reproducer reports `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. `scripts/agent_benchmark/manifest.py:729` routes the seed through `_require_pattern`, while `scripts/agent_benchmark/manifest.py:194` uses `pattern.match(s)`; Python `$` permits a match immediately before one terminal newline. This violates API-1's explicit non-token fail-closed criterion and the schema/loader parity checkpoint.
|
||||
- **Root Cause:** The shared manifest token validator tests only whether a prefix matches an anchored expression instead of requiring the entire string to match. The new seed field reused that helper, and `scripts/agent_benchmark/manifest_test.py:1492` did not include the terminal-newline variant in its invalid corpus.
|
||||
- **Selected Fix:** Change `_require_pattern` in `scripts/agent_benchmark/manifest.py` to use `pattern.fullmatch(s)`, then add `"seed-a\n"` to `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` in `scripts/agent_benchmark/manifest_test.py`. Keep the schema and seeded-order algorithm unchanged. Accept only after the named parity test, full manifest suite, three example validations, manifest/attempt/connectivity regression suite, and `git diff --check` all pass freshly.
|
||||
|
||||
No Suggested or Nit findings.
|
||||
|
||||
### Routing Signals
|
||||
|
||||
- `review_rework_count=1`
|
||||
- `evidence_integrity_failure=true`
|
||||
|
||||
### Next Step
|
||||
|
||||
Create the mandatory reviewer-closed follow-up plan for Required R1, rerun isolated task routing, archive this pair, and materialize the routed follow-up pair. Do not write `complete.log`.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/01_execution_order_contract
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
실행 순서 seed 계약과 terminal-newline fail-closed 보완을 공식 리뷰 2회(FAIL 1회, PASS 1회)로 완료했으며 최종 판정은 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | FAIL | seeded order/digest/slot 계약은 구현됐으나 `"seed-a\n"`이 loader token 검증을 우회하는 Required R1을 확인했다. |
|
||||
| `plan_cloud_G05_2.log` | `code_review_cloud_G05_2.log` | PASS | 공유 pattern helper를 full-string 검증으로 강화하고 terminal-newline parity regression을 추가해 R1을 닫았다. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- optional `execution_order_seed`가 manifest canonical order와 digest에 결정론적으로 반영되고 omission 호환성과 `RunStore.slots` 순서가 유지된다.
|
||||
- `_require_pattern`이 `Pattern.fullmatch`를 사용해 seed를 포함한 manifest token 전체 문자열을 fail-closed 검증한다.
|
||||
- schema/loader parity corpus에 `"seed-a\n"`을 추가했으며 schema, seeded ranking, digest, slot allocation과 shipped fixture는 follow-up 범위에서 변경하지 않았다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds` - PASS; 1 test, `OK`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.manifest_test` - PASS; 111 tests, `OK`.
|
||||
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 184 tests, `OK`.
|
||||
- `git diff --check` - PASS; 출력 없음.
|
||||
- 공개 `validate` entrypoint를 실제 fixture로 검증했다. 외부 provider 실행과 scored benchmark full-cycle은 이 parser-only packet의 검증 범위가 아니다.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- dependent `04+02,03_locked_benchmark_manifest` packet이 실제 C01-C09 immutable manifest evidence를 제공해야 하며, 이 완료 로그 단독으로 `matrix-lock` 전체 완료를 주장하지 않는다.
|
||||
|
|
@ -0,0 +1,198 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=2 tag=REVIEW_API milestone-task=matrix-lock -->
|
||||
|
||||
# Plan - Close Execution-Order Seed Token Validation
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G05.md` is the mandatory final implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The seeded execution-order implementation is otherwise complete, but the official review found one fail-closed boundary defect. Python's anchored `$` plus `Pattern.match` accepts a single trailing newline, so the loader can accept a seed that the declared full token grammar rejects. This follow-up applies the reviewer-selected full-string validation fix and locks the exact regression.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`.
|
||||
- Verdict: `FAIL` with Required R1; Suggested 0; Nit 0.
|
||||
- R1 evidence: the reviewer reproduced `execution_order_seed="seed-a\n"` as `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. Fresh focused suites passed (111 manifest tests and 1 slot test), all three example manifests validated, the combined manifest/attempt/connectivity suite passed 184 tests, and `git diff --check` was clean.
|
||||
- Closed fix: make the shared manifest pattern helper require a full-string match and add the terminal-newline seed to the invalid parity corpus. Keep the schema, seeded ranking, canonical digest behavior, slot allocation, fixtures, and Milestone scope unchanged.
|
||||
- Roadmap carryover remains `milestone-task=matrix-lock` and approved SDD scenario S03; this follow-up repairs the reusable seed contract but does not claim the concrete nine-cell immutable manifest evidence.
|
||||
|
||||
## Finding Resolution Map
|
||||
|
||||
| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands |
|
||||
|---------|-------------------|------------|--------------|------|----------------------|---------------------|
|
||||
| Required R1 | Fresh reproducer: `schema_fullmatch=False` and `loader_accepted='seed-a\n'`. | `_require_pattern` uses `pattern.match(s)`, whose `$` can stop before one terminal newline; the invalid seed corpus omits that value. | Use `pattern.fullmatch(s)` in `scripts/agent_benchmark/manifest.py` and add `"seed-a\n"` to the invalid seed corpus in `scripts/agent_benchmark/manifest_test.py`. Keep schema and ordering behavior unchanged. | `direct-fix` | The validator implementation and regression input change, so verification no longer repeats against the defective precondition. | Named parity test; full manifest suite; three example validations; manifest/attempt/connectivity suite; `git diff --check`. |
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_1.log`
|
||||
- `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_1.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, lock released, no user review.
|
||||
- First-line scope: `milestone-task=matrix-lock`.
|
||||
- Target: S03 requires the immutable scored manifest to retain a deterministic execution-order seed; the Evidence Map links that evidence to `matrix-lock`.
|
||||
- This checklist repairs the reusable seed grammar needed by S03. The dependent locked-manifest packet still owns concrete C01-C09 evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No external handoff or external runner is required. Repository-native local verification is authoritative.
|
||||
- Fresh review evidence: 111 manifest tests passed, the named slot test passed, three shipped manifests validated, the combined suite passed 184 tests, and `git diff --check` exited 0.
|
||||
- The focused reproducer accepted `"seed-a\n"` in the loader while the schema pattern's full-string check returned false.
|
||||
- Constraints: preserve valid seed behavior, omission compatibility, seeded rank, digest serialization, `RunStore.slots`, fixture bytes, and public schema.
|
||||
- Confidence: high. The defect is localized to one shared pattern helper and one missing regression value; all helper patterns are intended as whole-string token validators.
|
||||
- External verification preflight: not applicable.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing seed tests cover empty, uppercase, dotted, oversized, boolean, and null values.
|
||||
- Missing: the terminal-newline value that distinguishes `Pattern.match` from `Pattern.fullmatch`.
|
||||
- Add that value to the existing schema/loader parity corpus; no new test class or external fixture is required.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- `_require_pattern` remains the shared private validator for fixture version, model/effort/route tokens, viewport ids, cell ids, and `execution_order_seed`. The change tightens those existing whole-string grammars consistently.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
This is one compact atomic bug fix: the validator semantic change and its regression must land together. It remains child `01_execution_order_contract`; no new dependency or sibling split is introduced.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not change `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`; its seed grammar is already correct.
|
||||
- Do not change seeded ranking, canonical serialization/digest, `RunStore.slots`, attempt state, example manifests, contracts, living spec, or roadmap.
|
||||
- Do not broaden the packet into generic path normalization; only the shared regex token helper and the exact seed regression are writable production/test targets.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`; `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all true; `ownership_closed=true` through direct-fix R1.
|
||||
- Build scores: `1+0+2+1+1=G05`; base `local-fit`, final `recovery-boundary` because `review_rework_count=1` and `evidence_integrity_failure=true`; lane `cloud`, catalog `worker/cloud/G05`, filename `PLAN-cloud-G05.md`.
|
||||
- Review closures are all true; scores `1+0+2+1+1=G05`; route `official-review`, lane `cloud`, catalog `review/cloud/G05`, filename `CODE_REVIEW-cloud-G05.md`.
|
||||
- `large_indivisible_context=false`; matched loop risks: `boundary_contract`, `structured_interpretation` (`count=2`); `risk_boundary_matched=false`; `recovery_boundary_matched=true`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REVIEW_API-1] Enforce full-string manifest token validation and add the terminal-newline seed regression without changing schema, order, digest, or slot behavior.
|
||||
- [ ] [REVIEW_API-2] Run the focused parity test, full manifest suite, shipped manifest validations, combined benchmark regression suite, and whitespace check freshly.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REVIEW_API-1] Require complete token matches
|
||||
|
||||
**Problem**
|
||||
|
||||
At `scripts/agent_benchmark/manifest.py:192-198`, `_require_pattern` calls `pattern.match(s)`. With the anchored patterns used by the manifest, Python accepts a match immediately before a final newline, so `execution_order_seed="seed-a\n"` reaches ordering and digest code even though it is not a valid bounded token.
|
||||
|
||||
**Solution**
|
||||
|
||||
Apply the official review's selected fix exactly.
|
||||
|
||||
Before (`scripts/agent_benchmark/manifest.py:192-198`):
|
||||
|
||||
```python
|
||||
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if not pattern.match(s):
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' does not match required pattern"
|
||||
)
|
||||
return s
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if pattern.fullmatch(s) is None:
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' does not match required pattern"
|
||||
)
|
||||
return s
|
||||
```
|
||||
|
||||
Add `"seed-a\n"` to the existing `invalid_seeds` tuple in `TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds`. Do not change the schema pattern or add an alternate validator.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: replace prefix-capable pattern matching with full-string matching.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add the terminal-newline seed to the existing parity corpus.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the regression in the existing named parity test. It must prove the tracked pattern's full-string interpretation rejects the seed and `load_manifest` raises `ManifestValidationError` for the same value. Existing valid/boundary tests cover compatibility.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: both commands exit 0; the focused test rejects `"seed-a\n"` and the full manifest suite remains green.
|
||||
|
||||
### [REVIEW_API-2] Re-run the benchmark regression boundary
|
||||
|
||||
**Problem**
|
||||
|
||||
`_require_pattern` is shared by multiple manifest token fields, so the one-line fix must be checked against the full loader and the benchmark consumers that bind the manifest digest and matrix order.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run the exact fresh commands below after REVIEW_API-1. Do not alter product behavior or replace the failing value with a sanitized seed.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md`: record fresh command output and implementation notes.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional test file is needed beyond REVIEW_API-1. The shipped manifests and combined deterministic suite are the regression oracle; no external provider process may run.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: three validation success lines, every test reports `OK`, and `git diff --check` is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/manifest.py` | REVIEW_API-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | REVIEW_API-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md` | REVIEW_API-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0`. Fresh output is required; cached or summarized output is not accepted.
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.TestSchemaLoaderParity.test_schema_and_loader_reject_same_invalid_execution_order_seeds
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: all commands exit 0; the newline seed is rejected, all three manifests validate, all deterministic benchmark tests report `OK`, and the whitespace check is silent.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
||||
|
|
@ -0,0 +1,230 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=0 tag=API milestone-task=matrix-lock -->
|
||||
|
||||
# Plan - Seeded Benchmark Execution Order Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The approved benchmark SDD requires an immutable execution-order seed for C01-C09, but the loader currently canonicalizes every matrix only by cell id. This packet adds a backward-compatible optional seed whose deterministic order is part of the normalized manifest and is consumed by the existing slot allocator.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=matrix-lock`.
|
||||
- Target scenario: S03 requires `repetitions=1`, an execution-order seed, fresh session/setup-cache policy, timeout, and expected bindings in an immutable scored manifest.
|
||||
- Evidence Map row: S03 requires an immutable scored manifest and order seed, linked to `matrix-lock` evidence.
|
||||
- This packet implements only the reusable seed/order contract. The dependent locked-manifest packet supplies the concrete nine-cell S03 evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the loader/schema/tests above and from fresh commands run at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
- Current baseline passed all three example `validate` commands and `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` (`Ran 178 tests`, `OK`).
|
||||
- `Manifest.matrix` is sorted by id at `manifest.py:625-627`; `RunStore.slots` iterates that tuple at `attempts.py:679-681`. Therefore the loader can own the seeded order without a second scheduler or changes to run-state code.
|
||||
- The manifest schema is a public tracked config contract, so normal, omission-compatibility, permutation, invalid-token, digest, and slot-order tests are required. Fresh test output is required; cached output is not accepted.
|
||||
- No external verification is required for this packet.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing tests cover cell-id canonicalization and repetitions, but no field represents an execution-order seed.
|
||||
- No test proves that two differently ordered JSON inputs with the same explicit seed produce the same matrix order and digest.
|
||||
- No test proves that omitted seed behavior and existing example digests remain backward compatible.
|
||||
- No test binds seeded loader order to the slot sequence.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- `Manifest(...)` constructor call sites in `manifest.py` and tests must be updated only as required by the new defaulted field; keep the field optional to avoid unrelated call-site churn.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split child `01_execution_order_contract`. Its stable contract is: one optional seed deterministically canonicalizes `Manifest.matrix`, enters the digest only when explicitly present, and is consumed unchanged by `RunStore.slots`.
|
||||
- PASS evidence is loader/schema regression coverage plus the slot-order test.
|
||||
- It is independent of `02_route_preflight_contract`; neither packet claims the other's files. `03+01,02_locked_benchmark_manifest` depends on both.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not create the C01-C09 manifest here; the dependent integration packet owns it.
|
||||
- Do not change preflight observation coverage, caller adapters, run-record schemas, pipeline version, fixture bytes, contracts, or IOP configuration.
|
||||
- Do not reorder manifests that omit the field and do not rewrite the existing example manifests solely to exercise an optional field.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`.
|
||||
- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
|
||||
- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
|
||||
- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Add one canonical seeded-order contract
|
||||
|
||||
**Problem**
|
||||
|
||||
`Manifest` has no seed field (`manifest.py:112-126`), the top-level allowlist accepts only `repetitions` as optional (`manifest.py:681-694`), and `_validate_matrix` always sorts by id (`manifest.py:611-627`). Consequently S03 cannot persist or reproduce a deliberately seeded C01-C09 order.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep pipeline version 2 backward compatible. Add `execution_order_seed: str | None = None` to `Manifest`, accept it as an optional top-level field using the existing cell-id token constraints, and add the same optional constraint to the JSON schema. When absent, preserve exact cell-id sorting and omit the field from canonical serialization so prior normalized digests remain unchanged. When present, order cells by a fixed domain-separated SHA-256 rank with cell id as the collision tie-breaker, include the seed in canonical serialization, and therefore bind it into `digest_manifest_and_resolved_inputs`.
|
||||
|
||||
Before (`scripts/agent_benchmark/manifest.py:625-627`):
|
||||
|
||||
```python
|
||||
# Sort cells by id for canonical ordering
|
||||
cells.sort(key=lambda c: c.id)
|
||||
return tuple(cells)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]:
|
||||
rank = hashlib.sha256(
|
||||
b"iop-benchmark-order-v1\0" + seed.encode("ascii") + b"\0" + cell.id.encode("ascii")
|
||||
).digest()
|
||||
return rank, cell.id
|
||||
|
||||
# No seed preserves the pipeline-v2 cell-id order. An explicit seed is stable
|
||||
# across JSON input permutations and platforms.
|
||||
cells.sort(
|
||||
key=(lambda cell: cell.id)
|
||||
if execution_order_seed is None
|
||||
else (lambda cell: _execution_order_key(execution_order_seed, cell))
|
||||
)
|
||||
```
|
||||
|
||||
Construct the canonical dict first, then conditionally add `execution_order_seed`; never serialize a synthetic default. Add comments documenting the domain string and compatibility behavior.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: model, validate, normalize, order, and digest the explicit seed.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: accept only the optional bounded seed token while retaining `additionalProperties: false`.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add exact regression tests named `test_execution_order_seed_is_canonical_and_permutation_stable`, `test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract`, and schema/loader invalid-seed parity coverage.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests. Use at least three cells whose SHA-256 seeded order differs from lexical id order. Assert input permutation independence, same-seed repeatability, different-seed digest/order differentiation, invalid empty/oversize/non-token rejection, and omission compatibility. Existing example validation must remain unchanged.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: all manifest loader/schema/digest tests pass freshly.
|
||||
|
||||
### [API-2] Bind slot allocation to the canonical matrix order
|
||||
|
||||
**Problem**
|
||||
|
||||
`RunStore.slots` already iterates `manifest.matrix` (`attempts.py:679-681`), but the current test only checks repetitions for a one-cell manifest (`attempts_test.py:457`). There is no regression proof that a seeded multi-cell order reaches attempt allocation without being re-sorted.
|
||||
|
||||
**Solution**
|
||||
|
||||
Do not modify `RunStore.slots`. Extend the attempts-test manifest helper to accept an explicit seed and multiple cells, then assert the exact sequence is the loader's seeded `Manifest.matrix` order with repetitions `1..N` adjacent for each cell.
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:679-681`):
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def slots(manifest: Manifest) -> tuple[Slot, ...]:
|
||||
return tuple(Slot(cell.id, repetition) for cell in manifest.matrix for repetition in range(1, manifest.repetitions + 1))
|
||||
```
|
||||
|
||||
After: keep this production code unchanged; lock its behavior with `test_slots_follow_seeded_manifest_order_before_repetitions`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: add the seeded multi-cell slot-order regression without changing production attempts code.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named test. Assert both `(cell_id, repetition)` tuples and that the first occurrence order equals `manifest.matrix`. This prevents a future alphabetical sort in the store from silently defeating the seed.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
```
|
||||
|
||||
Expected: the exact seeded slot sequence passes.
|
||||
|
||||
### [API-3] Preserve the complete benchmark baseline
|
||||
|
||||
**Problem**
|
||||
|
||||
The field touches canonical JSON and a public schema; focused tests alone could miss example or integration drift.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run fresh benchmark suites and all shipped validation fixtures after the code and tests are complete. Do not accept cached output.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G08.md`: record actual implementation decisions and full command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional test file beyond API-1/API-2. The full existing suite is the regression oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: three `ok: manifest is valid` lines, all tests `OK`, and no whitespace errors.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md` | API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0`; fresh output is required and cached test output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits 0, the three manifests validate, all benchmark tests report `OK`, and `git diff --check` is silent.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,237 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/01_execution_order_contract plan=1 tag=API milestone-task=matrix-lock -->
|
||||
|
||||
# Plan - Seeded Benchmark Execution Order Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The approved benchmark SDD requires an immutable execution-order seed for C01-C09, but the loader currently canonicalizes every matrix only by cell id. This packet adds a backward-compatible optional seed whose deterministic order is part of the normalized manifest and is consumed by the existing slot allocator.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/plan_local_G08_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/code_review_cloud_G08_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found one stale dependency statement: it named a nonexistent integration child and incorrectly implied that child depended directly on both indices 01 and 02.
|
||||
- Replan keeps the implementation contract unchanged and corrects the DAG to `01 -> 03+01`, `02` independent, and `(02,03) -> 04+02,03`. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=matrix-lock`.
|
||||
- Target scenario: S03 requires `repetitions=1`, an execution-order seed, fresh session/setup-cache policy, timeout, and expected bindings in an immutable scored manifest.
|
||||
- Evidence Map row: S03 requires an immutable scored manifest and order seed, linked to `matrix-lock` evidence.
|
||||
- This packet implements only the reusable seed/order contract. The dependent locked-manifest packet supplies the concrete nine-cell S03 evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the loader/schema/tests above and from fresh commands run at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
- Current baseline passed all three example `validate` commands and `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` (`Ran 178 tests`, `OK`).
|
||||
- `Manifest.matrix` is sorted by id at `manifest.py:625-627`; `RunStore.slots` iterates that tuple at `attempts.py:679-681`. Therefore the loader can own the seeded order without a second scheduler or changes to run-state code.
|
||||
- The manifest schema is a public tracked config contract, so normal, omission-compatibility, permutation, invalid-token, digest, and slot-order tests are required. Fresh test output is required; cached output is not accepted.
|
||||
- No external verification is required for this packet.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing tests cover cell-id canonicalization and repetitions, but no field represents an execution-order seed.
|
||||
- No test proves that two differently ordered JSON inputs with the same explicit seed produce the same matrix order and digest.
|
||||
- No test proves that omitted seed behavior and existing example digests remain backward compatible.
|
||||
- No test binds seeded loader order to the slot sequence.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- `Manifest(...)` constructor call sites in `manifest.py` and tests must be updated only as required by the new defaulted field; keep the field optional to avoid unrelated call-site churn.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split child `01_execution_order_contract`. Its stable contract is: one optional seed deterministically canonicalizes `Manifest.matrix`, enters the digest only when explicitly present, and is consumed unchanged by `RunStore.slots`.
|
||||
- PASS evidence is loader/schema regression coverage plus the slot-order test.
|
||||
- It is independent of `02_route_preflight_contract`; neither packet claims the other's files. Only `03+01_rubric_version_contract` depends directly on this packet. The final `04+02,03_locked_benchmark_manifest` packet depends on indices 02 and 03.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not create the C01-C09 manifest here; the dependent integration packet owns it.
|
||||
- Do not change preflight observation coverage, caller adapters, run-record schemas, pipeline version, fixture bytes, contracts, or IOP configuration.
|
||||
- Do not reorder manifests that omit the field and do not rewrite the existing example manifests solely to exercise an optional field.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`.
|
||||
- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add the optional, backward-compatible execution-order seed to loader normalization, schema validation, and manifest digest behavior; verify normal, boundary, omission, and permutation cases.
|
||||
- [ ] [API-2] Prove `RunStore.slots` consumes the seeded canonical matrix order with repetitions nested per cell and does not add a second ordering source.
|
||||
- [ ] [API-3] Run the focused and full benchmark regression commands and confirm all shipped example manifests still validate.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Add one canonical seeded-order contract
|
||||
|
||||
**Problem**
|
||||
|
||||
`Manifest` has no seed field (`manifest.py:112-126`), the top-level allowlist accepts only `repetitions` as optional (`manifest.py:681-694`), and `_validate_matrix` always sorts by id (`manifest.py:611-627`). Consequently S03 cannot persist or reproduce a deliberately seeded C01-C09 order.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep pipeline version 2 backward compatible. Append `execution_order_seed: str | None = None` after the existing required `digest` field in `Manifest` so dataclass default ordering stays valid, accept it as an optional top-level field using the existing cell-id token constraints, and add the same optional constraint to the JSON schema. Pass an explicit seed into both loader-created `Manifest` values before digesting. When absent, preserve exact cell-id sorting and omit the field from canonical serialization so prior normalized digests remain unchanged. When present, order cells by a fixed domain-separated SHA-256 rank with cell id as the collision tie-breaker, include the seed in canonical serialization, and therefore bind it into `digest_manifest_and_resolved_inputs`.
|
||||
|
||||
Before (`scripts/agent_benchmark/manifest.py:625-627`):
|
||||
|
||||
```python
|
||||
# Sort cells by id for canonical ordering
|
||||
cells.sort(key=lambda c: c.id)
|
||||
return tuple(cells)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]:
|
||||
rank = hashlib.sha256(
|
||||
b"iop-benchmark-order-v1\0" + seed.encode("ascii") + b"\0" + cell.id.encode("ascii")
|
||||
).digest()
|
||||
return rank, cell.id
|
||||
|
||||
# No seed preserves the pipeline-v2 cell-id order. An explicit seed is stable
|
||||
# across JSON input permutations and platforms.
|
||||
cells.sort(
|
||||
key=(lambda cell: cell.id)
|
||||
if execution_order_seed is None
|
||||
else (lambda cell: _execution_order_key(execution_order_seed, cell))
|
||||
)
|
||||
```
|
||||
|
||||
Construct the canonical dict first, then conditionally add `execution_order_seed`; never serialize a synthetic default. Add comments documenting the domain string and compatibility behavior.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: model, validate, normalize, order, and digest the explicit seed.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: accept only the optional bounded seed token while retaining `additionalProperties: false`.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add exact regression tests named `test_execution_order_seed_is_canonical_and_permutation_stable`, `test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract`, and schema/loader invalid-seed parity coverage.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests. Use at least three cells whose SHA-256 seeded order differs from lexical id order. Assert input permutation independence, same-seed repeatability, different-seed digest/order differentiation, invalid empty/oversize/non-token rejection, and omission compatibility. Existing example validation must remain unchanged.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: all manifest loader/schema/digest tests pass freshly.
|
||||
|
||||
### [API-2] Bind slot allocation to the canonical matrix order
|
||||
|
||||
**Problem**
|
||||
|
||||
`RunStore.slots` already iterates `manifest.matrix` (`attempts.py:679-681`), but the current test only checks repetitions for a one-cell manifest (`attempts_test.py:457`). There is no regression proof that a seeded multi-cell order reaches attempt allocation without being re-sorted.
|
||||
|
||||
**Solution**
|
||||
|
||||
Do not modify `RunStore.slots`. Extend the attempts-test manifest helper to accept an explicit seed and multiple cells, then assert the exact sequence is the loader's seeded `Manifest.matrix` order with repetitions `1..N` adjacent for each cell.
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:679-681`):
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def slots(manifest: Manifest) -> tuple[Slot, ...]:
|
||||
return tuple(Slot(cell.id, repetition) for cell in manifest.matrix for repetition in range(1, manifest.repetitions + 1))
|
||||
```
|
||||
|
||||
After: keep this production code unchanged; lock its behavior with `test_slots_follow_seeded_manifest_order_before_repetitions`.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts_test.py`: add the seeded multi-cell slot-order regression without changing production attempts code.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named test. Assert both `(cell_id, repetition)` tuples and that the first occurrence order equals `manifest.matrix`. This prevents a future alphabetical sort in the store from silently defeating the seed.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
```
|
||||
|
||||
Expected: the exact seeded slot sequence passes.
|
||||
|
||||
### [API-3] Preserve the complete benchmark baseline
|
||||
|
||||
**Problem**
|
||||
|
||||
The field touches canonical JSON and a public schema; focused tests alone could miss example or integration drift.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run fresh benchmark suites and all shipped validation fixtures after the code and tests are complete. Do not accept cached output.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G08.md`: record actual implementation decisions and full command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional test file beyond API-1/API-2. The full existing suite is the regression oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: three `ok: manifest is valid` lines, all tests `OK`, and no whitespace errors.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1 |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md` | API-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0`; fresh output is required and cached test output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test.AttemptStoreTest.test_slots_follow_seeded_manifest_order_before_repetitions
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits 0, the three manifests validate, all benchmark tests report `OK`, and `git diff --check` is silent.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=0 tag=API milestone-task=route-readiness -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Require exact all-cell preflight evidence | [ ] |
|
||||
| API-2 Isolate live agy preflight state per cell | [ ] |
|
||||
| API-3 Replace obsolete direct-only assertions | [ ] |
|
||||
| API-4 Preserve append-only and redaction invariants | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
|
||||
- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
|
||||
- [ ] [API-3] Replace direct-only contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
|
||||
- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the durable result set equals all `manifest.matrix` cells in canonical order on both write and read.
|
||||
- Confirm preset blockers publish exact closed taxonomy and allocate no attempt.
|
||||
- Confirm agy runtime state is keyed by cell id and a blocked re-preflight clears stale ready state.
|
||||
- Confirm no adapter-specific endpoint, token, or raw caller value enters evidence.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### API-1/API-2 Focused Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-3 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-4 and Final Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,254 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=1 tag=API milestone-task=route-readiness -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found that it changed public preflight semantics but omitted the project benchmark skill and its contract tests, both of which explicitly require the obsolete direct-only behavior.
|
||||
- Replan makes this packet the sole owner of all-cell preflight documentation/tests, adds the concrete agy preflight type source to analysis, and leaves rubric-selection wording for the dependent final-manifest packet. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Require exact all-cell preflight evidence | [x] |
|
||||
| API-2 Isolate live agy preflight state per cell | [x] |
|
||||
| API-3 Replace obsolete direct-only code and skill assertions | [x] |
|
||||
| API-4 Preserve append-only and redaction invariants | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
|
||||
- [x] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
|
||||
- [x] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
|
||||
- [x] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`.
|
||||
- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` and update this checklist at the final archive path.
|
||||
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Preflight collection and durable record validation use the same immutable `manifest.matrix` tuple, so cardinality, identity, and order are checked against every declared direct or execution-preset cell.
|
||||
- `_LiveAdapter` clears prior admission before every cell preflight and publishes agy runtime state only after the complete live/config observation is ready. Invocation retrieves that state once by exact cell id and rejects missing or mismatched bindings.
|
||||
- The project skill and its mutation tests now require affirmative all-cell live evidence while retaining closed blocker taxonomy, zero-attempt blockers, and the rule that local fixture validation alone is not live readiness.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the durable result set equals all `manifest.matrix` cells in canonical order on both write and read.
|
||||
- Confirm preset blockers publish exact closed taxonomy and allocate no attempt.
|
||||
- Confirm agy runtime state is keyed by cell id and a blocked re-preflight clears stale ready state.
|
||||
- Confirm the project benchmark skill and contract tests require one observation for every immutable matrix cell and contain no direct-only/preset-local exception.
|
||||
- Confirm no adapter-specific endpoint, token, or raw caller value enters evidence.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### API-1/API-2 Focused Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
....
|
||||
----------------------------------------------------------------------
|
||||
Ran 4 tests in 2.414s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Exit code: `0`.
|
||||
|
||||
_Reviewer fresh rerun:_
|
||||
|
||||
```text
|
||||
....
|
||||
----------------------------------------------------------------------
|
||||
Ran 4 tests in 2.492s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Exit code: `0`.
|
||||
|
||||
### API-3 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
.................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 81 tests in 14.040s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Exit code: `0`.
|
||||
|
||||
_Reviewer fresh rerun:_
|
||||
|
||||
```text
|
||||
.................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 81 tests in 13.933s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Exit code: `0`.
|
||||
|
||||
### API-4 and Final Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
.........................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 233 tests in 47.536s
|
||||
|
||||
OK
|
||||
|
||||
$ git diff --check
|
||||
(no output)
|
||||
```
|
||||
|
||||
Both commands exited `0`.
|
||||
|
||||
_Reviewer fresh rerun:_
|
||||
|
||||
```text
|
||||
$ python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
.........................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 233 tests in 47.606s
|
||||
|
||||
OK
|
||||
|
||||
$ git diff --check
|
||||
(no output)
|
||||
```
|
||||
|
||||
Both commands exited `0`.
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
### Overall Verdict
|
||||
|
||||
PASS
|
||||
|
||||
### Dimension Assessment
|
||||
|
||||
| Dimension | Assessment | Evidence |
|
||||
|-----------|------------|----------|
|
||||
| Correctness | Pass | Full-matrix collection/write/read order, blocker handling, and cell-keyed agy admission matched the planned invariants. |
|
||||
| Completeness | Pass | API-1 through API-4 are implemented, and the public skill contract was updated to all-cell semantics. |
|
||||
| Test coverage | Pass | Fresh focused 4-test, connectivity/skill 81-test, and full 233-test suites passed. |
|
||||
| API contract | Pass | No caller wire/config contract changed; direct and execution-preset bindings remain exact and fail closed without substitution. |
|
||||
| Code quality | Pass | No stale direct-only production reference remains after the review-time CLI docstring repair; `git diff --check` passed. |
|
||||
| Implementation deviation | Pass | Planned implementation scope is present; unrelated sibling execution-order work was excluded from this verdict. |
|
||||
| Verification trust | Pass | Reviewer reran every planned command from the current worktree and reproduced the recorded success. |
|
||||
| Spec conformance | Pass | SDD S02 all-cell redacted preflight evidence and the `route-readiness` Evidence Map are satisfied by deterministic contract tests without claiming live credential readiness. |
|
||||
|
||||
### Findings
|
||||
|
||||
- Nit (repaired): `scripts/agent_comparison_benchmark.py:14` still described exit 0 as requiring every “direct preflight cell”; review changed it to “every matrix preflight cell” to match the implemented public semantics.
|
||||
- Required: None.
|
||||
- Suggested: None.
|
||||
|
||||
### Routing Signals
|
||||
|
||||
- `review_rework_count=0`
|
||||
- `evidence_integrity_failure=false`
|
||||
|
||||
### Next Step
|
||||
|
||||
- PASS: archive this pair, write `complete.log`, and emit the milestone completion metadata for runtime aggregation.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=1 tag=API milestone-task=route-readiness -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/02_route_preflight_contract
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
계획 개정 2회와 공식 코드 리뷰 1회를 거쳐 전체 matrix cell preflight 및 cell별 agy 상태 격리 계약을 구현했고 최종 판정은 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | 재계획 | Epic self-review에서 project benchmark skill과 contract test 범위 누락을 발견해 구현 전 계획을 보강했다. |
|
||||
| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | PASS | 전체 cell의 append-only preflight, blocker 무할당, cell별 agy 상태 격리 및 skill 계약을 확인했다. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- preflight 수집과 durable record 검증을 immutable `manifest.matrix` 전체 및 canonical order 기준으로 통일했다.
|
||||
- agy live preflight/admission 상태를 cell id별로 격리하고 재-preflight 시 stale 상태를 제거했다.
|
||||
- mixed/preset-only 성공·blocker·corruption·cell isolation 회귀 테스트와 project benchmark skill 계약을 all-cell 의미로 갱신했다.
|
||||
- 리뷰 중 공개 CLI 종료 코드 설명의 남은 direct-only 문구를 matrix 전체 의미로 정정했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state` - PASS; 4 tests, 2.492s.
|
||||
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 81 tests, 13.933s.
|
||||
- `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test` - PASS; 233 tests, 47.606s.
|
||||
- `git diff --check` - PASS; 출력 없음.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- 없음
|
||||
|
|
@ -0,0 +1,276 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=0 tag=API milestone-task=route-readiness -->
|
||||
|
||||
# Plan - All-Cell Route Preflight Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The benchmark currently validates preset capability locally but deliberately omits preset cells from durable preflight. A mixed direct/preset manifest therefore records a misleading ready subset and allocates no attempts, while the live `agy` adapter retains only the last cell's preflight state. Route readiness for C01-C09 requires every declared cell to be observed and bound independently before any attempt starts.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=route-readiness`.
|
||||
- Target scenario: S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker.
|
||||
- Evidence Map row: S02 requires a redacted C01-C09 preflight matrix with auth/route/effort/terminal evidence.
|
||||
- The checklist therefore replaces direct-only subset semantics with exact all-cell evidence and adds per-cell caller-state regressions. Live credentials remain the dependent manifest packet's external gate.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the attempts/live adapter paths and network-free integration tests above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three example manifests validated and the 178-test manifest/attempts/connectivity suite passed.
|
||||
- Current tests explicitly lock the old limitation: `test_cli_mixed_manifest_never_invokes_unobserved_preset_cells`, `test_generic_preset_cells_are_local_contract_only`, and `test_generic_preset_only_public_preflight_fails_closed_without_run_state` (`connectivity_integration_test.py:1009-1214`). These must be replaced, not preserved.
|
||||
- `collect_preflight_observations` skips non-direct cells (`attempts.py:386-423`), durable read/write validates against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`), and public preflight rejects an empty direct subset (`attempts.py:1869-1881`).
|
||||
- `_LiveAdapter` stores one `_agy_preflight` value for every cell (`live_iop.py:843-902`) and reuses it at invocation (`live_iop.py:935-950`), so multiple agy routes can cross-bind.
|
||||
- This packet uses only network-free adapters and mocks. Fresh output is required; cached output is not accepted.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing tests cover direct cells and intentionally assert that preset cells are omitted.
|
||||
- No mixed manifest test expects both direct and preset results in the same append-only record and then executes both.
|
||||
- No preset-only public preflight test expects a durable ready or blocker record.
|
||||
- No live-adapter test preflights two agy cells and proves each invocation receives its own preflight capability/runtime binding.
|
||||
- Existing corruption and secret-redaction coverage is broad and must continue to pass after the result-set invariant changes.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- Rename private `RunStore._direct_cells` to `_preflight_cells` or remove the helper. Its only references are `attempts.py:563` and `attempts.py:632`; update both together.
|
||||
- Rename private `_LiveAdapter._agy_preflight` to `_agy_preflights`; all references are `live_iop.py:858`, `885`, `893-894`, and `936-949`.
|
||||
- No public import or CLI symbol is renamed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split child `02_route_preflight_contract`. Its stable invariant is: the preflight result set equals the entire immutable matrix, and every admitted live invocation consumes the preflight state for its own cell id.
|
||||
- PASS evidence is all-cell append/read corruption coverage, mixed/preset-only CLI coverage, and a two-cell agy state test.
|
||||
- It is independent of `01_execution_order_contract`; it consumes `manifest.matrix` order without owning that order. `03+01,02_locked_benchmark_manifest` depends on both.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not add the final nine-cell manifest or an execution-order field here.
|
||||
- Do not change caller wire formats, IOP contracts, provider config, credentials, retry policy, attempt allocation semantics, or scoring.
|
||||
- Do not synthesize readiness for missing routes. Every preset must pass the same live config/catalog/capability checks and retain exact `registration_required` or `implementation_gap` taxonomy.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+2+2+1+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`.
|
||||
- Review closures: all closed. Scores `2+2+2+1+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but does not replace grade basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
|
||||
- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
|
||||
- [ ] [API-3] Replace direct-only contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
|
||||
- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Require exact all-cell preflight evidence
|
||||
|
||||
**Problem**
|
||||
|
||||
The collector skips every `execution_preset` (`attempts.py:390-410`), and both record validation and publication compare against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`). A mixed manifest can publish `status=ready` for only its direct subset, after which `run_slots` correctly refuses to execute because the record does not cover the matrix.
|
||||
|
||||
**Solution**
|
||||
|
||||
Validate capabilities and call `adapter.preflight(cell)` for every cell in canonical `manifest.matrix` order. Rename the private helper to `_preflight_cells` returning the full matrix, and use it for durable result cardinality, identity, order, and publication. Remove the direct-cell-empty special case from `preflight_manifest`; a valid manifest is already non-empty. Preserve canonical evidence validation and closed aggregate taxonomy.
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:386-423`):
|
||||
|
||||
```python
|
||||
"""Validate the full registry, then probe direct cells in manifest order."""
|
||||
...
|
||||
for cell in manifest.matrix:
|
||||
if cell.iop.route_kind != "direct":
|
||||
continue
|
||||
observation = adapters[cell.caller].preflight(cell)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
"""Validate the full registry, then probe every cell in manifest order."""
|
||||
...
|
||||
for cell in manifest.matrix:
|
||||
observation = adapters[cell.caller].preflight(cell)
|
||||
...
|
||||
observations[cell.id] = observation
|
||||
```
|
||||
|
||||
Use the exact same full tuple on write and read so missing, extra, reordered, or foreign results fail closed. If any cell is blocked, publish the all-cell record, allocate zero attempts, and return the existing closed CLI status.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: collect, write, read, and aggregate the exact full matrix.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: update the public preflight/CLI contract tests for all-cell behavior.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write/replace tests. Assert preset-only ready records are durable, preset-only blockers are durable with zero cells directory, mixed records include both cells in matrix order, a ready mixed run invokes both exactly once, and missing/extra/reordered record payloads remain rejected.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation
|
||||
```
|
||||
|
||||
Expected: all named all-cell tests pass.
|
||||
|
||||
### [API-2] Isolate live agy preflight state per cell
|
||||
|
||||
**Problem**
|
||||
|
||||
`_LiveAdapter._agy_preflight` is overwritten on each preflight (`live_iop.py:858`, `885`) and the last value is used for every agy invocation (`live_iop.py:935-949`). C03 and C07 in the same matrix can therefore use the wrong route observation/capability state.
|
||||
|
||||
**Solution**
|
||||
|
||||
Replace the scalar with `dict[str, AgyIopPreflight]` (use the concrete existing return type if exported; otherwise retain `Any` only at the value boundary). At the start of every cell preflight, remove that cell's prior agy state and admitted binding. Store both only after the current result is ready. At invocation, retrieve by `cell.id` once and pass that same object to `build_agy_invocation`, the invoker, and `observed_result`; reject absent state before building a spec.
|
||||
|
||||
Before (`scripts/agent_benchmark/live_iop.py:935-950`):
|
||||
|
||||
```python
|
||||
if self.caller == AGY_CALLER:
|
||||
if self._agy_preflight is None:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
...
|
||||
result = self._invokers.agy(spec, parser, self._agy_preflight, ...)
|
||||
observed = parser.observed_result(self._agy_preflight.capability, result)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
if self.caller == AGY_CALLER:
|
||||
agy_preflight = self._agy_preflights.get(cell.id)
|
||||
if agy_preflight is None:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
...
|
||||
result = self._invokers.agy(spec, parser, agy_preflight, ...)
|
||||
observed = parser.observed_result(agy_preflight.capability, result)
|
||||
```
|
||||
|
||||
Do not persist raw endpoint or credential values; the new map remains process-local runtime state.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: clear, store, and retrieve agy preflight/admission by exact cell id.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: add `test_live_agy_multiple_cells_consume_their_own_preflight_state` with two distinct route kinds/ids and mocked invokers.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named network-free regression. Preflight direct and preset agy cells with distinguishable observations, invoke in reverse preflight order, and assert each invocation receives its own object and binding. Add a re-preflight blocker case proving stale ready state cannot be invoked.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
```
|
||||
|
||||
Expected: both reverse-order isolation and stale-state rejection pass.
|
||||
|
||||
### [API-3] Replace the obsolete direct-only assertions
|
||||
|
||||
**Problem**
|
||||
|
||||
Three integration tests at `connectivity_integration_test.py:1009-1214` intentionally require preset omission. Leaving them unchanged would preserve the exact S02 gap this packet closes.
|
||||
|
||||
**Solution**
|
||||
|
||||
Replace those tests with the all-cell cases named in API-1 and update exact adapter call/result assertions everywhere affected. Retain the existing three-caller direct case, registration-vs-implementation taxonomy, append-only corruption, missing-adapter, CLI redaction, and zero-attempt blocker assertions.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove only obsolete local-only expectations and add full-matrix expectations.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests, do not skip. All expected result ids must be derived from `manifest.matrix` and compared in stable order. No test may invent a ready observation without going through the typed fake or live adapter boundary.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: all network-free public connectivity tests pass.
|
||||
|
||||
### [API-4] Preserve append-only and redaction invariants
|
||||
|
||||
**Problem**
|
||||
|
||||
Expanding the record set changes durable cardinality and exercises more caller variants; regressions could appear outside the focused cases.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run the complete manifest/attempt/connectivity suites and `git diff --check`. Record full output in the review stub.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G09.md`: record implementation decisions, deviations, and actual command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No further files. Existing corruption, retry/resume, measurement, web-validation, secret-redaction, and live-adapter tests provide the whole-suite oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: all tests report `OK`; whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/attempts.py` | API-1 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-2, API-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md` | API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0`; fresh output is required and cached output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits 0, all benchmark suites report `OK`, and no secret/raw runtime values appear in durable-record assertions.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,292 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/02_route_preflight_contract plan=1 tag=API milestone-task=route-readiness -->
|
||||
|
||||
# Plan - All-Cell Route Preflight Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The benchmark currently validates preset capability locally but deliberately omits preset cells from durable preflight. A mixed direct/preset manifest therefore records a misleading ready subset and allocates no attempts, while the live `agy` adapter retains only the last cell's preflight state. Route readiness for C01-C09 requires every declared cell to be observed and bound independently before any attempt starts.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/plan_cloud_G09_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/code_review_cloud_G09_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found that it changed public preflight semantics but omitted the project benchmark skill and its contract tests, both of which explicitly require the obsolete direct-only behavior.
|
||||
- Replan makes this packet the sole owner of all-cell preflight documentation/tests, adds the concrete agy preflight type source to analysis, and leaves rubric-selection wording for the dependent final-manifest packet. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=route-readiness`.
|
||||
- Target scenario: S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker.
|
||||
- Evidence Map row: S02 requires a redacted C01-C09 preflight matrix with auth/route/effort/terminal evidence.
|
||||
- The checklist therefore replaces direct-only subset semantics with exact all-cell evidence and adds per-cell caller-state regressions. Live credentials remain the dependent manifest packet's external gate.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the attempts/live adapter paths and network-free integration tests above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three example manifests validated and the 178-test manifest/attempts/connectivity suite passed.
|
||||
- Current tests explicitly lock the old limitation: `test_cli_mixed_manifest_never_invokes_unobserved_preset_cells`, `test_generic_preset_cells_are_local_contract_only`, and `test_generic_preset_only_public_preflight_fails_closed_without_run_state` (`connectivity_integration_test.py:1009-1214`). These must be replaced, not preserved.
|
||||
- `collect_preflight_observations` skips non-direct cells (`attempts.py:386-423`), durable read/write validates against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`), and public preflight rejects an empty direct subset (`attempts.py:1869-1881`).
|
||||
- `_LiveAdapter` stores one `_agy_preflight` value for every cell (`live_iop.py:843-902`) and reuses it at invocation (`live_iop.py:935-950`), so multiple agy routes can cross-bind.
|
||||
- This packet uses only network-free adapters and mocks. Fresh output is required; cached output is not accepted.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing tests cover direct cells and intentionally assert that preset cells are omitted.
|
||||
- No mixed manifest test expects both direct and preset results in the same append-only record and then executes both.
|
||||
- No preset-only public preflight test expects a durable ready or blocker record.
|
||||
- No live-adapter test preflights two agy cells and proves each invocation receives its own preflight capability/runtime binding.
|
||||
- Existing corruption and secret-redaction coverage is broad and must continue to pass after the result-set invariant changes.
|
||||
- The project benchmark skill and `skill_contract_test.py` explicitly describe/require direct-only preflight, so they would contradict production immediately after this packet unless changed in the same boundary.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- Rename private `RunStore._direct_cells` to `_preflight_cells` or remove the helper. Its only references are `attempts.py:563` and `attempts.py:632`; update both together.
|
||||
- Rename private `_LiveAdapter._agy_preflight` to `_agy_preflights`; all references are `live_iop.py:858`, `885`, `893-894`, and `936-949`.
|
||||
- No public import or CLI symbol is renamed.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split child `02_route_preflight_contract`. Its stable invariant is: the preflight result set equals the entire immutable matrix, and every admitted live invocation consumes the preflight state for its own cell id.
|
||||
- PASS evidence is all-cell append/read corruption coverage, mixed/preset-only CLI coverage, and a two-cell agy state test.
|
||||
- It is independent of `01_execution_order_contract`; it consumes `manifest.matrix` order without owning that order. The final `04+02,03_locked_benchmark_manifest` packet depends on this packet and index 03.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not add the final nine-cell manifest or an execution-order field here.
|
||||
- Do not change caller wire formats, IOP contracts, provider config, credentials, retry policy, attempt allocation semantics, or scoring.
|
||||
- Do not synthesize readiness for missing routes. Every preset must pass the same live config/catalog/capability checks and retain exact `registration_required` or `implementation_gap` taxonomy.
|
||||
- Update only the skill's preflight/run/resume/validation/safety wording here. Do not add the new rubric-selection wording; the dependent final-manifest packet owns that operational transition after the rubric implementation lands.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+2+2+1+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`.
|
||||
- Review closures: all closed. Scores `2+2+2+1+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but does not replace grade basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Make collection and append-only validation require one canonical observation for every matrix cell, including preset-only and mixed manifests, with no attempt allocation on blockers.
|
||||
- [ ] [API-2] Store and consume agy live preflight state by cell id, clearing stale admission on every re-preflight and rejecting missing or mismatched state.
|
||||
- [ ] [API-3] Replace direct-only code and project-skill contract tests with mixed/preset all-cell success, blocker, persistence, execution, and per-cell agy regressions.
|
||||
- [ ] [API-4] Run the focused and full benchmark regression suites and confirm secret-free append-only evidence remains valid.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Require exact all-cell preflight evidence
|
||||
|
||||
**Problem**
|
||||
|
||||
The collector skips every `execution_preset` (`attempts.py:390-410`), and both record validation and publication compare against `_direct_cells` (`attempts.py:514-516`, `563-568`, `632-651`). A mixed manifest can publish `status=ready` for only its direct subset, after which `run_slots` correctly refuses to execute because the record does not cover the matrix.
|
||||
|
||||
**Solution**
|
||||
|
||||
Validate capabilities and call `adapter.preflight(cell)` for every cell in canonical `manifest.matrix` order. Rename the private helper to `_preflight_cells` returning the full matrix, and use it for durable result cardinality, identity, order, and publication. Remove the direct-cell-empty special case from `preflight_manifest`; a valid manifest is already non-empty. Preserve canonical evidence validation and closed aggregate taxonomy.
|
||||
|
||||
Before (`scripts/agent_benchmark/attempts.py:386-423`):
|
||||
|
||||
```python
|
||||
"""Validate the full registry, then probe direct cells in manifest order."""
|
||||
...
|
||||
for cell in manifest.matrix:
|
||||
if cell.iop.route_kind != "direct":
|
||||
continue
|
||||
observation = adapters[cell.caller].preflight(cell)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
"""Validate the full registry, then probe every cell in manifest order."""
|
||||
...
|
||||
for cell in manifest.matrix:
|
||||
observation = adapters[cell.caller].preflight(cell)
|
||||
...
|
||||
observations[cell.id] = observation
|
||||
```
|
||||
|
||||
Use the exact same full tuple on write and read so missing, extra, reordered, or foreign results fail closed. If any cell is blocked, publish the all-cell record, allocate zero attempts, and return the existing closed CLI status.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/attempts.py`: collect, write, read, and aggregate the exact full matrix.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: update the public preflight/CLI contract tests for all-cell behavior.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write/replace tests. Assert preset-only ready records are durable, preset-only blockers are durable with zero cells directory, mixed records include both cells in matrix order, a ready mixed run invokes both exactly once, and missing/extra/reordered record payloads remain rejected.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation
|
||||
```
|
||||
|
||||
Expected: all named all-cell tests pass.
|
||||
|
||||
### [API-2] Isolate live agy preflight state per cell
|
||||
|
||||
**Problem**
|
||||
|
||||
`_LiveAdapter._agy_preflight` is overwritten on each preflight (`live_iop.py:858`, `885`) and the last value is used for every agy invocation (`live_iop.py:935-949`). C03 and C07 in the same matrix can therefore use the wrong route observation/capability state.
|
||||
|
||||
**Solution**
|
||||
|
||||
Replace the scalar with `dict[str, AgyPreflightResult]`, importing the concrete exported dataclass from `agy_iop.py`. At the start of every cell preflight, remove that cell's prior agy state and admitted binding. Store both only after the current result is ready. At invocation, retrieve by `cell.id` once and pass that same object to `build_agy_invocation`, the invoker, and `observed_result`; reject absent state before building a spec.
|
||||
|
||||
Before (`scripts/agent_benchmark/live_iop.py:935-950`):
|
||||
|
||||
```python
|
||||
if self.caller == AGY_CALLER:
|
||||
if self._agy_preflight is None:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
...
|
||||
result = self._invokers.agy(spec, parser, self._agy_preflight, ...)
|
||||
observed = parser.observed_result(self._agy_preflight.capability, result)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
if self.caller == AGY_CALLER:
|
||||
agy_preflight = self._agy_preflights.get(cell.id)
|
||||
if agy_preflight is None:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
...
|
||||
result = self._invokers.agy(spec, parser, agy_preflight, ...)
|
||||
observed = parser.observed_result(agy_preflight.capability, result)
|
||||
```
|
||||
|
||||
Do not persist raw endpoint or credential values; the new map remains process-local runtime state.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: clear, store, and retrieve agy preflight/admission by exact cell id.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: add `test_live_agy_multiple_cells_consume_their_own_preflight_state` with two distinct route kinds/ids and mocked invokers.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named network-free regression. Preflight direct and preset agy cells with distinguishable observations, invoke in reverse preflight order, and assert each invocation receives its own object and binding. Add a re-preflight blocker case proving stale ready state cannot be invoked.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
```
|
||||
|
||||
Expected: both reverse-order isolation and stale-state rejection pass.
|
||||
|
||||
### [API-3] Replace the obsolete direct-only code and skill assertions
|
||||
|
||||
**Problem**
|
||||
|
||||
Three integration tests at `connectivity_integration_test.py:1009-1214` intentionally require preset omission. The project benchmark skill repeats that contract in its preflight procedure, validation, and safety sections, and `_assert_preflight_contract` plus its mutation at `skill_contract_test.py:256-271,786-794` enforce it. Leaving either surface unchanged would preserve or publish the exact S02 gap this packet closes.
|
||||
|
||||
**Solution**
|
||||
|
||||
Replace those integration tests with the all-cell cases named in API-1 and update exact adapter call/result assertions everywhere affected. Update the project skill so preflight, run, and resume describe a fresh append-only observation for every immutable matrix cell, while retaining closed blocker taxonomy and zero-attempt behavior. Rewrite `_assert_preflight_contract` and its mutation to reject any return of direct-only/preset-local language. Retain the existing three-caller direct case, registration-vs-implementation taxonomy, append-only corruption, missing-adapter, CLI redaction, and zero-attempt blocker assertions.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove only obsolete local-only expectations and add full-matrix expectations.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: replace direct-only preflight wording with the exact all-cell durable contract.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require all-cell skill wording and reject restoration of preset-local semantics.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests, do not skip. All expected result ids must be derived from `manifest.matrix` and compared in stable order. No test may invent a ready observation without going through the typed fake or live adapter boundary. Skill tests must assert the affirmative all-cell contract and mutation-fail on the former direct-only statements.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: all network-free public connectivity tests pass.
|
||||
|
||||
### [API-4] Preserve append-only and redaction invariants
|
||||
|
||||
**Problem**
|
||||
|
||||
Expanding the record set changes durable cardinality and exercises more caller variants; regressions could appear outside the focused cases.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run the complete manifest/attempt/connectivity/skill-contract suites and `git diff --check`. Record full output in the review stub.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G09.md`: record implementation decisions, deviations, and actual command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No further files. Existing corruption, retry/resume, measurement, web-validation, secret-redaction, and live-adapter tests provide the whole-suite oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: all tests report `OK`; whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/attempts.py` | API-1 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-1, API-2, API-3 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | API-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md` | API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0`; fresh output is required and cached output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 -m unittest \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_cli_mixed_manifest_preflights_and_invokes_every_cell \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_only_public_preflight_appends_exact_results \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_preset_blocker_appends_without_attempt_allocation \
|
||||
scripts.agent_benchmark.connectivity_integration_test.ConnectivityIntegrationTest.test_live_agy_multiple_cells_consume_their_own_preflight_state
|
||||
python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits 0, all benchmark suites report `OK`, and no secret/raw runtime values appear in durable-record assertions.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=0 tag=API milestone-task=fixture-lock -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract, plan=0, tag=API
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_0.log` and `PLAN-local-G08.md` → `plan_local_G08_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Add a closed dual-version rubric catalog | [ ] |
|
||||
| API-2 Bind prompt and durable validation to manifest version | [ ] |
|
||||
| API-3 Preserve historical compatibility and exact new semantics | [ ] |
|
||||
| API-4 Run the complete benchmark regression | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
|
||||
- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
|
||||
- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility, and update the project benchmark skill to the manifest-selected contract.
|
||||
- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_0.log`.
|
||||
- [ ] Archive active `PLAN-local-G08.md` to `plan_local_G08_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the legacy version/category tuple remains exact and reloadable.
|
||||
- Confirm the new seven categories and maxima exactly match the approved SDD and total 100.
|
||||
- Confirm prompt and both worksheet load paths use `manifest.rubric_version`.
|
||||
- Confirm unknown and cross-version worksheets fail before scored result publication.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Dependency Verification
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")
|
||||
archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, candidates
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-1 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-2 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-3 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### API-4 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,260 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=1 tag=API milestone-task=fixture-lock -->
|
||||
|
||||
# Code Review Reference - API
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract, plan=1, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found a semantic contradiction: Scope Rationale prohibited a project-skill edit while API-3 and the file summary required that same edit, creating ownership overlap with the route-preflight packet.
|
||||
- Replan keeps this packet on the dormant additive rubric implementation and dual-version code tests. The dependent final-manifest packet owns the operational skill transition when it selects the new rubric. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G08.md` → `code_review_cloud_G08_1.log` and `PLAN-local-G08.md` → `plan_local_G08_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| API-1 Add a closed dual-version rubric catalog | [x] |
|
||||
| API-2 Bind prompt and durable validation to manifest version | [x] |
|
||||
| API-3 Preserve historical compatibility and exact new semantics | [x] |
|
||||
| API-4 Run the complete benchmark regression | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
|
||||
- [x] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
|
||||
- [x] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric.
|
||||
- [x] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G08.md` to `code_review_cloud_G08_1.log`.
|
||||
- [x] Archive active `PLAN-local-G08.md` to `plan_local_G08_1.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` and update this checklist at the final archive path.
|
||||
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
없음. 지정된 파일 범위와 검증 명령을 그대로 수행했으며, project benchmark skill과 living spec은 변경하지 않았다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- `RUBRIC_VERSION`과 `RUBRIC_CATEGORIES`를 legacy alias/table로 그대로 유지하고, `ONE_SHOT_RUBRIC_VERSION`, `RUBRIC_VERSIONS`, immutable `RUBRIC_CATEGORIES_BY_VERSION` catalog를 additive하게 추가했다.
|
||||
- worksheet가 선언한 version의 ordered table로 먼저 검증되고, `expected_version`이 있으면 exact match를 추가로 요구하도록 해 unknown·cross-version 출력을 fail-closed했다.
|
||||
- evaluator prompt와 신규/내구 score reload의 두 worksheet load path 모두 `manifest.rubric_version`을 사용한다. legacy prompt exact bytes와 canonical worksheet SHA-256도 회귀 테스트로 고정했다.
|
||||
- 새-version 성공과 structurally valid legacy worksheet의 cross-version 실패를 각각 fresh score attempt로 검증했으며 실패 result에는 scored worksheet를 publish하지 않음을 확인했다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the legacy version/category tuple remains exact and reloadable.
|
||||
- Confirm the new seven categories and maxima exactly match the approved SDD and total 100.
|
||||
- Confirm prompt and both worksheet load paths use `manifest.rubric_version`.
|
||||
- Confirm unknown and cross-version worksheets fail before scored result publication.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Dependency Verification
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")
|
||||
archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, candidates
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log
|
||||
```
|
||||
|
||||
### API-1 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
......................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 118 tests in 1.181s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### API-2 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.448s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### API-3 Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
.......................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 135 tests in 5.746s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### API-4 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 208 tests in 51.235s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
`git diff --check`: exit 0, 출력 없음.
|
||||
|
||||
### Reviewer Fresh Verification
|
||||
|
||||
2026-08-12 공식 리뷰에서 아래 검증을 현재 worktree 기준으로 다시 실행했다.
|
||||
|
||||
```text
|
||||
go version go1.26.2 linux/arm64
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log
|
||||
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
Ran 118 tests in 1.202s
|
||||
OK
|
||||
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
Ran 1 test in 0.453s
|
||||
OK
|
||||
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
Ran 135 tests in 5.848s
|
||||
OK
|
||||
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
Ran 208 tests in 50.790s
|
||||
OK
|
||||
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
Ran 432 tests in 119.960s
|
||||
OK
|
||||
|
||||
git diff --check
|
||||
exit 0, no output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: PASS
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — manifest-selected rubric lookup, prompt rendering, output validation, and durable reload validation preserve one exact version/table binding.
|
||||
- Completeness: Pass — API-1 through API-4 are implemented within the planned files; the dependent manifest packet remains the owner of the first tracked new-version manifest.
|
||||
- Test coverage: Pass — focused dual-version tests, the planned 208-test regression, and the 432-test full benchmark discovery all pass.
|
||||
- API contract: Pass — `landing-quality-v1` remains byte-compatible while the schema and loader expose only the closed two-version catalog.
|
||||
- Code quality: Pass — immutable tuples plus `MappingProxyType` keep the catalog closed; no stale production literal, debug print, TODO, or whitespace error was found.
|
||||
- Implementation deviation: Pass — no deviation from the revalidated plan; unrelated sibling-task worktree changes were excluded from this verdict.
|
||||
- Verification trust: Pass — implementation-reported results were independently reproduced with fresh reviewer commands.
|
||||
- Spec conformance: Pass — SDD S01's seven-category 100-point rubric and `fixture-lock` evidence boundary match the implementation; concrete manifest/digest evidence remains assigned to the dependent manifest packet.
|
||||
- Findings: None
|
||||
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
|
||||
- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock` for runtime aggregation.
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=1 tag=API milestone-task=fixture-lock -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
구현 전 ownership 충돌 재계획 1회 뒤 dual-version rubric 계약 구현을 공식 리뷰 1회로 검증했으며 최종 판정은 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_local_G08_0.log` | `code_review_cloud_G08_0.log` | 재계획 | project benchmark skill 수정 ownership 충돌을 구현 전에 제거했으며 공식 판정은 없었다. |
|
||||
| `plan_local_G08_1.log` | `code_review_cloud_G08_1.log` | PASS | legacy rubric을 보존한 closed dual-version catalog, manifest-selected prompt/worksheet 검증, 전체 benchmark 회귀를 확인했다. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- `landing-quality-v1`의 기존 category tuple과 canonical worksheet 의미를 유지하면서 `one-shot-agent-comparison-v1`의 SDD 승인 7개 category를 immutable catalog에 추가했다.
|
||||
- manifest schema/loader, evaluator prompt, 최초 score publication과 durable score reload가 같은 manifest-selected rubric version을 사용하고 unknown/cross-version worksheet를 fail-closed한다.
|
||||
- dual-version manifest/rubric/scoring regression과 legacy prompt/worksheet compatibility assertion을 추가했으며 운영 skill과 첫 tracked new-version manifest 전환은 후속 packet 범위로 유지했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test` - PASS; 118 tests, `OK`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation` - PASS; 1 test, `OK`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test` - PASS; 135 tests, `OK`.
|
||||
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 세 manifest 모두 `ok: manifest is valid`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` - PASS; 208 tests, `OK`.
|
||||
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 432 tests, `OK`.
|
||||
- `git diff --check` - PASS; 출력 없음.
|
||||
- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic benchmark rubric/schema packet 범위가 아니어서 실행하지 않았다.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- dependent `04+02,03_locked_benchmark_manifest` packet이 첫 tracked `one-shot-agent-comparison-v1` manifest와 concrete rubric digest evidence를 제공해야 하며, 이 완료 로그 단독으로 `fixture-lock` 전체 완료를 주장하지 않는다.
|
||||
|
|
@ -0,0 +1,304 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=0 tag=API milestone-task=fixture-lock -->
|
||||
|
||||
# Plan - Versioned One-Shot Benchmark Rubric Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until exactly one active or archived `01_execution_order_contract/complete.log` is resolved by the dependency command below. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The approved one-shot comparison SDD fixes seven rubric categories totaling 100, while the existing `landing-quality-v1` worksheet contains five different categories. Reusing that version name would mutate historical meaning, so this packet preserves legacy runs and adds a manifest-selected version for the approved benchmark rubric.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/rubric.py`
|
||||
- `scripts/agent_benchmark/rubric_test.py`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=fixture-lock`.
|
||||
- Target scenario: S01 requires the same fixture, viewport, rubric checksum/version for every cell. The approved rubric at SDD lines 90-91 is 요구사항 충족 25, 시각 완성도 25, 반응형·접근성 15, 이미지 활용·디테일 10, 동작 안정성 10, 코드 품질 10, 자체 검증 완결성 5.
|
||||
- Evidence Map row: S01 requires fixture prompt/assets/workspace/rubric digest evidence.
|
||||
- The checklist therefore creates a distinct immutable rubric version, retains legacy validation, and binds prompt/output validation to the manifest-selected version. The final manifest packet supplies the concrete version/digest evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the complete manifest, rubric, scoring, and test files above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all example manifests validated and the 178-test manifest/attempt/connectivity suite passed. `rubric_test` and `scoring_test` are additional required fresh suites for this packet.
|
||||
- Current `RUBRIC_CATEGORIES` is five entries totaling 100 (`rubric.py:21-27`). `validate_worksheet` accepts only global `RUBRIC_VERSION` (`rubric.py:82-126`), and `_prompt` hardcodes those categories and `landing-quality-v1` (`scoring.py:1207-1222`).
|
||||
- The safe compatibility rule is additive: keep `landing-quality-v1` and its five-category table readable; add `one-shot-agent-comparison-v1` with the approved seven-category table; reject unknown and cross-version worksheets.
|
||||
- Fresh output is required; cached output is not accepted. No external runner is required.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing rubric tests validate only the five-category legacy table.
|
||||
- Existing scoring tests generate only legacy worksheets and cannot prove prompt/worksheet selection from the manifest.
|
||||
- Existing manifest/schema tests accept only one rubric constant.
|
||||
- No test rejects a valid legacy worksheet when the allocation requires the new rubric, or vice versa.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `manifest.RUBRIC_VERSION` is referenced only by `manifest.py:711` and `rubric.py:16,83,126`. Replace internal single-version validation with a closed version catalog while preserving a legacy alias if needed for compatibility.
|
||||
- `rubric.RUBRIC_CATEGORIES` is imported by `scoring.py:44-49`, `scoring_test.py:56`, and `rubric_test.py:10`. Preserve it as the legacy tuple for existing callers, add a version lookup, and migrate production scoring to the lookup.
|
||||
- `_prompt` is private and called at `scoring.py:1879`; change both definition and call together.
|
||||
- `load_worksheet` call sites are `scoring.py:1641` and `scoring.py:1972`; both must pass the manifest-selected expected version.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is dependent child `03+01_rubric_version_contract`. Its stable invariant is: rubric version selects exactly one immutable ordered category table from manifest load through evaluator prompt, worksheet validation, durable result, and historical reload.
|
||||
- PASS evidence is dual-version loader/rubric/scoring tests plus full regression.
|
||||
- Predecessor index 01 is currently active at `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` but has no `complete.log`; implementation is dependency-blocked until that exact file exists. The dependency is required because both packets modify `manifest.py`, `manifest_test.py`, and the schema.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not delete, rename, or reinterpret `landing-quality-v1`; old manifests and immutable run snapshots must still load and score under the old table.
|
||||
- Do not change automatic web gate eligibility, evaluator identity blinding, score persistence, caller bindings, fixture bytes, or report layout.
|
||||
- Do not update the project benchmark skill or living spec in this packet; their current general 100-point description remains true, and operational selection remains manifest-owned.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`.
|
||||
- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one predecessor `complete.log`: the active sibling path or one matching `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log`.
|
||||
2. Implement this packet against the reviewed 01 source. Do not infer completion from an active PLAN or review stub.
|
||||
3. On PASS, `04+02,03_locked_benchmark_manifest` may proceed only after both index 02 and this index 03 have `complete.log`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
|
||||
- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
|
||||
- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility, and update the project benchmark skill to the manifest-selected contract.
|
||||
- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Add a closed dual-version rubric catalog
|
||||
|
||||
**Problem**
|
||||
|
||||
`manifest.py:30` exposes one allowed rubric version, and `rubric.py:21-27` gives that version a five-category table that differs from the approved SDD. Changing the existing tuple in place would silently rewrite historical `landing-quality-v1` meaning.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep the legacy string and tuple unchanged. Add `ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"`, a closed allowed-version tuple in `manifest.py`, and an ordered category mapping in `rubric.py`. The new table is exactly:
|
||||
|
||||
```python
|
||||
ONE_SHOT_RUBRIC_CATEGORIES = (
|
||||
("requirements_fidelity", 25),
|
||||
("visual_completeness", 25),
|
||||
("responsive_accessibility", 15),
|
||||
("image_detail_usage", 10),
|
||||
("behavior_stability", 10),
|
||||
("code_quality", 10),
|
||||
("self_verification", 5),
|
||||
)
|
||||
```
|
||||
|
||||
Add `rubric_categories(version)` that fails closed for unknown versions. Update the manifest schema from a single `const` to an exact two-value enum. `validate_worksheet` must read the worksheet version, optionally require an `expected_version`, and validate against only that version's ordered table.
|
||||
|
||||
Before (`scripts/agent_benchmark/rubric.py:21-27`):
|
||||
|
||||
```python
|
||||
RUBRIC_CATEGORIES = (
|
||||
("task_fidelity", 25),
|
||||
("visual_hierarchy", 25),
|
||||
("responsive_composition", 20),
|
||||
("typography_readability", 15),
|
||||
("polish_consistency", 15),
|
||||
)
|
||||
```
|
||||
|
||||
After: retain that exact tuple as the legacy value and add the new tuple/mapping without mutation.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: accept the closed legacy/new rubric version catalog.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: mirror the exact two-version enum.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`: provide versioned category lookup and expected-version worksheet validation.
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: verify exact tables, totals, ordering, unknown versions, and cross-version rejection.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: verify loader/schema parity for both allowed versions and an unknown version.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests. Preserve every existing legacy assertion. Add `test_one_shot_rubric_exact_categories_and_total_are_accepted`, `test_cross_version_worksheet_is_rejected`, and manifest/schema two-version parity. Assert both category sums equal exactly 100.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: both versions pass their exact table; malformed, unknown, reordered, and cross-version cases fail closed.
|
||||
|
||||
### [API-2] Bind prompt and durable score validation to the manifest version
|
||||
|
||||
**Problem**
|
||||
|
||||
`_prompt` uses global categories and a literal version (`scoring.py:1207-1222`), while both `load_worksheet` calls (`scoring.py:1641`, `1972`) validate only the global legacy version. A new manifest version could therefore ask for or accept the wrong worksheet.
|
||||
|
||||
**Solution**
|
||||
|
||||
Change `_prompt` to accept `rubric_version`, look up its exact table, render that version string, and call it with `manifest.rubric_version`. Extend `load_worksheet`/`validate_worksheet` with `expected_version` and pass `manifest.rubric_version` at both scoring call sites. Keep allocation/result binding to the manifest version and reject a worksheet whose internal version differs before publication.
|
||||
|
||||
Before (`scripts/agent_benchmark/scoring.py:1207-1222`):
|
||||
|
||||
```python
|
||||
def _prompt(blind: BlindWorkspace) -> bytes:
|
||||
categories = ", ".join(
|
||||
f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES
|
||||
)
|
||||
...
|
||||
"landing-quality-v1, integer scores within each maximum ..."
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes:
|
||||
categories = ", ".join(
|
||||
f"{ident} ({maximum})"
|
||||
for ident, maximum in rubric_categories(rubric_version)
|
||||
)
|
||||
...
|
||||
f"{rubric_version}, integer scores within each maximum ..."
|
||||
```
|
||||
|
||||
The `blind` parameter may remain for interface stability even though it is not interpolated. Do not expose producer identity.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: select prompt categories/version and worksheet validation from `manifest.rubric_version`.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: add a new-version scoring success and cross-version failure test.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write `test_manifest_selected_rubric_drives_prompt_and_worksheet_validation`. Build a new-version manifest, emit the exact seven-category worksheet, and assert success plus prompt category/version content. On a fresh score attempt, emit a structurally valid legacy worksheet and assert `invalid_worksheet` with no scored worksheet publication.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
```
|
||||
|
||||
Expected: selected-version success and cross-version rejection both pass.
|
||||
|
||||
### [API-3] Preserve historical compatibility and exact new semantics
|
||||
|
||||
**Problem**
|
||||
|
||||
An additive version is safe only if old manifests, prompts, worksheets, and immutable score reloads retain byte-level meaning.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep existing example manifests on `landing-quality-v1`. Make test helpers accept a version instead of replacing their legacy default. Add explicit assertions that legacy canonical worksheet bytes and prompts remain unchanged, while the new version has seven ordered categories and total 100. Update the project benchmark skill's scoring phase so it requires the exact rubric selected by the immutable manifest and names both supported versions; do not leave its current line 87 fixed to the legacy version.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: dual-version canonical and boundary assertions.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: legacy default plus new-version selected behavior.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: both versions and shipped examples.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact rubric behavior and the closed supported version set.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests; do not mass-replace legacy version literals in unrelated fixtures. The new final manifest is added only by the dependent packet.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md
|
||||
```
|
||||
|
||||
Expected: all legacy and new rubric tests report `OK`.
|
||||
|
||||
### [API-4] Run the complete benchmark regression
|
||||
|
||||
**Problem**
|
||||
|
||||
Rubric validation is used by scoring recovery and durable result reads, so focused tests are not enough.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run all rubric/scoring and the existing benchmark suites, validate shipped legacy examples, and check whitespace.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G08.md`: record exact decisions, deviations, and all actual command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No further files. Existing recovery, redaction, scoring retry, manifest digest, and connectivity suites are the full oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: three legacy manifests validate, all tests report `OK`, and whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric.py` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/scoring.py` | API-2 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | API-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md` | API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` after predecessor 01 is complete; fresh output is required and cached output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")
|
||||
archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, candidates
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
rg --sort path -n --fixed-strings 'one-shot-agent-comparison-v1' agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: dependency check and every command exit 0, legacy examples validate, all suites report `OK`, and `git diff --check` is silent.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,307 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract plan=1 tag=API milestone-task=fixture-lock -->
|
||||
|
||||
# Plan - Versioned One-Shot Benchmark Rubric Contract
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until exactly one active or archived `01_execution_order_contract/complete.log` is resolved by the dependency command below. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G08.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The approved one-shot comparison SDD fixes seven rubric categories totaling 100, while the existing `landing-quality-v1` worksheet contains five different categories. Reusing that version name would mutate historical meaning, so this packet preserves legacy runs and adds a manifest-selected version for the approved benchmark rubric.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/plan_local_G08_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/code_review_cloud_G08_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found a semantic contradiction: Scope Rationale prohibited a project-skill edit while API-3 and the file summary required that same edit, creating ownership overlap with the route-preflight packet.
|
||||
- Replan keeps this packet on the dormant additive rubric implementation and dual-version code tests. The dependent final-manifest packet owns the operational skill transition when it selects the new rubric. Baseline manifest validation and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/rubric.py`
|
||||
- `scripts/agent_benchmark/rubric_test.py`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=fixture-lock`.
|
||||
- Target scenario: S01 requires the same fixture, viewport, rubric checksum/version for every cell. The approved rubric at SDD lines 90-91 is 요구사항 충족 25, 시각 완성도 25, 반응형·접근성 15, 이미지 활용·디테일 10, 동작 안정성 10, 코드 품질 10, 자체 검증 완결성 5.
|
||||
- Evidence Map row: S01 requires fixture prompt/assets/workspace/rubric digest evidence.
|
||||
- The checklist therefore creates a distinct immutable rubric version, retains legacy validation, and binds prompt/output validation to the manifest-selected version. The final manifest packet supplies the concrete version/digest evidence.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the complete manifest, rubric, scoring, and test files above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all example manifests validated and the 178-test manifest/attempt/connectivity suite passed. `rubric_test` and `scoring_test` are additional required fresh suites for this packet.
|
||||
- Current `RUBRIC_CATEGORIES` is five entries totaling 100 (`rubric.py:21-27`). `validate_worksheet` accepts only global `RUBRIC_VERSION` (`rubric.py:82-126`), and `_prompt` hardcodes those categories and `landing-quality-v1` (`scoring.py:1207-1222`).
|
||||
- The safe compatibility rule is additive: keep `landing-quality-v1` and its five-category table readable; add `one-shot-agent-comparison-v1` with the approved seven-category table; reject unknown and cross-version worksheets.
|
||||
- Fresh output is required; cached output is not accepted. No external runner is required.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing rubric tests validate only the five-category legacy table.
|
||||
- Existing scoring tests generate only legacy worksheets and cannot prove prompt/worksheet selection from the manifest.
|
||||
- Existing manifest/schema tests accept only one rubric constant.
|
||||
- No test rejects a valid legacy worksheet when the allocation requires the new rubric, or vice versa.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `manifest.RUBRIC_VERSION` is referenced only by `manifest.py:711` and `rubric.py:16,83,126`. Replace internal single-version validation with a closed version catalog while preserving a legacy alias if needed for compatibility.
|
||||
- `rubric.RUBRIC_CATEGORIES` is imported by `scoring.py:44-49`, `scoring_test.py:56`, and `rubric_test.py:10`. Preserve it as the legacy tuple for existing callers, add a version lookup, and migrate production scoring to the lookup.
|
||||
- `_prompt` is private and called at `scoring.py:1879`; change both definition and call together.
|
||||
- `load_worksheet` call sites are `scoring.py:1641` and `scoring.py:1972`; both must pass the manifest-selected expected version.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is dependent child `03+01_rubric_version_contract`. Its stable invariant is: rubric version selects exactly one immutable ordered category table from manifest load through evaluator prompt, worksheet validation, durable result, and historical reload.
|
||||
- PASS evidence is dual-version loader/rubric/scoring tests plus full regression.
|
||||
- Predecessor index 01 is currently active at `agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/` but has no `complete.log`; implementation is dependency-blocked until that exact file exists. The dependency is required because both packets modify `manifest.py`, `manifest_test.py`, and the schema.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not delete, rename, or reinterpret `landing-quality-v1`; old manifests and immutable run snapshots must still load and score under the old table.
|
||||
- Do not change automatic web gate eligibility, evaluator identity blinding, score persistence, caller bindings, fixture bytes, or report layout.
|
||||
- Do not update the project benchmark skill or living spec in this packet. The currently tracked manifests still select the legacy rubric; dependent packet 04 owns the atomic skill transition when it adds the first tracked new-version manifest.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+1+2=G08`; base/final route `local-fit`, lane `local`, catalog `worker/local/G08`, filename `PLAN-local-G08.md`.
|
||||
- Review closures: all closed. Scores `2+1+2+1+2=G08`; route `official-review`, lane `cloud`, catalog `review/cloud/G08`, filename `CODE_REVIEW-cloud-G08.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one predecessor `complete.log`: the active sibling path or one matching `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log`.
|
||||
2. Implement this packet against the reviewed 01 source. Do not infer completion from an active PLAN or review stub.
|
||||
3. On PASS, `04+02,03_locked_benchmark_manifest` may proceed only after both index 02 and this index 03 have `complete.log`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [API-1] Add an immutable `one-shot-agent-comparison-v1` seven-category rubric while preserving exact `landing-quality-v1` loader/schema/worksheet behavior.
|
||||
- [ ] [API-2] Make evaluator prompt generation and worksheet loading use the manifest-selected rubric version and reject cross-version output.
|
||||
- [ ] [API-3] Add dual-version manifest, rubric, and scoring regressions for exact category order, maxima, total 100, prompt text, and historical compatibility without changing operational skill wording before a tracked manifest selects the new rubric.
|
||||
- [ ] [API-4] Run all rubric/scoring and benchmark regression suites plus shipped manifest validation.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Add a closed dual-version rubric catalog
|
||||
|
||||
**Problem**
|
||||
|
||||
`manifest.py:30` exposes one allowed rubric version, and `rubric.py:21-27` gives that version a five-category table that differs from the approved SDD. Changing the existing tuple in place would silently rewrite historical `landing-quality-v1` meaning.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep the legacy string and tuple unchanged. Add `ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"`, a closed allowed-version tuple in `manifest.py`, and an ordered category mapping in `rubric.py`. The new table is exactly:
|
||||
|
||||
```python
|
||||
ONE_SHOT_RUBRIC_CATEGORIES = (
|
||||
("requirements_fidelity", 25),
|
||||
("visual_completeness", 25),
|
||||
("responsive_accessibility", 15),
|
||||
("image_detail_usage", 10),
|
||||
("behavior_stability", 10),
|
||||
("code_quality", 10),
|
||||
("self_verification", 5),
|
||||
)
|
||||
```
|
||||
|
||||
Add `rubric_categories(version)` that fails closed for unknown versions. Update the manifest schema from a single `const` to an exact two-value enum. `validate_worksheet` must read the worksheet version, optionally require an `expected_version`, and validate against only that version's ordered table.
|
||||
|
||||
Before (`scripts/agent_benchmark/rubric.py:21-27`):
|
||||
|
||||
```python
|
||||
RUBRIC_CATEGORIES = (
|
||||
("task_fidelity", 25),
|
||||
("visual_hierarchy", 25),
|
||||
("responsive_composition", 20),
|
||||
("typography_readability", 15),
|
||||
("polish_consistency", 15),
|
||||
)
|
||||
```
|
||||
|
||||
After: retain that exact tuple as the legacy value and add the new tuple/mapping without mutation.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/manifest.py`: accept the closed legacy/new rubric version catalog.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`: mirror the exact two-version enum.
|
||||
- [ ] `scripts/agent_benchmark/rubric.py`: provide versioned category lookup and expected-version worksheet validation.
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: verify exact tables, totals, ordering, unknown versions, and cross-version rejection.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: verify loader/schema parity for both allowed versions and an unknown version.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests. Preserve every existing legacy assertion. Add `test_one_shot_rubric_exact_categories_and_total_are_accepted`, `test_cross_version_worksheet_is_rejected`, and manifest/schema two-version parity. Assert both category sums equal exactly 100.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: both versions pass their exact table; malformed, unknown, reordered, and cross-version cases fail closed.
|
||||
|
||||
### [API-2] Bind prompt and durable score validation to the manifest version
|
||||
|
||||
**Problem**
|
||||
|
||||
`_prompt` uses global categories and a literal version (`scoring.py:1207-1222`), while both `load_worksheet` calls (`scoring.py:1641`, `1972`) validate only the global legacy version. A new manifest version could therefore ask for or accept the wrong worksheet.
|
||||
|
||||
**Solution**
|
||||
|
||||
Change `_prompt` to accept `rubric_version`, look up its exact table, render that version string, and call it with `manifest.rubric_version`. Extend `load_worksheet`/`validate_worksheet` with `expected_version` and pass `manifest.rubric_version` at both scoring call sites. Keep allocation/result binding to the manifest version and reject a worksheet whose internal version differs before publication.
|
||||
|
||||
Before (`scripts/agent_benchmark/scoring.py:1207-1222`):
|
||||
|
||||
```python
|
||||
def _prompt(blind: BlindWorkspace) -> bytes:
|
||||
categories = ", ".join(
|
||||
f"{ident} ({maximum})" for ident, maximum in RUBRIC_CATEGORIES
|
||||
)
|
||||
...
|
||||
"landing-quality-v1, integer scores within each maximum ..."
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
def _prompt(blind: BlindWorkspace, rubric_version: str) -> bytes:
|
||||
categories = ", ".join(
|
||||
f"{ident} ({maximum})"
|
||||
for ident, maximum in rubric_categories(rubric_version)
|
||||
)
|
||||
...
|
||||
f"{rubric_version}, integer scores within each maximum ..."
|
||||
```
|
||||
|
||||
The `blind` parameter may remain for interface stability even though it is not interpolated. Do not expose producer identity.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: select prompt categories/version and worksheet validation from `manifest.rubric_version`.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: add a new-version scoring success and cross-version failure test.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write `test_manifest_selected_rubric_drives_prompt_and_worksheet_validation`. Build a new-version manifest, emit the exact seven-category worksheet, and assert success plus prompt category/version content. On a fresh score attempt, emit a structurally valid legacy worksheet and assert `invalid_worksheet` with no scored worksheet publication.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
```
|
||||
|
||||
Expected: selected-version success and cross-version rejection both pass.
|
||||
|
||||
### [API-3] Preserve historical compatibility and exact new semantics
|
||||
|
||||
**Problem**
|
||||
|
||||
An additive version is safe only if old manifests, prompts, worksheets, and immutable score reloads retain byte-level meaning.
|
||||
|
||||
**Solution**
|
||||
|
||||
Keep existing example manifests on `landing-quality-v1`. Make test helpers accept a version instead of replacing their legacy default. Add explicit assertions that legacy canonical worksheet bytes and prompts remain unchanged, while the new version has seven ordered categories and total 100. Do not change the project benchmark skill in this dependency packet: all currently tracked manifests still select the legacy version, and the dependent final-manifest packet owns the atomic operational transition to manifest-selected wording.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/rubric_test.py`: dual-version canonical and boundary assertions.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: legacy default plus new-version selected behavior.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: both versions and shipped examples.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write tests; do not mass-replace legacy version literals in unrelated fixtures. The new final manifest is added only by the dependent packet.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
```
|
||||
|
||||
Expected: all legacy and new rubric tests report `OK`.
|
||||
|
||||
### [API-4] Run the complete benchmark regression
|
||||
|
||||
**Problem**
|
||||
|
||||
Rubric validation is used by scoring recovery and durable result reads, so focused tests are not enough.
|
||||
|
||||
**Solution**
|
||||
|
||||
Run all rubric/scoring and the existing benchmark suites, validate shipped legacy examples, and check whitespace.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G08.md`: record exact decisions, deviations, and all actual command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No further files. Existing recovery, redaction, scoring retry, manifest digest, and connectivity suites are the full oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: three legacy manifests validate, all tests report `OK`, and whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/agent_benchmark/manifest.py` | API-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric.py` | API-1 |
|
||||
| `scripts/agent_benchmark/rubric_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | API-1, API-3 |
|
||||
| `scripts/agent_benchmark/scoring.py` | API-2 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | API-2, API-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md` | API-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` after predecessor 01 is complete; fresh output is required and cached output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
active = Path("agent-task/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log")
|
||||
archived = sorted(Path("agent-task/archive").glob("*/*/m-iop-one-shot-agent-model-comparison/01_execution_order_contract/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, candidates
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test.ScoringTest.test_manifest_selected_rubric_drives_prompt_and_worksheet_validation
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: dependency check and every command exit 0, legacy examples validate, all suites report `OK`, and `git diff --check` is silent.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,248 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=2 tag=TEST milestone-task=fixture-lock,matrix-lock -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=2, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan/review before Epic self-review: `plan_local_G07_0.log`, `code_review_cloud_G07_0.log`.
|
||||
- Refinement source pair: `plan_cloud_G09_1.log`, `code_review_cloud_G09_1.log`.
|
||||
- Neither prior pair contains implementation evidence or an official verdict. The refinement preserves the original manifest, skill-contract, and regression scope while moving only the external readiness preflight into child `05+04_readiness_preflight`.
|
||||
- Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b` in the source plan evidence.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_2.log` and `PLAN-local-G06.md` → `plan_local_G06_2.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 Create one exact immutable readiness manifest | [x] |
|
||||
| TEST-3 Preserve the full benchmark baseline | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
|
||||
- [x] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G06.md` to `code_review_cloud_G06_2.log`.
|
||||
- [x] Archive active `PLAN-local-G06.md` to `plan_local_G06_2.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path.
|
||||
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, keep parent `agent-task/m-iop-one-shot-agent-model-comparison/` because child `05+04_readiness_preflight` remains active until its own review completes.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- 없음. PLAN의 파일 범위와 검증 명령을 그대로 유지했다.
|
||||
- 보조 검증으로 `skill-creator`의 `quick_validate.py`를 추가 실행했으며 계획된 검증을 대체하지 않았다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- 새 manifest의 JSON matrix는 C01-C09 순서로 읽기 쉽게 유지하고, loader가 explicit seed로 산출한 immutable 실행 순서는 별도로 exact assertion했다.
|
||||
- manifest regression은 raw JSON 전체 cell payload와 loader의 canonical payload를 모두 비교해 route kind/id, request effort, 다섯 stage 순서, cloud effort, `ornith-fast` effort 생략을 함께 잠갔다.
|
||||
- scoring skill 문구는 기존 Procedure의 generic fallback-language 금지 계약을 유지하기 위해 `no substitute rubric or reinterpretation`으로 표현하고, 닫힌 두 rubric version literal과 manifest-selected semantics를 contract test로 고정했다.
|
||||
- Spec update not needed: 기존 living spec의 manifest-selected rubric 및 immutable manifest 설명과 동작 범위는 그대로 정확하며 새 tracked bench-02 입력만 추가됐다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm both predecessor completion files exist before reviewing implementation.
|
||||
- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact.
|
||||
- Confirm hybrid routes use route kind `execution_preset`, include `repair`, bind every cloud stage to its exact model with high effort, and bind work to `ornith-fast` with no invented effort.
|
||||
- Confirm project-skill scoring uses the exact manifest-selected rubric, names the closed two-version set, and preserves predecessor 02's all-cell preflight contract.
|
||||
- Confirm no public live preflight or scored run/score/report command was executed in this child.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`.
|
||||
|
||||
### Dependency Verification
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log
|
||||
```
|
||||
|
||||
### TEST-1 Verification
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: manifest is valid
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.003s
|
||||
|
||||
OK
|
||||
..................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 50 tests in 1.705s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
Supplemental skill validation:
|
||||
|
||||
```text
|
||||
$ python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/iop-agent-comparison-benchmark
|
||||
Skill is valid!
|
||||
```
|
||||
|
||||
### TEST-3 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
```text
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
..................................................................................................................................................................................................................................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 434 tests in 118.623s
|
||||
|
||||
OK
|
||||
git diff --check: no output (exit 0)
|
||||
```
|
||||
|
||||
### Reviewer Fresh Verification (2026-08-12)
|
||||
|
||||
```text
|
||||
$ python3 - <<'PY' # exact dependency verification block above
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/complete.log
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/complete.log
|
||||
|
||||
$ python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
ok: manifest is valid
|
||||
|
||||
$ python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
.
|
||||
----------------------------------------------------------------------
|
||||
Ran 1 test in 0.004s
|
||||
|
||||
OK
|
||||
|
||||
$ python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
..................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 50 tests in 1.716s
|
||||
|
||||
OK
|
||||
|
||||
$ for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
ok: manifest is valid
|
||||
|
||||
$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
----------------------------------------------------------------------
|
||||
Ran 434 tests in 124.077s
|
||||
|
||||
OK
|
||||
|
||||
$ git diff --check
|
||||
(no output; exit 0)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: PASS
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — tracked manifest fields, fixture checksum, two image paths, evaluator, C01-C09 direct/preset bindings, omitted work-stage effort, and seeded order match the approved plan and SDD.
|
||||
- Completeness: Pass — TEST-1 and TEST-3 are implemented; both predecessor completion records resolve exactly once, and external readiness remains correctly isolated in child `05+04_readiness_preflight`.
|
||||
- Test coverage: Pass — fresh focused manifest/skill tests, four-manifest validation, and all 434 discovered benchmark tests pass.
|
||||
- API contract: Pass — direct and execution-preset route identities preserve the OpenAI/Anthropic ingress and Edge config ownership boundaries without fallback or substitution.
|
||||
- Code quality: Pass — the exact regression compares raw and canonical payloads, the skill contract mutation-tests legacy-only wording, and `git diff --check` is clean.
|
||||
- Implementation deviation: Pass — planned production/documentation files are present; the supplemental skill validation did not replace required verification, and predecessor/sibling worktree changes were excluded from this verdict.
|
||||
- Verification trust: Pass — every applicable plan command was rerun against the current worktree and reproduced the implementation handoff; no `agent-test/runs/bench-02` execution state exists.
|
||||
- Spec conformance: Pass — SDD S01/S03 and Evidence Map entries for `fixture-lock,matrix-lock` are satisfied; the current living spec already describes immutable manifest-selected rubric behavior, so a spec update is not needed.
|
||||
- Findings: None
|
||||
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
|
||||
- Next Step: PASS — archive the active pair, write `complete.log`, and move this split task to the monthly archive while preserving `milestone-task=fixture-lock,matrix-lock` for runtime aggregation.
|
||||
|
|
@ -0,0 +1,209 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=0 tag=TEST milestone-task=fixture-lock,route-readiness,matrix-lock -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=0, tag=TEST
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G07.md` → `code_review_cloud_G07_0.log` and `PLAN-local-G07.md` → `plan_local_G07_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 Create one exact immutable readiness manifest | [ ] |
|
||||
| TEST-2 Produce one redacted nine-cell readiness record | [ ] |
|
||||
| TEST-3 Preserve the full benchmark baseline | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order.
|
||||
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G07.md` to `code_review_cloud_G07_0.log`.
|
||||
- [ ] Archive active `PLAN-local-G07.md` to `plan_local_G07_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm both predecessor completion files exist before reviewing implementation.
|
||||
- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact.
|
||||
- Confirm hybrid routes include `repair`, cloud stages use high, and work uses `ornith-fast` with no invented effort.
|
||||
- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value.
|
||||
- Confirm no scored run/score/report command was executed in this Epic.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. TEST-2's full command block is fixed in the plan and must be copied with its actual output or exact blocker here.
|
||||
|
||||
### Dependency Verification
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### TEST-1 Verification
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### TEST-2 External Preflight
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)"
|
||||
test -x ../iop-s2/build/bin/iop-edge
|
||||
test -x ../iop-s2/build/dev/iop-node
|
||||
test -f ../iop-s2/configs/edge.yaml
|
||||
claude --version
|
||||
agy --version
|
||||
codex --version
|
||||
agy_help="$(agy --help 2>&1)"; for token in --print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json; do grep -F -- "$token" <<<"$agy_help" >/dev/null; done
|
||||
python3 - <<'PY'
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)"
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
python3 - "$run_id" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert record["status"] == "ready"
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine ready results are manifest-bound and runtime values are absent")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output or exact blocker and resume condition:_
|
||||
|
||||
### TEST-3 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,243 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=1 tag=TEST milestone-task=fixture-lock,route-readiness,matrix-lock -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest, plan=1, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found that the final manifest would select the new rubric while the public project skill remained fixed to `landing-quality-v1`; it also found an incomplete agy resume gate and ambiguous testbed source provenance.
|
||||
- Replan atomically assigns rubric skill/contract-test synchronization to this dependent packet, requires the adapter-known agy version plus its documented transport surface, and records the independent read-only testbed's exact clean commit without requiring equality to the orchestration repository. A fresh read-only probe also found the current Edge artifact is Mach-O and both runtime artifacts predate that commit, so operator rebuild is an explicit resume condition. Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 Create one exact immutable readiness manifest | [ ] |
|
||||
| TEST-2 Produce one redacted nine-cell readiness record | [ ] |
|
||||
| TEST-3 Preserve the full benchmark baseline | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
|
||||
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_1.log`.
|
||||
- [ ] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm both predecessor completion files exist before reviewing implementation.
|
||||
- Confirm the complete C01-C09 map, seed-derived order, fixture checksum, two images, policies, and new rubric version are exact.
|
||||
- Confirm hybrid routes include `repair`, cloud stages use high, and work uses `ornith-fast` with no invented effort.
|
||||
- Confirm project-skill scoring uses the exact manifest-selected rubric, names the closed two-version set, and preserves predecessor 02's all-cell preflight contract.
|
||||
- Confirm the external gate captures the clean independent `../iop-s2` dev HEAD and accepts only the production adapter's exact known agy version/help contract.
|
||||
- Confirm both iop-s2 artifacts are current-HEAD AArch64 ELF executables before accepting endpoint readiness; `test -x` alone is insufficient.
|
||||
- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value.
|
||||
- Confirm no scored run/score/report command was executed in this Epic.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr under each command. If a command changes, document the replacement and reason in `Deviations from Plan`. TEST-2's full command block is fixed in the plan and must be copied with its actual output or exact blocker here.
|
||||
|
||||
### Dependency Verification
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### TEST-1 Verification
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
### TEST-2 External Preflight
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
test -n "$testbed_head"
|
||||
printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head"
|
||||
command -v readelf >/dev/null
|
||||
test -x ../iop-s2/build/bin/iop-edge
|
||||
test -x ../iop-s2/build/dev/iop-node
|
||||
test -f ../iop-s2/configs/edge.yaml
|
||||
commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test "$(stat -c %Y "$binary")" -ge "$commit_epoch"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null
|
||||
done
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)"
|
||||
printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n'
|
||||
claude --version
|
||||
agy --version
|
||||
codex --version
|
||||
python3 - <<'PY'
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
version = (version_run.stdout + version_run.stderr).strip()
|
||||
help_text = help_run.stdout + help_run.stderr
|
||||
capability = inspect_agy_iop_capability(version, help_text)
|
||||
assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION)
|
||||
assert capability.iop_transport_supported, capability
|
||||
print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented")
|
||||
PY
|
||||
python3 - <<'PY'
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)"
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
python3 - "$run_id" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert record["status"] == "ready"
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine ready results are manifest-bound and runtime values are absent")
|
||||
PY
|
||||
```
|
||||
|
||||
_Actual output or exact blocker and resume condition:_
|
||||
|
||||
### TEST-3 and Final Verification
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
_Actual output:_
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=2 tag=TEST milestone-task=fixture-lock,matrix-lock -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
두 차례 구현 전 범위 정리와 한 차례 공식 코드 리뷰를 거쳐 immutable C01-C09 manifest와 manifest-selected rubric skill 계약을 검증했으며 최종 판정은 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | 재계획 | Epic self-review에서 rubric skill 동기화와 외부 readiness gate 보강 필요를 확인했으며 구현 및 공식 판정은 없었다. |
|
||||
| `plan_cloud_G09_1.log` | `code_review_cloud_G09_1.log` | 분할 재계획 | 외부 readiness preflight를 child `05+04_readiness_preflight`로 분리하고 현재 child를 deterministic manifest 계약으로 한정했다. |
|
||||
| `plan_local_G06_2.log` | `code_review_cloud_G06_2.log` | PASS | fixture checksum, C01-C09 payload와 seed 순서, dual-rubric skill 계약 및 전체 434-test 회귀를 확인했다. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- 승인된 fixture, 두 이미지, viewport, timeout, evaluator와 C01-C09 direct/execution-preset binding을 하나의 tracked bench-02 manifest로 고정했다.
|
||||
- raw manifest와 canonical loader payload, explicit-seed 실행 순서, fixture checksum 및 manifest digest를 exact regression으로 잠갔다.
|
||||
- benchmark project skill이 manifest-selected rubric의 닫힌 두 version만 사용하고 legacy fallback 또는 재해석을 금지하도록 contract test와 함께 동기화했다.
|
||||
- 기존 living spec은 immutable manifest-selected rubric 동작을 이미 정확히 설명하므로 갱신하지 않았다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 - <<'PY' ... PY` - PASS; 선행 `02_route_preflight_contract`, `03+01_rubric_version_contract` 완료 로그가 각각 정확히 한 건 확인됐다.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness` - PASS; 1 test, `OK`.
|
||||
- `python3 -m unittest scripts.agent_benchmark.skill_contract_test` - PASS; 50 tests, `OK`.
|
||||
- `for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done` - PASS; 네 manifest 모두 유효하다.
|
||||
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 434 tests, `OK`.
|
||||
- `git diff --check` - PASS; 출력 없음.
|
||||
- repo 내부 edge-node 진단, 보조 E2E smoke, 외부 provider full-cycle은 deterministic manifest child 범위가 아니며 외부 readiness는 child `05+04_readiness_preflight`가 소유하므로 실행하지 않았다.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- `05+04_readiness_preflight`가 이 reviewed manifest를 사용해 별도 external readiness gate를 수행한다.
|
||||
|
|
@ -0,0 +1,363 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=1 tag=TEST milestone-task=fixture-lock,route-readiness,matrix-lock -->
|
||||
|
||||
# Plan - Locked C01-C09 Benchmark Readiness Manifest
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact live route evidence. This packet creates that final tracked input and runs the public preflight without substituting missing callers, credentials, models, or presets.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_local_G07_0.log`.
|
||||
- Prior review: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/code_review_cloud_G07_0.log`.
|
||||
- The prior pair contains no implementation evidence or official verdict. Epic self-review found that the final manifest would select the new rubric while the public project skill remained fixed to `landing-quality-v1`; it also found an incomplete agy resume gate and ambiguous testbed source provenance.
|
||||
- Replan atomically assigns rubric skill/contract-test synchronization to this dependent packet, requires the adapter-known agy version plus its documented transport surface, and records the independent read-only testbed's exact clean commit without requiring equality to the orchestration repository. A fresh read-only probe also found the current Edge artifact is Mach-O and both runtime artifacts predate that commit, so operator rebuild is an explicit resume condition. Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b`.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/agent_benchmark/rubric.py`
|
||||
- `scripts/agent_benchmark/rubric_test.py`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=fixture-lock,route-readiness,matrix-lock`.
|
||||
- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version; S02 requires a redacted all-cell preflight or exact blocker; S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings.
|
||||
- Evidence Map rows S01-S03 drive the static manifest assertions, secret-safe external preflight, and durable nine-result inspection below.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the complete fixture, manifest/schema, public CLI, adapter/store tests, SDD, spec, and contracts listed above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three shipped manifests validated; `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` ran 178 tests and passed.
|
||||
- Fixture evidence: prompt requires exactly root `index.html`, `styles.css`, `script.js`, vanilla HTML/CSS/JS, both local images, responsive desktop/mobile behavior, accessibility, and no external dependency. The assets resolve to exactly two images plus one reference file; checksum is `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`. Viewports are 1920x1080 and 375x812.
|
||||
- The direct example already fixes C01-C05 model/effort pairs. The approved SDD names the hybrid scenarios `gemini-hybrid` and `gpt-hybrid`; this packet uses those exact strings as both virtual request model and preset route id. Missing registration must remain `registration_required`, never fallback.
|
||||
- Planned seeded order for `bench-02-c01-c09-v1` under the reviewed domain-separated algorithm is: C02, C05, C03, C06, C08, C09, C01, C07, C04.
|
||||
- Fresh output is required; cached output is not accepted.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`.
|
||||
- Testbed: `/config/workspace/iop-s2`, branch `dev`, exact observed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean at planning time. It is the manifest-selected independent read-only runtime, so equality with the iop-s0 orchestration HEAD is neither expected nor a valid sync test; execution must capture its exact current `dev` HEAD and clean state as provenance before using existing artifacts.
|
||||
- Artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`, and `../iop-s2/configs/edge.yaml` exist, but existence is insufficient. The Edge magic bytes are Mach-O (`cf fa ed fe`) and it exits 126 with `Exec format error` on Linux; Node is AArch64 ELF. Both artifact mtimes predate current testbed HEAD (`2026-08-10T00:09:38+09:00`), so they are stale and Edge is host-incompatible. The tracked example config exposes only local example models and no benchmark route/preset catalog; do not treat it as live registration.
|
||||
- Callers: Claude Code `2.1.227`, agy `1.1.12`, Codex CLI `0.147.0` were installed. Claude/Codex help probes completed. The production adapter accepts only `agy` `1.1.11`; current `1.1.12` also lacks the required `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, and `AGY_OPENAI_API_KEY` help tokens, so it is definitively incompatible rather than merely missing a few tokens.
|
||||
- Runtime/config: every `IOP_BENCH_{CLAUDE,AGY,CODEX}_{BASE_URL,SECRET_ENV}` and `IOP_BENCH_CONFIG_OBSERVATION_ENV` was missing. Consequently runtime identity, endpoint hosts/ports, live catalog, auth, and route registration could not be observed. `ss` is unavailable; the public preflight's endpoint probe is the authoritative reachability check once URLs exist.
|
||||
- Exact resume condition: an operator must rebuild both iop-s2 artifacts from its exact clean `dev` HEAD for Linux AArch64, provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run those rebuilt Edge/Node endpoints, and install adapter-known `agy` `1.1.11` whose help satisfies every documented transport token. Execution proves both artifacts are AArch64 ELF, are not older than the captured commit, and expose their help/version commands. If adapter version support changes in a reviewed predecessor, use that code's exact `AGY_KNOWN_VERSION` and capability parser instead. No raw value is written to the repository or review file.
|
||||
- If the public preflight returns `registration_required` or `implementation_gap`, preserve its run record, record only the closed summary and run id in the review evidence, and stop. Do not edit `../iop-s2`, install tools, substitute models, or invoke callers outside the benchmark CLI.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version.
|
||||
- Existing generic preset examples do not express Gemini/GPT plan→ornith-fast work→review/repair stage bindings.
|
||||
- No static regression asserts the fixture checksum, exact two-image paths, policy fields, cell map, seeded order, and hybrid binding table together.
|
||||
- The project benchmark skill still hardcodes the legacy worksheet, and its contract test does not require the closed version catalog or manifest-selected scoring language.
|
||||
- Live readiness is currently blocked by stale/host-incompatible testbed artifacts, missing environment/config, and incompatible agy version/help evidence; the plan contains the exact secret-safe resume/preflight command.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Direct-small classification produced zero code changes. The existing fixture bytes already meet the prompt/image/workspace requirements, but their final version/checksum evidence is inseparable from the planned seeded/rubric manifest and all-cell preflight. Editing a partial manifest before those contracts land would collide with planned work.
|
||||
- This is dependent child `04+02,03_locked_benchmark_manifest`. Its stable invariant is one tracked manifest whose fixture, policies, nine exact cells, order, rubric, and durable preflight all agree.
|
||||
- Predecessor index 02 is active at `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` without `complete.log`; predecessor index 03 is active at `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` without `complete.log`. Both are currently unsatisfied; do not implement until both exact completion files exist.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations.
|
||||
- Update only the project skill's scoring wording and matching contract assertions here; preserve the all-cell preflight wording delivered by predecessor 02.
|
||||
- Do not run scored `run`, `resume`, `score`, or `report`; this Epic ends at immutable input plus readiness preflight.
|
||||
- Do not use the generic example aliases, omit `repair`, lower efforts, or replace unavailable models/presets.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `2+1+2+2+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`.
|
||||
- Review closures: all closed. Scores `2+1+2+2+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`, risk boundary matched but grade remains the basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. External rebuild/registration is a named execution precondition, not a cloud-resolvable planning gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`.
|
||||
2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`; its own index-01 dependency is already encoded there.
|
||||
3. Create and statically validate the locked manifest and regression test.
|
||||
4. Run the secret-safe external preflight once. Stop on its exact closed blocker; do not proceed to scored execution.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
|
||||
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Create one exact immutable readiness manifest
|
||||
|
||||
**Problem**
|
||||
|
||||
The generic fixture manifest has only three homogeneous presets (`agent-comparison-benchmark-manifest.example.json:42-91`), while the direct preflight example has only C01-C05 equivalents (`agent-comparison-benchmark-direct-preflight.example.json:42-108`). Neither can satisfy S01-S03.
|
||||
|
||||
**Solution**
|
||||
|
||||
Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-c01-c09-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5},
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"output_root": "agent-test/runs/bench-02"
|
||||
}
|
||||
```
|
||||
|
||||
Reuse the existing evaluator, fixture paths/checksum, and viewports byte-for-byte: `desktop_1080` at 1920x1080 and `mobile_375` at 375x812. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort.
|
||||
|
||||
| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings |
|
||||
|----|---------|--------|---------------|----------------------|-------------------|
|
||||
| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request Sonnet/max |
|
||||
| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high |
|
||||
| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high |
|
||||
| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh |
|
||||
| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh |
|
||||
| C06 | `c06-claude-gemini-hybrid` | claude | preset / `gemini-hybrid` | `gemini-hybrid` / high | Gemini/high, Gemini/high, ornith-fast, Gemini/high, Gemini/high |
|
||||
| C07 | `c07-agy-gemini-hybrid` | agy | preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 |
|
||||
| C08 | `c08-claude-gpt-hybrid` | claude | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | GPT/high, GPT/high, ornith-fast, GPT/high, GPT/high |
|
||||
| C09 | `c09-codex-gpt-hybrid` | codex | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 |
|
||||
|
||||
Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order:
|
||||
|
||||
```text
|
||||
c02-claude-gemini-direct
|
||||
c05-codex-gpt-direct
|
||||
c03-agy-gemini-direct
|
||||
c06-claude-gemini-hybrid
|
||||
c08-claude-gpt-hybrid
|
||||
c09-codex-gpt-hybrid
|
||||
c01-claude-sonnet-direct
|
||||
c07-agy-gemini-hybrid
|
||||
c04-claude-gpt-direct
|
||||
```
|
||||
|
||||
In the same dependent change, replace the project skill's fixed `landing-quality-v1` scoring sentence with a requirement to use the exact immutable manifest-selected rubric. Name the closed supported set (`landing-quality-v1`, `one-shot-agent-comparison-v1`) and forbid fallback or reinterpretation. Extend `skill_contract_test.py` so its base contract requires that wording and both version literals, while a mutation back to fixed-legacy wording fails. Preserve predecessor 02's all-cell preflight assertions unchanged.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact scoring and the closed two-version rubric set.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require that scoring contract without weakening the all-cell preflight contract.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named manifest test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters. In the skill contract test, assert both exact supported rubric versions and manifest selection, then mutation-test the legacy-only regression.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: the manifest validates, the exact lock test passes, and the skill contract suite accepts only manifest-selected rubric wording.
|
||||
|
||||
### [TEST-2] Produce one redacted nine-cell readiness record
|
||||
|
||||
**Problem**
|
||||
|
||||
Planning-time external preflight is blocked: benchmark environment references are missing, the dev runtime identity is unknown, and agy help lacks three adapter-required tokens. Static fixtures cannot claim live readiness.
|
||||
|
||||
**Solution**
|
||||
|
||||
First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them.
|
||||
|
||||
If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G09.md` and stop. Do not mutate external state.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G09.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional unit test; this is required external execution evidence. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
test -n "$testbed_head"
|
||||
printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head"
|
||||
command -v readelf >/dev/null
|
||||
test -x ../iop-s2/build/bin/iop-edge
|
||||
test -x ../iop-s2/build/dev/iop-node
|
||||
test -f ../iop-s2/configs/edge.yaml
|
||||
commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test "$(stat -c %Y "$binary")" -ge "$commit_epoch"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null
|
||||
done
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)"
|
||||
printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n'
|
||||
claude --version
|
||||
agy --version
|
||||
codex --version
|
||||
python3 - <<'PY'
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
version = (version_run.stdout + version_run.stderr).strip()
|
||||
help_text = help_run.stdout + help_run.stderr
|
||||
capability = inspect_agy_iop_capability(version, help_text)
|
||||
assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION)
|
||||
assert capability.iop_transport_supported, capability
|
||||
print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented")
|
||||
PY
|
||||
python3 - <<'PY'
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)"
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
python3 - "$run_id" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert record["status"] == "ready"
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine ready results are manifest-bound and runtime values are absent")
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: every setup command exits 0; preflight prints one ready summary with nine results; durable inspection prints its safe success line. At planning time this block is expected to stop at agy/environment setup until the recorded resume condition is satisfied.
|
||||
|
||||
### [TEST-3] Preserve the full benchmark baseline
|
||||
|
||||
**Problem**
|
||||
|
||||
The final file consumes the new order, rubric, and all-cell contracts. It must not regress legacy fixtures or other benchmark behavior.
|
||||
|
||||
**Solution**
|
||||
|
||||
Validate all four manifests, run the focused static test and complete rubric/scoring/manifest/attempt/connectivity suites, and check the diff.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G09.md`: record final command output after live readiness succeeds; if TEST-2 is blocked, leave this item unchecked and record the resume condition.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional files beyond TEST-1. Existing suites are the regression oracle; external preflight output is not cached.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: four manifests validate, all suites report `OK`, and whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | TEST-1 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | TEST-1 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G09.md` | TEST-2, TEST-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` in dependency order. Fresh output is required; cached output is not acceptable. Run the full TEST-2 command block exactly once after its setup checks pass, then run:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.skill_contract_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: static validation/test pass, TEST-2 has one nine-ready durable preflight record with no runtime values, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No scored benchmark command is run.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,244 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=2 tag=TEST milestone-task=fixture-lock,matrix-lock -->
|
||||
|
||||
# Plan - Locked C01-C09 Benchmark Manifest
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G06.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact policies. This packet creates and statically validates that tracked input before the dependent readiness-preflight packet contacts any external caller or runtime.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan/review before Epic self-review: `plan_local_G07_0.log`, `code_review_cloud_G07_0.log`.
|
||||
- Refinement source pair: `plan_cloud_G09_1.log`, `code_review_cloud_G09_1.log`.
|
||||
- Neither prior pair contains implementation evidence or an official verdict. The refinement preserves the original manifest, skill-contract, and regression scope while moving only the external readiness preflight into child `05+04_readiness_preflight`.
|
||||
- Baseline fixture checksum/shape and the 178-test manifest/attempt/connectivity suite were green at `b197e5db70637f87017a024a847e3e53fdc72e8b` in the source plan evidence.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/rubric.py`
|
||||
- `scripts/agent_benchmark/rubric_test.py`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/agent_benchmark/skill_contract_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD status is `[승인됨]`, implementation lock is released, and no unresolved user review exists.
|
||||
- First-line scope is narrowed to `milestone-task=fixture-lock,matrix-lock`.
|
||||
- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version. S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings.
|
||||
- The static manifest assertions and complete benchmark regression provide this child’s S01/S03 evidence. Live S02 evidence belongs to child 05.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- The source plan fixed fixture checksum `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`, viewports 1920x1080 and 375x812, and seeded order C02, C05, C03, C06, C08, C09, C01, C07, C04.
|
||||
- The approved hybrid route ids are `gemini-hybrid` and `gpt-hybrid`; stage order is selector, plan, work, review, repair. Cloud stages use `high`, while `ornith-fast` work omits effort.
|
||||
- The manifest/schema route-kind contract accepts only `direct` and `execution_preset`; the locked hybrid cells must use `execution_preset` exactly.
|
||||
- Fresh static validation and regression output is required; cached output is not accepted. No external runner is required for this child.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version.
|
||||
- No static regression asserts fixture checksum, exact two-image paths, policies, full cell map, seeded order, and hybrid binding table together.
|
||||
- The project benchmark skill still hardcodes the legacy worksheet, and its contract test does not require manifest-selected scoring with the closed rubric catalog.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This child owns the complete production/documentation change and every repository regression required for its PASS.
|
||||
- Child `05+04_readiness_preflight` depends on this reviewed manifest and owns only the additional external closure verification. It does not modify the manifest or duplicate the static regression.
|
||||
- Existing indices 01-04 remain fixed because their task directories contain logs; the new closure child uses the next free index 05.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations.
|
||||
- Update only the project skill's scoring wording and matching contract assertions; preserve predecessor 02's all-cell preflight wording.
|
||||
- Do not run public live preflight or any scored `run`, `resume`, `score`, or `report` command in this child.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures are all closed. Scores `2+0+2+1+1=G06`; base/final route `local-fit`, lane `local`, catalog `worker/local/G06`, filename `PLAN-local-G06.md`.
|
||||
- Review closures are all closed. Scores `2+0+2+1+1=G06`; route `official-review`, lane `cloud`, catalog `review/cloud/G06`, filename `CODE_REVIEW-cloud-G06.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`.
|
||||
2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`.
|
||||
3. Create and statically validate the locked manifest and project-skill scoring contract.
|
||||
4. On PASS, child `05+04_readiness_preflight` may run the external readiness gate.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order; atomically update project-skill scoring language and its contract test to the manifest-selected rubric catalog.
|
||||
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving live preflight and scored execution untouched.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Create one exact immutable readiness manifest
|
||||
|
||||
**Problem**
|
||||
|
||||
The generic fixture manifest has only three homogeneous presets (`scripts/fixtures/agent-comparison-benchmark-manifest.example.json:44`), while the direct preflight example has only five direct cells (`scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json:44`). Neither can satisfy S01 and S03, and the current route-kind contract accepts `direct` or `execution_preset` only (`scripts/agent_benchmark/manifest.py:33`).
|
||||
|
||||
**Solution**
|
||||
|
||||
Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-c01-c09-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5},
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"output_root": "agent-test/runs/bench-02"
|
||||
}
|
||||
```
|
||||
|
||||
Reuse the existing evaluator, fixture paths/checksum, and viewports byte-for-byte. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort.
|
||||
|
||||
| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings |
|
||||
|----|---------|--------|---------------|----------------------|-------------------|
|
||||
| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request `claude-sonnet-5`/max |
|
||||
| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / high | request `gemini-3.6-flash`/high |
|
||||
| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / high | request `gemini-3.6-flash`/high |
|
||||
| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / xhigh | request `gpt-5.6-luna`/xhigh |
|
||||
| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / xhigh | request `gpt-5.6-luna`/xhigh |
|
||||
| C06 | `c06-claude-gemini-hybrid` | claude | execution_preset / `gemini-hybrid` | `gemini-hybrid` / high | selector=`gemini-3.6-flash`/high; plan=`gemini-3.6-flash`/high; work=`ornith-fast`/omitted; review=`gemini-3.6-flash`/high; repair=`gemini-3.6-flash`/high |
|
||||
| C07 | `c07-agy-gemini-hybrid` | agy | execution_preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 |
|
||||
| C08 | `c08-claude-gpt-hybrid` | claude | execution_preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | selector=`gpt-5.6-terra`/high; plan=`gpt-5.6-terra`/high; work=`ornith-fast`/omitted; review=`gpt-5.6-terra`/high; repair=`gpt-5.6-terra`/high |
|
||||
| C09 | `c09-codex-gpt-hybrid` | codex | execution_preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 |
|
||||
|
||||
Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order:
|
||||
|
||||
```text
|
||||
c02-claude-gemini-direct
|
||||
c05-codex-gpt-direct
|
||||
c03-agy-gemini-direct
|
||||
c06-claude-gemini-hybrid
|
||||
c08-claude-gpt-hybrid
|
||||
c09-codex-gpt-hybrid
|
||||
c01-claude-sonnet-direct
|
||||
c07-agy-gemini-hybrid
|
||||
c04-claude-gpt-direct
|
||||
```
|
||||
|
||||
In the same change, replace the project skill's fixed `landing-quality-v1` scoring sentence with a requirement to use the exact immutable manifest-selected rubric. Name the closed supported set (`landing-quality-v1`, `one-shot-agent-comparison-v1`) and forbid fallback or reinterpretation. Extend `skill_contract_test.py` so its base contract requires that wording and both version literals, while a mutation back to fixed-legacy wording fails. Preserve predecessor 02's all-cell preflight assertions unchanged.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs.
|
||||
- [ ] `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`: document manifest-selected exact scoring and the closed two-version rubric set.
|
||||
- [ ] `scripts/agent_benchmark/skill_contract_test.py`: require that scoring contract without weakening the all-cell preflight contract.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named manifest test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters. In the skill contract test, assert both exact supported rubric versions and manifest selection, then mutation-test the legacy-only regression.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: the manifest validates, the exact lock test passes, and the skill contract suite accepts only manifest-selected rubric wording.
|
||||
|
||||
### [TEST-3] Preserve the full benchmark baseline
|
||||
|
||||
**Problem**
|
||||
|
||||
The final file consumes the new order, rubric, and all-cell contracts through manifest loading (`scripts/agent_benchmark/manifest.py:651`), run storage/execution (`scripts/agent_benchmark/attempts.py:426`), scoring (`scripts/agent_benchmark/scoring.py:2006`), and the public CLI (`scripts/agent_comparison_benchmark.py:273`). It must not regress legacy fixtures or any other benchmark behavior.
|
||||
|
||||
**Solution**
|
||||
|
||||
Validate all four manifests, run the focused static test and the full benchmark discovery suite, and check the diff. Do not run live preflight in this child.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G06.md`: record final static command output.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional files beyond TEST-1. Existing suites are the regression oracle.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: four manifests validate, the complete discovered benchmark suite reports `OK`, and whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | TEST-1 |
|
||||
| `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md` | TEST-1 |
|
||||
| `scripts/agent_benchmark/skill_contract_test.py` | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md` | TEST-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` after both predecessors complete; fresh output is required and cached output is not acceptable.
|
||||
|
||||
```bash
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
python3 -m unittest scripts.agent_benchmark.skill_contract_test
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: dependency check and every command exit 0, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No external preflight or scored benchmark command is run.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,320 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest plan=0 tag=TEST milestone-task=fixture-lock,route-readiness,matrix-lock -->
|
||||
|
||||
# Plan - Locked C01-C09 Benchmark Readiness Manifest
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until the dependency command resolves exactly one active or archived `complete.log` for both predecessor indices 02 and 03. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G07.md` is the mandatory last implementation step. Run every verification command, record actual notes and stdout/stderr, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. If blocked, record only the exact blocker, attempted commands/output, and resume condition in the implementation-owned evidence fields. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The repository already contains the reusable two-image vanilla landing-page fixture, but no immutable manifest combines it with the approved C01-C09 matrix, explicit seed, new rubric version, and exact live route evidence. This packet creates that final tracked input and runs the public preflight without substituting missing callers, credentials, models, or presets.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/manifest_test.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/attempts_test.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/connectivity_integration_test.py`
|
||||
- `scripts/agent_benchmark/rubric.py`
|
||||
- `scripts/agent_benchmark/rubric_test.py`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`
|
||||
- `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, status `[승인됨]`, implementation lock released, no unresolved user review.
|
||||
- First-line scope: `milestone-task=fixture-lock,route-readiness,matrix-lock`.
|
||||
- S01 requires identical prompt/assets/workspace/viewports/rubric checksum/version; S02 requires a redacted all-cell preflight or exact blocker; S03 requires the immutable nine-cell manifest, repetitions 1, explicit seed, fresh/isolated policy, timeout, and bindings.
|
||||
- Evidence Map rows S01-S03 drive the static manifest assertions, secret-safe external preflight, and durable nine-result inspection below.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No separate handoff was supplied. Repository-native evidence came from the complete fixture, manifest/schema, public CLI, adapter/store tests, SDD, spec, and contracts listed above.
|
||||
- Fresh baseline at starting HEAD `b197e5db70637f87017a024a847e3e53fdc72e8b`: all three shipped manifests validated; `python3 -m unittest scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test` ran 178 tests and passed.
|
||||
- Fixture evidence: prompt requires exactly root `index.html`, `styles.css`, `script.js`, vanilla HTML/CSS/JS, both local images, responsive desktop/mobile behavior, accessibility, and no external dependency. The assets resolve to exactly two images plus one reference file; checksum is `sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98`. Viewports are 1920x1080 and 375x812.
|
||||
- The direct example already fixes C01-C05 model/effort pairs. The approved SDD names the hybrid scenarios `gemini-hybrid` and `gpt-hybrid`; this packet uses those exact strings as both virtual request model and preset route id. Missing registration must remain `registration_required`, never fallback.
|
||||
- Planned seeded order for `bench-02-c01-c09-v1` under the reviewed domain-separated algorithm is: C02, C05, C03, C06, C08, C09, C01, C07, C04.
|
||||
- Fresh output is required; cached output is not accepted.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`.
|
||||
- Testbed: `/config/workspace/iop-s2`, branch `dev`, observed HEAD `1f2f7f...`, clean at planning time. Source sync beyond the clean dev checkout was not asserted.
|
||||
- Artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`, and `../iop-s2/configs/edge.yaml` exist. The tracked example config exposes only local example models and no benchmark route/preset catalog; do not treat it as live registration.
|
||||
- Callers: Claude Code `2.1.227`, agy `1.1.12`, Codex CLI `0.147.0` were installed. Claude/Codex help probes completed. Current agy help contains `--print`, `--output-format`, `--sandbox`, `--model`, `--effort`, and `stream-json`, but lacks the adapter-required `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, and `AGY_OPENAI_API_KEY` tokens.
|
||||
- Runtime/config: every `IOP_BENCH_{CLAUDE,AGY,CODEX}_{BASE_URL,SECRET_ENV}` and `IOP_BENCH_CONFIG_OBSERVATION_ENV` was missing. Consequently runtime identity, endpoint hosts/ports, live catalog, auth, and route registration could not be observed. `ss` is unavailable; the public preflight's endpoint probe is the authoritative reachability check once URLs exist.
|
||||
- Exact resume condition: an operator must provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run the dev Edge/Node endpoints, and install an agy build whose help satisfies the current adapter capability tokens. No raw value is written to the repository or review file.
|
||||
- If the public preflight returns `registration_required` or `implementation_gap`, preserve its run record, record only the closed summary and run id in the review evidence, and stop. Do not edit `../iop-s2`, install tools, substitute models, or invoke callers outside the benchmark CLI.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No shipped manifest contains exactly C01-C09 with one explicit seed and the approved new rubric version.
|
||||
- Existing generic preset examples do not express Gemini/GPT plan→ornith-fast work→review/repair stage bindings.
|
||||
- No static regression asserts the fixture checksum, exact two-image paths, policy fields, cell map, seeded order, and hybrid binding table together.
|
||||
- Live readiness is currently blocked by missing environment/config and incompatible agy help evidence; the plan contains the exact secret-safe resume/preflight command.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol is renamed or removed.
|
||||
- The new fixture path is consumed by `load_manifest`, public CLI commands, `RunStore`, and the manifest regression test; no new API is introduced.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Direct-small classification produced zero code changes. The existing fixture bytes already meet the prompt/image/workspace requirements, but their final version/checksum evidence is inseparable from the planned seeded/rubric manifest and all-cell preflight. Editing a partial manifest before those contracts land would collide with planned work.
|
||||
- This is dependent child `04+02,03_locked_benchmark_manifest`. Its stable invariant is one tracked manifest whose fixture, policies, nine exact cells, order, rubric, and durable preflight all agree.
|
||||
- Predecessor index 02 is active at `agent-task/m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/` without `complete.log`; predecessor index 03 is active at `agent-task/m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/` without `complete.log`. Both are currently unsatisfied; do not implement until both exact completion files exist.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not modify fixture prompt/reference/image bytes, benchmark runtime code, scoring/reporting logic, caller adapters, `../iop-s2`, credentials, or external registrations.
|
||||
- Do not run scored `run`, `resume`, `score`, or `report`; this Epic ends at immutable input plus readiness preflight.
|
||||
- Do not use the generic example aliases, omit `repair`, lower efforts, or replace unavailable models/presets.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures: scope/context/verification/evidence/ownership/decision all closed. Scores `1+1+1+2+2=G07`; base/final route `local-fit`, lane `local`, catalog `worker/local/G07`, filename `PLAN-local-G07.md`.
|
||||
- Review closures: all closed. Scores `1+1+1+2+2=G07`; route `official-review`, lane `cloud`, catalog `review/cloud/G07`, filename `CODE_REVIEW-cloud-G07.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `boundary_contract`, `structured_interpretation`, `variant_product` (`count=3`); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap. External registration is a named execution precondition, not a cloud-resolvable planning gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one active-sibling or matching archived `complete.log` for `02_route_preflight_contract`.
|
||||
2. Resolve exactly one active-sibling or matching archived `complete.log` for `03+01_rubric_version_contract`; its own index-01 dependency is already encoded there.
|
||||
3. Create and statically validate the locked manifest and regression test.
|
||||
4. Run the secret-safe external preflight once. Stop on its exact closed blocker; do not proceed to scored execution.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add the exact tracked bench-02 manifest and a static regression that locks fixture checksum/version, viewports, rubric, policies, C01-C09 bindings, seed, and seeded order.
|
||||
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [ ] [TEST-3] Run final schema, focused regression, full benchmark suite, and whitespace verification while leaving scored execution untouched.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Create one exact immutable readiness manifest
|
||||
|
||||
**Problem**
|
||||
|
||||
The generic fixture manifest has only three homogeneous presets (`agent-comparison-benchmark-manifest.example.json:42-91`), while the direct preflight example has only C01-C05 equivalents (`agent-comparison-benchmark-direct-preflight.example.json:42-108`). Neither can satisfy S01-S03.
|
||||
|
||||
**Solution**
|
||||
|
||||
Create `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` with these immutable common fields:
|
||||
|
||||
```json
|
||||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-c01-c09-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {"run_seconds": 300, "idle_seconds": 30, "quiet_seconds": 10, "cleanup_grace_seconds": 5},
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"output_root": "agent-test/runs/bench-02"
|
||||
}
|
||||
```
|
||||
|
||||
Reuse the existing evaluator, fixture paths/checksum, and desktop/mobile viewports byte-for-byte. Use the exact matrix below; stage order is selector, plan, work, review, repair. Cloud selector/plan/review/repair effort is `high`; `ornith-fast` work omits effort.
|
||||
|
||||
| ID | Cell id | Caller | Route kind/id | Request model/effort | Expected bindings |
|
||||
|----|---------|--------|---------------|----------------------|-------------------|
|
||||
| C01 | `c01-claude-sonnet-direct` | claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / max | request Sonnet/max |
|
||||
| C02 | `c02-claude-gemini-direct` | claude | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high |
|
||||
| C03 | `c03-agy-gemini-direct` | agy | direct / `gemini-3.6-flash` | Gemini / high | request Gemini/high |
|
||||
| C04 | `c04-claude-gpt-direct` | claude | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh |
|
||||
| C05 | `c05-codex-gpt-direct` | codex | direct / `gpt-5.6-luna` | GPT / xhigh | request GPT/xhigh |
|
||||
| C06 | `c06-claude-gemini-hybrid` | claude | preset / `gemini-hybrid` | `gemini-hybrid` / high | Gemini/high, Gemini/high, ornith-fast, Gemini/high, Gemini/high |
|
||||
| C07 | `c07-agy-gemini-hybrid` | agy | preset / `gemini-hybrid` | `gemini-hybrid` / high | same as C06 |
|
||||
| C08 | `c08-claude-gpt-hybrid` | claude | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | GPT/high, GPT/high, ornith-fast, GPT/high, GPT/high |
|
||||
| C09 | `c09-codex-gpt-hybrid` | codex | preset / `gpt-hybrid` | `gpt-hybrid` / xhigh | same as C08 |
|
||||
|
||||
Add `test_iop_one_shot_manifest_locks_benchmark_readiness` to `manifest_test.py`. Assert exact fixture version/checksum/assets, exactly two image workspace paths, viewports, rubric, timeout/session/cache/repetition/output root, seed, complete cell payloads, digest shape, and loaded seeded order:
|
||||
|
||||
```text
|
||||
c02-claude-gemini-direct
|
||||
c05-codex-gpt-direct
|
||||
c03-agy-gemini-direct
|
||||
c06-claude-gemini-hybrid
|
||||
c08-claude-gpt-hybrid
|
||||
c09-codex-gpt-hybrid
|
||||
c01-claude-sonnet-direct
|
||||
c07-agy-gemini-hybrid
|
||||
c04-claude-gpt-direct
|
||||
```
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: add the exact locked readiness manifest.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: add the exact static regression and recompute fixture checksum from resolved inputs.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
Write the named test. Compare complete nested cell payloads, not just counts. Recalculate the fixture checksum with production helpers and assert the manifest digest is `sha256:` plus 64 lowercase hex characters.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
```
|
||||
|
||||
Expected: the manifest validates and the exact lock test passes.
|
||||
|
||||
### [TEST-2] Produce one redacted nine-cell readiness record
|
||||
|
||||
**Problem**
|
||||
|
||||
Planning-time external preflight is blocked: benchmark environment references are missing, the dev runtime identity is unknown, and agy help lacks three adapter-required tokens. Static fixtures cannot claim live readiness.
|
||||
|
||||
**Solution**
|
||||
|
||||
First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them.
|
||||
|
||||
If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G07.md` and stop. Do not mutate external state.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G07.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional unit test; this is required external execution evidence. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
for subtask in ("02_route_preflight_contract", "03+01_rubric_version_contract"):
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)"
|
||||
test -x ../iop-s2/build/bin/iop-edge
|
||||
test -x ../iop-s2/build/dev/iop-node
|
||||
test -f ../iop-s2/configs/edge.yaml
|
||||
claude --version
|
||||
agy --version
|
||||
codex --version
|
||||
agy_help="$(agy --help 2>&1)"; for token in --print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json; do grep -F -- "$token" <<<"$agy_help" >/dev/null; done
|
||||
python3 - <<'PY'
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json)"
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
python3 - "$run_id" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert record["status"] == "ready"
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9 and all(item["status"] == "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine ready results are manifest-bound and runtime values are absent")
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: every setup command exits 0; preflight prints one ready summary with nine results; durable inspection prints its safe success line. At planning time this block is expected to stop at agy/environment setup until the recorded resume condition is satisfied.
|
||||
|
||||
### [TEST-3] Preserve the full benchmark baseline
|
||||
|
||||
**Problem**
|
||||
|
||||
The final file consumes the new order, rubric, and all-cell contracts. It must not regress legacy fixtures or other benchmark behavior.
|
||||
|
||||
**Solution**
|
||||
|
||||
Validate all four manifests, run the focused static test and complete rubric/scoring/manifest/attempt/connectivity suites, and check the diff.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G07.md`: record final command output after live readiness succeeds; if TEST-2 is blocked, leave this item unchecked and record the resume condition.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No additional files beyond TEST-1. Existing suites are the regression oracle; external preflight output is not cached.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: four manifests validate, all suites report `OK`, and whitespace check is silent.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | TEST-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G07.md` | TEST-2, TEST-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` in dependency order. Fresh output is required; cached output is not acceptable. Run the full TEST-2 command block exactly once after its setup checks pass, then run:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test.ManifestValidationTest.test_iop_one_shot_manifest_locks_benchmark_readiness
|
||||
for manifest in scripts/fixtures/agent-comparison-benchmark-manifest.example.json scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json; do python3 scripts/agent_comparison_benchmark.py validate --manifest "$manifest"; done
|
||||
python3 -m unittest scripts.agent_benchmark.rubric_test scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test scripts.agent_benchmark.attempts_test scripts.agent_benchmark.connectivity_integration_test
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: static validation/test pass, TEST-2 has one nine-ready durable preflight record with no runtime values, four manifests validate, all suites report `OK`, and `git diff --check` is silent. No scored benchmark command is run.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,288 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight plan=0 tag=TEST milestone-task=route-readiness -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight, plan=0, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refinement source pair: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log` and matching review log.
|
||||
- The source pair contains no implementation evidence or official verdict. This child preserves only its TEST-2 external readiness scope and depends on the reviewed static manifest child.
|
||||
- Source-plan read-only evidence found the current testbed blocked by host-incompatible/stale artifacts, missing benchmark environment references, and an incompatible agy version/help surface. Those facts are planning evidence, not a substitute for execution-time checks.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare the execution evidence against the fixed gate. Rerun only when the plan's setup prerequisites are present and a fresh readiness observation is required; do not run scored benchmark commands. If the gate is blocked, verify that the exact safe blocker and resume condition are recorded without leaking runtime values.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G09.md` → `code_review_cloud_G09_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-2 Produce one redacted nine-cell readiness record | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G09.md` to `code_review_cloud_G09_0.log`.
|
||||
- [x] Archive active `PLAN-cloud-G09.md` to `plan_cloud_G09_0.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/` and update this checklist at the final archive path.
|
||||
- [x] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- The fixed gate's unchanged setup commands were executed in dependency, testbed, and artifact segments so execution could stop at the first exact blocker. The artifact segment exited 69 at the current-HEAD freshness check, so the later caller/version/help, environment-reference, public preflight, and durable-record commands were not executed. No acceptance condition or command predicate was changed.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Preserved `../iop-s2` as a read-only independent testbed and captured its exact clean `dev` HEAD without changing artifacts, configuration, credentials, or route registration.
|
||||
- Honored the fail-closed boundary: no caller/model/preset/effort substitution was attempted, and the public benchmark preflight was not invoked after the setup blocker.
|
||||
- Made no production, manifest, fixture, benchmark runtime, adapter, skill, test, scoring, reporting, or external testbed change; this review evidence file is the only implementation write.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm predecessor 04 has exactly one active or archived `complete.log` before accepting any readiness evidence.
|
||||
- Confirm the external gate captures the clean independent `../iop-s2` dev HEAD and accepts only the production adapter's exact known agy version/help contract.
|
||||
- Confirm both iop-s2 artifacts are current-HEAD AArch64 ELF executables before accepting endpoint readiness; `test -x` alone is insufficient.
|
||||
- Confirm live readiness came only from public preflight and durable evidence contains no endpoint, secret, or raw config value.
|
||||
- Confirm every locked manifest cell is represented exactly once and all nine are ready.
|
||||
- Confirm no tracked production/config/test file and no scored run/score/report surface was modified or executed in this child.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual safe stdout/stderr below. If a command changes, document the replacement and reason in `Deviations from Plan`. The full external block is fixed and must be recorded with actual safe output or the exact blocker and resume condition.
|
||||
|
||||
### TEST-2 External Preflight
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() {
|
||||
printf 'blocked: %s\n' "$1" >&2
|
||||
exit 69
|
||||
}
|
||||
python3 - <<'PY' || blocked "predecessor 04 must have exactly one active or archived complete.log"
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
subtask = "04+02,03_locked_benchmark_manifest"
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev" || blocked "../iop-s2 must be on branch dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)" || blocked "../iop-s2 must be clean"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" || blocked "../iop-s2 HEAD must resolve"
|
||||
test -n "$testbed_head" || blocked "../iop-s2 HEAD must be non-empty"
|
||||
printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head"
|
||||
command -v readelf >/dev/null || blocked "readelf must be installed"
|
||||
test -x ../iop-s2/build/bin/iop-edge || blocked "current iop-edge artifact must be executable"
|
||||
test -x ../iop-s2/build/dev/iop-node || blocked "current iop-node artifact must be executable"
|
||||
test -f ../iop-s2/configs/edge.yaml || blocked "../iop-s2/configs/edge.yaml must exist"
|
||||
commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" || blocked "testbed commit time must resolve"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test "$(stat -c %Y "$binary")" -ge "$commit_epoch" || blocked "$binary must be built from the current testbed HEAD"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute on this host"
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute on this host"
|
||||
printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n'
|
||||
claude --version || blocked "claude caller must be installed"
|
||||
agy --version || blocked "agy caller must be installed"
|
||||
codex --version || blocked "codex caller must be installed"
|
||||
python3 - <<'PY' || blocked "agy must match the adapter-known version and transport help contract"
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
version = (version_run.stdout + version_run.stderr).strip()
|
||||
help_text = help_run.stdout + help_run.stderr
|
||||
capability = inspect_agy_iop_capability(version, help_text)
|
||||
assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION)
|
||||
assert capability.iop_transport_supported, capability
|
||||
print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented")
|
||||
PY
|
||||
python3 - <<'PY' || blocked "benchmark endpoint, secret, and config-observation references must be present"
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
set +e
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json 2>&1)"
|
||||
preflight_status=$?
|
||||
set -e
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_status" in
|
||||
0) case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac ;;
|
||||
69) case "$preflight_output" in *"error: preflight blocked "*) ;; *) exit 1 ;; esac ;;
|
||||
*) exit "$preflight_status" ;;
|
||||
esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
test -n "$run_id"
|
||||
python3 - "$run_id" "$preflight_status" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9
|
||||
if sys.argv[2] == "0":
|
||||
assert record["status"] == "ready" and all(item["status"] == "ready" for item in record["results"])
|
||||
else:
|
||||
assert record["status"] in {"registration_required", "implementation_gap"}
|
||||
assert any(item["status"] != "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine preflight results are manifest-bound and runtime values are absent")
|
||||
for item in record["results"]:
|
||||
issues = ",".join(f'{issue["code"]}:{issue["resume_code"]}' for issue in item["issues"]) or "none"
|
||||
print(f'cell={item["cell"]["id"]} status={item["status"]} issues={issues}')
|
||||
PY
|
||||
if [ "$preflight_status" -eq 69 ]; then
|
||||
exit 69
|
||||
fi
|
||||
```
|
||||
|
||||
_Actual safe output or exact blocker and resume condition:_
|
||||
|
||||
```text
|
||||
dependency check (exit 0):
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log
|
||||
|
||||
testbed provenance check (exit 0):
|
||||
ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
|
||||
|
||||
artifact check (exit 69):
|
||||
blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD
|
||||
```
|
||||
|
||||
Exact blocker: `../iop-s2/build/bin/iop-edge` did not satisfy the required modification-time freshness bound for clean testbed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`. The fixed gate stopped before accepting its Linux AArch64 ELF identity or executability as current-HEAD evidence.
|
||||
|
||||
Resume condition: from the exact clean `../iop-s2` `dev` HEAD, rebuild `build/bin/iop-edge` as a Linux AArch64 ELF artifact and ensure both required Edge/Node artifacts satisfy the fixed current-HEAD freshness, ELF, and host-execution checks. Then rerun the fixed gate from the dependency check. The later agy version/help, benchmark environment-reference, and public nine-cell preflight checks remain unexecuted and must also pass in that resumed gate; do not infer their current state from planning evidence.
|
||||
|
||||
Not executed after the first blocker:
|
||||
|
||||
```text
|
||||
caller version/help checks: not run
|
||||
benchmark environment-reference check: not run
|
||||
public benchmark preflight: not invoked (no run id created)
|
||||
durable nine-result and secret-absence inspection: not run
|
||||
scored run/resume/score/report commands: not run
|
||||
```
|
||||
|
||||
### Reviewer Fresh Verification (2026-08-12)
|
||||
|
||||
The reviewer reran the read-only dependency, testbed provenance, and artifact setup sequence. It stopped at the same first blocker without invoking the public preflight or any caller/provider:
|
||||
|
||||
```text
|
||||
exit 69
|
||||
ok: predecessor complete agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/complete.log
|
||||
ok: testbed branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
|
||||
blocked: ../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD
|
||||
```
|
||||
|
||||
Focused repository checks:
|
||||
|
||||
```text
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
ok: manifest is valid
|
||||
|
||||
manifest inspection
|
||||
ok: manifest_order=c02-claude-gemini-direct,c05-codex-gpt-direct,c03-agy-gemini-direct,c06-claude-gemini-hybrid,c08-claude-gpt-hybrid,c09-codex-gpt-hybrid,c01-claude-sonnet-direct,c07-agy-gemini-hybrid,c04-claude-gpt-direct
|
||||
ok: cell_prefixes=c01,c02,c03,c04,c05,c06,c07,c08,c09 unique=9 agy_known_version=1.1.11
|
||||
|
||||
git diff --check
|
||||
ok: no output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: PASS
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — the fixed setup gate stopped at the first exact current-HEAD artifact blocker and did not bypass or substitute the blocked route.
|
||||
- Completeness: Pass — TEST-2 explicitly accepts either nine ready results or an exact safe blocker and resume condition; the latter is complete and independently reproduced.
|
||||
- Test Coverage: Pass — the reviewer reproduced the blocker, validated the locked manifest, confirmed one-to-one C01-C09 coverage, and ran `git diff --check`.
|
||||
- API Contract: Pass — no caller/provider request was made after the prerequisite failure, and no route/model/effort/auth contract was substituted.
|
||||
- Code Quality: Pass — this child changes only its review evidence and contains no debug artifact, raw runtime value, or unrelated production edit.
|
||||
- Implementation Deviation: Pass — segmented execution preserves the plan's fixed order and fail-closed semantics; later checks were correctly not executed.
|
||||
- Verification Trust: Pass — fresh reviewer output matches the recorded predecessor path, clean testbed HEAD, and exact artifact blocker.
|
||||
- Spec Conformance: Pass — SDD S02 permits an exact blocker, and the evidence records the safe blocker and deterministic resume condition without asserting Milestone Task completion.
|
||||
- Findings: None
|
||||
- Routing Signals: `review_rework_count=0`, `evidence_integrity_failure=false`
|
||||
- Next Step: PASS — write `complete.log`, archive this pair and task directory, and report `milestone-task=route-readiness` only as runtime aggregation metadata.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight plan=0 tag=TEST milestone-task=route-readiness -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
한 차례 구현과 공식 코드 리뷰에서 고정 readiness gate의 첫 외부 artifact blocker와 재개 조건을 secret-safe하게 확정했으며 최종 판정은 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | PASS | clean `../iop-s2` dev HEAD의 stale `iop-edge` artifact에서 fail-closed했고, exact blocker와 전체 gate 재개 조건을 독립 재현했다. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- 선행 manifest child의 완료 로그가 정확히 한 건임을 확인하고 독립 testbed의 branch, HEAD, clean 상태를 기록했다.
|
||||
- current-HEAD artifact freshness gate의 첫 실패를 안전하게 기록하고 caller/version/help, 환경 reference, public preflight와 scored 명령을 실행하지 않았다.
|
||||
- `route-readiness`는 Milestone 완료 선언이 아닌 기여 범위 metadata로만 보존했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `TEST-2 dependency/testbed/artifact setup sequence` - BLOCKED(expected accepted outcome); exit 69, `../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD`, recorded HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 구현 evidence가 일치했다.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; `ok: manifest is valid`.
|
||||
- `python3 - <<'PY' ... load_manifest/AGY_KNOWN_VERSION inspection ... PY` - PASS; C01-C09 prefix가 각각 정확히 한 번 존재하고 known agy version은 `1.1.11`이다.
|
||||
- `git diff --check` - PASS; 출력 없음.
|
||||
- 공개 benchmark preflight, repo 내부 edge-node 진단, 보조 E2E smoke와 provider full-cycle은 artifact prerequisite에서 fail-closed했으므로 실행하지 않았다.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- 정확한 clean `../iop-s2` dev HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller/version/help, benchmark environment reference와 public nine-cell preflight를 포함한 고정 gate 전체를 다시 실행해야 한다.
|
||||
|
|
@ -0,0 +1,239 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight plan=0 tag=TEST milestone-task=route-readiness -->
|
||||
|
||||
# Plan - Redacted Nine-Cell Readiness Preflight
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Do not start until the dependency command resolves exactly one active or archived `04+02,03_locked_benchmark_manifest/complete.log`. Filling the implementation-owned sections in `CODE_REVIEW-cloud-G09.md` is the mandatory last implementation step. Run the fixed external gate exactly once after its setup checks pass, record safe output or the exact blocker and resume condition, keep both active files in place, and report ready for review; only the code-review skill may finalize or archive this task. Do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The reviewed static child creates the immutable C01-C09 input. This packet performs the remaining execution-day readiness closure through the public benchmark preflight without substituting missing callers, credentials, models, presets, runtime artifacts, or effort settings.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Refinement source pair: `agent-task/m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/plan_cloud_G09_1.log` and matching review log.
|
||||
- The source pair contains no implementation evidence or official verdict. This child preserves only its TEST-2 external readiness scope and depends on the reviewed static manifest child.
|
||||
- Source-plan read-only evidence found the current testbed blocked by host-incompatible/stale artifacts, missing benchmark environment references, and an incompatible agy version/help surface. Those facts are planning evidence, not a substitute for execution-time checks.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/agent_benchmark/manifest.py`
|
||||
- `scripts/agent_benchmark/attempts.py`
|
||||
- `scripts/agent_benchmark/connectivity.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-manifest.schema.json`
|
||||
- `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD status is `[승인됨]`, implementation lock is released, and no unresolved user review exists.
|
||||
- First-line scope is `milestone-task=route-readiness`.
|
||||
- S02 requires execution-day auth, model/preset, effort, stream/finish/idle checks for all C01-C09 or an exact blocker.
|
||||
- The required evidence is one redacted C01-C09 preflight matrix with auth/route/effort/terminal status, or the exact safe blocker and resume condition.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Runner/workdir: local Linux `aarch64`, `/config/workspace/iop-s0`.
|
||||
- Testbed: `/config/workspace/iop-s2`, branch `dev`; execution must capture its exact current clean HEAD as independent read-only provenance.
|
||||
- Source-plan evidence found `../iop-s2/build/bin/iop-edge` to be Mach-O and both runtime artifacts older than the observed testbed HEAD. It also found agy `1.1.12` while the reviewed adapter accepted `1.1.11`, and all benchmark environment references were missing.
|
||||
- Exact resume condition remains: rebuild both iop-s2 artifacts from its exact clean `dev` HEAD for Linux AArch64, provide all benchmark env references and their non-empty referenced secret/config values, register all direct/hybrid routes with exact bindings, run the rebuilt endpoints, and install the adapter-known agy version whose help satisfies every documented transport token.
|
||||
- No raw endpoint, secret, or config-observation value may be written to the repository or review file.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Static fixtures cannot prove execution-day runtime identity, artifact compatibility, credentials, route registration, caller transport, or finish/idle readiness.
|
||||
- No durable nine-cell all-ready record exists for the locked manifest.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No production symbol or tracked config is changed.
|
||||
- The public `preflight` command and durable record are the only acceptance boundary; never invoke a caller/provider directly.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is the allowed closure-verification child from the source pair. It has no production write set and consumes the exact reviewed manifest from child 04.
|
||||
- Static manifest construction and all repository regressions remain in predecessor 04; this child does not duplicate them.
|
||||
- No child created in this refinement pass is split again.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Do not edit `../iop-s2`, install tools, change credentials/registrations, or substitute models, presets, callers, or effort.
|
||||
- Do not modify the locked manifest, fixture bytes, benchmark runtime, adapters, project skill, tests, scoring, or reporting.
|
||||
- Do not run scored `run`, `resume`, `score`, or `report`; this child ends at readiness preflight evidence.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; `finalizer=finalize-task-policy.sh`, `finalizer_mode=pair`.
|
||||
- Build closures are all closed. Scores `2+1+2+2+2=G09`; base/final route `grade-boundary`, lane `cloud`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`.
|
||||
- Review closures are all closed. Scores `2+1+2+2+2=G09`; route `official-review`, lane `cloud`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product` (`count=4`; grade remains the route basis); `review_rework_count=0`; `evidence_integrity_failure=false`; no capability gap.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Resolve exactly one active-sibling or matching archived `complete.log` for `04+02,03_locked_benchmark_manifest`.
|
||||
2. Run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks in the fixed order.
|
||||
3. Only after every setup check passes, invoke the public benchmark `preflight` once and inspect its durable nine-result record.
|
||||
4. Stop on the first exact blocker or after recording the all-ready redacted evidence. Do not proceed to scored execution.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-2] Run the secret-safe caller/testbed/environment gate and public all-cell preflight; require nine ready results or record the exact blocker and resume condition without substitution.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-2] Produce one redacted nine-cell readiness record
|
||||
|
||||
**Problem**
|
||||
|
||||
Static validation cannot establish live readiness. The public CLI returns closed preflight blockers with exit 69 (`scripts/agent_comparison_benchmark.py:208`), while canonical per-cell issue/resume evidence and preflight collection are owned by `scripts/agent_benchmark/connectivity.py:406` and `scripts/agent_benchmark/attempts.py:386`. Planning-time external preflight was blocked by stale/host-incompatible artifacts, missing environment references, and an incompatible agy capability surface.
|
||||
|
||||
**Solution**
|
||||
|
||||
First run dependency, clean-testbed, artifact, caller-version/help, and secret-reference checks. The environment check dereferences names without printing names or values. Then invoke only the public benchmark `preflight` command once. Parse its run id, require `status=ready ready=9 registration_required=0 implementation_gap=0`, and inspect the durable result list against the loaded manifest. Search durable bytes for exact runtime base URLs, dereferenced secrets, and raw config observation JSON without printing them.
|
||||
|
||||
If any setup check fails or preflight returns 69, record the exact safe output and resume condition in `CODE_REVIEW-cloud-G09.md` and stop. The shell must capture preflight stderr and status without allowing `set -e` to exit before the blocker can be recorded. Do not mutate external state.
|
||||
|
||||
**Modified Files and Checklist**
|
||||
|
||||
- [ ] `CODE_REVIEW-cloud-G09.md`: record safe preflight output, run id, nine-result inspection, or exact blocker/resume evidence.
|
||||
|
||||
**Test Strategy**
|
||||
|
||||
No unit test is added. The public CLI and durable record are the acceptance oracle. Never call a caller/provider directly.
|
||||
|
||||
**Verification**
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() {
|
||||
printf 'blocked: %s\n' "$1" >&2
|
||||
exit 69
|
||||
}
|
||||
python3 - <<'PY' || blocked "predecessor 04 must have exactly one active or archived complete.log"
|
||||
from pathlib import Path
|
||||
root = Path("agent-task")
|
||||
group = "m-iop-one-shot-agent-model-comparison"
|
||||
subtask = "04+02,03_locked_benchmark_manifest"
|
||||
active = root / group / subtask / "complete.log"
|
||||
archived = sorted((root / "archive").glob(f"*/*/{group}/{subtask}/complete.log"))
|
||||
candidates = [path for path in (active, *archived) if path.is_file()]
|
||||
assert len(candidates) == 1, (subtask, candidates)
|
||||
print(f"ok: predecessor complete {candidates[0]}")
|
||||
PY
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = "dev" || blocked "../iop-s2 must be on branch dev"
|
||||
test -z "$(git -C ../iop-s2 status --short)" || blocked "../iop-s2 must be clean"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)" || blocked "../iop-s2 HEAD must resolve"
|
||||
test -n "$testbed_head" || blocked "../iop-s2 HEAD must be non-empty"
|
||||
printf 'ok: testbed branch=dev head=%s clean=true\n' "$testbed_head"
|
||||
command -v readelf >/dev/null || blocked "readelf must be installed"
|
||||
test -x ../iop-s2/build/bin/iop-edge || blocked "current iop-edge artifact must be executable"
|
||||
test -x ../iop-s2/build/dev/iop-node || blocked "current iop-node artifact must be executable"
|
||||
test -f ../iop-s2/configs/edge.yaml || blocked "../iop-s2/configs/edge.yaml must exist"
|
||||
commit_epoch="$(git -C ../iop-s2 show -s --format=%ct HEAD)" || blocked "testbed commit time must resolve"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test "$(stat -c %Y "$binary")" -ge "$commit_epoch" || blocked "$binary must be built from the current testbed HEAD"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute on this host"
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute on this host"
|
||||
printf 'ok: current-HEAD Linux AArch64 Edge/Node artifacts are executable\n'
|
||||
claude --version || blocked "claude caller must be installed"
|
||||
agy --version || blocked "agy caller must be installed"
|
||||
codex --version || blocked "codex caller must be installed"
|
||||
python3 - <<'PY' || blocked "agy must match the adapter-known version and transport help contract"
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
version = (version_run.stdout + version_run.stderr).strip()
|
||||
help_text = help_run.stdout + help_run.stderr
|
||||
capability = inspect_agy_iop_capability(version, help_text)
|
||||
assert capability.version == AGY_KNOWN_VERSION, (capability.version, AGY_KNOWN_VERSION)
|
||||
assert capability.iop_transport_supported, capability
|
||||
print(f"ok: adapter-known agy {AGY_KNOWN_VERSION} transport is documented")
|
||||
PY
|
||||
python3 - <<'PY' || blocked "benchmark endpoint, secret, and config-observation references must be present"
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references present")
|
||||
PY
|
||||
set +e
|
||||
preflight_output="$(python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json 2>&1)"
|
||||
preflight_status=$?
|
||||
set -e
|
||||
printf '%s\n' "$preflight_output"
|
||||
case "$preflight_status" in
|
||||
0) case "$preflight_output" in *"status=ready ready=9 registration_required=0 implementation_gap=0"*) ;; *) exit 1 ;; esac ;;
|
||||
69) case "$preflight_output" in *"error: preflight blocked "*) ;; *) exit 1 ;; esac ;;
|
||||
*) exit "$preflight_status" ;;
|
||||
esac
|
||||
run_id="${preflight_output#*run_id=}"; run_id="${run_id%% *}"
|
||||
test -n "$run_id"
|
||||
python3 - "$run_id" "$preflight_status" <<'PY'
|
||||
import json, os, sys
|
||||
from pathlib import Path
|
||||
from scripts.agent_benchmark.manifest import load_manifest
|
||||
manifest_path = Path("scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json")
|
||||
manifest = load_manifest(manifest_path, repo_root=Path.cwd())
|
||||
run_root = Path(manifest.output_root) / sys.argv[1]
|
||||
record = json.loads((run_root / "preflight/preflight-000001.json").read_text(encoding="ascii"))
|
||||
assert [item["cell"]["id"] for item in record["results"]] == [cell.id for cell in manifest.matrix]
|
||||
assert len(record["results"]) == 9
|
||||
if sys.argv[2] == "0":
|
||||
assert record["status"] == "ready" and all(item["status"] == "ready" for item in record["results"])
|
||||
else:
|
||||
assert record["status"] in {"registration_required", "implementation_gap"}
|
||||
assert any(item["status"] != "ready" for item in record["results"])
|
||||
sensitive = []
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
sensitive.append(os.environ[f"IOP_BENCH_{caller}_BASE_URL"].encode())
|
||||
sensitive.append(os.environ[os.environ[f"IOP_BENCH_{caller}_SECRET_ENV"]].encode())
|
||||
config_ref = os.environ["IOP_BENCH_CONFIG_OBSERVATION_ENV"]
|
||||
sensitive.append(os.environ[config_ref].encode())
|
||||
durable = b"".join(path.read_bytes() for path in run_root.rglob("*") if path.is_file())
|
||||
assert all(value and value not in durable for value in sensitive)
|
||||
print("ok: nine preflight results are manifest-bound and runtime values are absent")
|
||||
for item in record["results"]:
|
||||
issues = ",".join(f'{issue["code"]}:{issue["resume_code"]}' for issue in item["issues"]) or "none"
|
||||
print(f'cell={item["cell"]["id"]} status={item["status"]} issues={issues}')
|
||||
PY
|
||||
if [ "$preflight_status" -eq 69 ]; then
|
||||
exit 69
|
||||
fi
|
||||
```
|
||||
|
||||
Expected: every setup command exits 0. A ready preflight prints one ready summary, verifies nine ready results, and exits 0. A closed preflight blocker is captured despite `set -e`, verifies and prints only per-cell status plus canonical issue/resume codes from the durable record, and exits 69 so implementation records that exact safe blocker and the resume condition before stopping.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md` | TEST-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
Run from `/config/workspace/iop-s0` after predecessor 04 completes. Run the TEST-2 verification block exactly once after its setup checks pass. No other validation or scored benchmark command belongs to this child.
|
||||
|
||||
Expected: dependency and setup checks pass, one public preflight reports nine ready results, and durable evidence contains none of the exact runtime values. Otherwise the review stub records the exact safe blocker and resume condition.
|
||||
|
||||
After completing the readiness check, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,200 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=5 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - REVIEW_TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, run public benchmark `run|resume`, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=5, tag=REVIEW_TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다.
|
||||
- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다.
|
||||
- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다.
|
||||
- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다.
|
||||
- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G06.md` → `code_review_cloud_G06_5.log` and `PLAN-local-G06.md` → `plan_local_G06_5.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata in `complete.log` and report it for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| FIX-1: Current Codex usage schema | [x] |
|
||||
| FIX-2: Config-owned effective binding consumption | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals.
|
||||
- [x] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G06_5.log`.
|
||||
- [x] Archive active `PLAN-*-G??.md` to `plan_local_G06_5.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent or verify it was kept due to remaining siblings/files.
|
||||
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
없음. public benchmark `run|resume|score`, caller/provider 직접 호출, run tree 수정 없이 plan의 다섯 source/test/fixture 파일만 수정했다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Codex의 `cache_write_input_tokens`는 caller가 명시적으로 보고한 값만 canonical `cache_write_tokens`로 보존하고 `total_tokens`는 합성하지 않는다.
|
||||
- independently validated config observation을 execution/scoring의 canonical binding으로 사용한다. optional caller binding은 absent일 수 있지만, 보고된 non-`None` 값은 admitted tuple과 정확히 일치해야 한다.
|
||||
- scoring result에는 admitted tuple을 기록해 downstream이 caller observation 유무와 무관하게 검증된 effective binding을 소비하도록 했다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Verify `_CODEX_USAGE_FIELDS` accepts only the current explicit keys and maps `cache_write_input_tokens` to `cache_write_tokens` without deriving `total_tokens`.
|
||||
- Verify the tracked Codex fixture no longer relies on a fabricated `iop_effective_binding` event for production execution correctness.
|
||||
- Verify execution and scoring both accept `None` caller observation only after ready config admission and still reject any contradictory non-`None` observation.
|
||||
- Verify no attempt recovery, run tree, provider, remote runtime, or unrelated caller behavior changed.
|
||||
- Verify no public `run|resume|score` or direct caller/provider command was executed in this follow-up.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Focused parser and integration coverage
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
.........................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 41 tests in 14.500s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### Full benchmark suite
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
```
|
||||
|
||||
Expected: all benchmark tests pass with no skipped required test or network/provider call.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
.............................................................................................................................................................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 429 tests in 115.180s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
### Manifest and repository diff
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: `ok: manifest is valid`; `git diff --check` has no output.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
ok: manifest is valid
|
||||
```
|
||||
|
||||
`git diff --check` stdout/stderr: `(none)`
|
||||
|
||||
Reviewer fresh rerun:
|
||||
|
||||
```text
|
||||
focused: Ran 41 tests in 14.154s — OK
|
||||
full: Ran 429 tests in 116.096s — OK
|
||||
manifest: ok: manifest is valid
|
||||
git diff --check: (none)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---|---|---|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns applicable commands and records fresh output |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: FAIL
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — R1 current usage parsing and R2 config-owned execution/scoring binding now match production behavior while preserving mismatch rejection.
|
||||
- Completeness: Fail — no replacement C01-C09 scored execution exists after the retained plan=4 failure.
|
||||
- Test coverage: Pass — production-shaped usage, absent observation success, non-None mismatch failure, execution, and scoring paths have meaningful regression coverage.
|
||||
- API contract: Pass — optional caller observation and canonical config admission now have one consistent ownership contract.
|
||||
- Code quality: Pass — changes are bounded to the selected parser/consumer/test/fixture paths with no debug or dead code.
|
||||
- Implementation deviation: Pass — implementation followed R1/R2 exactly and did not invoke a caller/provider or mutate retained run state.
|
||||
- Verification trust: Pass — reviewer fresh reruns match implementation evidence: focused 41/OK, full 429/OK, manifest valid, diff check clean.
|
||||
- Spec conformance: Fail — S04-S08 still require nine retained C01-C09 terminal attempts from an authorized scored execution.
|
||||
- Findings:
|
||||
- Required R3 — Authorized replacement scored execution is still required.
|
||||
- Evidence: source-fix reviewer verification passes, but read-only public status for `run-20260812T040619Z-00a5e6664764` remains `failed=1, running=1`; `run_id.log` is absent and no replacement execution was invoked by plan=5. The benchmark skill requires stopping after a retained execution failure until the user explicitly authorizes another stateful execution.
|
||||
- Root Cause: the single scored run authorized for plan=4 was consumed before R1/R2 were known; append-only policy forbids rewriting that attempt, and plan=5 intentionally repaired only deterministic source/test boundaries.
|
||||
- Selected Fix: after explicit user authorization, create a new execution plan that rebinds the protected local token/CA references, requires fresh public preflight `ready=9`, invokes `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once, stores only its CLI-emitted run id, and accepts only nine retained terminal attempts with zero `running|interrupted`. Preserve every older run and do not call direct callers/providers or edit state.
|
||||
- Routing Signals: review_rework_count=2 evidence_integrity_failure=false
|
||||
- Next Step: USER_REVIEW external-execution — obtain explicit authorization for exactly one new public C01-C09 scored run after the retained failure; then re-enter the plan skill for this exact task path.
|
||||
|
|
@ -0,0 +1,208 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=0, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`다. 미해결 Required/Suggested는 기재되지 않았고 잔여 Nit은 없다.
|
||||
- 선행 작업은 source를 변경하지 않았고, clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 stale `../iop-s2/build/bin/iop-edge` 사이의 freshness blocker를 exit 69로 재현했다. Manifest validate는 `ok: manifest is valid`였다.
|
||||
- Roadmap carryover는 위 exact HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller version/help, secret-safe environment reference, public nine-cell preflight를 포함한 gate를 다시 통과하는 것이다. 구현자는 archive 전체를 탐색하지 말고 추가 세부가 꼭 필요할 때만 위 `complete.log` 한 건을 읽는다.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_0.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_0.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Confirm the fixed external preconditions without exposing secrets, execute the immutable C01-C09 `run` exactly once, persist its canonical run id, and verify all nine attempts reach success with unresolved 0; if the gate blocks or an attempt fails, do not retry and record the exact evidence and resume condition.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_0.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_0.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm `run_id.log` contains the id emitted by the one and only `run` invocation and was not fabricated or replaced.
|
||||
- Confirm verbatim execution output reports `completed=9`, `unresolved=0`, `success=9`, and zero failed/timed_out/cancelled/interrupted/running attempts; independently rerun only `status`, never `run` or `resume`.
|
||||
- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid`, with timing/usage/workspace and hybrid stage evidence validated by the run.
|
||||
- Confirm no secret, raw private endpoint/config payload, caller substitution, retry, score, or report was introduced and `../iop-s2` stayed clean.
|
||||
|
||||
## Verification Results
|
||||
|
||||
> The implementing agent must replace each placeholder below with the exact command result and verbatim stdout/stderr. If blocked, retain unchecked completion boxes and record the exit code, exact public error, and resume condition. The review agent reruns only commands marked repeatable; the one-time provider execution must be evaluated from its preserved evidence and public `status`.
|
||||
|
||||
### Static manifest and harness tests — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: manifest valid; 91 tests pass.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### External execution preconditions — repeatable and secret-safe
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
test "$(uname -s)" = Linux
|
||||
test "$(uname -m)" = aarch64
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
|
||||
for tool in python3 claude agy codex git; do command -v "$tool"; done
|
||||
python3 - <<'PY'
|
||||
import os
|
||||
|
||||
ok = True
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
base = bool(os.environ.get(f"IOP_BENCH_{caller}_BASE_URL"))
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
secret = bool(ref and os.environ.get(ref))
|
||||
print(f"{caller}: base_reference={base} secret_reference={bool(ref)} referenced_secret={secret}")
|
||||
ok = ok and base and bool(ref) and secret
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
config = bool(config_ref and os.environ.get(config_ref))
|
||||
print(f"CONFIG: reference={bool(config_ref)} referenced_value={config}")
|
||||
ok = ok and bool(config_ref) and config
|
||||
raise SystemExit(0 if ok else 69)
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: exact clean testbed branch/HEAD; five binaries; every printed boolean is `True`.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### C01-C09 run — implementation-only, do not repeat in review
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
bench_stdout="$(mktemp)"
|
||||
bench_stderr="$(mktemp)"
|
||||
trap 'rm -f "$bench_stdout" "$bench_stderr"' EXIT
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$bench_stdout" 2>"$bench_stderr"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'exit=%s\n' "$bench_exit"
|
||||
printf '%s\n' '--- stdout ---'
|
||||
cat "$bench_stdout"
|
||||
printf '%s\n' '--- stderr ---'
|
||||
cat "$bench_stderr"
|
||||
bench_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$bench_stdout" "$bench_stderr" | sed -n '1p')"
|
||||
test -n "$bench_run_id"
|
||||
printf '%s\n' "$bench_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id"
|
||||
test "$bench_exit" -eq 0
|
||||
```
|
||||
|
||||
Expected: exit 0; completed 9, unresolved 0, success 9, all other counts 0; one canonical `run_id.log`.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output, including the exit line and both labeled streams._
|
||||
|
||||
### Recorded run status and worktree — repeatable, no provider invocation
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: valid run id; status success 9 and every other count 0; `git diff --check` has no output.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,273 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=1 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=1, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 readiness 확인 절차가 exact blocker를 보존했다는 뜻이며 `route-readiness` 완료 선언이 아니다.
|
||||
- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`에서 Edge/Node artifact freshness, Linux AArch64 identity, caller version/help, environment reference를 모두 통과한 뒤에만 live benchmark를 시작하도록 요구한다.
|
||||
- 최초 active pair는 공식 verdict 없이 `plan_cloud_G09_0.log`와 `code_review_cloud_G10_0.log`로 보존했다. 그 pair의 success-only acceptance와 불완전한 artifact/caller preflight는 이 generation이 대체하며 구현자는 이전 pair를 다시 읽지 않는다.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_1.log` and `PLAN-cloud-G09.md` → `plan_cloud_G09_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Prove every fixed external precondition, invoke the immutable C01-C09 `run` exactly once, preserve its canonical run id and verbatim CLI result, and accept only either nine success results or nine retained non-interrupted terminal results; never retry, resume, substitute, or treat preflight/partial execution as completion.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_1.log`.
|
||||
- [ ] Archive active `PLAN-*-G??.md` to `plan_cloud_G09_1.log`.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores local `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-iop-one-shot-agent-model-comparison/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm every fixed gate passed before the one and only `run` invocation; in particular both artifacts are current-HEAD Linux AArch64 ELF, agy matches the adapter-known version/help contract, and environment references were dereferenced without disclosure.
|
||||
- Confirm `run_id.log` names the one exact newly created run and matches any CLI-emitted id; it must not point to the preparation-only preflight run.
|
||||
- Confirm public status validates exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted`. Retained failures are valid execution evidence; zero or partial attempts are not.
|
||||
- Confirm the CLI exit/output and state classification agree, no retry/resume/score/report was invoked, no secret or raw private endpoint/config payload was recorded, and `../iop-s2` stayed clean.
|
||||
- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid` under the immutable manifest.
|
||||
|
||||
## Verification Results
|
||||
|
||||
> The implementing agent must replace each placeholder below with the exact command result and verbatim stdout/stderr. If a fixed gate or partial execution blocks, retain unchecked completion boxes and record the exact public error and resume condition. The review agent reruns only commands marked repeatable; the one-time provider execution must be evaluated from preserved evidence and public `status`.
|
||||
|
||||
### Static manifest and harness tests — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
Expected: manifest valid; 91 tests pass.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### Fixed external execution preconditions — repeatable and secret-safe
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
|
||||
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
|
||||
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
|
||||
command -v readelf >/dev/null || blocked "readelf must be installed"
|
||||
command -v go >/dev/null || blocked "go must be installed"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test -x "$binary" || blocked "$binary must be executable"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
|
||||
import subprocess, sys
|
||||
expected = sys.argv[1]
|
||||
for binary in sys.argv[2:]:
|
||||
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
|
||||
build = {}
|
||||
for line in output.splitlines():
|
||||
fields = line.strip().split("\t", 1)
|
||||
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
|
||||
key, value = fields[1].split("=", 1)
|
||||
build[key] = value
|
||||
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
|
||||
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
|
||||
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
|
||||
print("ok: Edge/Node build identities match the clean testbed HEAD")
|
||||
PY
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute"
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute"
|
||||
for tool in python3 claude agy codex git; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
|
||||
python3 - <<'PY' || blocked "agy must match the adapter-known version and transport contract"
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
capability = inspect_agy_iop_capability((version_run.stdout + version_run.stderr).strip(), help_run.stdout + help_run.stderr)
|
||||
assert capability.version == AGY_KNOWN_VERSION
|
||||
assert capability.iop_transport_supported
|
||||
print("ok: agy IOP transport capability")
|
||||
PY
|
||||
python3 - <<'PY' || blocked "benchmark environment references must be present"
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references")
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: all checks exit 0. Any blocker stops before the scored command.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### C01-C09 run — implementation-only, do not repeat in review
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
run_root=agent-test/runs/bench-02
|
||||
before="$(mktemp)"; after="$(mktemp)"; new_runs="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
|
||||
trap 'rm -f "$before" "$after" "$new_runs" "$out" "$err"' EXIT
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
|
||||
printf '%s\n' 'stderr:'; cat "$err"
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' | sort >"$after"
|
||||
comm -13 "$before" "$after" >"$new_runs"
|
||||
test "$(wc -l <"$new_runs")" -eq 1
|
||||
discovered_run_id="$(cat "$new_runs")"
|
||||
emitted_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$out" "$err" | sed -n '1p')"
|
||||
if test -n "$emitted_run_id"; then test "$emitted_run_id" = "$discovered_run_id"; fi
|
||||
printf '%s\n' "$discovered_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$discovered_run_id")"
|
||||
printf 'status_stdout:\n%s\n' "$status_output"
|
||||
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
|
||||
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
|
||||
bench_exit = int(os.environ["BENCH_EXIT"])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
|
||||
assert bench_exit == (0 if states["success"] == 9 else 69)
|
||||
print(f"classification=execution_complete states={states}")
|
||||
raise SystemExit(0)
|
||||
if sum(states.values()) == 0 and bench_exit == 69:
|
||||
print(f"classification=blocked_preflight states={states}")
|
||||
else:
|
||||
print(f"classification=blocked_partial states={states}")
|
||||
raise SystemExit(69)
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: exactly one new run id and `classification=execution_complete`. The original CLI exit may be 69 only when all nine non-interrupted terminal attempts are retained and public status proves their states.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output, including command, exit code, both streams, status and classification._
|
||||
|
||||
### Recorded run status and worktree — repeatable, no provider invocation
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
|
||||
printf '%s\n' "$status_output"
|
||||
STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
|
||||
print(f"ok: nine retained terminal attempts states={states}")
|
||||
PY
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: valid run id; nine retained accepted terminal attempts; zero interrupted/running; no diff whitespace errors.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,265 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=2 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - C01-C09 원샷 비교 실행
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=2, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` proves task-protocol dependency completion while preserving a fail-closed live-readiness blocker; it does not mark Milestone `route-readiness` complete.
|
||||
- Generation 0/1 are preserved as `plan_cloud_G09_0.log`, `code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`, `code_review_cloud_G10_1.log`, all without an official verdict. Active plan=2 replaces their semantic defects; do not execute archived commands.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. Never repeat the one-time provider execution in Final Verification step 3. If implementation is present, repair missing or stale repeatable verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating a follow-up plan.
|
||||
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-{review_lane}-{review_grade}.md` and `PLAN-{build_lane}-{build_grade}.md` to the next canonical log indices.
|
||||
3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/{task_name}/`. If WARN/FAIL, write the next filesystem state required by the code-review skill.
|
||||
4. If PASS, preserve the first-line `milestone-task` metadata and report it for runtime aggregation. Roadmap evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| TEST-1: Immutable C01-C09 comparison run | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [ ] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [ ] Run applicable required repeatable verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [ ] Archive active `CODE_REVIEW-*-G??.md` and `PLAN-*-G??.md` to the next canonical log indices.
|
||||
- [ ] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` from the canonical template and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory to `agent-task/archive/YYYY/MM/{task_name}/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report the five exact `milestone-task` ids for runtime aggregation without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove an empty active parent or verify it remains because siblings/files exist.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching the verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record any deviations from the plan and the rationale here._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record key design decisions here._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm active PLAN/review headers both use plan=2 and the exact five-task union: `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`.
|
||||
- Confirm active index `06+05`, exact archived predecessor `05+04/.../complete.log`, and the preserved readiness blocker; do not infer that `route-readiness` is complete.
|
||||
- Confirm S04-S08 map C01-C09 to those five tasks and source/spec/contract/SDD/Milestone remained outside the write set.
|
||||
- Confirm the implementer did not directly invoke a caller/provider, inspect secret values, duplicate live-preflight policy, rebuild/mutate `../iop-s2`, or invoke `resume`, retry, score, evaluator, or report.
|
||||
- Confirm step 2 proved fixed host/testbed/artifact identity and command presence before any public `run` call.
|
||||
- Confirm step 3 was invoked at most once, the pointer came only from one unique CLI-emitted id, that id did not preexist, and its exact run directory exists. Ignore unrelated concurrent run directories.
|
||||
- Confirm public status validates exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted`. Retained failure is valid execution evidence; zero/partial/invalid/unidentified state is not.
|
||||
- Confirm CLI exit/output, state classification and checklist agree, no secret/raw private endpoint/config payload was recorded, and `../iop-s2` stayed clean.
|
||||
- Confirm C01 maps to `claude-standalone`, C02-C03 to `gemini-standalone`, C04-C05 to `gpt-standalone`, C06-C07 to `gemini-hybrid`, and C08-C09 to `gpt-hybrid`.
|
||||
|
||||
## Verification Results
|
||||
|
||||
> The implementing agent must replace each placeholder with exact command results and verbatim stdout/stderr. If a fixed gate or execution state blocks, retain unchecked completion boxes and record the exact public error and resume condition. The review agent reruns only sections marked repeatable; the one-time provider execution is evaluated from preserved evidence and public `status`.
|
||||
|
||||
### Static manifest and full benchmark suite — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
```
|
||||
|
||||
Expected: manifest valid and all discovered benchmark tests pass.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### Fixed host/testbed/artifact gate — repeatable, no caller/provider invocation
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
|
||||
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
|
||||
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
|
||||
for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test -x "$binary" || blocked "$binary must be executable"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
|
||||
import subprocess, sys
|
||||
expected = sys.argv[1]
|
||||
for binary in sys.argv[2:]:
|
||||
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
|
||||
build = {}
|
||||
for line in output.splitlines():
|
||||
fields = line.strip().split("\t", 1)
|
||||
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
|
||||
key, value = fields[1].split("=", 1)
|
||||
build[key] = value
|
||||
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
|
||||
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
|
||||
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
|
||||
print("ok: Edge/Node build identities match the clean testbed HEAD")
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: all checks exit 0. Any blocker stops before the scored command.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
### C01-C09 run — implementation-only, do not repeat in review
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
|
||||
run_root=agent-test/runs/bench-02
|
||||
test ! -e "$task_dir/run_id.log"
|
||||
before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
|
||||
trap 'rm -f "$before" "$out" "$err"' EXIT
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
|
||||
printf '%s\n' 'stderr:'; cat "$err"
|
||||
emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)"
|
||||
emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)"
|
||||
if test "$emitted_count" -ne 1; then
|
||||
printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count"
|
||||
exit 69
|
||||
fi
|
||||
bench_run_id="$(printf '%s\n' "$emitted_ids")"
|
||||
if grep -Fxq "$bench_run_id" "$before"; then
|
||||
printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id"
|
||||
exit 69
|
||||
fi
|
||||
test -d "$run_root/$bench_run_id"
|
||||
printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log"
|
||||
set +e
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)"
|
||||
status_exit=$?
|
||||
set -e
|
||||
printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output"
|
||||
if test "$status_exit" -ne 0; then
|
||||
printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id"
|
||||
exit 69
|
||||
fi
|
||||
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
|
||||
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
|
||||
bench_exit = int(os.environ["BENCH_EXIT"])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
|
||||
assert bench_exit == (0 if states["success"] == 9 else 69)
|
||||
print(f"classification=execution_complete states={states}")
|
||||
raise SystemExit(0)
|
||||
if sum(states.values()) == 0 and bench_exit == 69:
|
||||
print(f"classification=blocked_preflight states={states}")
|
||||
else:
|
||||
print(f"classification=blocked_partial states={states}")
|
||||
raise SystemExit(69)
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: one CLI-emitted new run id and `classification=execution_complete`. Exit 69 is accepted only when all nine non-interrupted terminal attempts are retained.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output, including command, exit code, both streams, status and classification._
|
||||
|
||||
### Recorded run status and worktrees — repeatable, no provider invocation
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
test "$(wc -l <"$pointer")" -eq 1
|
||||
bench_run_id="$(tr -d '\n' <"$pointer")"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
|
||||
printf '%s\n' "$status_output"
|
||||
STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
|
||||
print(f"ok: nine retained terminal attempts states={states}")
|
||||
PY
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed and no whitespace errors.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
_Fill with verbatim output._
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
|
@ -0,0 +1,206 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=3 tag=API milestone-task=agy-iop-compatibility,managed-credential-dev,route-readiness,claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - official agy IOP 호환, managed dev 배포와 C01-C09 실행
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Finalization (`Code Review Result`, archive, `complete.log`) is review-agent-only.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=3, tag=API
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Generation 0/1/2 are preserved in the same task directory and have no official verdict. Active plan=3 replaces the invalid synthetic agy/legacy credential premise while preserving the immutable one-run rule.
|
||||
- Exact predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` satisfies the index dependency but does not remove the current live-readiness blocker.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
Compare every implementation item with source and fresh repeatable verification. Never repeat the one-time C01-C09 provider run. Append verdict and routing signals, archive the active pair, and create `complete.log` only on PASS.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| API-1: Gemini-native Edge ingress | [x] |
|
||||
| API-2: official agy 1.1.12 adapter | [x] |
|
||||
| API-3: managed dev deployment and live readiness | [x] |
|
||||
| API-4: immutable C01-C09 run | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] API-1: Implement and test the route-qualified Gemini request/auth/SSE bridge through existing route and preset admission.
|
||||
- [x] API-2: Use official agy 1.1.12 Gemini API-key transport and real stream-json lifecycle with config-owned binding evidence.
|
||||
- [x] API-3: Deploy fresh artifacts and a complete managed credential/TLS composition to dev, proving direct/hybrid readiness without legacy fallback.
|
||||
- [ ] API-4: Invoke the public C01-C09 run exactly once after readiness and retain exactly nine non-interrupted terminal attempts.
|
||||
- [x] Fill all implementation-owned sections below with actual notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
- [ ] Append one verdict and verified `review_rework_count` / `evidence_integrity_failure` signals.
|
||||
- [ ] Run all applicable repeatable verification and compare it with implementation evidence.
|
||||
- [ ] Confirm no inbound principal token can become provider authorization and no secret appears in tracked evidence.
|
||||
- [ ] Confirm official agy argv/env/events and config-owned effective binding match the contract.
|
||||
- [ ] Confirm dev has CA-signed TLS, projection, slot-route authorization and no legacy credential fallback.
|
||||
- [ ] Confirm the scored run occurred at most once and has exactly nine retained terminal attempts.
|
||||
- [ ] Archive the active pair and write `complete.log` only for PASS.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- 공식 `agy` 1.1.12의 실제 JSONL은 flat `status`/`usage`가 아니라 `event`와 같은 이름의 `init`/`step_update`/`result` 중첩 payload를 사용했다. adapter, fixture와 integration seam을 실제 출력 구조로 교정했다.
|
||||
- `/tmp`는 실행 불가라 deterministic credential smoke의 임시 실행 경로를 ignored `build/test-tmp`로 옮겼다. 이 과정에서 smoke가 내부 route id를 기다리던 오래된 판정을 찾아, 계약대로 public route alias를 기다리도록 수정했다.
|
||||
- 공식 agy hybrid와 raw Chat hybrid에서 plan output의 마지막 LF 생략, work/review tool path와 review JSON strictness 문제가 드러나 parser의 최종 LF 경계, strict tool schema/한 번의 bounded correction, review response schema를 보완했다.
|
||||
- 공개 `run`은 all-cell preflight 뒤 정확히 한 번 호출했지만 첫 caller launch 전에 shared filesystem이 symlink 경유 Unix socket bind를 `EINVAL`로 거부했다. 상대 basename bind와 0700 parent fallback, pre-registration reconcile을 구현하고 회귀 테스트를 추가했다. one-run 원칙에 따라 `run`/`resume`은 다시 호출하지 않았다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Gemini ingress는 별도 provider 우회 경로를 만들지 않고 인증된 내부 Chat request로 변환해 기존 route/preset admission, managed credential projection과 stage authorization을 그대로 사용한다.
|
||||
- inbound `x-goog-api-key`는 IOP principal token으로만 사용하고 provider credential은 slot/lease projection에서만 가져온다. caller token, provider token과 private key는 tracked evidence에 기록하지 않는다.
|
||||
- official caller child에는 최소 환경을 유지하되 private dev CA가 필요한 경우 표준 `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS` 두 값만 명시적으로 전달한다.
|
||||
- pre-registration interruption은 invocation identity가 존재하지 않으므로 불가능한 measurement/web sidecar를 발명하지 않는다. locator 또는 일부 sidecar가 존재하는 경우에는 기존 엄격 검증을 유지한다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm `/gemini/{route}/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse` is authenticated using the IOP principal, while provider credentials still come only from managed projection.
|
||||
- Confirm malformed/conflicting auth and unsupported Gemini bodies fail before dispatch, and stream cancellation/error/tool/usage paths remain bounded and caller-safe.
|
||||
- Confirm agy uses `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`, official model label and no `--effort` or invented `AGY_*` variables/events.
|
||||
- Confirm remote config and live evidence prove managed credentials, HTTPS/mTLS, exact direct/hybrid routes and no static provider credential source.
|
||||
- Confirm C01-C09 is invoked only after all-cell preflight; no retry/resume/manual result editing is allowed.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Focused local suites — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/edge/internal/openai
|
||||
python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
ok iop/apps/edge/internal/openai 8.326s
|
||||
ok iop/apps/edge/internal/service 8.252s
|
||||
ok iop/packages/go/singlerequesttemplate 0.006s
|
||||
|
||||
Ran 62 tests in 19.224s
|
||||
OK
|
||||
```
|
||||
|
||||
### Full local suites and contract hygiene — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
go test -p 1 -count=1 ./...
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
go test -p 1 -count=1 ./...: PASS (all packages)
|
||||
Ran 429 tests in 115.981s
|
||||
OK
|
||||
ok: manifest is valid
|
||||
git diff --check: PASS
|
||||
```
|
||||
|
||||
첫 full Go 병렬 검증에서는 unrelated transport duplicate-registration test 한 건이 45초 timeout이었으나 단독 0.01초, package 4.766초에 통과했고 부하 없이 반복한 전체 `-p 1` run은 모두 통과했다.
|
||||
|
||||
### Managed credential deterministic qualification — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)"
|
||||
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
|
||||
rmdir "$credential_smoke_parent"
|
||||
```
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
mode=deterministic
|
||||
profiles=seulgi_chat,seulgi_messages
|
||||
same_model_two_slot=true
|
||||
exact_auth.chat=2
|
||||
exact_auth.messages=1
|
||||
ciphertext_only=true
|
||||
tls_negative_matrix=passed
|
||||
post_revoke.counters_unchanged=true
|
||||
post_revoke.no_fallback=true
|
||||
result=success
|
||||
```
|
||||
|
||||
실제 slot/route 식별자는 secret-safe evidence 요구에 따라 생략했다.
|
||||
|
||||
### Remote managed dev and live caller readiness — repeatable except provisioning
|
||||
|
||||
Record sanitized source/artifact/config identity, runtime health, official agy direct/hybrid smoke and public preflight output. Do not record secrets, raw request content, private keys, slot aliases or lease ids.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
edge_sha256=c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc
|
||||
edge_pid=172
|
||||
ports=18082,18083,18084,19093,19101 all_up
|
||||
nodes=4 connected=4 providers=8 provider_states=available/healthy
|
||||
nodes=mac-codex-node,gx10-vllm-node,onexplayer-lemonade-node,rtx5090-lemonade-node
|
||||
|
||||
raw gemini-hybrid smoke #1: HTTP 200, finish=stop, errors=0, done=true
|
||||
raw gemini-hybrid smoke #2: HTTP 200, finish=stop, errors=0, done=true
|
||||
official agy direct: exit=0, result.status=SUCCESS, stderr_bytes=0
|
||||
official agy hybrid: exit=0, result.status=SUCCESS, workspace_unchanged=true, stderr_bytes=0
|
||||
ok: preflight status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
```
|
||||
|
||||
### C01-C09 run — implementation-only, never repeat in review
|
||||
|
||||
Record the single public `run` command exit/output, its one CLI-emitted run id, and public `status` classification.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
$ python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
error: benchmark state is unavailable
|
||||
exit=69
|
||||
```
|
||||
|
||||
CLI-emitted run id는 없었으므로 `run_id.log`는 생성하지 않았다. Controller가 만든 state는 첫 slot에서 `caller_launched=false`, `terminal_reason=supervisor_error`, cleanup 완료였다. 원인은 symlink path Unix socket `bind(2)`의 `EINVAL`이며 수정 후 symlink control socket 및 pre-registration reconcile 회귀 테스트가 통과했다. 같은 run을 재시도하거나 resume하지 않았다.
|
||||
|
||||
### Recorded run status and worktrees — repeatable
|
||||
|
||||
Record public status for the stored pointer, nine terminal attempts, zero running/interrupted, and `git diff --check`.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
controller_state=run-20260812T031657Z-0a24376c2414
|
||||
status before reconcile: running=1, all other terminal counts=0
|
||||
status after harness reconcile: interrupted=1, running=0, success=0, failed=0, timed_out=0, cancelled=0
|
||||
caller_launched=false
|
||||
live_supervisor=false
|
||||
control_alias_count=0
|
||||
run_id.log=absent
|
||||
testbed=../iop-s2 branch=dev head=1f2f7f1066fcf165a9e469bae77203b569b6f772 clean=true
|
||||
git diff --check: PASS
|
||||
```
|
||||
|
||||
API-4 acceptance인 nine terminal attempts/zero interrupted는 충족하지 못했다. resume/retry 없이 새 실행 정책 승인이 있어야 C01-C09를 다시 시작할 수 있다.
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section and check only completed items. Leave review-only sections unchanged.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
_Review-agent only._
|
||||
|
|
@ -0,0 +1,267 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=4 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected scope and state boundary as written. Do not choose another owner, narrow/expand the write boundary, or replace the run with an alternate caller path.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=4, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다.
|
||||
- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다.
|
||||
- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다.
|
||||
- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against the active Plan and immutable run state. Repeat only local tests, manifest validation, secret-safe preflight and public `status`; never repeat `run`, invoke `resume`, use `--retry-failed`, call a caller/provider directly, or edit a run tree. Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_4.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_4.log`.
|
||||
3. If PASS, write `complete.log` and move the active task directory to `agent-task/archive/YYYY/MM/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; if WARN/FAIL, write the next state required by the code-review skill.
|
||||
4. Preserve first-line `milestone-task` metadata for runtime aggregation. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1: Protected runtime binding and fresh readiness | [x] |
|
||||
| TEST-2: One replacement C01-C09 execution run | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output.
|
||||
- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable repeatable verification and record fresh output; do not repeat the execution run.
|
||||
- [x] Confirm prior interrupted/preflight-only run roots remain append-only and no stored pointer was written for the non-emitted execution id.
|
||||
- [x] Confirm public status is `failed=1, running=1` rather than the required nine terminal attempts and classify the gap as Required.
|
||||
- [x] Confirm no secret value, raw prompt/response, private key, slot alias or lease id appears in tracked evidence.
|
||||
- [x] Archive active `CODE_REVIEW-cloud-G10.md` to `code_review_cloud_G10_4.log`.
|
||||
- [x] Archive active `PLAN-cloud-G10.md` to `plan_cloud_G10_4.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` and leave no active `.md` files.
|
||||
- [ ] If PASS, move the active task directory to the archive path and update this checklist at the final archive path.
|
||||
- [ ] If PASS, preserve and report `milestone-task` metadata without modifying roadmap directly.
|
||||
- [x] If WARN/FAIL, write the next filesystem state matching the code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- Planning 중 보호된 runtime reference가 없는 셸에서 공개 preflight를 호출해 `implementation_gap=9`가 발생했다. attempt는 생성되지 않았으며 `token/.iop-bench`, private dev CA와 배포 config observation을 process-local environment에 연결한 뒤 `ready=9`를 확인했다.
|
||||
- 최초 execution wrapper는 임시파일 cleanup 구문이 host safety policy에서 process 시작 전에 거부됐다. public benchmark CLI는 호출되지 않았고 active Plan/Review의 command를 direct CLI output 방식으로 교정한 뒤 실행했다.
|
||||
- 교정 후 public `run`은 정확히 한 번 호출됐다. 첫 read-only status는 `failed=1,running=1`이었고 CLI는 이후 `error: benchmark state is unavailable`, exit 69로 종료했다. CLI-emitted run id가 없고 최종 public status도 `failed=1,running=1`이라 TEST-2 acceptance를 충족하지 못했다.
|
||||
- observed controller directory는 `run-20260812T040619Z-00a5e6664764`지만 CLI-emitted canonical id가 아니므로 `run_id.log`를 생성하지 않았다. 해당 run을 resume/retry/reconcile하거나 run tree를 편집하지 않았다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- preflight/run/status는 모두 public deterministic benchmark CLI만 사용했다. caller/provider 직접 호출이나 route/model/effort 대체는 하지 않았다.
|
||||
- retained failure와 dangling status를 성공으로 재분류하지 않는다. 실행 process가 없는 것을 read-only로 확인했지만 내부 `RunStore`나 reconcile API로 상태를 바꾸지 않는다.
|
||||
- TEST-2는 exact blocker 상태로 남긴다. 재개 조건은 official review가 `benchmark state is unavailable`의 root cause와 one selected fix를 확정하고, append-only evidence를 보존하는 follow-up plan을 생성하는 것이다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Confirm the no-reference `implementation_gap=9` preflight and the corrected `ready=9` preflight allocated no scored attempts.
|
||||
- Confirm TEST-2 uses one new public `run` identity and does not resume or modify `run-20260812T031657Z-0a24376c2414`.
|
||||
- Confirm each C01-C09 cell has one retained accepted terminal attempt under the new run, with fresh workspace/session identity and IOP route binding.
|
||||
- Confirm `run_id.log` contains exactly the CLI-emitted new execution run id and public `status` matches the recorded evidence.
|
||||
- Never use execution success alone to complete the third Epic; validation/scoring/report remain separately planned S09-S12 work.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Fresh local contract checks — repeatable
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
.....................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 69 tests in 59.949s
|
||||
|
||||
OK
|
||||
ok: manifest is valid
|
||||
```
|
||||
|
||||
`git diff --check` stdout/stderr: `(none)`
|
||||
|
||||
### Protected all-cell preflight — repeatable, no scored attempt
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
|
||||
export IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083
|
||||
export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083
|
||||
export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1
|
||||
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
|
||||
BENCH_CONFIG="$(python3 - <<'PY'
|
||||
import json
|
||||
routes = [
|
||||
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
|
||||
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
|
||||
]
|
||||
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
|
||||
PY
|
||||
)"
|
||||
export BENCH_CONFIG
|
||||
for tool in python3 claude agy codex git; do command -v "$tool"; done
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
/bin/python3
|
||||
/config/.npm-global/bin/claude
|
||||
/config/.local/bin/agy
|
||||
/config/.npm-global/bin/codex
|
||||
/bin/git
|
||||
ok: preflight run_id=run-20260812T040619Z-1ed70fffdbda status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
```
|
||||
|
||||
### C01-C09 execution — implementation-only, never repeat in review
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Expected: the CLI emits exactly one canonical execution id. Record its exit code and stdout/stderr verbatim; public status is checked separately. A retained failure is preserved and never retried.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
```text
|
||||
command: run
|
||||
exit_code: 69
|
||||
stdout:
|
||||
benchmark_exit_code=69
|
||||
stderr:
|
||||
error: benchmark state is unavailable
|
||||
cli_emitted_run_id=(none)
|
||||
```
|
||||
|
||||
Intermediate public status while the command was active:
|
||||
|
||||
```text
|
||||
ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}
|
||||
```
|
||||
|
||||
### Stored execution status — repeatable, provider-free
|
||||
|
||||
Command:
|
||||
|
||||
```bash
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error.
|
||||
|
||||
Actual stdout/stderr:
|
||||
|
||||
The planned stored-pointer command was not run because `run_id.log` was correctly absent. Read-only blocker checks were:
|
||||
|
||||
```text
|
||||
$ python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id run-20260812T040619Z-00a5e6664764
|
||||
ok: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}
|
||||
|
||||
$ protected path/process check
|
||||
run_id.log=absent
|
||||
live_execution_process=false
|
||||
```
|
||||
|
||||
`git diff --check` stdout/stderr: `(none)`
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan | Implementing agent uses it as prior-loop context and does not search archive broadly |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results | Implementing agent, then review agent | Implementation records initial output; reviewer reruns only repeatable commands and never the execution run |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: FAIL
|
||||
- Dimension Assessment:
|
||||
- Correctness: Fail — production Codex output is rejected and the public run aborts with one dead attempt retained as `running`.
|
||||
- Completeness: Fail — TEST-2 requires nine non-interrupted terminal attempts, but public status is `failed=1, running=1` after only two allocated cells.
|
||||
- Test coverage: Fail — the fixture and integration seam omit a production usage key and synthesize a caller binding event that the real Codex CLI does not emit.
|
||||
- API contract: Fail — parser and consumers disagree about both the current Codex JSONL usage schema and the documented optional caller binding observation.
|
||||
- Code quality: Pass — no unrelated debug output, TODO, or formatting defect was found in the reviewed scope.
|
||||
- Implementation deviation: Pass — the implementation preserved the failed run, did not write `run_id.log`, and did not resume, retry, or edit state.
|
||||
- Verification trust: Pass — recorded exit/status evidence matches fresh read-only status, process inspection, lifecycle replay, and retained attempt files.
|
||||
- Spec conformance: Fail — S04-S08 still lack the required C01-C09 terminal execution evidence.
|
||||
- Findings:
|
||||
- Required R1 — Current Codex usage JSONL is rejected.
|
||||
- Evidence: replaying the retained C05 `lifecycle-result.json` stdout through `CodexJSONLParser` fails on line 26 with `CodexJSONLError: invalid Codex usage observation`; the `turn.completed.usage` keys are `cache_write_input_tokens`, `cached_input_tokens`, `input_tokens`, `output_tokens`, and `reasoning_output_tokens`. `scripts/agent_benchmark/codex_iop.py:79-85,334-345` does not admit `cache_write_input_tokens`. Fresh `python3 -m unittest scripts.agent_benchmark.codex_iop_test` still reports 9 tests/OK, proving the current test set misses the production shape.
|
||||
- Root Cause: the Codex adapter's exact usage-key mapping predates the caller's `cache_write_input_tokens` field, while the tracked fixture and parser unit test still use the older subset.
|
||||
- Selected Fix: add `cache_write_input_tokens -> cache_write_tokens` to `_CODEX_USAGE_FIELDS`; update `scripts/agent_benchmark/codex_iop_test.py` and `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` with the retained production-shaped usage object and assertions that the value is preserved without reconstructing totals. Acceptance commands: `python3 -m unittest scripts.agent_benchmark.codex_iop_test` and the focused lifecycle/integration suite in the follow-up plan.
|
||||
- Required R2 — Config-owned binding admission is contradicted by caller-owned enforcement.
|
||||
- Evidence: `CodexJSONLParser.effective_binding` is explicitly optional and `test_bridge_proves_finish_then_idle_after_child_exit` asserts `None`, but `scripts/agent_benchmark/live_iop.py:992-995` rejects every live invocation unless the caller emits an exact `iop_effective_binding`. The retained real Codex stdout contains no such synthetic event. The live scoring consumer at `scripts/agent_benchmark/live_iop.py:1123-1131` applies the same impossible requirement. `scripts/agent_benchmark/connectivity_integration_test.py:217-227` currently injects the synthetic event and `:2403-2436` asserts that absence is an error, hiding the production mismatch.
|
||||
- Root Cause: preflight moved effective route/model/stage ownership to the independently validated config observation, but execution and scoring consumers retained the earlier caller-event equality gate and their seams continued fabricating that event.
|
||||
- Selected Fix: in `_LiveAdapter.invoke` and `_LiveScoringAdapter.invoke`, use the admitted config binding as the canonical result; treat an optional caller observation as an additional exact-match check only when present, and still reject any non-`None` mismatch. Remove the synthetic Codex binding from the integration fixture and add execution plus scoring assertions that absent observation succeeds from admitted config while mismatches fail. Acceptance commands: the focused `codex_iop_test` and named connectivity integration tests, followed by the full Python benchmark suite and `git diff --check`.
|
||||
- Routing Signals: review_rework_count=1 evidence_integrity_failure=false
|
||||
- Next Step: Run the mandatory plan-skill WARN/FAIL follow-up for Required R1 and R2; after deterministic fixes pass, a new scored execution remains authorization-gated because this retained run must not be retried implicitly.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=6 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=6, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained execution defect evidence.
|
||||
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 fixed with 41/429 tests passing.
|
||||
- `user_review_0.log`: continuing execution authorization resolved.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| TEST-1 deterministic regression | [x] |
|
||||
| TEST-2 protected ready=9 preflight | [x] |
|
||||
| TEST-3 new C01-C09 public run | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh.
|
||||
- [x] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`.
|
||||
- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
- [ ] Append verdict and routing signals.
|
||||
- [ ] Run repeatable verification without repeating scored execution.
|
||||
- [ ] Archive active pair to `plan_cloud_G10_6.log` and `code_review_cloud_G10_6.log`.
|
||||
- [ ] If PASS, write complete.log and archive task; if non-PASS, materialize the required next state.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
TEST-3의 public `run`은 계획대로 정확히 한 번 호출했으나 exit 69로 종료됐다. CLI가 canonical id를 출력하지 않아 `run_id.log`를 쓰지 않았고, 새 run은 read-only blocker evidence로만 식별했다. resume/retry/direct caller/provider/state edit는 수행하지 않았다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
Codex C05 success가 종결된 뒤 C03 agy의 ordinary nonzero lifecycle에서 controller exception이 발생한 사실을 보존했다. retained run은 재개하지 않고, ordinary lifecycle failure를 terminal attempt로 처리하는 source fix를 후속 plan에서 적용한다.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### TEST-1 deterministic regression
|
||||
|
||||
```text
|
||||
focused: Ran 41 tests in 15.084s — OK
|
||||
full: Ran 429 tests in 118.691s — OK
|
||||
manifest: ok: manifest is valid
|
||||
git diff --check: (none)
|
||||
```
|
||||
|
||||
### TEST-2 protected preflight
|
||||
|
||||
```text
|
||||
ok: preflight run_id=run-20260812T043640Z-91c437c5c7cd status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
```
|
||||
|
||||
### TEST-3 one public run and status
|
||||
|
||||
```text
|
||||
command: run
|
||||
exit_code: 69
|
||||
stdout: (none)
|
||||
stderr: error: benchmark state is unavailable
|
||||
cli_emitted_run_id: (none)
|
||||
identified_new_run: run-20260812T043704Z-041320764a11
|
||||
public_status: {'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 1, 'timed_out': 0}
|
||||
attempts: c02=failed, c05=success, c03=running
|
||||
```
|
||||
|
||||
C03 retained lifecycle is `success=false`, `terminal_reason=nonzero_exit`, `exit_code=1`, `cleanup_complete=true`, `process_group_alive=false`, but attempt-measurement/web-validation are absent and `attempt.json` remains `running`.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
- Implementation owns completion status, deviations, decisions, and verification output.
|
||||
- Review owns verdict, archive, complete.log, next state, and roadmap completion event.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: FAIL
|
||||
- Dimension Assessment:
|
||||
- Correctness: Fail — an ordinary agy nonzero lifecycle is converted to an adapter exception and leaves a dead attempt running.
|
||||
- Completeness: Fail — only three attempts were allocated and C01-C09 terminal evidence is incomplete.
|
||||
- Test coverage: Fail — current live integration coverage does not assert that an agy failed lifecycle is returned for terminal publication and that later slots continue.
|
||||
- API contract: Fail — execution adapter treats caller execution failure as connectivity-contract failure after preflight was already admitted.
|
||||
- Code quality: Pass — retained evidence and prior Codex fix are consistent and bounded.
|
||||
- Implementation deviation: Pass — exactly one run was called and no forbidden recovery path was used.
|
||||
- Verification trust: Pass — public status and retained lifecycle/sidecar evidence agree.
|
||||
- Spec conformance: Fail — S04-S08 still lack nine terminal attempts.
|
||||
- Findings:
|
||||
- Required R4 — agy ordinary failed lifecycle aborts the whole run instead of terminalizing the attempt.
|
||||
- Evidence: C03 `lifecycle-result.json` is a clean `nonzero_exit` with process cleanup complete; `live_iop.py` calls `parser.observed_result`, receives non-ready because lifecycle success is false, then raises `LiveIopError`. This occurs before `run_slots.invoke_bound` publishes measurement/web validation, so `attempt.json` stays `running` and the CLI returns unavailable state. C05 immediately before it is `success`, proving the earlier Codex fix works.
|
||||
- Root Cause: `_LiveAdapter.invoke` conflates post-admission execution outcome with preflight binding verification. It applies the successful-stream `observed_result` gate even when lifecycle already has a valid terminal failure.
|
||||
- Selected Fix: in the agy branch, return `_bound_observations(result, admitted)` immediately for any `result.success == false`; apply `parser.observed_result` exact binding checks only to successful lifecycle results. Add credential-free integration coverage for failed agy lifecycle terminal publication and later-slot continuation, then run full benchmark tests, fresh ready=9 preflight, and one new public run under the user's continuing authorization.
|
||||
- Routing Signals: review_rework_count=3 evidence_integrity_failure=false
|
||||
- Next Step: Run the mandatory plan-skill FAIL follow-up for R4, including one authorized new public run after the deterministic fix; preserve this run without resume/retry/edit.
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - REVIEW_TEST
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs, plan=7
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: R4 evidence and retained run.
|
||||
- `user_review_0.log`: continuing execution authorization.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| FIX-1 agy failure terminalization | [x] |
|
||||
| FIX-2 regression coverage | [x] |
|
||||
| TEST-1 deterministic verification | [x] |
|
||||
| TEST-2 ready=9 and one new run | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams.
|
||||
- [x] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots.
|
||||
- [x] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check.
|
||||
- [x] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted.
|
||||
- [x] Fill implementation evidence.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
- [x] Append PASS verdict/routing signals, run repeatable verification, archive to `_7.log`, write `complete.log`, move the task directory, and preserve milestone completion metadata.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
없음. 새 public run은 정확히 한 번 호출했으며 exit 69은 retained failed cells 때문이었다. CLI-emitted canonical id만 `run_id.log`에 기록했고 old run resume/retry/state edit 또는 direct caller/provider 호출은 없었다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- agy lifecycle 자체가 실패하면 connectivity binding을 재판정하지 않고 admitted binding에 대해 typed observation만 검증해 RunStore가 정상 terminal publication을 수행하게 했다.
|
||||
- 성공 lifecycle에만 `observed_result` exact config-binding gate를 유지해 contract mismatch는 계속 fail-closed한다.
|
||||
- 개별 failed 결과를 재시도하지 않고 D10 retained evidence로 수용했다.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Source/tests
|
||||
|
||||
```text
|
||||
focused: Ran 75 tests in 51.600s — OK
|
||||
full: Ran 430 tests in 122.446s — OK
|
||||
manifest: ok: manifest is valid
|
||||
git diff --check: (none)
|
||||
```
|
||||
|
||||
### Protected preflight/run/status
|
||||
|
||||
```text
|
||||
preflight: ok: preflight run_id=run-20260812T044743Z-40dd65f85a35 status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
run exit_code: 69
|
||||
run stderr: error: benchmark execution failed run_id=run-20260812T044800Z-412e05fc80df completed=9 unresolved=7 success=2 failed=7 timed_out=0 cancelled=0 interrupted=0 running=0
|
||||
status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0}
|
||||
attempt_count: 9
|
||||
all_terminal: True
|
||||
all_sidecars: True
|
||||
```
|
||||
|
||||
## Section Ownership
|
||||
|
||||
Implementation owns status/evidence; review owns verdict/archive/complete/next state.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Overall Verdict: PASS
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — agy failed lifecycles now terminalize and later slots continue; successful binding checks remain exact.
|
||||
- Completeness: Pass — C01-C09 all have retained terminal attempts, required measurement/web sidecars, and zero running/interrupted.
|
||||
- Test coverage: Pass — focused 75 and full 430 tests include adapter failure return and later-slot continuation.
|
||||
- API contract: Pass — post-admission execution failure is distinct from successful-stream binding validation.
|
||||
- Code quality: Pass — the source change is a bounded branch guard with focused regression coverage.
|
||||
- Implementation deviation: Pass — one fresh run, no resume/retry/substitution/state edit/direct caller/provider.
|
||||
- Verification trust: Pass — reviewer fresh tests, manifest, diff, pointer format, and public status all agree.
|
||||
- Spec conformance: Pass — S04-S08/D06/D10 accept the complete nine-cell success/failure terminal matrix with preserved failures.
|
||||
- Findings: None
|
||||
- Routing Signals: review_rework_count=3 evidence_integrity_failure=false
|
||||
- Next Step: PASS — archive the pair, write complete.log, move the task directory, and emit milestone completion metadata for runtime aggregation.
|
||||
|
||||
### Reviewer Fresh Verification
|
||||
|
||||
```text
|
||||
focused: Ran 75 tests in 45.609s — OK
|
||||
full: Ran 430 tests in 117.951s — OK
|
||||
manifest: ok: manifest is valid
|
||||
status: {'cancelled': 0, 'failed': 7, 'interrupted': 0, 'running': 0, 'success': 2, 'timed_out': 0}
|
||||
git diff --check: (none)
|
||||
```
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
8개 plan/review revision과 지속 실행 승인 1건을 거쳐 C01-C09의 fresh one-shot terminal evidence를 완성했고 최종 verdict는 PASS다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|---|---|---|---|
|
||||
| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | 초기 실행 준비 revision. |
|
||||
| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | readiness 정합화 revision. |
|
||||
| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | runtime/provider 준비 revision. |
|
||||
| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, managed dev, all-cell readiness와 pre-launch controller fix. |
|
||||
| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Codex current usage와 config-owned binding 결함 발견. |
|
||||
| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | Codex 결함 수정 및 실행 지속 승인 대기. |
|
||||
| `plan_cloud_G10_6.log` | `code_review_cloud_G10_6.log` | FAIL | Codex success 확인 후 agy failed-lifecycle terminalization 결함 발견. |
|
||||
| `plan_cloud_G10_7.log` | `code_review_cloud_G10_7.log` | PASS | R4 수정, 9개 terminal attempt 및 sidecar evidence 완성. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- Gemini-native Edge ingress, official agy transport, managed credential dev binding과 all-cell readiness를 정합화했다.
|
||||
- Codex `cache_write_input_tokens`를 canonical metric으로 수용하고 config-owned binding을 execution/scoring에서 일관되게 적용했다.
|
||||
- agy ordinary failed lifecycle을 terminal attempt로 보존해 이후 슬롯 실행이 계속되도록 수정했다.
|
||||
- canonical run `run-20260812T044800Z-412e05fc80df`에 C01-C09 9개 attempt, `success=2`, `failed=7`, `running=0`, `interrupted=0`과 모든 measurement/web-validation sidecar를 보존했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test` - PASS; reviewer fresh 75 tests/OK.
|
||||
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; reviewer fresh 430 tests/OK.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` - PASS; manifest valid.
|
||||
- public `status --run-id run-20260812T044800Z-412e05fc80df` - PASS; 9 terminal, running/interrupted 0.
|
||||
- `git diff --check` - PASS; output 없음.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- 다음 Epic의 objective validation, blind scoring, aggregation/reporting은 별도 task로 진행한다.
|
||||
|
|
@ -0,0 +1,201 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - C01-C09 원샷 비교 실행
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 아래 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다.
|
||||
|
||||
## Background
|
||||
|
||||
승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행하도록 정의한다. `run` 경로가 단일 writer 안에서 fresh all-cell preflight 뒤 모든 slot을 순회하므로 caller/model별로 분할 실행하면 동일 조건과 실패 보존 계약이 깨진다. 이 작업은 외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아니며, 준비된 harness를 한 번 실행하고 검증 evidence를 고정하는 하나의 large slice다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`다. 미해결 Required/Suggested는 기재되지 않았고 잔여 Nit은 없다.
|
||||
- 선행 작업은 source를 변경하지 않았고, clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 stale `../iop-s2/build/bin/iop-edge` 사이의 freshness blocker를 exit 69로 재현했다. Manifest validate는 `ok: manifest is valid`였다.
|
||||
- Roadmap carryover는 위 exact HEAD에서 Linux AArch64 Edge/Node artifact를 다시 빌드한 뒤 caller version/help, secret-safe environment reference, public nine-cell preflight를 포함한 gate를 다시 통과하는 것이다. 구현자는 archive 전체를 탐색하지 말고 추가 세부가 꼭 필요할 때만 위 `complete.log` 한 건을 읽는다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- Runner source: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/codex_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/workspace.py`
|
||||
- Tests/rules: `scripts/agent_benchmark/attempts_test.py`, `scripts/agent_benchmark/skill_contract_test.py`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`
|
||||
- Fixture: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- Prior evidence: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD는 `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`이며 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다.
|
||||
- 첫 줄 metadata는 `milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`를 그대로 보존한다.
|
||||
- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다 (`SDD.md:105-109`).
|
||||
- Evidence Map S04-S08은 각각 event/timing/usage/workspace evidence와 hybrid stage/terminal evidence를 요구한다 (`SDD.md:122-126`). 공통 완료 조건은 모든 cell의 terminal evidence 및 성공 결과의 gate/screenshot, 모든 attempt의 timing/usage source다 (`SDD.md:132`).
|
||||
- 따라서 checklist는 C01-C09를 한 번만 제출하고 성공/실패 evidence를 보존하도록 구성했으며, final verification은 canonical `run_id`의 9개 success와 unresolved 0을 public `status`로 확인한다. Scoring/report evidence인 S09-S12는 이 Epic 범위가 아니다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- 별도 handoff는 없었다. 위 Files Read의 runner, tests, fixture, SDD, spec, contracts 및 exact predecessor `complete.log`에서 실행 계약을 재구성했다.
|
||||
- 적용 기준은 local/dev testing smoke, 동일 fixture/checksum, repetitions 1, fresh session, isolated setup cache, fixed seed, no manual retry다.
|
||||
- 준비 단계에서 `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`은 `ok: manifest is valid`, `python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test`는 91 tests/OK, `git diff --check`는 출력 없이 통과했다. Cached test output은 허용하지 않으며 구현·review에서 fresh 실행한다.
|
||||
- Repository-native fallback evidence는 manifest validator, 91개 unit/contract test, public `run`/`status` CLI다. Provider 호출 성공을 정적 test로 대체하지 않는다.
|
||||
- 제약: secret 값과 raw private endpoint/config payload를 출력하지 않고, `../iop-s2`는 clean/read-only fixture source로 유지하며, run state는 CLI가 검증한 `agent-test/runs/bench-02/<run-id>/` 아래에만 생성한다. 한 번 시작한 `run`을 재실행하거나 `resume --retry-failed`로 실패 evidence를 대체하지 않는다.
|
||||
- Gap은 현재 준비 host의 `IOP_BENCH_*` runtime reference가 unset이고 선행 evidence의 artifact freshness blocker가 아직 해소됐다는 증거가 없다는 점이다. 이는 제품/범위 결정이 아니라 실행 환경 선행조건이며, 해소되지 않으면 `run`을 호출하지 않고 review evidence에 exact blocker와 resume condition을 남긴다.
|
||||
- Confidence는 medium이다. Repository-native 실행/검증 계약은 명확하지만 실제 provider와 current-HEAD artifact readiness는 외부 runtime이 준비된 실행 시점에만 증명된다. Maintenance mode 전환은 필요 없다.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/repo: local Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, starting/current HEAD `e86113f0ae3faecf8dfe715990e4755cdc42bedf`, 준비 시 clean.
|
||||
- Testbed: `/config/workspace/iop-s2`, branch `dev`, HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, 준비 시 clean. Source sync는 수행하지 않았고 이 exact HEAD를 고정한다.
|
||||
- Binaries: `/bin/python3` (3.12.3), `/config/.npm-global/bin/claude`, `/config/.local/bin/agy`, `/config/.npm-global/bin/codex`, `/bin/git`. Caller version/help는 public run preflight가 secret-safe하게 검사한다.
|
||||
- Artifact/config/runtime: stale blocker는 `../iop-s2/build/bin/iop-edge must be built from the current testbed HEAD`다. Exact setup 조건은 위 testbed HEAD에서 Linux AArch64 Edge/Node artifact를 runtime owner가 재빌드하고, 세 caller의 `BASE_URL`/`SECRET_ENV` reference 및 `IOP_BENCH_CONFIG_OBSERVATION_ENV` reference가 유효한 환경에서 실행하는 것이다. Config는 file path가 아니라 environment-owned JSON reference다.
|
||||
- Ports/process/external hosts: raw 값은 benchmark evidence 계약상 출력하지 않는다. Public all-cell preflight가 endpoint/catalog/config identity를 hash로 관측하고 실패 시 exit 69로 닫는다.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Source behavior change는 없다. Existing tests는 run 생성, single writer, fresh preflight, terminal state, retry 보존, skill command/report contract를 다룬다.
|
||||
- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 실제 C01-C09 run 대상이다.
|
||||
- 새 test는 추가하지 않는다. 검증된 production harness의 외부 실행이 이 slice의 변화이며, 별도 mock test는 acceptance evidence를 늘리지 못한다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
None. Rename/remove/change하는 symbol이 없다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가지므로 작은 독립 변경 기준을 충족하지 않는다.
|
||||
- 다섯 task id는 `execution_order_seed=bench-02-c01-c09-v1`, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유한다 (`agent-comparison-benchmark-iop-one-shot.json:5-8,43-177`, `attempts.py:1928-1969`). 따라서 하나의 plan으로만 실행한다.
|
||||
- 이 split subtask의 predecessor index `05`는 archived `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log` 한 건으로 만족한다. 다만 그 PASS가 보존한 current-HEAD artifact freshness 조건은 TEST-1의 pre-execution gate로 남는다.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- `scripts/**`, `agent-spec/**`, `agent-contract/**`, SDD/Milestone 문서는 변경하지 않는다. Harness/manifest/계약은 이미 확정돼 있고 이 Epic은 실제 실행만 소유한다.
|
||||
- `score`, blind evaluator, aggregate/report, Milestone task S09-S12는 후속 Epic 범위이므로 호출하거나 생성하지 않는다.
|
||||
- Caller/model/effort/preset 대체, extra repetition, manual edit, retry/resume, testbed source 변경은 immutable comparison과 failure preservation을 깨므로 제외한다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=pair`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh`.
|
||||
- Build target: closure=closed, grade=`G09`, route=`cloud`, canonical filename=`PLAN-cloud-G09.md`.
|
||||
- Review target: closure=closed, grade=`G10`, route=`cloud`, canonical filename=`CODE_REVIEW-cloud-G10.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks는 `temporal_state`, `concurrent_consistency`, `boundary_contract`, `variant_product`의 4개다.
|
||||
- Recovery signals는 `review_rework_count=0`, `evidence_integrity_failure=false`다.
|
||||
- Capability gap은 없다. 실행 환경이 미준비이면 exact external blocker로 닫는 방법과 재개 조건이 정해져 있고, 실행/검증은 repository-native CLI로 완결된다.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. 디렉터리명 `06+05_comparison_runs`의 유일한 runtime dependency는 predecessor `05`다. `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`가 이를 충족한다.
|
||||
2. 위 predecessor가 보존한 exact testbed HEAD artifact rebuild와 secret-safe environment references가 준비된 뒤 TEST-1을 시작한다. 이것은 새 task dependency가 아니라 TEST-1 자체의 fail-closed execution precondition이다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Confirm the fixed external preconditions without exposing secrets, execute the immutable C01-C09 `run` exactly once, persist its canonical run id, and verify all nine attempts reach success with unresolved 0; if the gate blocks or an attempt fails, do not retry and record the exact evidence and resume condition.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Immutable C01-C09 comparison run
|
||||
|
||||
#### Problem
|
||||
|
||||
`scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json:43-177` binds C01-C09 to five Milestone tasks, but no canonical scored run exists yet. `scripts/agent_comparison_benchmark.py:124-176` creates one run and reports unresolved slots, while `scripts/agent_benchmark/attempts.py:1928-1969` appends one fresh all-cell preflight and serializes every slot under one writer. Per-cell plans or repeated commands would break the SDD S04-S08 same-condition and failure-preservation invariants.
|
||||
|
||||
#### Solution
|
||||
|
||||
After the exact external preconditions are available, run the repository CLI once and save only the emitted canonical run id to deterministic task evidence. Let the CLI create all dynamic run/attempt artifacts; do not edit them. On exit 69 or any nonzero attempt result, retain the run id and public status, do not invoke `run` or `resume` again, and record the blocker in the review stub.
|
||||
|
||||
No before/after code snippet applies because this item changes no source: it performs one stateful CLI execution. The state transition is “no canonical C01-C09 run pointer” → “one `run_id.log` pointing to the append-only C01-C09 run and a public status proving its terminal result.”
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one CLI-emitted `run-...` id; never fabricate or replace it.
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: paste verbatim stdout/stderr and record deviations/design decisions without secrets.
|
||||
- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02/<run-id>/` artifacts or `../iop-s2`.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No test file is added because there is no source/API change. Fresh existing unit/contract tests verify the harness, and the actual one-shot run plus public status supplies the provider/caller acceptance evidence unavailable to mocks.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the external-precondition command in Final Verification step 2. Only when it passes, run the one-time execution command in Final Verification step 3 exactly once. Expected success is exit 0, `completed=9`, `unresolved=0`, `success=9`, every other terminal/nonterminal count 0, and one valid `run_id.log`; any other result is preserved blocker evidence, not a retry trigger.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Validate the immutable manifest and run fresh repository-native tests. Expected: manifest valid, 91 tests pass.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
2. Prove the fixed checkout/tool/environment assumptions without printing secret or endpoint values. Expected: Linux/AArch64, exact clean testbed branch/HEAD, all binaries found, and every boolean `True`. If the artifact has not been rebuilt from this HEAD, stop before step 3 and record the predecessor blocker and resume condition.
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
test "$(uname -s)" = Linux
|
||||
test "$(uname -m)" = aarch64
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
|
||||
for tool in python3 claude agy codex git; do command -v "$tool"; done
|
||||
python3 - <<'PY'
|
||||
import os
|
||||
|
||||
ok = True
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
base = bool(os.environ.get(f"IOP_BENCH_{caller}_BASE_URL"))
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
secret = bool(ref and os.environ.get(ref))
|
||||
print(f"{caller}: base_reference={base} secret_reference={bool(ref)} referenced_secret={secret}")
|
||||
ok = ok and base and bool(ref) and secret
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
config = bool(config_ref and os.environ.get(config_ref))
|
||||
print(f"CONFIG: reference={bool(config_ref)} referenced_value={config}")
|
||||
ok = ok and bool(config_ref) and config
|
||||
raise SystemExit(0 if ok else 69)
|
||||
PY
|
||||
```
|
||||
|
||||
3. Execute the nine cells exactly once, preserve stdout/stderr, save the emitted run id, and inspect that same run. This command is implementation-only and non-repeatable; the review agent verifies step 4 instead of rerunning it. Expected success: exit 0, `completed=9 unresolved=0 success=9 failed=0 timed_out=0 cancelled=0 interrupted=0 running=0`, followed by valid status.
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
bench_stdout="$(mktemp)"
|
||||
bench_stderr="$(mktemp)"
|
||||
trap 'rm -f "$bench_stdout" "$bench_stderr"' EXIT
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$bench_stdout" 2>"$bench_stderr"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'exit=%s\n' "$bench_exit"
|
||||
printf '%s\n' '--- stdout ---'
|
||||
cat "$bench_stdout"
|
||||
printf '%s\n' '--- stderr ---'
|
||||
cat "$bench_stderr"
|
||||
bench_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$bench_stdout" "$bench_stderr" | sed -n '1p')"
|
||||
test -n "$bench_run_id"
|
||||
printf '%s\n' "$bench_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id"
|
||||
test "$bench_exit" -eq 0
|
||||
```
|
||||
|
||||
4. Revalidate the recorded run without provider re-execution and check the worktree. Expected: one well-formed id, public status with success 9 and all other counts 0, and `git diff --check` passes. The active PLAN/review pair and `run_id.log` are intentional changes.
|
||||
|
||||
```bash
|
||||
set -eu
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,264 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=1 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - C01-C09 원샷 비교 실행
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 아래 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다.
|
||||
|
||||
## Background
|
||||
|
||||
승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행하도록 정의한다. 이 generation은 최초 계획의 두 semantic defect를 바로잡는다. scored 실행 전에 current-HEAD Edge/Node artifact와 caller capability를 모두 증명하고, 실제 제출이 9개 모두 terminal이면 성공뿐 아니라 보존된 실패·timeout·cancel도 SDD에 맞는 비교 실행 evidence로 인정한다. 외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아닌 하나의 indivisible execution slice다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 readiness 확인 절차가 exact blocker를 보존했다는 뜻이며 `route-readiness` 완료 선언이 아니다.
|
||||
- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`에서 Edge/Node artifact freshness, Linux AArch64 identity, caller version/help, environment reference를 모두 통과한 뒤에만 live benchmark를 시작하도록 요구한다.
|
||||
- 최초 active pair는 공식 verdict 없이 `plan_cloud_G09_0.log`와 `code_review_cloud_G10_0.log`로 보존했다. 그 pair의 success-only acceptance와 불완전한 artifact/caller preflight는 이 generation이 대체하며 구현자는 이전 pair를 다시 읽지 않는다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- Runner/fixture: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- Tests/rules: `scripts/agent_benchmark/attempts_test.py`, `scripts/agent_benchmark/skill_contract_test.py`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- Exact prior evidence: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`, its directly connected `plan_cloud_G09_0.log` and `code_review_cloud_G09_0.log`, and the current task's archived generation-0 pair.
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD는 `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`이며 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다.
|
||||
- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다. Evidence Map은 각 cell의 event/timing/usage/workspace evidence와 hybrid stage/terminal evidence를 요구한다.
|
||||
- D10과 state invariant는 scored failure를 보존하고 성공 결과만 골라 대표하지 않도록 요구한다. 따라서 preflight 차단이나 1~8개 partial attempt는 미완료지만, fresh run에서 C01-C09 각각 정확히 한 번 제출되어 9개 모두 `success|failed|timed_out|cancelled` terminal이면 이 execution Epic의 evidence는 완결된다. 실패가 있더라도 `resume --retry-failed`를 호출하지 않는다.
|
||||
- Scoring/report evidence인 S09-S12는 이 Epic 범위가 아니다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- 별도 handoff는 없었다. repository-native CLI, manifest, SDD, spec/contracts, focused tests와 exact predecessor evidence에서 실행 계약을 재구성했다.
|
||||
- 2026-08-12 self-review에서 manifest validate는 `ok: manifest is valid`, focused unit/contract suite는 `Ran 91 tests ... OK`, `git diff --check`는 출력 없이 통과했다. Cached output은 구현·review evidence로 재사용하지 않는다.
|
||||
- 같은 self-review의 secret-safe preflight snapshot은 local Linux/AArch64, exact clean testbed branch/HEAD, `iop-edge` build metadata가 exact HEAD/clean, `iop-node`가 다른 revision의 dirty build, agy observed `1.1.12`/adapter-known `1.1.11`/IOP transport unsupported, 세 caller와 config observation environment reference 모두 absent였다. 현재는 scored run을 시작할 수 없다.
|
||||
- `agent-test/runs/bench-02/run-20260811T210154Z-925c0f26af88/`은 기준 커밋 뒤 생성된 preparation-only preflight state다. public status는 모든 attempt count 0이고 preflight는 `implementation_gap=9`다. Append-only evidence로 보존하되 readiness나 C01-C09 실행 evidence로 재사용하지 않는다.
|
||||
- 제약: secret 값과 raw private endpoint/config payload를 출력하지 않고, `../iop-s2`는 clean/read-only fixture source로 유지하며, dynamic run state는 CLI가 검증한 `agent-test/runs/bench-02/<run-id>/` 아래에만 둔다. `run_id.log`는 그 canonical state를 가리키는 task-local pointer일 뿐 run data를 복제하지 않는다.
|
||||
- Confidence는 high다. 현재 blocker와 실행/terminal 분류가 deterministic하며 provider 호출은 fixed gate가 모두 통과한 뒤 public `run` 한 번으로만 발생한다.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/repo: local Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, source HEAD `e86113f0ae3faecf8dfe715990e4755cdc42bedf`.
|
||||
- Testbed: `/config/workspace/iop-s2`, branch `dev`, fixed HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean/read-only.
|
||||
- Required artifacts: `../iop-s2/build/bin/iop-edge`, `../iop-s2/build/dev/iop-node`; both must be executable current-HEAD Linux AArch64 ELF artifacts and execute their safe help/version probes.
|
||||
- Required callers: `claude`, `agy`, `codex`; agy must equal the production adapter's `AGY_KNOWN_VERSION` and expose its IOP transport help contract.
|
||||
- Required environment: each caller's base-url reference, secret environment-name reference and dereferenced non-empty secret, plus valid config-observation JSON reference. Commands print only booleans or fixed success text.
|
||||
- Ports/process/external hosts are observed by the public all-cell preflight inside `run`; raw values remain out of task evidence.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Source behavior change는 없다. Existing tests는 run 생성, single writer, fresh preflight, terminal state, retry 보존과 skill command contract를 다룬다.
|
||||
- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, workspace/web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 실제 C01-C09 run 대상이다.
|
||||
- 새 test는 추가하지 않는다. Fresh focused tests와 actual one-shot run/public status가 이 slice의 acceptance oracle이다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
None. Rename/remove/change하는 symbol이 없다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가진다.
|
||||
- 다섯 task id는 fixed seed, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유하므로 하나의 plan으로 실행한다.
|
||||
- 디렉터리 predecessor index `05`는 archived `05+04_readiness_preflight/complete.log`로 task-protocol상 충족한다. 그 로그가 보존한 live readiness blocker는 TEST-1의 fail-closed precondition이며 아직 해소되지 않았다.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- `scripts/**`, `agent-spec/**`, `agent-contract/**`, SDD/Milestone 문서는 변경하지 않는다. 이 Epic은 준비된 harness의 actual run만 소유한다.
|
||||
- `score`, evaluator, report, S09-S12는 후속 Epic 범위다.
|
||||
- Caller/model/effort/preset 대체, extra repetition, manual edit, `resume`, retry, testbed source/config mutation은 제외한다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh` mode=`pair`.
|
||||
- Build: `scope/context/verification/evidence/ownership/decision=true`, capability gap=none, grade scores `1/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G09`, filename=`PLAN-cloud-G09.md`, catalog route=`worker/cloud/G09`.
|
||||
- Review: `scope/context/verification/evidence/ownership/decision=true`, capability gap=none, grade scores `2/2/2/2/2`, route basis=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog route=`review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; positive loop risks=`temporal_state,concurrent_consistency,boundary_contract,variant_product` (4); `review_rework_count=0`; `evidence_integrity_failure=false`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `06+05_comparison_runs`의 runtime dependency는 predecessor `05`이며 exact archived `complete.log`가 task-protocol dependency를 충족한다.
|
||||
2. TEST-1의 fixed gate가 current-HEAD artifacts, caller capability와 environment references를 모두 통과해야 scored `run`을 시작할 수 있다. 현재 blocker가 하나라도 남으면 run을 호출하지 않고 active pair를 유지한다.
|
||||
3. Gate 통과 뒤 public `run`을 정확히 한 번 호출한다. 그 fresh run의 9개 terminal attempt를 확인한 뒤에만 TEST-1을 완료로 표시한다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Prove every fixed external precondition, invoke the immutable C01-C09 `run` exactly once, preserve its canonical run id and verbatim CLI result, and accept only either nine success results or nine retained non-interrupted terminal results; never retry, resume, substitute, or treat preflight/partial execution as completion.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Immutable C01-C09 comparison run
|
||||
|
||||
#### Problem
|
||||
|
||||
The generation-0 plan omitted the predecessor's full Edge/Node artifact and agy capability gate, so it could allocate a blocked run against stale or unsupported runtime inputs. It also required `success=9` even though SDD D10 requires retained scored failures to remain valid comparison evidence. `scripts/agent_comparison_benchmark.py` returns exit 69 for both preflight blockers and retained failures, so acceptance must distinguish zero/partial attempts from nine terminal attempts without rerunning.
|
||||
|
||||
#### Solution
|
||||
|
||||
Run the complete fixed precondition command first. Only after it exits 0, snapshot existing run roots and invoke the public CLI exactly once. Capture its stdout/stderr and exit code verbatim, resolve the new run id from the CLI output or the one exact newly created run root, and write only that id to `run_id.log`.
|
||||
|
||||
Use the public read-only `status` command on that exact run. A preflight blocker with zero attempts or any partial/running/interrupted state leaves TEST-1 unchecked and records a resume condition. Exactly nine attempts in `success|failed|timed_out|cancelled`, with zero `running` and `interrupted`, completes the execution slice; exit 0 must correspond to nine successes and exit 69 to retained non-success terminals. Do not invoke `run` or `resume` again.
|
||||
|
||||
No before/after code snippet applies because this item changes no source. The durable transition is no scored run → one append-only C01-C09 run plus one task-local canonical pointer.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one resolved `run-...` id; never fabricate or replace it.
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim command output, classification, deviations and blocker/resume evidence without secrets.
|
||||
- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02/<run-id>/` artifacts or `../iop-s2`.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No test file is added because there is no source/API change. Fresh existing unit/contract tests verify the harness; the actual one-shot run and public status supply the external acceptance evidence unavailable to mocks.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run Final Verification steps 1 and 2. Only if step 2 passes, run step 3 exactly once. Mark TEST-1 complete only when step 3 classifies `execution_complete`; otherwise keep it pending with the exact blocker and resume condition. The review agent reruns steps 1, 2 and 4 only, never step 3.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|------|------|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Validate the immutable manifest and run fresh focused tests. Expected: manifest valid; 91 tests pass.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest scripts.agent_benchmark.attempts_test scripts.agent_benchmark.skill_contract_test
|
||||
```
|
||||
|
||||
2. Prove the fixed testbed, artifact, caller and secret-reference assumptions without printing secret or endpoint values. Expected: every command succeeds. If any check fails, stop before step 3.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
|
||||
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
|
||||
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
|
||||
command -v readelf >/dev/null || blocked "readelf must be installed"
|
||||
command -v go >/dev/null || blocked "go must be installed"
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test -x "$binary" || blocked "$binary must be executable"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
|
||||
import subprocess, sys
|
||||
expected = sys.argv[1]
|
||||
for binary in sys.argv[2:]:
|
||||
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
|
||||
build = {}
|
||||
for line in output.splitlines():
|
||||
fields = line.strip().split("\t", 1)
|
||||
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
|
||||
key, value = fields[1].split("=", 1)
|
||||
build[key] = value
|
||||
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
|
||||
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
|
||||
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
|
||||
print("ok: Edge/Node build identities match the clean testbed HEAD")
|
||||
PY
|
||||
../iop-s2/build/bin/iop-edge --help >/dev/null || blocked "iop-edge help must execute"
|
||||
test -n "$(../iop-s2/build/dev/iop-node version)" || blocked "iop-node version must execute"
|
||||
for tool in python3 claude agy codex git; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
|
||||
python3 - <<'PY' || blocked "agy must match the adapter-known version and transport contract"
|
||||
import subprocess
|
||||
from scripts.agent_benchmark.agy_iop import AGY_KNOWN_VERSION, inspect_agy_iop_capability
|
||||
version_run = subprocess.run(["agy", "--version"], check=True, capture_output=True, text=True)
|
||||
help_run = subprocess.run(["agy", "--help"], check=True, capture_output=True, text=True)
|
||||
capability = inspect_agy_iop_capability((version_run.stdout + version_run.stderr).strip(), help_run.stdout + help_run.stderr)
|
||||
assert capability.version == AGY_KNOWN_VERSION
|
||||
assert capability.iop_transport_supported
|
||||
print("ok: agy IOP transport capability")
|
||||
PY
|
||||
python3 - <<'PY' || blocked "benchmark environment references must be present"
|
||||
import json, os, re
|
||||
name = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
|
||||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
assert os.environ.get(f"IOP_BENCH_{caller}_BASE_URL")
|
||||
ref = os.environ.get(f"IOP_BENCH_{caller}_SECRET_ENV", "")
|
||||
assert name.fullmatch(ref) and os.environ.get(ref)
|
||||
config_ref = os.environ.get("IOP_BENCH_CONFIG_OBSERVATION_ENV", "")
|
||||
assert name.fullmatch(config_ref) and os.environ.get(config_ref)
|
||||
value = json.loads(os.environ[config_ref])
|
||||
assert value.get("schema_version") == "1" and isinstance(value.get("routes"), list)
|
||||
print("ok: benchmark environment references")
|
||||
PY
|
||||
```
|
||||
|
||||
3. Execute the nine cells exactly once. This block is implementation-only and non-repeatable. Expected task completion is `classification=execution_complete`; the recorded benchmark exit remains 0 for all-success or 69 for retained failures.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
run_root=agent-test/runs/bench-02
|
||||
before="$(mktemp)"; after="$(mktemp)"; new_runs="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
|
||||
trap 'rm -f "$before" "$after" "$new_runs" "$out" "$err"' EXIT
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
|
||||
printf '%s\n' 'stderr:'; cat "$err"
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' | sort >"$after"
|
||||
comm -13 "$before" "$after" >"$new_runs"
|
||||
test "$(wc -l <"$new_runs")" -eq 1
|
||||
discovered_run_id="$(cat "$new_runs")"
|
||||
emitted_run_id="$(sed -nE 's/.*run_id=(run-[^ ]+).*/\1/p' "$out" "$err" | sed -n '1p')"
|
||||
if test -n "$emitted_run_id"; then test "$emitted_run_id" = "$discovered_run_id"; fi
|
||||
printf '%s\n' "$discovered_run_id" > agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$discovered_run_id")"
|
||||
printf 'status_stdout:\n%s\n' "$status_output"
|
||||
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
|
||||
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
|
||||
bench_exit = int(os.environ["BENCH_EXIT"])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
|
||||
assert bench_exit == (0 if states["success"] == 9 else 69)
|
||||
print(f"classification=execution_complete states={states}")
|
||||
raise SystemExit(0)
|
||||
if sum(states.values()) == 0 and bench_exit == 69:
|
||||
print(f"classification=blocked_preflight states={states}")
|
||||
else:
|
||||
print(f"classification=blocked_partial states={states}")
|
||||
raise SystemExit(69)
|
||||
PY
|
||||
```
|
||||
|
||||
4. Revalidate only the recorded run and worktree; do not invoke a provider. Expected after completed execution: one valid id, public status with nine accepted terminals and zero interrupted/running, `git diff --check` success.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
bench_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
|
||||
printf '%s\n' "$status_output"
|
||||
STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
|
||||
print(f"ok: nine retained terminal attempts states={states}")
|
||||
PY
|
||||
git diff --check
|
||||
```
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,248 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=2 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - C01-C09 원샷 비교 실행
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 구현의 필수 마지막 단계다. 검증 명령을 정확히 실행하고 실제 메모와 stdout/stderr를 기록한 뒤 active 파일을 그대로 두고 review 준비 상태를 보고한다. 구현이 막히면 exact blocker, 시도한 명령/출력, 재개 조건만 구현 담당 evidence 필드에 기록한다. 사용자에게 질문하거나 user-input 도구·control-plane stop 파일을 사용하거나 다음 상태를 분류하지 않으며, 로그 archive·`complete.log` 작성·task 디렉터리 이동은 code-review 담당에게 남긴다.
|
||||
|
||||
## Background
|
||||
|
||||
승인된 immutable manifest는 다섯 Milestone task에 해당하는 C01-C09를 한 run identity와 고정 seed 아래 실행한다. checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`의 generation 0 의도인 “fresh preflight 뒤 public run 한 번, retry/resume 없음, 한 run의 아홉 cell 보존”을 유지한다. Generation 1의 caller 직접 호출·preflight 정책 중복과 global run-directory 차집합 의존을 제거한다. Caller capability, environment, live route readiness는 public `run`의 fresh all-cell preflight만 판정하며, CLI가 방출한 run id만 canonical pointer로 채택한다.
|
||||
|
||||
외부 caller/provider와 append-only run state를 실제로 변경하므로 direct-small이 아닌 하나의 indivisible execution slice다. Scored failure도 SDD D10에 따라 보존하므로 아홉 cell이 모두 non-interrupted terminal이면 이 Epic의 실행 evidence는 완결된다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- 선행 task `m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight`의 exact `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 `PASS`지만, 이는 blocker를 보존한 readiness 절차의 완료이지 Milestone `route-readiness` 완료 선언이 아니다.
|
||||
- 선행 evidence는 clean `../iop-s2` `dev` HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`와 current-HEAD Edge/Node artifact를 요구한다. 이 dependency는 TEST-1에서 fail-closed로 다시 확인한다.
|
||||
- Generation 0은 `plan_cloud_G09_0.log` / `code_review_cloud_G10_0.log`, generation 1은 `plan_cloud_G09_1.log` / `code_review_cloud_G10_1.log`로 공식 verdict 없이 보존됐다. Generation 0의 atomic intent는 유지하고 success-only acceptance를 바로잡았으며, generation 1의 직접 caller probe와 concurrent run 오인 가능성을 이 generation이 대체한다. 구현자는 이 plan과 active review만 실행하며 archive를 다시 읽지 않는다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- Spec/contracts: `agent-spec/index.md`, `agent-spec/testing/agent-comparison-benchmark.md`, `agent-contract/index.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- Runner/fixture: `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/manifest.py`, `scripts/agent_benchmark/workspace.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/agy_iop.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/fixtures/agent-comparison-benchmark/prompt.md`, `scripts/fixtures/agent-comparison-benchmark/reference.txt`, `scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg`, `scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg`
|
||||
- Rules/skills: `agent-ops/rules/project/domain/testing/rules.md`, `agent-test/local/rules.md`, `agent-test/local/testing-smoke.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- Exact prior evidence: predecessor `complete.log`와 직접 연결된 plan/review log, 그리고 current task의 generation 0/1 pair
|
||||
|
||||
### SDD Criteria and Task Union
|
||||
|
||||
- SDD 상태는 `[승인됨]`, 잠금은 `해제`, 추가 사용자 결정은 없다.
|
||||
- 이 pair의 Milestone task union은 정확히 `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다. S04는 C01, S05는 C02-C03, S06은 C04-C05, S07은 C06-C07, S08은 C08-C09에 대응한다. 다른 Milestone task는 소유하지 않는다.
|
||||
- 각 cell은 event/timing/usage/workspace evidence를 보존하고 hybrid cell은 stage/terminal evidence도 보존해야 한다.
|
||||
- D10과 state invariant는 scored failure를 보존하고 성공 결과만 선별하지 않도록 요구한다. 따라서 preflight 차단이나 1~8개 partial attempt는 미완료지만, fresh run에서 C01-C09가 각각 정확히 한 번 제출되어 9개 모두 `success|failed|timed_out|cancelled`이면 실행 Epic evidence는 완결된다. `resume --retry-failed`는 호출하지 않는다.
|
||||
- S09-S12의 score/evaluator/report evidence는 이 Epic 범위가 아니다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- 별도 handoff는 없었다. repository-native CLI, manifest, SDD, living spec/contracts, current source와 exact predecessor evidence에서 실행 계약을 재구성했다.
|
||||
- 현재 repository HEAD는 checkpoint `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`다. 제품 source는 직전 `e86113f0ae3faecf8dfe715990e4755cdc42bedf` 이후 변경되지 않았고 checkpoint에는 task pair만 추가됐다.
|
||||
- 2026-08-12 read-only audit에서 `../iop-s2`는 clean `dev`의 fixed HEAD이고 `iop-edge` metadata는 그 exact HEAD/clean과 일치하지만, `iop-node` metadata는 revision `9b2f...`, `vcs.modified=true`로 stale다. 현재는 step 2가 fail-closed해야 하며 scored `run`을 호출하면 안 된다.
|
||||
- Generic dev runner 경로보다 SDD/immutable manifest의 task-specific local sibling `../iop-s2`가 이 benchmark의 고정 testbed 계약을 우선한다. Testbed source/config는 read-only다.
|
||||
- Caller/provider를 public CLI 밖에서 실행하지 않는다. `command -v`는 설치 여부만 확인하고 caller version/help, secret dereference, endpoint/process/config observation과 live connectivity는 public `run`의 fresh all-cell preflight가 단독 소유한다.
|
||||
- Secret 값과 raw private endpoint/config payload를 출력하지 않는다. Dynamic state는 CLI가 검증한 `agent-test/runs/bench-02/<emitted-run-id>/` 아래에만 쓰고, `run_id.log`는 그 exact state를 가리키는 task-local pointer다.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Source behavior change는 없다. Repository full benchmark suite는 manifest, run allocation, single writer, fresh preflight, terminal state, retry 보존, caller registry와 skill command contract를 검증한다.
|
||||
- 실제 Claude/agy/Codex → dev IOP → provider route, model/stage effort, timing/usage, workspace/web evidence는 unit test로 증명할 수 없다. 이 gap이 TEST-1의 C01-C09 run 대상이다.
|
||||
- 새 test는 추가하지 않는다. Fresh full benchmark suite와 actual one-shot run/public status가 acceptance oracle이다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
None. Rename/remove/change하는 symbol이 없다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- Direct-small slice는 0개다. 각 cell은 provider 비용/credential, 외부 caller invocation, append-only attempt state라는 side effect를 가진다.
|
||||
- 다섯 task id는 fixed seed, repetitions 1, 동일 run identity, fresh all-cell preflight와 single writer라는 indivisible invariant를 공유하므로 하나의 plan으로 실행한다.
|
||||
- 디렉터리 predecessor index `05`는 archived `05+04_readiness_preflight/complete.log`로 task-protocol상 충족한다. 보존된 readiness blocker는 TEST-1의 fail-closed precondition이며 아직 해소되지 않았다.
|
||||
|
||||
### Scope and Write Ownership
|
||||
|
||||
- Source/spec/contract/SDD/Milestone는 변경하지 않는다. Caller/model/effort/preset 대체, extra repetition, manual run-state edit, `resume`, retry, score, evaluator, report, testbed rebuild/source/config mutation은 제외한다.
|
||||
- Implementer-owned write: task-local `run_id.log`와 active `CODE_REVIEW-cloud-G10.md`의 implementation-owned evidence/checklist sections.
|
||||
- CLI-owned dynamic write: public `run`이 방출한 exact `agent-test/runs/bench-02/<run-id>/**`. Implementer는 이 tree를 직접 편집하지 않는다.
|
||||
- Review-owned write/archive는 review verdict 이후 절차이며 이 implementation slice가 실행하지 않는다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`agent-ops/skills/common/finalize-task-routing/scripts/finalize-task-policy.sh`, mode=`pair`, status=`routed`.
|
||||
- Build closures는 모두 true, capability gap은 none, grade scores=`1/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G09`, filename=`PLAN-cloud-G09.md`, catalog route=`worker/cloud/G09`.
|
||||
- Review closures는 모두 true, capability gap은 none, grade scores=`2/2/2/2/2`, route basis=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog route=`review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; matched risks=`temporal_state,concurrent_consistency,boundary_contract,variant_product` (4); `review_rework_count=0`; `evidence_integrity_failure=false`. Grade boundary가 build cloud route를 결정한다.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Active index `06+05`의 predecessor는 archived index `05+04`이고 exact `complete.log`가 task-protocol dependency를 충족한다.
|
||||
2. Step 2가 host/testbed/artifact identity와 tool presence를 모두 통과해야 step 3을 시작할 수 있다. 현재 stale `iop-node`가 남아 있으면 stop하고 TEST-1을 unchecked로 유지한다.
|
||||
3. Step 3의 public `run`은 caller/environment/live readiness를 fresh all-cell preflight로 판정하고, gate 통과 시 C01-C09를 동일 run에서 각각 한 번 제출한다. 이 명령은 이 plan에서 정확히 한 번만 실행한다.
|
||||
4. CLI가 한 canonical run id를 방출하고 exact run root가 새로 생긴 경우에만 pointer를 쓴다. Id가 없거나 ambiguous하면 global directory 차집합으로 추측하지 않고 `blocked_unidentified`로 중단하며 이 plan에서 rerun하지 않는다.
|
||||
5. Public status가 아홉 non-interrupted terminal attempt를 증명한 뒤에만 TEST-1을 완료로 표시한다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Pass the fixed read-only artifact gate, invoke the immutable public C01-C09 `run` exactly once, preserve only its CLI-emitted canonical run id and verbatim result, and accept only nine retained non-interrupted terminal results; never directly invoke callers/providers, retry, resume, substitute, or treat preflight/partial execution as completion.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Immutable C01-C09 comparison run
|
||||
|
||||
#### Problem
|
||||
|
||||
Generation 1 directly invoked `agy --version/--help`, dereferenced benchmark environment references, and duplicated policy already owned by the public all-cell preflight. It also inferred identity from “exactly one new global run directory,” which can reject the correct non-repeatable run when another actor concurrently creates an unrelated run. These defects violate the benchmark skill boundary and weaken exactly-once evidence.
|
||||
|
||||
#### Solution
|
||||
|
||||
Use a pre-run gate only for task-specific host/testbed/artifact identity and command presence. Let the public `run` exclusively invoke callers and assess capability, secrets, endpoints, processes, config observation and connectivity. Invoke it once, preserve both streams and exit code, and extract exactly one unique `run-...` id only from CLI output. Confirm that id was absent before the command and its exact directory now exists; unrelated concurrent run roots are ignored.
|
||||
|
||||
Write `run_id.log` only after those checks. Use public read-only `status` on that exact id. Zero attempts, partial attempts, `running`, `interrupted`, invalid state, or an unidentified run leave TEST-1 unchecked with an exact resume condition. Exactly nine `success|failed|timed_out|cancelled` attempts with zero `running`/`interrupted` complete this execution slice. Exit 0 must mean nine successes; retained non-success terminal results require exit 69. Never invoke `run` or `resume` again under this plan.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: write exactly one CLI-emitted canonical `run-...` id; never infer, fabricate, or replace it.
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim command output, classification, deviations and blocker/resume evidence without secrets.
|
||||
- [ ] Do not manually modify CLI-owned `agent-test/runs/bench-02/<run-id>/**` or anything under `../iop-s2`.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No test file is added because there is no source/API change. Fresh repository benchmark tests verify the harness. The actual one-shot run and public status supply the external acceptance evidence unavailable to mocks.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run Final Verification steps 1 and 2. Only if step 2 passes, run step 3 exactly once. Mark TEST-1 complete only when step 3 prints `classification=execution_complete`; otherwise retain unchecked items and exact blocker/resume evidence. The review agent reruns steps 1, 2 and 4 only, never step 3.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Owner | Item |
|
||||
|------|-------|------|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | Implementer | TEST-1 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | Implementer evidence fields | TEST-1 |
|
||||
| `agent-test/runs/bench-02/<CLI-emitted-run-id>/**` | Public benchmark CLI | TEST-1 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Validate the immutable manifest and run the full repository benchmark suite. Expected: manifest valid and all discovered benchmark tests pass.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
```
|
||||
|
||||
2. Prove only the fixed host/testbed/artifact and command-presence assumptions. This command does not invoke a caller/provider or reproduce public live-preflight policy. Expected: every check succeeds. If any check fails, stop before step 3.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
blocked() { printf 'blocked: %s\n' "$1" >&2; exit 69; }
|
||||
test "$(uname -s)" = Linux || blocked "benchmark host must be Linux"
|
||||
test "$(uname -m)" = aarch64 || blocked "benchmark host must be AArch64"
|
||||
test "$(git -C ../iop-s2 branch --show-current)" = dev || blocked "../iop-s2 must be on branch dev"
|
||||
test "$(git -C ../iop-s2 rev-parse HEAD)" = 1f2f7f1066fcf165a9e469bae77203b569b6f772 || blocked "../iop-s2 HEAD changed"
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)" || blocked "../iop-s2 must be clean"
|
||||
for tool in python3 git readelf go claude agy codex; do command -v "$tool" >/dev/null || blocked "$tool must be installed"; done
|
||||
testbed_head="$(git -C ../iop-s2 rev-parse HEAD)"
|
||||
for binary in ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node; do
|
||||
test -x "$binary" || blocked "$binary must be executable"
|
||||
readelf -h "$binary" | rg 'Machine:\s+AArch64' >/dev/null || blocked "$binary must be a Linux AArch64 ELF artifact"
|
||||
done
|
||||
python3 - "$testbed_head" ../iop-s2/build/bin/iop-edge ../iop-s2/build/dev/iop-node <<'PY' || blocked "Edge/Node build identity must match the clean testbed HEAD"
|
||||
import subprocess, sys
|
||||
expected = sys.argv[1]
|
||||
for binary in sys.argv[2:]:
|
||||
output = subprocess.run(["go", "version", "-m", binary], check=True, capture_output=True, text=True).stdout
|
||||
build = {}
|
||||
for line in output.splitlines():
|
||||
fields = line.strip().split("\t", 1)
|
||||
if len(fields) == 2 and fields[0] == "build" and "=" in fields[1]:
|
||||
key, value = fields[1].split("=", 1)
|
||||
build[key] = value
|
||||
assert build.get("vcs.revision") == expected, (binary, build.get("vcs.revision"))
|
||||
assert build.get("vcs.modified") == "false", (binary, build.get("vcs.modified"))
|
||||
assert build.get("GOOS") == "linux" and build.get("GOARCH") == "arm64", (binary, build.get("GOOS"), build.get("GOARCH"))
|
||||
print("ok: Edge/Node build identities match the clean testbed HEAD")
|
||||
PY
|
||||
```
|
||||
|
||||
3. Execute the nine cells exactly once. This block is implementation-only and non-repeatable. Expected task completion is `classification=execution_complete`; the benchmark exit remains 0 for all-success or 69 for retained failures.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
task_dir=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
|
||||
run_root=agent-test/runs/bench-02
|
||||
test ! -e "$task_dir/run_id.log"
|
||||
before="$(mktemp)"; out="$(mktemp)"; err="$(mktemp)"
|
||||
trap 'rm -f "$before" "$out" "$err"' EXIT
|
||||
find "$run_root" -mindepth 1 -maxdepth 1 -type d -name 'run-*' -printf '%f\n' 2>/dev/null | sort >"$before"
|
||||
set +e
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json >"$out" 2>"$err"
|
||||
bench_exit=$?
|
||||
set -e
|
||||
printf 'command: run\nexit_code: %s\nstdout:\n' "$bench_exit"; cat "$out"
|
||||
printf '%s\n' 'stderr:'; cat "$err"
|
||||
emitted_ids="$(sed -nE 's/.*run_id=(run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}).*/\1/p' "$out" "$err" | sort -u)"
|
||||
emitted_count="$(printf '%s\n' "$emitted_ids" | sed '/^$/d' | wc -l)"
|
||||
if test "$emitted_count" -ne 1; then
|
||||
printf 'classification=blocked_unidentified emitted_run_ids=%s\n' "$emitted_count"
|
||||
exit 69
|
||||
fi
|
||||
bench_run_id="$(printf '%s\n' "$emitted_ids")"
|
||||
if grep -Fxq "$bench_run_id" "$before"; then
|
||||
printf 'classification=blocked_preexisting_run_id run_id=%s\n' "$bench_run_id"
|
||||
exit 69
|
||||
fi
|
||||
test -d "$run_root/$bench_run_id"
|
||||
printf '%s\n' "$bench_run_id" >"$task_dir/run_id.log"
|
||||
set +e
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id" 2>&1)"
|
||||
status_exit=$?
|
||||
set -e
|
||||
printf 'status_exit_code: %s\nstatus_output:\n%s\n' "$status_exit" "$status_output"
|
||||
if test "$status_exit" -ne 0; then
|
||||
printf 'classification=blocked_invalid_status run_id=%s\n' "$bench_run_id"
|
||||
exit 69
|
||||
fi
|
||||
BENCH_EXIT="$bench_exit" STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
expected = {"success", "failed", "timed_out", "cancelled", "interrupted", "running"}
|
||||
assert set(states) == expected and all(isinstance(value, int) and value >= 0 for value in states.values())
|
||||
bench_exit = int(os.environ["BENCH_EXIT"])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
if accepted == 9 and states["interrupted"] == 0 and states["running"] == 0:
|
||||
assert bench_exit == (0 if states["success"] == 9 else 69)
|
||||
print(f"classification=execution_complete states={states}")
|
||||
raise SystemExit(0)
|
||||
if sum(states.values()) == 0 and bench_exit == 69:
|
||||
print(f"classification=blocked_preflight states={states}")
|
||||
else:
|
||||
print(f"classification=blocked_partial states={states}")
|
||||
raise SystemExit(69)
|
||||
PY
|
||||
```
|
||||
|
||||
4. Re-read the recorded run without invoking a provider and check both worktrees. Expected: valid pointer, nine retained terminal attempts, zero interrupted/running, clean testbed, no whitespace errors.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
pointer=agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log
|
||||
test "$(wc -l <"$pointer")" -eq 1
|
||||
bench_run_id="$(tr -d '\n' <"$pointer")"
|
||||
printf '%s\n' "$bench_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
status_output="$(python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$bench_run_id")"
|
||||
printf '%s\n' "$status_output"
|
||||
STATUS_OUTPUT="$status_output" python3 - <<'PY'
|
||||
import ast, os
|
||||
text = os.environ["STATUS_OUTPUT"]
|
||||
assert text.startswith("ok: "), text
|
||||
states = ast.literal_eval(text[4:])
|
||||
accepted = sum(states[key] for key in ("success", "failed", "timed_out", "cancelled"))
|
||||
assert accepted == 9 and states["interrupted"] == 0 and states["running"] == 0, states
|
||||
print(f"ok: nine retained terminal attempts states={states}")
|
||||
PY
|
||||
test -z "$(git -C ../iop-s2 status --porcelain=v1)"
|
||||
git diff --check
|
||||
```
|
||||
|
|
@ -0,0 +1,294 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=3 tag=API milestone-task=agy-iop-compatibility,managed-credential-dev,route-readiness,claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - official agy IOP 호환, managed dev 배포와 C01-C09 실행
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 설계와 검증을 그대로 구현하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행한 명령/출력과 재개 조건만 evidence 필드에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다.
|
||||
|
||||
## Background
|
||||
|
||||
기존 active plan은 이미 준비됐다고 가정하고 C01-C09만 실행하도록 했으나, official `agy 1.1.12`의 실제 Gemini API-key transport와 dev managed credential runtime이 그 전제와 맞지 않았다. 실제 CLI는 Gemini-native `streamGenerateContent`, `x-goog-api-key`, `GEMINI_API_KEY`, `GOOGLE_GEMINI_BASE_URL`과 `event=init|step_update|result` stream JSON을 사용하며 API-key mode에서 `--effort`를 거부한다. Edge에는 이 ingress가 없고 dev는 legacy credential mode라 marked hybrid preset admission도 불가능하다. 이 plan은 caller→Edge→preset/provider→official lifecycle 전체가 한 번의 실제 preflight에서 닫힌 뒤에만 immutable 9-cell run을 시작한다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G09_0.log`/`code_review_cloud_G10_0.log`, `plan_cloud_G09_1.log`/`code_review_cloud_G10_1.log`, `plan_cloud_G09_2.log`/`code_review_cloud_G10_2.log`는 공식 verdict 없이 종료된 이전 실행 계획이다. 그중 plan=2의 one-run 원칙은 API-4에 유지하지만 “source/contract/dev 변경 없음” 전제는 폐기한다.
|
||||
- predecessor `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/complete.log`는 index `05` 의존성을 충족하지만 live blocker를 완료로 바꾸지 않는다.
|
||||
- remote dev checkout은 `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이며 runtime config는 `build/dev-runtime/edge.yaml`, `control-plane.yaml`, `node-*.yaml`이다. public Edge는 현재 legacy HTTP `:18083`이다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Rules: `agent-ops/rules/project/domain/control-plane/rules.md`, `agent-ops/rules/project/domain/edge/rules.md`, `agent-ops/rules/project/domain/node/rules.md`, `agent-ops/rules/project/domain/platform-common/rules.md`, `agent-test/local/rules.md`, `agent-test/dev/rules.md`
|
||||
- Roadmap/SDD: `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- Contracts/docs: `agent-contract/index.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`, `agent-contract/inner/edge-config-runtime-refresh.md`, `docs/edge-local-dev-guide.md`
|
||||
- Edge: `apps/edge/internal/openai/routes.go`, `principal.go`, `server.go`, `route_resolution.go`, `dispatch_context.go`, `chat_handler.go`, `provider_tunnel.go`, `anthropic_bridge.go`, `single_request_preset_binding.go`, `apps/edge/internal/service/provider_tunnel.go`
|
||||
- Config/Node/Control Plane: `packages/go/config/protocol_profile.go`, `edge_types.go`, `node_types.go`, `validate.go`, `credential_plane_config_test.go`, `apps/node/internal/node/tunnel_handler.go`, `apps/control-plane/cmd/control-plane/main.go`, `credential_commands.go`, `credential_http_handlers.go`, `secure_delivery_integration_test.go`, `configs/control-plane.yaml`, `scripts/e2e-credential-slot-smoke.sh`
|
||||
- Benchmark: `scripts/agent_benchmark/agy_iop.py`, `agy_iop_test.py`, `live_iop.py`, `manifest.py`, `lifecycle.py`, `scripts/agent_comparison_benchmark.py`, `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- Active/prior evidence: current task plan=0/1/2 logs and active pair, exact predecessor `complete.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD는 `[승인됨]`, 잠금 `해제`다. 이 pair의 first-line task ids는 `agy-iop-compatibility`, `managed-credential-dev`, `route-readiness`, `claude-standalone`, `gemini-standalone`, `gpt-standalone`, `gemini-hybrid`, `gpt-hybrid`다.
|
||||
- S13은 official agy 1.1.12의 route별 Gemini base URL, IOP principal auth, Gemini request/tool/SSE와 실제 stream-json lifecycle을 요구하므로 API-1/API-2를 정한다.
|
||||
- S14는 CA-signed mTLS, Edge HTTPS, projection, slot-route와 sealed lease/no-fallback evidence를 요구하므로 API-3을 정한다.
|
||||
- S02는 모든 caller의 auth/route/effort/terminal live preflight를 요구하고, S04-S08은 C01-C09 one-submission evidence를 요구하므로 API-3 통과 후에만 API-4를 실행한다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- handoff는 없으며 repository source, contracts, official CLI 실행과 remote read-only preflight를 직접 확인했다.
|
||||
- official `agy 1.1.12` known model은 `Gemini 3.6 Flash`다. API-key call은 `POST /v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse`, `x-goog-api-key`와 `contents/generationConfig/systemInstruction/toolConfig/tools`를 보냈다. `GOOGLE_GEMINI_BASE_URL` invalid endpoint가 network failure를 만들었고 `GEMINI_BASE_URL`은 override가 아니었다.
|
||||
- official stream JSON은 `event=init`, `event=step_update`와 `event=result`; result는 `status`, `duration_seconds`, `num_turns`, `usage(input_tokens/output_tokens/thinking_tokens/cache_read_tokens/total_tokens)`를 가진다. 합성 `iop effective_binding`, metric, `system idle` event는 없다.
|
||||
- remote dev는 clean release checkout과 `build/dev-runtime` config/artifacts를 갖지만 `credential_plane`/TLS material이 없고 legacy static provider credential이 남아 있다. 모든 raw provider token은 workspace `token/.gemini`, `.claude`, `.gpt`에서만 읽어 credential HTTPS stdin으로 등록하며 출력·명령 인자·tracked file에 넣지 않는다.
|
||||
- external target: benchmark host Linux/AArch64의 current checkout, Mac dev Edge/Control Plane/local Node, inventory의 GX10/OneXPlayer/RTX5090 Nodes. Source sync→fresh build→config check→bounded restart→health/identity→direct/hybrid smoke 순으로 진행한다.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Gemini-native ingress/auth/body conversion/SSE projection은 기존 test가 없다. normal text, tool, usage, invalid path/body, conflicting auth, route denial, stream error/cancel regression을 새 Go test로 작성한다.
|
||||
- agy adapter test는 fake 1.1.11 help와 합성 events라 실제 1.1.12를 가리지 못한다. official help/event fixtures와 exact invocation env/argv tests로 교체하고 live registry integration test를 갱신한다.
|
||||
- managed credential code 자체는 deterministic full-cycle smoke가 있다. gap은 실제 dev security material, slot/route와 multi-node rollout이므로 config check와 live secret-safe smoke가 필요하다.
|
||||
- C01-C09 결과는 unit test로 대체할 수 없다. API-4의 public run/status가 유일한 scored evidence다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
삭제·rename symbol은 없다. 새 Gemini handler/bridge symbol은 `routes.go` 한 곳에서 등록되고, benchmark 상수 변경 call site는 `agy_iop.py`, `agy_iop_test.py`, `live_iop.py`와 connectivity integration tests다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- API translator, official event parser, managed route authorization과 actual caller smoke는 각각 독립 test가 있지만, 이 task의 correctness 조건은 `GEMINI_API_KEY`가 upstream key가 아닌 projected IOP token이고 route-specific base가 direct/hybrid binding으로 연결된다는 하나의 cross-boundary invariant다.
|
||||
- API-4는 그 invariant가 live에서 닫힌 후 exactly once여야 하므로 별도 활성 sibling으로 이동하면 기존 fixed index와 non-repeatable state coordination을 분리할 수 없다. 하나의 순차 plan으로 유지한다.
|
||||
- index `05` predecessor는 archived `complete.log`로 충족됐다.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Gemini non-stream/batch/files/cache/tuning API, arbitrary third-party Gemini SDK, unofficial agy custom model, new protocol-profile driver, pricing과 report/scoring task는 제외한다.
|
||||
- 기존 OpenAI/Anthropic caller schema, preset stage semantics와 provider OpenAI-compatible upstream은 변경하지 않는다. Gemini ingress는 이를 재사용하는 outer translation만 소유한다.
|
||||
- remote dev config와 operator secret material은 repository 밖 external state다. raw token/cert private key/slot alias/lease id는 plan/review/run evidence에 쓰지 않는다.
|
||||
- scored run 전 preflight가 실패하면 API-4는 실행하지 않는다. 한 번 시작한 public run은 retry/resume하지 않는다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
|
||||
- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`.
|
||||
- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. API-1 Go tests와 API-2 Python tests/full local suite가 통과해야 API-3 remote build/deploy를 시작한다.
|
||||
2. API-3는 current source의 fresh artifacts, managed config validation, all runtime health, official agy direct/hybrid smoke와 all-cell public preflight가 모두 통과해야 완료다.
|
||||
3. API-3 완료 전 API-4 public `run`을 호출하지 않는다. API-4는 exactly once이며 CLI-emitted run id만 보존한다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] API-1: Implement the route-qualified Gemini `streamGenerateContent` ingress, `x-goog-api-key` principal authentication, bounded Gemini↔Chat/tool/SSE translation, caller-safe errors and full Edge regression tests without bypassing existing route/preset admission.
|
||||
- [x] API-2: Replace the invented agy 1.1.11/OpenAI env/event contract with the official agy 1.1.12 Gemini API-key invocation and stream-json parser, config-owned effective binding evidence, exact redaction and updated unit/integration fixtures.
|
||||
- [x] API-3: Build and deploy fresh artifacts to dev, provision the complete operator-owned managed credential/TLS composition and projected provider routes, then pass deterministic credential qualification plus official agy direct/hybrid and all-caller live preflight without secret leakage or legacy fallback.
|
||||
- [ ] API-4: Invoke the immutable public C01-C09 run exactly once only after API-3, preserve its CLI-emitted canonical run id and all nine non-interrupted terminal attempts, and never retry, resume, substitute, or manually edit run state.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [API-1] Gemini-native Edge ingress
|
||||
|
||||
#### Problem
|
||||
|
||||
`apps/edge/internal/openai/routes.go:11-19` registers only OpenAI, Anthropic and Ollama surfaces. `principal.go:164-183` understands bearer and Anthropic `X-Api-Key` but not official agy `x-goog-api-key`. Sending agy's native body into `chat_handler.go:55-190` is impossible without a schema and streaming translation, so current official calls either bypass IOP or fail before route/preset admission.
|
||||
|
||||
#### Solution
|
||||
|
||||
Register `/gemini/` under the shared auth wrapper. Parse only `/gemini/{route-id}/v1beta/models/{caller-model}:streamGenerateContent?alt=sse`, freeze the bounded body, validate the contract fields, and translate to a synthetic internal streaming Chat request whose `model` is `{route-id}`. Call the existing Chat handler directly with the already-authenticated context and a streaming response writer that maps Chat SSE text/reasoning/tool/usage/terminal/error frames to Gemini SSE. Accept `x-goog-api-key` as a surface-specific IOP token, require equality with bearer if both exist, and never forward it as provider auth.
|
||||
|
||||
Before:
|
||||
|
||||
```go
|
||||
// apps/edge/internal/openai/routes.go:11-19
|
||||
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
|
||||
s.registerAnthropicRoutes(mux)
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
|
||||
mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent))
|
||||
s.registerAnthropicRoutes(mux)
|
||||
```
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [x] `apps/edge/internal/openai/routes.go`: register Gemini route and Gemini-native auth/error selection.
|
||||
- [x] `apps/edge/internal/openai/principal.go`: add constant-time `x-goog-api-key` principal extraction/conflict checks.
|
||||
- [x] `apps/edge/internal/openai/gemini_types.go`: define bounded request/response/error DTOs for the contracted subset.
|
||||
- [x] `apps/edge/internal/openai/gemini_handler.go`: validate path/query/body, construct internal Chat request and preserve cancellation.
|
||||
- [x] `apps/edge/internal/openai/gemini_bridge.go`: incrementally translate Chat SSE to Gemini SSE with bounded tool assembly and usage mapping.
|
||||
- [x] `apps/edge/internal/openai/gemini_handler_test.go`: cover normal, boundary, auth, route, tool, usage, error and cancellation behavior.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Write table-driven tests using existing mock run/provider/preset services. Assert zero dispatch for malformed path/body/conflicting key, direct and preset route identity, no inbound key in provider headers/log body, streaming text/tool/usage shape, one terminal and cancellation propagation.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run `gofmt -w` on changed Go files, then `go test -count=1 ./apps/edge/internal/openai`. Expected: all package tests pass fresh.
|
||||
|
||||
### [API-2] official agy 1.1.12 benchmark adapter
|
||||
|
||||
#### Problem
|
||||
|
||||
`scripts/agent_benchmark/agy_iop.py:48-66` hard-codes 1.1.11, nonexistent `AGY_*` OpenAI transport and synthetic duration events. `build_agy_invocation` adds unsupported `--effort`, while `AgyEventParser` waits for events official agy never emits. `live_iop.py:881-908` therefore classifies the installed valid CLI as an implementation gap.
|
||||
|
||||
#### Solution
|
||||
|
||||
Use `GEMINI_API_KEY` and `GOOGLE_GEMINI_BASE_URL`; map `gemini-3.6-flash` to official label `Gemini 3.6 Flash`; omit `--effort`. Version-gate 1.1.12 and its real help options. Parse `init`, `step_update`, `result`; preserve result/step usage as caller-reported count metrics and duration seconds as a duration metric, derive finish+idle from one successful result plus process exit/quiescence, and classify non-success as failure. Effective route/stage binding comes only from the independently hashed config observation validated before launch, never a fabricated caller event. Structural redaction persists only event/state/step type/status and numeric metrics.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [x] `scripts/agent_benchmark/agy_iop.py`: replace constants, invocation, capability and official event parser/redactor.
|
||||
- [x] `scripts/agent_benchmark/agy_iop_test.py`: replace fake transport/synthetic event tests with captured official 1.1.12 shapes and boundary cases.
|
||||
- [x] `scripts/agent_benchmark/live_iop.py`: derive catalog root from route-qualified Gemini base and bind config-owned effective route before agy invocation.
|
||||
- [x] `scripts/agent_benchmark/connectivity_integration_test.py`: update live agy seams and exact admitted binding assertions.
|
||||
- [x] `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl`: replace synthetic fixture with content-free official event shapes.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Tests assert exact env allowlist/argv, no `--effort`, known-model rejection, 1.1.12 help/version gate, official success/error/result order, usage/duration units, malformed/duplicate result rejection, config-binding mismatch and durable secret/content redaction. No unit test contacts a provider.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run `python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test`. Expected: all tests pass.
|
||||
|
||||
### [API-3] Managed dev deployment and live readiness
|
||||
|
||||
#### Problem
|
||||
|
||||
Remote `build/dev-runtime/edge.yaml` serves legacy HTTP and contains static provider auth. Marked presets require managed principal projection/stage authorization, and the new Gemini ingress contract requires HTTPS plus `x-goog-api-key` as an IOP token. Source compatibility alone cannot prove C03/C07 or hybrid readiness.
|
||||
|
||||
#### Solution
|
||||
|
||||
First run the repository deterministic credential-slot smoke. Build current Control Plane/Edge/Node artifacts for each target and stage them on the clean remote release checkout. Under remote `build/dev-runtime/.secrets/credential-plane/` generate one CA, role/name-bound CP/Edge/Node certs, Edge HTTP cert for `toki-labs.com`, at-rest keyring, issuer Ed25519 and per-node X25519 recipient keys with restrictive modes. Create managed candidate configs that remove all legacy bearer/principal/provider header/env values, enable CP credential HTTPS/edge-wire mTLS, Edge CP/Node mTLS and public HTTPS, and Node lease crypto. Bootstrap one benchmark principal into local protected `token/.iop-bench`, register `.gemini`, `.claude`, `.gpt` through credential HTTPS stdin as slots/routes for the canonical direct and stage models, then bounded-restart CP→Edge→Nodes. Verify fresh projection, advertised route ids, safe slot revision/lease attribution, direct agy and hybrid agy through route-qualified bases, Claude/Codex compatibility and the public all-cell preflight. Update the Gemini contract status to active and the human dev guide only after these pass.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [x] `agent-contract/outer/gemini-compatible-api.md`: change planned→active and record final implemented constraints only after live proof.
|
||||
- [x] `agent-contract/index.md`: keep source pointers and trigger terms aligned with implemented files.
|
||||
- [x] `docs/edge-local-dev-guide.md`: add secret-safe route-qualified official agy and managed dev verification commands without token values.
|
||||
- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record sanitized deterministic/remote/live outputs and exact external config paths.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Reuse `make test-credential-slot-smoke` for deterministic security lifecycle. Remote checks use each binary's `config check` where available, TLS health, authenticated model discovery, official agy direct/hybrid minimal workspace calls and benchmark `preflight`; raw provider responses and credentials are not retained.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the Final Verification local suite, deterministic credential smoke, remote clean/source/artifact/config checks, then official agy direct/hybrid and `agent_comparison_benchmark.py preflight`. Expected: all cells ready, no legacy source accepted, and sanitized evidence only.
|
||||
|
||||
### [API-4] Immutable C01-C09 run
|
||||
|
||||
#### Problem
|
||||
|
||||
The benchmark must retain one attempt per cell and scored failures, but invoking it before API-3 or rerunning after partial output would violate the approved manifest and D10.
|
||||
|
||||
#### Solution
|
||||
|
||||
After API-3, invoke public `run` exactly once. Extract exactly one CLI-emitted new run id, write only that id to `run_id.log`, and use public `status` to require exactly nine `success|failed|timed_out|cancelled` with zero running/interrupted. Never call `resume`, retry failed cells, directly invoke scored callers, or edit the run tree.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store one CLI-emitted canonical run id after the one-time command.
|
||||
- [x] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record verbatim sanitized run/status output and classification.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No additional test file. The immutable manifest validation, full benchmark suite, fresh all-cell preflight, one-time public run and read-only status are the acceptance oracle.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run Final Verification step 5 exactly once. Mark API-4 complete only for nine retained non-interrupted terminal attempts; otherwise record blocker and do not rerun.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Item |
|
||||
|------|------|
|
||||
| `apps/edge/internal/openai/routes.go` | API-1 |
|
||||
| `apps/edge/internal/openai/principal.go` | API-1 |
|
||||
| `apps/edge/internal/openai/gemini_types.go` | API-1 |
|
||||
| `apps/edge/internal/openai/gemini_handler.go` | API-1 |
|
||||
| `apps/edge/internal/openai/gemini_bridge.go` | API-1 |
|
||||
| `apps/edge/internal/openai/gemini_handler_test.go` | API-1 |
|
||||
| `scripts/agent_benchmark/agy_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/agy_iop_test.py` | API-2 |
|
||||
| `scripts/agent_benchmark/claude_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/claude_iop_test.py` | API-2 |
|
||||
| `scripts/agent_benchmark/codex_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/codex_iop_test.py` | API-2 |
|
||||
| `scripts/agent_benchmark/lifecycle.py` | API-2, API-4 recovery |
|
||||
| `scripts/agent_benchmark/lifecycle_test.py` | API-2, API-4 recovery |
|
||||
| `scripts/agent_benchmark/attempts.py` | API-4 recovery |
|
||||
| `scripts/agent_benchmark/attempts_test.py` | API-4 recovery |
|
||||
| `scripts/agent_benchmark/live_iop.py` | API-2 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | API-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl` | API-2 |
|
||||
| `scripts/e2e-credential-slot-smoke.sh` | API-3 |
|
||||
| `agent-contract/outer/openai-compatible-api.md` | API-3 |
|
||||
| `agent-contract/outer/gemini-compatible-api.md` | API-3 |
|
||||
| `agent-contract/index.md` | API-3 |
|
||||
| `docs/edge-local-dev-guide.md` | API-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | API-4 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | API-1, API-2, API-3, API-4 evidence |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Format and run fresh focused suites:
|
||||
|
||||
```bash
|
||||
gofmt -w apps/edge/internal/openai/routes.go apps/edge/internal/openai/principal.go apps/edge/internal/openai/gemini_types.go apps/edge/internal/openai/gemini_handler.go apps/edge/internal/openai/gemini_bridge.go apps/edge/internal/openai/gemini_handler_test.go
|
||||
go test -count=1 ./apps/edge/internal/openai
|
||||
python3 -m unittest scripts.agent_benchmark.agy_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: all focused tests pass.
|
||||
|
||||
2. Run the full local suites and contract hygiene:
|
||||
|
||||
```bash
|
||||
go test -p 1 -count=1 ./...
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: all packages/tests pass, manifest is valid, no whitespace error.
|
||||
|
||||
3. Run deterministic managed credential qualification outside the repo temp tree:
|
||||
|
||||
```bash
|
||||
credential_smoke_parent="$(mktemp -d /tmp/iop-bench02-credential.XXXXXX)"
|
||||
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
|
||||
rmdir "$credential_smoke_parent"
|
||||
```
|
||||
|
||||
Expected: CA/mTLS, two-slot, rotation, revoke/no-fallback and Messages/Chat qualification pass.
|
||||
|
||||
4. After fresh dev deploy, run the sanitized live readiness commands documented in `docs/edge-local-dev-guide.md`: official agy direct and hybrid minimal calls through `https://toki-labs.com:18083/gemini/<route-id>`, authenticated model discovery, and:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Expected: C01-C09 all report ready with official caller version, managed route/stage binding and no secret output.
|
||||
|
||||
5. Implementation-only, exactly once after step 4: run the immutable public benchmark and preserve its CLI-emitted id.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Then write the emitted `run-...` id to `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` and run:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
```
|
||||
|
||||
Expected: exactly nine retained non-interrupted terminal attempts. Do not rerun step 5.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,203 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=4 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - pre-scored interruption 이후 C01-C09 replacement 실행
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 명령과 상태 경계를 그대로 실행하고 실제 stdout/stderr를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, control-plane stop 파일, verdict·archive·`complete.log` 작성은 하지 않는다.
|
||||
|
||||
## Background
|
||||
|
||||
plan=3의 제품 코드, managed dev 배포와 caller readiness는 완료됐지만 최초 public `run`은 caller launch 전에 control socket `EINVAL`로 중단됐다. 그 결함과 pre-registration reconcile은 수정·회귀 검증됐고, SDD D10은 실패 evidence를 보존한 새 attempt를 허용한다. 이 revision은 이전 interruption을 덮어쓰지 않고 새 public run identity에서 C01-C09를 실행하는 작업만 소유한다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G09_0.log`~`plan_cloud_G09_2.log`, `code_review_cloud_G10_0.log`~`code_review_cloud_G10_2.log`는 verdict 없는 이전 준비 revision이다.
|
||||
- `plan_cloud_G10_3.log`와 `code_review_cloud_G10_3.log`는 Gemini ingress, official agy 1.1.12, managed credential dev 배포와 readiness 구현 evidence를 보존한다. fresh local suite는 Go 전체 PASS, Python 429 tests/OK였고 remote Edge hash는 `c72a926ec2a5e39b9b59c39e8630c70f093dfb332c004f2b214c76a940a592cc`다.
|
||||
- 기존 controller state `run-20260812T031657Z-0a24376c2414`는 `caller_launched=false`, terminal `interrupted=1`, `running=0`인 pre-scored infrastructure failure다. 수정 뒤 같은 run을 resume하거나 state tree를 편집하지 않았다.
|
||||
- planning 중 runtime reference가 없는 셸에서 수행한 preflight-only `run-20260812T035510Z-d484c647ebac`은 attempt 없이 `implementation_gap=9`를 보존한다. 보호된 reference를 주입한 fresh preflight-only `run-20260812T035838Z-4c4a056c63b1`은 `ready=9`다. 둘 다 scored caller를 호출하지 않았다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Rules/skills: `agent-ops/rules/project/rules.md`, `agent-ops/rules/common/rules-roadmap.md`, `agent-ops/rules/project/domain/testing/rules.md`, `agent-ops/skills/common/router.md`, `agent-ops/skills/common/plan/SKILL.md`, `agent-ops/skills/common/finalize-task-routing/SKILL.md`, `agent-ops/skills/common/update-test/SKILL.md`, `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- Test rules: `agent-test/local/rules.md`, `agent-test/dev/rules.md`, `agent-test/dev/testing-smoke.md`, `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md`
|
||||
- Roadmap/SDD: `agent-roadmap/current.md`, `agent-roadmap/priority-queue.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/PHASE.md`, `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`, `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- Benchmark/runtime: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`, `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/live_iop.py`, `scripts/agent_benchmark/claude_iop.py`, `scripts/agent_benchmark/codex_iop.py`, `docs/edge-local-dev-guide.md`
|
||||
- Active evidence: `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_3.log`, `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_3.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, 상태 `[승인됨]`, 잠금 `해제`, 사용자 리뷰 없음.
|
||||
- `milestone-task`는 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`다.
|
||||
- 대상 Acceptance Scenario는 S04(C01), S05(C02-C03), S06(C04-C05), S07(C06-C07), S08(C08-C09)다. Evidence Map은 caller별 event/timing/usage/workspace와 hybrid stage/terminal evidence를 요구한다.
|
||||
- D06은 cell별 repetitions=1과 fresh session을, D10은 실패 보존과 새 attempt 기록을 요구한다. 따라서 TEST-1은 S02 readiness를 재확인하고 TEST-2는 S04-S08의 새 run evidence를 append-only로 생성한다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Environment: dev. Rules state는 usable이며 matching profile은 testing/edge/node smoke다. 파일 수정 없는 `resolve-context`로 local current checkout, remote dev runtime과 public benchmark CLI를 판정했다.
|
||||
- Local runner: Linux AArch64, `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, HEAD `8dcf2a3246b1cc36ad15f4c9015fa3b66fd09832`, 현재 task 구현 변경 때문에 dirty다. `../iop-s2`는 branch `dev`, HEAD `1f2f7f1066fcf165a9e469bae77203b569b6f772`, clean이다.
|
||||
- Remote runtime: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`, Darwin/arm64, branch `release/dev-936`, HEAD `d40e4494e767e3fec7796c670f58130a9a194e80`, clean이다. `build/dev-runtime/bin/edge` hash는 위 snapshot과 일치하고 PID 172, ports 18082/18083/18084/19093/19101은 모두 up이다.
|
||||
- CLI: `scripts/agent_comparison_benchmark.py`는 validate/preflight/run/resume/status/score/report를 제공하고 manifest validate는 `ok: manifest is valid`다. caller binary/version/help와 model catalog는 fresh public preflight가 실제로 확인했다.
|
||||
- Runtime references: principal은 보호된 `token/.iop-bench`, private dev CA는 `token/iop-dev-ca.pem`에서 같은 process environment로만 읽는다. caller base/secret reference와 config observation 값만 child allowlist에 전달하며 값은 출력·tracked 파일·run metadata에 기록하지 않는다.
|
||||
- Read-only preflight: 무설정 호출은 `implementation_gap=9`로 fail-closed했고 attempt를 만들지 않았다. reference 주입 뒤 `ok: preflight run_id=run-20260812T035838Z-4c4a056c63b1 status=ready ready=9 registration_required=0 implementation_gap=0`으로 통과했다.
|
||||
- Constraints: benchmark CLI 밖 caller/provider 호출, route/model/effort 대체, old run resume, `--retry-failed`, run tree 편집, testbed 쓰기, raw secret 출력은 금지한다. 새 execution run이 retained failure로 끝나면 status만 기록하고 재실행하지 않는다.
|
||||
- Gaps: 없음. Confidence는 high이며 test rule 유지보수는 필요 없다.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- 새 제품 동작이나 source 변경은 없다. control socket symlink와 pre-registration reconcile 회귀는 plan=3에서 lifecycle/attempt tests 및 전체 Python suite로 검증됐다.
|
||||
- 남은 gap은 실제 C01-C09 결과뿐이며 unit test로 대체할 수 없다. public preflight/run/status가 acceptance oracle이다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- 없음. symbol rename, remove, dependency 변경이 없다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- 새 run identity의 preflight, attempt allocation과 terminal status는 하나의 append-only state invariant다. 별도 sibling으로 나누면 runtime identity와 one-submission evidence가 분리되므로 같은 task path의 단일 plan으로 유지한다.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Edge/Node/Control Plane, caller adapter, manifest, SDD, contract와 dev config를 수정하지 않는다. 현재 ready runtime을 소비하는 검증 실행만 범위다.
|
||||
- objective validation, blind scoring, performance aggregation과 report는 세 번째 Epic S09-S12의 후속 plan으로 남긴다.
|
||||
- old interrupted run resume/retry/reconcile, 실패 cell 재시도, provider 직접 호출은 제외한다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
|
||||
- Build closures(scope/context/verification/evidence/ownership/decision)는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, base/route basis=`grade-boundary`, lane=`cloud`, grade=`G10`, filename=`PLAN-cloud-G10.md`, catalog=`worker/cloud/G10`.
|
||||
- Review closures는 모두 true, capability gap은 none, scores=`2/2/2/2/2`, route=`official-review`, lane=`cloud`, grade=`G10`, filename=`CODE_REVIEW-cloud-G10.md`, catalog=`review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; risks=`temporal_state,concurrent_consistency,boundary_contract,structured_interpretation,variant_product` (5); `review_rework_count=0`; `evidence_integrity_failure=false`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. TEST-1의 manifest validation과 fresh all-cell preflight가 `ready=9`여야 TEST-2를 시작한다.
|
||||
2. TEST-2는 public `run`을 한 번만 호출한다. CLI가 만든 새 canonical run id를 기록한 뒤에는 read-only `status`만 사용한다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Bind protected benchmark runtime references and pass fresh manifest validation plus C01-C09 all-cell preflight with `ready=9` and no secret output.
|
||||
- [ ] TEST-2: Execute one new public C01-C09 run, preserve its canonical run id, and verify exactly nine retained non-interrupted terminal attempts without resume, retry, substitution, or state editing.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Protected runtime binding and fresh readiness
|
||||
|
||||
#### Problem
|
||||
|
||||
`scripts/agent_benchmark/live_iop.py:1-5,259-266` requires caller base/secret references and an operator-owned config observation. A fresh shell without those references produced preflight-only `implementation_gap=9`, even though the deployed runtime itself remained healthy. Starting scored work before a closed `ready=9` violates S02 and the benchmark skill stop condition.
|
||||
|
||||
#### Solution
|
||||
|
||||
Read the existing principal and CA only inside the execution process, bind Claude/agy/Codex to their deployed HTTPS surfaces, and provide the already-verified route/preset observation. Run manifest validation, the focused lifecycle regression, caller version/help checks and public preflight. No repository source or configuration is changed.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record exact secret-safe validation/preflight output.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No new test file is required because there is no behavior change. Run fresh lifecycle/attempt regressions and the public preflight; cached output is not accepted.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the Final Verification readiness block. Expected: manifest valid, focused tests pass, caller commands resolve, environment reference booleans are true, and preflight reports `ready=9` with no registration or implementation gap.
|
||||
|
||||
### [TEST-2] One replacement C01-C09 execution run
|
||||
|
||||
#### Problem
|
||||
|
||||
`code_review_cloud_G10_3.log:159-197` records that the former execution command ended before caller launch and retained no C01-C09 result. S04-S08 therefore remain incomplete even though the infrastructure fault is fixed and readiness is green.
|
||||
|
||||
#### Solution
|
||||
|
||||
With TEST-1 references still bound, call the public `run` exactly once to create a distinct execution run. Preserve all older run roots, store only the CLI-emitted canonical execution id in `run_id.log`, and use public `status` to require nine `success|failed|timed_out|cancelled`, zero `interrupted`, and zero `running`. A retained failed/timed-out/cancelled cell remains evidence and is not retried.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log`: store exactly one CLI-emitted execution run id.
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md`: record the verbatim run output, exit code, canonical id and public status.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
No new test file is required. The deterministic CLI's append-only run/status contract is the only valid live acceptance path; callers are never invoked separately.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the Final Verification execution block once. Expected: exactly one new execution run id and nine retained accepted terminal attempts with zero interrupted/running. Do not invoke `run` or `resume` again in implementation or review.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Item |
|
||||
|------|------|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Run fresh local contract checks:
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.lifecycle_test scripts.agent_benchmark.attempts_test
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: fresh tests pass, manifest is valid, and `git diff --check` has no output.
|
||||
|
||||
2. In one protected shell, bind runtime references and run fresh all-cell preflight. Read token values only into variables; never print them:
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
|
||||
export IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CLAUDE_BASE_URL=https://toki-labs.com:18083
|
||||
export IOP_BENCH_AGY_BASE_URL=https://toki-labs.com:18083
|
||||
export IOP_BENCH_CODEX_BASE_URL=https://toki-labs.com:18083/v1
|
||||
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
|
||||
BENCH_CONFIG="$(python3 - <<'PY'
|
||||
import json
|
||||
routes = [
|
||||
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
|
||||
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
|
||||
]
|
||||
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
|
||||
PY
|
||||
)"
|
||||
export BENCH_CONFIG
|
||||
for tool in python3 claude agy codex git; do command -v "$tool"; done
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Expected: every tool resolves and preflight reports `ready=9 registration_required=0 implementation_gap=0`. If it does not, stop before TEST-2.
|
||||
|
||||
3. In the same protected shell, invoke the execution command exactly once and preserve its direct CLI output. A nonzero exit is acceptable only when public status later proves nine retained accepted terminal attempts:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
Expected: the CLI emits one canonical execution id. Record the command exit code and stdout/stderr verbatim, then write that exact emitted id to `run_id.log` and use step 4 for status. Never rerun this block.
|
||||
|
||||
4. Verify the stored pointer without invoking a provider:
|
||||
|
||||
```bash
|
||||
test "$(wc -l < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)" -eq 1
|
||||
execution_run_id="$(tr -d '\n' < agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log)"
|
||||
printf '%s\n' "$execution_run_id" | grep -Eq '^run-[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$'
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json --run-id "$execution_run_id"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: valid id, nine retained accepted terminal attempts, zero interrupted/running, and no whitespace error.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=6 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - 지속 승인 기반 C01-C09 fresh scored execution
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md` 구현 섹션을 채우는 것이 필수 마지막 단계다. protected local references를 process environment에서만 읽고, public benchmark CLI를 통해 preflight와 새 `run`을 수행한다. caller/provider 직접 호출, old run resume, `--retry-failed`, run tree 편집, secret 출력은 금지한다. 새 run이 retained failure로 끝나더라도 사용자의 지속 승인은 이후 새 plan 진행 권한으로 유지되지만, 이 plan 안에서는 `run`을 한 번만 호출하고 status evidence를 기록한 뒤 멈춘다.
|
||||
|
||||
## Background
|
||||
|
||||
plan=5에서 Codex production usage parser와 config-owned execution/scoring binding 결함을 수정했고 focused 41 tests와 전체 benchmark 429 tests가 통과했다. 이후 `USER_REVIEW.md`는 사용자가 “이후는 계속 승인 상태이니 이어서 진행”이라고 명시해 external-execution 조건이 해소됐으며 `user_review_0.log`로 보존됐다. 이 revision은 모든 기존 run을 보존한 distinct new run identity에서 C01-C09를 실행한다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G10_4.log`/`code_review_cloud_G10_4.log`: retained run `run-20260812T040619Z-00a5e6664764`의 parser/binding failure와 FAIL evidence.
|
||||
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: R1/R2 source fix, focused 41/OK, full 429/OK와 새 execution authorization gate.
|
||||
- `user_review_0.log`: 사용자 지속 승인으로 exact new public run 실행 조건이 RESOLVED 됐다.
|
||||
- 이전 run은 어떤 방식으로도 resume/retry/reconcile/edit하지 않는다. 새 execution CLI가 출력한 id만 `run_id.log`에 기록한다.
|
||||
|
||||
## Analysis
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- Approved/unlocked SDD의 S04-S08은 C01-C09 one-submission, fresh session, caller lifecycle/usage/workspace 및 hybrid terminal evidence를 요구한다.
|
||||
- D06은 repetitions=1, D10은 실패 evidence 보존과 새 attempt identity를 요구한다.
|
||||
- milestone-task는 기존 다섯 id를 그대로 유지한다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Local source regression: focused 41/OK, full 429/OK, manifest valid, diff check clean.
|
||||
- Dev runtime: prior fresh public preflight가 `ready=9`; existing protected principal과 dev CA는 `token/.iop-bench`, `token/iop-dev-ca.pem`에 있다.
|
||||
- Execution environment binds Claude/agy/Codex base URLs and independently verified config observation in one process only. Raw values are never printed or persisted in tracked task evidence.
|
||||
- Constraints: deterministic public CLI only; no direct caller/provider, no route/model/effort substitution, no old run resume/retry, no run tree mutation.
|
||||
- User authorization is continuing for subsequent necessary new executions. Each individual plan still calls public `run` at most once and preserves failure before another plan.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Deterministic source tests are closed. Remaining gap is real C01-C09 scored evidence, which only public `run/status` can produce.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
All-cell preflight, immutable run creation, nine attempt allocations and terminal status form one append-only execution invariant. Splitting would separate readiness and the exact authorized run identity, so one plan is retained.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- No product/runtime/config/source change is planned.
|
||||
- This plan owns only fresh verification, protected preflight, one new public run, stored pointer and status evidence.
|
||||
- score/report and objective validation remain later Epic work.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- isolated reassessment; closures true; capability gap none.
|
||||
- Build scores `2/2/2/2/2`, risks all five, rework=2, integrity=false; route `grade-boundary`, cloud G10, `PLAN-cloud-G10.md`.
|
||||
- Review scores `2/2/2/2/2`; `official-review`, cloud G10, `CODE_REVIEW-cloud-G10.md`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Fresh deterministic tests and manifest validation pass.
|
||||
2. Protected public preflight reports exactly `ready=9` and zero gaps.
|
||||
3. Call public `run` exactly once.
|
||||
4. Store only a CLI-emitted canonical run id; use read-only status to verify nine accepted terminal attempts, `running=0`, `interrupted=0`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] TEST-1: Re-run focused Codex/integration regression, full benchmark suite, manifest validation, and diff check fresh.
|
||||
- [ ] TEST-2: Bind existing protected runtime references and pass a fresh public all-cell preflight with `ready=9`.
|
||||
- [ ] TEST-3: Invoke one new public C01-C09 `run`, preserve only its CLI-emitted canonical id, and verify public status without resume/retry/direct calls/state editing.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual output.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Item |
|
||||
|---|---|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 evidence |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test`
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
3. `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` and `git diff --check`
|
||||
4. In one secret-safe process, read `token/.iop-bench`, bind the existing dev endpoints/CA/config observation, and call public `preflight`. Require `ready=9 registration_required=0 implementation_gap=0`.
|
||||
5. In that protected process, call `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exactly once. Record verbatim stdout/stderr and exit code. Write `run_id.log` only if the CLI emits one canonical id.
|
||||
6. If an id was emitted, call public `status`. Completion requires nine retained terminal attempts, zero running/interrupted. If no id was emitted, identify the newly created run root read-only for blocker evidence but do not fabricate `run_id.log`.
|
||||
|
||||
After completing, fill CODE_REVIEW implementation-owned sections and keep the active pair for review.
|
||||
|
|
@ -0,0 +1,79 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=7 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - agy failed lifecycle terminalization 및 fresh C01-C09 run
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
R4의 선택된 수정만 구현하고 credential-free tests를 통과한 뒤 protected preflight와 public `run`을 각 계획대로 수행한다. 원인 재조사, old run resume, retry-failed, caller/provider 직접 호출, state tree 편집, secret 출력은 금지한다. 새 run은 이 plan에서 정확히 한 번 호출한다.
|
||||
|
||||
## Background
|
||||
|
||||
plan=6 new run `run-20260812T043704Z-041320764a11`은 C05 Codex를 success로 종결해 R1/R2 fix를 검증했지만, C03 agy의 valid `nonzero_exit`를 live adapter가 `stream_incompatible` 예외로 바꿔 attempt를 running으로 남겼다. 사용자의 이후 실행 지속 승인은 유효하다. 기존 run은 보존하고 R4 fix 이후 distinct new run을 수행한다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `plan_cloud_G10_6.log`/`code_review_cloud_G10_6.log`: ready=9, one public run, C02 failed/C05 success/C03 running과 R4 FAIL evidence.
|
||||
- `plan_local_G06_5.log`/`code_review_cloud_G06_5.log`: Codex R1/R2 fixes and 429/OK.
|
||||
- `user_review_0.log`: subsequent execution remains authorized.
|
||||
|
||||
## Finding Resolution Map
|
||||
|
||||
| Finding | Evidence | Root Cause | Selected Fix | Mode | Acceptance |
|
||||
|---|---|---|---|---|---|
|
||||
| Required R4 | C03 lifecycle is clean nonzero_exit/cleanup complete, but measurement/web absent and attempt running; agy branch raises after `observed_result` | post-admission execution failure is incorrectly reclassified as connectivity failure | return bound failed lifecycle directly; apply observed binding gate only to successful lifecycle; add integration test proving failure terminalization/continuation | direct-fix | focused integration, full 429+, ready=9, one new run with nine terminal attempts |
|
||||
|
||||
## Analysis
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
S04-S08/D06/D10 and the five milestone-task ids remain unchanged. Failed cells are accepted terminal evidence; `running|interrupted` are not.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Local/dev context and protected `token/.iop-bench`, `token/iop-dev-ca.pem` bindings are unchanged.
|
||||
- Previous fresh ready=9 and Codex success prove runtime and R1/R2.
|
||||
- New gap is deterministic in `_LiveAdapter.invoke` and can be tested without provider access.
|
||||
- Continuing authorization permits a new run after the fix; each plan still invokes run once.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
No test proves agy non-success lifecycle is returned to RunStore and later slots continue. Add the smallest live-registry/run-slots regression using invoker seams and existing production sidecar assertions.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
R4 terminalization and its verification in a new nine-cell run are one correctness boundary; separating them would leave S04-S08 unverified.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Modify only `live_iop.py`, its integration test, active evidence and `run_id.log`. Do not change agy transport/parser, dev runtime, generic recovery, other callers, scoring/report.
|
||||
|
||||
### Final Routing
|
||||
|
||||
isolated reassessment, closures true, scores 2/2/2/2/2, five loop risks, rework=3, integrity=false; build grade-boundary cloud G10, review official cloud G10.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] FIX-1: Return a valid failed agy lifecycle for normal terminal publication; retain exact binding checks for successful streams.
|
||||
- [ ] FIX-2: Add credential-free regression coverage proving failed agy lifecycle terminalizes and does not prevent later eligible slots.
|
||||
- [ ] TEST-1: Pass focused and full benchmark tests, manifest validation, and diff check.
|
||||
- [ ] TEST-2: Pass protected public preflight ready=9, then invoke one new public run and require nine terminal attempts with zero running/interrupted.
|
||||
- [ ] Fill CODE_REVIEW implementation-owned evidence.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Item |
|
||||
|---|---|
|
||||
| `scripts/agent_benchmark/live_iop.py` | FIX-1 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/run_id.log` | TEST-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G10.md` | evidence |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.connectivity_integration_test scripts.agent_benchmark.attempts_test`
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
3. manifest validate and `git diff --check`.
|
||||
4. Protected public preflight must report ready=9.
|
||||
5. Public `run` exactly once. Store only CLI-emitted id and use status. PASS requires nine terminal, running=0, interrupted=0.
|
||||
|
||||
After completing, fill active CODE_REVIEW evidence and stop for review.
|
||||
|
|
@ -0,0 +1,178 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/06+05_comparison_runs plan=5 tag=REVIEW_TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Plan - Codex production stream 및 config-owned binding 정합화
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
`CODE_REVIEW-*-G??.md`의 구현 담당 섹션을 채우는 것이 필수 마지막 단계다. 아래 R1/R2의 선택된 수정만 구현하고 검증 결과를 기록한 뒤 active 파일을 유지한 채 review 준비 상태를 보고한다. 원인을 재조사하거나 다른 해결책을 선택하지 않는다. 막히면 exact blocker, 실행 명령/출력과 재개 조건만 구현 evidence에 남긴다. 사용자 질문, user-input 도구, public benchmark `run|resume`, provider 직접 호출, verdict·archive·`complete.log` 작성은 하지 않는다.
|
||||
|
||||
## Background
|
||||
|
||||
plan=4의 새 public run `run-20260812T040619Z-00a5e6664764`은 C02를 failed로 종결한 뒤 C05 Codex 프로세스가 실제로 끝났음에도 attempt를 `running`으로 남기고 exit 69로 중단됐다. 리뷰가 retained lifecycle stdout을 동일 parser에 재생해 current Codex usage key 누락과 config-owned binding/caller-owned enforcement 모순을 확정했다. 이 revision은 그 두 source defect와 회귀 테스트만 수정한다. 실패 run은 append-only evidence로 보존하며 재개·retry·state 편집·새 scored 실행을 하지 않는다.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/plan_cloud_G10_4.log`와 `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/code_review_cloud_G10_4.log`는 plan=4 실행과 FAIL 판정을 보존한다.
|
||||
- verdict는 Required R1/R2의 `FAIL`, routing signal은 `review_rework_count=1`, `evidence_integrity_failure=false`다.
|
||||
- R1 evidence는 retained C05 stdout line 26의 `turn.completed.usage`가 `cache_write_input_tokens`를 포함하고 현재 parser가 `invalid Codex usage observation`으로 거부한다는 것이다.
|
||||
- R2 evidence는 real Codex stdout에 synthetic `iop_effective_binding`이 없고 parser/test는 이를 optional로 정의하지만 live 실행·채점 소비자는 exact caller observation을 의무화한다는 것이다.
|
||||
- run public status는 `failed=1, running=1`, live execution process는 없고 `run_id.log`는 없다. 이 follow-up은 해당 run을 읽기 전용 evidence로만 취급한다.
|
||||
|
||||
## Finding Resolution Map
|
||||
|
||||
| Finding | Reviewer Evidence | Root Cause | Selected Fix | Mode | Changed Precondition | Acceptance Commands |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Required R1 | C05 retained stdout replay가 `cache_write_input_tokens`에서 `CodexJSONLError`; 기존 9 parser tests는 모두 통과해 coverage gap 확인 | `_CODEX_USAGE_FIELDS`와 tracked fixture가 current caller usage schema보다 오래됨 | `cache_write_input_tokens -> cache_write_tokens`를 exact mapping에 추가하고 unit fixture/assertion을 production shape로 갱신하며 total은 합성하지 않음 | direct-fix | 실제 `turn.completed.usage`가 parser terminal success 및 typed cache-write metric으로 보존됨 | `python3 -m unittest scripts.agent_benchmark.codex_iop_test`; focused integration/full benchmark tests |
|
||||
| Required R2 | parser는 absent binding을 허용하지만 `_LiveAdapter.invoke`와 `_LiveScoringAdapter.invoke`는 absent를 mismatch로 거부; integration fake가 synthetic event 주입 | config observation으로 admission ownership을 이전한 뒤 두 consumer와 fixture가 구 caller-event gate를 유지함 | admitted config binding을 canonical result로 사용하고 optional non-None caller observation만 exact mismatch 검사; execution/scoring 테스트에서 absent PASS와 mismatch FAIL 검증 | direct-fix | real Codex의 absent observation이 admitted binding으로 진행되고 실제 mismatch만 fail-closed함 | named connectivity integration tests; full benchmark tests; `git diff --check` |
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- Reviewer evidence: archived plan/review pair 위 두 exact 경로와 retained run의 C05 `attempt.json`, `lifecycle-result.json`, `cleanup-receipt.json`.
|
||||
- Source/tests: `scripts/agent_benchmark/codex_iop.py`, `codex_iop_test.py`, `live_iop.py`, `connectivity_integration_test.py`, `lifecycle.py`, `attempts.py`, `scoring.py`, tracked Codex JSONL fixture.
|
||||
- Rules/skills: project/testing rules, local test rules, `code-review`, `plan`, `finalize-task-routing`, project benchmark skill.
|
||||
- Roadmap/SDD: active comparison Milestone와 approved/unlocked SDD S04-S08/D06/D10.
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- `milestone-task`는 기존과 동일한 `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`를 유지한다.
|
||||
- S04-S08은 C01-C09 retained terminal evidence를 요구한다. 이 source-fix revision은 실패한 실행 경계를 복구하지만 그 scored evidence 자체를 대체하지 않는다.
|
||||
- D10에 따라 failed/incomplete run을 수정하거나 재사용하지 않는다. 이후 새 scored 실행은 별도 명시적 authorization과 새 plan에서만 수행한다.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Environment는 local source/test다. provider, remote dev, protected token이 필요하지 않다.
|
||||
- Reviewer가 production retained output으로 exact parser failure를 재현했고 current test가 gap을 놓치는 것도 확인했다.
|
||||
- 변경 후 credential-free unit/integration/full Python suite와 manifest validation을 fresh 실행한다. cached output은 허용하지 않는다.
|
||||
- Constraints: archived/run evidence 수정 금지, public `run|resume` 금지, caller/provider 직접 호출 금지, optional observation mismatch의 fail-closed 성질 유지.
|
||||
- Confidence는 high이고 verification context gap은 없다.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- parser test/fixture에 `cache_write_input_tokens`가 없다.
|
||||
- live integration fake는 실제 Codex가 내보내지 않는 `iop_effective_binding`을 합성한다.
|
||||
- execution consumer의 absent-observation PASS와 scoring consumer의 absent-observation PASS/non-None mismatch FAIL 조합이 없다.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `_CODEX_USAGE_FIELDS`: exact accepted caller usage schema.
|
||||
- `CodexJSONLParser._turn_observations`: typed metric conversion and unknown-key fail-closed boundary.
|
||||
- `_LiveAdapter.invoke`: execution-time config admission consumer.
|
||||
- `_LiveScoringAdapter.invoke`: scoring-time config admission consumer.
|
||||
- `CodexInvocationResult.effective_binding`: optional caller observation; canonical config binding과 동일한 소유물이 아니다.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
R1 parser terminal success와 R2 admitted binding 소비는 하나의 real Codex invocation이 lifecycle result로 반환되는 compact boundary다. 분리하면 parser가 성공한 직후 동일 caller-binding 모순으로 다시 중단되므로 한 plan에서 함께 수정한다. predecessor index 05는 기존 archived completion evidence로 이미 충족되며 새 dependency는 없다.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Codex parser, live execution/scoring consumer, 해당 tests/fixture만 수정한다.
|
||||
- generic attempt recovery, CLI exception policy, Claude의 실제 tool-use failure, agy/Edge/provider/runtime config는 변경하지 않는다.
|
||||
- scored result 생성·재개·채점·report와 roadmap 완료 갱신은 제외한다.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`, finalizer=`finalize-task-policy.sh`, mode=`pair`, status=`routed`.
|
||||
- Build closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06, risks=`boundary_contract,structured_interpretation`, `large_indivisible_context=false`, `review_rework_count=1`, `evidence_integrity_failure=false`; route=`local-fit`, lane=`local`, filename=`PLAN-local-G06.md`, catalog=`worker/local/G06`.
|
||||
- Review closures는 모두 true, capability gap none, scores=`2/1/1/1/1`→G06; route=`official-review`, lane=`cloud`, filename=`CODE_REVIEW-cloud-G06.md`, catalog=`review/cloud/G06`.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. FIX-1이 production usage shape를 terminal success로 parse해야 한다.
|
||||
2. FIX-2가 그 lifecycle result를 optional caller observation 없이 admitted config binding으로 실행·채점 소비하되 non-None mismatch는 거부해야 한다.
|
||||
3. focused tests가 통과한 뒤 전체 Python benchmark suite와 manifest validation을 실행한다.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] FIX-1: Accept the current Codex `cache_write_input_tokens` usage key as canonical `cache_write_tokens`, update the production-shaped parser fixture/test, and preserve exact reported counts without reconstructing totals.
|
||||
- [ ] FIX-2: Make admitted config observation the canonical execution/scoring binding, accept an absent caller binding observation, reject any present mismatch, and replace synthetic integration coverage with real absent/mismatch cases.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [FIX-1] Current Codex usage schema
|
||||
|
||||
#### Problem
|
||||
|
||||
`CodexJSONLParser._turn_observations` validates usage keys as an exact subset of `_CODEX_USAGE_FIELDS`. The production `turn.completed` adds `cache_write_input_tokens`; omission converts an otherwise completed invocation into `parser_error`.
|
||||
|
||||
#### Solution
|
||||
|
||||
Add only the observed key with canonical metric name `cache_write_tokens`. Update the unit event and tracked stream fixture to include the retained production-shaped five fields, assert every exact reported count, and keep `total_tokens` unavailable when the caller omits it.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/agent_benchmark/codex_iop.py`: extend exact usage mapping.
|
||||
- [ ] `scripts/agent_benchmark/codex_iop_test.py`: assert production-shaped cache-write parsing and no derived total.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl`: align deterministic fixture with current caller output.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
The parser unit must fail before the mapping and pass after it. Existing unknown/fractional/negative-key/value cases continue proving fail-closed behavior.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the focused parser suite in Final Verification step 1.
|
||||
|
||||
### [FIX-2] Config-owned effective binding consumption
|
||||
|
||||
#### Problem
|
||||
|
||||
Preflight admits exact route/model/stage from independently validated config. Real Codex reports no custom binding event, but execution/scoring consumers currently require one, making all real successful Codex invocations impossible.
|
||||
|
||||
#### Solution
|
||||
|
||||
For execution, allow `result.effective_binding is None`, reject only a present value different from admitted, then retain the lifecycle observations already checked against admitted models/stages. For scoring, apply the same optional mismatch check and return the expected admitted tuple as the canonical scoring result. Remove the synthetic event from the integration fake; assert absent observation succeeds and present mismatch fails in both consumer paths.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/agent_benchmark/live_iop.py`: align execution and scoring consumers with config-owned admission.
|
||||
- [ ] `scripts/agent_benchmark/connectivity_integration_test.py`: remove synthetic Codex event and cover absent/mismatch execution and scoring behavior.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Use credential-free invoker seams returning real-shaped `CodexInvocationResult`. Verify absent optional observation reaches success using admitted binding, while a contradictory non-None tuple remains fail-closed.
|
||||
|
||||
#### Verification
|
||||
|
||||
Run the focused integration suite in Final Verification step 1, then the full suite in step 2.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| Path | Item |
|
||||
|---|---|
|
||||
| `scripts/agent_benchmark/codex_iop.py` | FIX-1 |
|
||||
| `scripts/agent_benchmark/codex_iop_test.py` | FIX-1 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl` | FIX-1 |
|
||||
| `scripts/agent_benchmark/live_iop.py` | FIX-2 |
|
||||
| `scripts/agent_benchmark/connectivity_integration_test.py` | FIX-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/CODE_REVIEW-cloud-G06.md` | FIX-1, FIX-2 evidence |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. Run focused parser and integration coverage:
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
```
|
||||
|
||||
Expected: all focused credential-free tests pass, including production cache-write usage, absent config-owned binding success, and mismatch rejection.
|
||||
|
||||
2. Run the fresh full benchmark suite:
|
||||
|
||||
```bash
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
```
|
||||
|
||||
Expected: all benchmark tests pass with no skipped required test or network/provider call.
|
||||
|
||||
3. Validate manifest and repository diff:
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: `ok: manifest is valid`; `git diff --check` has no output. Do not run public `run`, `resume`, `score`, caller, or provider commands in this follow-up.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1 @@
|
|||
run-20260812T044800Z-412e05fc80df
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
# User Review Required - m-iop-one-shot-agent-model-comparison/06+05_comparison_runs
|
||||
|
||||
## Requested At
|
||||
|
||||
2026-08-12
|
||||
|
||||
## Status
|
||||
|
||||
RESOLVED
|
||||
|
||||
## Reason
|
||||
|
||||
- Type: external-execution
|
||||
- Target: `scripts/agent_comparison_benchmark.py run` against `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` using the existing protected `token/.iop-bench` and `token/iop-dev-ca.pem` dev binding
|
||||
- Current review number: 5
|
||||
- Final verdict: FAIL
|
||||
- Summary: R1/R2 source defects are fixed and all deterministic tests pass, but the one scored run previously authorized was retained as `failed=1,running=1`. Benchmark policy requires explicit user authorization before another stateful scored execution.
|
||||
|
||||
## Loop History
|
||||
|
||||
| Plan | Review | Verdict | Note |
|
||||
|---|---|---|---|
|
||||
| `plan_cloud_G09_0.log` | `code_review_cloud_G10_0.log` | unknown | Initial execution preparation; no official verdict. |
|
||||
| `plan_cloud_G09_1.log` | `code_review_cloud_G10_1.log` | unknown | Readiness revision; no official verdict. |
|
||||
| `plan_cloud_G09_2.log` | `code_review_cloud_G10_2.log` | unknown | Runtime/provider readiness revision; no official verdict. |
|
||||
| `plan_cloud_G10_3.log` | `code_review_cloud_G10_3.log` | unknown | Gemini ingress, official agy, managed dev and caller readiness completed; first controller run interrupted before scored caller launch. |
|
||||
| `plan_cloud_G10_4.log` | `code_review_cloud_G10_4.log` | FAIL | Replacement run exposed current Codex usage parser and config-binding consumer defects. |
|
||||
| `plan_local_G06_5.log` | `code_review_cloud_G06_5.log` | FAIL | R1/R2 fixed; focused 41 and full 429 tests pass, but a new scored execution remains authorization-gated. |
|
||||
|
||||
## Blocking Evidence
|
||||
|
||||
- Problem: S04-S08 still lack one C01-C09 run containing exactly nine retained terminal attempts with zero `running|interrupted`.
|
||||
- Current archived plan: `plan_local_G06_5.log`
|
||||
- Current archived review: `code_review_cloud_G06_5.log`
|
||||
- Verification command: `python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test && python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' && python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json && git diff --check`
|
||||
- Actual output: focused `Ran 41 tests in 14.154s — OK`; full `Ran 429 tests in 116.096s — OK`; `ok: manifest is valid`; diff check `(none)`. Existing run status remains `{'cancelled': 0, 'failed': 1, 'interrupted': 0, 'running': 1, 'success': 0, 'timed_out': 0}`.
|
||||
- Blocking rationale: the declared CLI and protected runtime binding are available, but another scored run repeats stateful/cost-bearing model execution after a retained failure. The benchmark skill forbids implicit retry or replacement execution without explicit user authorization.
|
||||
|
||||
## Required User Action
|
||||
|
||||
- [x] Explicitly authorize exactly one new public C01-C09 scored `run` after the retained failure. Resolved by the user's instruction that subsequent execution remains continuously authorized; no token value or provider key handoff is needed.
|
||||
|
||||
## Resume Condition
|
||||
|
||||
- The user explicitly states that one new public C01-C09 scored run is authorized. The next plan must preserve all old run roots, use fresh public preflight `ready=9`, call public `run` once, and never call `resume`, `--retry-failed`, direct callers/providers, or edit run state.
|
||||
|
||||
## Next Execution Hint
|
||||
|
||||
- Re-enter the `plan` skill for `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/`; archive this file as `user_review_0.log`, create a fresh routed execution pair, consume `token/.iop-bench` and `token/iop-dev-ca.pem` only inside the protected process environment, and use the deterministic public CLI.
|
||||
|
||||
## Closure Rules
|
||||
|
||||
- If the recorded user action and evidence resolve this stop as complete/PASS, update `USER_REVIEW.md` to the resolved state, write `complete.log` from `agent-ops/skills/common/code-review/templates/complete-log-template.md`, and move the task directory to the archive.
|
||||
- If new implementation is required, the `plan` skill archives `USER_REVIEW.md` as `user_review_N.log` before writing a new `PLAN-*-G??.md` / `CODE_REVIEW-*-G??.md` pair.
|
||||
|
|
@ -0,0 +1,148 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST]** Fill every implementation-owned section, run verification, keep active files in place, and report ready for review. Finalization, archive moves, and `complete.log` are review-only.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts, plan=0, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor terminal run: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`.
|
||||
- Failed-result report: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md` (`success=2`, `failed=7`, no required workspace files).
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| TEST-1 Claude write tools/parser | [x] |
|
||||
| TEST-2 Codex sandbox/metrics | [x] |
|
||||
| TEST-3 agy HOME | [x] |
|
||||
| TEST-4 guide/full tests | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage.
|
||||
- [x] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage.
|
||||
- [x] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage.
|
||||
- [x] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
- [x] Append PASS/WARN/FAIL and routing signals after fresh review verification.
|
||||
- [x] Verify finding classifications and dimension assessment.
|
||||
- [x] Archive active plan/review, write `complete.log` on PASS, and move the task directory to the dated archive.
|
||||
- [x] Preserve milestone-task metadata and keep the active parent because child 08 remains.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
없음. 계획에 명시된 파일과 동작 경계 안에서 구현했다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Claude는 shell/network를 열지 않고 `Read,Write,Edit`만 허용하며 같은 목록만 사전 승인한다.
|
||||
- Claude partial snapshot은 message id로 중복 제거하고 user tool-result 경계에서 미완료 tool-use turn을 확정한다.
|
||||
- Codex tool completion count와 선택적 duration metric을 분리하고, 검증 실패 전에는 count 상태를 바꾸지 않는다.
|
||||
- agy는 prepared session directory를 HOME으로 사용하며 workspace/session/attempt 경로가 모두 존재해야 실행한다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Claude has only Read/Write/Edit and accepts actual tool-use stream lifecycle without accepting contradictory terminals.
|
||||
- Codex always uses workspace-write and tool counts are not coupled to optional duration.
|
||||
- agy receives the fresh prepared HOME and no ambient user config.
|
||||
- No secrets or task contents enter captures.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Focused adapter tests
|
||||
|
||||
Command: `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test`
|
||||
|
||||
```text
|
||||
.................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 33 tests in 6.742s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
exit status: 0
|
||||
|
||||
### Complete benchmark tests
|
||||
|
||||
Command: `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'`
|
||||
|
||||
```text
|
||||
.................................................................................................................................................................................................................................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 433 tests in 119.205s
|
||||
|
||||
OK
|
||||
```
|
||||
|
||||
exit status: 0
|
||||
|
||||
### Contract search
|
||||
|
||||
Command: `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md`
|
||||
|
||||
```text
|
||||
scripts/agent_benchmark/agy_iop.py:344: "HOME": prepared.session_dir,
|
||||
scripts/agent_benchmark/claude_iop.py:433: "--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
|
||||
scripts/agent_benchmark/codex_iop.py:228: *executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
|
||||
scripts/agent_benchmark/codex_iop.py:243: "HOME": prepared.session_dir,
|
||||
docs/agent-comparison-benchmark-dev-guide.md:176:| Claude Code | Anthropic-compatible Messages | ... `--tools Read,Write,Edit --allowedTools Read,Write,Edit` ... |
|
||||
docs/agent-comparison-benchmark-dev-guide.md:177:| agy | Gemini-native `streamGenerateContent` | fresh session `HOME` ... |
|
||||
docs/agent-comparison-benchmark-dev-guide.md:178:| Codex | OpenAI-compatible Responses | fresh session `HOME` ... `exec --sandbox workspace-write` ... |
|
||||
```
|
||||
|
||||
exit status: 0
|
||||
|
||||
## Section Ownership
|
||||
|
||||
Implementation item status, checklist, deviations, decisions, and verification output are implementation-owned. Review checklist and verdict are review-only.
|
||||
|
||||
## Reviewer Fresh Verification
|
||||
|
||||
```text
|
||||
$ python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test
|
||||
.................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 33 tests in 6.763s
|
||||
|
||||
OK
|
||||
|
||||
$ python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
.................................................................................................................................................................................................................................................................................................................................................................................................................................................
|
||||
----------------------------------------------------------------------
|
||||
Ran 433 tests in 118.282s
|
||||
|
||||
OK
|
||||
|
||||
$ git diff --check -- <plan write boundary>
|
||||
(no output)
|
||||
```
|
||||
|
||||
All commands exited 0.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- Verdict: PASS
|
||||
- Required: 0
|
||||
- Suggested: 0
|
||||
- Nit: 0
|
||||
- Dimension Assessment:
|
||||
- Correctness: Pass — restricted capabilities and parser state transitions are explicit and fail closed on contradictions.
|
||||
- Completeness: Pass — all four plan items and documentation are implemented.
|
||||
- Test coverage: Pass — launch argv, partial/tool-use streams, durationless Codex tools, and missing agy paths have regressions.
|
||||
- API contract: Pass — existing builder/parser call sites remain compatible; documented CLI contracts match argv.
|
||||
- Code quality: Pass — no debug output, placeholder, or ambient-secret inheritance added.
|
||||
- Plan deviation: Pass — no out-of-scope implementation change.
|
||||
- Verification trust: Pass — reviewer reran 33 focused and 433 complete tests plus diff check.
|
||||
- Findings: 없음.
|
||||
- Routing Signals: review_rework_count=0, evidence_integrity_failure=false.
|
||||
- Next Step: PASS finalization and dependent child 08 execution.
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Complete - m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-12
|
||||
|
||||
## 요약
|
||||
|
||||
Claude/Codex/agy의 격리 workspace 쓰기 계약과 parser/metric 회귀를 1회 루프 PASS로 정리했다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_local_G07_0.log` | `code_review_cloud_G07_0.log` | PASS | 33 focused/433 full tests와 diff check 통과 |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- Claude에 `Read,Write,Edit`만 허용하고 partial/tool-use/user-result stream lifecycle을 처리한다.
|
||||
- Codex에 `workspace-write` sandbox를 지정하고 duration 없는 완료 tool도 정확히 집계한다.
|
||||
- agy에 prepared session HOME을 제공하고 모든 prepared 경로 존재를 검증한다.
|
||||
- dev guide의 세 caller 실행 계약과 격리 경계를 현재 구현에 맞췄다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` - PASS; 33 tests.
|
||||
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - PASS; 433 tests.
|
||||
- `git diff --check -- <plan write boundary>` - PASS; no output.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- `08+07_comparison_rerun`에서 managed dev runtime과 fresh 9-cell execution을 검증한다.
|
||||
|
|
@ -0,0 +1,163 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Caller workspace-write contract remediation
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement every item, run the specified verification, and fill the implementation-owned sections of `CODE_REVIEW-cloud-G07.md` with actual output. Keep both active files in place and report ready for review. If blocked, record only the exact blocker, attempted commands/output, and resume condition. Do not ask the user, create control-plane stop files, archive logs, or write `complete.log`; finalization is code-review-skill only.
|
||||
|
||||
## Background
|
||||
|
||||
The first terminal 9-cell smoke run ended with seven failed cells and no generated workspace files. The adapters launched real callers, but Claude had all tools disabled and rejected normal stream snapshots, Codex did not grant workspace-write sandbox access, and agy omitted the isolated HOME required for its runtime configuration.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log` (PASS for terminal matrix execution).
|
||||
- Canonical failed-result evidence: `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df/report.md`; success=2, failed=7, all web gates failed because required files were absent.
|
||||
- The predecessor review accepted terminal-evidence collection, not successful benchmark artifacts. This plan repairs the concrete caller launch/parser contracts exposed by that run.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop_test.py`
|
||||
- `scripts/agent_benchmark/codex_iop.py`
|
||||
- `scripts/agent_benchmark/codex_iop_test.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop_test.py`
|
||||
- `scripts/agent_benchmark/lifecycle.py`
|
||||
- `scripts/agent_benchmark/lifecycle_test.py`
|
||||
- `docs/agent-comparison-benchmark-dev-guide.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-spec/testing/agent-comparison-benchmark.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/local/testing-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
The approved SDD is `READY`; this plan contributes milestone-task ids `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid`. S04-S08 and Evidence Map rows require terminal evidence plus generated workspace results for C01-C09. The checklist therefore repairs all three caller write paths and requires deterministic adapter regression tests before any new scored run.
|
||||
|
||||
### Verification Context
|
||||
|
||||
No separate handoff was supplied. Repository source, archived predecessor `complete.log`, its canonical run report, and official caller CLI documentation were used. The old run is immutable evidence; no direct caller/provider invocation is permitted. Confidence is high for the launch-contract defects and medium for complete external recovery until the dependent live run is executed.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Claude tests assert the empty `--tools=` launch and only a one-shot `end_turn`; they do not cover partial snapshots or tool-use cycles.
|
||||
- Codex tests do not require `--sandbox workspace-write` and count a tool only when duration is reported.
|
||||
- agy tests do not require isolated HOME or prepared-session directory availability.
|
||||
|
||||
### Symbol References
|
||||
|
||||
No public symbols are renamed or removed. Existing call sites of the modified builders/parsers remain unchanged.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
This is child 07 of a split remediation. It has a stable contract: every caller receives the minimum isolated filesystem capability and its parser accepts the caller's documented event lifecycle. PASS is the focused/full Python test suite. Predecessor 06 is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Child 08 depends on this PASS and owns runtime recovery/scored execution.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not change IOP routing, provider credentials, benchmark task content, scoring, roadmap state, or archived runs. This packet is limited to caller launch/parser contracts, regression tests, and the human dev guide.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair`
|
||||
- build closures: all true; scores `2/1/1/2/1` => G07; base/route `local-fit`; filename `PLAN-local-G07.md`
|
||||
- review closures: all true; scores `2/1/1/2/1` => G07; route `official-review`; filename `CODE_REVIEW-cloud-G07.md`
|
||||
- large_indivisible_context=false; positive risks=`boundary_contract,structured_interpretation,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Give Claude the restricted Read/Write/Edit tool set and support partial/tool-use stream cycles with regression coverage.
|
||||
- [ ] [TEST-2] Grant Codex `workspace-write` sandbox access and count completed tool items even without a duration, with regression coverage.
|
||||
- [ ] [TEST-3] Give agy the prepared isolated HOME and reject missing prepared directories, with regression coverage.
|
||||
- [ ] [TEST-4] Update the human dev guide and run focused plus complete local benchmark tests.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Claude write-tool and stream lifecycle
|
||||
|
||||
**Problem:** `scripts/agent_benchmark/claude_iop.py:220-238` rejects assistant snapshots whose `stop_reason` is null/tool-use, and `:351-356` passes `--tools=` while the task requires file creation.
|
||||
|
||||
**Solution:** Launch Claude with exact `Read,Write,Edit` tools, pre-approve only those tools, retain `dontAsk`, and make the parser deduplicate message ids while accepting partial snapshots, tool-use completion, user tool results, then one final `end_turn` and successful result.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/claude_iop.py`: restricted tool argv and multi-message lifecycle.
|
||||
- [ ] `scripts/agent_benchmark/claude_iop_test.py`: exact argv and partial/tool-use regression cases.
|
||||
|
||||
**Test Strategy:** Add normal, duplicate-terminal, session/model mismatch, and final result ordering assertions to the existing credential-free unit tests.
|
||||
|
||||
**Verification:** `python3 -m unittest scripts.agent_benchmark.claude_iop_test` exits 0.
|
||||
|
||||
### [TEST-2] Codex workspace sandbox and tool metrics
|
||||
|
||||
**Problem:** `scripts/agent_benchmark/codex_iop.py:221-224` starts ephemeral Codex without a writable sandbox. `:330-351` discards real completed command items when `duration_ms` is absent, reporting zero tool calls.
|
||||
|
||||
**Solution:** Add `--sandbox workspace-write` to the isolated exec argv. Count a completed tool item once by safe id/type regardless of duration; publish a duration metric only when the caller explicitly reports one.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/codex_iop.py`: sandbox argv and independent completed-tool count.
|
||||
- [ ] `scripts/agent_benchmark/codex_iop_test.py`: argv and durationless command regressions.
|
||||
|
||||
**Test Strategy:** Extend the existing fixture/unit coverage for exact sandbox arguments, durationless tool completion, uniqueness, and explicit-duration preservation.
|
||||
|
||||
**Verification:** `python3 -m unittest scripts.agent_benchmark.codex_iop_test` exits 0.
|
||||
|
||||
### [TEST-3] agy isolated HOME
|
||||
|
||||
**Problem:** `scripts/agent_benchmark/agy_iop.py` constructs a minimal environment without HOME even though agy stores runtime configuration beneath the user's home, causing startup to exit before JSONL output.
|
||||
|
||||
**Solution:** Require both prepared workspace and session directories and pass `HOME=prepared.session_dir` without treating it as a secret-bearing allowlist key.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `scripts/agent_benchmark/agy_iop.py`: prepared-path validation and HOME environment.
|
||||
- [ ] `scripts/agent_benchmark/agy_iop_test.py`: create session fixture and assert HOME/path rejection.
|
||||
|
||||
**Test Strategy:** Extend the builder tests; no network or live caller execution.
|
||||
|
||||
**Verification:** `python3 -m unittest scripts.agent_benchmark.agy_iop_test` exits 0.
|
||||
|
||||
### [TEST-4] Contract guide and full local tests
|
||||
|
||||
**Problem:** `docs/agent-comparison-benchmark-dev-guide.md` still documents the broken launch contracts.
|
||||
|
||||
**Solution:** Document the minimum write-capability arguments, isolated HOME, and failure classification so future setup does not reintroduce the smoke-only state.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `docs/agent-comparison-benchmark-dev-guide.md`: align caller launch and troubleshooting guidance.
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md`: record actual implementation evidence.
|
||||
|
||||
**Test Strategy:** Run the complete benchmark module suite fresh; documentation is checked by exact searches in review.
|
||||
|
||||
**Verification:** `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` exits 0.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
Predecessor `06+05_comparison_runs` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/complete.log`. Complete TEST-1 through TEST-4 before child `08+07_comparison_rerun` starts.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|---|---|
|
||||
| `scripts/agent_benchmark/claude_iop.py` | TEST-1 |
|
||||
| `scripts/agent_benchmark/claude_iop_test.py` | TEST-1 |
|
||||
| `scripts/agent_benchmark/codex_iop.py` | TEST-2 |
|
||||
| `scripts/agent_benchmark/codex_iop_test.py` | TEST-2 |
|
||||
| `scripts/agent_benchmark/agy_iop.py` | TEST-3 |
|
||||
| `scripts/agent_benchmark/agy_iop_test.py` | TEST-3 |
|
||||
| `docs/agent-comparison-benchmark-dev-guide.md` | TEST-4 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/CODE_REVIEW-cloud-G07.md` | TEST-4 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `python3 -m unittest scripts.agent_benchmark.claude_iop_test scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.agy_iop_test` — exits 0.
|
||||
2. `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` — exits 0 with fresh execution; cached output is not accepted.
|
||||
3. `rg --sort path -n -- '--tools=|--allowedTools|--sandbox|"HOME"' scripts/agent_benchmark docs/agent-comparison-benchmark-dev-guide.md` — shows only the intended restricted/write-enabled contracts.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
# Milestone Work Log
|
||||
|
||||
> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.
|
||||
|
||||
| seq | time | event | task | loop | role | attempt | model | result | locator |
|
||||
|---:|---|---|---|---:|---|---:|---|---|---|
|
||||
| 1 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json |
|
||||
| 2 | 26-08-12 03:28:04 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json |
|
||||
| 3 | 26-08-12 03:36:10 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__worker__a00/locator.json |
|
||||
| 4 | 26-08-12 03:36:10 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json |
|
||||
| 5 | 26-08-12 03:38:25 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/PLAN-cloud-G09.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T032804+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__worker__a00/locator.json |
|
||||
| 6 | 26-08-12 03:38:25 KST | START | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json |
|
||||
| 7 | 26-08-12 03:45:17 KST | FINISH | m-iop-one-shot-agent-model-comparison/02_route_preflight_contract/CODE_REVIEW-cloud-G09.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033825+0900__m-iop-one-shot-agent-model-comparison__02_route_preflight_contract__p1__review__a00/locator.json |
|
||||
| 8 | 26-08-12 03:46:32 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T033610+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p1__review__a00/locator.json |
|
||||
| 9 | 26-08-12 03:46:32 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json |
|
||||
| 10 | 26-08-12 03:50:49 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/PLAN-cloud-G05.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T034632+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__worker__a00/locator.json |
|
||||
| 11 | 26-08-12 03:50:50 KST | START | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json |
|
||||
| 12 | 26-08-12 03:57:29 KST | FINISH | m-iop-one-shot-agent-model-comparison/01_execution_order_contract/CODE_REVIEW-cloud-G05.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035050+0900__m-iop-one-shot-agent-model-comparison__01_execution_order_contract__p2__review__a00/locator.json |
|
||||
| 13 | 26-08-12 03:57:29 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json |
|
||||
| 14 | 26-08-12 04:06:20 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/PLAN-local-G08.md | 1 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T035729+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__worker__a00/locator.json |
|
||||
| 15 | 26-08-12 04:06:21 KST | START | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json |
|
||||
| 16 | 26-08-12 04:14:46 KST | FINISH | m-iop-one-shot-agent-model-comparison/03+01_rubric_version_contract/CODE_REVIEW-cloud-G08.md | 1 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T040621+0900__m-iop-one-shot-agent-model-comparison__03__01_rubric_version_contract__p1__review__a00/locator.json |
|
||||
| 17 | 26-08-12 04:14:46 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json |
|
||||
| 18 | 26-08-12 04:24:01 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/PLAN-local-G06.md | 2 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T041446+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__worker__a00/locator.json |
|
||||
| 19 | 26-08-12 04:24:01 KST | START | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json |
|
||||
| 20 | 26-08-12 04:31:38 KST | FINISH | m-iop-one-shot-agent-model-comparison/04+02,03_locked_benchmark_manifest/CODE_REVIEW-cloud-G06.md | 2 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T042401+0900__m-iop-one-shot-agent-model-comparison__04__02__03_locked_benchmark_manifest__p2__review__a00/locator.json |
|
||||
| 21 | 26-08-12 04:31:38 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json |
|
||||
| 22 | 26-08-12 04:34:59 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043138+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__worker__a00/locator.json |
|
||||
| 23 | 26-08-12 04:34:59 KST | START | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | running | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json |
|
||||
| 24 | 26-08-12 04:41:22 KST | FINISH | m-iop-one-shot-agent-model-comparison/05+04_readiness_preflight/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol | succeeded:0 | /config/workspace/iop-s0/.git/agent-task-dispatcher/runs/20260812T043459+0900__m-iop-one-shot-agent-model-comparison__05__04_readiness_preflight__p0__review__a00/locator.json |
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST]** Start only after child 07 PASS. Fill implementation evidence, leave active files in place, and report ready for review. Finalization is review-only.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-12
|
||||
task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun, plan=0, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Preserve `run-20260812T044800Z-412e05fc80df` and `run-20260812T050756Z-18293db0b83f`; do not reuse them.
|
||||
- Dependency: archived PASS `07+06_caller_write_contracts/complete.log`.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| TEST-1 runtime recovery | [ ] |
|
||||
| TEST-2 ready=9 preflight | [ ] |
|
||||
| TEST-3 one scored run | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source.
|
||||
- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9.
|
||||
- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
- [ ] Append verdict/routing signals after fresh verification.
|
||||
- [ ] Verify no direct caller/provider invocation or scored retry occurred.
|
||||
- [ ] Archive active files and write/move `complete.log` only on PASS.
|
||||
- [ ] Preserve milestone-task metadata.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Record actual deviations or state none._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Record actual decisions._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- Runtime recovery used the existing managed deployment and did not expose secrets.
|
||||
- Preflight is ready=9 before run.
|
||||
- Exactly one new scored run id exists, with 9 success and all web gates passing.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Dependency and runtime recovery
|
||||
|
||||
_Paste exact commands/output with secret values omitted by command construction._
|
||||
|
||||
### Public preflight
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
|
||||
_Paste actual stdout/stderr and exit status._
|
||||
|
||||
### Public scored run
|
||||
|
||||
Command: `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
|
||||
_Paste actual stdout/stderr and exit status._
|
||||
|
||||
## Section Ownership
|
||||
|
||||
Implementation status/checklist/evidence is implementation-owned. Review checklist and verdict are review-only.
|
||||
|
|
@ -0,0 +1,137 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun plan=0 tag=TEST milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid -->
|
||||
|
||||
# Dev runtime recovery and scored comparison rerun
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Start only after child 07 has archived PASS evidence. Restore the existing managed dev runtime, use only the public benchmark CLI, store emitted ids in the exact pointer files, and fill `CODE_REVIEW-cloud-G10.md`. Do not invoke callers/providers directly, reuse a scored run id, overwrite evidence, ask the user, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The adapter fixes are not proven until the managed dev endpoint is available and a fresh preflight reports all nine cells ready. A new append-only scored run must then prove every caller can create the required website artifacts.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Old terminal run remains at `agent-test/runs/bench-02/run-20260812T044800Z-412e05fc80df` and must not be reused or overwritten.
|
||||
- A later read-only preflight created `run-20260812T050756Z-18293db0b83f` with ready=0 because the current shell had no runtime endpoint variables and no local managed listener.
|
||||
- Child 07 must supply the corrected caller contracts before this plan begins.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `scripts/agent_comparison_benchmark.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- `docs/agent-comparison-benchmark-dev-guide.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/local/testing-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
SDD is `READY`; milestone-task ids are all five comparison tasks. S04-S08 and Evidence Map rows require C01-C09 event/timing/usage/workspace evidence, so PASS requires ready=9 followed by exactly one fresh scored run whose nine cells pass terminal and web gates.
|
||||
|
||||
### Verification Context
|
||||
|
||||
Declared runner is this local checkout; managed dev runtime inventory resolves to SSH `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, branch `dev`, CP status port 18001 and edge ports 18082-18084/19093. Current local preflight is blocked by unavailable endpoint environment. Before mutation, inspect remote branch/HEAD/dirty state, process/service ownership, config, listeners, and health without exposing secret values. Restore the existing deployed runtime/config rather than changing IOP source. The public CLI is the only allowed benchmark/provider execution surface.
|
||||
|
||||
### External Verification Preflight
|
||||
|
||||
- Runner/workdir: `/config/workspace/iop-s0`, branch `feature/iop-one-shot-agent-model-comparison`, dirty feature worktree; caller binaries are installed for Linux aarch64.
|
||||
- Remote: `toki@toki-labs.com:/Users/toki/agent-work/iop-dev`; exact current HEAD/dirty/service state must be captured before restart.
|
||||
- Credentials/CA: protected files under `token/`; never print values. Export only in the benchmark command process according to the dev guide.
|
||||
- Resume gate: remote health succeeds and public CLI preflight emits `ready=9`, `registration_required=0`, `implementation_gap=0`.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
Local tests cannot prove external endpoint health, caller installation, TLS, or generated files. This plan closes those gaps with preflight and one fresh scored run.
|
||||
|
||||
### Symbol References
|
||||
|
||||
None; this packet changes no production symbols.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
This is dependent child 08. Its stable contract is external full-cycle evidence and its PASS oracle is a fresh all-success run. Directory dependency `+07` requires archived `07+06_caller_write_contracts/complete.log`; implementation must not start while it is missing.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
Do not deploy new IOP source, alter routes/models/task content, invoke direct callers/providers, retry a scored run, score subjective quality, or modify roadmap state. If the new run exposes a new code defect, preserve evidence and return it to review for a concrete follow-up.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- evaluation_mode: `first-pass`; finalizer: `finalize-task-policy.sh pair`
|
||||
- build closures all true; scores `1/2/2/2/2` => G09; base/route `grade-boundary`; filename `PLAN-cloud-G09.md`
|
||||
- review closures all true; scores `2/2/2/2/2` => G10; route `official-review`; filename `CODE_REVIEW-cloud-G10.md`
|
||||
- large_indivisible_context=false; positive risks=`temporal_state,boundary_contract,variant_product` (3); review_rework_count=0; evidence_integrity_failure=false; capability gap absent.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Prove child 07 PASS, inspect and restore the existing managed dev runtime without changing source.
|
||||
- [ ] [TEST-2] Run a fresh public CLI preflight and record its emitted id with ready=9.
|
||||
- [ ] [TEST-3] Run exactly one fresh public CLI scored matrix and record its run id and nine-cell terminal/web results.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Runtime recovery
|
||||
|
||||
**Problem:** The current public preflight reports ready=0 because the declared managed dev endpoint/config is unavailable to the runner.
|
||||
|
||||
**Solution:** Verify the child-07 completion log, inspect remote git/runtime state read-only, then restart/rebuild only the existing deployed managed services needed to restore health. Bind local command environment from protected `token/` files without logging values.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md`: record remote preflight/recovery evidence.
|
||||
|
||||
**Test Strategy:** Use remote health/listener and public CLI preflight only; no direct provider call.
|
||||
|
||||
**Verification:** Remote health succeeds and the public preflight gate can execute.
|
||||
|
||||
### [TEST-2] Fresh ready=9 preflight
|
||||
|
||||
**Problem:** A scored run is forbidden until every matrix cell has proven connectivity/effective binding.
|
||||
|
||||
**Solution:** Execute the manifest preflight once runtime inputs are present and store only the emitted id in `preflight_id.log`.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log`: exact emitted preflight id.
|
||||
|
||||
**Test Strategy:** Assert summary `ready=9 registration_required=0 implementation_gap=0`.
|
||||
|
||||
**Verification:** `python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0.
|
||||
|
||||
### [TEST-3] One scored 9-cell rerun
|
||||
|
||||
**Problem:** The old run proves process termination but not successful benchmark artifacts.
|
||||
|
||||
**Solution:** Execute exactly one new public `run`, store its id, and inspect immutable result/workspace validation evidence. Do not retry that id.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log`: exact emitted scored run id.
|
||||
|
||||
**Test Strategy:** Require 9 success, 0 failed/interrupted/running and all required HTML/CSS/JS workspace gates PASS.
|
||||
|
||||
**Verification:** `python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` exits 0 and emitted run evidence satisfies the stated counts.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
`07+06_caller_write_contracts` must first produce an archived `complete.log`. Then perform TEST-1, TEST-2, and TEST-3 in order. No dependency beyond index 07 is implied.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Item |
|
||||
|---|---|
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/CODE_REVIEW-cloud-G10.md` | TEST-1, TEST-2, TEST-3 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/preflight_id.log` | TEST-2 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/08+07_comparison_rerun/run_id.log` | TEST-3 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
1. `test -f agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/07+06_caller_write_contracts/complete.log` — exits 0.
|
||||
2. Public preflight command above — exits 0 with ready=9.
|
||||
3. Public run command above — executes exactly once and exits 0 with nine passing terminal/web results.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
|
@ -16,7 +16,7 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest)
|
|||
}
|
||||
for key := range raw {
|
||||
switch key {
|
||||
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning":
|
||||
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body":
|
||||
default:
|
||||
return fmt.Errorf("%s is not supported for /v1/chat/completions", key)
|
||||
}
|
||||
|
|
@ -40,6 +40,9 @@ func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest)
|
|||
if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) {
|
||||
return fmt.Errorf("top_p must be between 0 and 1")
|
||||
}
|
||||
if req.TopK != nil && *req.TopK <= 0 {
|
||||
return fmt.Errorf("top_k must be greater than zero")
|
||||
}
|
||||
if err := validateThinkControl(req); err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -83,7 +86,7 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe
|
|||
}
|
||||
for key := range raw {
|
||||
switch key {
|
||||
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning":
|
||||
case "model", "messages", "stream", "metadata", "max_tokens", "max_completion_tokens", "temperature", "top_p", "top_k", "presence_penalty", "frequency_penalty", "seed", "stop", "response_format", "tools", "tool_choice", "parallel_tool_calls", "stream_options", "store", "think", "reasoning_effort", "thinking_token_budget", "include_reasoning", "extra_body":
|
||||
default:
|
||||
// Unknown fields are tolerated for provider-pool passthrough.
|
||||
}
|
||||
|
|
@ -108,6 +111,9 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe
|
|||
if req.TopP != nil && (*req.TopP < 0 || *req.TopP > 1) {
|
||||
return fmt.Errorf("top_p must be between 0 and 1")
|
||||
}
|
||||
if req.TopK != nil && *req.TopK <= 0 {
|
||||
return fmt.Errorf("top_k must be greater than zero")
|
||||
}
|
||||
if err := validateThinkControl(req); err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -102,6 +102,20 @@ func (s *Server) handleChatCompletions(w http.ResponseWriter, r *http.Request) {
|
|||
writeError(w, http.StatusBadRequest, "invalid_request_error", "messages are required")
|
||||
return
|
||||
}
|
||||
if dispatch.SingleRequest != nil {
|
||||
capability, ok := s.service.(singleRequestService)
|
||||
if !ok {
|
||||
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
|
||||
return
|
||||
}
|
||||
recordSingleRequestIngress()
|
||||
if req.Stream {
|
||||
s.handleChatSingleRequestStream(w, r, capability, dispatch, rawBody)
|
||||
} else {
|
||||
s.handleChatSingleRequest(w, r, capability, dispatch, rawBody)
|
||||
}
|
||||
return
|
||||
}
|
||||
outputPolicy := s.resolveOutputPolicy(basePrompt)
|
||||
// The mutable model-catalog generation policy is applied here, at the single
|
||||
// ingress point, before req is frozen into the dispatch context. Every stage
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ type chatCompletionRequest struct {
|
|||
MaxCompletionTokens *int `json:"max_completion_tokens,omitempty"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
TopP *float64 `json:"top_p,omitempty"`
|
||||
TopK *int `json:"top_k,omitempty"`
|
||||
PresencePenalty *float64 `json:"presence_penalty,omitempty"`
|
||||
FrequencyPenalty *float64 `json:"frequency_penalty,omitempty"`
|
||||
Seed *int `json:"seed,omitempty"`
|
||||
|
|
@ -28,6 +29,7 @@ type chatCompletionRequest struct {
|
|||
ReasoningEffort *string `json:"reasoning_effort,omitempty"`
|
||||
ThinkingTokenBudget *int `json:"thinking_token_budget,omitempty"`
|
||||
IncludeReasoning *bool `json:"include_reasoning,omitempty"`
|
||||
ExtraBody any `json:"extra_body,omitempty"`
|
||||
}
|
||||
|
||||
type chatMessage struct {
|
||||
|
|
@ -76,6 +78,7 @@ func (req chatCompletionRequest) runInput(prompt string, messages []chatMessage,
|
|||
}
|
||||
setOptionFloat(options, "temperature", req.Temperature)
|
||||
setOptionFloat(options, "top_p", req.TopP)
|
||||
setOptionInt(options, "top_k", req.TopK)
|
||||
setOptionFloat(options, "presence_penalty", req.PresencePenalty)
|
||||
setOptionFloat(options, "frequency_penalty", req.FrequencyPenalty)
|
||||
setOptionInt(options, "seed", req.Seed)
|
||||
|
|
|
|||
310
apps/edge/internal/openai/gemini_bridge.go
Normal file
310
apps/edge/internal/openai/gemini_bridge.go
Normal file
|
|
@ -0,0 +1,310 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type geminiBridgeResponseWriter struct {
|
||||
target http.ResponseWriter
|
||||
header http.Header
|
||||
status int
|
||||
committed bool
|
||||
buffer bytes.Buffer
|
||||
stream *geminiBridgeStream
|
||||
}
|
||||
|
||||
func newGeminiBridgeResponseWriter(target http.ResponseWriter, model string) *geminiBridgeResponseWriter {
|
||||
return &geminiBridgeResponseWriter{
|
||||
target: target, header: make(http.Header), stream: newGeminiBridgeStream(target, model),
|
||||
}
|
||||
}
|
||||
|
||||
func (w *geminiBridgeResponseWriter) Header() http.Header { return w.header }
|
||||
|
||||
func (w *geminiBridgeResponseWriter) WriteHeader(status int) {
|
||||
if w.status == 0 {
|
||||
w.status = status
|
||||
}
|
||||
}
|
||||
|
||||
func (w *geminiBridgeResponseWriter) Write(payload []byte) (int, error) {
|
||||
if w.status == 0 {
|
||||
w.status = http.StatusOK
|
||||
}
|
||||
if w.status >= http.StatusBadRequest {
|
||||
return w.buffer.Write(payload)
|
||||
}
|
||||
w.commit()
|
||||
if err := w.stream.Feed(payload); err != nil {
|
||||
_ = w.stream.Error("upstream stream could not be translated")
|
||||
return 0, err
|
||||
}
|
||||
return len(payload), nil
|
||||
}
|
||||
|
||||
func (w *geminiBridgeResponseWriter) Flush() {
|
||||
if w.status == 0 {
|
||||
w.status = http.StatusOK
|
||||
}
|
||||
if w.status < http.StatusBadRequest {
|
||||
w.commit()
|
||||
}
|
||||
if flusher, ok := w.target.(http.Flusher); ok {
|
||||
flusher.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
func (w *geminiBridgeResponseWriter) commit() {
|
||||
if w.committed {
|
||||
return
|
||||
}
|
||||
w.target.Header().Set("Content-Type", "text/event-stream")
|
||||
w.target.Header().Set("Cache-Control", "no-cache")
|
||||
w.target.Header().Del("Content-Length")
|
||||
w.target.WriteHeader(http.StatusOK)
|
||||
w.committed = true
|
||||
}
|
||||
|
||||
func (w *geminiBridgeResponseWriter) Finish() {
|
||||
if w.status == 0 {
|
||||
writeGeminiError(w.target, http.StatusBadGateway, "UNAVAILABLE", "runtime request failed")
|
||||
return
|
||||
}
|
||||
if w.status >= http.StatusBadRequest {
|
||||
status, code := geminiHTTPError(w.status)
|
||||
writeGeminiError(w.target, status, code, geminiSafeErrorMessage(status))
|
||||
return
|
||||
}
|
||||
w.commit()
|
||||
if err := w.stream.Finish(); err != nil {
|
||||
_ = w.stream.Error("upstream stream could not be translated")
|
||||
}
|
||||
if flusher, ok := w.target.(http.Flusher); ok {
|
||||
flusher.Flush()
|
||||
}
|
||||
}
|
||||
|
||||
func geminiHTTPError(status int) (int, string) {
|
||||
switch {
|
||||
case status == http.StatusUnauthorized || status == http.StatusForbidden:
|
||||
return http.StatusUnauthorized, "UNAUTHENTICATED"
|
||||
case status >= 400 && status < 500:
|
||||
return http.StatusBadRequest, "INVALID_ARGUMENT"
|
||||
default:
|
||||
return http.StatusBadGateway, "UNAVAILABLE"
|
||||
}
|
||||
}
|
||||
|
||||
func geminiSafeErrorMessage(status int) string {
|
||||
if status == http.StatusUnauthorized {
|
||||
return "authentication failed"
|
||||
}
|
||||
if status == http.StatusBadRequest {
|
||||
return "request is invalid"
|
||||
}
|
||||
return "runtime request failed"
|
||||
}
|
||||
|
||||
type geminiBridgeToolState struct {
|
||||
name string
|
||||
thoughtSignature string
|
||||
arguments strings.Builder
|
||||
}
|
||||
|
||||
type geminiBridgeStream struct {
|
||||
w http.ResponseWriter
|
||||
model string
|
||||
pendingSSE []byte
|
||||
tools map[int]*geminiBridgeToolState
|
||||
finish string
|
||||
usage map[string]int
|
||||
stopped bool
|
||||
errored bool
|
||||
}
|
||||
|
||||
func newGeminiBridgeStream(w http.ResponseWriter, model string) *geminiBridgeStream {
|
||||
return &geminiBridgeStream{w: w, model: model, tools: make(map[int]*geminiBridgeToolState), usage: make(map[string]int)}
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) Feed(chunk []byte) error {
|
||||
if s.stopped {
|
||||
return nil
|
||||
}
|
||||
s.pendingSSE = append(s.pendingSSE, chunk...)
|
||||
s.pendingSSE = bytes.ReplaceAll(s.pendingSSE, []byte("\r\n"), []byte("\n"))
|
||||
for {
|
||||
index := bytes.Index(s.pendingSSE, []byte("\n\n"))
|
||||
if index < 0 {
|
||||
return nil
|
||||
}
|
||||
event := append([]byte(nil), s.pendingSSE[:index]...)
|
||||
s.pendingSSE = s.pendingSSE[index+2:]
|
||||
if err := s.consumeSSEEvent(event); err != nil {
|
||||
return err
|
||||
}
|
||||
if s.stopped {
|
||||
s.pendingSSE = nil
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) consumeSSEEvent(event []byte) error {
|
||||
var lines [][]byte
|
||||
for _, line := range bytes.Split(event, []byte("\n")) {
|
||||
line = bytes.TrimSpace(line)
|
||||
if bytes.HasPrefix(line, []byte("data:")) {
|
||||
lines = append(lines, bytes.TrimSpace(bytes.TrimPrefix(line, []byte("data:"))))
|
||||
}
|
||||
}
|
||||
if len(lines) == 0 {
|
||||
return nil
|
||||
}
|
||||
payload := bytes.Join(lines, []byte("\n"))
|
||||
if bytes.Equal(payload, []byte("[DONE]")) {
|
||||
return s.Finish()
|
||||
}
|
||||
var chunk geminiChatStreamChunk
|
||||
if err := json.Unmarshal(payload, &chunk); err != nil {
|
||||
return fmt.Errorf("decode Chat SSE: %w", err)
|
||||
}
|
||||
if chunk.Error != nil {
|
||||
return s.Error("upstream request failed")
|
||||
}
|
||||
if chunk.Usage != nil {
|
||||
setGeminiUsage(s.usage, "promptTokenCount", chunk.Usage.PromptTokens)
|
||||
setGeminiUsage(s.usage, "candidatesTokenCount", chunk.Usage.CompletionTokens)
|
||||
setGeminiUsage(s.usage, "totalTokenCount", chunk.Usage.TotalTokens)
|
||||
if chunk.Usage.PromptTokensDetails != nil {
|
||||
setGeminiUsage(s.usage, "cachedContentTokenCount", chunk.Usage.PromptTokensDetails.CachedTokens)
|
||||
}
|
||||
if chunk.Usage.CompletionTokensDetails != nil {
|
||||
setGeminiUsage(s.usage, "thoughtsTokenCount", chunk.Usage.CompletionTokensDetails.ReasoningTokens)
|
||||
}
|
||||
}
|
||||
for _, choice := range chunk.Choices {
|
||||
reasoning := choice.Delta.ReasoningContent
|
||||
if reasoning == "" {
|
||||
reasoning = choice.Delta.Reasoning
|
||||
}
|
||||
if reasoning != "" {
|
||||
if err := s.emitParts([]any{map[string]any{"text": reasoning, "thought": true}}, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if choice.Delta.Content != "" {
|
||||
if err := s.emitParts([]any{map[string]any{"text": choice.Delta.Content}}, ""); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, delta := range choice.Delta.ToolCalls {
|
||||
state := s.tools[delta.Index]
|
||||
if state == nil {
|
||||
state = &geminiBridgeToolState{}
|
||||
s.tools[delta.Index] = state
|
||||
}
|
||||
if delta.Function.Name != "" {
|
||||
state.name = delta.Function.Name
|
||||
}
|
||||
if delta.ExtraContent.Google != nil && delta.ExtraContent.Google.ThoughtSignature != "" {
|
||||
state.thoughtSignature = delta.ExtraContent.Google.ThoughtSignature
|
||||
}
|
||||
if state.arguments.Len()+len(delta.Function.Arguments) > geminiToolArgumentLimit {
|
||||
return fmt.Errorf("tool arguments exceed limit")
|
||||
}
|
||||
state.arguments.WriteString(delta.Function.Arguments)
|
||||
}
|
||||
if choice.FinishReason != nil {
|
||||
s.finish = *choice.FinishReason
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func setGeminiUsage(target map[string]int, key string, value *int) {
|
||||
if value != nil {
|
||||
target[key] = *value
|
||||
}
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) emitParts(parts []any, finish string) error {
|
||||
candidate := map[string]any{"content": map[string]any{"role": "model", "parts": parts}}
|
||||
if finish != "" {
|
||||
candidate["finishReason"] = finish
|
||||
}
|
||||
payload := map[string]any{"candidates": []any{candidate}, "modelVersion": s.model}
|
||||
if len(s.usage) > 0 {
|
||||
payload["usageMetadata"] = s.usage
|
||||
}
|
||||
return writeGeminiSSE(s.w, payload)
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) emitTools() error {
|
||||
if len(s.tools) == 0 {
|
||||
return nil
|
||||
}
|
||||
indices := make([]int, 0, len(s.tools))
|
||||
for index := range s.tools {
|
||||
indices = append(indices, index)
|
||||
}
|
||||
sort.Ints(indices)
|
||||
parts := make([]any, 0, len(indices))
|
||||
for _, index := range indices {
|
||||
state := s.tools[index]
|
||||
if !geminiPathToken.MatchString(state.name) {
|
||||
return fmt.Errorf("tool name is invalid")
|
||||
}
|
||||
var args map[string]any
|
||||
if json.Unmarshal([]byte(state.arguments.String()), &args) != nil {
|
||||
return fmt.Errorf("tool arguments are invalid")
|
||||
}
|
||||
part := map[string]any{"functionCall": map[string]any{"name": state.name, "args": args}}
|
||||
if state.thoughtSignature != "" {
|
||||
part["thoughtSignature"] = state.thoughtSignature
|
||||
}
|
||||
parts = append(parts, part)
|
||||
}
|
||||
return s.emitParts(parts, "")
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) Finish() error {
|
||||
if s.stopped {
|
||||
return nil
|
||||
}
|
||||
if len(bytes.TrimSpace(s.pendingSSE)) > 0 {
|
||||
return fmt.Errorf("truncated Chat SSE")
|
||||
}
|
||||
if err := s.emitTools(); err != nil {
|
||||
return err
|
||||
}
|
||||
finish := "STOP"
|
||||
if s.finish == "length" {
|
||||
finish = "MAX_TOKENS"
|
||||
}
|
||||
s.stopped = true
|
||||
return s.emitParts([]any{}, finish)
|
||||
}
|
||||
|
||||
func (s *geminiBridgeStream) Error(message string) error {
|
||||
if s.errored || s.stopped {
|
||||
return nil
|
||||
}
|
||||
s.errored, s.stopped = true, true
|
||||
return writeGeminiSSE(s.w, geminiErrorResponse{Error: geminiErrorBody{
|
||||
Code: http.StatusBadGateway, Message: message, Status: "UNAVAILABLE",
|
||||
}})
|
||||
}
|
||||
|
||||
func writeGeminiSSE(w http.ResponseWriter, value any) error {
|
||||
payload, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = fmt.Fprintf(w, "data: %s\n\n", payload)
|
||||
return err
|
||||
}
|
||||
375
apps/edge/internal/openai/gemini_handler.go
Normal file
375
apps/edge/internal/openai/gemini_handler.go
Normal file
|
|
@ -0,0 +1,375 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var geminiPathToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`)
|
||||
|
||||
func isGeminiRequest(r *http.Request) bool {
|
||||
return r != nil && strings.HasPrefix(r.URL.Path, geminiPathPrefix)
|
||||
}
|
||||
|
||||
func writeGeminiError(w http.ResponseWriter, status int, code, message string) {
|
||||
writeJSON(w, status, geminiErrorResponse{Error: geminiErrorBody{
|
||||
Code: status, Message: message, Status: code,
|
||||
}})
|
||||
}
|
||||
|
||||
func (s *Server) handleGeminiStreamGenerateContent(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost {
|
||||
writeGeminiError(w, http.StatusMethodNotAllowed, "INVALID_ARGUMENT", "method not allowed")
|
||||
return
|
||||
}
|
||||
routeID, callerModel, err := parseGeminiStreamPath(r)
|
||||
if err != nil {
|
||||
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request path is invalid")
|
||||
return
|
||||
}
|
||||
defer r.Body.Close()
|
||||
body, err := readOpenAIIngressBody(w, r, s.maxIngressSnapshotBytes())
|
||||
if err != nil {
|
||||
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid")
|
||||
return
|
||||
}
|
||||
chatBody, err := prepareGeminiChatBridge(body, routeID)
|
||||
if err != nil {
|
||||
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid")
|
||||
return
|
||||
}
|
||||
|
||||
internal := r.Clone(r.Context())
|
||||
internal.URL.Path = "/v1/chat/completions"
|
||||
internal.URL.RawPath = ""
|
||||
internal.URL.RawQuery = ""
|
||||
internal.RequestURI = "/v1/chat/completions"
|
||||
internal.Body = io.NopCloser(bytes.NewReader(chatBody))
|
||||
internal.ContentLength = int64(len(chatBody))
|
||||
internal.Header = r.Header.Clone()
|
||||
internal.Header.Del("Authorization")
|
||||
internal.Header.Del("X-Goog-Api-Key")
|
||||
internal.Header.Set("Content-Type", "application/json")
|
||||
|
||||
bridge := newGeminiBridgeResponseWriter(w, callerModel)
|
||||
s.handleChatCompletions(bridge, internal)
|
||||
bridge.Finish()
|
||||
}
|
||||
|
||||
func parseGeminiStreamPath(r *http.Request) (string, string, error) {
|
||||
if r == nil || r.URL == nil {
|
||||
return "", "", fmt.Errorf("missing URL")
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(r.URL.Path, geminiPathPrefix), "/")
|
||||
if len(parts) != 4 || parts[1] != "v1beta" || parts[2] != "models" {
|
||||
return "", "", fmt.Errorf("unexpected path")
|
||||
}
|
||||
const suffix = ":streamGenerateContent"
|
||||
if !strings.HasSuffix(parts[3], suffix) {
|
||||
return "", "", fmt.Errorf("unexpected method")
|
||||
}
|
||||
routeID := parts[0]
|
||||
callerModel := strings.TrimSuffix(parts[3], suffix)
|
||||
if !geminiPathToken.MatchString(routeID) || !geminiPathToken.MatchString(callerModel) {
|
||||
return "", "", fmt.Errorf("invalid path token")
|
||||
}
|
||||
query := r.URL.Query()
|
||||
if len(query) != 1 || len(query["alt"]) != 1 || query.Get("alt") != "sse" {
|
||||
return "", "", fmt.Errorf("alt=sse is required")
|
||||
}
|
||||
return routeID, callerModel, nil
|
||||
}
|
||||
|
||||
func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) {
|
||||
if err := validateJSONMembers(body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||
decoder.DisallowUnknownFields()
|
||||
var req geminiRequest
|
||||
if err := decoder.Decode(&req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := requireGeminiJSONEOF(decoder); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(req.Contents) == 0 {
|
||||
return nil, fmt.Errorf("contents are required")
|
||||
}
|
||||
chat := map[string]any{
|
||||
"model": routeID, "stream": true,
|
||||
"stream_options": map[string]any{"include_usage": true},
|
||||
}
|
||||
messages := make([]map[string]any, 0, len(req.Contents)+1)
|
||||
if req.SystemInstruction != nil {
|
||||
text, err := geminiTextOnly(*req.SystemInstruction)
|
||||
if err != nil || strings.TrimSpace(text) == "" {
|
||||
return nil, fmt.Errorf("systemInstruction is invalid")
|
||||
}
|
||||
messages = append(messages, map[string]any{"role": "system", "content": text})
|
||||
}
|
||||
pendingCalls := make(map[string][]string)
|
||||
for contentIndex, content := range req.Contents {
|
||||
converted, err := geminiContentToChat(content, contentIndex, pendingCalls)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
messages = append(messages, converted...)
|
||||
}
|
||||
chat["messages"] = messages
|
||||
if config := req.GenerationConfig; config != nil {
|
||||
if config.CandidateCount != nil && *config.CandidateCount != 1 {
|
||||
return nil, fmt.Errorf("candidateCount must be one")
|
||||
}
|
||||
if config.MaxOutputTokens != nil {
|
||||
if *config.MaxOutputTokens <= 0 {
|
||||
return nil, fmt.Errorf("maxOutputTokens must be positive")
|
||||
}
|
||||
chat["max_tokens"] = *config.MaxOutputTokens
|
||||
}
|
||||
if len(config.StopSequences) > 0 {
|
||||
chat["stop"] = config.StopSequences
|
||||
}
|
||||
if config.Temperature != nil {
|
||||
if *config.Temperature < 0 || *config.Temperature > 2 {
|
||||
return nil, fmt.Errorf("temperature is invalid")
|
||||
}
|
||||
}
|
||||
if config.TopK != nil {
|
||||
if *config.TopK <= 0 {
|
||||
return nil, fmt.Errorf("topK is invalid")
|
||||
}
|
||||
}
|
||||
if config.TopP != nil {
|
||||
if *config.TopP < 0 || *config.TopP > 1 {
|
||||
return nil, fmt.Errorf("topP is invalid")
|
||||
}
|
||||
}
|
||||
if thinking := config.ThinkingConfig; thinking != nil {
|
||||
googleThinking := make(map[string]any)
|
||||
if thinking.IncludeThoughts != nil {
|
||||
googleThinking["include_thoughts"] = *thinking.IncludeThoughts
|
||||
}
|
||||
if thinking.ThinkingBudget != nil {
|
||||
if *thinking.ThinkingBudget < -1 {
|
||||
return nil, fmt.Errorf("thinkingBudget is invalid")
|
||||
}
|
||||
googleThinking["thinking_budget"] = *thinking.ThinkingBudget
|
||||
}
|
||||
if len(googleThinking) > 0 {
|
||||
// Gemini's OpenAI-compatible endpoint accepts native options only
|
||||
// below extra_body.google; generic think/include_reasoning fields
|
||||
// are rejected by that endpoint.
|
||||
chat["extra_body"] = map[string]any{
|
||||
"google": map[string]any{"thinking_config": googleThinking},
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(req.Tools) > 0 {
|
||||
tools := make([]map[string]any, 0)
|
||||
for _, group := range req.Tools {
|
||||
if len(group.FunctionDeclarations) == 0 {
|
||||
return nil, fmt.Errorf("functionDeclarations are required")
|
||||
}
|
||||
for _, declaration := range group.FunctionDeclarations {
|
||||
if !geminiPathToken.MatchString(declaration.Name) || len(declaration.ParametersJSONSchema) == 0 {
|
||||
return nil, fmt.Errorf("function declaration is invalid")
|
||||
}
|
||||
var schema map[string]any
|
||||
if json.Unmarshal(declaration.ParametersJSONSchema, &schema) != nil {
|
||||
return nil, fmt.Errorf("function schema is invalid")
|
||||
}
|
||||
function := map[string]any{"name": declaration.Name, "parameters": schema}
|
||||
if declaration.Description != "" {
|
||||
function["description"] = declaration.Description
|
||||
}
|
||||
tools = append(tools, map[string]any{"type": "function", "function": function})
|
||||
}
|
||||
}
|
||||
chat["tools"] = tools
|
||||
}
|
||||
if req.ToolConfig != nil && req.ToolConfig.FunctionCallingConfig != nil {
|
||||
switch strings.ToUpper(req.ToolConfig.FunctionCallingConfig.Mode) {
|
||||
case "", "AUTO":
|
||||
chat["tool_choice"] = "auto"
|
||||
case "ANY":
|
||||
chat["tool_choice"] = "required"
|
||||
case "NONE":
|
||||
chat["tool_choice"] = "none"
|
||||
default:
|
||||
return nil, fmt.Errorf("function calling mode is invalid")
|
||||
}
|
||||
}
|
||||
return json.Marshal(chat)
|
||||
}
|
||||
|
||||
func geminiTextOnly(content geminiContent) (string, error) {
|
||||
// Gemini represents systemInstruction as Content and official agy 1.1.12
|
||||
// labels that Content with the API-native "user" role.
|
||||
if content.Role != "" && content.Role != "user" {
|
||||
return "", fmt.Errorf("role is invalid")
|
||||
}
|
||||
texts := make([]string, 0, len(content.Parts))
|
||||
for _, part := range content.Parts {
|
||||
if part.Text == nil || part.FunctionCall != nil || part.FunctionResponse != nil || part.Thought || part.ThoughtSignature != "" {
|
||||
return "", fmt.Errorf("only text is supported")
|
||||
}
|
||||
texts = append(texts, *part.Text)
|
||||
}
|
||||
return strings.Join(texts, "\n"), nil
|
||||
}
|
||||
|
||||
func geminiContentToChat(content geminiContent, contentIndex int, pending map[string][]string) ([]map[string]any, error) {
|
||||
role := strings.ToLower(strings.TrimSpace(content.Role))
|
||||
if role != "user" && role != "model" {
|
||||
return nil, fmt.Errorf("content role is invalid")
|
||||
}
|
||||
if len(content.Parts) == 0 {
|
||||
return nil, fmt.Errorf("content parts are required")
|
||||
}
|
||||
var texts []string
|
||||
var reasoning []string
|
||||
var toolCalls []any
|
||||
var toolMessages []map[string]any
|
||||
for partIndex, part := range content.Parts {
|
||||
set := 0
|
||||
if part.Text != nil {
|
||||
set++
|
||||
}
|
||||
if part.FunctionCall != nil {
|
||||
set++
|
||||
}
|
||||
if part.FunctionResponse != nil {
|
||||
set++
|
||||
}
|
||||
if set != 1 {
|
||||
return nil, fmt.Errorf("content part is invalid")
|
||||
}
|
||||
if part.Text != nil {
|
||||
if role == "model" && part.Thought {
|
||||
reasoning = append(reasoning, *part.Text)
|
||||
} else if part.Thought {
|
||||
return nil, fmt.Errorf("user thought is invalid")
|
||||
} else {
|
||||
texts = append(texts, *part.Text)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if part.FunctionCall != nil {
|
||||
if role != "model" || !geminiPathToken.MatchString(part.FunctionCall.Name) {
|
||||
return nil, fmt.Errorf("functionCall is invalid")
|
||||
}
|
||||
var args map[string]any
|
||||
if json.Unmarshal(part.FunctionCall.Args, &args) != nil {
|
||||
return nil, fmt.Errorf("functionCall args are invalid")
|
||||
}
|
||||
callID := fmt.Sprintf("gemini_call_%d_%d", contentIndex, partIndex)
|
||||
pending[part.FunctionCall.Name] = append(pending[part.FunctionCall.Name], callID)
|
||||
call := map[string]any{"id": callID, "type": "function", "function": map[string]any{"name": part.FunctionCall.Name, "arguments": string(part.FunctionCall.Args)}}
|
||||
if part.ThoughtSignature != "" {
|
||||
call["extra_content"] = openAIChatThoughtSignature(part.ThoughtSignature)
|
||||
}
|
||||
toolCalls = append(toolCalls, call)
|
||||
continue
|
||||
}
|
||||
response := part.FunctionResponse
|
||||
if role != "user" || !geminiPathToken.MatchString(response.Name) {
|
||||
return nil, fmt.Errorf("functionResponse is invalid")
|
||||
}
|
||||
ids := pending[response.Name]
|
||||
if len(ids) == 0 || !json.Valid(response.Response) {
|
||||
return nil, fmt.Errorf("functionResponse has no matching call")
|
||||
}
|
||||
callID := ids[0]
|
||||
pending[response.Name] = ids[1:]
|
||||
toolMessages = append(toolMessages, map[string]any{"role": "tool", "tool_call_id": callID, "tool_name": response.Name, "content": string(response.Response)})
|
||||
}
|
||||
if role == "model" {
|
||||
message := map[string]any{"role": "assistant", "content": strings.Join(texts, "\n")}
|
||||
if len(reasoning) > 0 {
|
||||
message["reasoning_content"] = strings.Join(reasoning, "")
|
||||
}
|
||||
if len(toolCalls) > 0 {
|
||||
message["tool_calls"] = toolCalls
|
||||
}
|
||||
return []map[string]any{message}, nil
|
||||
}
|
||||
if len(texts) > 0 {
|
||||
toolMessages = append(toolMessages, map[string]any{"role": "user", "content": strings.Join(texts, "\n")})
|
||||
}
|
||||
if len(toolMessages) == 0 {
|
||||
return nil, fmt.Errorf("user content is empty")
|
||||
}
|
||||
return toolMessages, nil
|
||||
}
|
||||
|
||||
func validateJSONMembers(raw []byte) error {
|
||||
dec := json.NewDecoder(bytes.NewReader(raw))
|
||||
if err := validateGeminiJSONValue(dec); err != nil {
|
||||
return err
|
||||
}
|
||||
return requireGeminiJSONEOF(dec)
|
||||
}
|
||||
|
||||
func validateGeminiJSONValue(dec *json.Decoder) error {
|
||||
token, err := dec.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
delim, ok := token.(json.Delim)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
switch delim {
|
||||
case '{':
|
||||
seen := map[string]struct{}{}
|
||||
for dec.More() {
|
||||
keyToken, err := dec.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
key, ok := keyToken.(string)
|
||||
if !ok {
|
||||
return fmt.Errorf("object key is invalid")
|
||||
}
|
||||
if _, duplicate := seen[key]; duplicate {
|
||||
return fmt.Errorf("duplicate member")
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
if err := validateGeminiJSONValue(dec); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
closeToken, err := dec.Token()
|
||||
if err != nil || closeToken != json.Delim('}') {
|
||||
return fmt.Errorf("object is invalid")
|
||||
}
|
||||
case '[':
|
||||
for dec.More() {
|
||||
if err := validateGeminiJSONValue(dec); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
closeToken, err := dec.Token()
|
||||
if err != nil || closeToken != json.Delim(']') {
|
||||
return fmt.Errorf("array is invalid")
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("JSON delimiter is invalid")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func requireGeminiJSONEOF(dec *json.Decoder) error {
|
||||
var extra any
|
||||
if err := dec.Decode(&extra); err != io.EOF {
|
||||
return fmt.Errorf("trailing JSON value")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
173
apps/edge/internal/openai/gemini_handler_test.go
Normal file
173
apps/edge/internal/openai/gemini_handler_test.go
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"iop/packages/go/config"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
func TestGeminiIngressAuthenticatesAndStreamsThroughChatRoute(t *testing.T) {
|
||||
fake := &fakeRunService{events: bufferedRunEvents(
|
||||
&iop.RunEvent{Type: "delta", Delta: "hello"},
|
||||
&iop.RunEvent{Type: "complete", Usage: &iop.Usage{InputTokens: 3, OutputTokens: 2}, Metadata: map[string]string{"finish_reason": "length"}},
|
||||
)}
|
||||
srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama", Target: "provider-model"}, fake, nil)
|
||||
body := `{
|
||||
"systemInstruction":{"role":"user","parts":[{"text":"be concise"}]},
|
||||
"contents":[{"role":"user","parts":[{"text":"say hello"}]}],
|
||||
"generationConfig":{"candidateCount":1,"maxOutputTokens":16,"temperature":0.2,"topK":8,"topP":0.9,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}}
|
||||
}`
|
||||
req := httptest.NewRequest(http.MethodPost, "/gemini/gemini-direct/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", strings.NewReader(body))
|
||||
req.Header.Set("X-Goog-Api-Key", "iop-principal")
|
||||
w := httptest.NewRecorder()
|
||||
srv.routes().ServeHTTP(w, req)
|
||||
|
||||
if w.Code != http.StatusOK || w.Header().Get("Content-Type") != "text/event-stream" {
|
||||
t.Fatalf("status/header: %d %q body=%s", w.Code, w.Header().Get("Content-Type"), w.Body.String())
|
||||
}
|
||||
if fake.req.ModelGroupKey != "gemini-direct" || fake.req.Target != "provider-model" {
|
||||
t.Fatalf("route bypassed Chat admission: %+v", fake.req)
|
||||
}
|
||||
if !strings.Contains(fake.req.Prompt, "system: be concise") || !strings.Contains(fake.req.Prompt, "user: say hello") {
|
||||
t.Fatalf("prompt conversion mismatch: %q", fake.req.Prompt)
|
||||
}
|
||||
options := fake.req.Input["options"].(map[string]any)
|
||||
if options["max_tokens"] != 16 {
|
||||
t.Fatalf("generation config mismatch: input=%+v", fake.req.Input)
|
||||
}
|
||||
if _, exists := options["top_k"]; exists {
|
||||
t.Fatalf("deprecated Gemini sampling option reached Chat upstream: input=%+v", fake.req.Input)
|
||||
}
|
||||
response := w.Body.String()
|
||||
for _, want := range []string{`"text":"hello"`, `"finishReason":"MAX_TOKENS"`} {
|
||||
if !strings.Contains(response, want) {
|
||||
t.Fatalf("missing %s in %s", want, response)
|
||||
}
|
||||
}
|
||||
if strings.Contains(response, "iop-principal") {
|
||||
t.Fatal("principal token leaked to response")
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeminiIngressRejectsAuthenticationAndShapeBeforeDispatch(t *testing.T) {
|
||||
base := `{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}`
|
||||
for _, tc := range []struct {
|
||||
name, path, body, bearer, key string
|
||||
status int
|
||||
}{
|
||||
{"missing key", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "", "", 401},
|
||||
{"conflicting auth", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", base, "Bearer other", "iop-principal", 401},
|
||||
{"wrong query", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent", base, "", "iop-principal", 400},
|
||||
{"duplicate member", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[],"contents":[]}`, "", "iop-principal", 400},
|
||||
{"two candidates", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"candidateCount":2}}`, "", "iop-principal", 400},
|
||||
{"invalid thinking budget", "/gemini/r/v1beta/models/gemini-3.6-flash:streamGenerateContent?alt=sse", `{"contents":[{"role":"user","parts":[{"text":"hi"}]}],"generationConfig":{"thinkingConfig":{"thinkingBudget":-2}}}`, "", "iop-principal", 400},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
fake := &fakeRunService{}
|
||||
srv := NewServer(config.EdgeOpenAIConf{BearerToken: "iop-principal", Adapter: "ollama"}, fake, nil)
|
||||
req := httptest.NewRequest(http.MethodPost, tc.path, strings.NewReader(tc.body))
|
||||
if tc.bearer != "" {
|
||||
req.Header.Set("Authorization", tc.bearer)
|
||||
}
|
||||
if tc.key != "" {
|
||||
req.Header.Set("X-Goog-Api-Key", tc.key)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
srv.routes().ServeHTTP(w, req)
|
||||
if w.Code != tc.status {
|
||||
t.Fatalf("status=%d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
var failure geminiErrorResponse
|
||||
if json.Unmarshal(w.Body.Bytes(), &failure) != nil || failure.Error.Code != tc.status {
|
||||
t.Fatalf("not a Gemini error: %s", w.Body.String())
|
||||
}
|
||||
if fake.req.ModelGroupKey != "" {
|
||||
t.Fatalf("unexpected dispatch: %+v", fake.req)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeminiRequestBridgePreservesToolsAndThoughtSignature(t *testing.T) {
|
||||
body := []byte(`{
|
||||
"contents":[
|
||||
{"role":"model","parts":[{"text":"thinking","thought":true},{"functionCall":{"name":"lookup","args":{"q":"x"}},"thoughtSignature":"opaque"}]},
|
||||
{"role":"user","parts":[{"functionResponse":{"name":"lookup","response":{"value":1}}}]},
|
||||
{"role":"user","parts":[{"text":"continue"}]}
|
||||
],
|
||||
"tools":[{"functionDeclarations":[{"name":"lookup","description":"find","parametersJsonSchema":{"type":"object"}}]}],
|
||||
"toolConfig":{"functionCallingConfig":{"mode":"ANY"}}
|
||||
}`)
|
||||
converted, err := prepareGeminiChatBridge(body, "preset-gemini-hybrid")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
text := string(converted)
|
||||
for _, want := range []string{`"model":"preset-gemini-hybrid"`, `"reasoning_content":"thinking"`, `"thought_signature":"opaque"`, `"tool_call_id":"gemini_call_0_1"`, `"tool_choice":"required"`} {
|
||||
if !strings.Contains(text, want) {
|
||||
t.Fatalf("missing %s in %s", want, text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeminiRequestBridgeUsesProviderNativeThinkingEnvelope(t *testing.T) {
|
||||
converted, err := prepareGeminiChatBridge([]byte(`{
|
||||
"contents":[{"role":"user","parts":[{"text":"hello"}]}],
|
||||
"generationConfig":{"temperature":1,"topK":50,"topP":1,"thinkingConfig":{"includeThoughts":true,"thinkingBudget":-1}}
|
||||
}`), "gemini-3.6-flash")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var body map[string]any
|
||||
if json.Unmarshal(converted, &body) != nil {
|
||||
t.Fatal("converted body is invalid")
|
||||
}
|
||||
for _, forbidden := range []string{"temperature", "top_k", "top_p", "think", "include_reasoning", "thinking_token_budget"} {
|
||||
if _, exists := body[forbidden]; exists {
|
||||
t.Fatalf("unsupported field %q in converted body: %s", forbidden, converted)
|
||||
}
|
||||
}
|
||||
extra := body["extra_body"].(map[string]any)
|
||||
google := extra["google"].(map[string]any)
|
||||
thinking := google["thinking_config"].(map[string]any)
|
||||
if thinking["include_thoughts"] != true || thinking["thinking_budget"] != float64(-1) {
|
||||
t.Fatalf("thinking envelope mismatch: %+v", thinking)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) {
|
||||
w := httptest.NewRecorder()
|
||||
stream := newGeminiBridgeStream(w, "gemini-3.6-flash")
|
||||
input := "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\"}}]}}]}\n\n" +
|
||||
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"x\\\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\n" +
|
||||
"data: [DONE]\n\n"
|
||||
if err := stream.Feed([]byte(input)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Count(w.Body.String(), `"functionCall"`) != 1 || !strings.Contains(w.Body.String(), `"finishReason":"STOP"`) {
|
||||
t.Fatalf("tool terminal mismatch: %s", w.Body.String())
|
||||
}
|
||||
usageWriter := httptest.NewRecorder()
|
||||
usageStream := newGeminiBridgeStream(usageWriter, "m")
|
||||
if err := usageStream.Feed([]byte("data: {\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":2,\"total_tokens\":5,\"prompt_tokens_details\":{\"cached_tokens\":1},\"completion_tokens_details\":{\"reasoning_tokens\":1}}}\n\ndata: [DONE]\n\n")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{`"promptTokenCount":3`, `"candidatesTokenCount":2`, `"cachedContentTokenCount":1`, `"thoughtsTokenCount":1`, `"totalTokenCount":5`} {
|
||||
if !strings.Contains(usageWriter.Body.String(), want) {
|
||||
t.Fatalf("missing usage %s in %s", want, usageWriter.Body.String())
|
||||
}
|
||||
}
|
||||
oversize := newGeminiBridgeStream(httptest.NewRecorder(), "m")
|
||||
state := &geminiBridgeToolState{name: "lookup"}
|
||||
state.arguments.WriteString(strings.Repeat("x", geminiToolArgumentLimit))
|
||||
oversize.tools[0] = state
|
||||
chunk := `data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"x"}}]}}]}` + "\n\n"
|
||||
if err := oversize.Feed([]byte(chunk)); err == nil {
|
||||
t.Fatal("oversize tool arguments must fail")
|
||||
}
|
||||
}
|
||||
117
apps/edge/internal/openai/gemini_types.go
Normal file
117
apps/edge/internal/openai/gemini_types.go
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
package openai
|
||||
|
||||
import "encoding/json"
|
||||
|
||||
const (
|
||||
geminiPathPrefix = "/gemini/"
|
||||
geminiToolArgumentLimit = 1 << 20
|
||||
)
|
||||
|
||||
type geminiRequest struct {
|
||||
Contents []geminiContent `json:"contents"`
|
||||
SystemInstruction *geminiContent `json:"systemInstruction,omitempty"`
|
||||
GenerationConfig *geminiGenerationConfig `json:"generationConfig,omitempty"`
|
||||
Tools []geminiTool `json:"tools,omitempty"`
|
||||
ToolConfig *geminiToolConfig `json:"toolConfig,omitempty"`
|
||||
}
|
||||
|
||||
type geminiContent struct {
|
||||
Role string `json:"role,omitempty"`
|
||||
Parts []geminiPart `json:"parts"`
|
||||
}
|
||||
|
||||
type geminiPart struct {
|
||||
Text *string `json:"text,omitempty"`
|
||||
Thought bool `json:"thought,omitempty"`
|
||||
ThoughtSignature string `json:"thoughtSignature,omitempty"`
|
||||
FunctionCall *geminiFunctionCall `json:"functionCall,omitempty"`
|
||||
FunctionResponse *geminiFunctionResponse `json:"functionResponse,omitempty"`
|
||||
}
|
||||
|
||||
type geminiFunctionCall struct {
|
||||
Name string `json:"name"`
|
||||
Args json.RawMessage `json:"args"`
|
||||
}
|
||||
|
||||
type geminiFunctionResponse struct {
|
||||
Name string `json:"name"`
|
||||
Response json.RawMessage `json:"response"`
|
||||
}
|
||||
|
||||
type geminiGenerationConfig struct {
|
||||
CandidateCount *int `json:"candidateCount,omitempty"`
|
||||
MaxOutputTokens *int `json:"maxOutputTokens,omitempty"`
|
||||
StopSequences []string `json:"stopSequences,omitempty"`
|
||||
Temperature *float64 `json:"temperature,omitempty"`
|
||||
TopK *int `json:"topK,omitempty"`
|
||||
TopP *float64 `json:"topP,omitempty"`
|
||||
ThinkingConfig *geminiThinkingConfig `json:"thinkingConfig,omitempty"`
|
||||
}
|
||||
|
||||
type geminiThinkingConfig struct {
|
||||
IncludeThoughts *bool `json:"includeThoughts,omitempty"`
|
||||
ThinkingBudget *int `json:"thinkingBudget,omitempty"`
|
||||
}
|
||||
|
||||
type geminiTool struct {
|
||||
FunctionDeclarations []geminiFunctionDeclaration `json:"functionDeclarations"`
|
||||
}
|
||||
|
||||
type geminiFunctionDeclaration struct {
|
||||
Name string `json:"name"`
|
||||
Description string `json:"description,omitempty"`
|
||||
ParametersJSONSchema json.RawMessage `json:"parametersJsonSchema"`
|
||||
}
|
||||
|
||||
type geminiToolConfig struct {
|
||||
FunctionCallingConfig *geminiFunctionCallingConfig `json:"functionCallingConfig,omitempty"`
|
||||
}
|
||||
|
||||
type geminiFunctionCallingConfig struct {
|
||||
Mode string `json:"mode"`
|
||||
}
|
||||
|
||||
type geminiErrorResponse struct {
|
||||
Error geminiErrorBody `json:"error"`
|
||||
}
|
||||
|
||||
type geminiErrorBody struct {
|
||||
Code int `json:"code"`
|
||||
Message string `json:"message"`
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
type geminiChatStreamChunk struct {
|
||||
Choices []struct {
|
||||
Delta struct {
|
||||
Content string `json:"content"`
|
||||
ReasoningContent string `json:"reasoning_content"`
|
||||
Reasoning string `json:"reasoning"`
|
||||
ToolCalls []struct {
|
||||
Index int `json:"index"`
|
||||
ID string `json:"id"`
|
||||
ExtraContent openAIChatToolExtraContent `json:"extra_content,omitempty"`
|
||||
Function struct {
|
||||
Name string `json:"name"`
|
||||
Arguments string `json:"arguments"`
|
||||
} `json:"function"`
|
||||
} `json:"tool_calls"`
|
||||
} `json:"delta"`
|
||||
FinishReason *string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
Usage *struct {
|
||||
PromptTokens *int `json:"prompt_tokens,omitempty"`
|
||||
CompletionTokens *int `json:"completion_tokens,omitempty"`
|
||||
TotalTokens *int `json:"total_tokens,omitempty"`
|
||||
PromptTokensDetails *struct {
|
||||
CachedTokens *int `json:"cached_tokens,omitempty"`
|
||||
} `json:"prompt_tokens_details,omitempty"`
|
||||
CompletionTokensDetails *struct {
|
||||
ReasoningTokens *int `json:"reasoning_tokens,omitempty"`
|
||||
} `json:"completion_tokens_details,omitempty"`
|
||||
} `json:"usage,omitempty"`
|
||||
Error *struct {
|
||||
Type string `json:"type"`
|
||||
Message string `json:"message"`
|
||||
} `json:"error,omitempty"`
|
||||
}
|
||||
|
|
@ -167,6 +167,8 @@ func principalTokenFromRequest(r *http.Request) (string, bool) {
|
|||
apiKey := ""
|
||||
if isAnthropicRequest(r) {
|
||||
apiKey = strings.TrimSpace(r.Header.Get("X-Api-Key"))
|
||||
} else if isGeminiRequest(r) {
|
||||
apiKey = strings.TrimSpace(r.Header.Get("X-Goog-Api-Key"))
|
||||
}
|
||||
if authorization != "" && bearer == "" {
|
||||
return "", false
|
||||
|
|
|
|||
|
|
@ -54,7 +54,10 @@ func (s *Server) advertisedModelsForPrincipal(ctx context.Context) ([]advertised
|
|||
}
|
||||
|
||||
for _, r := range routes {
|
||||
id := strings.TrimSpace(r.RouteID)
|
||||
id := strings.TrimSpace(r.RouteAlias)
|
||||
if id == "" {
|
||||
id = strings.TrimSpace(r.RouteID)
|
||||
}
|
||||
addModel(id, id)
|
||||
}
|
||||
|
||||
|
|
@ -241,8 +244,10 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M
|
|||
if selector == "" {
|
||||
return managedCatalogBinding{}, ErrRouteNotFound
|
||||
}
|
||||
routeAlias := strings.TrimSpace(route.RouteAlias)
|
||||
explicit := !strings.EqualFold(selector, "default")
|
||||
var matches []managedCatalogBinding
|
||||
var aliasMatches []managedCatalogBinding
|
||||
for _, entry := range catalog {
|
||||
group := strings.TrimSpace(entry.ID)
|
||||
if group == "" {
|
||||
|
|
@ -260,9 +265,23 @@ func resolveManagedCatalogBinding(route authprojection.Route, catalog []config.M
|
|||
binding.ProviderID = strings.TrimSpace(providerID)
|
||||
}
|
||||
matches = append(matches, binding)
|
||||
if routeAlias != "" && routeAlias == group {
|
||||
aliasMatches = append(aliasMatches, binding)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
// A principal-owned public alias that exactly names a compatible catalog
|
||||
// group is an explicit disambiguator. This is required when two catalog
|
||||
// groups intentionally expose the same provider/upstream pair (for example
|
||||
// a general model and a preset-pinned fast alias). It never bypasses the
|
||||
// resource selector or upstream-model checks above.
|
||||
if len(aliasMatches) == 1 {
|
||||
return aliasMatches[0], nil
|
||||
}
|
||||
if len(aliasMatches) > 1 {
|
||||
return managedCatalogBinding{}, ErrRouteNotFound
|
||||
}
|
||||
if len(matches) != 1 {
|
||||
return managedCatalogBinding{}, ErrRouteNotFound
|
||||
}
|
||||
|
|
|
|||
|
|
@ -72,7 +72,7 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) {
|
|||
"token-p1": "principal-1",
|
||||
"token-p2": "principal-2",
|
||||
}, map[string]authprojection.Route{
|
||||
"r1": {RouteID: "route-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
|
||||
"r1": {RouteID: "route-p1", RouteAlias: "model-p1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
|
||||
"r2": {RouteID: "route-p2", PrincipalRef: "principal-2", CredentialSlotRef: "slot-2", ProfileID: "openai", UpstreamModel: "gpt-4o", ResourceSelector: "default"},
|
||||
})
|
||||
if err := cache.Apply(proj); err != nil {
|
||||
|
|
@ -96,8 +96,8 @@ func TestManagedModelsListIsPrincipalScoped(t *testing.T) {
|
|||
if err := json.Unmarshal(wP1.Body.Bytes(), &respP1); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(respP1.Data) != 1 || respP1.Data[0].ID != "route-p1" {
|
||||
t.Fatalf("P1 models: expected [route-p1], got %+v", respP1.Data)
|
||||
if len(respP1.Data) != 1 || respP1.Data[0].ID != "model-p1" {
|
||||
t.Fatalf("P1 models: expected public alias [model-p1], got %+v", respP1.Data)
|
||||
}
|
||||
|
||||
// Call for P2
|
||||
|
|
@ -123,7 +123,7 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) {
|
|||
proj := makeTestProjection(1, now, time.Hour, map[string]string{
|
||||
"token-p1": "principal-1",
|
||||
}, map[string]authprojection.Route{
|
||||
"r1": {RouteID: "claude-route-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"},
|
||||
"r1": {RouteID: "claude-route-1", RouteAlias: "claude-public-1", PrincipalRef: "principal-1", CredentialSlotRef: "slot-1", ProfileID: "anthropic", UpstreamModel: "claude-3-5-sonnet", ResourceSelector: "default"},
|
||||
})
|
||||
if err := cache.Apply(proj); err != nil {
|
||||
t.Fatal(err)
|
||||
|
|
@ -148,8 +148,8 @@ func TestManagedAnthropicModelsListUsesRouteIDs(t *testing.T) {
|
|||
if err := json.Unmarshal(w.Body.Bytes(), &anthropicResp); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-route-1" {
|
||||
t.Fatalf("expected [claude-route-1], got %+v", anthropicResp.Data)
|
||||
if len(anthropicResp.Data) != 1 || anthropicResp.Data[0].ID != "claude-public-1" {
|
||||
t.Fatalf("expected public alias [claude-public-1], got %+v", anthropicResp.Data)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -337,9 +337,15 @@ func TestManagedCatalogBindingExplicitSelectorAndNoFallback(t *testing.T) {
|
|||
t.Fatalf("missing selector err=%v", err)
|
||||
}
|
||||
route.ResourceSelector = "default"
|
||||
if _, err := resolveManagedCatalogBinding(route, append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}})); !errors.Is(err, ErrRouteNotFound) {
|
||||
ambiguous := append(catalog, config.ModelCatalogEntry{ID: "another-group", Providers: map[string]string{"provider-resource": "served"}})
|
||||
if _, err := resolveManagedCatalogBinding(route, ambiguous); !errors.Is(err, ErrRouteNotFound) {
|
||||
t.Fatalf("ambiguous catalog err=%v", err)
|
||||
}
|
||||
route.RouteAlias = "another-group"
|
||||
binding, err = resolveManagedCatalogBinding(route, ambiguous)
|
||||
if err != nil || binding.ModelGroupKey != "another-group" || binding.ProviderID != "" {
|
||||
t.Fatalf("alias-disambiguated binding=%+v err=%v", binding, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataSpoofIsOverwrittenByManagedBinding(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -386,6 +386,12 @@ func (s *Server) compilePresetArtifactBinding(dispatch routeDispatch, protocol s
|
|||
preset = found
|
||||
}
|
||||
}
|
||||
// Marked single-request presets use the operator-owned internal workspace
|
||||
// capability compiled into dispatch.SingleRequest. They intentionally reject
|
||||
// legacy caller workspace_tools, so artifact binding must not run here.
|
||||
if preset.SingleRequest != nil || dispatch.SingleRequest != nil {
|
||||
return nil, false, nil
|
||||
}
|
||||
if !isModeAllowed(preset, modeLight) {
|
||||
return nil, false, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ func (s *Server) routes() *http.ServeMux {
|
|||
mux.HandleFunc("/v1/models", s.withAuth(s.handleModels))
|
||||
mux.HandleFunc("/v1/chat/completions", s.withAuth(s.handleChatCompletions))
|
||||
mux.HandleFunc("/v1/responses", s.withAuth(s.handleResponses))
|
||||
mux.HandleFunc("/gemini/", s.withAuth(s.handleGeminiStreamGenerateContent))
|
||||
s.registerAnthropicRoutes(mux)
|
||||
mux.HandleFunc("/api/", s.withAuth(s.handleOllamaAPI))
|
||||
return mux
|
||||
|
|
@ -54,6 +55,10 @@ func (s *Server) writeCallerProviderCredentialRejection(w http.ResponseWriter, r
|
|||
writeAnthropicError(w, http.StatusBadRequest, "invalid_request_error", message)
|
||||
return
|
||||
}
|
||||
if isGeminiRequest(r) {
|
||||
writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", message)
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", message)
|
||||
}
|
||||
|
||||
|
|
@ -63,6 +68,10 @@ func (s *Server) writeAuthenticationFailure(w http.ResponseWriter, r *http.Reque
|
|||
writeAnthropicError(w, http.StatusUnauthorized, "authentication_error", "authentication failed")
|
||||
return
|
||||
}
|
||||
if isGeminiRequest(r) {
|
||||
writeGeminiError(w, http.StatusUnauthorized, "UNAUTHENTICATED", "authentication failed")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "unauthorized")
|
||||
}
|
||||
|
||||
|
|
|
|||
366
apps/edge/internal/openai/single_request_chat.go
Normal file
366
apps/edge/internal/openai/single_request_chat.go
Normal file
|
|
@ -0,0 +1,366 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
)
|
||||
|
||||
type singleRequestChatPolicy struct {
|
||||
status int
|
||||
finishReason string
|
||||
errorType string
|
||||
message string
|
||||
silent bool
|
||||
error bool
|
||||
}
|
||||
|
||||
func singleRequestChatTerminalPolicy(disposition edgeservice.SingleRequestTerminalDisposition) singleRequestChatPolicy {
|
||||
if disposition.Kind == "" {
|
||||
disposition.Kind = edgeservice.SingleRequestTerminalEndTurn
|
||||
}
|
||||
if disposition.Validate() != nil {
|
||||
disposition = edgeservice.SingleRequestTerminalDisposition{
|
||||
Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider,
|
||||
}
|
||||
}
|
||||
switch disposition.Kind {
|
||||
case edgeservice.SingleRequestTerminalEndTurn:
|
||||
return singleRequestChatPolicy{status: http.StatusOK, finishReason: "stop"}
|
||||
case edgeservice.SingleRequestTerminalLength:
|
||||
return singleRequestChatPolicy{status: http.StatusOK, finishReason: "length"}
|
||||
case edgeservice.SingleRequestTerminalCancelled:
|
||||
return singleRequestChatPolicy{silent: true}
|
||||
case edgeservice.SingleRequestTerminalError:
|
||||
switch disposition.ErrorClass {
|
||||
case edgeservice.SingleRequestTerminalErrorValidation, edgeservice.SingleRequestTerminalErrorContext:
|
||||
return singleRequestChatPolicy{status: http.StatusBadRequest, errorType: "invalid_request_error", message: "single-request execution was rejected", error: true}
|
||||
case edgeservice.SingleRequestTerminalErrorTimeout:
|
||||
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution timed out", error: true}
|
||||
default:
|
||||
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true}
|
||||
}
|
||||
default:
|
||||
return singleRequestChatPolicy{status: http.StatusBadGateway, errorType: "run_error", message: "single-request execution failed", error: true}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Server) handleChatSingleRequestStream(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
capability singleRequestService,
|
||||
dispatch routeDispatch,
|
||||
body []byte,
|
||||
) {
|
||||
requestID, err := newLogicalRequestRandomID()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
|
||||
return
|
||||
}
|
||||
requestID = "req_" + requestID
|
||||
stream, err := newSingleRequestChatStream(w, requestID, dispatch.SingleRequest.PublicModel)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "single-request streaming is unavailable")
|
||||
return
|
||||
}
|
||||
execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{
|
||||
RequestID: requestID,
|
||||
Binding: dispatch.SingleRequest.Clone(),
|
||||
Prompt: string(append([]byte(nil), body...)),
|
||||
})
|
||||
if err != nil || execution == nil {
|
||||
if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) {
|
||||
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started")
|
||||
return
|
||||
}
|
||||
defer execution.Cancel()
|
||||
_ = pumpSingleRequestChatStream(r.Context(), execution, stream, newWallClockSingleRequestAnthropicTicker)
|
||||
}
|
||||
|
||||
func (s *Server) handleChatSingleRequest(
|
||||
w http.ResponseWriter,
|
||||
r *http.Request,
|
||||
capability singleRequestService,
|
||||
dispatch routeDispatch,
|
||||
body []byte,
|
||||
) {
|
||||
requestID, err := newLogicalRequestRandomID()
|
||||
if err != nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
|
||||
return
|
||||
}
|
||||
requestID = "req_" + requestID
|
||||
execution, err := capability.StartSingleRequest(r.Context(), edgeservice.SingleRequestRequest{
|
||||
RequestID: requestID,
|
||||
Binding: dispatch.SingleRequest.Clone(),
|
||||
Prompt: string(append([]byte(nil), body...)),
|
||||
})
|
||||
if err != nil || execution == nil {
|
||||
if errors.Is(err, edgeservice.ErrSingleRequestExecutorUnavailable) {
|
||||
writeError(w, http.StatusServiceUnavailable, "run_error", "single-request execution is unavailable")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadGateway, "run_error", "single-request execution could not be started")
|
||||
return
|
||||
}
|
||||
defer execution.Cancel()
|
||||
for {
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
execution.Cancel()
|
||||
return
|
||||
case progress, ok := <-execution.Progress():
|
||||
if !ok {
|
||||
if r.Context().Err() == nil && execution.State() != edgeservice.SingleRequestStateCancelled {
|
||||
writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed")
|
||||
}
|
||||
return
|
||||
}
|
||||
switch progress.Stage {
|
||||
case edgeservice.SingleRequestStateFinalizing:
|
||||
if progress.Result == nil {
|
||||
_ = execution.AcknowledgeTerminal(false)
|
||||
writeError(w, http.StatusBadGateway, "run_error", "single-request execution failed")
|
||||
return
|
||||
}
|
||||
writeErr := writeChatSingleRequestTerminal(w, requestID, dispatch.SingleRequest.PublicModel, *progress.Result)
|
||||
_ = execution.AcknowledgeTerminal(writeErr == nil)
|
||||
return
|
||||
case edgeservice.SingleRequestStateFailed:
|
||||
policy := singleRequestChatTerminalPolicy(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}))
|
||||
writeError(w, policy.status, policy.errorType, policy.message)
|
||||
return
|
||||
case edgeservice.SingleRequestStateCancelled:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func writeChatSingleRequestTerminal(w http.ResponseWriter, requestID, model string, result edgeservice.SingleRequestResult) error {
|
||||
policy := singleRequestChatTerminalPolicy(result.Terminal)
|
||||
if policy.silent || policy.error || policy.finishReason == "" {
|
||||
return errors.New("single-request result has no Chat terminal")
|
||||
}
|
||||
content := result.Output
|
||||
if policy.finishReason == "length" {
|
||||
content = ""
|
||||
}
|
||||
response := chatCompletionResponse{
|
||||
ID: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), Object: "chat.completion",
|
||||
Created: time.Now().Unix(), Model: model,
|
||||
Choices: []chatCompletionChoice{{Index: 0, Message: chatMessage{Role: "assistant", Content: content}, FinishReason: policy.finishReason}},
|
||||
}
|
||||
encoded, err := json.Marshal(response)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
encoded = append(encoded, '\n')
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(policy.status)
|
||||
n, err := w.Write(encoded)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n != len(encoded) {
|
||||
return io.ErrShortWrite
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type singleRequestChatStream struct {
|
||||
mu sync.Mutex
|
||||
w http.ResponseWriter
|
||||
flusher http.Flusher
|
||||
id string
|
||||
model string
|
||||
created int64
|
||||
started bool
|
||||
terminal bool
|
||||
terminalErr error
|
||||
}
|
||||
|
||||
func newSingleRequestChatStream(w http.ResponseWriter, requestID, model string) (*singleRequestChatStream, error) {
|
||||
flusher, ok := w.(http.Flusher)
|
||||
if !ok || strings.TrimSpace(requestID) == "" || strings.TrimSpace(model) == "" {
|
||||
return nil, errors.New("single-request Chat stream is unavailable")
|
||||
}
|
||||
return &singleRequestChatStream{w: w, flusher: flusher, id: "chatcmpl_iop_" + strings.TrimPrefix(requestID, "req_"), model: model, created: time.Now().Unix()}, nil
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) startLocked() error {
|
||||
if s.started {
|
||||
return nil
|
||||
}
|
||||
s.w.Header().Set("Content-Type", "text/event-stream")
|
||||
s.w.Header().Set("Cache-Control", "no-cache")
|
||||
s.w.WriteHeader(http.StatusOK)
|
||||
s.flusher.Flush()
|
||||
s.started = true
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) Start() error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal {
|
||||
return s.terminalErr
|
||||
}
|
||||
return s.startLocked()
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) Ping() error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal {
|
||||
return s.terminalErr
|
||||
}
|
||||
if err := s.startLocked(); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := io.WriteString(s.w, ": ping\n\n")
|
||||
if err == nil {
|
||||
s.flusher.Flush()
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) writeSSELocked(value any) error {
|
||||
payload, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = fmt.Fprintf(s.w, "data: %s\n\n", payload); err != nil {
|
||||
return err
|
||||
}
|
||||
s.flusher.Flush()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) Final(result edgeservice.SingleRequestResult) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal {
|
||||
return s.terminalErr
|
||||
}
|
||||
if err := s.startLocked(); err != nil {
|
||||
return err
|
||||
}
|
||||
policy := singleRequestChatTerminalPolicy(result.Terminal)
|
||||
if policy.silent || policy.error || policy.finishReason == "" {
|
||||
return errors.New("single-request result has no Chat stream terminal")
|
||||
}
|
||||
s.terminal = true
|
||||
if policy.finishReason != "length" && result.Output != "" {
|
||||
if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{Content: result.Output}}}}); err != nil {
|
||||
s.terminalErr = err
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := s.writeSSELocked(chatCompletionChunk{ID: s.id, Object: "chat.completion.chunk", Created: s.created, Model: s.model, Choices: []chatCompletionChunkChoice{{Index: 0, Delta: chatDelta{}, FinishReason: policy.finishReason}}}); err != nil {
|
||||
s.terminalErr = err
|
||||
return err
|
||||
}
|
||||
if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil {
|
||||
s.terminalErr = err
|
||||
return err
|
||||
}
|
||||
s.flusher.Flush()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *singleRequestChatStream) TerminalError(disposition edgeservice.SingleRequestTerminalDisposition) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal {
|
||||
return s.terminalErr
|
||||
}
|
||||
policy := singleRequestChatTerminalPolicy(disposition)
|
||||
if policy.silent {
|
||||
s.terminal = true
|
||||
return nil
|
||||
}
|
||||
if !policy.error {
|
||||
policy = singleRequestChatTerminalPolicy(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
|
||||
}
|
||||
if err := s.startLocked(); err != nil {
|
||||
return err
|
||||
}
|
||||
s.terminal = true
|
||||
if err := s.writeSSELocked(errorResponse{Error: errorBody{Type: policy.errorType, Message: policy.message}}); err != nil {
|
||||
s.terminalErr = err
|
||||
return err
|
||||
}
|
||||
if _, err := io.WriteString(s.w, "data: [DONE]\n\n"); err != nil {
|
||||
s.terminalErr = err
|
||||
return err
|
||||
}
|
||||
s.flusher.Flush()
|
||||
return nil
|
||||
}
|
||||
|
||||
func pumpSingleRequestChatStream(ctx context.Context, execution edgeservice.SingleRequestExecution, stream *singleRequestChatStream, tickerFactory singleRequestAnthropicTickerFactory) error {
|
||||
if execution == nil || stream == nil || tickerFactory == nil {
|
||||
return errors.New("single-request Chat stream is unavailable")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
execution.Cancel()
|
||||
return err
|
||||
}
|
||||
if err := stream.Start(); err != nil {
|
||||
execution.Cancel()
|
||||
return err
|
||||
}
|
||||
ticker := tickerFactory()
|
||||
if ticker == nil {
|
||||
execution.Cancel()
|
||||
return errors.New("single-request Chat stream is unavailable")
|
||||
}
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
execution.Cancel()
|
||||
return ctx.Err()
|
||||
case <-ticker.Ticks():
|
||||
if err := stream.Ping(); err != nil {
|
||||
execution.Cancel()
|
||||
return err
|
||||
}
|
||||
case progress, ok := <-execution.Progress():
|
||||
if !ok {
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
if execution.State() == edgeservice.SingleRequestStateCompleted {
|
||||
return nil
|
||||
}
|
||||
return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
|
||||
}
|
||||
switch progress.Stage {
|
||||
case edgeservice.SingleRequestStateFinalizing:
|
||||
if progress.Result == nil {
|
||||
err := stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider})
|
||||
return errors.Join(err, execution.AcknowledgeTerminal(false))
|
||||
}
|
||||
err := stream.Final(*progress.Result)
|
||||
return errors.Join(err, execution.AcknowledgeTerminal(err == nil))
|
||||
case edgeservice.SingleRequestStateFailed:
|
||||
return stream.TerminalError(singleRequestProgressTerminal(progress, edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalError, ErrorClass: edgeservice.SingleRequestTerminalErrorProvider}))
|
||||
case edgeservice.SingleRequestStateCancelled:
|
||||
return stream.TerminalError(edgeservice.SingleRequestTerminalDisposition{Kind: edgeservice.SingleRequestTerminalCancelled})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
96
apps/edge/internal/openai/single_request_chat_test.go
Normal file
96
apps/edge/internal/openai/single_request_chat_test.go
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
)
|
||||
|
||||
func TestSingleRequestChatStreamPumpProjectsOnlyFinalResult(t *testing.T) {
|
||||
execution := startSingleRequestAnthropicTestExecution(t, func(
|
||||
_ context.Context,
|
||||
req edgeservice.SingleRequestRequest,
|
||||
ctrl edgeservice.SingleRequestController,
|
||||
) error {
|
||||
for index, stage := range []edgeservice.SingleRequestState{
|
||||
edgeservice.SingleRequestStatePlanning,
|
||||
edgeservice.SingleRequestStateWorking,
|
||||
edgeservice.SingleRequestStateReviewing,
|
||||
edgeservice.SingleRequestStateFinalizing,
|
||||
} {
|
||||
envelope := edgeservice.SingleRequestEnvelope{
|
||||
RequestID: req.RequestID, Sequence: uint64(index + 1), Stage: stage,
|
||||
}
|
||||
if stage == edgeservice.SingleRequestStateFinalizing {
|
||||
envelope.Result = &edgeservice.SingleRequestResult{Output: "safe final"}
|
||||
}
|
||||
if err := ctrl.SubmitEnvelope(envelope); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
w := httptest.NewRecorder()
|
||||
stream, err := newSingleRequestChatStream(w, "req_chat", "virtual-model")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ticker := newManualSingleRequestAnthropicTicker()
|
||||
if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body := w.Body.String()
|
||||
if !strings.Contains(body, `"content":"safe final"`) || !strings.Contains(body, `"finish_reason":"stop"`) || strings.Count(body, "data: [DONE]") != 1 {
|
||||
t.Fatalf("unexpected Chat stream: %s", body)
|
||||
}
|
||||
for _, private := range []string{"Planning the requested work", "Executing the requested work", "Reviewing the completed work"} {
|
||||
if strings.Contains(body, private) {
|
||||
t.Fatalf("internal progress leaked: %s", body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestChatStreamSanitizesExecutorFailure(t *testing.T) {
|
||||
execution := startSingleRequestAnthropicTestExecution(t, func(
|
||||
_ context.Context,
|
||||
req edgeservice.SingleRequestRequest,
|
||||
ctrl edgeservice.SingleRequestController,
|
||||
) error {
|
||||
if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: 1, Stage: edgeservice.SingleRequestStatePlanning}); err != nil {
|
||||
return err
|
||||
}
|
||||
return errors.New("PRIVATE_EXECUTOR_FAILURE")
|
||||
})
|
||||
w := httptest.NewRecorder()
|
||||
stream, err := newSingleRequestChatStream(w, "req_chat_error", "virtual-model")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ticker := newManualSingleRequestAnthropicTicker()
|
||||
if err := pumpSingleRequestChatStream(context.Background(), execution, stream, func() singleRequestAnthropicTicker { return ticker }); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body := w.Body.String()
|
||||
if strings.Contains(body, "PRIVATE_EXECUTOR_FAILURE") || !strings.Contains(body, "single-request execution failed") || strings.Count(body, "data: [DONE]") != 1 {
|
||||
t.Fatalf("unexpected sanitized Chat error: %s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMarkedSingleRequestSkipsLegacyWorkspaceToolBinding(t *testing.T) {
|
||||
dispatch := routeDispatch{
|
||||
Preset: validSingleRequestPreset(),
|
||||
SingleRequest: newSingleRequestAnthropicTestBinding(t),
|
||||
}
|
||||
binding, pinArtifact, err := (&Server{}).compilePresetArtifactBinding(
|
||||
dispatch,
|
||||
"openai",
|
||||
[]byte(`{"model":"virtual-model","messages":[{"role":"user","content":"task"}]}`),
|
||||
)
|
||||
if err != nil || binding != nil || pinArtifact {
|
||||
t.Fatalf("marked single request entered legacy workspace binding: binding=%+v pin=%t err=%v", binding, pinArtifact, err)
|
||||
}
|
||||
}
|
||||
|
|
@ -185,8 +185,10 @@ type singleRequestProviderOutputSchema struct {
|
|||
}
|
||||
|
||||
type singleRequestProviderOutputProperty struct {
|
||||
Type string `json:"type"`
|
||||
Description string `json:"description"`
|
||||
Type string `json:"type"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Enum []string `json:"enum,omitempty"`
|
||||
MinLength int `json:"minLength,omitempty"`
|
||||
}
|
||||
|
||||
// buildSingleRequestChatBody owns all request authority. Stage options are
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ import (
|
|||
)
|
||||
|
||||
const (
|
||||
singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
singleRequestReviewPrompt = "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
singleRequestReviewStageID = "review"
|
||||
)
|
||||
|
||||
|
|
@ -110,7 +110,9 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
|
|||
{Role: "user", Content: "Task:\n" + strings.TrimSpace(req.Task) + "\n\nPLAN:\n" + string(plan) + "\n\nWORK COMPLETION:\n" + strings.TrimSpace(req.Work.Completion) + "\n\nWORK VERIFICATION:\n" + strings.TrimSpace(req.Work.Verification)},
|
||||
}
|
||||
repairRequired := false
|
||||
for attempts := 0; attempts <= req.Limits.MaxToolIterations; attempts++ {
|
||||
invalidToolCorrections := 0
|
||||
toolAttempts := 0
|
||||
for {
|
||||
response, err := s.submit(ctx, req, messages, tools, repairRequired)
|
||||
if err != nil {
|
||||
return nil, quality.reclassify(err, errSingleRequestReviewStage)
|
||||
|
|
@ -132,7 +134,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
|
|||
}
|
||||
return result, nil
|
||||
}
|
||||
if response.call == nil || attempts == req.Limits.MaxToolIterations {
|
||||
if response.call == nil || toolAttempts >= req.Limits.MaxToolIterations {
|
||||
if response.call != nil {
|
||||
return nil, quality.budget(errSingleRequestReviewStage)
|
||||
}
|
||||
|
|
@ -142,6 +144,7 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
|
|||
if err != nil {
|
||||
return nil, quality.malformed(errSingleRequestReviewStage)
|
||||
}
|
||||
arguments = normalizeSingleRequestProviderToolArguments(response.call.Function.Name, arguments)
|
||||
isInspection := response.call.Function.Name == edgeservice.InternalWorkspaceToolRead || response.call.Function.Name == edgeservice.InternalWorkspaceToolList
|
||||
if !isInspection && !isSingleRequestReviewRepairTool(response.call.Function.Name) {
|
||||
return nil, quality.malformed(errSingleRequestReviewStage)
|
||||
|
|
@ -168,12 +171,24 @@ func (s *singleRequestReviewStage) run(ctx context.Context, req singleRequestRev
|
|||
}
|
||||
}
|
||||
key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestReviewStageID, toolCallID: response.call.ID}
|
||||
call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments}
|
||||
if err := edgeservice.ValidateInternalWorkspaceToolCall(call); err != nil {
|
||||
if invalidToolCorrections >= 1 {
|
||||
return nil, quality.malformed(errSingleRequestReviewStage)
|
||||
}
|
||||
invalidToolCorrections++
|
||||
messages = append(messages,
|
||||
chatMessage{Role: "assistant", ToolCalls: []any{response.call.asChatToolCall()}},
|
||||
chatMessage{Role: "tool", ToolCallID: response.call.ID, ToolName: response.call.Function.Name, Content: singleRequestWorkToolResultContent(edgeservice.InternalWorkspaceToolResult{Status: "invalid", ErrorCode: "invalid_request"}, req.Limits.MaxOutputBytes)},
|
||||
)
|
||||
continue
|
||||
}
|
||||
toolAttempts++
|
||||
resultCh, err := s.bridge.register(key)
|
||||
if err != nil {
|
||||
return nil, quality.internalTool(errSingleRequestReviewStage)
|
||||
}
|
||||
sequence++
|
||||
call := &edgeservice.InternalWorkspaceToolCall{RequestID: req.RequestID, StageID: key.stageID, ToolCallID: key.toolCallID, Name: response.call.Function.Name, Arguments: arguments}
|
||||
if err := ctrl.SubmitEnvelope(edgeservice.SingleRequestEnvelope{RequestID: req.RequestID, Sequence: sequence, Stage: edgeservice.SingleRequestStateInternalTool, SavedStage: stage, ToolCall: call}); err != nil {
|
||||
s.bridge.unregister(key)
|
||||
return nil, quality.serviceFailure(ctx, err, errSingleRequestReviewStage)
|
||||
|
|
@ -249,7 +264,7 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any
|
|||
if repairRequired {
|
||||
toolChoice = "required"
|
||||
}
|
||||
body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "stream": false}
|
||||
body := map[string]any{"model": target, "messages": messages, "tools": tools, "tool_choice": toolChoice, "parallel_tool_calls": false, "response_format": singleRequestReviewResponseFormat(), "stream": false}
|
||||
for key, value := range options {
|
||||
folded := strings.ToLower(key)
|
||||
if isSingleRequestReviewReservedOption(folded) && key != folded {
|
||||
|
|
@ -269,13 +284,35 @@ func buildSingleRequestReviewBody(messages []chatMessage, options map[string]any
|
|||
|
||||
func isSingleRequestReviewReservedOption(key string) bool {
|
||||
switch key {
|
||||
case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "stream", "credential", "credential_binding", "reasoning_effort":
|
||||
case "model", "messages", "tools", "tool_choice", "parallel_tool_calls", "response_format", "stream", "credential", "credential_binding", "reasoning_effort":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func singleRequestReviewResponseFormat() *singleRequestProviderResponseFormat {
|
||||
return &singleRequestProviderResponseFormat{
|
||||
Type: "json_schema",
|
||||
JSONSchema: singleRequestProviderResponseJSONSchema{
|
||||
Name: "single_request_review",
|
||||
Strict: true,
|
||||
Schema: singleRequestProviderOutputSchema{
|
||||
Type: "object",
|
||||
Properties: map[string]singleRequestProviderOutputProperty{
|
||||
"decision": {Type: "string", Enum: []string{"pass"}},
|
||||
"output": {Type: "string", MinLength: 1},
|
||||
"checks": {Type: "string", MinLength: 1},
|
||||
"verification": {Type: "string", MinLength: 1},
|
||||
"summary": {Type: "string", MinLength: 1},
|
||||
},
|
||||
Required: []string{"decision", "output", "checks", "verification", "summary"},
|
||||
AdditionalProperties: false,
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
type singleRequestReviewProviderEnvelope struct {
|
||||
ID string `json:"id"`
|
||||
Object string `json:"object"`
|
||||
|
|
|
|||
|
|
@ -383,6 +383,52 @@ func TestSingleRequestReviewStageInspectionAndRepairRemainInLegalStates(t *testi
|
|||
})
|
||||
}
|
||||
|
||||
func TestSingleRequestReviewStageBoundedInvalidToolCorrection(t *testing.T) {
|
||||
t.Run("one malformed provider call is corrected before coordinator admission", func(t *testing.T) {
|
||||
bridge := newSingleRequestWorkToolBridge()
|
||||
ctrl := newReviewController(t, bridge)
|
||||
var bodies [][]byte
|
||||
responses := [][]byte{
|
||||
reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
|
||||
reviewToolBody("inspect-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":"result.txt"}`),
|
||||
reviewPassBody("Approved.", "Corrected inspection passed."),
|
||||
}
|
||||
stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge)
|
||||
if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(bodies) != 3 || !strings.Contains(string(bodies[1]), "invalid_request") {
|
||||
t.Fatalf("provider correction bodies=%d", len(bodies))
|
||||
}
|
||||
want := []edgeservice.SingleRequestState{edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateInternalTool, edgeservice.SingleRequestStateReviewing, edgeservice.SingleRequestStateFinalizing}
|
||||
if len(ctrl.envelopes) != len(want) {
|
||||
t.Fatalf("envelopes=%+v", ctrl.envelopes)
|
||||
}
|
||||
for index, state := range want {
|
||||
if ctrl.envelopes[index].Stage != state {
|
||||
t.Fatalf("envelopes=%+v", ctrl.envelopes)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("second malformed provider call fails closed", func(t *testing.T) {
|
||||
bridge := newSingleRequestWorkToolBridge()
|
||||
ctrl := newReviewController(t, bridge)
|
||||
var bodies [][]byte
|
||||
responses := [][]byte{
|
||||
reviewToolBody("invalid-1", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
|
||||
reviewToolBody("invalid-2", edgeservice.InternalWorkspaceToolRead, `{"relative_path":""}`),
|
||||
}
|
||||
stage := newSingleRequestReviewStage(scriptedReviewProvider(t, ctrl, responses, &bodies), bridge)
|
||||
if _, err := stage.run(context.Background(), reviewRequest(t), ctrl); !errors.Is(err, errSingleRequestReviewStage) {
|
||||
t.Fatalf("err=%v, want review stage failure", err)
|
||||
}
|
||||
if len(bodies) != 2 || len(ctrl.envelopes) != 1 || ctrl.envelopes[0].Stage != edgeservice.SingleRequestStateReviewing || bridge.pendingCount() != 0 {
|
||||
t.Fatalf("bodies=%d envelopes=%+v pending=%d", len(bodies), ctrl.envelopes, bridge.pendingCount())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestSingleRequestReviewStageFailsClosed(t *testing.T) {
|
||||
for _, raw := range []string{
|
||||
`{"decision":"pass","output":"x","checks":"c","verification":"v","summary":"y","extra":1}`,
|
||||
|
|
@ -471,11 +517,41 @@ func TestSingleRequestReviewBodyRejectsOptionAliases(t *testing.T) {
|
|||
if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "low"}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) {
|
||||
t.Fatalf("err=%v", err)
|
||||
}
|
||||
if _, err := buildSingleRequestReviewBody([]chatMessage{{Role: "user", Content: "x"}}, map[string]any{"reasoning_effort": "high", "Response_Format": map[string]any{"type": "caller"}}, []any{singleRequestWorkToolSchema(edgeservice.InternalWorkspaceToolRead, map[string]any{"type": "object"})}, "gemini", false); !errors.Is(err, errSingleRequestReviewStage) {
|
||||
t.Fatalf("response format alias err=%v", err)
|
||||
}
|
||||
if _, _, err := renderSingleRequestReview(singlerequesttemplate.DefaultReviewTemplate, singleRequestReviewDecision{Decision: "pass", Output: strings.Repeat("x", 10), Checks: "c", Verification: "v", Summary: "summary"}, 9); !errors.Is(err, errSingleRequestReviewStage) {
|
||||
t.Fatalf("err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestReviewResponseFormatIsClosed(t *testing.T) {
|
||||
format := singleRequestReviewResponseFormat()
|
||||
if format.Type != "json_schema" || format.JSONSchema.Name != "single_request_review" || !format.JSONSchema.Strict {
|
||||
t.Fatalf("response format=%+v", format)
|
||||
}
|
||||
schema := format.JSONSchema.Schema
|
||||
if schema.Type != "object" || schema.AdditionalProperties || len(schema.Properties) != 5 || len(schema.Required) != 5 {
|
||||
t.Fatalf("review schema=%+v", schema)
|
||||
}
|
||||
if decision := schema.Properties["decision"]; !reflect.DeepEqual(decision.Enum, []string{"pass"}) {
|
||||
t.Fatalf("decision schema=%+v", decision)
|
||||
}
|
||||
for _, name := range []string{"output", "checks", "verification", "summary"} {
|
||||
if schema.Properties[name].MinLength != 1 {
|
||||
t.Fatalf("property %q schema=%+v", name, schema.Properties[name])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestReviewPromptRequiresCanonicalWorkspaceRelativePaths(t *testing.T) {
|
||||
for _, requirement := range []string{"canonical", "workspace-relative", "README.md", "./README.md", "absolute path", "parent traversal"} {
|
||||
if !strings.Contains(singleRequestReviewPrompt, requirement) {
|
||||
t.Fatalf("review prompt does not describe %q path requirement", requirement)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// customReviewStageTemplate is an operator-authored effective Review template
|
||||
// that differs from the built-in default, so a rendered artifact cannot pass by
|
||||
// accidentally falling back.
|
||||
|
|
@ -645,7 +721,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
|
|
@ -664,7 +740,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -681,7 +759,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -698,7 +778,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path", "content"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
},
|
||||
"content": {
|
||||
"type": "string"
|
||||
|
|
@ -718,7 +800,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -764,7 +848,7 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
"content": "Review the task, plan, completed work, and verification evidence. Return exactly one JSON object with decision=pass, non-empty output, non-empty checks, non-empty verification, and non-empty summary when approved. Otherwise make exactly one approved workspace tool call to inspect or repair, with no text content. Every relative_path argument must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal. After a tool result with error_code=invalid_request, correct the arguments and make exactly one valid tool call; do not pass. After a tool result with error_code=not_found, do not pass or inspect again; make one repair tool call."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
|
|
@ -808,7 +892,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -825,7 +911,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -842,7 +930,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path", "content"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
},
|
||||
"content": {
|
||||
"type": "string"
|
||||
|
|
@ -862,7 +952,9 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
"required": ["relative_path"],
|
||||
"properties": {
|
||||
"relative_path": {
|
||||
"type": "string"
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"description": "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -900,6 +992,15 @@ func expectedSingleRequestReviewBodyAuthority(isResumed bool) map[string]any {
|
|||
}
|
||||
var res map[string]any
|
||||
_ = json.Unmarshal([]byte(raw), &res)
|
||||
for _, rawTool := range res["tools"].([]any) {
|
||||
tool := rawTool.(map[string]any)
|
||||
function := tool["function"].(map[string]any)
|
||||
function["strict"] = true
|
||||
}
|
||||
encodedFormat, _ := json.Marshal(singleRequestReviewResponseFormat())
|
||||
var expectedFormat any
|
||||
_ = json.Unmarshal(encodedFormat, &expectedFormat)
|
||||
res["response_format"] = expectedFormat
|
||||
return res
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -16,8 +16,9 @@ import (
|
|||
)
|
||||
|
||||
const (
|
||||
singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification."
|
||||
singleRequestWorkStageID = "work"
|
||||
singleRequestWorkPrompt = "Read the supplied plan, use only the supplied workspace tools when needed, then return exactly one JSON object with non-empty string fields completion and verification. Every relative_path argument and every workspace path mentioned in the completion or verification must be canonical and workspace-relative: use README.md, never ./README.md, an absolute path, or a parent traversal."
|
||||
singleRequestWorkStageID = "work"
|
||||
singleRequestCanonicalRelativePathDescription = "Canonical path relative to the workspace root. Never start with /, ./, or ../; use README.md rather than ./README.md."
|
||||
)
|
||||
|
||||
var errSingleRequestWorkStage = errors.New("single-request work stage: failed")
|
||||
|
|
@ -240,6 +241,7 @@ func (s *singleRequestWorkStage) run(ctx context.Context, req singleRequestWorkS
|
|||
if err != nil {
|
||||
return nil, quality.malformed(errSingleRequestWorkStage)
|
||||
}
|
||||
arguments = normalizeSingleRequestProviderToolArguments(call.Function.Name, arguments)
|
||||
key := singleRequestWorkToolKey{requestID: req.RequestID, stageID: singleRequestWorkStageID, toolCallID: call.ID}
|
||||
resultCh, err := s.bridge.register(key)
|
||||
if err != nil {
|
||||
|
|
@ -283,14 +285,15 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding
|
|||
return false
|
||||
}
|
||||
tools := make([]any, 0, len(workspace.OperationIDs))
|
||||
path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}}}
|
||||
pathProperty := map[string]any{"type": "string", "minLength": 1, "description": singleRequestCanonicalRelativePathDescription}
|
||||
path := map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path"}, "properties": map[string]any{"relative_path": pathProperty}}
|
||||
for _, pair := range []struct{ operation, name string }{{"read", edgeservice.InternalWorkspaceToolRead}, {"list", edgeservice.InternalWorkspaceToolList}, {"write", edgeservice.InternalWorkspaceToolWrite}, {"delete", edgeservice.InternalWorkspaceToolDelete}} {
|
||||
if !has(pair.operation) {
|
||||
continue
|
||||
}
|
||||
parameters := path
|
||||
if pair.operation == "write" {
|
||||
parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": map[string]any{"type": "string"}, "content": map[string]any{"type": "string"}}}
|
||||
parameters = map[string]any{"type": "object", "additionalProperties": false, "required": []string{"relative_path", "content"}, "properties": map[string]any{"relative_path": pathProperty, "content": map[string]any{"type": "string"}}}
|
||||
}
|
||||
tools = append(tools, singleRequestWorkToolSchema(pair.name, parameters))
|
||||
}
|
||||
|
|
@ -316,7 +319,7 @@ func singleRequestWorkTools(workspace *edgeservice.SingleRequestWorkspaceBinding
|
|||
}
|
||||
|
||||
func singleRequestWorkToolSchema(name string, parameters map[string]any) map[string]any {
|
||||
return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "parameters": parameters}}
|
||||
return map[string]any{"type": "function", "function": map[string]any{"name": name, "description": "Approved IOP workspace operation.", "strict": true, "parameters": parameters}}
|
||||
}
|
||||
|
||||
type singleRequestWorkProviderResponse struct {
|
||||
|
|
@ -514,6 +517,30 @@ func decodeSingleRequestWorkToolArguments(arguments string) (json.RawMessage, er
|
|||
return append(json.RawMessage(nil), bytes.TrimSpace(raw)...), nil
|
||||
}
|
||||
|
||||
// normalizeSingleRequestProviderToolArguments canonicalizes the one
|
||||
// containment-safe provider convention that differs from the internal tool
|
||||
// contract: an empty workspace_list path means the workspace root. All other
|
||||
// operations and non-canonical paths remain unchanged and are rejected by the
|
||||
// service-owned decoder.
|
||||
func normalizeSingleRequestProviderToolArguments(name string, arguments json.RawMessage) json.RawMessage {
|
||||
if name != edgeservice.InternalWorkspaceToolList {
|
||||
return arguments
|
||||
}
|
||||
var pathArguments struct {
|
||||
RelativePath *string `json:"relative_path"`
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(arguments))
|
||||
decoder.DisallowUnknownFields()
|
||||
if err := decoder.Decode(&pathArguments); err != nil || pathArguments.RelativePath == nil || *pathArguments.RelativePath != "" {
|
||||
return arguments
|
||||
}
|
||||
var trailing any
|
||||
if err := decoder.Decode(&trailing); err != io.EOF {
|
||||
return arguments
|
||||
}
|
||||
return json.RawMessage(`{"relative_path":"."}`)
|
||||
}
|
||||
|
||||
func decodeSingleRequestWorkProviderResponse(body []byte, maximum int) (*singleRequestWorkProviderResponse, error) {
|
||||
if len(body) == 0 || len(body) > maximum || validateSingleRequestJSON(body) != nil {
|
||||
return nil, errSingleRequestWorkStage
|
||||
|
|
|
|||
|
|
@ -877,6 +877,29 @@ func TestSingleRequestWorkStageRejectsMalformedResponsesAndOptions(t *testing.T)
|
|||
}
|
||||
}
|
||||
|
||||
func TestNormalizeSingleRequestProviderToolArguments(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
tool string
|
||||
arguments string
|
||||
want string
|
||||
}{
|
||||
{name: "empty list root becomes canonical dot", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":""}`, want: `{"relative_path":"."}`},
|
||||
{name: "empty read remains invalid for service", tool: edgeservice.InternalWorkspaceToolRead, arguments: `{"relative_path":""}`, want: `{"relative_path":""}`},
|
||||
{name: "dot-prefixed list remains invalid for service", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"./"}`, want: `{"relative_path":"./"}`},
|
||||
{name: "canonical list path is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"src"}`, want: `{"relative_path":"src"}`},
|
||||
{name: "unknown list field is unchanged", tool: edgeservice.InternalWorkspaceToolList, arguments: `{"relative_path":"","extra":true}`, want: `{"relative_path":"","extra":true}`},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got := normalizeSingleRequestProviderToolArguments(test.tool, json.RawMessage(test.arguments))
|
||||
if string(got) != test.want {
|
||||
t.Fatalf("normalized arguments=%s, want %s", got, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestWorkStageRejectsReservedOptionAliases(t *testing.T) {
|
||||
messageSets := map[string][]chatMessage{
|
||||
"initial": {{Role: "user", Content: "immutable task"}},
|
||||
|
|
@ -977,6 +1000,37 @@ func TestSingleRequestWorkStageProjectsClosedEnvironmentSchema(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestWorkspaceToolSchemasDescribeCanonicalRelativePaths(t *testing.T) {
|
||||
tools, err := singleRequestWorkTools(workBinding(t).Workspace)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pathTools := 0
|
||||
for _, raw := range tools {
|
||||
tool, _ := raw.(map[string]any)
|
||||
function, _ := tool["function"].(map[string]any)
|
||||
if function["strict"] != true {
|
||||
t.Fatalf("tool %q strict=%v, want true", function["name"], function["strict"])
|
||||
}
|
||||
parameters, _ := function["parameters"].(map[string]any)
|
||||
properties, _ := parameters["properties"].(map[string]any)
|
||||
pathProperty, hasPath := properties["relative_path"].(map[string]any)
|
||||
if !hasPath {
|
||||
continue
|
||||
}
|
||||
pathTools++
|
||||
if pathProperty["description"] != singleRequestCanonicalRelativePathDescription {
|
||||
t.Fatalf("tool %q relative_path description=%v", function["name"], pathProperty["description"])
|
||||
}
|
||||
if pathProperty["minLength"] != 1 {
|
||||
t.Fatalf("tool %q relative_path minLength=%v, want 1", function["name"], pathProperty["minLength"])
|
||||
}
|
||||
}
|
||||
if pathTools == 0 {
|
||||
t.Fatal("workspace tools did not expose a relative_path schema")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSingleRequestWorkToolBridgeCorrelatesAndCleansUp(t *testing.T) {
|
||||
b := newSingleRequestWorkToolBridge()
|
||||
keys := []singleRequestWorkToolKey{{"request-a", "working", "one"}, {"request-b", "working", "two"}}
|
||||
|
|
|
|||
|
|
@ -81,6 +81,15 @@ func (r InternalWorkspaceToolResult) Clone() InternalWorkspaceToolResult {
|
|||
return r
|
||||
}
|
||||
|
||||
// ValidateInternalWorkspaceToolCall applies the service-owned closed decoder
|
||||
// without executing the call. Private provider stages use it to reject or
|
||||
// boundedly correct malformed model output before the coordinator enters an
|
||||
// internal-tool state.
|
||||
func ValidateInternalWorkspaceToolCall(call *InternalWorkspaceToolCall) error {
|
||||
_, err := decodeInternalWorkspaceToolCall(call.Clone())
|
||||
return err
|
||||
}
|
||||
|
||||
// SingleRequestToolContinuation is optional. An executor that emits an
|
||||
// internal workspace call must implement it so the coordinator can deliver the
|
||||
// correlated Node result without involving an HTTP caller.
|
||||
|
|
|
|||
|
|
@ -53,6 +53,18 @@ func TestInternalWorkspaceToolDecodeClosedOperations(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestValidateInternalWorkspaceToolCall(t *testing.T) {
|
||||
if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolList, `{"relative_path":"."}`)); err != nil {
|
||||
t.Fatalf("canonical call rejected: %v", err)
|
||||
}
|
||||
if err := ValidateInternalWorkspaceToolCall(internalToolCall(InternalWorkspaceToolRead, `{"relative_path":""}`)); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) {
|
||||
t.Fatalf("empty read error=%v, want invalid call", err)
|
||||
}
|
||||
if err := ValidateInternalWorkspaceToolCall(nil); !errors.Is(err, ErrSingleRequestInternalToolInvalidCall) {
|
||||
t.Fatalf("nil call error=%v, want invalid call", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInternalWorkspaceToolDecodeRejectsMalformed(t *testing.T) {
|
||||
const rawSentinel = "RAW-ARGUMENT-SENTINEL-DO-NOT-LEAK"
|
||||
tests := map[string]*InternalWorkspaceToolCall{
|
||||
|
|
|
|||
563
docs/agent-comparison-benchmark-dev-guide.md
Normal file
563
docs/agent-comparison-benchmark-dev-guide.md
Normal file
|
|
@ -0,0 +1,563 @@
|
|||
# Agent Comparison Benchmark Dev Guide
|
||||
|
||||
이 문서는 IOP one-shot agent/model comparison benchmark의 현재 구현, dev 환경 구성, caller 연결 방식, managed credential 경계, 실행 절차와 장애 대응 기준을 한곳에 모은 운영 가이드다.
|
||||
|
||||
raw token, provider API key, private key, slot alias, lease id와 개인 endpoint는 이 문서에 기록하지 않는다. 실제 host, checkout, Node/provider endpoint와 최신 process 상태는 아래 source of truth에서 확인한다.
|
||||
|
||||
- benchmark manifest: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- public CLI: `scripts/agent_comparison_benchmark.py`
|
||||
- dev environment: `agent-test/dev/rules.md`
|
||||
- machine-readable dev inventory: `agent-test/inventory-dev.yaml`
|
||||
- Edge/Node verification: `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md`
|
||||
- API contracts: `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`
|
||||
- benchmark SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
|
||||
이 문서는 위 계약과 manifest를 설명할 뿐 덮어쓰지 않는다. 값이 다르면 manifest, contract, inventory, environment rule 순서로 최신 상태를 확인한다.
|
||||
|
||||
## 1. 현재 상태
|
||||
|
||||
2026-08-12 기준 상태는 다음과 같다.
|
||||
|
||||
| 영역 | 상태 |
|
||||
|---|---|
|
||||
| Gemini-native Edge ingress | 구현 완료. route-qualified `streamGenerateContent`, `x-goog-api-key` IOP principal auth, request/tool/SSE 변환을 지원한다. |
|
||||
| official agy adapter | 구현 완료. `agy 1.1.12`, Gemini API-key provider, real `init/step_update/result` JSONL을 사용한다. |
|
||||
| managed dev credential runtime | 구성 완료. Control Plane projection, credential slot/route, sealed lease, Edge HTTPS, CP↔Edge/Edge↔Node mTLS를 사용한다. |
|
||||
| caller readiness | 최근 완료된 readiness evidence에서 C01-C09 `ready=9`를 확인했다. 실제 실행일에는 fresh preflight가 다시 필요하다. |
|
||||
| controller recovery | control socket symlink와 caller launch 전 interruption 회귀 수정 및 테스트가 완료됐다. |
|
||||
| Codex production JSONL | `cache_write_input_tokens` 수용과 config-owned effective binding 정합화가 완료됐다. |
|
||||
| deterministic tests | focused 41 tests, 전체 benchmark 429 tests가 통과했다. 숫자는 당시 snapshot이며 현재 suite 결과는 fresh 실행으로 판단한다. |
|
||||
| C01-C09 scored execution | 아직 완료되지 않았다. 과거 incomplete/failed run은 append-only evidence로 보존하며 정상 결과로 간주하지 않는다. |
|
||||
| blind scoring/report | 유효한 9-cell execution이 생긴 뒤 수행하는 후속 단계다. |
|
||||
|
||||
과거 실행 실패는 새 run을 정당화하는 완료 evidence가 아니다. 이전 run tree를 수정하거나 old run을 암묵적으로 `resume`/retry하지 않는다. 새 scored execution은 그 실행을 명시적으로 소유하는 현재 plan과 사용자 권한이 있을 때만 한 번 수행한다. 이 문서 자체는 지속적 실행 승인을 부여하지 않는다.
|
||||
|
||||
## 2. 시스템 구성
|
||||
|
||||
```text
|
||||
Benchmark runner
|
||||
├─ Claude Code ── Anthropic Messages ───────────────┐
|
||||
├─ agy ───────── Gemini streamGenerateContent ─────┤
|
||||
└─ Codex ─────── OpenAI Responses ─────────────────┤
|
||||
v
|
||||
IOP Edge HTTPS
|
||||
│
|
||||
┌───────────────────────────┴───────────────────────────┐
|
||||
│ │
|
||||
direct model route execution preset
|
||||
│ selector/plan/work/review/repair
|
||||
└───────────────────────────┬───────────────────────────┘
|
||||
v
|
||||
managed credential lease
|
||||
Control Plane projection + Node sealed lease
|
||||
│
|
||||
v
|
||||
Node-owned provider
|
||||
```
|
||||
|
||||
핵심 보안 경계는 다음과 같다.
|
||||
|
||||
- benchmark caller는 하나의 IOP principal token으로 Edge만 인증한다.
|
||||
- caller token은 upstream provider credential이 아니다.
|
||||
- provider credential은 Control Plane에 등록된 slot과 Node 대상 sealed lease에서만 온다.
|
||||
- Edge는 provider credential을 복호화하지 않고 caller가 보낸 token을 upstream auth로 재사용하지 않는다.
|
||||
- managed mode는 legacy static principal/provider credential과 혼용하지 않는다.
|
||||
- config observation은 secret이 아닌 route/model/stage binding의 독립 증거다.
|
||||
|
||||
## 3. 환경 프로필
|
||||
|
||||
### 3.1 Benchmark runner
|
||||
|
||||
현재 검증된 runner class는 Linux/AArch64다. 명령은 repository root에서 실행한다.
|
||||
|
||||
필수 command:
|
||||
|
||||
```bash
|
||||
command -v python3
|
||||
command -v git
|
||||
command -v claude
|
||||
command -v agy
|
||||
command -v codex
|
||||
```
|
||||
|
||||
2026-08-12 확인 snapshot:
|
||||
|
||||
| Tool | 확인된 버전 | 정책 |
|
||||
|---|---|---|
|
||||
| Claude Code | `2.1.228` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`/`--help`를 확인한다. |
|
||||
| agy | `1.1.12` | adapter가 이 버전을 명시적으로 gate한다. 다른 버전은 재검증 전 fail closed한다. |
|
||||
| Codex CLI | `0.147.0` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`과 `exec --help`를 확인한다. |
|
||||
|
||||
### 3.2 Testbed
|
||||
|
||||
- path: `../iop-s2`
|
||||
- 현재 확인 branch: `dev`
|
||||
- 현재 확인 상태: clean
|
||||
- benchmark는 testbed를 read-only provenance로 취급한다.
|
||||
- caller별 workspace와 session은 run tree 아래에 새로 만들며 서로 공유하지 않는다.
|
||||
- testbed를 benchmark 결과로 수정하거나 결과 파일을 다시 복사하지 않는다.
|
||||
|
||||
fixture checksum과 source file 목록은 manifest가 고정한다. testbed HEAD와 clean 상태는 실행일에 다시 확인한다.
|
||||
|
||||
### 3.3 Dev runtime
|
||||
|
||||
현재 검증된 runtime class는 macOS/ARM64 remote dev runner다. exact SSH target과 checkout은 `agent-test/dev/rules.md` 및 `agent-test/inventory-dev.yaml`을 따른다.
|
||||
|
||||
benchmark 관련 runtime 역할:
|
||||
|
||||
| Port | 역할 |
|
||||
|---:|---|
|
||||
| `18082` | Edge artifact/bootstrap HTTP |
|
||||
| `18083` | managed Edge public HTTPS; Anthropic/OpenAI/Gemini caller ingress |
|
||||
| `18084` | native dev-runtime Edge↔Node TCP |
|
||||
| `19093` | Edge admin/config refresh |
|
||||
| `19101` | Edge metrics |
|
||||
|
||||
2026-08-12 read-only 확인에서 위 listener와 managed Edge process는 모두 active였고 remote checkout은 clean release 상태였다. exact commit, binary checksum, process id와 endpoint는 실행 evidence에만 기록하고 이 가이드에 고정하지 않는다.
|
||||
|
||||
최근 완료된 live readiness evidence는 4 connected Nodes와 8 healthy/available providers를 확인했다. 최신 Node/provider 세부와 접속 위치는 반드시 `agent-test/inventory-dev.yaml`에서 다시 확인한다.
|
||||
|
||||
이 benchmark는 compose dev stack의 Edge-Node TCP `19003`이 아니라 native dev-runtime provider pool의 `18084`를 사용한다. compose와 native profile은 포트, process와 판정 evidence가 서로 다르므로 한 실행에서 섞지 않는다. 현재 inventory가 가리키는 배포 산출물은 다음과 같다.
|
||||
|
||||
| Artifact | Path |
|
||||
|---|---|
|
||||
| native Edge config | `build/dev-runtime/edge.yaml` |
|
||||
| Edge binary | `build/dev-runtime/bin/edge` |
|
||||
| macOS Node binary | `build/dev-runtime/bin/iop-node` |
|
||||
| Linux ARM64 Node binary | `build/dev-runtime/bin/iop-node-linux-arm64` |
|
||||
| Windows AMD64 Node binary | `build/dev-runtime/bin/iop-node-windows-amd64.exe` |
|
||||
|
||||
모든 Edge/Node binary는 scored execution 전에 동일 source ref로 rebuild·redeploy·restart한다. `build/dev-runtime/**`의 runtime config와 untracked credential material은 원격 runner가 소유하며 tracked 문서나 testbed로 복사하지 않는다.
|
||||
|
||||
## 4. 보호 파일과 credential 역할
|
||||
|
||||
benchmark runner의 `token/` 아래에는 다음 파일이 준비돼 있다. 파일 존재와 mode만 확인하며 내용을 출력하지 않는다.
|
||||
|
||||
| Path | 역할 | 실행 시 사용 |
|
||||
|---|---|---|
|
||||
| `token/.iop-bench` | benchmark용 IOP principal token | preflight/run/score caller가 Edge를 인증할 때 사용 |
|
||||
| `token/iop-dev-ca.pem` | managed dev Edge HTTPS CA certificate | `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`로 전달 |
|
||||
| `token/.claude` | Claude provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
| `token/.gemini` | Gemini provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
| `token/.gpt` | GPT provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
|
||||
현재 secret source 파일은 `0600`, CA certificate는 `0644`로 확인됐다. CA certificate는 public trust material이지만 private key는 아니다.
|
||||
|
||||
안전 확인:
|
||||
|
||||
```bash
|
||||
for benchmark_secret_file in token/.iop-bench token/.claude token/.gemini token/.gpt; do
|
||||
test -f "$benchmark_secret_file"
|
||||
test "$(stat -c '%a' "$benchmark_secret_file")" = 600
|
||||
done
|
||||
test -f token/iop-dev-ca.pem
|
||||
```
|
||||
|
||||
macOS에서 동일 검사를 수행할 때는 BSD `stat` 문법을 사용한다. 어떤 경우에도 `cat`, `echo`, shell tracing(`set -x`)으로 secret 내용을 출력하지 않는다.
|
||||
|
||||
## 5. Provider와 managed credential 설정
|
||||
|
||||
현재 dev 구성은 다음 절차로 만들어졌다.
|
||||
|
||||
1. Control Plane credential plane용 CA, workload certificate, at-rest keyring과 lease issuer/recipient key를 operator-owned untracked 경로에 생성했다.
|
||||
2. Control Plane, Edge와 각 Node에 role/name-bound mTLS identity를 배치했다.
|
||||
3. Edge public ingress를 HTTPS로 구성하고 benchmark runner에 CA certificate만 전달했다.
|
||||
4. benchmark principal을 bootstrap하고 one-time token을 `token/.iop-bench`에 저장했다.
|
||||
5. `token/.claude`, `token/.gemini`, `token/.gpt`의 raw provider key를 credential HTTPS request body로 직접 등록했다. command argument, YAML, tracked docs나 task evidence에는 넣지 않았다.
|
||||
6. credential slot과 public route를 별도로 생성하고 principal projection에 direct route와 hybrid preset stage authorization을 연결했다.
|
||||
7. Control Plane → Edge → Nodes 순서로 bounded restart하고 fresh projection, sealed lease, route revision과 no-fallback 동작을 검증했다.
|
||||
|
||||
재구성이 필요하면 `docs/edge-local-dev-guide.md`의 “Managed credential plane and TLS startup”과 “Safe slot lifecycle”을 따른다. 실제 slot id, alias, revision과 lease id는 운영 상태이므로 이 문서에 복사하지 않는다.
|
||||
|
||||
## 6. Caller별 연결 방식
|
||||
|
||||
| Caller | Edge surface | Child 설정 | 중요한 제한 |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic-compatible Messages | `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`; `--bare --print --verbose --output-format stream-json --no-session-persistence --permission-mode dontAsk --tools Read,Write,Edit --allowedTools Read,Write,Edit` | task는 stdin으로 한 번 제출한다. network/shell 도구 없이 격리 workspace 파일 작업만 허용한다. model/effort는 manifest 값을 그대로 전달한다. |
|
||||
| agy | Gemini-native `streamGenerateContent` | fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`; `--sandbox --output-format stream-json --model ... --print ...` | `agy 1.1.12`만 승인된다. API-key provider에 `--effort`를 전달하지 않으며 ambient user config를 읽지 않는다. |
|
||||
| Codex | OpenAI-compatible Responses | fresh session `HOME`, isolated `iop_benchmark` provider override, `IOP_BENCHMARK_API_KEY`; `exec --sandbox workspace-write --json --ephemeral --ignore-user-config --strict-config` | user config를 읽지 않고 격리 workspace에만 쓸 수 있다. current adapter effort는 `xhigh`; caller binding event가 없으면 admitted config binding을 사용하고, 보고된 mismatch는 거부한다. |
|
||||
|
||||
세 caller child에는 필요한 `PATH`와 CA 변수만 allowlist로 전달한다. parent의 caller/provider 설정이나 unrelated secret은 상속하지 않는다.
|
||||
|
||||
agy의 public base는 adapter가 cell별로 다음처럼 route-qualified 한다.
|
||||
|
||||
```text
|
||||
<edge-origin>/gemini/<route-id>
|
||||
```
|
||||
|
||||
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 benchmark transport가 아니다.
|
||||
|
||||
## 7. Benchmark 고정 설정
|
||||
|
||||
| 설정 | 값 |
|
||||
|---|---|
|
||||
| pipeline version | `2` |
|
||||
| environment | `dev` |
|
||||
| execution seed | `bench-02-c01-c09-v1` |
|
||||
| repetitions | `1` |
|
||||
| session policy | `fresh` |
|
||||
| setup cache policy | `isolated` |
|
||||
| run timeout | 300 seconds |
|
||||
| idle timeout | 30 seconds |
|
||||
| quiet window | 10 seconds |
|
||||
| cleanup grace | 5 seconds |
|
||||
| desktop viewport | `1920x1080` |
|
||||
| mobile viewport | `375x812` |
|
||||
| output root | `agent-test/runs/bench-02` |
|
||||
| rubric | `one-shot-agent-comparison-v1` |
|
||||
|
||||
execution seed로 결정되는 현재 slot 순서는 다음과 같다. 표의 C 번호 순서와 실제 실행 순서는 다르다.
|
||||
|
||||
1. C02 Claude→Gemini direct
|
||||
2. C05 Codex→GPT direct
|
||||
3. C03 agy→Gemini direct
|
||||
4. C06 Claude→Gemini hybrid
|
||||
5. C08 Claude→GPT hybrid
|
||||
6. C09 Codex→GPT hybrid
|
||||
7. C01 Claude→Sonnet direct
|
||||
8. C07 agy→Gemini hybrid
|
||||
9. C04 Claude→GPT direct
|
||||
|
||||
## 8. C01-C09 matrix와 route binding
|
||||
|
||||
| Cell | Caller | Route kind/id | Requested model/effort | Effective stage binding |
|
||||
|---|---|---|---|---|
|
||||
| C01 | Claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / `max` | request=`claude-sonnet-5` max |
|
||||
| C02 | Claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
|
||||
| C03 | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
|
||||
| C04 | Claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
|
||||
| C05 | Codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
|
||||
| C06 | Claude | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
|
||||
| C07 | agy | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
|
||||
| C08 | Claude | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
|
||||
| C09 | Codex | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
|
||||
|
||||
hybrid preset은 caller가 stage를 따로 호출하는 구조가 아니다. 하나의 caller request 안에서 Edge가 selector/plan/work/review/repair를 소유한다.
|
||||
|
||||
## 9. Fixture와 결과 조건
|
||||
|
||||
공통 task는 fictional product “Lumen Atlas”의 responsive one-page landing page다.
|
||||
|
||||
- prompt: `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- copy: `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- images: `aurora-grid.svg`, `orbit-rings.svg`
|
||||
- 생성 파일: workspace root의 `index.html`, `styles.css`, `script.js` 정확히 세 개
|
||||
- 외부 asset, framework, package manager, build tool, analytics와 network dependency 금지
|
||||
- desktop/mobile responsive, semantic HTML, focus/contrast/accessibility 요구
|
||||
- 각 attempt는 fresh workspace/session에서 task를 한 번만 제출
|
||||
|
||||
fixture checksum은 manifest의 값이 유일한 기준이다. prompt나 asset을 변경하면 기존 run과 비교하지 말고 manifest/version/checksum을 함께 갱신하는 별도 작업으로 처리한다.
|
||||
|
||||
## 10. Process environment 준비
|
||||
|
||||
다음은 value를 출력하지 않는 process-local 예시다. `<edge-host>`를 문서에 실제 값으로 치환하지 말고 실행 환경에서만 주입한다.
|
||||
|
||||
Public live registry가 소비하는 environment contract는 다음과 같다.
|
||||
|
||||
| Variable | 값/의미 | Durable evidence |
|
||||
|---|---|---|
|
||||
| `IOP_BENCH_CLAUDE_BASE_URL` | managed Edge HTTPS origin | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_AGY_BASE_URL` | managed Edge HTTPS origin; adapter가 `/gemini/<route-id>`를 추가 | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_CODEX_BASE_URL` | managed Edge OpenAI-compatible `/v1` base | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_CLAUDE_SECRET_ENV` | Claude가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_AGY_SECRET_ENV` | agy가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_CODEX_SECRET_ENV` | Codex가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_SHARED_TOKEN` | 이 가이드 예시의 secret-bearing variable | 값 기록 금지 |
|
||||
| `SSL_CERT_FILE` | dev Edge HTTPS CA certificate path | repository-relative file reference만 허용 |
|
||||
| `NODE_EXTRA_CA_CERTS` | Node.js caller용 동일 CA certificate path | repository-relative file reference만 허용 |
|
||||
| `IOP_BENCH_CONFIG_OBSERVATION_ENV` | config JSON을 보유한 variable 이름 | variable 이름만 허용 |
|
||||
| `BENCH_CONFIG` | schema v1 route/model/stage observation JSON | secret은 없지만 runtime과 일치하는 canonical digest만 evidence에 기록 |
|
||||
| `PATH` | caller binary resolution | resolved executable path/version만 preflight에서 확인 |
|
||||
|
||||
Child adapter가 내부적으로 만드는 값은 caller별로 격리된다.
|
||||
|
||||
- Claude: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, restricted `Read,Write,Edit`, nonessential traffic/autoupdater disable flags와 CA variables
|
||||
- agy: fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `LANG=C.UTF-8`, `LC_ALL=C.UTF-8`, `TZ=UTC`와 CA variables
|
||||
- Codex: fresh session `HOME`, `IOP_BENCHMARK_API_KEY`, strict ephemeral provider override와 CA variables
|
||||
|
||||
이 child variable은 사용자가 별도로 준비할 값이 아니다. live registry가 위 public contract에서 파생하며, parent의 같은 이름 값을 그대로 신뢰하거나 상속하지 않는다.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
|
||||
benchmark_edge_origin="${BENCHMARK_EDGE_ORIGIN:?set BENCHMARK_EDGE_ORIGIN to the managed dev Edge HTTPS origin}"
|
||||
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
|
||||
export IOP_BENCH_SHARED_TOKEN
|
||||
|
||||
export IOP_BENCH_CLAUDE_BASE_URL="$benchmark_edge_origin"
|
||||
export IOP_BENCH_AGY_BASE_URL="$benchmark_edge_origin"
|
||||
export IOP_BENCH_CODEX_BASE_URL="$benchmark_edge_origin/v1"
|
||||
|
||||
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
|
||||
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
|
||||
BENCH_CONFIG="$(python3 - <<'PY'
|
||||
import json
|
||||
|
||||
routes = [
|
||||
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
|
||||
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
|
||||
]
|
||||
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
|
||||
PY
|
||||
)"
|
||||
export BENCH_CONFIG
|
||||
```
|
||||
|
||||
중요한 의미:
|
||||
|
||||
- `IOP_BENCH_*_SECRET_ENV`의 값은 secret이 아니라 실제 secret을 보유한 environment variable의 이름이다.
|
||||
- 세 caller는 같은 IOP principal을 사용하지만 서로 다른 protocol base를 받는다.
|
||||
- `BENCH_CONFIG`는 runtime에서 독립적으로 확인한 route/stage snapshot이어야 한다. manifest를 보고 임의 합성한 값을 live readiness evidence로 사용하면 안 된다.
|
||||
- endpoint와 config는 evidence에 raw 값 대신 digest identity로만 남는다.
|
||||
|
||||
작업 후에는 같은 shell에서 다음 변수를 제거한다.
|
||||
|
||||
```bash
|
||||
unset IOP_BENCH_SHARED_TOKEN
|
||||
unset IOP_BENCH_CLAUDE_BASE_URL IOP_BENCH_AGY_BASE_URL IOP_BENCH_CODEX_BASE_URL
|
||||
unset IOP_BENCH_CLAUDE_SECRET_ENV IOP_BENCH_AGY_SECRET_ENV IOP_BENCH_CODEX_SECRET_ENV
|
||||
unset IOP_BENCH_CONFIG_OBSERVATION_ENV BENCH_CONFIG
|
||||
unset SSL_CERT_FILE NODE_EXTRA_CA_CERTS
|
||||
unset BENCHMARK_EDGE_ORIGIN benchmark_edge_origin
|
||||
```
|
||||
|
||||
## 11. 실행 절차
|
||||
|
||||
manifest path는 모든 명령에서 동일하게 사용한다.
|
||||
|
||||
```bash
|
||||
benchmark_manifest=scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
### 11.1 Source와 deterministic verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest "$benchmark_manifest"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
필요하면 변경 범위에 따라 Go tests와 managed credential qualification도 실행한다.
|
||||
|
||||
```bash
|
||||
go test -count=1 ./...
|
||||
credential_smoke_parent="$(mktemp -d /tmp/iop-benchmark-credential.XXXXXX)"
|
||||
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
|
||||
rmdir "$credential_smoke_parent"
|
||||
```
|
||||
|
||||
### 11.2 Public preflight
|
||||
|
||||
10절의 환경을 같은 shell에 준비한 뒤 실행한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest "$benchmark_manifest"
|
||||
```
|
||||
|
||||
성공 조건:
|
||||
|
||||
```text
|
||||
status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
```
|
||||
|
||||
preflight는 다음을 함께 확인한다.
|
||||
|
||||
- caller binary/version/help
|
||||
- Edge model catalog
|
||||
- principal auth와 endpoint compatibility
|
||||
- direct/preset route 존재
|
||||
- requested model/effort
|
||||
- exact stage binding과 order
|
||||
- official agy transport capability
|
||||
|
||||
`registration_required` 또는 `implementation_gap`이면 즉시 중단한다. alias, model, effort, route나 caller를 대체하지 않는다. preflight-only run root는 evidence이므로 삭제하지 않는다.
|
||||
|
||||
### 11.3 Scored execution
|
||||
|
||||
fresh preflight와 명시적 실행 권한이 있는 현재 plan에서만 다음 명령을 한 번 호출한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest "$benchmark_manifest"
|
||||
```
|
||||
|
||||
- direct CLI stdout/stderr와 exit code를 보존한다.
|
||||
- CLI가 출력한 canonical `run-...` id만 이후 `status`, `score`, `report`에 사용한다.
|
||||
- command가 nonzero여도 같은 plan에서 `run`을 다시 호출하지 않는다.
|
||||
- CLI가 run id를 출력하지 않으면 임의 id나 성공 pointer를 만들지 않는다.
|
||||
- caller나 provider를 CLI 밖에서 별도로 호출해 scored result를 보충하지 않는다.
|
||||
|
||||
### 11.4 Status
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py status \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
현재 comparison execution 완료 조건:
|
||||
|
||||
- `success + failed + timed_out + cancelled = 9`
|
||||
- `running = 0`
|
||||
- `interrupted = 0`
|
||||
- 각 cell/repetition에 retained terminal attempt가 존재
|
||||
|
||||
실패/timed-out/cancelled는 terminal evidence지만 성공 결과가 아니므로 scoring eligibility와 최종 비교에서 별도로 표시된다.
|
||||
|
||||
### 11.5 Blind scoring
|
||||
|
||||
유효한 execution run에 대해서만 수행한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py score \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
evaluator는 Codex→`gpt-5.6-luna` xhigh direct route다. identity가 제거된 blind workspace만 보며 source cell identity mapping은 blind tree 밖에 유지한다.
|
||||
|
||||
automatic gate 실패나 lifecycle failure는 `unscored`이고 0점으로 바꾸지 않는다. `scoring_failed`도 명시적 `--retry-scoring-failed` 권한 없이 재시도하지 않는다.
|
||||
|
||||
### 11.6 Report
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py report \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
report는 run root의 immutable evidence를 읽어 idempotent `report.md`를 만든다. 기존 report 내용과 새 projection이 다르면 덮어쓰지 않고 실패한다.
|
||||
|
||||
## 12. 수집 evidence
|
||||
|
||||
각 attempt는 다음 범주의 evidence를 가진다.
|
||||
|
||||
| 범주 | 내용 |
|
||||
|---|---|
|
||||
| lifecycle | submission, first output, finish, idle, exit, quiet, cleanup와 terminal reason |
|
||||
| timeline | submitted, first output, first workspace write observation/mtime, total duration |
|
||||
| usage | input/output/reasoning/cache read/cache write/total tokens, model/tool calls와 duration |
|
||||
| workspace | fresh session identity, fixture checksum, testbed provenance, generated file tree |
|
||||
| web validation | generated files, static safety, images, network, console, responsive, accessibility |
|
||||
| screenshots | desktop `1920x1080`, mobile `375x812` |
|
||||
| scoring | eligibility, blind allocation, rubric worksheet, score status |
|
||||
|
||||
caller가 보고하지 않은 metric은 0으로 만들지 않고 `unavailable`과 reason/source를 보존한다.
|
||||
|
||||
Codex current usage mapping:
|
||||
|
||||
| Caller JSONL field | Canonical metric |
|
||||
|---|---|
|
||||
| `input_tokens` | `input_tokens` |
|
||||
| `cached_input_tokens` | `cached_input_tokens` |
|
||||
| `cache_write_input_tokens` | `cache_write_tokens` |
|
||||
| `output_tokens` | `output_tokens` |
|
||||
| `reasoning_output_tokens` | `reasoning_tokens` |
|
||||
| `total_tokens` | `total_tokens` |
|
||||
|
||||
누락된 `total_tokens`는 하위 category 합으로 재구성하지 않는다.
|
||||
|
||||
## 13. 자동 gate와 100점 rubric
|
||||
|
||||
automatic web gates는 scoring eligibility만 결정하고 점수에 포함되지 않는다.
|
||||
|
||||
| Gate |
|
||||
|---|
|
||||
| generated files |
|
||||
| static safety |
|
||||
| local images |
|
||||
| no external network dependency |
|
||||
| console safety |
|
||||
| responsive layout |
|
||||
| accessibility |
|
||||
|
||||
quality rubric:
|
||||
|
||||
| Category | Max |
|
||||
|---|---:|
|
||||
| requirements fidelity | 25 |
|
||||
| visual completeness | 25 |
|
||||
| responsive accessibility | 15 |
|
||||
| image/detail usage | 10 |
|
||||
| behavior stability | 10 |
|
||||
| code quality | 10 |
|
||||
| self verification | 5 |
|
||||
| Total | 100 |
|
||||
|
||||
## 14. 실패 처리와 재개 원칙
|
||||
|
||||
| 상황 | 조치 |
|
||||
|---|---|
|
||||
| manifest invalid | source/manifest를 수정하고 validate부터 다시 시작한다. run을 만들지 않는다. |
|
||||
| preflight not ready | blocker를 해결하고 fresh preflight한다. attempt를 할당하지 않는다. |
|
||||
| caller launch 전 interruption | retained evidence를 보존한다. run tree를 직접 수정하지 않는다. |
|
||||
| lifecycle/parser failure | exact retained output으로 source 원인을 수정하고 deterministic regression을 추가한다. |
|
||||
| terminal failed/timed_out/cancelled | evidence로 보존한다. 암묵 retry하지 않는다. |
|
||||
| state가 `running`이지만 process가 없음 | manual JSON 수정/삭제/reconcile을 하지 않는다. reviewer evidence로 남기고 승인된 새 plan에서만 다음 상태를 결정한다. |
|
||||
| scoring_failed | 0점 처리하지 않는다. 명시적 retry 권한 없이는 중단한다. |
|
||||
| report unavailable | run evidence를 수정하거나 report를 수작업 생성하지 않는다. |
|
||||
|
||||
현재 milestone의 원칙은 old incomplete/failed run을 `resume --retry-failed`하지 않고 distinct fresh run을 만드는 것이다. 일반 CLI가 `resume`을 지원한다는 사실이 현재 benchmark에서 사용 권한을 뜻하지 않는다.
|
||||
|
||||
## 15. Secret-safe 기록 규칙
|
||||
|
||||
다음 값은 tracked docs, task review, run metadata, log, metric label과 command argument에 남기지 않는다.
|
||||
|
||||
- IOP principal token
|
||||
- raw provider credential
|
||||
- private key, at-rest key, issuer/recipient private key
|
||||
- slot alias와 lease id
|
||||
- credential-bearing URL
|
||||
- raw prompt/response, tool input/output
|
||||
- caller/provider session content
|
||||
|
||||
허용되는 내용:
|
||||
|
||||
- secret file의 상대 path와 존재/mode
|
||||
- route/model/stage 이름
|
||||
- safe credential slot reference/revision
|
||||
- hashed endpoint/config/spec identity
|
||||
- redacted lifecycle 상태, duration과 usage count
|
||||
- run id와 attempt identity가 필요한 controller evidence
|
||||
|
||||
incident evidence를 보존하기 전에 repository와 run output에서 secret 원문이 없는지 확인한다. 의심되는 artifact는 내용을 복사하지 말고 path와 redaction failure만 보고한다.
|
||||
|
||||
## 16. 실행 전 체크리스트
|
||||
|
||||
- [ ] 현재 manifest validation 통과
|
||||
- [ ] focused/full deterministic tests fresh PASS
|
||||
- [ ] benchmark runner와 `../iop-s2` provenance 확인
|
||||
- [ ] Claude/agy/Codex command와 current version/help 확인
|
||||
- [ ] `token/.iop-bench`, CA와 provider provisioning source 존재/mode 확인; 내용 출력 없음
|
||||
- [ ] dev runtime source/build identity, process와 listener 확인
|
||||
- [ ] managed projection, provider slot/route와 no-legacy-fallback 확인
|
||||
- [ ] config observation이 runtime route/stage와 정확히 일치
|
||||
- [ ] public preflight `ready=9`
|
||||
- [ ] 현재 plan이 exactly one scored run을 소유하고 사용자 권한이 명확함
|
||||
- [ ] old run resume/retry/state edit 계획 없음
|
||||
- [ ] run 이후 status, scoring, report의 run id 전달 경로 준비
|
||||
|
||||
## 17. 관련 구현
|
||||
|
||||
- controller/state: `scripts/agent_benchmark/attempts.py`
|
||||
- manifest: `scripts/agent_benchmark/manifest.py`
|
||||
- lifecycle: `scripts/agent_benchmark/lifecycle.py`
|
||||
- workspace isolation: `scripts/agent_benchmark/workspace.py`
|
||||
- live routing/admission: `scripts/agent_benchmark/live_iop.py`
|
||||
- Claude adapter: `scripts/agent_benchmark/claude_iop.py`
|
||||
- agy adapter: `scripts/agent_benchmark/agy_iop.py`
|
||||
- Codex adapter: `scripts/agent_benchmark/codex_iop.py`
|
||||
- measurement: `scripts/agent_benchmark/measurement.py`
|
||||
- browser/web gate: `scripts/agent_benchmark/web_validation.py`
|
||||
- blind scoring: `scripts/agent_benchmark/scoring.py`
|
||||
- rubric: `scripts/agent_benchmark/rubric.py`
|
||||
- report: `scripts/agent_benchmark/reporting.py`
|
||||
|
|
@ -253,6 +253,31 @@ rmdir "$credential_smoke_parent"
|
|||
|
||||
The deterministic Messages qualification succeeds alongside Chat: the Control Plane canonicalizes built-in lowercase API-key header names (for example `x-api-key` to `X-Api-Key`) before signing the lease scope, so both managed profiles reach Node/upstream exactly once with their exact header semantics. A lease failure fails closed before dispatch and never falls back to caller auth or another slot; treat a Chat-only result or any fallback as a qualification failure.
|
||||
|
||||
### Official agy route smoke
|
||||
|
||||
공식 `agy` 1.1.12 API-key provider는 upstream Gemini key가 아니라 관리형 IOP principal token을 사용한다. dev operator가 이미 발급한 token과 CA 파일을 보호된 `token/` 아래에 둔 경우 값을 명령행에 직접 쓰지 않고 다음처럼 읽는다.
|
||||
|
||||
```bash
|
||||
read -r IOP_BENCH_TOKEN < token/.iop-bench
|
||||
export GEMINI_API_KEY="$IOP_BENCH_TOKEN"
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
|
||||
GOOGLE_GEMINI_BASE_URL="https://<edge-host>:<https-port>/gemini/<direct-route-id>" \
|
||||
agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \
|
||||
--print 'Reply only with OK. Do not use tools or modify files.'
|
||||
|
||||
GOOGLE_GEMINI_BASE_URL="https://<edge-host>:<https-port>/gemini/<hybrid-preset-id>" \
|
||||
agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \
|
||||
--print 'Inspect README.md and report only its first Markdown heading. Do not modify files.'
|
||||
|
||||
unset IOP_BENCH_TOKEN GEMINI_API_KEY SSL_CERT_FILE NODE_EXTRA_CA_CERTS
|
||||
```
|
||||
|
||||
`--effort`는 API-key provider 호출에 넣지 않는다. direct와 hybrid 모두 JSONL의 마지막 record가 `event=result`, 중첩 `result.status=SUCCESS` 한 건이어야 한다. hybrid는 plan/work/review가 포함되므로 direct보다 오래 걸릴 수 있으며, caller timeout을 이유로 같은 scored attempt를 재실행하지 않는다. Gemini ingress와 공식 event 구조의 상세 계약은 `agent-contract/outer/gemini-compatible-api.md`를 기준으로 한다.
|
||||
|
||||
benchmark 전체 preflight에서는 Claude/agy/Codex에 같은 IOP principal을 secret environment reference로 연결하고 `IOP_BENCH_CONFIG_OBSERVATION_ENV`가 가리키는 operator-owned route/binding observation을 함께 제공한다. 사설 CA 환경은 각 isolated caller child에 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`로 전달된다. preflight가 모든 cell을 `ready`로 판정하기 전에는 scored `run`을 시작하지 않는다.
|
||||
|
||||
### Incident redaction check
|
||||
|
||||
Before retaining logs or evidence, reject any artifact containing an IOP bearer token, provider credential, slot alias, lease id, certificate private key, keyring material, recipient/issuer private key, target URL with credentials, prompt, or response body. Public metrics may contain only stable safe references such as `credential_slot_ref` and `credential_revision`; request/run/session/attempt/node ids and raw payloads are not credential-attribution labels.
|
||||
|
|
|
|||
|
|
@ -244,10 +244,20 @@ func ParsePlan(tmpl string, rawOutput string, maxOutputBytes int) ([]byte, error
|
|||
if f3 == "" {
|
||||
vVerif = rem
|
||||
} else {
|
||||
if !strings.HasSuffix(rem, f3) {
|
||||
return nil, ErrMalformedPlan
|
||||
suffix := f3
|
||||
if !strings.HasSuffix(rem, suffix) {
|
||||
// Provider chat APIs commonly omit the model's final line feed. Treat
|
||||
// only that last byte as optional; all other static suffix text must
|
||||
// still match the configured template exactly.
|
||||
if !strings.HasSuffix(f3, "\n") {
|
||||
return nil, ErrMalformedPlan
|
||||
}
|
||||
suffix = strings.TrimSuffix(f3, "\n")
|
||||
if !strings.HasSuffix(rem, suffix) {
|
||||
return nil, ErrMalformedPlan
|
||||
}
|
||||
}
|
||||
vVerif = rem[:len(rem)-len(f3)]
|
||||
vVerif = rem[:len(rem)-len(suffix)]
|
||||
}
|
||||
|
||||
trimmedGoal := strings.TrimSpace(vGoal)
|
||||
|
|
|
|||
|
|
@ -290,6 +290,63 @@ Fix single-request template handling bug.
|
|||
maxOutputBytes: 1024,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "provider omits final line feed",
|
||||
tmpl: singlerequesttemplate.DefaultPlanTemplate,
|
||||
raw: strings.TrimSuffix(validOutput, "\n"),
|
||||
maxOutputBytes: 1024,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "provider omits only final line feed after static suffix",
|
||||
tmpl: `# Plan
|
||||
|
||||
## Goal
|
||||
{{goal}}
|
||||
|
||||
## Steps
|
||||
{{steps}}
|
||||
|
||||
## Verification
|
||||
{{verification}}
|
||||
|
||||
END
|
||||
`,
|
||||
raw: `# Plan
|
||||
|
||||
## Goal
|
||||
Fix suffix parsing.
|
||||
|
||||
## Steps
|
||||
- Keep the static suffix.
|
||||
- Allow the final line feed omission.
|
||||
|
||||
## Verification
|
||||
- Run the parser tests.
|
||||
|
||||
END`,
|
||||
maxOutputBytes: 1024,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "provider omits static suffix text",
|
||||
tmpl: `# Plan
|
||||
|
||||
## Goal
|
||||
{{goal}}
|
||||
|
||||
## Steps
|
||||
{{steps}}
|
||||
|
||||
## Verification
|
||||
{{verification}}
|
||||
|
||||
END
|
||||
`,
|
||||
raw: strings.TrimSuffix(validOutput, "\n"),
|
||||
maxOutputBytes: 1024,
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
name: "boundary steps = 6, verif = 3",
|
||||
tmpl: singlerequesttemplate.DefaultPlanTemplate,
|
||||
|
|
|
|||
|
|
@ -22,48 +22,48 @@ from scripts.agent_benchmark.connectivity import (
|
|||
CallerCapability,
|
||||
ConnectivityResult,
|
||||
ConnectivityIssue,
|
||||
EffectiveBinding,
|
||||
RequestedEffectiveBinding,
|
||||
classify_issues,
|
||||
make_result,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
COMPLETION_EXIT_AFTER_IDLE,
|
||||
SUBMISSION_STDIN_ONCE,
|
||||
SUBMISSION_ARGV_TASK,
|
||||
InvocationResult,
|
||||
InvocationSpec,
|
||||
LifecycleMetricError,
|
||||
ParsedMetric,
|
||||
SupervisorLocator,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
env_pairs,
|
||||
exact_value_redactor,
|
||||
is_reported_number,
|
||||
inherited_tls_ca_environment,
|
||||
run_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, TOKEN_RE, Timeout
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import PreparedWorkspace
|
||||
|
||||
|
||||
AGY_CALLER = "agy"
|
||||
AGY_KNOWN_VERSION = "1.1.11"
|
||||
AGY_PROVIDER_ENV = "AGY_PROVIDER"
|
||||
AGY_ENDPOINT_ENV = "AGY_OPENAI_BASE_URL"
|
||||
AGY_AUTH_ENV = "AGY_OPENAI_API_KEY"
|
||||
_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE)
|
||||
_SAFE_EVENT_FIELDS = ("type", "subtype", "model", "effort", "route_kind", "route_id")
|
||||
_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model", "--effort")
|
||||
_DOCUMENTED_ENVIRONMENT = (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV)
|
||||
AGY_SAFE_METRIC_LABELS = ("metric:duration_ms",)
|
||||
# agy reports whole durations in milliseconds. Only these subtypes are
|
||||
# converted, and a model-stage duration is marked as overlapping because it is
|
||||
# reported inside the same window as the total.
|
||||
AGY_DURATION_METRICS = {
|
||||
"duration_ms": ("total_duration", False),
|
||||
"model_duration_ms": ("model_duration", True),
|
||||
"queue_duration_ms": ("queue_duration", False),
|
||||
AGY_KNOWN_VERSION = "1.1.12"
|
||||
AGY_ENDPOINT_ENV = "GOOGLE_GEMINI_BASE_URL"
|
||||
AGY_AUTH_ENV = "GEMINI_API_KEY"
|
||||
AGY_MODEL_LABELS = {
|
||||
"gemini-3.6-flash": "Gemini 3.6 Flash",
|
||||
"gemini-hybrid": "Gemini 3.6 Flash",
|
||||
}
|
||||
_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE)
|
||||
_SAFE_EVENT_FIELDS = ("event", "state", "step_type", "status")
|
||||
_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--sandbox", "--model")
|
||||
_AGY_USAGE_METRICS = {
|
||||
"input_tokens": "input_tokens",
|
||||
"cache_read_tokens": "cached_input_tokens",
|
||||
"output_tokens": "output_tokens",
|
||||
"thinking_tokens": "reasoning_tokens",
|
||||
"total_tokens": "total_tokens",
|
||||
}
|
||||
_AGY_METRIC_KEYS = frozenset({"type", "subtype", "value", "model"})
|
||||
_IDENTITY_RE = re.compile(r"sha256:[0-9a-f]{64}\Z")
|
||||
|
||||
|
||||
|
|
@ -162,7 +162,7 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili
|
|||
return AgyDocumentedCapabilities((), (), ())
|
||||
return AgyDocumentedCapabilities(
|
||||
tuple(token for token in _DOCUMENTED_OPTIONS if _exact_token_present(help_output, token)),
|
||||
tuple(token for token in _DOCUMENTED_ENVIRONMENT if _exact_token_present(help_output, token)),
|
||||
(),
|
||||
("stream-json",) if _exact_token_present(help_output, "stream-json") else (),
|
||||
)
|
||||
|
||||
|
|
@ -170,9 +170,9 @@ def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabili
|
|||
def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapability:
|
||||
"""Inspect only public, versioned help text for the closed IOP transport.
|
||||
|
||||
A version string is accepted only when it names the known agy release and
|
||||
every required transport variable is documented. This prevents a new or
|
||||
partially documented client from silently inheriting ambient provider state.
|
||||
The known release is pinned because its Gemini provider environment is not
|
||||
printed by ``--help``. The public options and stream format still have to
|
||||
match exactly; a changed release fails closed until re-qualified.
|
||||
"""
|
||||
if not isinstance(version_output, str) or not isinstance(help_output, str):
|
||||
return AgyCapability(None, False, False, False, False, False, (), ())
|
||||
|
|
@ -180,11 +180,11 @@ def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapa
|
|||
version = matched.group(1) if matched else None
|
||||
known_version = version == AGY_KNOWN_VERSION
|
||||
documented = parse_documented_agy_capabilities(help_output)
|
||||
endpoint_supported = AGY_ENDPOINT_ENV in documented.environment
|
||||
auth_supported = AGY_AUTH_ENV in documented.environment
|
||||
endpoint_supported = known_version
|
||||
auth_supported = known_version
|
||||
protocol_supported = known_version and all(
|
||||
option in documented.options for option in _DOCUMENTED_OPTIONS
|
||||
) and AGY_PROVIDER_ENV in documented.environment
|
||||
)
|
||||
stream_supported = "stream-json" in documented.output_formats
|
||||
supported = endpoint_supported and auth_supported and protocol_supported and stream_supported
|
||||
if not supported:
|
||||
|
|
@ -238,7 +238,14 @@ def validate_agy_iop_runtime(
|
|||
if not all(isinstance(value, str) and value for value in (runtime.binary, runtime.endpoint, runtime.credential)):
|
||||
raise AgyAdapterError("agy runtime values are unavailable")
|
||||
endpoint = urlsplit(runtime.endpoint)
|
||||
if endpoint.scheme not in ("http", "https") or not endpoint.netloc or endpoint.query or endpoint.fragment:
|
||||
route_path = f"/gemini/{cell.iop.route_id}"
|
||||
if (
|
||||
endpoint.scheme != "https"
|
||||
or not endpoint.netloc
|
||||
or endpoint.query
|
||||
or endpoint.fragment
|
||||
or endpoint.path.rstrip("/") != route_path
|
||||
):
|
||||
raise AgyAdapterError("agy IOP endpoint is invalid")
|
||||
_validate_config_owner_observation(cell, observation)
|
||||
if observation.endpoint_identity != _runtime_identity("endpoint", runtime.endpoint):
|
||||
|
|
@ -269,7 +276,7 @@ def preflight_agy_iop(
|
|||
issues.append(_issue("endpoint_incompatible"))
|
||||
if not runtime.credential:
|
||||
issues.append(_issue("credential_missing"))
|
||||
if not cell.iop.request_model:
|
||||
if cell.iop.request_model not in AGY_MODEL_LABELS:
|
||||
issues.append(_issue("model_missing"))
|
||||
if not runtime.endpoint:
|
||||
issues.append(_issue("endpoint_incompatible"))
|
||||
|
|
@ -307,16 +314,25 @@ def build_agy_invocation(
|
|||
timeout: Timeout,
|
||||
preflight: AgyPreflightResult,
|
||||
) -> InvocationSpec:
|
||||
"""Build one isolated stdin-only agy invocation after a ready preflight."""
|
||||
"""Build one isolated official agy print invocation after a ready preflight."""
|
||||
if preflight.status != "ready" or not preflight.capability.iop_transport_supported or preflight.runtime is None:
|
||||
raise AgyAdapterError("agy IOP transport is not proven")
|
||||
runtime = preflight.runtime
|
||||
if not runtime.binary or not Path(runtime.binary).is_file():
|
||||
raise AgyAdapterError("agy binary is unavailable")
|
||||
if not isinstance(prepared, PreparedWorkspace) or not prepared.workspace_dir:
|
||||
if (
|
||||
not isinstance(prepared, PreparedWorkspace)
|
||||
or not Path(prepared.workspace_dir).is_dir()
|
||||
or not Path(prepared.session_dir).is_dir()
|
||||
or not Path(prepared.attempt_root).is_dir()
|
||||
):
|
||||
raise AgyAdapterError("prepared workspace is unavailable")
|
||||
if not isinstance(task_payload, bytes) or not task_payload:
|
||||
raise AgyAdapterError("agy task payload is unavailable")
|
||||
try:
|
||||
task_text = task_payload.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise AgyAdapterError("agy task payload must be UTF-8") from exc
|
||||
|
||||
# The child receives a minimal environment and explicit IOP-only provider
|
||||
# settings. No parent agy/Gemini config or session variable is inherited.
|
||||
|
|
@ -325,33 +341,32 @@ def build_agy_invocation(
|
|||
"LANG": "C.UTF-8",
|
||||
"LC_ALL": "C.UTF-8",
|
||||
"TZ": "UTC",
|
||||
AGY_PROVIDER_ENV: "iop-openai",
|
||||
"HOME": prepared.session_dir,
|
||||
AGY_ENDPOINT_ENV: runtime.endpoint,
|
||||
AGY_AUTH_ENV: runtime.credential,
|
||||
}
|
||||
environment.update(inherited_tls_ca_environment())
|
||||
return InvocationSpec(
|
||||
argv=(
|
||||
runtime.binary,
|
||||
"--print",
|
||||
"--sandbox",
|
||||
"--output-format", "stream-json",
|
||||
"--model", cell.iop.request_model,
|
||||
"--effort", cell.iop.requested_effort,
|
||||
"--model", AGY_MODEL_LABELS[cell.iop.request_model],
|
||||
"--print", task_text,
|
||||
),
|
||||
cwd=prepared.workspace_dir,
|
||||
env=env_pairs(environment),
|
||||
env_allowlist=(AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
env_allowlist=(AGY_ENDPOINT_ENV, AGY_AUTH_ENV, *TLS_CA_ENV_KEYS),
|
||||
submission_mode=SUBMISSION_ARGV_TASK,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=timeout,
|
||||
evidence_dir=prepared.attempt_root,
|
||||
task_payload=task_payload,
|
||||
control_dir=str(Path(prepared.attempt_root) / "agy-control"),
|
||||
)
|
||||
|
||||
|
||||
def _safe_identifier(value: Any) -> str | None:
|
||||
return value if isinstance(value, str) and TOKEN_RE.fullmatch(value) else None
|
||||
return value if isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:+-]{0,63}", value) else None
|
||||
|
||||
|
||||
def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> str:
|
||||
|
|
@ -362,115 +377,105 @@ def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> s
|
|||
return '{"event":"unparseable"}'
|
||||
if not isinstance(parsed, dict):
|
||||
return '{"event":"unparseable"}'
|
||||
safe: dict[str, str] = {}
|
||||
for field in _SAFE_EVENT_FIELDS:
|
||||
value = _safe_identifier(parsed.get(field))
|
||||
event = _safe_identifier(parsed.get("event"))
|
||||
if event is None or event in sensitive_values:
|
||||
return '{"event":"unparseable"}'
|
||||
safe: dict[str, str] = {"event": event}
|
||||
payload = parsed.get(event)
|
||||
if not isinstance(payload, dict):
|
||||
payload = {}
|
||||
for field in _SAFE_EVENT_FIELDS[1:]:
|
||||
value = _safe_identifier(payload.get(field))
|
||||
if value is not None and value not in sensitive_values:
|
||||
safe[field] = value
|
||||
if "type" not in safe or "subtype" not in safe:
|
||||
return '{"event":"unparseable"}'
|
||||
return json.dumps(safe, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
|
||||
|
||||
|
||||
class AgyEventParser:
|
||||
"""Strict stream-json parser bound to exactly one requested IOP cell."""
|
||||
"""Parse the official 1.1.12 stream while trusting binding only from config."""
|
||||
|
||||
def __init__(self, cell: MatrixCell) -> None:
|
||||
def __init__(self, cell: MatrixCell, admitted_binding: RequestedEffectiveBinding) -> None:
|
||||
if not isinstance(admitted_binding, RequestedEffectiveBinding):
|
||||
raise AgyAdapterError("agy admitted binding is invalid")
|
||||
if admitted_binding.cell_id != cell.id or admitted_binding.caller != cell.caller:
|
||||
raise AgyAdapterError("agy admitted binding mismatch")
|
||||
self._cell = cell
|
||||
self._observed_binding: RequestedEffectiveBinding | None = None
|
||||
self._binding_invalid = False
|
||||
self._admitted_binding = admitted_binding
|
||||
self._init_seen = False
|
||||
self._result_seen = False
|
||||
self._latest_usage: dict[str, Any] | None = None
|
||||
|
||||
def __call__(self, stream: str, raw_line: str) -> str | ParsedMetric | None:
|
||||
def __call__(self, stream: str, raw_line: str) -> str | ParsedMetric | tuple[Any, ...] | None:
|
||||
return self.parse(stream, raw_line)
|
||||
|
||||
def parse(self, stream: str, raw_line: str) -> str | ParsedMetric | None:
|
||||
def parse(self, stream: str, raw_line: str) -> str | ParsedMetric | tuple[Any, ...] | None:
|
||||
if stream != "stdout":
|
||||
return None
|
||||
try:
|
||||
event = json.loads(raw_line)
|
||||
item = json.loads(raw_line)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
return "malformed"
|
||||
if not isinstance(event, dict):
|
||||
if not isinstance(item, dict):
|
||||
return "malformed"
|
||||
event_type = event.get("type")
|
||||
subtype = event.get("subtype")
|
||||
if (event_type, subtype) == ("iop", "effective_binding"):
|
||||
self._observe_effective_binding(event)
|
||||
event = item.get("event")
|
||||
payload = item.get(event) if isinstance(event, str) else None
|
||||
if not isinstance(payload, dict):
|
||||
return "malformed"
|
||||
if event == "init":
|
||||
if self._init_seen or self._result_seen:
|
||||
return "malformed"
|
||||
self._init_seen = True
|
||||
return None
|
||||
if event_type == "metric":
|
||||
return self._observe_duration(event, subtype)
|
||||
if event_type == "result" and subtype == "error":
|
||||
# Quota/provider errors can never be interpreted as finish/idle.
|
||||
return "quota_error" if event.get("reason") == "quota" else "malformed"
|
||||
terminal = (
|
||||
"finish" if (event_type, subtype) == ("result", "success")
|
||||
else "idle" if (event_type, subtype) == ("system", "idle")
|
||||
else None
|
||||
)
|
||||
if terminal is None or not self._matches_exact_binding(event):
|
||||
if event == "step_update":
|
||||
if not self._init_seen or self._result_seen:
|
||||
return "malformed"
|
||||
usage = payload.get("usage")
|
||||
if usage is not None:
|
||||
if self._usage_metrics(usage) is None:
|
||||
return "malformed"
|
||||
self._latest_usage = usage
|
||||
return None
|
||||
if event != "result" or not self._init_seen or self._result_seen:
|
||||
return "malformed"
|
||||
return terminal
|
||||
|
||||
def _observe_duration(self, event: dict[str, Any], subtype: Any) -> str | ParsedMetric:
|
||||
"""Convert one allowlisted agy duration losslessly, or fail closed."""
|
||||
mapped = AGY_DURATION_METRICS.get(subtype) if isinstance(subtype, str) else None
|
||||
value = event.get("value")
|
||||
if (
|
||||
mapped is None
|
||||
or not set(event) <= _AGY_METRIC_KEYS
|
||||
or not is_reported_number(value)
|
||||
or ("model" in event and event["model"] != self._cell.iop.request_model)
|
||||
):
|
||||
self._result_seen = True
|
||||
if payload.get("status") != "SUCCESS":
|
||||
return "malformed"
|
||||
name, overlap = mapped
|
||||
metrics: list[ParsedMetric] = []
|
||||
usage = payload.get("usage", self._latest_usage)
|
||||
if usage is not None:
|
||||
parsed_usage = self._usage_metrics(usage)
|
||||
if parsed_usage is None:
|
||||
return "malformed"
|
||||
metrics.extend(parsed_usage)
|
||||
try:
|
||||
return duration_metric(
|
||||
name, value, reported_unit="ms",
|
||||
model=self._cell.iop.request_model, overlap=overlap,
|
||||
)
|
||||
if "duration_seconds" in payload:
|
||||
metrics.append(duration_metric(
|
||||
"total_duration", payload["duration_seconds"], reported_unit="s",
|
||||
model=self._cell.iop.request_model,
|
||||
))
|
||||
if "num_turns" in payload:
|
||||
metrics.append(count_metric(
|
||||
"model_calls", payload["num_turns"], model=self._cell.iop.request_model,
|
||||
))
|
||||
except LifecycleMetricError:
|
||||
return "malformed"
|
||||
return tuple(metrics) + ("finish", "idle")
|
||||
|
||||
def _matches_exact_binding(self, event: dict[str, Any]) -> bool:
|
||||
expected = self._cell.iop
|
||||
return (
|
||||
event.get("model") == expected.request_model
|
||||
and event.get("effort") == expected.requested_effort
|
||||
and event.get("route_kind") == expected.route_kind
|
||||
and event.get("route_id") == expected.route_id
|
||||
)
|
||||
|
||||
def _observe_effective_binding(self, event: dict[str, Any]) -> None:
|
||||
expected_keys = {"type", "subtype", "route_kind", "route_id", "model", "effort", "stages"}
|
||||
if set(event) != expected_keys or self._observed_binding is not None:
|
||||
self._binding_invalid = True
|
||||
return
|
||||
values = tuple(event[key] for key in ("route_kind", "route_id", "model", "effort"))
|
||||
stages = event.get("stages")
|
||||
if not all(isinstance(value, str) and TOKEN_RE.fullmatch(value) for value in values) or not isinstance(stages, list):
|
||||
self._binding_invalid = True
|
||||
return
|
||||
parsed_stages: list[EffectiveBinding] = []
|
||||
for stage in stages:
|
||||
if not isinstance(stage, dict) or set(stage) != {"stage", "model", "effort"}:
|
||||
self._binding_invalid = True
|
||||
return
|
||||
if not isinstance(stage["stage"], str) or not isinstance(stage["model"], str):
|
||||
self._binding_invalid = True
|
||||
return
|
||||
if stage["effort"] is not None and not isinstance(stage["effort"], str):
|
||||
self._binding_invalid = True
|
||||
return
|
||||
parsed_stages.append(EffectiveBinding(stage["stage"], stage["model"], stage["effort"]))
|
||||
self._observed_binding = RequestedEffectiveBinding(
|
||||
self._cell.id, self._cell.caller,
|
||||
self._cell.iop.route_kind, self._cell.iop.route_id,
|
||||
self._cell.iop.request_model, self._cell.iop.requested_effort,
|
||||
values[0], values[1], values[2], values[3], tuple(parsed_stages),
|
||||
)
|
||||
def _usage_metrics(self, usage: Any) -> tuple[ParsedMetric, ...] | None:
|
||||
if not isinstance(usage, dict) or not set(usage) <= set(_AGY_USAGE_METRICS):
|
||||
return None
|
||||
metrics: list[ParsedMetric] = []
|
||||
try:
|
||||
for wire_name, metric_name in _AGY_USAGE_METRICS.items():
|
||||
if wire_name in usage:
|
||||
metrics.append(count_metric(
|
||||
metric_name, usage[wire_name], model=self._cell.iop.request_model,
|
||||
))
|
||||
except LifecycleMetricError:
|
||||
return None
|
||||
return tuple(metrics)
|
||||
|
||||
def observed_result(self, capability: AgyCapability, lifecycle: InvocationResult) -> ConnectivityResult:
|
||||
"""Report ready only for successful lifecycle-owned explicit evidence."""
|
||||
requested = _requested_binding(self._cell)
|
||||
closed_gap = (_issue("stream_incompatible"),)
|
||||
caller_capability = CallerCapability(AGY_CALLER, capability.route_kinds, capability.efforts)
|
||||
|
|
@ -478,12 +483,11 @@ class AgyEventParser:
|
|||
not isinstance(lifecycle, InvocationResult)
|
||||
or not lifecycle.success
|
||||
or not lifecycle.finish_then_idle_then_quiet
|
||||
or self._binding_invalid
|
||||
or self._observed_binding is None
|
||||
or not self._result_seen
|
||||
):
|
||||
return make_result(self._cell, caller_capability, requested, closed_gap)
|
||||
try:
|
||||
return make_result(self._cell, caller_capability, self._observed_binding)
|
||||
return make_result(self._cell, caller_capability, self._admitted_binding)
|
||||
except Exception:
|
||||
return make_result(self._cell, caller_capability, requested, closed_gap)
|
||||
|
||||
|
|
@ -511,5 +515,5 @@ def run_agy_invocation(
|
|||
spec,
|
||||
parse_event=parser,
|
||||
on_started=on_started,
|
||||
redact=lambda line: line if line in AGY_SAFE_METRIC_LABELS else structural(exact(line)),
|
||||
redact=lambda line: structural(exact(line)),
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Credential-free tests for the fail-closed agy IOP adapter."""
|
||||
"""Credential-free tests for the official agy 1.1.12 IOP adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -9,25 +9,27 @@ import tempfile
|
|||
import unittest
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.agy_iop import (
|
||||
AGY_AUTH_ENV,
|
||||
AGY_ENDPOINT_ENV,
|
||||
AGY_KNOWN_VERSION,
|
||||
AGY_PROVIDER_ENV,
|
||||
AgyAdapterError,
|
||||
AgyEventParser,
|
||||
AgyRuntimeInputs,
|
||||
AgyRuntimeObservation,
|
||||
_runtime_identity,
|
||||
build_agy_invocation,
|
||||
inspect_agy_iop_capability,
|
||||
preflight_agy_iop,
|
||||
redact_agy_event,
|
||||
run_agy_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.connectivity import EffectiveBinding, RequestedEffectiveBinding
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
REASON_DUPLICATE_EVENT,
|
||||
REASON_MALFORMED_EVENT,
|
||||
SUBMISSION_ARGV_TASK,
|
||||
InvocationSpec,
|
||||
env_pairs,
|
||||
)
|
||||
|
|
@ -35,29 +37,24 @@ from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCel
|
|||
from scripts.agent_benchmark.workspace import AttemptIdentity, PreparedWorkspace, TestbedProvenance
|
||||
|
||||
|
||||
def _help(*, transport: bool = True) -> str:
|
||||
basic = "--print --output-format stream-json --sandbox --model --effort"
|
||||
return basic + (f" {AGY_PROVIDER_ENV} {AGY_ENDPOINT_ENV} {AGY_AUTH_ENV}" if transport else "")
|
||||
def _help() -> str:
|
||||
return "--print --output-format stream-json --sandbox --model --effort"
|
||||
|
||||
|
||||
def _cell() -> MatrixCell:
|
||||
return MatrixCell(
|
||||
"agy-direct", "agy",
|
||||
IopCell("gemini-2.0-flash", "high", "direct", "agy-direct", (
|
||||
ExpectedBinding("request", "gemini-2.0-flash", "high"),
|
||||
IopCell("gemini-3.6-flash", "high", "direct", "agy-direct", (
|
||||
ExpectedBinding("request", "gemini-3.6-flash", "high"),
|
||||
)),
|
||||
)
|
||||
|
||||
|
||||
def _iop_config_observation() -> AgyRuntimeObservation:
|
||||
"""Fixed evidence from the independent IOP config owner for this cell."""
|
||||
return AgyRuntimeObservation(
|
||||
"agy-direct",
|
||||
"direct",
|
||||
"agy-direct",
|
||||
"sha256:feb4c33d4e775c775bfb3c333fdb7d4f97069af31c8e824094fb13181fad53d3",
|
||||
"sha256:ab1b96f33fc4a662c870f349d92c54bc8e2574028fa41b79526d4edaf6f49daa",
|
||||
"sha256:" + "c" * 64,
|
||||
def _binding() -> RequestedEffectiveBinding:
|
||||
return RequestedEffectiveBinding(
|
||||
"agy-direct", "agy", "direct", "agy-direct", "gemini-3.6-flash", "high",
|
||||
"direct", "agy-direct", "gemini-3.6-flash", "high",
|
||||
(EffectiveBinding("request", "gemini-3.6-flash", "high"),),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -67,316 +64,198 @@ class AgyIopTest(unittest.TestCase):
|
|||
self.root = Path(self.temp.name)
|
||||
self.workspace = self.root / "workspace"
|
||||
self.workspace.mkdir()
|
||||
self.runtime = AgyRuntimeInputs(sys.executable, "https://private.invalid/v1", "iop_secret_123456789")
|
||||
self.session = self.root / "session"
|
||||
self.session.mkdir()
|
||||
self.runtime = AgyRuntimeInputs(
|
||||
sys.executable, "https://private.invalid/gemini/agy-direct", "iop_secret_123456789"
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def _observation(self, runtime: AgyRuntimeInputs | None = None) -> AgyRuntimeObservation:
|
||||
value = runtime or self.runtime
|
||||
return AgyRuntimeObservation(
|
||||
"agy-direct", "direct", "agy-direct",
|
||||
_runtime_identity("endpoint", value.endpoint),
|
||||
_runtime_identity("credential", value.credential),
|
||||
"sha256:" + "c" * 64,
|
||||
)
|
||||
|
||||
def _prepared(self) -> PreparedWorkspace:
|
||||
return PreparedWorkspace(
|
||||
AttemptIdentity("run", "agy-direct", 1, 1), str(self.root), str(self.workspace),
|
||||
str(self.root / "session"), "fresh-session", True, "sha256:" + "0" * 64,
|
||||
str(self.session), "fresh-session", True, "sha256:" + "0" * 64,
|
||||
"isolated", TestbedProvenance("/testbed", "main", "0" * 40, "sha256:" + "1" * 64, True),
|
||||
"2026-01-01T00:00:00+00:00",
|
||||
)
|
||||
|
||||
def _preflight(self, *, runtime: AgyRuntimeInputs | None = None, help_text: str | None = None):
|
||||
values = self.runtime if runtime is None else runtime
|
||||
def _preflight(self, runtime: AgyRuntimeInputs | None = None):
|
||||
value = runtime or self.runtime
|
||||
return preflight_agy_iop(
|
||||
_cell(),
|
||||
inspect_agy_iop_capability("agy 1.1.11", _help() if help_text is None else help_text),
|
||||
values,
|
||||
_iop_config_observation(),
|
||||
_cell(), inspect_agy_iop_capability("agy 1.1.12", _help()), value,
|
||||
self._observation(value),
|
||||
)
|
||||
|
||||
def _run_lines(self, lines: list[str], parser: AgyEventParser, preflight):
|
||||
def _run_lines(self, lines: list[str], parser: AgyEventParser):
|
||||
evidence = self.root / f"evidence-{len(list(self.root.glob('evidence-*')))}"
|
||||
evidence.mkdir()
|
||||
source = "import sys; lines=" + repr(lines) + "; [print(line) for line in lines]"
|
||||
source = "lines=" + repr(lines) + "; [print(line) for line in lines]"
|
||||
spec = InvocationSpec(
|
||||
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
|
||||
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
|
||||
submission_mode="stdin_once", completion_mode="exit_after_idle",
|
||||
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
|
||||
)
|
||||
return run_agy_invocation(spec, parser, preflight, lambda _: None)
|
||||
return run_agy_invocation(spec, parser, self._preflight(), lambda _: None)
|
||||
|
||||
def test_absent_or_unknown_transport_never_constructs_launch(self) -> None:
|
||||
for version, help_text, expected in (
|
||||
("agy 1.1.11", _help(transport=False), ("endpoint_incompatible", "auth_incompatible", "protocol_incompatible")),
|
||||
("agy 9.9.9", _help(), "protocol_incompatible"),
|
||||
):
|
||||
with self.subTest(version=version):
|
||||
preflight = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability(version, help_text), self.runtime,
|
||||
_iop_config_observation(),
|
||||
)
|
||||
self.assertEqual(preflight.status, "implementation_gap")
|
||||
expected_codes = (expected,) if isinstance(expected, str) else expected
|
||||
self.assertEqual([item.code for item in preflight.issues], list(expected_codes))
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
|
||||
|
||||
def test_installed_public_surface_is_exact_fail_closed_gap(self) -> None:
|
||||
public_help = "--print --output-format stream-json --sandbox --model --effort"
|
||||
for transport_name in (AGY_PROVIDER_ENV, AGY_ENDPOINT_ENV, AGY_AUTH_ENV):
|
||||
self.assertNotIn(transport_name, public_help)
|
||||
|
||||
capability = inspect_agy_iop_capability("1.1.11", public_help)
|
||||
def test_official_public_surface_is_pinned_without_invented_environment(self) -> None:
|
||||
capability = inspect_agy_iop_capability("1.1.12", _help())
|
||||
self.assertEqual(capability.version, AGY_KNOWN_VERSION)
|
||||
self.assertTrue(capability.stream_supported)
|
||||
self.assertFalse(capability.endpoint_supported)
|
||||
self.assertFalse(capability.auth_supported)
|
||||
self.assertFalse(capability.protocol_supported)
|
||||
self.assertFalse(capability.iop_transport_supported)
|
||||
self.assertTrue(capability.iop_transport_supported)
|
||||
self.assertTrue(capability.endpoint_supported)
|
||||
self.assertTrue(capability.auth_supported)
|
||||
self.assertFalse(inspect_agy_iop_capability("1.1.11", _help()).iop_transport_supported)
|
||||
self.assertFalse(inspect_agy_iop_capability("1.1.12", _help().replace("stream-json", "json")).stream_supported)
|
||||
|
||||
preflight = preflight_agy_iop(
|
||||
_cell(), capability, self.runtime, _iop_config_observation()
|
||||
)
|
||||
self.assertEqual(preflight.status, "implementation_gap")
|
||||
self.assertEqual(
|
||||
[item.code for item in preflight.issues],
|
||||
["endpoint_incompatible", "auth_incompatible", "protocol_incompatible"],
|
||||
)
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(
|
||||
_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight
|
||||
)
|
||||
|
||||
def test_non_ready_preflight_cannot_start_supplied_invocation(self) -> None:
|
||||
preflight = self._preflight(help_text=_help(transport=False))
|
||||
self.assertEqual(preflight.status, "implementation_gap")
|
||||
self.assertIsNotNone(preflight.runtime)
|
||||
marker = self.root / "caller-launched"
|
||||
evidence = self.root / "blocked-evidence"
|
||||
source = "from pathlib import Path; Path(" + repr(str(marker)) + ").write_text('launched')"
|
||||
spec = InvocationSpec(
|
||||
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
|
||||
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
|
||||
submission_mode="stdin_once", completion_mode="exit_after_idle",
|
||||
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
|
||||
)
|
||||
started: list[object] = []
|
||||
|
||||
with self.assertRaisesRegex(AgyAdapterError, "agy IOP transport is not proven"):
|
||||
run_agy_invocation(spec, AgyEventParser(_cell()), preflight, started.append)
|
||||
|
||||
self.assertFalse(marker.exists())
|
||||
self.assertEqual(started, [])
|
||||
self.assertFalse(evidence.exists())
|
||||
|
||||
def test_registration_gaps_remain_distinct_from_implementation_gap(self) -> None:
|
||||
no_credential = AgyRuntimeInputs(sys.executable, self.runtime.endpoint, "")
|
||||
supported = inspect_agy_iop_capability("agy 1.1.11", _help())
|
||||
result = preflight_agy_iop(
|
||||
_cell(), supported, no_credential, _iop_config_observation()
|
||||
)
|
||||
self.assertEqual(result.status, "registration_required")
|
||||
self.assertEqual([item.code for item in result.issues], ["credential_missing"])
|
||||
gap = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help(transport=False)), no_credential,
|
||||
_iop_config_observation(),
|
||||
)
|
||||
self.assertEqual(gap.status, "implementation_gap")
|
||||
self.assertEqual([item.code for item in gap.issues], ["credential_missing", "endpoint_incompatible", "auth_incompatible", "protocol_incompatible"])
|
||||
|
||||
def test_endpoint_auth_and_protocol_gaps_are_exact(self) -> None:
|
||||
cases = (
|
||||
(_help().replace(AGY_ENDPOINT_ENV, ""), "endpoint_incompatible"),
|
||||
(_help().replace(AGY_AUTH_ENV, ""), "auth_incompatible"),
|
||||
(_help().replace("--sandbox", ""), "protocol_incompatible"),
|
||||
)
|
||||
for help_text, expected in cases:
|
||||
with self.subTest(expected=expected):
|
||||
outcome = self._preflight(help_text=help_text)
|
||||
self.assertEqual([item.code for item in outcome.issues], [expected])
|
||||
unknown = inspect_agy_iop_capability(None, None) # type: ignore[arg-type]
|
||||
self.assertFalse(unknown.iop_transport_supported)
|
||||
|
||||
def test_build_is_fresh_stdin_sandbox_and_iop_only(self) -> None:
|
||||
preflight = self._preflight()
|
||||
spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), preflight)
|
||||
self.assertEqual(spec.submission_mode, "stdin_once")
|
||||
self.assertIn("--print", spec.argv)
|
||||
self.assertIn("--sandbox", spec.argv)
|
||||
self.assertNotIn("--resume", spec.argv)
|
||||
def test_build_uses_official_gemini_api_key_transport(self) -> None:
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), self._preflight())
|
||||
environment = dict(spec.env)
|
||||
self.assertEqual(environment[AGY_PROVIDER_ENV], "iop-openai")
|
||||
self.assertEqual(environment[AGY_ENDPOINT_ENV], self.runtime.endpoint)
|
||||
self.assertEqual(environment[AGY_AUTH_ENV], self.runtime.credential)
|
||||
self.assertEqual(environment["HOME"], str(self.session))
|
||||
self.assertNotIn("AGY_PROVIDER", environment)
|
||||
self.assertNotIn("AGY_OPENAI_BASE_URL", environment)
|
||||
self.assertNotIn("AGY_OPENAI_API_KEY", environment)
|
||||
self.assertNotIn("--effort", spec.argv)
|
||||
self.assertEqual(spec.argv[spec.argv.index("--model") + 1], "Gemini 3.6 Flash")
|
||||
self.assertEqual(spec.argv[-2:], ("--print", "one task"))
|
||||
self.assertEqual(spec.submission_mode, SUBMISSION_ARGV_TASK)
|
||||
self.assertEqual(spec.task_payload, b"")
|
||||
self.assertEqual(environment["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(environment["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(spec.env_allowlist, (AGY_ENDPOINT_ENV, AGY_AUTH_ENV, "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS"))
|
||||
|
||||
def test_exact_help_tokens_and_stream_format_gate(self) -> None:
|
||||
lookalike = _help().replace("--print", "--print-json").replace(
|
||||
AGY_ENDPOINT_ENV, AGY_ENDPOINT_ENV + "_EXTRA"
|
||||
).replace("stream-json", "stream-jsonl")
|
||||
capability = inspect_agy_iop_capability("agy 1.1.11", lookalike)
|
||||
self.assertFalse(capability.iop_transport_supported)
|
||||
self.assertFalse(capability.endpoint_supported)
|
||||
self.assertFalse(capability.stream_supported)
|
||||
missing_stream = self._preflight(help_text=_help().replace("stream-json", ""))
|
||||
self.assertEqual([issue.code for issue in missing_stream.issues], ["stream_incompatible"])
|
||||
def test_build_rejects_missing_prepared_workspace_or_session(self) -> None:
|
||||
prepared = self._prepared()
|
||||
for field in ("workspace_dir", "session_dir", "attempt_root"):
|
||||
missing = prepared.__class__(**{
|
||||
**prepared.__dict__, field: str(self.root / f"missing-{field}"),
|
||||
})
|
||||
with self.subTest(field=field):
|
||||
with self.assertRaisesRegex(AgyAdapterError, "prepared workspace"):
|
||||
build_agy_invocation(
|
||||
_cell(), missing, b"task", Timeout(5, 1, 1, 1), self._preflight()
|
||||
)
|
||||
|
||||
def test_unvalidated_runtime_cannot_launch(self) -> None:
|
||||
observation = _iop_config_observation()
|
||||
for mismatched in (
|
||||
replace(observation, cell_id="other-cell"),
|
||||
replace(observation, route_id="other-route"),
|
||||
replace(observation, endpoint_identity="sha256:" + "d" * 64),
|
||||
replace(observation, config_identity="not-a-config-identity"),
|
||||
def test_build_rejects_non_utf8_task_for_print_argument(self) -> None:
|
||||
with self.assertRaisesRegex(AgyAdapterError, "must be UTF-8"):
|
||||
build_agy_invocation(
|
||||
_cell(), self._prepared(), b"\xff", Timeout(5, 1, 1, 1), self._preflight()
|
||||
)
|
||||
|
||||
def test_route_qualified_https_runtime_is_required(self) -> None:
|
||||
for endpoint in (
|
||||
"http://private.invalid/gemini/agy-direct",
|
||||
"https://private.invalid/v1",
|
||||
"https://private.invalid/gemini/other",
|
||||
):
|
||||
with self.subTest(observation=mismatched):
|
||||
preflight = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), self.runtime, mismatched
|
||||
)
|
||||
self.assertEqual(preflight.status, "implementation_gap")
|
||||
self.assertIsNone(preflight.runtime)
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
|
||||
|
||||
def test_arbitrary_runtime_cannot_self_issue_iop_proof(self) -> None:
|
||||
arbitrary = AgyRuntimeInputs(sys.executable, "https://api.openai.com/v1", "unrelated_token_123456789")
|
||||
preflight = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("agy 1.1.11", _help()), arbitrary,
|
||||
_iop_config_observation(),
|
||||
runtime = replace(self.runtime, endpoint=endpoint)
|
||||
result = self._preflight(runtime)
|
||||
self.assertEqual(result.status, "implementation_gap")
|
||||
self.assertEqual([issue.code for issue in result.issues], ["endpoint_incompatible"])
|
||||
missing = replace(self.runtime, credential="")
|
||||
result = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("1.1.12", _help()), missing,
|
||||
self._observation(self.runtime),
|
||||
)
|
||||
self.assertEqual(preflight.status, "implementation_gap")
|
||||
self.assertIsNone(preflight.runtime)
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), preflight)
|
||||
self.assertEqual(result.status, "registration_required")
|
||||
self.assertEqual([issue.code for issue in result.issues], ["credential_missing"])
|
||||
|
||||
def test_lifecycle_fixture_success_and_metric_preservation(self) -> None:
|
||||
parser = AgyEventParser(_cell())
|
||||
def test_unknown_model_and_unvalidated_observation_fail_closed(self) -> None:
|
||||
unsupported = replace(_cell(), iop=replace(_cell().iop, request_model="gemini-unknown"))
|
||||
result = preflight_agy_iop(
|
||||
unsupported, inspect_agy_iop_capability("1.1.12", _help()), self.runtime,
|
||||
self._observation(),
|
||||
)
|
||||
self.assertIn("model_missing", [issue.code for issue in result.issues])
|
||||
mismatched = replace(self._observation(), endpoint_identity="sha256:" + "d" * 64)
|
||||
result = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("1.1.12", _help()), self.runtime, mismatched,
|
||||
)
|
||||
self.assertIsNone(result.runtime)
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), result)
|
||||
|
||||
def test_official_fixture_completes_and_preserves_metrics(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
fixture = Path("scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl")
|
||||
result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser, self._preflight())
|
||||
result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser)
|
||||
self.assertTrue(result.success)
|
||||
self.assertTrue(result.finish_then_idle_then_quiet)
|
||||
journal = Path(result.journal_path).read_text(encoding="utf-8")
|
||||
self.assertIn('"kind": "metric:total_duration"', journal)
|
||||
self.assertIn('"kind": "metric:model_duration"', journal)
|
||||
observed = {metric.name: metric for metric in result.metrics}
|
||||
self.assertEqual(set(observed), {"total_duration", "model_duration"})
|
||||
self.assertEqual(observed["total_duration"].value, 12 * 10 ** 6)
|
||||
self.assertFalse(observed["total_duration"].overlap)
|
||||
# 8.5 ms is preserved exactly; agy's model stage overlaps its total.
|
||||
self.assertEqual(observed["model_duration"].value, 8_500_000)
|
||||
self.assertTrue(observed["model_duration"].overlap)
|
||||
for metric in result.metrics:
|
||||
self.assertEqual(metric.unit, "ns")
|
||||
self.assertEqual(metric.clock, "caller_reported")
|
||||
self.assertEqual(metric.source, "caller_output")
|
||||
self.assertEqual(metric.model, "gemini-2.0-flash")
|
||||
capability = inspect_agy_iop_capability("agy 1.1.11", _help())
|
||||
self.assertEqual(parser.observed_result(capability, result).status, "ready")
|
||||
|
||||
def test_only_allowlisted_bound_representable_durations_are_observed(self) -> None:
|
||||
parser = AgyEventParser(_cell())
|
||||
rejected = (
|
||||
{"type": "metric", "subtype": "unknown_ms", "value": 5},
|
||||
{"type": "metric", "subtype": "duration_ms", "value": "5"},
|
||||
{"type": "metric", "subtype": "duration_ms", "value": True},
|
||||
{"type": "metric", "subtype": "duration_ms", "value": -1},
|
||||
# 0.0000001 ms is 0.1 ns and cannot be represented without invention.
|
||||
{"type": "metric", "subtype": "duration_ms", "value": 0.0000001},
|
||||
{"type": "metric", "subtype": "duration_ms", "value": 5, "model": "other"},
|
||||
{"type": "metric", "subtype": "duration_ms", "value": 5, "extra": 1},
|
||||
metrics = {metric.name: metric for metric in result.metrics}
|
||||
self.assertEqual(
|
||||
set(metrics),
|
||||
{"total_duration", "model_calls", "input_tokens", "cached_input_tokens", "output_tokens", "reasoning_tokens", "total_tokens"},
|
||||
)
|
||||
for event in rejected:
|
||||
with self.subTest(event=event):
|
||||
self.assertEqual(parser("stdout", json.dumps(event)), "malformed")
|
||||
bound = parser("stdout", json.dumps({
|
||||
"type": "metric", "subtype": "queue_duration_ms", "value": 2,
|
||||
"model": "gemini-2.0-flash",
|
||||
}))
|
||||
self.assertEqual((bound.name, bound.value, bound.overlap), ("queue_duration", 2_000_000, False))
|
||||
self.assertEqual(metrics["total_duration"].value, 12_000_000)
|
||||
self.assertEqual(metrics["model_calls"].value, 1)
|
||||
self.assertEqual(metrics["total_tokens"].value, 12)
|
||||
capability = inspect_agy_iop_capability("1.1.12", _help())
|
||||
self.assertEqual(parser.observed_result(capability, result).binding, _binding())
|
||||
|
||||
def test_unrepresentable_duration_fails_the_run_without_partial_metric(self) -> None:
|
||||
parser = AgyEventParser(_cell())
|
||||
result = self._run_lines(
|
||||
[json.dumps({"type": "metric", "subtype": "duration_ms", "value": 0.0000001})],
|
||||
parser, self._preflight(),
|
||||
)
|
||||
self.assertFalse(result.success)
|
||||
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
|
||||
self.assertEqual(result.metrics, ())
|
||||
def test_result_must_follow_init_be_unique_and_successful(self) -> None:
|
||||
result = {
|
||||
"event": "result", "result": {
|
||||
"status": "SUCCESS", "duration_seconds": 0.1, "num_turns": 1,
|
||||
"usage": {"input_tokens": 1}, "response": "secret content",
|
||||
},
|
||||
}
|
||||
for lines, reason in (
|
||||
([json.dumps(result)], REASON_MALFORMED_EVENT),
|
||||
([json.dumps({"event": "init", "init": {}}), json.dumps(result), json.dumps(result)], REASON_MALFORMED_EVENT),
|
||||
([json.dumps({"event": "init", "init": {}}), json.dumps({**result, "result": {**result["result"], "status": "ERROR"}})], REASON_MALFORMED_EVENT),
|
||||
):
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
invocation = self._run_lines(lines, parser)
|
||||
self.assertFalse(invocation.success)
|
||||
self.assertEqual(invocation.terminal_reason, reason)
|
||||
|
||||
def test_metric_prefix_cannot_bypass_durable_redaction(self) -> None:
|
||||
parser = AgyEventParser(_cell())
|
||||
raw_lines = [f"metric:{self.runtime.endpoint}", f"metric:{self.runtime.credential}", "metric:not-json"]
|
||||
result = self._run_lines(raw_lines, parser, self._preflight())
|
||||
self.assertFalse(result.success)
|
||||
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
|
||||
persisted = Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8")
|
||||
for forbidden in (*raw_lines, self.runtime.endpoint, self.runtime.credential):
|
||||
self.assertNotIn(forbidden, persisted)
|
||||
def test_latest_step_usage_is_used_only_when_result_omits_usage(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
lines = [
|
||||
json.dumps({"event": "init", "conversation_id": "c", "init": {"model": "Gemini 3.6 Flash"}}),
|
||||
json.dumps({"event": "step_update", "step_update": {"state": "DONE", "step_index": 0, "step_type": "agent_response", "text_delta": "private", "usage": {"input_tokens": 2, "output_tokens": 1, "total_tokens": 3}}}),
|
||||
json.dumps({"event": "result", "result": {"status": "SUCCESS", "duration_seconds": 0.2, "num_turns": 1, "response": "private"}}),
|
||||
]
|
||||
invocation = self._run_lines(lines, parser)
|
||||
self.assertTrue(invocation.success)
|
||||
self.assertEqual({m.name: m.value for m in invocation.metrics}["total_tokens"], 3)
|
||||
|
||||
def test_mismatch_duplicate_and_quota_cannot_pass(self) -> None:
|
||||
event = {"type": "result", "subtype": "success", "model": "other", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
|
||||
self.assertEqual(AgyEventParser(_cell())("stdout", json.dumps(event)), "malformed")
|
||||
parser = AgyEventParser(_cell())
|
||||
finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
|
||||
self.assertEqual(parser("stdout", json.dumps(finish)), "finish")
|
||||
self.assertEqual(parser("stdout", json.dumps(finish)), "finish")
|
||||
self.assertEqual(AgyEventParser(_cell())("stdout", '{"type":"result","subtype":"error","reason":"quota"}'), "quota_error")
|
||||
def test_malformed_usage_fails_without_partial_metric(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
invocation = self._run_lines([
|
||||
'{"event":"init","init":{}}',
|
||||
'{"event":"result","result":{"status":"SUCCESS","usage":{"input_tokens":"1"}}}',
|
||||
], parser)
|
||||
self.assertFalse(invocation.success)
|
||||
self.assertEqual(invocation.terminal_reason, REASON_MALFORMED_EVENT)
|
||||
self.assertEqual(invocation.metrics, ())
|
||||
|
||||
evidence = self.root / "duplicate-evidence"
|
||||
evidence.mkdir()
|
||||
source = "import json; event=" + repr(finish) + "; print(json.dumps(event)); print(json.dumps(event))"
|
||||
spec = InvocationSpec(
|
||||
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
|
||||
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
|
||||
submission_mode="stdin_once", completion_mode="exit_after_idle",
|
||||
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
|
||||
)
|
||||
duplicate = run_agy_invocation(spec, AgyEventParser(_cell()), self._preflight(), lambda _: None)
|
||||
self.assertFalse(duplicate.success)
|
||||
self.assertEqual(duplicate.terminal_reason, REASON_DUPLICATE_EVENT)
|
||||
|
||||
def test_structural_redaction_excludes_content_tools_endpoints_and_secrets(self) -> None:
|
||||
raw = json.dumps({"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "content": "raw prompt", "tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint, "token": self.runtime.credential})
|
||||
def test_structural_redaction_excludes_response_tools_endpoint_and_secret(self) -> None:
|
||||
raw = json.dumps({
|
||||
"event": "result", "result": {
|
||||
"status": "SUCCESS", "response": "raw prompt",
|
||||
"tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint,
|
||||
"token": self.runtime.credential,
|
||||
},
|
||||
})
|
||||
redacted = redact_agy_event(raw, (self.runtime.endpoint, self.runtime.credential))
|
||||
self.assertEqual(redacted, '{"model":"gemini-2.0-flash","subtype":"success","type":"result"}')
|
||||
self.assertEqual(redacted, '{"event":"result","status":"SUCCESS"}')
|
||||
for forbidden in ("raw prompt", "tool_input", self.runtime.endpoint, self.runtime.credential):
|
||||
self.assertNotIn(forbidden, redacted)
|
||||
|
||||
def test_lifecycle_rejects_quota_without_durable_leak(self) -> None:
|
||||
evidence = self.root / "evidence"
|
||||
evidence.mkdir()
|
||||
parser = AgyEventParser(_cell())
|
||||
secret = self.runtime.credential
|
||||
endpoint = self.runtime.endpoint
|
||||
source = "import json; print(json.dumps(" + repr({
|
||||
"type": "result", "subtype": "error", "reason": "quota",
|
||||
"content": secret, "endpoint": endpoint,
|
||||
}) + "))"
|
||||
spec = InvocationSpec(
|
||||
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
|
||||
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
|
||||
submission_mode="stdin_once", completion_mode="exit_after_idle",
|
||||
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
|
||||
)
|
||||
result = run_agy_invocation(spec, parser, self._preflight(), lambda _: None)
|
||||
self.assertFalse(result.success)
|
||||
self.assertEqual(result.terminal_reason, REASON_MALFORMED_EVENT)
|
||||
persisted = (Path(result.journal_path).read_text(encoding="utf-8") + Path(result.result_path).read_text(encoding="utf-8"))
|
||||
self.assertNotIn(secret, persisted)
|
||||
self.assertNotIn(endpoint, persisted)
|
||||
|
||||
def test_ready_requires_observed_stage_binding_and_successful_lifecycle(self) -> None:
|
||||
capability = inspect_agy_iop_capability("agy 1.1.11", _help())
|
||||
finish = {"type": "result", "subtype": "success", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
|
||||
idle = {"type": "system", "subtype": "idle", "model": "gemini-2.0-flash", "effort": "high", "route_kind": "direct", "route_id": "agy-direct"}
|
||||
binding = {"type": "iop", "subtype": "effective_binding", "route_kind": "direct", "route_id": "agy-direct", "model": "gemini-2.0-flash", "effort": "high", "stages": [{"stage": "request", "model": "gemini-2.0-flash", "effort": "high"}]}
|
||||
for lines in (
|
||||
[json.dumps(finish), json.dumps(idle)],
|
||||
[json.dumps(binding), json.dumps(idle), json.dumps(finish)],
|
||||
[json.dumps(binding), json.dumps(finish), json.dumps(finish), json.dumps(idle)],
|
||||
[json.dumps({**binding, "route_id": "other"}), json.dumps(finish), json.dumps(idle)],
|
||||
):
|
||||
with self.subTest(lines=lines):
|
||||
parser = AgyEventParser(_cell())
|
||||
result = self._run_lines(lines, parser, self._preflight())
|
||||
self.assertEqual(parser.observed_result(capability, result).status, "implementation_gap")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
|
|||
|
|
@ -363,7 +363,7 @@ def _connectivity_result_from_payload(
|
|||
tuple(stages),
|
||||
)
|
||||
# Capability is deliberately not serialized. The persisted proof is
|
||||
# revalidated against the exact immutable cell and its one direct route.
|
||||
# revalidated against the exact immutable cell and its declared route.
|
||||
capability = CallerCapability(
|
||||
cell.caller, (cell.iop.route_kind,), (cell.iop.requested_effort,)
|
||||
)
|
||||
|
|
@ -387,11 +387,7 @@ def collect_preflight_observations(
|
|||
manifest: Manifest,
|
||||
adapters: Mapping[str, PreflightAdapter],
|
||||
) -> dict[str, PreflightObservation]:
|
||||
"""Validate the full registry, then probe direct cells in manifest order.
|
||||
|
||||
Execution-preset cells exercise only the local adapter capability contract in
|
||||
this milestone. They never become a synthetic live-ready observation.
|
||||
"""
|
||||
"""Validate the full registry, then probe every cell in manifest order."""
|
||||
if not isinstance(adapters, Mapping):
|
||||
raise CapabilityUnavailable("capability-unavailable: caller-adapter")
|
||||
required_callers = {cell.caller for cell in manifest.matrix}
|
||||
|
|
@ -405,8 +401,6 @@ def collect_preflight_observations(
|
|||
|
||||
observations: dict[str, PreflightObservation] = {}
|
||||
for cell in manifest.matrix:
|
||||
if cell.iop.route_kind != "direct":
|
||||
continue
|
||||
observation = adapters[cell.caller].preflight(cell)
|
||||
if not isinstance(observation, PreflightObservation):
|
||||
raise AttemptStateError("preflight observation is invalid")
|
||||
|
|
@ -512,8 +506,8 @@ class RunStore:
|
|||
os.close(fd)
|
||||
|
||||
@staticmethod
|
||||
def _direct_cells(manifest: Manifest) -> tuple[MatrixCell, ...]:
|
||||
return tuple(cell for cell in manifest.matrix if cell.iop.route_kind == "direct")
|
||||
def _preflight_cells(manifest: Manifest) -> tuple[MatrixCell, ...]:
|
||||
return tuple(manifest.matrix)
|
||||
|
||||
def _preflight_root(self, run: RunIdentity, *, create: bool) -> Path:
|
||||
root = Path(run.root) / "preflight"
|
||||
|
|
@ -560,11 +554,11 @@ class RunStore:
|
|||
or not isinstance(record["results"], list)
|
||||
):
|
||||
raise AttemptStateError("preflight record identity is invalid")
|
||||
direct_cells = self._direct_cells(manifest)
|
||||
if len(record["results"]) != len(direct_cells):
|
||||
preflight_cells = self._preflight_cells(manifest)
|
||||
if len(record["results"]) != len(preflight_cells):
|
||||
raise AttemptStateError("preflight result set is invalid")
|
||||
statuses: list[str] = []
|
||||
for cell, result_payload in zip(direct_cells, record["results"]):
|
||||
for cell, result_payload in zip(preflight_cells, record["results"]):
|
||||
result, _, _ = _connectivity_result_from_payload(result_payload, cell)
|
||||
statuses.append(result.status)
|
||||
if record["status"] != _overall_preflight_status(iter(statuses)):
|
||||
|
|
@ -629,11 +623,11 @@ class RunStore:
|
|||
bound_run = self.open(manifest, run.run_id)
|
||||
if bound_run != run:
|
||||
raise AttemptStateError("run identity is invalid")
|
||||
direct_cells = self._direct_cells(manifest)
|
||||
if set(observations) != {cell.id for cell in direct_cells}:
|
||||
preflight_cells = self._preflight_cells(manifest)
|
||||
if set(observations) != {cell.id for cell in preflight_cells}:
|
||||
raise AttemptStateError("preflight observation set is invalid")
|
||||
results: list[dict[str, Any]] = []
|
||||
for cell in direct_cells:
|
||||
for cell in preflight_cells:
|
||||
observation = observations[cell.id]
|
||||
if not isinstance(observation, PreflightObservation):
|
||||
raise AttemptStateError("preflight observation is invalid")
|
||||
|
|
@ -782,7 +776,18 @@ class RunStore:
|
|||
if "lifecycle" in record and (not isinstance(record["lifecycle"], dict) or set(record["lifecycle"]) != {"terminal_reason"} or not isinstance(record["lifecycle"]["terminal_reason"], str)):
|
||||
raise AttemptStateError("attempt lifecycle is invalid")
|
||||
if record["state"] in TERMINAL_STATES:
|
||||
self._validate_web_validation(root, run, identity, web_policy)
|
||||
pre_registration_interrupted = (
|
||||
record["state"] == "interrupted"
|
||||
and "locator" not in record
|
||||
and record.get("lifecycle") == {"terminal_reason": "interrupted"}
|
||||
)
|
||||
self._validate_web_validation(
|
||||
root,
|
||||
run,
|
||||
identity,
|
||||
web_policy,
|
||||
allow_pre_registration_absence=pre_registration_interrupted,
|
||||
)
|
||||
elif (root / WEB_VALIDATION_FILENAME).exists() or (
|
||||
root / WEB_VALIDATION_FILENAME
|
||||
).is_symlink():
|
||||
|
|
@ -989,9 +994,12 @@ class RunStore:
|
|||
def _validate_web_validation(
|
||||
self, root: Path, run: RunIdentity, identity: AttemptIdentity,
|
||||
policy: str | None,
|
||||
*, allow_pre_registration_absence: bool = False,
|
||||
) -> None:
|
||||
path = root / WEB_VALIDATION_FILENAME
|
||||
if not path.exists() and not path.is_symlink():
|
||||
if allow_pre_registration_absence:
|
||||
return
|
||||
if policy == WEB_VALIDATION_POLICY_REQUIRED_V1:
|
||||
raise AttemptStateError("required web validation is unavailable")
|
||||
return
|
||||
|
|
@ -1054,6 +1062,20 @@ class RunStore:
|
|||
occurs before ``attempt.json`` is replaced, leaving recovery resumable.
|
||||
"""
|
||||
policy = record.get("web_validation_policy")
|
||||
if (
|
||||
policy == WEB_VALIDATION_POLICY_REQUIRED_V1
|
||||
and record.get("locator") is None
|
||||
and terminal.get("terminal_reason") == "interrupted"
|
||||
):
|
||||
path = root / WEB_VALIDATION_FILENAME
|
||||
measurement = root / MEASUREMENT_FILENAME
|
||||
if not (
|
||||
path.exists()
|
||||
or path.is_symlink()
|
||||
or measurement.exists()
|
||||
or measurement.is_symlink()
|
||||
):
|
||||
return
|
||||
if policy != WEB_VALIDATION_POLICY_REQUIRED_V1:
|
||||
self._validate_web_validation(root, run, identity, policy)
|
||||
return
|
||||
|
|
@ -1873,10 +1895,8 @@ def preflight_manifest(
|
|||
*,
|
||||
adapters: Mapping[str, PreflightAdapter],
|
||||
) -> tuple[RunIdentity, dict[str, Any]]:
|
||||
"""Collect direct observations, then create one run and append one record."""
|
||||
"""Collect all-cell observations, then create one run and append one record."""
|
||||
observations = collect_preflight_observations(manifest, adapters)
|
||||
if not observations:
|
||||
raise AttemptStateError("preflight requires a direct cell")
|
||||
run = store.create(manifest, manifest_bytes)
|
||||
return run, store.record_preflight(run, manifest, observations)
|
||||
|
||||
|
|
@ -1946,8 +1966,6 @@ def run_slots(
|
|||
raise AttemptStateError("workspace preparer is invalid")
|
||||
|
||||
observations = collect_preflight_observations(manifest, adapters)
|
||||
if not observations:
|
||||
raise AttemptStateError("preflight requires a direct cell")
|
||||
bound_run = store.open(manifest, run.run_id)
|
||||
if bound_run != run:
|
||||
raise AttemptStateError("run identity is invalid")
|
||||
|
|
|
|||
|
|
@ -72,7 +72,12 @@ from scripts.agent_benchmark.web_validation import (
|
|||
from scripts.agent_benchmark.workspace import AttemptIdentity, prepare_workspace
|
||||
|
||||
|
||||
def _manifest(root: Path, repetitions: int = 1):
|
||||
def _manifest(
|
||||
root: Path,
|
||||
repetitions: int = 1,
|
||||
execution_order_seed: str | None = None,
|
||||
cell_ids: tuple[str, ...] = ("a",),
|
||||
):
|
||||
fixtures = root / "scripts/fixtures"
|
||||
fixtures.mkdir(parents=True, exist_ok=True)
|
||||
(fixtures / "prompt.md").write_text("prompt", encoding="utf-8")
|
||||
|
|
@ -89,8 +94,13 @@ def _manifest(root: Path, repetitions: int = 1):
|
|||
"viewports": [{"id": "desktop", "width": 1, "height": 1}], "rubric_version": "landing-quality-v1",
|
||||
"evaluator": {"caller": "codex", "iop": {"request_model": "judge", "requested_effort": "high", "route_kind": "direct", "route_id": "judge", "expected_bindings": [{"stage": "request", "model": "judge", "effort": "high"}]}},
|
||||
"output_root": "agent-test/runs/a", "fixture": fixture, "repetitions": repetitions,
|
||||
"matrix": [{"id": "a", "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}}],
|
||||
"matrix": [
|
||||
{"id": cell_id, "caller": "claude", "iop": {"request_model": "model", "requested_effort": "high", "route_kind": "direct", "route_id": "route", "expected_bindings": [{"stage": "request", "model": "model", "effort": "high"}]}}
|
||||
for cell_id in cell_ids
|
||||
],
|
||||
}
|
||||
if execution_order_seed is not None:
|
||||
data["execution_order_seed"] = execution_order_seed
|
||||
path = root / "manifest.json"
|
||||
raw = json.dumps(data, sort_keys=True).encode("utf-8")
|
||||
path.write_bytes(raw)
|
||||
|
|
@ -454,6 +464,33 @@ class AttemptBase(unittest.TestCase):
|
|||
|
||||
|
||||
class AttemptStoreTest(AttemptBase):
|
||||
def test_slots_follow_seeded_manifest_order_before_repetitions(self):
|
||||
self.manifest, self.raw, self.manifest_path = _manifest(
|
||||
self.root,
|
||||
repetitions=2,
|
||||
execution_order_seed="seed-a",
|
||||
cell_ids=("cell-a", "cell-b", "cell-c"),
|
||||
)
|
||||
matrix_order = [cell.id for cell in self.manifest.matrix]
|
||||
self.assertEqual(matrix_order, ["cell-c", "cell-a", "cell-b"])
|
||||
|
||||
slots = self.store.slots(self.manifest)
|
||||
self.assertEqual(
|
||||
[(slot.cell_id, slot.repetition) for slot in slots],
|
||||
[
|
||||
("cell-c", 1),
|
||||
("cell-c", 2),
|
||||
("cell-a", 1),
|
||||
("cell-a", 2),
|
||||
("cell-b", 1),
|
||||
("cell-b", 2),
|
||||
],
|
||||
)
|
||||
self.assertEqual(
|
||||
[slot.cell_id for slot in slots if slot.repetition == 1],
|
||||
matrix_order,
|
||||
)
|
||||
|
||||
def test_slots_and_append_only_terminals(self):
|
||||
self.manifest, self.raw, self.manifest_path = _manifest(self.root, repetitions=2)
|
||||
run = self.create_run()
|
||||
|
|
@ -1091,6 +1128,29 @@ class AttemptMeasurementTest(AttemptBase):
|
|||
class AttemptWebValidationTest(AttemptBase):
|
||||
"""Required S12 policy, lifecycle mapping, and recovery-before-terminal."""
|
||||
|
||||
def test_pre_registration_interruption_needs_no_impossible_sidecars(self):
|
||||
run = self.create_run()
|
||||
with self.store.writer(run):
|
||||
attempt = self.store.allocate(run, Slot("a", 1))
|
||||
with self.assertRaisesRegex(ControllerCrash, "before registration"):
|
||||
self.store.execute_attempt(
|
||||
attempt,
|
||||
prepare=lambda _: None,
|
||||
invoke=lambda _attempt, _started: (_ for _ in ()).throw(
|
||||
ControllerCrash("before registration")
|
||||
),
|
||||
require_measurement=True,
|
||||
require_web_validation=True,
|
||||
)
|
||||
terminal = self.store.reconcile(attempt)
|
||||
self.assertEqual(terminal.state, "interrupted")
|
||||
root = Path(terminal.root)
|
||||
self.assertFalse((root / MEASUREMENT_FILENAME).exists())
|
||||
self.assertFalse((root / WEB_VALIDATION_FILENAME).exists())
|
||||
self.assertEqual(
|
||||
self.store.status(run, self.manifest)["attempts"]["interrupted"], 1
|
||||
)
|
||||
|
||||
def _run(self, mode: str = "success"):
|
||||
self._init_testbed()
|
||||
run = self.create_run()
|
||||
|
|
|
|||
|
|
@ -28,9 +28,11 @@ from scripts.agent_benchmark.lifecycle import (
|
|||
InvocationSpec,
|
||||
LifecycleMetricError,
|
||||
ParsedMetric,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
exact_value_redactor,
|
||||
inherited_tls_ca_environment,
|
||||
is_reported_number,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
|
||||
|
|
@ -221,6 +223,8 @@ class ClaudeStreamParser:
|
|||
self.claude_session_id: str | None = None
|
||||
self._phase = "await_init"
|
||||
self._assistant_messages = 0
|
||||
self._active_message_id: str | None = None
|
||||
self._completed_message_ids: set[str] = set()
|
||||
|
||||
def _require_bound_session(self, event: dict[str, Any]) -> None:
|
||||
if self.claude_session_id is None:
|
||||
|
|
@ -241,14 +245,59 @@ class ClaudeStreamParser:
|
|||
raise ClaudeIopProtocolError("duplicate or out-of-order Claude assistant")
|
||||
self._require_bound_session(event)
|
||||
message = event.get("message")
|
||||
if not isinstance(message, dict) or message.get("stop_reason") != "end_turn":
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant terminal")
|
||||
if not isinstance(message, dict):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant event")
|
||||
if _required_string(message, "model") != self.cell.iop.request_model:
|
||||
raise ClaudeIopProtocolError("Claude model binding mismatch")
|
||||
stop_reason = message.get("stop_reason")
|
||||
if stop_reason not in (None, "tool_use", "end_turn"):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant stop reason")
|
||||
message_id = message.get("id")
|
||||
if message_id is not None and (not isinstance(message_id, str) or not message_id):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant message id")
|
||||
if stop_reason is None:
|
||||
if message_id is None:
|
||||
raise ClaudeIopProtocolError("unbound Claude assistant snapshot")
|
||||
if message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
if self._active_message_id not in (None, message_id):
|
||||
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
|
||||
self._active_message_id = message_id
|
||||
return None
|
||||
|
||||
if message_id is None:
|
||||
# Older fixture-shaped output did not include a message id. It is
|
||||
# admissible only for the single final assistant event.
|
||||
if stop_reason != "end_turn" or self._active_message_id is not None:
|
||||
raise ClaudeIopProtocolError("unbound Claude assistant terminal")
|
||||
else:
|
||||
if message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
if self._active_message_id not in (None, message_id):
|
||||
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
|
||||
self._completed_message_ids.add(message_id)
|
||||
self._active_message_id = None
|
||||
self._assistant_messages += 1
|
||||
if stop_reason == "tool_use":
|
||||
self._phase = "await_tool_result"
|
||||
return None
|
||||
self._phase = "await_result"
|
||||
return "finish"
|
||||
|
||||
def _consume_user(self, event: dict[str, Any]) -> None:
|
||||
if self._phase not in ("await_assistant", "await_tool_result"):
|
||||
raise ClaudeIopProtocolError("out-of-order Claude user event")
|
||||
self._require_bound_session(event)
|
||||
if self._active_message_id is not None:
|
||||
if self._active_message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
self._completed_message_ids.add(self._active_message_id)
|
||||
self._active_message_id = None
|
||||
self._assistant_messages += 1
|
||||
elif self._phase != "await_tool_result":
|
||||
raise ClaudeIopProtocolError("unexpected Claude user event")
|
||||
self._phase = "await_assistant"
|
||||
|
||||
def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...]:
|
||||
if self._phase != "await_result":
|
||||
raise ClaudeIopProtocolError("duplicate or out-of-order Claude result")
|
||||
|
|
@ -301,6 +350,9 @@ class ClaudeStreamParser:
|
|||
return None
|
||||
if event_type == "assistant":
|
||||
return self._consume_assistant(event)
|
||||
if event_type == "user":
|
||||
self._consume_user(event)
|
||||
return None
|
||||
if event_type == "result":
|
||||
return self._consume_result(event)
|
||||
# Informational events are deliberately ignored only after they have
|
||||
|
|
@ -371,19 +423,21 @@ class ClaudeIopAdapter:
|
|||
("ANTHROPIC_API_KEY", self.api_key),
|
||||
("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "1"),
|
||||
("CLAUDE_CODE_DISABLE_AUTOUPDATER", "1"),
|
||||
)
|
||||
) + inherited_tls_ca_environment()
|
||||
return InvocationSpec(
|
||||
argv=(
|
||||
self.binary, "--bare", "--print", "--verbose",
|
||||
"--input-format", "text", "--output-format", "stream-json",
|
||||
"--model", self.cell.iop.request_model, "--effort", self.cell.iop.requested_effort,
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk", "--tools=",
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk",
|
||||
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
|
||||
),
|
||||
cwd=str(cwd),
|
||||
env=env,
|
||||
env_allowlist=(
|
||||
"ANTHROPIC_BASE_URL", "ANTHROPIC_API_KEY",
|
||||
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "CLAUDE_CODE_DISABLE_AUTOUPDATER",
|
||||
*TLS_CA_ENV_KEYS,
|
||||
),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import tempfile
|
|||
import textwrap
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.claude_iop import (
|
||||
ClaudeIopAdapter,
|
||||
|
|
@ -124,18 +125,22 @@ class ClaudeIopTest(unittest.TestCase):
|
|||
|
||||
def test_exact_iop_only_invocation_and_fresh_workspace(self) -> None:
|
||||
adapter = self._adapter()
|
||||
spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1))
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1))
|
||||
self.assertEqual(spec.cwd, self.workspace.workspace_dir)
|
||||
self.assertEqual(spec.submission_mode, "stdin_once")
|
||||
self.assertEqual(spec.task_payload, SENTINELS[0].encode())
|
||||
self.assertEqual(spec.argv[1:], (
|
||||
"--bare", "--print", "--verbose", "--input-format", "text",
|
||||
"--output-format", "stream-json", "--model", "claude-sonnet", "--effort", "high",
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk", "--tools=",
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk",
|
||||
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
|
||||
))
|
||||
env = dict(spec.env)
|
||||
self.assertEqual(env["ANTHROPIC_BASE_URL"], SENTINELS[1])
|
||||
self.assertEqual(env["ANTHROPIC_API_KEY"], SENTINELS[2])
|
||||
self.assertEqual(env["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(env["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertNotIn("ANTHROPIC_AUTH_TOKEN", env)
|
||||
self.assertNotIn("CLAUDE_CONFIG_DIR", env)
|
||||
(self.root / "other").mkdir()
|
||||
|
|
@ -215,6 +220,62 @@ class ClaudeIopTest(unittest.TestCase):
|
|||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
self.assertEqual([parser("stdout", event) for event in missing_result], [None, "finish"])
|
||||
|
||||
def test_parser_accepts_partial_snapshots_and_tool_result_cycles(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
partial = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-tool-1", "model": "claude-sonnet", "stop_reason": None,
|
||||
"content": [{"type": "tool_use", "id": "tool-1"}]},
|
||||
})
|
||||
user = json.dumps({
|
||||
"type": "user", "session_id": "claude-session-fixture",
|
||||
"message": {"content": [{"type": "tool_result", "tool_use_id": "tool-1"}]},
|
||||
})
|
||||
final = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-final", "model": "claude-sonnet",
|
||||
"stop_reason": "end_turn", "content": []},
|
||||
})
|
||||
self.assertEqual(
|
||||
[parser("stdout", line) for line in (init, partial, partial, user, final)],
|
||||
[None, None, None, None, "finish"],
|
||||
)
|
||||
observations = parser("stdout", result)
|
||||
metrics = {metric.name: metric.value for metric in observations[:-1]}
|
||||
self.assertEqual(metrics["model_calls"], 2)
|
||||
|
||||
def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
tool_use = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-tool-1", "model": "claude-sonnet",
|
||||
"stop_reason": "tool_use", "content": []},
|
||||
})
|
||||
user = json.dumps({
|
||||
"type": "user", "session_id": "claude-session-fixture", "message": {"content": []},
|
||||
})
|
||||
final = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-final", "model": "claude-sonnet",
|
||||
"stop_reason": "end_turn", "content": []},
|
||||
})
|
||||
self.assertEqual(
|
||||
[parser("stdout", line) for line in (init, tool_use, user, final)],
|
||||
[None, None, None, "finish"],
|
||||
)
|
||||
self.assertEqual(
|
||||
{metric.name: metric.value for metric in parser("stdout", result)[:-1]}["model_calls"],
|
||||
2,
|
||||
)
|
||||
duplicate = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
duplicate("stdout", init)
|
||||
duplicate("stdout", tool_use)
|
||||
duplicate("stdout", user)
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
duplicate("stdout", tool_use)
|
||||
|
||||
def test_reported_result_values_become_bound_normalized_observations(self) -> None:
|
||||
init, assistant, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
|
|
|
|||
|
|
@ -37,10 +37,12 @@ from scripts.agent_benchmark.lifecycle import (
|
|||
LifecycleValidationError,
|
||||
ParsedMetric,
|
||||
SupervisorLocator,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
env_pairs,
|
||||
exact_value_redactor,
|
||||
inherited_tls_ca_environment,
|
||||
is_reported_number,
|
||||
run_invocation,
|
||||
)
|
||||
|
|
@ -77,12 +79,16 @@ _SAFE_STRING_KEYS = frozenset({
|
|||
_CODEX_USAGE_FIELDS = {
|
||||
"input_tokens": "input_tokens",
|
||||
"cached_input_tokens": "cached_input_tokens",
|
||||
"cache_write_input_tokens": "cache_write_tokens",
|
||||
"output_tokens": "output_tokens",
|
||||
"reasoning_output_tokens": "reasoning_tokens",
|
||||
"total_tokens": "total_tokens",
|
||||
}
|
||||
_CODEX_ITEM_COMPLETED = "item.completed"
|
||||
_CODEX_CALL_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.:-]{0,63}$")
|
||||
_CODEX_TOOL_ITEM_TYPES = frozenset({
|
||||
"command_execution", "file_change", "mcp_tool_call", "web_search",
|
||||
})
|
||||
|
||||
|
||||
class CodexIOPError(Exception):
|
||||
|
|
@ -219,7 +225,7 @@ def build_codex_spec(
|
|||
raise CodexRuntimeError("invalid Codex executable")
|
||||
|
||||
codex_argv: list[str] = [
|
||||
*executable_argv, "exec", "--json", "--ephemeral", "--ignore-user-config",
|
||||
*executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
|
||||
"--strict-config", "--skip-git-repo-check", "-C", prepared.workspace_dir,
|
||||
"-m", cell.iop.request_model,
|
||||
]
|
||||
|
|
@ -232,8 +238,13 @@ def build_codex_spec(
|
|||
# import resolution in the child bridge.
|
||||
argv=(sys.executable, str(Path(__file__).resolve()), "--bridge", "--", *codex_argv),
|
||||
cwd=prepared.workspace_dir,
|
||||
env=env_pairs({"PATH": runtime.path, "HOME": prepared.session_dir, SECRET_ENV_KEY: runtime.api_key}),
|
||||
env_allowlist=(SECRET_ENV_KEY,),
|
||||
env=env_pairs({
|
||||
"PATH": runtime.path,
|
||||
"HOME": prepared.session_dir,
|
||||
SECRET_ENV_KEY: runtime.api_key,
|
||||
**dict(inherited_tls_ca_environment()),
|
||||
}),
|
||||
env_allowlist=(SECRET_ENV_KEY, *TLS_CA_ENV_KEYS),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=timeout,
|
||||
|
|
@ -346,17 +357,20 @@ class CodexJSONLParser:
|
|||
return tuple(observations)
|
||||
|
||||
def _tool_interval(self, record: dict[str, Any]) -> ParsedMetric | None:
|
||||
"""Observe one explicitly paired tool interval; never infer a pairing."""
|
||||
"""Count a completed tool and optionally observe its reported interval."""
|
||||
item = record.get("item")
|
||||
if not isinstance(item, dict) or "duration_ms" not in item:
|
||||
if not isinstance(item, dict) or item.get("type") not in _CODEX_TOOL_ITEM_TYPES:
|
||||
return None
|
||||
call_id = item.get("id")
|
||||
if not isinstance(call_id, str) or _CODEX_CALL_ID_RE.fullmatch(call_id) is None:
|
||||
raise CodexJSONLError("unpaired Codex tool interval")
|
||||
raise CodexJSONLError("unpaired Codex tool completion")
|
||||
if call_id in self._tool_calls:
|
||||
raise CodexJSONLError("duplicate Codex tool completion")
|
||||
if "duration_ms" not in item:
|
||||
self._tool_calls.add(call_id)
|
||||
return None
|
||||
if not is_reported_number(item["duration_ms"]):
|
||||
raise CodexJSONLError("invalid Codex tool interval")
|
||||
if call_id in self._tool_calls:
|
||||
raise CodexJSONLError("duplicate Codex tool interval")
|
||||
try:
|
||||
observation = duration_metric(
|
||||
"tool_duration", item["duration_ms"], reported_unit="ms",
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import sys
|
|||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.codex_iop import (
|
||||
BASE_URL_ENV_KEY,
|
||||
|
|
@ -95,13 +96,14 @@ class CodexIOPTest(unittest.TestCase):
|
|||
return str(path)
|
||||
|
||||
def test_exact_isolated_responses_spec_uses_one_stdin_submission(self) -> None:
|
||||
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
|
||||
argv = list(spec.argv)
|
||||
codex = argv[argv.index("--") + 1:]
|
||||
self.assertFalse((self.workspace / ".git").exists())
|
||||
self.assertEqual(codex.count("--skip-git-repo-check"), 1)
|
||||
self.assertEqual(codex[:12], [
|
||||
"codex", "exec", "--json", "--ephemeral", "--ignore-user-config",
|
||||
self.assertEqual(codex[:14], [
|
||||
"codex", "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
|
||||
"--strict-config", "--skip-git-repo-check", "-C", str(self.workspace),
|
||||
"-m", "gpt-5.6-luna", "-c",
|
||||
])
|
||||
|
|
@ -118,6 +120,8 @@ class CodexIOPTest(unittest.TestCase):
|
|||
self.assertEqual(spec.submission_mode, "stdin_once")
|
||||
self.assertEqual(spec.completion_mode, "exit_after_idle")
|
||||
self.assertEqual(dict(spec.env)[SECRET_ENV_KEY], _SECRET)
|
||||
self.assertEqual(dict(spec.env)["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(dict(spec.env)["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertNotIn(BASE_URL_ENV_KEY, dict(spec.env))
|
||||
self.assertNotIn("OPENAI_API_KEY", dict(spec.env))
|
||||
|
||||
|
|
@ -142,8 +146,9 @@ class CodexIOPTest(unittest.TestCase):
|
|||
self.assertEqual(events[2][-1], "finish")
|
||||
turn = {metric.name: metric.value for metric in events[2][:-1]}
|
||||
self.assertEqual(turn, {
|
||||
"cached_input_tokens": 8, "input_tokens": 31, "output_tokens": 12,
|
||||
"reasoning_tokens": 4, "model_calls": 1, "tool_calls": 1,
|
||||
"cache_write_tokens": 6, "cached_input_tokens": 8,
|
||||
"input_tokens": 31, "output_tokens": 12, "reasoning_tokens": 4,
|
||||
"model_calls": 1, "tool_calls": 1,
|
||||
})
|
||||
# The fixture omits the provider total, so it is never reconstructed
|
||||
# from the reported categories.
|
||||
|
|
@ -197,25 +202,31 @@ class CodexIOPTest(unittest.TestCase):
|
|||
|
||||
def test_tool_intervals_require_one_explicit_unique_pairing(self) -> None:
|
||||
parser = CodexJSONLParser(_cell(), "0123456789abcdef")
|
||||
# An item without a reported duration carries no interval to pair.
|
||||
# A non-tool item carries neither a tool count nor an interval.
|
||||
self.assertIsNone(parser.parse("stdout", json.dumps(
|
||||
{"type": "item.completed", "item": {"type": "agent_message", "text": "x"}}
|
||||
)))
|
||||
paired = json.dumps({"type": "item.completed", "item": {"id": "call-1", "duration_ms": 3}})
|
||||
# Real Codex command completions omit duration; they must still count.
|
||||
self.assertIsNone(parser.parse("stdout", json.dumps(
|
||||
{"type": "item.completed", "item": {"id": "call-0", "type": "command_execution"}}
|
||||
)))
|
||||
paired = json.dumps({"type": "item.completed", "item": {
|
||||
"id": "call-1", "type": "command_execution", "duration_ms": 3,
|
||||
}})
|
||||
self.assertEqual(parser.parse("stdout", paired).call_id, "call-1")
|
||||
for name, item in {
|
||||
"duplicate-call": {"id": "call-1", "duration_ms": 4},
|
||||
"unpaired-duration": {"duration_ms": 4},
|
||||
"unsafe-call-id": {"id": "call 1", "duration_ms": 4},
|
||||
"string-duration": {"id": "call-2", "duration_ms": "4"},
|
||||
"negative-duration": {"id": "call-3", "duration_ms": -4},
|
||||
"duplicate-call": {"id": "call-1", "type": "command_execution", "duration_ms": 4},
|
||||
"unpaired-duration": {"type": "command_execution", "duration_ms": 4},
|
||||
"unsafe-call-id": {"id": "call 1", "type": "command_execution", "duration_ms": 4},
|
||||
"string-duration": {"id": "call-2", "type": "command_execution", "duration_ms": "4"},
|
||||
"negative-duration": {"id": "call-3", "type": "command_execution", "duration_ms": -4},
|
||||
}.items():
|
||||
with self.subTest(name=name):
|
||||
with self.assertRaises(CodexJSONLError):
|
||||
parser.parse("stdout", json.dumps({"type": "item.completed", "item": item}))
|
||||
turn = parser.parse("stdout", json.dumps({"type": "turn.completed"}))
|
||||
counts = {metric.name: metric.value for metric in turn[:-1]}
|
||||
self.assertEqual(counts, {"model_calls": 1, "tool_calls": 1})
|
||||
self.assertEqual(counts, {"model_calls": 1, "tool_calls": 2})
|
||||
|
||||
def test_unknown_or_fractional_turn_usage_fails_closed(self) -> None:
|
||||
for usage in (
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ from scripts import agent_comparison_benchmark as benchmark_cli
|
|||
from scripts.agent_benchmark import live_iop
|
||||
from scripts.agent_benchmark.browser_cdp import RenderObservation, ViewportObservation
|
||||
from scripts.agent_benchmark.attempts import (
|
||||
AttemptStateError,
|
||||
CapabilityUnavailable,
|
||||
PreflightObservation,
|
||||
RunBusyError,
|
||||
|
|
@ -42,7 +43,7 @@ from scripts.agent_benchmark.connectivity import (
|
|||
canonical_evidence_bytes,
|
||||
make_result,
|
||||
)
|
||||
from scripts.agent_benchmark.codex_iop import CodexInvocationResult
|
||||
from scripts.agent_benchmark.codex_iop import CodexInvocation, CodexInvocationResult
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
AssetMapping,
|
||||
ExpectedBinding,
|
||||
|
|
@ -200,15 +201,17 @@ if name == "claude":
|
|||
"result": leak},
|
||||
]
|
||||
elif name == "agy":
|
||||
binding = {"route_kind": "direct", "route_id": route_id,
|
||||
"model": option("--model"), "effort": option("--effort")}
|
||||
events = [
|
||||
{"type": "metric", "subtype": "duration_ms", "value": 12.5},
|
||||
dict(binding, type="iop", subtype="effective_binding",
|
||||
stages=[{"stage": "request", "model": binding["model"],
|
||||
"effort": binding["effort"]}]),
|
||||
dict(binding, type="result", subtype="success", text=leak),
|
||||
dict(binding, type="system", subtype="idle"),
|
||||
{"event": "init", "conversation_id": "fixture",
|
||||
"init": {"model": "Gemini 3.6 Flash", "permission_mode": "sandbox", "tools": []}},
|
||||
{"event": "step_update", "step_update": {
|
||||
"state": "DONE", "step_index": 0, "step_type": "agent_response",
|
||||
"text_delta": leak,
|
||||
"usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}},
|
||||
{"event": "result", "result": {
|
||||
"conversation_id": "fixture", "status": "SUCCESS",
|
||||
"duration_seconds": 0.0125, "num_turns": 1, "response": leak,
|
||||
"usage": {"input_tokens": 11, "output_tokens": 22, "total_tokens": 33}}},
|
||||
]
|
||||
else:
|
||||
events = [
|
||||
|
|
@ -217,10 +220,8 @@ else:
|
|||
"item": {"id": "call-1", "type": "command_execution",
|
||||
"duration_ms": 7.25, "output": leak}},
|
||||
{"type": "turn.completed", "status": "completed",
|
||||
"usage": {"input_tokens": 31, "cached_input_tokens": 8, "output_tokens": 12},
|
||||
"iop_effective_binding": {
|
||||
"route_kind": "direct", "route_id": route_id, "model": option("-m"),
|
||||
"effort": override("model_reasoning_effort=")}},
|
||||
"usage": {"input_tokens": 31, "cached_input_tokens": 8,
|
||||
"cache_write_input_tokens": 6, "output_tokens": 12}},
|
||||
]
|
||||
|
||||
for event in events:
|
||||
|
|
@ -465,6 +466,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
for caller in ("CLAUDE", "AGY", "CODEX"):
|
||||
environment[f"IOP_BENCH_{caller}_BASE_URL"] = "http://127.0.0.1:18083/v1"
|
||||
environment[f"IOP_BENCH_{caller}_SECRET_ENV"] = "BENCH_TOKEN"
|
||||
environment["IOP_BENCH_AGY_BASE_URL"] = "https://127.0.0.1:18083"
|
||||
return environment
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -691,6 +693,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
environment[secret_env] = branch["secret"]
|
||||
environment[prefix + "SECRET_ENV"] = secret_env
|
||||
environment[prefix + "BASE_URL"] = f"http://{branch['endpoint']}.invalid:18083/v1"
|
||||
if caller == "agy":
|
||||
environment[prefix + "BASE_URL"] = f"https://{branch['endpoint']}.invalid:18083"
|
||||
return environment
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -824,10 +828,13 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
"total_duration", "model_duration", "model_calls", "input_tokens",
|
||||
"output_tokens", "cached_input_tokens",
|
||||
},
|
||||
"agy": {"total_duration"},
|
||||
"agy": {
|
||||
"total_duration", "model_calls", "input_tokens", "output_tokens",
|
||||
"total_tokens",
|
||||
},
|
||||
"codex": {
|
||||
"model_calls", "tool_calls", "input_tokens", "output_tokens",
|
||||
"cached_input_tokens",
|
||||
"cached_input_tokens", "cache_write_tokens",
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -848,9 +855,13 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
if item.status == "observed"
|
||||
}
|
||||
self.assertEqual(observed, self._EXPECTED_TOTALS[caller], attempt_root)
|
||||
# No caller reports a provider total, and none is reconstructed.
|
||||
self.assertEqual(measurement.usage["total_tokens"].status, "unavailable")
|
||||
self.assertIsNone(measurement.usage["total_tokens"].value)
|
||||
# Official agy reports total_tokens; other callers in this fixture
|
||||
# do not, and no total is reconstructed for them.
|
||||
if caller == "agy":
|
||||
self.assertEqual(measurement.usage["total_tokens"].value, 33)
|
||||
else:
|
||||
self.assertEqual(measurement.usage["total_tokens"].status, "unavailable")
|
||||
self.assertIsNone(measurement.usage["total_tokens"].value)
|
||||
self.assertEqual(measurement.timeline["first_output_at"].status, "observed")
|
||||
self.assertEqual(
|
||||
measurement.timeline["first_write_mtime"].clock, "filesystem_mtime"
|
||||
|
|
@ -910,7 +921,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
root.mkdir()
|
||||
matrix = [
|
||||
_cell("claude", "claude", "sonnet", "max"),
|
||||
_cell("agy", "agy", "gemini", "high"),
|
||||
_cell("agy", "agy", "gemini-3.6-flash", "high"),
|
||||
_cell("codex", "codex", "gpt", "xhigh"),
|
||||
]
|
||||
sentinels = _branch_sentinels()
|
||||
|
|
@ -931,7 +942,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
tuple(sorted(cell.iop.request_model for cell in manifest.matrix)),
|
||||
"sha256:" + "f" * 64,
|
||||
True,
|
||||
"agy 1.1.11",
|
||||
"agy 1.1.12",
|
||||
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
|
||||
)
|
||||
|
||||
|
|
@ -1006,14 +1017,15 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
self.assertFalse((run_roots[0] / "cells").exists())
|
||||
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
|
||||
|
||||
def test_cli_mixed_manifest_never_invokes_unobserved_preset_cells(self) -> None:
|
||||
def test_cli_mixed_manifest_preflights_and_invokes_every_cell(self) -> None:
|
||||
self._init_testbed()
|
||||
manifest, _, path = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_cell("direct-ready", "claude", "claude-sonnet-5", "max"),
|
||||
_preset("preset-unobserved", "claude", "claude-sonnet-5", "max"),
|
||||
_preset("preset-ready", "claude", "claude-sonnet-5", "max"),
|
||||
],
|
||||
output_id="mixed-unobserved",
|
||||
output_id="mixed-ready",
|
||||
)
|
||||
registry = self._registry()
|
||||
stdout = io.StringIO()
|
||||
|
|
@ -1028,10 +1040,9 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
):
|
||||
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
|
||||
|
||||
self.assertEqual(exit_code, 69)
|
||||
self.assertEqual(stdout.getvalue(), "")
|
||||
self.assertIn("error: benchmark execution failed", stderr.getvalue())
|
||||
self.assertIn("completed=0 unresolved=2", stderr.getvalue())
|
||||
self.assertEqual(exit_code, 0, stderr.getvalue())
|
||||
self.assertIn("ok: run run_id=", stdout.getvalue())
|
||||
self.assertEqual(stderr.getvalue(), "")
|
||||
run_roots = list((self.root / manifest.output_root).glob("run-*"))
|
||||
self.assertEqual(len(run_roots), 1)
|
||||
preflight = json.loads(
|
||||
|
|
@ -1042,11 +1053,62 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
self.assertEqual(preflight["status"], "ready")
|
||||
self.assertEqual(
|
||||
[result["cell"]["id"] for result in preflight["results"]],
|
||||
["direct-ready"],
|
||||
[cell.id for cell in manifest.matrix],
|
||||
)
|
||||
self.assertFalse((run_roots[0] / "cells").exists())
|
||||
self.assertEqual(registry["claude"].calls, ["direct-ready"])
|
||||
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
|
||||
self.assertEqual(
|
||||
registry["claude"].calls,
|
||||
[cell.id for cell in manifest.matrix],
|
||||
)
|
||||
self.assertEqual(
|
||||
[item[0] for item in registry["claude"].invocations],
|
||||
[cell.id for cell in manifest.matrix],
|
||||
)
|
||||
self.assertEqual(
|
||||
len(list(run_roots[0].glob("cells/*/repetition-*/attempt-*"))),
|
||||
len(manifest.matrix),
|
||||
)
|
||||
|
||||
def test_cli_failed_attempt_does_not_prevent_later_slot(self) -> None:
|
||||
self._init_testbed()
|
||||
manifest, _, path = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_cell("agy-first", "agy", "gemini-3.6-flash", "high"),
|
||||
_cell("codex-later", "codex", "gpt-5.6-luna", "xhigh"),
|
||||
],
|
||||
output_id="terminal-failure-continues",
|
||||
)
|
||||
agy = FakeAdapter("agy", ("high",))
|
||||
agy.fail_invocation = True
|
||||
codex = FakeAdapter("codex", ("xhigh",))
|
||||
self.addCleanup(agy.cleanup)
|
||||
self.addCleanup(codex.cleanup)
|
||||
stdout = io.StringIO()
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root),
|
||||
mock.patch.object(
|
||||
benchmark_cli,
|
||||
"build_adapter_registry",
|
||||
return_value={"agy": agy, "codex": codex},
|
||||
),
|
||||
contextlib.redirect_stdout(stdout),
|
||||
contextlib.redirect_stderr(stderr),
|
||||
):
|
||||
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
|
||||
|
||||
self.assertEqual(exit_code, 69)
|
||||
self.assertEqual(stdout.getvalue(), "")
|
||||
match = re.search(r"run_id=(run-[0-9A-Za-z-]+)", stderr.getvalue())
|
||||
self.assertIsNotNone(match)
|
||||
self.assertEqual([item[0] for item in agy.invocations], ["agy-first"])
|
||||
self.assertEqual([item[0] for item in codex.invocations], ["codex-later"])
|
||||
status = self.store.status(manifest=manifest, run=self.store.open(
|
||||
manifest, match.group(1) # type: ignore[union-attr]
|
||||
))
|
||||
self.assertEqual(status["attempts"]["failed"], 1)
|
||||
self.assertEqual(status["attempts"]["success"], 1)
|
||||
self.assertEqual(status["attempts"]["running"], 0)
|
||||
|
||||
def test_cli_resume_retries_append_only_and_status_is_read_only(self) -> None:
|
||||
self._init_testbed()
|
||||
|
|
@ -1163,8 +1225,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
)
|
||||
self.assertFalse(output_root.exists())
|
||||
|
||||
def test_generic_preset_cells_are_local_contract_only(self) -> None:
|
||||
generic, _, _ = _write_manifest(
|
||||
def test_preset_only_public_preflight_appends_exact_results(self) -> None:
|
||||
generic, _, path = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
|
||||
|
|
@ -1174,31 +1236,6 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
output_id="generic",
|
||||
)
|
||||
registry = self._registry()
|
||||
observations = collect_preflight_observations(generic, registry)
|
||||
self.assertEqual(observations, {})
|
||||
self.assertTrue(all(adapter.calls == [] for adapter in registry.values()))
|
||||
|
||||
def test_generic_preset_only_public_preflight_fails_closed_without_run_state(self) -> None:
|
||||
generic, raw, path = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
|
||||
_preset("agy-generic", "agy", "gemini-3.6-flash", "high"),
|
||||
_preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"),
|
||||
],
|
||||
output_id="generic",
|
||||
)
|
||||
registry = self._registry()
|
||||
output_root = self.root / generic.output_root
|
||||
with self.assertRaises(Exception) as ctx:
|
||||
preflight_manifest(
|
||||
self.store, generic, raw, adapters=registry
|
||||
)
|
||||
self.assertIn("preflight requires a direct cell", str(ctx.exception))
|
||||
self.assertFalse(output_root.exists())
|
||||
self.assertTrue(all(adapter.calls == [] for adapter in registry.values()))
|
||||
|
||||
sentinel = "private_endpoint_and_token_must_not_appear"
|
||||
stdout = io.StringIO()
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
|
|
@ -1208,10 +1245,103 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
contextlib.redirect_stderr(stderr),
|
||||
):
|
||||
exit_code = benchmark_cli.main(["preflight", "--manifest", str(path)])
|
||||
|
||||
self.assertEqual(exit_code, 0, stderr.getvalue())
|
||||
self.assertIn("status=ready", stdout.getvalue())
|
||||
self.assertEqual(stderr.getvalue(), "")
|
||||
run_roots = list((self.root / generic.output_root).glob("run-*"))
|
||||
self.assertEqual(len(run_roots), 1)
|
||||
record = json.loads(
|
||||
(run_roots[0] / "preflight/preflight-000001.json").read_text(
|
||||
encoding="ascii"
|
||||
)
|
||||
)
|
||||
self.assertEqual(
|
||||
[result["cell"]["id"] for result in record["results"]],
|
||||
[cell.id for cell in generic.matrix],
|
||||
)
|
||||
self.assertFalse((run_roots[0] / "cells").exists())
|
||||
self.assertEqual(
|
||||
{caller: adapter.calls for caller, adapter in registry.items()},
|
||||
{
|
||||
"claude": ["claude-generic"],
|
||||
"agy": ["agy-generic"],
|
||||
"codex": ["codex-generic"],
|
||||
},
|
||||
)
|
||||
|
||||
def test_preset_blocker_appends_without_attempt_allocation(self) -> None:
|
||||
generic, _, path = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_preset("claude-generic", "claude", "claude-sonnet-5", "high"),
|
||||
_preset("agy-generic", "agy", "gemini-3.6-flash", "high"),
|
||||
_preset("codex-generic", "codex", "gpt-5.6-luna", "xhigh"),
|
||||
],
|
||||
output_id="generic-blocked",
|
||||
)
|
||||
registry = self._registry(
|
||||
{"agy-generic": ("credential_missing",)}
|
||||
)
|
||||
stdout = io.StringIO()
|
||||
stderr = io.StringIO()
|
||||
with (
|
||||
mock.patch.object(benchmark_cli, "_REPO_ROOT", self.root),
|
||||
mock.patch.object(benchmark_cli, "build_adapter_registry", return_value=registry),
|
||||
contextlib.redirect_stdout(stdout),
|
||||
contextlib.redirect_stderr(stderr),
|
||||
):
|
||||
exit_code = benchmark_cli.main(["run", "--manifest", str(path)])
|
||||
self.assertEqual(exit_code, 69)
|
||||
self.assertEqual(stdout.getvalue(), "")
|
||||
self.assertNotIn(sentinel, stderr.getvalue())
|
||||
self.assertFalse(output_root.exists())
|
||||
self.assertIn("error: preflight blocked", stderr.getvalue())
|
||||
run_roots = list((self.root / generic.output_root).glob("run-*"))
|
||||
self.assertEqual(len(run_roots), 1)
|
||||
record = json.loads(
|
||||
(run_roots[0] / "preflight/preflight-000001.json").read_text(
|
||||
encoding="ascii"
|
||||
)
|
||||
)
|
||||
self.assertEqual(record["status"], "registration_required")
|
||||
self.assertEqual(
|
||||
[result["cell"]["id"] for result in record["results"]],
|
||||
[cell.id for cell in generic.matrix],
|
||||
)
|
||||
self.assertFalse((run_roots[0] / "cells").exists())
|
||||
self.assertTrue(all(adapter.invocations == [] for adapter in registry.values()))
|
||||
|
||||
def test_all_cell_preflight_rejects_missing_extra_and_reordered_results(self) -> None:
|
||||
manifest, raw, _ = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_cell("direct", "claude", "claude-sonnet-5", "max"),
|
||||
_preset("preset", "claude", "claude-sonnet-5", "max"),
|
||||
],
|
||||
output_id="all-cell-corruption",
|
||||
)
|
||||
run, record = preflight_manifest(
|
||||
self.store, manifest, raw, adapters=self._registry()
|
||||
)
|
||||
path = Path(run.root) / "preflight/preflight-000001.json"
|
||||
original = path.read_bytes()
|
||||
mutations = {
|
||||
"missing": record["results"][:-1],
|
||||
"extra": [*record["results"], record["results"][0]],
|
||||
"reordered": list(reversed(record["results"])),
|
||||
}
|
||||
for label, results in mutations.items():
|
||||
with self.subTest(label=label):
|
||||
mutated = {**record, "results": results}
|
||||
path.write_bytes(
|
||||
json.dumps(
|
||||
mutated, sort_keys=True, separators=(",", ":")
|
||||
).encode("ascii")
|
||||
+ b"\n"
|
||||
)
|
||||
with self.assertRaises(AttemptStateError):
|
||||
self.store.preflights(run, manifest)
|
||||
path.write_bytes(original)
|
||||
self.assertEqual(self.store.preflights(run, manifest), (record,))
|
||||
|
||||
def test_live_registry_dereferences_secret_names_without_persisting_values(self) -> None:
|
||||
sentinel = "live-token-must-not-persist"
|
||||
|
|
@ -1223,7 +1353,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)),
|
||||
"sha256:" + "c" * 64,
|
||||
True,
|
||||
"agy 1.1.11",
|
||||
"agy 1.1.12",
|
||||
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
|
||||
)
|
||||
|
||||
|
|
@ -1294,6 +1424,8 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
)
|
||||
captured = {}
|
||||
|
||||
caller_binding = {"value": None}
|
||||
|
||||
def invoke(invocation, _on_started):
|
||||
captured["spec"] = invocation.spec
|
||||
stream = CaptureStream("stdout", "", 0, 0, False)
|
||||
|
|
@ -1304,13 +1436,9 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
"2026-08-11T00:00:00+00:00",
|
||||
"2026-08-11T00:00:01+00:00", 1, (),
|
||||
)
|
||||
binding = (
|
||||
evaluator.iop.route_kind,
|
||||
evaluator.iop.route_id,
|
||||
evaluator.iop.request_model,
|
||||
evaluator.iop.requested_effort,
|
||||
return CodexInvocationResult(
|
||||
lifecycle, caller_binding["value"]
|
||||
)
|
||||
return CodexInvocationResult(lifecycle, binding)
|
||||
|
||||
adapter = live_iop.build_live_scoring_adapter(
|
||||
environment,
|
||||
|
|
@ -1349,10 +1477,45 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
lambda *_args: None,
|
||||
)
|
||||
self.assertTrue(result.success)
|
||||
self.assertEqual(result.effective_binding[0], route_kind)
|
||||
expected_binding = (
|
||||
evaluator.iop.route_kind,
|
||||
evaluator.iop.route_id,
|
||||
evaluator.iop.request_model,
|
||||
evaluator.iop.requested_effort,
|
||||
)
|
||||
self.assertEqual(result.effective_binding, expected_binding)
|
||||
finalized = adapter.finalize_evidence(blind)
|
||||
self.assertTrue(finalized.safe)
|
||||
|
||||
caller_binding["value"] = (
|
||||
evaluator.iop.route_kind,
|
||||
"contradictory-route",
|
||||
evaluator.iop.request_model,
|
||||
evaluator.iop.requested_effort,
|
||||
)
|
||||
mismatch_root = self.root / "runs" / f"blind-{route_kind}-mismatch"
|
||||
for name in ("input", "session", "output"):
|
||||
(mismatch_root / name).mkdir(parents=True, exist_ok=True)
|
||||
mismatch_blind = BlindWorkspace(
|
||||
f"blind-{route_kind}-mismatch", str(mismatch_root),
|
||||
str(mismatch_root / "input"), str(mismatch_root / "session"),
|
||||
str(mismatch_root / "output"), "sha256:" + "e" * 64,
|
||||
"sha256:" + "f" * 64,
|
||||
)
|
||||
mismatch = adapter.invoke(
|
||||
evaluator,
|
||||
mismatch_blind,
|
||||
prompt,
|
||||
self.manifest.timeout,
|
||||
lambda *_args: None,
|
||||
)
|
||||
self.assertFalse(mismatch.success)
|
||||
self.assertEqual(mismatch.terminal_reason, "binding_mismatch")
|
||||
self.assertEqual(
|
||||
mismatch.effective_binding, caller_binding["value"]
|
||||
)
|
||||
self.assertTrue(adapter.finalize_evidence(mismatch_blind).safe)
|
||||
|
||||
spec = captured["spec"]
|
||||
visible = "\n".join(
|
||||
(*spec.argv, spec.cwd, *(value for pair in spec.env for value in pair))
|
||||
|
|
@ -2289,7 +2452,7 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
]
|
||||
no_model = {**environment, "BENCH_CONFIG": json.dumps({"schema_version": "1", "routes": no_model_routes})}
|
||||
unsupported = replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy"), iop=replace(next(cell for cell in self.manifest.matrix if cell.caller == "agy").iop, requested_effort="max"))
|
||||
observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.11", "--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json")
|
||||
observed = lambda _runtime: live_iop._Observation(tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)), "sha256:" + "1" * 64, True, "agy 1.1.12", "--print --output-format --sandbox --model --effort stream-json")
|
||||
checks = (
|
||||
(live_iop.build_live_adapter_registry(no_route, observer=observed)["claude"].preflight(claude).result, "route_missing"),
|
||||
(live_iop.build_live_adapter_registry(no_model, observer=observed)["claude"].preflight(claude).result, "model_missing"),
|
||||
|
|
@ -2310,12 +2473,12 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
(("effort_unsupported", ISSUE_RESUME_CODES["effort_unsupported"]),),
|
||||
)
|
||||
|
||||
def test_live_invocation_rejects_missing_or_mismatched_caller_binding(self) -> None:
|
||||
def test_live_invocation_uses_config_owned_codex_binding_and_rejects_mismatch(self) -> None:
|
||||
environment = self._live_environment()
|
||||
observed = lambda _runtime: live_iop._Observation(
|
||||
tuple(sorted(cell.iop.request_model for cell in self.manifest.matrix)),
|
||||
"sha256:" + "2" * 64, True, "agy 1.1.11",
|
||||
"--print --output-format --sandbox --model --effort AGY_PROVIDER AGY_OPENAI_BASE_URL AGY_OPENAI_API_KEY stream-json",
|
||||
"sha256:" + "2" * 64, True, "agy 1.1.12",
|
||||
"--print --output-format --sandbox --model --effort stream-json",
|
||||
)
|
||||
registry = live_iop.build_live_adapter_registry(
|
||||
environment, observer=observed, binary_resolver=lambda _name: "/bin/true"
|
||||
|
|
@ -2332,11 +2495,6 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
lambda *_args: None,
|
||||
live_iop._DEFAULT_INVOKERS.codex,
|
||||
)
|
||||
codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
|
||||
live_iop._DEFAULT_INVOKERS.claude,
|
||||
live_iop._DEFAULT_INVOKERS.agy,
|
||||
lambda *_args: type("Mismatch", (), {"effective_binding": None, "lifecycle": None})(),
|
||||
)
|
||||
with mock.patch.object(live_iop, "build_agy_invocation", return_value=object()):
|
||||
with self.assertRaises(live_iop.LiveIopError) as raised:
|
||||
agy_adapter.invoke(
|
||||
|
|
@ -2344,7 +2502,74 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
self.manifest.timeout, lambda *_args: None,
|
||||
)
|
||||
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
|
||||
with mock.patch.object(live_iop, "build_codex_invocation", return_value=type("Invocation", (), {"spec": object()})()):
|
||||
|
||||
failed_stream = CaptureStream("stdout", "", 0, 0, False)
|
||||
failed_lifecycle = InvocationResult(
|
||||
False, "nonzero_exit", 1, None, True, False, True, False,
|
||||
(), failed_stream, replace(failed_stream, stream="stderr"), "", "", None,
|
||||
"sha256:" + "9" * 64,
|
||||
"2026-08-11T00:00:00+00:00",
|
||||
"2026-08-11T00:00:01+00:00", 1, (),
|
||||
)
|
||||
agy_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
|
||||
live_iop._DEFAULT_INVOKERS.claude,
|
||||
lambda *_args: failed_lifecycle,
|
||||
live_iop._DEFAULT_INVOKERS.codex,
|
||||
)
|
||||
with (
|
||||
mock.patch.object(live_iop, "build_agy_invocation", return_value=object()),
|
||||
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
|
||||
):
|
||||
failed = agy_adapter.invoke(
|
||||
agy, object(), object(), "/tmp/control", b"task",
|
||||
self.manifest.timeout, lambda *_args: None,
|
||||
)
|
||||
self.assertIs(failed, failed_lifecycle)
|
||||
|
||||
stream = CaptureStream("stdout", "", 0, 0, False)
|
||||
lifecycle = InvocationResult(
|
||||
True, "success", 0, None, True, True, True, False,
|
||||
(), stream, replace(stream, stream="stderr"), "", "", None,
|
||||
"sha256:" + "a" * 64,
|
||||
"2026-08-11T00:00:00+00:00",
|
||||
"2026-08-11T00:00:01+00:00", 1, (),
|
||||
)
|
||||
invocation = CodexInvocation(
|
||||
InvocationSpec(
|
||||
argv=("codex",), cwd="/tmp", env=(),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=self.manifest.timeout, evidence_dir="/tmp",
|
||||
),
|
||||
None, # type: ignore[arg-type]
|
||||
lambda line: line,
|
||||
)
|
||||
observed_binding = {"value": None}
|
||||
|
||||
def invoke_codex(*_args):
|
||||
return CodexInvocationResult(lifecycle, observed_binding["value"])
|
||||
|
||||
codex_adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
|
||||
live_iop._DEFAULT_INVOKERS.claude,
|
||||
live_iop._DEFAULT_INVOKERS.agy,
|
||||
invoke_codex,
|
||||
)
|
||||
with (
|
||||
mock.patch.object(live_iop, "build_codex_invocation", return_value=invocation),
|
||||
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
|
||||
):
|
||||
result = codex_adapter.invoke(
|
||||
codex, object(), object(), "/tmp/control", b"task",
|
||||
self.manifest.timeout, lambda *_args: None,
|
||||
)
|
||||
self.assertIs(result, lifecycle)
|
||||
|
||||
observed_binding["value"] = (
|
||||
codex.iop.route_kind,
|
||||
"contradictory-route",
|
||||
codex.iop.request_model,
|
||||
codex.iop.requested_effort,
|
||||
)
|
||||
with self.assertRaises(live_iop.LiveIopError) as raised:
|
||||
codex_adapter.invoke(
|
||||
codex, object(), object(), "/tmp/control", b"task",
|
||||
|
|
@ -2352,6 +2577,145 @@ class ConnectivityIntegrationTest(unittest.TestCase):
|
|||
)
|
||||
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
|
||||
|
||||
def test_live_agy_multiple_cells_consume_their_own_preflight_state(self) -> None:
|
||||
manifest, _, _ = _write_manifest(
|
||||
self.root,
|
||||
[
|
||||
_cell("agy-direct", "agy", "gemini-3.6-flash", "high"),
|
||||
_preset("agy-preset", "agy", "gemini-hybrid", "high"),
|
||||
],
|
||||
output_id="agy-cell-state",
|
||||
)
|
||||
environment = self._live_environment(manifest=manifest)
|
||||
observed_models = tuple(
|
||||
sorted(cell.iop.request_model for cell in manifest.matrix)
|
||||
)
|
||||
ready_observation = lambda _runtime: live_iop._Observation(
|
||||
observed_models,
|
||||
"sha256:" + "2" * 64,
|
||||
True,
|
||||
"agy 1.1.12",
|
||||
"--print --output-format --sandbox --model --effort stream-json",
|
||||
)
|
||||
adapter = live_iop.build_live_adapter_registry(
|
||||
environment,
|
||||
observer=ready_observation,
|
||||
binary_resolver=lambda _name: "/bin/true",
|
||||
)["agy"]
|
||||
cells = {cell.id: cell for cell in manifest.matrix}
|
||||
for cell in manifest.matrix:
|
||||
self.assertEqual(adapter.preflight(cell).result.status, "ready")
|
||||
|
||||
preflights = dict(adapter._agy_preflights) # type: ignore[attr-defined]
|
||||
self.assertEqual(set(preflights), set(cells))
|
||||
self.assertIsNot(preflights["agy-direct"], preflights["agy-preset"])
|
||||
self.assertEqual(
|
||||
preflights["agy-direct"].runtime.observation.cell_id,
|
||||
"agy-direct",
|
||||
)
|
||||
self.assertEqual(
|
||||
preflights["agy-preset"].runtime.observation.cell_id,
|
||||
"agy-preset",
|
||||
)
|
||||
|
||||
stream = CaptureStream("stdout", "", 0, 0, False)
|
||||
lifecycle = InvocationResult(
|
||||
True,
|
||||
"success",
|
||||
0,
|
||||
None,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
False,
|
||||
(),
|
||||
stream,
|
||||
replace(stream, stream="stderr"),
|
||||
"",
|
||||
"",
|
||||
None,
|
||||
"sha256:" + "a" * 64,
|
||||
"2026-08-12T00:00:00+00:00",
|
||||
"2026-08-12T00:00:01+00:00",
|
||||
1,
|
||||
(),
|
||||
)
|
||||
built: list[tuple[str, object]] = []
|
||||
invoked: list[tuple[str, object]] = []
|
||||
|
||||
def build(cell, _prepared, _task, _timeout, preflight):
|
||||
built.append((cell.id, preflight))
|
||||
return object()
|
||||
|
||||
def invoke(_spec, parser, preflight, _on_started):
|
||||
invoked.append((parser._cell.id, preflight))
|
||||
return lifecycle
|
||||
|
||||
def observed_result(parser, capability, _result):
|
||||
cell = parser._cell
|
||||
admitted = adapter._admitted_bindings[cell.id] # type: ignore[attr-defined]
|
||||
caller_capability = CallerCapability(
|
||||
"agy", capability.route_kinds, capability.efforts
|
||||
)
|
||||
return make_result(cell, caller_capability, admitted)
|
||||
|
||||
adapter._invokers = live_iop._InvokerSeams( # type: ignore[attr-defined]
|
||||
live_iop._DEFAULT_INVOKERS.claude,
|
||||
invoke,
|
||||
live_iop._DEFAULT_INVOKERS.codex,
|
||||
)
|
||||
with (
|
||||
mock.patch.object(live_iop, "build_agy_invocation", side_effect=build),
|
||||
mock.patch.object(live_iop, "_bind_live_spec", side_effect=lambda *_args: _args[-1]),
|
||||
mock.patch.object(
|
||||
live_iop.AgyEventParser,
|
||||
"observed_result",
|
||||
new=observed_result,
|
||||
),
|
||||
):
|
||||
for cell_id in ("agy-preset", "agy-direct"):
|
||||
cell = cells[cell_id]
|
||||
result = adapter.invoke(
|
||||
cell,
|
||||
object(),
|
||||
object(),
|
||||
"/tmp/control",
|
||||
b"task",
|
||||
manifest.timeout,
|
||||
lambda *_args: None,
|
||||
)
|
||||
self.assertIs(result, lifecycle)
|
||||
|
||||
self.assertEqual(
|
||||
built,
|
||||
[
|
||||
("agy-preset", preflights["agy-preset"]),
|
||||
("agy-direct", preflights["agy-direct"]),
|
||||
],
|
||||
)
|
||||
self.assertEqual(invoked, built)
|
||||
|
||||
adapter._observer = lambda _runtime: live_iop._Observation( # type: ignore[attr-defined]
|
||||
observed_models,
|
||||
"sha256:" + "2" * 64,
|
||||
False,
|
||||
)
|
||||
self.assertEqual(
|
||||
adapter.preflight(cells["agy-direct"]).result.status,
|
||||
"implementation_gap",
|
||||
)
|
||||
with self.assertRaises(live_iop.LiveIopError) as raised:
|
||||
adapter.invoke(
|
||||
cells["agy-direct"],
|
||||
object(),
|
||||
object(),
|
||||
"/tmp/control",
|
||||
b"task",
|
||||
manifest.timeout,
|
||||
lambda *_args: None,
|
||||
)
|
||||
self.assertEqual(raised.exception.issue_code, "stream_incompatible")
|
||||
|
||||
def test_cli_missing_live_input_fails_closed_without_secret_or_attempt(self) -> None:
|
||||
sentinel = "missing-input-token"
|
||||
stdout = io.StringIO()
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ from __future__ import annotations
|
|||
|
||||
import ctypes
|
||||
import datetime
|
||||
import errno
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
|
|
@ -28,6 +29,7 @@ import secrets
|
|||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
|
@ -36,7 +38,7 @@ import time
|
|||
from dataclasses import dataclass
|
||||
from decimal import Decimal, InvalidOperation
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Optional
|
||||
from typing import Any, Callable, Mapping, Optional
|
||||
|
||||
from scripts.agent_benchmark.manifest import Timeout
|
||||
|
||||
|
|
@ -148,6 +150,7 @@ DEFAULT_ENV_ALLOWLIST = (
|
|||
"USER", "LOGNAME", "SHELL", "PWD", "PYTHONPATH", "PYTHONHASHSEED",
|
||||
"NO_COLOR", "CI",
|
||||
)
|
||||
TLS_CA_ENV_KEYS = ("SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS")
|
||||
|
||||
ENV_KEY_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]{0,63}$")
|
||||
METRIC_KIND_RE = re.compile(r"^metric:[a-z0-9][a-z0-9_.+-]{0,63}$")
|
||||
|
|
@ -681,6 +684,18 @@ def env_pairs(mapping: dict[str, str]) -> tuple[tuple[str, str], ...]:
|
|||
return tuple(sorted((str(k), str(v)) for k, v in mapping.items()))
|
||||
|
||||
|
||||
def inherited_tls_ca_environment(
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> tuple[tuple[str, str], ...]:
|
||||
"""Freeze only the two standard public CA bundle settings for a child."""
|
||||
source = os.environ if environment is None else environment
|
||||
return tuple(
|
||||
(key, value)
|
||||
for key in TLS_CA_ENV_KEYS
|
||||
if isinstance((value := source.get(key)), str) and value
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frame transport
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -759,11 +774,47 @@ class _Supervisor:
|
|||
socket_path = self.control_dir / SOCKET_FILENAME
|
||||
self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
previous_umask = os.umask(0o177)
|
||||
original_dir_fd = -1
|
||||
control_dir_fd = -1
|
||||
try:
|
||||
self.sock.bind(str(socket_path))
|
||||
# Some shared/container filesystems reject bind(2) when the socket
|
||||
# pathname traverses the short attempt symlink, even though the
|
||||
# resolved directory supports Unix sockets. Resolve the directory
|
||||
# once with fchdir and bind the basename; the published locator
|
||||
# remains the authenticated short absolute alias.
|
||||
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
|
||||
original_dir_fd = os.open(".", directory_flags)
|
||||
control_dir_fd = os.open(self.control_dir, directory_flags)
|
||||
os.fchdir(control_dir_fd)
|
||||
self.sock.bind(SOCKET_FILENAME)
|
||||
try:
|
||||
os.chmod(SOCKET_FILENAME, 0o600)
|
||||
except OSError as exc:
|
||||
# A small class of shared filesystems permits Unix sockets but
|
||||
# rejects chmod on the socket inode. The containing directory
|
||||
# is still an exclusive 0700 security boundary.
|
||||
directory = os.stat(".")
|
||||
unsupported = {errno.EINVAL}
|
||||
if hasattr(errno, "ENOTSUP"):
|
||||
unsupported.add(errno.ENOTSUP)
|
||||
if (
|
||||
exc.errno not in unsupported
|
||||
or stat.S_IMODE(directory.st_mode) != 0o700
|
||||
or directory.st_uid != os.geteuid()
|
||||
):
|
||||
raise
|
||||
else:
|
||||
socket_mode = stat.S_IMODE(os.lstat(SOCKET_FILENAME).st_mode)
|
||||
if socket_mode & 0o077:
|
||||
raise PermissionError("control socket permissions are too broad")
|
||||
finally:
|
||||
if original_dir_fd >= 0:
|
||||
os.fchdir(original_dir_fd)
|
||||
if control_dir_fd >= 0:
|
||||
os.close(control_dir_fd)
|
||||
if original_dir_fd >= 0:
|
||||
os.close(original_dir_fd)
|
||||
os.umask(previous_umask)
|
||||
os.chmod(socket_path, 0o600)
|
||||
self.sock.listen(4)
|
||||
locator = {
|
||||
"supervisor_pid": os.getpid(),
|
||||
|
|
|
|||
|
|
@ -627,6 +627,26 @@ class LifecycleTest(unittest.TestCase):
|
|||
self.assertTrue(checked)
|
||||
self.assertTrue(result.success)
|
||||
|
||||
def test_control_socket_bind_supports_short_symlink_alias(self) -> None:
|
||||
with tempfile.TemporaryDirectory(
|
||||
dir=Path.cwd(), prefix=".lifecycle-symlink-target-"
|
||||
) as target_name, tempfile.TemporaryDirectory(
|
||||
dir=tempfile.gettempdir(), prefix="iop-life-alias-parent-"
|
||||
) as alias_parent_name:
|
||||
target = Path(target_name)
|
||||
alias = Path(alias_parent_name) / "attempt"
|
||||
alias.symlink_to(target.resolve(), target_is_directory=True)
|
||||
evidence = target / "evidence"
|
||||
evidence.mkdir()
|
||||
result = self._run(replace(
|
||||
self._spec("print('FINISH'); print('IDLE')"),
|
||||
evidence_dir=str(evidence),
|
||||
control_dir=str(alias / "control"),
|
||||
))
|
||||
self.assertTrue(result.success)
|
||||
self.assertTrue((target / "control" / "locator.json").is_file())
|
||||
self.assertFalse((target / "control" / "control.sock").exists())
|
||||
|
||||
def test_owned_descendant_ignoring_term_is_killed_and_reaped(self) -> None:
|
||||
descendant_pid_path = self.root / "descendant.pid"
|
||||
descendant_source = (
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ from urllib.request import Request, urlopen
|
|||
from scripts.agent_benchmark.agy_iop import (
|
||||
AGY_CALLER,
|
||||
AgyEventParser,
|
||||
AgyPreflightResult,
|
||||
AgyRuntimeInputs,
|
||||
AgyRuntimeObservation,
|
||||
build_agy_invocation,
|
||||
|
|
@ -158,6 +159,9 @@ def _base_url(value: str) -> str:
|
|||
def _models_url(base_url: str) -> str:
|
||||
parsed = urlsplit(base_url)
|
||||
path = parsed.path.rstrip("/")
|
||||
marker = "/gemini/"
|
||||
if marker in path:
|
||||
path = path.split(marker, 1)[0]
|
||||
if path.endswith("/v1"):
|
||||
path += "/models"
|
||||
else:
|
||||
|
|
@ -165,6 +169,18 @@ def _models_url(base_url: str) -> str:
|
|||
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
|
||||
|
||||
|
||||
def _agy_route_base(base_url: str, route_id: str) -> str:
|
||||
parsed = urlsplit(base_url)
|
||||
path = parsed.path.rstrip("/")
|
||||
marker = "/gemini/"
|
||||
if marker in path:
|
||||
path = path.split(marker, 1)[0]
|
||||
if path.endswith("/v1"):
|
||||
path = path[:-3]
|
||||
path = path.rstrip("/") + f"/gemini/{route_id}"
|
||||
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
|
||||
|
||||
|
||||
def _command(argv: tuple[str, ...]) -> str:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
|
|
@ -855,12 +871,15 @@ class _LiveAdapter:
|
|||
self._observer = observer
|
||||
self._binary_resolver = binary_resolver
|
||||
self._invokers = invokers
|
||||
self._agy_preflight: Any = None
|
||||
self._agy_preflights: dict[str, AgyPreflightResult] = {}
|
||||
self._admitted_bindings: dict[str, RequestedEffectiveBinding] = {}
|
||||
|
||||
def preflight(self, cell: MatrixCell) -> PreflightObservation:
|
||||
if cell.caller != self.caller:
|
||||
raise LiveIopError("protocol_incompatible")
|
||||
self._admitted_bindings.pop(cell.id, None)
|
||||
if self.caller == AGY_CALLER:
|
||||
self._agy_preflights.pop(cell.id, None)
|
||||
resolution = self._runtime_resolution
|
||||
runtime = resolution.runtime
|
||||
if runtime is None:
|
||||
|
|
@ -874,31 +893,38 @@ class _LiveAdapter:
|
|||
if not observed.caller_ready:
|
||||
result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible"))
|
||||
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
|
||||
binding, issues = _binding_from_config(cell, self.capability, runtime.config)
|
||||
if not issues and binding.effective_model not in observed.catalog_models:
|
||||
binding, issues = _requested(cell), _issues("model_missing")
|
||||
if issues:
|
||||
result = make_result(cell, self.capability, binding, issues)
|
||||
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
|
||||
agy_preflight: AgyPreflightResult | None = None
|
||||
if self.caller == AGY_CALLER:
|
||||
capability = inspect_agy_iop_capability(observed.agy_version, observed.agy_help)
|
||||
agy_endpoint = _agy_route_base(runtime.base_url, cell.iop.route_id)
|
||||
agy_observation = AgyRuntimeObservation(
|
||||
cell.id, cell.iop.route_kind, cell.iop.route_id,
|
||||
_agy_runtime_identity("endpoint", runtime.base_url),
|
||||
_agy_runtime_identity("endpoint", agy_endpoint),
|
||||
_agy_runtime_identity("credential", runtime.secret), runtime.config.identity,
|
||||
)
|
||||
try:
|
||||
self._agy_preflight = preflight_agy_iop(
|
||||
agy_preflight = preflight_agy_iop(
|
||||
cell, capability,
|
||||
AgyRuntimeInputs(self._binary_resolver("agy"), runtime.base_url, runtime.secret),
|
||||
AgyRuntimeInputs(self._binary_resolver("agy"), agy_endpoint, runtime.secret),
|
||||
agy_observation,
|
||||
)
|
||||
except Exception:
|
||||
result = make_result(cell, self.capability, _requested(cell), _issues("stream_incompatible"))
|
||||
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
|
||||
if self._agy_preflight.issues:
|
||||
result = make_result(cell, self.capability, _requested(cell), self._agy_preflight.issues)
|
||||
if agy_preflight.issues:
|
||||
result = make_result(cell, self.capability, _requested(cell), agy_preflight.issues)
|
||||
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
|
||||
binding, issues = _binding_from_config(cell, self.capability, runtime.config)
|
||||
if not issues and binding.effective_model not in observed.catalog_models:
|
||||
binding, issues = _requested(cell), _issues("model_missing")
|
||||
result = make_result(cell, self.capability, binding, issues)
|
||||
if result.status == "ready":
|
||||
self._admitted_bindings[cell.id] = result.binding
|
||||
if agy_preflight is not None:
|
||||
self._agy_preflights[cell.id] = agy_preflight
|
||||
return PreflightObservation(result, runtime.endpoint_identity, runtime.config.identity)
|
||||
|
||||
def invoke(
|
||||
|
|
@ -933,7 +959,8 @@ class _LiveAdapter:
|
|||
admitted,
|
||||
)
|
||||
if self.caller == AGY_CALLER:
|
||||
if self._agy_preflight is None:
|
||||
agy_preflight = self._agy_preflights.get(cell.id)
|
||||
if agy_preflight is None or agy_preflight.binding != _requested(cell):
|
||||
raise LiveIopError("stream_incompatible")
|
||||
spec = _bind_live_spec(
|
||||
cell,
|
||||
|
|
@ -941,12 +968,14 @@ class _LiveAdapter:
|
|||
attempt,
|
||||
control_dir,
|
||||
build_agy_invocation(
|
||||
cell, prepared, task_payload, timeout, self._agy_preflight
|
||||
cell, prepared, task_payload, timeout, agy_preflight
|
||||
),
|
||||
)
|
||||
parser = AgyEventParser(cell)
|
||||
result = self._invokers.agy(spec, parser, self._agy_preflight, lambda locator: on_started(locator, spec_digest(spec)))
|
||||
observed = parser.observed_result(self._agy_preflight.capability, result)
|
||||
parser = AgyEventParser(cell, admitted)
|
||||
result = self._invokers.agy(spec, parser, agy_preflight, lambda locator: on_started(locator, spec_digest(spec)))
|
||||
if not result.success:
|
||||
return _bound_observations(result, admitted)
|
||||
observed = parser.observed_result(agy_preflight.capability, result)
|
||||
if observed.status != "ready" or observed.binding != admitted:
|
||||
raise LiveIopError("stream_incompatible")
|
||||
return _bound_observations(result, admitted)
|
||||
|
|
@ -964,7 +993,10 @@ class _LiveAdapter:
|
|||
)
|
||||
result = self._invokers.codex(invocation, lambda locator: on_started(locator, spec_digest(invocation.spec)))
|
||||
expected = (admitted.effective_route_kind, admitted.effective_route_id, admitted.effective_model, admitted.effective_effort)
|
||||
if result.effective_binding != expected:
|
||||
if (
|
||||
result.effective_binding is not None
|
||||
and result.effective_binding != expected
|
||||
):
|
||||
raise LiveIopError("stream_incompatible")
|
||||
return _bound_observations(result.lifecycle, admitted)
|
||||
raise LiveIopError("protocol_incompatible")
|
||||
|
|
@ -1093,7 +1125,10 @@ class _LiveScoringAdapter:
|
|||
admitted.effective_model,
|
||||
admitted.effective_effort,
|
||||
)
|
||||
if result.effective_binding != expected:
|
||||
if (
|
||||
result.effective_binding is not None
|
||||
and result.effective_binding != expected
|
||||
):
|
||||
return ScoringInvocationResult(
|
||||
False, "binding_mismatch", result.effective_binding
|
||||
)
|
||||
|
|
@ -1101,7 +1136,7 @@ class _LiveScoringAdapter:
|
|||
return ScoringInvocationResult(
|
||||
lifecycle.success,
|
||||
lifecycle.terminal_reason,
|
||||
result.effective_binding,
|
||||
expected,
|
||||
)
|
||||
|
||||
def finalize_evidence(
|
||||
|
|
|
|||
|
|
@ -28,6 +28,8 @@ SESSION_POLICY = "fresh"
|
|||
SETUP_CACHE_POLICY = "isolated"
|
||||
DEFAULT_REPETITIONS = 1
|
||||
RUBRIC_VERSION = "landing-quality-v1"
|
||||
ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"
|
||||
RUBRIC_VERSIONS = (RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION)
|
||||
|
||||
CALLER_ENUM = ("claude", "agy", "codex")
|
||||
ROUTE_KIND_ENUM = ("direct", "execution_preset")
|
||||
|
|
@ -124,6 +126,7 @@ class Manifest:
|
|||
fixture: Fixture
|
||||
matrix: tuple[MatrixCell, ...]
|
||||
digest: str
|
||||
execution_order_seed: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -190,7 +193,7 @@ def _require_enum(value: Any, field_name: str, allowed: tuple[str, ...]) -> str:
|
|||
|
||||
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if not pattern.match(s):
|
||||
if pattern.fullmatch(s) is None:
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' does not match required pattern"
|
||||
)
|
||||
|
|
@ -310,7 +313,7 @@ def digest_manifest_and_resolved_inputs(manifest: Manifest) -> str:
|
|||
|
||||
def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]:
|
||||
"""Convert a Manifest to a plain dict for canonical JSON serialization."""
|
||||
return {
|
||||
canonical = {
|
||||
"pipeline_version": manifest.pipeline_version,
|
||||
"environment": manifest.environment,
|
||||
"testbed": manifest.testbed,
|
||||
|
|
@ -377,6 +380,11 @@ def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]:
|
|||
for c in manifest.matrix
|
||||
],
|
||||
}
|
||||
# Pipeline-v2 manifests without an explicit seed keep their legacy
|
||||
# canonical bytes and digest. An explicit seed is part of the contract.
|
||||
if manifest.execution_order_seed is not None:
|
||||
canonical["execution_order_seed"] = manifest.execution_order_seed
|
||||
return canonical
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -608,7 +616,19 @@ def _validate_evaluator(data: dict[str, Any]) -> Evaluator:
|
|||
return Evaluator(caller=caller, iop=_validate_iop_cell(obj["iop"]))
|
||||
|
||||
|
||||
def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]:
|
||||
def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]:
|
||||
rank = hashlib.sha256(
|
||||
b"iop-benchmark-order-v1\0"
|
||||
+ seed.encode("ascii")
|
||||
+ b"\0"
|
||||
+ cell.id.encode("ascii")
|
||||
).digest()
|
||||
return rank, cell.id
|
||||
|
||||
|
||||
def _validate_matrix(
|
||||
data: list[Any], execution_order_seed: str | None
|
||||
) -> tuple[MatrixCell, ...]:
|
||||
arr = _require_array(data, "matrix")
|
||||
if len(arr) < 1:
|
||||
raise ManifestValidationError("matrix must have at least 1 item")
|
||||
|
|
@ -622,8 +642,13 @@ def _validate_matrix(data: list[Any]) -> tuple[MatrixCell, ...]:
|
|||
seen_ids.add(cell.id)
|
||||
cells.append(cell)
|
||||
|
||||
# Sort cells by id for canonical ordering
|
||||
cells.sort(key=lambda c: c.id)
|
||||
# No seed preserves the pipeline-v2 cell-id order. An explicit seed is
|
||||
# stable across JSON input permutations and platforms.
|
||||
cells.sort(
|
||||
key=(lambda cell: cell.id)
|
||||
if execution_order_seed is None
|
||||
else (lambda cell: _execution_order_key(execution_order_seed, cell))
|
||||
)
|
||||
return tuple(cells)
|
||||
|
||||
|
||||
|
|
@ -684,7 +709,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
|
|||
"session_policy", "setup_cache_policy", "timeout", "viewports",
|
||||
"rubric_version", "evaluator", "output_root",
|
||||
}
|
||||
optional_top = {"repetitions"}
|
||||
optional_top = {"repetitions", "execution_order_seed"}
|
||||
declared_keys = set(raw.keys())
|
||||
required_present = expected_top.issubset(declared_keys)
|
||||
extra = declared_keys - expected_top - optional_top
|
||||
|
|
@ -701,6 +726,12 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
|
|||
if "repetitions" in raw:
|
||||
repetitions = _require_positive_int(raw["repetitions"], "repetitions")
|
||||
|
||||
execution_order_seed: str | None = None
|
||||
if "execution_order_seed" in raw:
|
||||
execution_order_seed = _require_pattern(
|
||||
raw["execution_order_seed"], "execution_order_seed", CELL_ID_RE
|
||||
)
|
||||
|
||||
session_policy = _require_enum(raw["session_policy"], "session_policy", (SESSION_POLICY,))
|
||||
setup_cache_policy = _require_enum(
|
||||
raw["setup_cache_policy"], "setup_cache_policy", (SETUP_CACHE_POLICY,)
|
||||
|
|
@ -708,13 +739,13 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
|
|||
timeout = _validate_timeout(raw["timeout"])
|
||||
viewports = _validate_viewports(raw["viewports"])
|
||||
rubric_version = _require_enum(
|
||||
raw["rubric_version"], "rubric_version", (RUBRIC_VERSION,)
|
||||
raw["rubric_version"], "rubric_version", RUBRIC_VERSIONS
|
||||
)
|
||||
evaluator = _validate_evaluator(raw["evaluator"])
|
||||
output_root = _validate_output_root(raw["output_root"], repo_root)
|
||||
|
||||
fixture = _validate_fixture(raw["fixture"], repo_root)
|
||||
matrix = _validate_matrix(raw["matrix"])
|
||||
matrix = _validate_matrix(raw["matrix"], execution_order_seed)
|
||||
|
||||
# Compute manifest digest during load
|
||||
dummy_manifest = Manifest(
|
||||
|
|
@ -732,6 +763,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
|
|||
fixture=fixture,
|
||||
matrix=matrix,
|
||||
digest="",
|
||||
execution_order_seed=execution_order_seed,
|
||||
)
|
||||
computed_digest = digest_manifest_and_resolved_inputs(dummy_manifest)
|
||||
|
||||
|
|
@ -750,6 +782,7 @@ def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Mani
|
|||
fixture=fixture,
|
||||
matrix=matrix,
|
||||
digest=computed_digest,
|
||||
execution_order_seed=execution_order_seed,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ import hashlib
|
|||
import inspect
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
|
|
@ -38,11 +39,15 @@ from scripts.agent_benchmark.manifest import (
|
|||
ManifestPathError,
|
||||
ManifestValidationError,
|
||||
MatrixCell,
|
||||
ONE_SHOT_RUBRIC_VERSION,
|
||||
RUBRIC_VERSION,
|
||||
RUBRIC_VERSIONS,
|
||||
Timeout,
|
||||
Viewport,
|
||||
digest_manifest_and_resolved_inputs,
|
||||
digest_workspace_inputs,
|
||||
load_manifest,
|
||||
_manifest_to_dict,
|
||||
validate_manifest_bytes,
|
||||
)
|
||||
|
||||
|
|
@ -159,6 +164,230 @@ def _load_tmp_manifest(path: Path) -> Manifest:
|
|||
return load_manifest(path, repo_root=_REPO_ROOT)
|
||||
|
||||
|
||||
class ManifestValidationTest(unittest.TestCase):
|
||||
"""Exact tracked-manifest regressions used by benchmark readiness packets."""
|
||||
|
||||
def test_iop_one_shot_manifest_locks_benchmark_readiness(self):
|
||||
"""The bench-02 manifest locks identical inputs and the complete C01-C09 matrix."""
|
||||
manifest_path = (
|
||||
_REPO_ROOT
|
||||
/ "scripts"
|
||||
/ "fixtures"
|
||||
/ "agent-comparison-benchmark-iop-one-shot.json"
|
||||
)
|
||||
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
|
||||
|
||||
self.assertEqual(
|
||||
{
|
||||
key: raw[key]
|
||||
for key in (
|
||||
"pipeline_version",
|
||||
"environment",
|
||||
"testbed",
|
||||
"execution_order_seed",
|
||||
"repetitions",
|
||||
"session_policy",
|
||||
"setup_cache_policy",
|
||||
"timeout",
|
||||
"viewports",
|
||||
"rubric_version",
|
||||
"evaluator",
|
||||
"output_root",
|
||||
)
|
||||
},
|
||||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-c01-c09-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5,
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812},
|
||||
],
|
||||
"rubric_version": ONE_SHOT_RUBRIC_VERSION,
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{
|
||||
"stage": "request",
|
||||
"model": "gpt-5.6-luna",
|
||||
"effort": "xhigh",
|
||||
}
|
||||
],
|
||||
},
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-02",
|
||||
},
|
||||
)
|
||||
|
||||
expected_fixture = {
|
||||
"version": "landing-v1",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{
|
||||
"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt",
|
||||
"workspace_path": "brief/reference.txt",
|
||||
},
|
||||
{
|
||||
"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg",
|
||||
"workspace_path": "assets/aurora-grid.svg",
|
||||
},
|
||||
{
|
||||
"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg",
|
||||
"workspace_path": "assets/orbit-rings.svg",
|
||||
},
|
||||
],
|
||||
"checksum": "sha256:7dc1be6ed4a9f2f873016b708b99d827b0249c74f2ac287e1fcf8deade8dcd98",
|
||||
}
|
||||
self.assertEqual(raw["fixture"], expected_fixture)
|
||||
self.assertEqual(
|
||||
_compute_fixture_checksum(_REPO_ROOT, raw["fixture"]),
|
||||
expected_fixture["checksum"],
|
||||
)
|
||||
self.assertEqual(
|
||||
[
|
||||
asset["workspace_path"]
|
||||
for asset in raw["fixture"]["assets"]
|
||||
if asset["source"].endswith(".svg")
|
||||
],
|
||||
["assets/aurora-grid.svg", "assets/orbit-rings.svg"],
|
||||
)
|
||||
self.assertEqual(
|
||||
manifest.viewports,
|
||||
(
|
||||
Viewport(id="desktop_1080", width=1920, height=1080),
|
||||
Viewport(id="mobile_375", width=375, height=812),
|
||||
),
|
||||
)
|
||||
self.assertEqual(
|
||||
manifest.timeout,
|
||||
Timeout(
|
||||
run_seconds=300,
|
||||
idle_seconds=30,
|
||||
quiet_seconds=10,
|
||||
cleanup_grace_seconds=5,
|
||||
),
|
||||
)
|
||||
|
||||
def direct(cell_id: str, caller: str, model: str, effort: str) -> dict:
|
||||
return {
|
||||
"id": cell_id,
|
||||
"caller": caller,
|
||||
"iop": {
|
||||
"request_model": model,
|
||||
"requested_effort": effort,
|
||||
"route_kind": "direct",
|
||||
"route_id": model,
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": model, "effort": effort}
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
def hybrid(
|
||||
cell_id: str,
|
||||
caller: str,
|
||||
route_id: str,
|
||||
request_effort: str,
|
||||
cloud_model: str,
|
||||
) -> dict:
|
||||
return {
|
||||
"id": cell_id,
|
||||
"caller": caller,
|
||||
"iop": {
|
||||
"request_model": route_id,
|
||||
"requested_effort": request_effort,
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": route_id,
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": cloud_model, "effort": "high"},
|
||||
{"stage": "plan", "model": cloud_model, "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": cloud_model, "effort": "high"},
|
||||
{"stage": "repair", "model": cloud_model, "effort": "high"},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
expected_cells = [
|
||||
direct("c01-claude-sonnet-direct", "claude", "claude-sonnet-5", "max"),
|
||||
direct("c02-claude-gemini-direct", "claude", "gemini-3.6-flash", "high"),
|
||||
direct("c03-agy-gemini-direct", "agy", "gemini-3.6-flash", "high"),
|
||||
direct("c04-claude-gpt-direct", "claude", "gpt-5.6-luna", "xhigh"),
|
||||
direct("c05-codex-gpt-direct", "codex", "gpt-5.6-luna", "xhigh"),
|
||||
hybrid(
|
||||
"c06-claude-gemini-hybrid",
|
||||
"claude",
|
||||
"gemini-hybrid",
|
||||
"high",
|
||||
"gemini-3.6-flash",
|
||||
),
|
||||
hybrid(
|
||||
"c07-agy-gemini-hybrid",
|
||||
"agy",
|
||||
"gemini-hybrid",
|
||||
"high",
|
||||
"gemini-3.6-flash",
|
||||
),
|
||||
hybrid(
|
||||
"c08-claude-gpt-hybrid",
|
||||
"claude",
|
||||
"gpt-hybrid",
|
||||
"xhigh",
|
||||
"gpt-5.6-terra",
|
||||
),
|
||||
hybrid(
|
||||
"c09-codex-gpt-hybrid",
|
||||
"codex",
|
||||
"gpt-hybrid",
|
||||
"xhigh",
|
||||
"gpt-5.6-terra",
|
||||
),
|
||||
]
|
||||
self.assertEqual(raw["matrix"], expected_cells)
|
||||
|
||||
expected_order = [
|
||||
"c02-claude-gemini-direct",
|
||||
"c05-codex-gpt-direct",
|
||||
"c03-agy-gemini-direct",
|
||||
"c06-claude-gemini-hybrid",
|
||||
"c08-claude-gpt-hybrid",
|
||||
"c09-codex-gpt-hybrid",
|
||||
"c01-claude-sonnet-direct",
|
||||
"c07-agy-gemini-hybrid",
|
||||
"c04-claude-gpt-direct",
|
||||
]
|
||||
expected_by_id = {cell["id"]: cell for cell in expected_cells}
|
||||
self.assertEqual(
|
||||
[cell.id for cell in manifest.matrix],
|
||||
expected_order,
|
||||
)
|
||||
self.assertEqual(
|
||||
_manifest_to_dict(manifest)["matrix"],
|
||||
[expected_by_id[cell_id] for cell_id in expected_order],
|
||||
)
|
||||
self.assertEqual(manifest.execution_order_seed, "bench-02-c01-c09-v1")
|
||||
self.assertEqual(manifest.rubric_version, ONE_SHOT_RUBRIC_VERSION)
|
||||
self.assertEqual(manifest.fixture.checksum, expected_fixture["checksum"])
|
||||
self.assertEqual(manifest.digest, digest_manifest_and_resolved_inputs(manifest))
|
||||
self.assertRegex(manifest.digest, r"^sha256:[0-9a-f]{64}$")
|
||||
|
||||
|
||||
class TestLoadManifestValid(unittest.TestCase):
|
||||
"""Valid manifest loading tests."""
|
||||
|
||||
|
|
@ -374,6 +603,90 @@ class TestMatrixExtension(unittest.TestCase):
|
|||
class TestDeterministicOrdering(unittest.TestCase):
|
||||
"""Deterministic cell and binding ordering tests."""
|
||||
|
||||
@staticmethod
|
||||
def _direct_cells(*cell_ids: str) -> list[dict]:
|
||||
return [
|
||||
{
|
||||
"id": cell_id,
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-4-20250514",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-direct",
|
||||
"expected_bindings": [
|
||||
{
|
||||
"stage": "request",
|
||||
"model": "claude-sonnet-4-20250514",
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
for cell_id in cell_ids
|
||||
]
|
||||
|
||||
def test_execution_order_seed_is_canonical_and_permutation_stable(self):
|
||||
"""An explicit seed produces one repeatable order for every input permutation."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
cells = self._direct_cells("cell-a", "cell-b", "cell-c")
|
||||
seeded = _make_minimal_manifest_dict(
|
||||
execution_order_seed="seed-a", matrix=cells
|
||||
)
|
||||
permuted = _make_minimal_manifest_dict(
|
||||
execution_order_seed="seed-a", matrix=list(reversed(cells))
|
||||
)
|
||||
different_seed = _make_minimal_manifest_dict(
|
||||
execution_order_seed="seed-b", matrix=cells
|
||||
)
|
||||
|
||||
m1 = _load_tmp_manifest(
|
||||
_write_tmp_manifest(tmp_dir, seeded, "seeded.json")
|
||||
)
|
||||
m2 = _load_tmp_manifest(
|
||||
_write_tmp_manifest(tmp_dir, permuted, "permuted.json")
|
||||
)
|
||||
m3 = _load_tmp_manifest(
|
||||
_write_tmp_manifest(tmp_dir, different_seed, "different.json")
|
||||
)
|
||||
|
||||
self.assertEqual(m1.execution_order_seed, "seed-a")
|
||||
self.assertEqual(
|
||||
_manifest_to_dict(m1)["execution_order_seed"],
|
||||
"seed-a",
|
||||
)
|
||||
self.assertEqual(
|
||||
[cell.id for cell in m1.matrix],
|
||||
["cell-c", "cell-a", "cell-b"],
|
||||
)
|
||||
self.assertEqual(m1.matrix, m2.matrix)
|
||||
self.assertEqual(m1.digest, m2.digest)
|
||||
self.assertEqual(
|
||||
[cell.id for cell in m3.matrix],
|
||||
["cell-a", "cell-c", "cell-b"],
|
||||
)
|
||||
self.assertNotEqual(m1.matrix, m3.matrix)
|
||||
self.assertNotEqual(m1.digest, m3.digest)
|
||||
|
||||
def test_omitted_execution_order_seed_preserves_legacy_order_and_digest_contract(self):
|
||||
"""Omitting the seed keeps id order and excludes a synthetic default from canonical JSON."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
data = _make_minimal_manifest_dict(
|
||||
matrix=self._direct_cells("cell-c", "cell-a", "cell-b")
|
||||
)
|
||||
manifest = _load_tmp_manifest(_write_tmp_manifest(tmp_dir, data))
|
||||
|
||||
self.assertIsNone(manifest.execution_order_seed)
|
||||
self.assertEqual(
|
||||
[cell.id for cell in manifest.matrix],
|
||||
["cell-a", "cell-b", "cell-c"],
|
||||
)
|
||||
self.assertNotIn("execution_order_seed", _manifest_to_dict(manifest))
|
||||
self.assertEqual(
|
||||
manifest.digest, digest_manifest_and_resolved_inputs(manifest)
|
||||
)
|
||||
|
||||
def test_cells_sorted_by_id(self):
|
||||
"""Cells are sorted by id regardless of input order."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
|
|
@ -951,7 +1264,7 @@ class TestEnumsAndBounds(unittest.TestCase):
|
|||
_load_tmp_manifest(path)
|
||||
|
||||
def test_invalid_rubric_version_rejected(self):
|
||||
"""Invalid rubric_version pattern is rejected."""
|
||||
"""A rubric version outside the closed catalog is rejected."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
d = _make_minimal_manifest_dict(rubric_version="Invalid Version!")
|
||||
|
|
@ -1313,6 +1626,44 @@ class TestSchemaLoaderParity(unittest.TestCase):
|
|||
m = load_manifest(example_path)
|
||||
self.assertEqual(m.pipeline_version, "2")
|
||||
self.assertEqual(m.testbed, "../iop-s2")
|
||||
self.assertEqual(m.rubric_version, RUBRIC_VERSION)
|
||||
|
||||
def test_schema_and_loader_accept_exact_rubric_version_catalog(self):
|
||||
"""The schema and loader accept both immutable rubric versions only."""
|
||||
schema_path = (
|
||||
_REPO_ROOT
|
||||
/ "scripts"
|
||||
/ "fixtures"
|
||||
/ "agent-comparison-benchmark-manifest.schema.json"
|
||||
)
|
||||
schema = json.loads(schema_path.read_text(encoding="utf-8"))
|
||||
rubric_schema = schema["properties"]["rubric_version"]
|
||||
self.assertEqual(tuple(rubric_schema["enum"]), RUBRIC_VERSIONS)
|
||||
self.assertEqual(
|
||||
RUBRIC_VERSIONS,
|
||||
(RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION),
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
for index, version in enumerate(RUBRIC_VERSIONS):
|
||||
with self.subTest(version=version):
|
||||
path = _write_tmp_manifest(
|
||||
tmp_dir,
|
||||
_make_minimal_manifest_dict(rubric_version=version),
|
||||
f"rubric-{index}.json",
|
||||
)
|
||||
self.assertEqual(_load_tmp_manifest(path).rubric_version, version)
|
||||
|
||||
unknown = "one-shot-agent-comparison-v2"
|
||||
self.assertNotIn(unknown, rubric_schema["enum"])
|
||||
path = _write_tmp_manifest(
|
||||
tmp_dir,
|
||||
_make_minimal_manifest_dict(rubric_version=unknown),
|
||||
"rubric-unknown.json",
|
||||
)
|
||||
with self.assertRaises(ManifestValidationError):
|
||||
_load_tmp_manifest(path)
|
||||
|
||||
def test_tracked_fixtures_separate_generic_contract_from_direct_preflight(self):
|
||||
fixtures = _REPO_ROOT / "scripts" / "fixtures"
|
||||
|
|
@ -1377,6 +1728,52 @@ class TestSchemaLoaderParity(unittest.TestCase):
|
|||
with self.assertRaises(ManifestValidationError):
|
||||
_load_tmp_manifest(p)
|
||||
|
||||
def test_schema_and_loader_reject_same_invalid_execution_order_seeds(self):
|
||||
"""The tracked schema and loader enforce the same bounded seed token."""
|
||||
schema_path = (
|
||||
_REPO_ROOT
|
||||
/ "scripts"
|
||||
/ "fixtures"
|
||||
/ "agent-comparison-benchmark-manifest.schema.json"
|
||||
)
|
||||
schema = json.loads(schema_path.read_text(encoding="utf-8"))
|
||||
seed_schema = schema["properties"]["execution_order_seed"]
|
||||
self.assertEqual(seed_schema["type"], "string")
|
||||
self.assertEqual(seed_schema["pattern"], "^[a-z0-9][a-z0-9_-]{0,63}$")
|
||||
|
||||
boundary_seed = "a" * 64
|
||||
self.assertIsNotNone(re.fullmatch(seed_schema["pattern"], boundary_seed))
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
data = _make_minimal_manifest_dict(
|
||||
execution_order_seed=boundary_seed
|
||||
)
|
||||
path = _write_tmp_manifest(tmp_dir, data, "boundary-seed.json")
|
||||
self.assertEqual(
|
||||
_load_tmp_manifest(path).execution_order_seed,
|
||||
boundary_seed,
|
||||
)
|
||||
|
||||
invalid_seeds: tuple[object, ...] = (
|
||||
"",
|
||||
"Uppercase",
|
||||
"contains.dot",
|
||||
"seed-a\n",
|
||||
"a" * 65,
|
||||
True,
|
||||
None,
|
||||
)
|
||||
pattern = re.compile(seed_schema["pattern"])
|
||||
for index, seed in enumerate(invalid_seeds):
|
||||
schema_accepts = isinstance(seed, str) and pattern.fullmatch(seed) is not None
|
||||
self.assertFalse(schema_accepts)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp_dir = Path(tmp)
|
||||
data = _make_minimal_manifest_dict(execution_order_seed=seed)
|
||||
path = _write_tmp_manifest(tmp_dir, data, f"invalid-seed-{index}.json")
|
||||
with self.assertRaises(ManifestValidationError):
|
||||
_load_tmp_manifest(path)
|
||||
|
||||
def test_preset_with_request_stage_rejected(self):
|
||||
"""Execution preset cell with extra request stage raises ManifestValidationError."""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""Strict D12 landing-page worksheet contract.
|
||||
"""Strict versioned benchmark worksheet contract.
|
||||
|
||||
Automatic web gates establish scoring eligibility; they are intentionally not
|
||||
represented in this 100-point worksheet and can never contribute points.
|
||||
|
|
@ -11,9 +11,13 @@ import os
|
|||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from types import MappingProxyType
|
||||
from typing import Any, Mapping
|
||||
|
||||
from scripts.agent_benchmark.manifest import RUBRIC_VERSION
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
ONE_SHOT_RUBRIC_VERSION,
|
||||
RUBRIC_VERSION,
|
||||
)
|
||||
|
||||
|
||||
MAX_WORKSHEET_BYTES = 64 * 1024
|
||||
|
|
@ -25,12 +29,39 @@ RUBRIC_CATEGORIES = (
|
|||
("typography_readability", 15),
|
||||
("polish_consistency", 15),
|
||||
)
|
||||
ONE_SHOT_RUBRIC_CATEGORIES = (
|
||||
("requirements_fidelity", 25),
|
||||
("visual_completeness", 25),
|
||||
("responsive_accessibility", 15),
|
||||
("image_detail_usage", 10),
|
||||
("behavior_stability", 10),
|
||||
("code_quality", 10),
|
||||
("self_verification", 5),
|
||||
)
|
||||
RUBRIC_CATEGORIES_BY_VERSION: Mapping[str, tuple[tuple[str, int], ...]] = (
|
||||
MappingProxyType(
|
||||
{
|
||||
RUBRIC_VERSION: RUBRIC_CATEGORIES,
|
||||
ONE_SHOT_RUBRIC_VERSION: ONE_SHOT_RUBRIC_CATEGORIES,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class RubricError(Exception):
|
||||
"""A worksheet is missing, malformed, non-canonical, or out of bounds."""
|
||||
|
||||
|
||||
def rubric_categories(version: str) -> tuple[tuple[str, int], ...]:
|
||||
"""Return the immutable ordered category table for a known rubric version."""
|
||||
if not isinstance(version, str):
|
||||
raise RubricError("rubric version is invalid")
|
||||
try:
|
||||
return RUBRIC_CATEGORIES_BY_VERSION[version]
|
||||
except KeyError as exc:
|
||||
raise RubricError("rubric version is invalid") from exc
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CategoryScore:
|
||||
id: str
|
||||
|
|
@ -75,22 +106,28 @@ def canonical_worksheet_bytes(worksheet: Worksheet) -> bytes:
|
|||
)
|
||||
|
||||
|
||||
def validate_worksheet(value: Any) -> Worksheet:
|
||||
def validate_worksheet(
|
||||
value: Any, expected_version: str | None = None
|
||||
) -> Worksheet:
|
||||
if not isinstance(value, dict) or set(value) != {
|
||||
"rubric_version", "categories", "total",
|
||||
}:
|
||||
raise RubricError("worksheet schema is invalid")
|
||||
if value["rubric_version"] != RUBRIC_VERSION:
|
||||
rubric_version = value["rubric_version"]
|
||||
expected_categories = rubric_categories(rubric_version)
|
||||
if expected_version is not None:
|
||||
rubric_categories(expected_version)
|
||||
if expected_version is not None and rubric_version != expected_version:
|
||||
raise RubricError("worksheet rubric version is invalid")
|
||||
raw_categories = value["categories"]
|
||||
if not isinstance(raw_categories, list) or len(raw_categories) != len(
|
||||
RUBRIC_CATEGORIES
|
||||
expected_categories
|
||||
):
|
||||
raise RubricError("worksheet categories are invalid")
|
||||
|
||||
categories: list[CategoryScore] = []
|
||||
for raw, (expected_id, expected_max) in zip(
|
||||
raw_categories, RUBRIC_CATEGORIES
|
||||
raw_categories, expected_categories
|
||||
):
|
||||
if not isinstance(raw, dict) or set(raw) != {
|
||||
"id", "max_score", "score", "evidence",
|
||||
|
|
@ -123,10 +160,12 @@ def validate_worksheet(value: Any) -> Worksheet:
|
|||
or not 0 <= total <= 100
|
||||
):
|
||||
raise RubricError("worksheet total is invalid")
|
||||
return Worksheet(RUBRIC_VERSION, tuple(categories), total)
|
||||
return Worksheet(rubric_version, tuple(categories), total)
|
||||
|
||||
|
||||
def load_worksheet(path: str | Path) -> Worksheet:
|
||||
def load_worksheet(
|
||||
path: str | Path, expected_version: str | None = None
|
||||
) -> Worksheet:
|
||||
target = Path(path)
|
||||
flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
|
|
@ -155,4 +194,4 @@ def load_worksheet(path: str | Path) -> Worksheet:
|
|||
value = json.loads(bytes(raw).decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise RubricError("worksheet JSON is invalid") from exc
|
||||
return validate_worksheet(value)
|
||||
return validate_worksheet(value, expected_version=expected_version)
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue