refactor(benchmark): 전용 하네스를 폐기한다
제품 호출 안정성과 측정 경계를 다시 분리하고, 벤치 대상 경로의 최소 HTML 스모크와 초경량 단일 시도 비교만 남긴다.
This commit is contained in:
parent
15bc8879cf
commit
f25eda05c3
71 changed files with 369 additions and 35852 deletions
9
Makefile
9
Makefile
|
|
@ -1,4 +1,4 @@
|
|||
.PHONY: all build build-local build-edge build-edge-host build-node build-node-target build-node-targets pack-node-target pack-edge archive-edge tidy test test-e2e test-control-plane-edge-wire test-credential-slot-smoke test-openai-ollama test-openai-lemonade test-openai-glm-coding test-hot-path-agent-smoke-self-test test-hot-path-agent-smoke-preflight test-hot-path-agent-smoke test-single-request-claude-smoke-self-test test-single-request-claude-smoke-preflight test-single-request-claude-smoke-validate test-single-request-claude-smoke readability-audit proto proto-dart client-test client-build-web clean test-agent-comparison-benchmark
|
||||
.PHONY: all build build-local build-edge build-edge-host build-node build-node-target build-node-targets pack-node-target pack-edge archive-edge tidy test test-e2e test-control-plane-edge-wire test-credential-slot-smoke test-openai-ollama test-openai-lemonade test-openai-glm-coding test-hot-path-agent-smoke-self-test test-hot-path-agent-smoke-preflight test-hot-path-agent-smoke test-single-request-claude-smoke-self-test test-single-request-claude-smoke-preflight test-single-request-claude-smoke-validate test-single-request-claude-smoke readability-audit proto proto-dart client-test client-build-web clean
|
||||
|
||||
GOFLAGS ?= -trimpath
|
||||
BUILD_DIR ?= build
|
||||
|
|
@ -78,13 +78,6 @@ test:
|
|||
readability-audit:
|
||||
python3 scripts/readability_audit.py --check --input-mode worktree --output build/readability-audit.json
|
||||
|
||||
# Deterministic, credential-free benchmark manifest tests.
|
||||
# Fresh unittest discovery for *_test.py plus the tracked example validation.
|
||||
test-agent-comparison-benchmark:
|
||||
cd $(CURDIR) && PYTHONPATH=$(CURDIR) python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py' -v
|
||||
python3 scripts/agent_comparison_benchmark.py validate \
|
||||
--manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json
|
||||
|
||||
test-e2e:
|
||||
@echo "NOTE: test-e2e runs auxiliary smoke (Edge-Node + OpenAI) plus Control Plane-Edge wire smoke; completion still requires user-flow verification when changing runtime paths."
|
||||
./scripts/e2e-smoke.sh
|
||||
|
|
|
|||
|
|
@ -37,6 +37,5 @@ Start with `agent-contract/index.md` for protocol and runtime contracts, and `ag
|
|||
Operator and client setup guides:
|
||||
|
||||
- [Edge Local Quickstart](docs/edge-local-dev-guide.md)
|
||||
- [Agent Comparison Benchmark Dev Guide](docs/agent-comparison-benchmark-dev-guide.md)
|
||||
- [dev OpenCode Settings Guide](docs/dev-opencode-settings-guide.md)
|
||||
- [dev-corp Pi Settings Guide](docs/dev-corp-pi-settings-guide.md)
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@
|
|||
|----|-----------|-----------|------|
|
||||
| `iop.openai-compatible-api` | OpenAI-compatible API, Responses API, Chat Completions, legacy Completions, error envelope/SSE terminal error, `model` route, managed projection principal auth and slot-route binding, managed-versus-legacy provider credential selection, model-driven passthrough/normalized routing, provider-pool admission/unavailable error, safe credential-slot attribution, standard metadata, and provider-native extension fields such as `chat_template_kwargs` | `apps/edge/internal/openai/*`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/service/provider_tunnel.go`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/openai-compatible-api.md` |
|
||||
| `iop.anthropic-compatible-api` | Anthropic Messages API, count_tokens, models list, bearer or `X-Api-Key` principal auth, active managed projection auth and slot-route binding, `anthropic-version` routing, native Anthropic tunnel, Chat bridge, provider-pool-only admission, profile capability checks, managed-versus-legacy provider credentials, marked-preset single-request admission with Edge-owned internal Plan/Review template customization that leaves caller I/O unchanged, and current no-OpenAI-metric status | `apps/edge/internal/openai/anthropic_handler.go`, `apps/edge/internal/openai/anthropic_native.go`, `apps/edge/internal/openai/anthropic_bridge.go`, `apps/edge/internal/openai/anthropic_stream.go`, `apps/edge/internal/openai/anthropic_types.go`, `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/authprojection/*`, `apps/edge/internal/openai/provider_tunnel.go`, `apps/edge/internal/openai/provider_model_rewrite.go`, `apps/edge/internal/openai/single_request_preset_binding.go`, `apps/edge/internal/openai/single_request_plan_stage.go`, `apps/edge/internal/openai/single_request_review_stage.go`, `packages/go/singlerequesttemplate/template.go`, `packages/go/config/protocol_profile.go` | `agent-contract/outer/anthropic-compatible-api.md` |
|
||||
| `iop.gemini-compatible-api` | Gemini Developer API `streamGenerateContent`, route-qualified Gemini-native ingress, `x-goog-api-key` principal auth, `GOOGLE_GEMINI_BASE_URL`, official agy 1.1.12 API-key transport, Gemini function calls/thought signatures/SSE, and direct-versus-execution-preset binding | `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`, `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/live_iop.py` | `agent-contract/outer/gemini-compatible-api.md` |
|
||||
| `iop.gemini-compatible-api` | Gemini Developer API `streamGenerateContent`, route-qualified Gemini-native ingress, `x-goog-api-key` principal auth, `GOOGLE_GEMINI_BASE_URL`, official agy 1.1.12 API-key transport, Gemini function calls/thought signatures/SSE, and direct-versus-execution-preset binding | `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`, `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go` | `agent-contract/outer/gemini-compatible-api.md` |
|
||||
| `iop.a2a-json-rpc-api` | A2A JSON-RPC API, `message/send`, `tasks/get`, `tasks/cancel`, A2A task state, agent card, `a2a.bearer_token`, Edge A2A input surface | `apps/edge/internal/input/a2a/*`, `packages/go/config/config.go`, `configs/edge.yaml` | `agent-contract/outer/a2a-json-rpc-api.md` |
|
||||
|
||||
## Inner Contracts
|
||||
|
|
|
|||
|
|
@ -9,14 +9,12 @@
|
|||
- `apps/edge/internal/openai/routes.go`
|
||||
- `apps/edge/internal/openai/principal.go`
|
||||
- `apps/edge/internal/openai/chat_handler.go`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/live_iop.py`
|
||||
- external caller surface: official Antigravity CLI `agy` 1.1.12 Gemini API-key provider
|
||||
|
||||
## 읽는 조건
|
||||
|
||||
- Gemini Developer API `streamGenerateContent`, `x-goog-api-key`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `agy` API-key provider, Gemini-native tool/function call, 또는 Gemini-native SSE ingress를 구현·검증할 때 읽는다.
|
||||
- IOP execution preset을 Gemini-native caller에 노출하거나 `agy` benchmark transport를 변경할 때 읽는다.
|
||||
- IOP execution preset을 Gemini-native caller에 노출하거나 공식 `agy` transport를 변경할 때 읽는다.
|
||||
|
||||
## 범위
|
||||
|
||||
|
|
@ -41,7 +39,7 @@ x-goog-api-key: <IOP principal token>
|
|||
- `GEMINI_API_KEY`: upstream provider key가 아니라 IOP principal token이다.
|
||||
- `GOOGLE_GEMINI_BASE_URL`: `https://<edge>/gemini/{route-id}`다.
|
||||
- 사설 dev CA를 사용하는 경우 caller child에는 표준 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`만 명시적으로 전달한다.
|
||||
- `--model`: 공식 CLI가 인식하는 Gemini 모델 label을 사용한다. benchmark의 Gemini 3.6 Flash 호출은 `Gemini 3.6 Flash`다.
|
||||
- `--model`: 공식 CLI가 인식하는 Gemini 모델 label을 사용한다. dev 호환 확인에서 사용하는 label은 `Gemini 3.6 Flash`다.
|
||||
- `--effort`는 API-key provider 호출에 전달하지 않는다. 요청된 high effort는 인증된 IOP route/preset의 effective binding으로 검증한다.
|
||||
|
||||
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 계약의 transport가 아니다.
|
||||
|
|
@ -75,7 +73,7 @@ Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message,
|
|||
- provider-reported usage가 있으면 `usageMetadata.promptTokenCount`, `candidatesTokenCount`, `thoughtsTokenCount`, `cachedContentTokenCount`, `totalTokenCount`의 존재하는 값만 투영한다. 누락 값을 0으로 발명하지 않는다.
|
||||
- caller disconnect는 기존 request cancellation 경계를 사용하며 이후 frame을 쓰지 않는다.
|
||||
|
||||
## 공식 agy stream-json lifecycle
|
||||
## 공식 agy 수동 호출 확인 기준
|
||||
|
||||
`agy` 1.1.12의 각 JSONL record는 `event` discriminator와 같은 이름의 중첩 payload를 사용한다.
|
||||
|
||||
|
|
@ -83,7 +81,7 @@ Edge는 이를 기존 Chat/preset ingress의 system/user/assistant/tool message,
|
|||
- step: `{"event":"step_update","step_update":{"state":...,"step_type":...,"usage":{...}}}`
|
||||
- terminal: `{"event":"result","result":{"status":"SUCCESS","duration_seconds":...,"num_turns":...,"usage":{...}}}`
|
||||
|
||||
benchmark adapter는 중첩 payload만 파싱하며 `result.status=SUCCESS` 한 건과 process exit/quiet를 success terminal로 인정한다. 구조가 유효한 `result.status=ERROR`는 stream incompatibility가 아니므로 success terminal을 만들지 않고 caller process의 non-zero exit를 lifecycle failure authority로 보존한다. `response`, `text_delta`, tool payload, conversation id는 durable evidence에 보존하지 않는다. usage는 caller가 제공한 `input_tokens`, `cache_read_tokens`, `output_tokens`, `thinking_tokens`, `total_tokens`만 원래 단위의 count로 기록하고 누락값을 합성하지 않는다.
|
||||
수동 확인에서는 중첩 `result.status=SUCCESS` 한 건과 process exit 0을 성공 terminal로 본다. 구조가 유효한 `result.status=ERROR` 또는 non-zero exit는 실패로 남기며 성공으로 재해석하지 않는다. 확인 기록에는 raw 응답, tool payload, conversation id, credential을 남기지 않고 caller가 제공한 usage만 원래 단위로 요약한다.
|
||||
|
||||
## 오류
|
||||
|
||||
|
|
@ -101,9 +99,8 @@ HTTP commit 전 오류는 다음 Gemini envelope 한 건으로 반환한다.
|
|||
- route/auth: `apps/edge/internal/openai/routes.go`, `apps/edge/internal/openai/principal.go`
|
||||
- Gemini request/SSE bridge: `apps/edge/internal/openai/gemini_handler.go`, `apps/edge/internal/openai/gemini_bridge.go`, `apps/edge/internal/openai/gemini_types.go`
|
||||
- Edge regression: `apps/edge/internal/openai/gemini_handler_test.go`, existing Chat/preset/auth tests
|
||||
- caller adapter: `scripts/agent_benchmark/agy_iop.py`, `scripts/agent_benchmark/agy_iop_test.py`, `scripts/agent_benchmark/live_iop.py`
|
||||
- live proof: official `agy --output-format stream-json` through the dev Edge route-specific base URL, with direct and execution-preset effective binding evidence
|
||||
|
||||
2026-08-12 dev 검증에서 normal/boundary/auth/tool/SSE 회귀 테스트와 공식 `agy` 1.1.12 direct·hybrid 실호출이 통과해 이 계약을 active로 전환했다.
|
||||
|
||||
2026-08-12 benchmark 실측 보정에서 official agy planner의 structured-output schema field를 Chat `response_format`으로 변환하고, 구조가 유효한 ERROR result를 parser failure와 분리했다.
|
||||
2026-08-12 실호출 보정에서 official agy planner의 structured-output schema field를 Chat `response_format`으로 변환했다.
|
||||
|
|
|
|||
|
|
@ -111,5 +111,3 @@
|
|||
- field 테스트 포트, artifact/bootstrap HTTP, 외부 테스트 환경: `agent-test/local/rules.md`를 따른다.
|
||||
- bootstrap/install UX, Agent Bootstrap, specialized agent 등록, Control Plane enrollment: `testing` domain rule과 `agent-test/local/rules.md`를 따른다.
|
||||
- 반복 작업이 확인되면 `agent-ops/skills/project/<skill-name>/SKILL.md`를 생성하고 이 표에 등록한다.
|
||||
- 벤치마크 매니페스트 검증/실행/재개/상태 확인/익명 채점/리포트: `agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md`
|
||||
- 벤치마크 validate, run, resume, status, score, report 요청, 매니페스트 검증 요청, 벤치마크 실행 요청, 벤치마크 상태 확인 요청, 벤치마크 익명 채점 요청, 벤치마크 리포트 요청
|
||||
|
|
|
|||
|
|
@ -1,211 +0,0 @@
|
|||
---
|
||||
name: iop-agent-comparison-benchmark
|
||||
description: Recognize benchmark validate/preflight/run/resume/status/score/report requests, delegate supported operations to the deterministic CLI, and fail closed at execution or scoring blockers.
|
||||
---
|
||||
|
||||
# iop-agent-comparison-benchmark
|
||||
|
||||
## Purpose
|
||||
|
||||
Route agent comparison benchmark requests to the deterministic CLI while enforcing append-only evidence, no-substitution, capability, and safety boundaries. The skill is routing and documentation, not a second implementation or orchestration dispatcher.
|
||||
|
||||
## When to use
|
||||
|
||||
- User requests benchmark manifest validation: `validate`, `validate manifest`, `manifest 검증`
|
||||
- User requests route readiness: `preflight`, `preflight benchmark`, `연결 사전 점검`
|
||||
- User requests benchmark execution: `run`, `run benchmark`, `벤치마크 실행`, `시작해`
|
||||
- User requests benchmark resume: `resume`, `resume benchmark`, `재개`, `계속해`
|
||||
- User requests benchmark status: `status`, `status benchmark`, `상태 확인`, `어디까지 왔어`
|
||||
- User requests blind scoring: `score`, `score benchmark`, `익명 채점`, `품질 채점`
|
||||
- User requests report or output: `report`, `report output`, `결과 보고`, `리포트`, `리포트 보여줘`
|
||||
|
||||
## Inputs
|
||||
|
||||
- `manifest`: Path to the benchmark manifest JSON file. (required for validate, preflight, run, resume, status, score, report)
|
||||
- `run_id`: Harness-generated run id. (required for resume, status, score, report)
|
||||
- `retry_failed`: Boolean flag for resume. (optional, default: false)
|
||||
- `retry_scoring_failed`: Boolean flag for score. (optional, default: false)
|
||||
|
||||
## Preflight
|
||||
|
||||
- [ ] Confirm the request matches one of the supported trigger cases above.
|
||||
- [ ] For validate/preflight/run/resume/status/score/report: confirm a manifest path is provided. If missing, return `error: manifest path is required`.
|
||||
- [ ] For resume/status/score/report: confirm a harness-issued run id is provided. If missing, return `error: run id is required`.
|
||||
- [ ] Confirm the CLI exists: `scripts/agent_comparison_benchmark.py` is present at the repo root.
|
||||
- [ ] Confirm the manifest file exists and is readable before delegating.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Classify the request**
|
||||
- Map the user request to one of: `validate`, `preflight`, `run`, `resume`, `status`, `score`, `report`.
|
||||
- If the request does not match any trigger, report that the benchmark pipeline skill does not cover the request and route to the appropriate skill.
|
||||
|
||||
2. **Delegate report to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py report --manifest <manifest-path> --run-id <run-id>`
|
||||
- The CLI opens immutable run state and delegates to the strict reporter with no caller adapter path.
|
||||
- On exit 0, report `ok: report run_id=<run-id> path=<run-relative-path>`.
|
||||
- On exit 69, report `error: benchmark report is unavailable` from stderr.
|
||||
|
||||
3. **Delegate validate to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py validate --manifest <manifest-path>`
|
||||
- Report the CLI exit code and stdout/stderr verbatim.
|
||||
- On exit 0, report `ok: manifest is valid`.
|
||||
- On exit 69, report the validation error from stderr.
|
||||
- On exit 64, report the usage error from stderr.
|
||||
|
||||
4. **Delegate preflight to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py preflight --manifest <manifest-path>`
|
||||
- The CLI records one fresh live observation for every immutable matrix cell, including direct and execution-preset routes, in canonical matrix order in one append-only run record.
|
||||
- On exit 0, report the exact closed `ready` summary from stdout.
|
||||
- On exit 69, report the exact `registration_required` or `implementation_gap` summary from stderr and stop. Never bypass the blocker, substitute a route/model/effort, or treat local manifest validation as live readiness.
|
||||
|
||||
5. **Delegate run to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py run --manifest <manifest-path>`
|
||||
- On missing or invalid manifest, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before creating execution state.
|
||||
- The CLI creates one run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
|
||||
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69, allocates no attempt, and preserves the run id for a later resume.
|
||||
- On `ready`, it binds the exact caller, cell, fresh workspace, session, and attempt identity, then must invoke each eligible cell exactly once with the fixture task.
|
||||
- Exit 0 when every manifest slot has complete terminal evidence (`unresolved=0`); independent failure counts remain in stdout and are classified by `score`. Exit 69 only for preflight blockers or incomplete evidence (absent/running slots). Never performs an implicit retry of a failed gate.
|
||||
|
||||
6. **Delegate resume to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py resume --manifest <manifest-path> --run-id <run-id> [--retry-failed]`
|
||||
- On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64) before changing the run.
|
||||
- The CLI opens the exact immutable run and uses its single writer to append a fresh all-cell preflight before attempt allocation.
|
||||
- On `registration_required` or `implementation_gap`, it prints `error: preflight blocked ...` to stderr with exit 69 and allocates no attempt.
|
||||
- On `ready`, it reconciles interrupted state, skips only slots whose latest product/harness/process/artifact gates all pass, preserves prior attempt bytes, and allocates a new attempt only for eligible work. `--retry-failed` admits a new attempt for any latest terminal attempt whose independent gates do not all pass.
|
||||
- Exit 0 when every manifest slot has complete terminal evidence (`unresolved=0`); independent failure counts remain in stdout and are classified by `score`. Exit 69 only for preflight blockers or incomplete evidence. Never performs an implicit retry of a failed gate.
|
||||
- It must invoke each eligible cell exactly once with a new workspace and session identity.
|
||||
|
||||
7. **Delegate status to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py status --manifest <manifest-path> --run-id <run-id>`
|
||||
- On missing or invalid manifest or state, the CLI prints `error: benchmark state is unavailable` to stderr with exit 69 (or `error: invalid usage` with exit 64).
|
||||
- On success, the CLI prints the controller counts, product/harness/process/artifact counts, and `unresolved=<count>` to stdout with exit 0.
|
||||
- Report the exact CLI output.
|
||||
|
||||
8. **Delegate score to the CLI**
|
||||
- Run: `python3 scripts/agent_comparison_benchmark.py score --manifest <manifest-path> --run-id <run-id> [--retry-scoring-failed]`
|
||||
- The CLI classifies each failed product, harness, process, or required artifact gate with its own immutable `unscored` reason, without invoking the evaluator or assigning zero.
|
||||
- Eligible attempts receive an opaque blind workspace, one manifest-bound fresh Codex evaluator session, and the exact immutable manifest-selected rubric from the closed supported catalog (`landing-quality-v1`, `one-shot-agent-comparison-v1`); no substitute rubric or reinterpretation is permitted.
|
||||
- A prior `scored` result is terminal. A prior `scoring_failed` result is retried only with `--retry-scoring-failed`, which allocates a new score id and preserves every prior byte.
|
||||
- On exit 0, report the exact closed `scored`, `unscored`, `scoring_failed`, and `blocked` counts from stdout.
|
||||
- On exit 69, report the exact closed counts or unavailable-state line from stderr. Never substitute evaluator route/model/effort, fabricate a worksheet, or turn failure into zero.
|
||||
|
||||
9. **Report the result**
|
||||
- Report the command executed, the exact CLI exit code, and the full stdout/stderr.
|
||||
- Do not summarize, paraphrase, or fabricate CLI output.
|
||||
|
||||
## Validation
|
||||
|
||||
- [ ] The CLI command was executed and the exit code matches the documented contract.
|
||||
- [ ] The reported stdout/stderr matches the CLI output exactly.
|
||||
- [ ] Preflight evidence is append-only, covers every immutable matrix cell in canonical order, and uses only closed status/count fields.
|
||||
- [ ] A preflight blocker created no scored attempt and was not bypassed.
|
||||
- [ ] An ineligible execution attempt became `unscored` without an evaluator invocation or a zero score.
|
||||
- [ ] Product, harness, process, and artifact outcomes remain separately visible in run/resume/status output and report rows.
|
||||
- [ ] Each eligible score id used one opaque blind workspace and one fresh evaluator session; retry preserved prior bytes and used a new id.
|
||||
- [ ] `scoring_failed` used no fallback, synthetic worksheet, or implicit retry.
|
||||
- [ ] No caller or provider was invoked outside the deterministic CLI.
|
||||
- [ ] No report or output was fabricated for report requests; the CLI produced the deterministic artifact.
|
||||
- [ ] No public `prepare` operation was exposed or referenced.
|
||||
- [ ] Run/resume exit 0 requires `unresolved=0` (every slot has complete terminal evidence); independent failure axes remain visible and are classified by `score`.
|
||||
- [ ] Retry is explicit only (`--retry-failed`); a failed terminal gate is never reinterpreted as success.
|
||||
- If validation fails, report the mismatch and stop without fallback.
|
||||
|
||||
## Output format
|
||||
|
||||
For validate/run/resume/status:
|
||||
|
||||
```
|
||||
command: <validate|run|resume|status>
|
||||
exit_code: <int>
|
||||
stdout: <verbatim CLI stdout or "(none)">
|
||||
stderr: <verbatim CLI stderr or "(none)">
|
||||
```
|
||||
|
||||
For preflight:
|
||||
|
||||
```
|
||||
command: preflight
|
||||
exit_code: <0|69>
|
||||
stdout: <verbatim closed summary or "(none)">
|
||||
stderr: <verbatim closed summary or "(none)">
|
||||
```
|
||||
|
||||
For score:
|
||||
|
||||
```
|
||||
command: score
|
||||
exit_code: <0|69>
|
||||
stdout: <verbatim closed score summary or "(none)">
|
||||
stderr: <verbatim closed score summary or "(none)">
|
||||
```
|
||||
|
||||
For report:
|
||||
|
||||
```
|
||||
command: report
|
||||
exit_code: <0|69>
|
||||
stdout: <verbatim CLI stdout or "(none)">
|
||||
stderr: <verbatim CLI stderr or "(none)">
|
||||
```
|
||||
|
||||
For run/resume ready completion:
|
||||
|
||||
```
|
||||
command: <run|resume>
|
||||
exit_code: 0
|
||||
stdout: ok: <run|resume> run_id=<run-id> executed=<count> unresolved=0 completed=<retained-count> timed_out=<retained-count> cancelled=<retained-count> interrupted=<retained-count> running=0 product_succeeded=<count> product_failed=<count> product_unknown=<count> harness_passed=<count> harness_failed=<count> process_exited=<count> process_signalled=<count> process_timed_out=<count> process_cancelled=<count> process_not_started=<count> artifact_passed=<count> artifact_failed=<count> artifact_blocked=<count> artifact_not_run=0
|
||||
stderr: (none)
|
||||
```
|
||||
|
||||
Note: `unresolved=0` means every manifest slot has complete terminal evidence (web validation present). Independent failure axes (`product_failed`, `artifact_failed`, etc.) remain visible in the summary and are classified by `score` as `unscored` without invoking the evaluator or assigning zero.
|
||||
|
||||
For run/resume blocker or execution failure:
|
||||
|
||||
```
|
||||
command: <run|resume>
|
||||
exit_code: 69
|
||||
stdout: (none)
|
||||
stderr: <verbatim closed preflight or execution failure summary>
|
||||
```
|
||||
|
||||
## Safety rules
|
||||
|
||||
- The skill delegates every supported stateful operation verbatim to `scripts/agent_comparison_benchmark.py`. It does not reproduce pipeline policy in prose.
|
||||
- Durable run/attempt state and preflight evidence are persisted only under the validated run root (`agent-test/runs/<output-id>/<run-id>/`).
|
||||
- Caller sessions, output workspaces, and caches are fresh and isolated for every cell, repetition, and attempt; session or cache state is never shared within a run or across runs.
|
||||
- Read-only testbed/fixture inputs (such as `../iop-s2`) are not copied back or mutated; no writes occur outside the validated run root.
|
||||
- Preflight never allocates a scored attempt. Every immutable matrix cell requires its own fresh live observation.
|
||||
- Run/resume append a fresh all-cell preflight under the run writer before any attempt allocation; a blocker allocates no attempt.
|
||||
- Ready execution binds one exact cell and immutable attempt identity to one fresh workspace/session and one task submission.
|
||||
- Product, harness, process, and artifact are independent gates. Controller state `completed` only means the invocation controller reached a terminal state.
|
||||
- Resolution (`unresolved=0`) requires every manifest slot to have complete terminal evidence (web validation present); it does not require every gate to pass. Failed gates remain visible as independent failure counts and are classified by `score` as `unscored`.
|
||||
- Release qualification runs the five-cell direct manifest as one unscored canary and requires fresh `ready=5`, exactly five fresh attempts, `unresolved=0`, `running=0`, `interrupted=0`, terminal controller/product/harness/process/web-validation evidence for every slot, and no exhausted browser/CDP infrastructure block before a fresh nine-cell preflight. Product failure, upstream HTTP rejection, generated-missing after caller failure, and timeout remain measured outcomes and do not trigger an implicit retry. Edge pre-ingress incompatibility or an exhausted browser/CDP infrastructure block stops qualification; this step does not allocate hybrid or scored execution.
|
||||
- Scoring copies only anonymous generated files, two local images, and screenshots into an opaque run-owned blind tree; the identity mapping remains outside that tree.
|
||||
- Scoring records `unscored`, `scored`, and `scoring_failed` append-only, and a retry always allocates a fresh score id/session.
|
||||
- The internal workspace API (`RunStore`, `Manifest`, etc.) is not a user command. Do not expose it.
|
||||
|
||||
## Stop conditions
|
||||
|
||||
- Stop immediately on a preflight `registration_required` or `implementation_gap` result. Do not substitute an alias, change an effort, or continue to attempt allocation.
|
||||
- Stop immediately on a run/resume preflight blocker without allocating an attempt or invoking another execution path.
|
||||
- Do not resume or retry a retained execution failure unless the user explicitly requests resume with `--retry-failed`. A release-qualification run with complete terminal evidence may continue to the fresh nine-cell preflight under the Safety rules without retrying or reinterpreting the failed result.
|
||||
- Stop after `scoring_failed` unless the user explicitly requests score with `--retry-scoring-failed`.
|
||||
- Stop immediately and report the unavailable-state line for report requests that cannot project the deterministic report.
|
||||
- Stop immediately if the manifest path is missing or the file is not readable.
|
||||
- Stop immediately if the run id is missing for resume/status.
|
||||
- Stop without fallback, fabricated evidence, ad-hoc provider calls, subagents, or orchestration dispatchers.
|
||||
|
||||
## Prohibitions
|
||||
|
||||
- Do not expose a public `prepare` operation.
|
||||
- Do not invoke a caller or provider outside the deterministic benchmark CLI.
|
||||
- Do not bypass a preflight blocker or substitute caller, route, model, effort, or preset.
|
||||
- Do not retry scoring implicitly, replace the manifest evaluator, fabricate a worksheet, or convert `unscored`/`scoring_failed` to zero.
|
||||
- Do not claim execution-preset fixture validation as live readiness.
|
||||
- Do not use subagents, orchestration dispatchers, or dispatch.py-equivalent tools for benchmark execution.
|
||||
- Do not fabricate benchmark results, reports, or output.
|
||||
- Do not reproduce pipeline policy, state machine, or allocation logic in prose.
|
||||
- Do not allow user override of the read-only `../iop-s2` testbed provenance.
|
||||
- Do not write output or ad-hoc state outside the validated run root (`agent-test/runs/<output-id>/<run-id>/`).
|
||||
- Do not share session or cache state within a run across cells, repetitions, or attempts, or across run invocations.
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
# Milestone: [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
|
||||
## 위치
|
||||
|
||||
- Roadmap: [ROADMAP.md](../../../../ROADMAP.md)
|
||||
- Phase: [PHASE.md](../../../../phase/knowledge-tool-optimization-extension/PHASE.md)
|
||||
- SDD: [SDD.md](../../../sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md)
|
||||
|
||||
## 목표
|
||||
|
||||
동일한 정적 웹 과제를 9개 IOP 경유 caller/model 조합으로 실행해 속도·usage·품질을 비교하는 전용 benchmark를 수행하려 했다.
|
||||
|
||||
## 상태
|
||||
|
||||
[폐기]
|
||||
|
||||
## 구현 잠금
|
||||
|
||||
- 상태: 잠금
|
||||
- SDD: 필요
|
||||
- SDD 문서: [폐기된 SDD](../../../sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md)
|
||||
- SDD 상태: 폐기
|
||||
- SDD 잠금: 잠금
|
||||
- SDD 사용자 리뷰: 없음
|
||||
- 결정 필요: 없음
|
||||
|
||||
## 범위
|
||||
|
||||
- 전용 manifest/runner, 격리·재개·복구, 자동 웹 검증, 익명 채점과 deterministic report를 사용한 C01-C09 비교였다.
|
||||
- 2026-08-13 사용자 결정으로 제품 안정성과 측정 경계를 분리하기 위해 전용 harness와 함께 현재 실행 범위에서 철회했다.
|
||||
|
||||
## 기능
|
||||
|
||||
### Epic: [benchmark-rollback] 전용 측정 경계 폐기
|
||||
|
||||
- [x] [harness-retirement] 전용 benchmark 구현, 공식 CLI/skill, manifest/schema, 자동 검증·채점·보고와 현재 living spec을 제거했다.
|
||||
- [x] [milestone-retirement] 제품 안정성 전수 재검증을 이 측정 Milestone의 대체 작업으로 만들지 않고, 구체적 결함만 해당 제품 소유 영역에서 처리하도록 분리했다.
|
||||
|
||||
## 완료 리뷰
|
||||
|
||||
- 상태: 폐기
|
||||
- 요청일: 2026-08-13
|
||||
- 완료 근거: 사용자 요청과 전용 harness 삭제 diff
|
||||
- 검토 항목: 없음
|
||||
- 리뷰 코멘트: 기존 2/9 scored run과 후속 복구 계획은 완료 근거가 아니며 재개하지 않는다.
|
||||
|
||||
## 범위 제외
|
||||
|
||||
- IOP 제품 ingress, route/preset, stream terminal과 공식 caller 호환 코드의 롤백
|
||||
- 구체적 제품 결함의 소유 영역별 수정과 회귀 검증
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 폐기 사유: 측정 시스템의 정합성과 복구가 제품 안정성보다 우선되는 목적 역전
|
||||
- 후속 제품 Milestone: 없음. 현재 IOP 호출부를 기준선으로 유지하고 구체적 회귀만 국소 처리한다.
|
||||
- 복구 가능성: 삭제된 구현과 이전 문서는 Git 이력에서 복구할 수 있다.
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
# SDD: IOP 원샷 Agent 모델 비교 벤치마크
|
||||
|
||||
## 위치
|
||||
|
||||
- Milestone: [폐기된 Milestone](../../../phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md)
|
||||
- Phase: [PHASE.md](../../../../phase/knowledge-tool-optimization-extension/PHASE.md)
|
||||
|
||||
## 상태
|
||||
|
||||
[폐기]
|
||||
|
||||
## SDD 잠금
|
||||
|
||||
- 상태: 잠금
|
||||
- 사용자 리뷰: 없음
|
||||
- 잠금 항목:
|
||||
- 없음. 설계 자체를 폐기했다.
|
||||
|
||||
## 문제 / 비목표
|
||||
|
||||
- 문제: 전용 측정 harness의 lifecycle, 복구, 증거와 채점 정합성이 IOP 제품과 caller/model 호출 안정성보다 우선되는 목적 역전이 발생했다.
|
||||
- 비목표:
|
||||
- 이 SDD를 제품 안정성 설계로 재사용한다.
|
||||
|
||||
## Source of Truth
|
||||
|
||||
| 영역 | 기준 | 메모 |
|
||||
|------|------|------|
|
||||
| Roadmap | [폐기된 Milestone](../../../phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md) | 폐기 결정 원장 |
|
||||
| User Decision | 2026-08-13 하네스 폐기와 경계 분리 지시 | 측정 설계 철회 근거 |
|
||||
|
||||
## State Machine
|
||||
|
||||
| 상태 | 진입 조건 | 다음 상태 | 근거 |
|
||||
|------|-----------|-----------|------|
|
||||
| `retired` | 전용 harness와 Milestone 폐기 | 종료 | 삭제 diff와 archive 상태 |
|
||||
|
||||
## Interface Contract
|
||||
|
||||
- 계약 원문: 없음
|
||||
- 입력:
|
||||
- 없음
|
||||
- 출력:
|
||||
- 없음
|
||||
- 금지:
|
||||
- 이 폐기된 SDD를 현재 실행 또는 완료 조건으로 사용한다.
|
||||
|
||||
## Acceptance Scenarios
|
||||
|
||||
| ID | Milestone Task | Given | When | Then |
|
||||
|----|----------------|-------|------|------|
|
||||
| S01 | `harness-retirement` | 전용 측정 구현과 공식 포인터 | 폐기 반영 | 활성 구현과 라우팅에서 제거된다. |
|
||||
| S02 | `milestone-retirement` | 제품과 측정이 섞인 현재 Milestone | 경계 분리 | 측정 Milestone은 폐기되고 전수 안정성 대체 Milestone을 만들지 않는다. |
|
||||
|
||||
## Evidence Map
|
||||
|
||||
| Scenario | Required Evidence | `agent-task` 연결 | 완료 Evidence 기대 |
|
||||
|----------|-------------------|------------------|---------------------------|
|
||||
| S01 | 삭제 diff와 dead-reference 검색 | 해당 없음 | 전용 harness 표면 없음 |
|
||||
| S02 | archive, current, Phase와 queue diff | 해당 없음 | 대체 전수 검증 gate 없음 |
|
||||
|
||||
## Cross-repo Dependencies
|
||||
|
||||
- 폐기된 `[bench-02]`를 가리키던 workspace lock은 제거하며 대체 안정성 dependency를 만들지 않는다.
|
||||
|
||||
## Drift Check
|
||||
|
||||
- [x] Milestone의 폐기 상태와 일치한다.
|
||||
- [x] 현재 제품 계약을 이 SDD에 복제하지 않았다.
|
||||
- [x] 사용자 결정이 반영됐다.
|
||||
- [x] 전수 재검증을 요구하는 대체 Milestone과 실행 차단을 만들지 않았다.
|
||||
|
||||
## 사용자 리뷰 이력
|
||||
|
||||
- 2026-08-13: 사용자가 전용 harness 폐기, benchmark Milestone 롤백, IOP 제품 안정성 우선과 측정 경계 분리를 확정했다.
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 표준선: 폐기된 측정 상태 머신과 완료 조건을 후속 제품 작업에서 재사용하지 않는다.
|
||||
- 후속 SDD: 없음. 구체적 결함은 해당 제품 소유 영역에서 국소 처리한다.
|
||||
|
|
@ -12,7 +12,7 @@ Ollama serving 경로와 운영 기반이 안정화된 뒤, execution preset,
|
|||
cloud-first route evidence가 충분히 쌓이면 동일한 mode decision contract를 쓰는 RAG 기반 local routing model을 shadow/canary로 검증해 운영 기본 경로로 점진 전환한다.
|
||||
caller-neutral 누적 요청 컨텍스트 최적화, repository 장기 기억 RAG, advisor와 Context Hook은 routing evidence RAG와 서로 다른 후속 기능으로 분리한다.
|
||||
이 Phase는 특정 Agent Shell에 종속되지 않고 OpenAI-compatible, A2A, IOP native protocol 중 맞는 표면에서 공통 최적화 책임을 제공하는 방향을 다룬다.
|
||||
단일 요청 Agent 실행의 정식 smoke 이후 비교 검증은 별도 benchmark lane에서 수행하며, 준비 pipeline은 병렬 구축하고 실제 scored 비교는 route-02 완료 뒤 실행한다.
|
||||
단일 요청 Agent 실행은 현재 완료된 제품 상태를 기준선으로 유지한다. 구체적 결함이 재현되면 해당 제품 소유 영역에서 국소 수정·검증하며, 측정 도구는 제품 완료 조건이나 실행 차단을 소유하지 않는다.
|
||||
|
||||
## Milestone 흐름
|
||||
|
||||
|
|
@ -57,9 +57,17 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실
|
|||
- 경로: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](../../archive/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md)
|
||||
- 요약: 모델·caller·prompt·반복 횟수를 manifest로 바꾸고 Claude Code, agy, Codex의 IOP 연결부터 finish/idle, 시간·token·웹 검증·익명 채점·Markdown 보고까지 같은 pipeline으로 재현한다.
|
||||
|
||||
- [진행중] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
- 경로: [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](milestones/iop-one-shot-agent-model-comparison.md)
|
||||
- 요약: route-02 정식 smoke와 benchmark pipeline 준비 뒤 dev `../iop-s2`에서 동일 정적 웹 fixture로 9개 IOP 경유 단독·하이브리드 caller 조합을 각각 한 번 실행해 속도·token·품질을 비교한다.
|
||||
- [폐기] [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
- 경로: [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md)
|
||||
- 요약: 전용 harness의 정합성과 복구가 제품 안정성보다 우선되는 목적 역전으로 2026-08-13 폐기했다. 기존 결과와 계획은 재개하지 않는다.
|
||||
|
||||
- [진행중] [bench-route-01] 벤치 경로 최소 HTML 스모크
|
||||
- 경로: [[bench-route-01] 벤치 경로 최소 HTML 스모크](milestones/benchmark-route-minimal-html-smoke.md)
|
||||
- 요약: 벤치에 사용할 9개 caller/model/route 조합을 고정된 최소 `index.html` 생성 요청으로 한 번씩 직접 호출하고, 실패한 경로만 귀속·국소 수정·재검증한다.
|
||||
|
||||
- [계획] [bench-lite-01] 초경량 Agent 모델 비교
|
||||
- 경로: [[bench-lite-01] 초경량 Agent 모델 비교](milestones/thin-agent-model-comparison-benchmark.md)
|
||||
- 요약: 최소 HTML 스모크를 통과한 동일 경로를 복구·재개·자동 채점 없는 단일 시도로 실행하고, 성공 여부·경과 시간·제공된 usage·산출물만 한 표에 기록한다.
|
||||
|
||||
- [스케치] [output-03] OpenAI-compatible Runtime Output Integrity Filter
|
||||
- 경로: [[output-03] OpenAI-compatible Runtime Output Integrity Filter](milestones/openai-compatible-runtime-output-integrity-filter.md)
|
||||
|
|
@ -108,6 +116,6 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실
|
|||
- plan-bearing one-shot mode는 IOP Node가 승인된 workspace root 아래 `.iop/job/<request_id>/plan.md`와 `review.md`를 직접 생성·읽기·갱신·정리한다. 내부 model tool call/result는 IOP coordinator가 소비하며 Claude에 후속 tool result 요청을 요구하지 않는다.
|
||||
- 각 stage의 routing, plan, work, review, defect와 repair는 outer stream에 redacted 진행 요약으로만 투영한다. 내부 provider reasoning, control prompt, tool protocol·argument/result, credential과 stage terminal은 공개하지 않고 최종 사용자 결과와 outer terminal만 완결된 응답으로 반환한다.
|
||||
- target agent나 외부 workflow 제품의 process/state를 실행 의존성으로 연결하지 않는다. 범용 interactive shell과 장기 workflow는 제외하지만, execution preset의 request-scoped workspace/tool executor는 IOP가 소유한다.
|
||||
- benchmark skill/pipeline은 제품 coordinator가 아니라 dev 검증 harness다. 준비 작업은 route-02와 병렬일 수 있지만 실제 scored 비교는 route-02 정식 기능·필수 smoke와 benchmark pipeline 완료 뒤 별도 Milestone에서 수행한다.
|
||||
- 공식 caller 경로는 현재 제품 기준선을 유지한다. 벤치 준비에서는 벤치 대상 경로만 최소 HTML로 얕게 확인하고, 구체적 회귀가 발생한 경로만 해당 제품 소유 영역에서 검증한다. 전수 재검증 campaign이나 측정 도구를 제품 완료 gate로 두지 않는다.
|
||||
- cloud model은 초기 semantic judge/teacher 역할을 하고, 충분한 정제 evidence가 쌓인 뒤 RAG local router로 운영 기본을 전환한다. 두 경우 모두 최종 권한은 deterministic hard gate를 적용하는 Edge arbiter에 남는다.
|
||||
- routing evidence RAG는 route 판정 전용이고, repository 장기 기억 RAG·누적 요청 context·advisor·Context Hook과 corpus/index/평가를 공유하지 않는다.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,77 @@
|
|||
# Milestone: [bench-route-01] 벤치 경로 최소 HTML 스모크
|
||||
|
||||
## 위치
|
||||
|
||||
- Roadmap: [ROADMAP.md](../../../ROADMAP.md)
|
||||
- Phase: [PHASE.md](../PHASE.md)
|
||||
|
||||
## 목표
|
||||
|
||||
벤치마크에 사용할 caller/model/route 조합만 고정된 최소 HTML 생성 요청으로 빠르게 확인한다.
|
||||
IOP 전체 안정성을 처음부터 재검증하지 않고, 실패가 재현된 경로만 제품·caller·provider·환경 중 한 경계에 귀속해 국소 수정한 뒤 그 경로만 다시 확인한다.
|
||||
|
||||
## 상태
|
||||
|
||||
[진행중]
|
||||
|
||||
## 구현 잠금
|
||||
|
||||
- 상태: 해제
|
||||
- SDD: 불필요
|
||||
- SDD 문서: 없음
|
||||
- SDD 사유: 기존 공개 호출 경로를 임시 workspace에서 수동 확인하는 test-only 작업이며 API, wire, config, schema, lifecycle 또는 retry 계약을 새로 만들지 않는다.
|
||||
- 결정 필요: 없음
|
||||
|
||||
## 범위
|
||||
|
||||
- 아래 9개 기존 벤치 대상 조합을 순서와 무관하게 한 경로씩 직접 실행한다.
|
||||
- Claude Code → Claude direct
|
||||
- Claude Code → Gemini direct
|
||||
- agy → Gemini direct
|
||||
- Claude Code → GPT direct
|
||||
- Codex → GPT direct
|
||||
- Claude Code → Gemini execution preset
|
||||
- agy → Gemini execution preset
|
||||
- Claude Code → GPT execution preset
|
||||
- Codex → GPT execution preset
|
||||
- 모든 경로에 같은 요청을 사용한다: 외부 asset과 JavaScript 없이 exact marker가 있는 단일 `index.html`을 생성하고 종료한다.
|
||||
- 각 경로는 최초 1회만 실행하고 120초 안에 terminal과 `index.html` 생성 여부를 확인한다.
|
||||
- 실패하면 자동 retry나 전체 재실행을 하지 않는다. sanitized terminal/error와 IOP request stage만 확인해 소유 경계를 정하고, 변경된 원인이 있을 때 실패 경로만 1회 재검증한다.
|
||||
- 제품 코드를 수정한 결함에는 해당 소유 package의 focused regression test를 추가하고 관련 Go test를 실행한다.
|
||||
|
||||
## 기능
|
||||
|
||||
### Epic: [route-smoke] 벤치 경로 초경량 확인
|
||||
|
||||
- [ ] [minimal-html-calls] 9개 조합에 동일한 최소 HTML 요청을 한 번씩 직접 실행하고, 경로별 caller/model/route, terminal, 경과 시간, `index.html` marker 확인 결과를 한 개의 Markdown 표에 기록한다. 검증: 새 runner/manifest 없이 각 행에 실제 호출 결과가 하나만 있어야 한다.
|
||||
- [ ] [failed-path-fixes] 실패한 조합마다 제품·caller·provider·환경 중 소유 경계를 기록하고, IOP 제품 결함이 재현된 경우에만 국소 수정과 focused regression을 수행한 뒤 해당 조합만 다시 호출한다. 검증: 성공한 조합의 반복 실행이 없고, 재실행 행에는 변경된 원인과 연결된 수정·테스트 근거가 있어야 한다.
|
||||
- [ ] [thin-bench-handoff] 9개 조합의 통과 또는 구체적 외부 차단 상태를 짧게 정리해 `[bench-lite-01]` 실행 가능 여부를 남긴다. 검증: 비교 점수나 순위가 아니라 호출 가능 여부와 남은 소유자만 기록한다.
|
||||
|
||||
## 완료 리뷰
|
||||
|
||||
- 상태: 없음
|
||||
- 요청일: 없음
|
||||
- 완료 근거: 최소 HTML 호출 evidence가 아직 없다.
|
||||
- 검토 항목:
|
||||
- [ ] 새 benchmark script, runner, manifest, state store가 생성되지 않았다.
|
||||
- [ ] 성공 경로는 한 번만 실행했고 실패 경로만 변경된 원인 뒤 재검증했다.
|
||||
- [ ] 제품 수정은 재현된 결함과 focused regression으로 한정됐다.
|
||||
- agent-ui 상태 반영: 해당 없음
|
||||
- 리뷰 코멘트: 없음
|
||||
|
||||
## 범위 제외
|
||||
|
||||
- IOP ingress/provider/preset 전체 전수 안정화
|
||||
- benchmark runner, manifest/schema, resume/recovery store, 자동 retry
|
||||
- browser/CDP, screenshot, accessibility/network gate, 자동 품질 채점
|
||||
- 반복 측정, 통계, 순위, token 정규화
|
||||
- benchmark 성공을 다른 제품 Milestone이나 프로젝트의 완료 gate로 사용하는 것
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 계획 범위: 짧은 단일 test plan으로 즉시 실행할 수 있게 유지한다. 별도 설계·SDD·다단계 복구 계획으로 확장하지 않는다.
|
||||
- 실행 방식: 기존 공식 caller 명령을 한 번씩 직접 실행한다. 공통화가 필요해 보여도 이 Milestone에서는 script로 승격하지 않는다.
|
||||
- evidence 위치: `agent-test/dev/iop-benchmark-route-minimal-html-smoke.md`
|
||||
- 현재 사전 확인: 2026-08-13에 Claude Code 2.1.228, agy 1.1.12, Codex 0.147.0 실행 파일과 CA/token 파일의 존재·mode를 확인했으나, 기존 `token/.iop-bench`로 public `/v1/models`가 HTTP 401을 반환했다. 현재 public Edge 활성 config에는 이 token hash 매핑이 0건이며 caller/model 호출은 시작하지 않았다.
|
||||
- 재개 조건: 기존 5개 route를 볼 수 있는 유효한 dev-corp principal token을 operator-private 경로에 준비하거나 기존 test token을 active Edge에 안전하게 매핑·재시작하고, token 원문을 출력하지 않은 `/v1/models` 확인이 200이어야 한다.
|
||||
- 후속 측정: [초경량 Agent 모델 비교](thin-agent-model-comparison-benchmark.md)
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
# Milestone: [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
|
||||
## 위치
|
||||
|
||||
- Roadmap: [ROADMAP.md](../../../ROADMAP.md)
|
||||
- Phase: [PHASE.md](../PHASE.md)
|
||||
- SDD: [SDD.md](../../../sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md)
|
||||
|
||||
## 목표
|
||||
|
||||
`[route-02]`의 정식 기능과 필수 smoke가 완료된 뒤, 동일한 정적 웹페이지 과제를 IOP를 경유하는 3개 단독 모델과 Gemini/GPT 하이브리드 구조의 9개 caller 조합으로 각각 한 번 실행한다.
|
||||
첫 output·model/tool·전체시간, 호출 횟수와 세부 token, 자동 웹 검증과 익명 100점 품질 평가를 함께 비교하고 재현 가능한 Markdown 보고서를 현재 프로젝트에 남긴다.
|
||||
|
||||
## 상태
|
||||
|
||||
[진행중]
|
||||
|
||||
## 구현 잠금
|
||||
|
||||
- 상태: 해제
|
||||
- SDD: 필요
|
||||
- SDD 문서: [IOP 원샷 Agent 모델 비교 벤치마크 SDD](../../../sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md)
|
||||
- SDD 사유: 실제 dev provider/credential과 외부 CLI를 사용하는 field benchmark이며 실행 순서, 비용, 실패·재실행, secret-safe evidence와 비교 공정성을 고정해야 한다.
|
||||
- SDD 상태: 승인됨
|
||||
- SDD 잠금: 해제
|
||||
- SDD 사용자 리뷰: 없음
|
||||
- 잠금 해제 조건: 아래 체크리스트
|
||||
- [x] SDD 잠금이 해제되어 있다.
|
||||
- [x] SDD 사용자 리뷰가 없거나 승인/해결되었다.
|
||||
- [x] Acceptance Scenario가 Milestone 기능 Task와 연결되어 있다.
|
||||
- [x] Evidence Map이 완료 시 `complete.log`의 `milestone-task` id별 집계와 최종 검증 evidence로 검증 가능하게 연결되어 있다.
|
||||
- 결정 필요: 없음
|
||||
|
||||
## 범위
|
||||
|
||||
- 선행 조건은 `[bench-01]` benchmark pipeline 준비 완료와 `[route-02]` 정식 기능·필수 Claude smoke 완료다.
|
||||
- 모든 scored 실행은 dev 환경의 `../iop-s2` IOP runtime을 경유하고, 동일 checksum의 이미지 2장과 vanilla HTML/CSS/JS 단일 페이지 prompt를 run별 clean workspace와 fresh caller session에 제공한다.
|
||||
- 원샷은 사용자 작업 제출 1회 뒤 사람의 중간 feedback·수동 수정·재시작 없이 caller가 finish/complete event 후 idle이 될 때까지를 뜻하며 model/tool 호출 횟수는 제한하지 않고 측정한다.
|
||||
- 초기 benchmark는 아래 9개 비교군을 각각 1회 실행한다.
|
||||
|
||||
| ID | 유형 | Caller | IOP 실행 구성 |
|
||||
|----|------|--------|---------------|
|
||||
| C01 | Claude 단독 | Claude Code | Claude Sonnet 5 최고 effort |
|
||||
| C02 | Gemini 단독 | Claude Code | Gemini 3.6 Flash high |
|
||||
| C03 | Gemini 단독 | agy | Gemini 3.6 Flash high |
|
||||
| C04 | GPT 단독 | Claude Code | GPT-5.6 luna xhigh |
|
||||
| C05 | GPT 단독 | Codex | GPT-5.6 luna xhigh |
|
||||
| C06 | Gemini 하이브리드 | Claude Code | Gemini plan → ornith-fast work → Gemini review/repair |
|
||||
| C07 | Gemini 하이브리드 | agy | Gemini plan → ornith-fast work → Gemini review/repair |
|
||||
| C08 | GPT 하이브리드 | Claude Code | GPT plan → ornith-fast work → GPT review/repair |
|
||||
| C09 | GPT 하이브리드 | Codex | GPT plan → ornith-fast work → GPT review/repair |
|
||||
|
||||
- provider가 제공하는 input/output/reasoning/cached/total token을 model·stage별로 기록하고, 제공되지 않는 값은 추정 원본과 섞지 않고 `미제공`으로 표시한다.
|
||||
- 결과물 identity를 가린 뒤 Codex가 동일 rubric으로 품질을 채점하고 자동 검증 결과와 분리해 보고한다.
|
||||
|
||||
## 기능
|
||||
|
||||
### Epic: [benchmark-readiness] 비교 입력과 실행 준비 고정
|
||||
|
||||
실행 전에 공정한 fixture와 실제 IOP route/credential 상태를 고정한다.
|
||||
|
||||
- [x] [fixture-lock] 이미지 2장, 동일 one-page 요구사항, vanilla HTML/CSS/JS 초기 workspace, viewport와 자동 검증·100점 rubric을 checksum/version과 함께 고정한다.
|
||||
- [x] [agy-iop-compatibility] 공식 `agy 1.1.12`의 Gemini API-key provider가 route별 `GOOGLE_GEMINI_BASE_URL`과 `GEMINI_API_KEY=<IOP principal token>`으로 Edge의 Gemini-native `streamGenerateContent` ingress를 호출하고, Gemini request/tool/SSE를 기존 direct·execution-preset 실행에 연결하며 실제 `stream-json` lifecycle/usage를 benchmark adapter가 수집하도록 구현한다. `--effort`와 비공식 custom model에 의존하지 않고 high effort는 IOP effective binding으로 검증한다.
|
||||
- [x] [managed-credential-dev] dev Control Plane·Edge·Node에 CA-signed mTLS, Edge HTTPS, at-rest/issuer/recipient key material, principal projection과 Gemini/Claude/GPT slot-route를 operator-owned secret 경로로 구성한다. legacy static credential source를 제거하고 marked hybrid preset이 같은 principal의 고정 stage route를 managed lease로 실행하는지 secret-safe smoke로 확인한다.
|
||||
- [x] [route-readiness] dev `../iop-s2`에서 Claude Code·Codex의 호환 ingress와 공식 agy의 Gemini API-key transport가 각각 IOP 인증 경계를 통과하고, Sonnet/Gemini/GPT route, Gemini/GPT hybrid preset, effort와 stream/finish/idle이 모두 실제 caller preflight를 통과했는지 확인한다. 존재하지 않는 caller 환경 변수나 합성 event fixture는 live 호환 근거로 인정하지 않는다.
|
||||
- [x] [matrix-lock] C01-C09의 caller, IOP route/preset, model/effort, 반복 횟수 1, 실행 순서 seed, fresh-session과 setup/cache 정책 및 timeout을 immutable run manifest로 확정한다.
|
||||
|
||||
### Epic: [comparison-runs] 9개 원샷 실행
|
||||
|
||||
각 비교군을 clean workspace에서 한 번 실행하고 실패를 포함한 attempt evidence를 보존한다.
|
||||
|
||||
- [ ] [claude-standalone] C01 Claude Code→IOP→Claude Sonnet 5 최고 effort 단독 원샷을 실행한다.
|
||||
- [ ] [gemini-standalone] C02 Claude Code와 C03 agy가 각각 IOP→Gemini 3.6 Flash high 단독 원샷을 실행한다.
|
||||
- [ ] [gpt-standalone] C04 Claude Code와 C05 Codex가 각각 IOP→GPT-5.6 luna xhigh 단독 원샷을 실행한다.
|
||||
- [ ] [gemini-hybrid] C06 Claude Code와 C07 agy가 각각 IOP의 Gemini plan→ornith-fast work→Gemini review/repair 원샷을 실행한다.
|
||||
- [ ] [gpt-hybrid] C08 Claude Code와 C09 Codex가 각각 IOP의 GPT plan→ornith-fast work→GPT review/repair 원샷을 실행한다.
|
||||
|
||||
### Epic: [comparison-report] 검증·채점·보고서
|
||||
|
||||
정량 evidence와 익명 품질 평가를 결합하되 원본 수치와 해석을 분리한다.
|
||||
|
||||
- [ ] [objective-validation] 복구 후 새 C01-C09 full run의 각 결과에 build/serve, desktop·mobile screenshot, 이미지·asset, console 오류, 요구사항·반응형·접근성 gate와 최종 workspace 상태를 자동 검증한다. 생성물 존재·정적 안전·asset·render hard gate는 모두 통과해야 하며 품질 gate 실패는 익명 채점의 감점 evidence로 전달한다.
|
||||
- [ ] [quality-scoring] 익명화된 9개 결과 모두에 요구사항 25, 시각 완성도 25, 반응형·접근성 15, 이미지·디테일 10, 안정성 10, 코드 품질 10, 자체 검증 5의 동일 100점 rubric으로 Codex가 점수를 기록한다. 완료 집계는 `scored=9`, `unscored=0`, `scoring_failed=0`, `blocked=0`이어야 한다.
|
||||
- [ ] [performance-usage] 첫 output·첫 file write·model 호출별·tool·queue·전체 finish/idle 시간, 호출 횟수와 model/stage별 input/output/reasoning/cached/total token을 clock/source·미제공 여부와 함께 비교하고 중첩 구간이나 미관측 overhead를 임의 산술 분해하지 않는다.
|
||||
- [ ] [benchmark-report] 9개 결과의 속도·품질·token 표, 실행 조건·버전·실패·한계·raw evidence 링크를 포함한 날짜별 Markdown 보고서를 `agent-test/dev/`에 남긴다.
|
||||
|
||||
## 완료 리뷰
|
||||
|
||||
- 상태: 없음
|
||||
- 요청일: 없음
|
||||
- 완료 근거: 사용자 확정 9개 비교군과 post-smoke 실행·평가 기준을 SDD와 기능 Task로 정리했으며 실제 비교 evidence는 아직 없다.
|
||||
- 검토 항목: 없음
|
||||
- 리뷰 코멘트: 없음
|
||||
|
||||
## 범위 제외
|
||||
|
||||
- `[route-02]` 정식 기능이나 필수 smoke의 완료 여부를 이 비교 점수로 대체하거나 소급 변경하는 작업
|
||||
- 첫 보고서에서 비교군별 2회 이상 반복하는 실행
|
||||
- React/Vite 등 dependency 설치와 cache가 속도에 섞이는 frontend framework 과제
|
||||
- provider가 보고하지 않은 reasoning token을 exact 값처럼 추정하거나 서로 다른 tokenizer 수치를 무보정 단일 합계로 단정하는 방식
|
||||
- 실패 attempt를 삭제하고 성공 재실행만 대표값으로 선택하는 방식
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2`
|
||||
- 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다.
|
||||
- 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다.
|
||||
- 현재 차단: 보존 run `run-20260813T081326Z-4e1ac5152c6c`은 2개 scored/7개 unscored이므로 완료 근거가 아니다. Claude/agy caller·IOP route와 scoring eligibility를 복구하고 짧은 route별 smoke를 통과한 뒤, 기존 run과 다른 identity의 새 C01-C09 full run을 실행해 9개 모두 success/hard-gate-pass/scored가 되어야 한다. 이전 run과 attempt는 resume/retry/수정하지 않는다.
|
||||
- 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md)
|
||||
- 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md)
|
||||
- 확인 필요: 없음
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
# Milestone: [bench-lite-01] 초경량 Agent 모델 비교
|
||||
|
||||
## 위치
|
||||
|
||||
- Roadmap: [ROADMAP.md](../../../ROADMAP.md)
|
||||
- Phase: [PHASE.md](../PHASE.md)
|
||||
|
||||
## 목표
|
||||
|
||||
`[bench-route-01]`에서 호출 가능성이 확인된 동일 9개 조합을 고정된 실제 비교 과제로 한 번씩 실행해 최소 비교 결과를 남긴다.
|
||||
측정 지원 코드는 만들지 않고, caller 실행과 결과 기록만 측정 경계로 둔다.
|
||||
|
||||
## 상태
|
||||
|
||||
[계획]
|
||||
|
||||
## 구현 잠금
|
||||
|
||||
- 상태: 해제
|
||||
- SDD: 불필요
|
||||
- SDD 문서: 없음
|
||||
- SDD 사유: 기존 caller와 제품 경로를 단일 시도로 관측하는 test-only 작업이며 제품 API, 상태 머신, retry 또는 schema를 변경하지 않는다.
|
||||
- 결정 필요: 없음
|
||||
|
||||
## 범위
|
||||
|
||||
- `[bench-route-01]`과 동일한 9개 caller/model/route 조합
|
||||
- 모든 조합에 같은 고정 비교 prompt와 같은 빈 임시 workspace 사용
|
||||
- 조합별 정확히 1회 실행
|
||||
- 성공 여부, 전체 경과 시간, caller가 직접 제공한 usage, 산출물 경로와 짧은 수동 관찰만 기록
|
||||
- 실패한 조합은 실패로 기록하고 같은 측정 안에서 retry, resume 또는 대체 run을 하지 않음
|
||||
|
||||
## 기능
|
||||
|
||||
### Epic: [thin-run] 단일 시도 비교
|
||||
|
||||
- [ ] [single-attempt-matrix] 9개 조합을 같은 prompt와 초기 상태에서 정확히 한 번씩 실행한다. 검증: 조합별 producer attempt가 하나이며 retry/resume/recovery 기록이 없어야 한다.
|
||||
- [ ] [minimal-result-table] 성공 여부, 경과 시간, caller 제공 usage, 산출물 경로와 짧은 관찰을 단일 Markdown 표로 기록한다. 제공되지 않은 usage는 `미제공`으로 두고 추정하거나 0으로 바꾸지 않는다.
|
||||
- [ ] [bounded-conclusion] 성공한 결과만 비교하고 실패·미제공 데이터를 점수 0으로 취급하지 않는 짧은 결론을 남긴다. 자동 채점이나 통계적 일반화는 하지 않는다.
|
||||
|
||||
## 완료 리뷰
|
||||
|
||||
- 상태: 없음
|
||||
- 요청일: 없음
|
||||
- 완료 근거: `[bench-route-01]`과 단일 시도 결과가 아직 없다.
|
||||
- 검토 항목:
|
||||
- [ ] `[bench-route-01]`이 통과 또는 사용자 승인된 외부 차단 상태다.
|
||||
- [ ] 새 benchmark script와 자동화 state가 없다.
|
||||
- [ ] 조합별 정확히 한 번의 실행과 최소 결과 표만 남았다.
|
||||
- agent-ui 상태 반영: 해당 없음
|
||||
- 리뷰 코멘트: 없음
|
||||
|
||||
## 범위 제외
|
||||
|
||||
- manifest runner, orchestration framework, lifecycle store
|
||||
- retry, resume, recovery qualification, stale-run repair
|
||||
- browser/CDP 자동 검증, screenshot gate, accessibility/network gate
|
||||
- 익명 LLM 채점, 자동 순위, 반복·분산·유의성 분석
|
||||
- 이 측정을 IOP 제품 안정성 또는 다른 Milestone의 완료 조건으로 승격하는 것
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 선행 작업: [벤치 경로 최소 HTML 스모크](benchmark-route-minimal-html-smoke.md)
|
||||
- 실행 방식: 기존 공식 caller 명령을 한 번씩 직접 실행하며 공통 runner를 만들지 않는다.
|
||||
- 결과 위치: `agent-test/dev/iop-thin-agent-model-comparison.md`
|
||||
|
|
@ -4,11 +4,21 @@
|
|||
|
||||
## 실행 순서
|
||||
|
||||
### bench-route
|
||||
|
||||
1. [[bench-route-01] 벤치 경로 최소 HTML 스모크](phase/knowledge-tool-optimization-extension/milestones/benchmark-route-minimal-html-smoke.md)
|
||||
벤치 대상 9개 caller/model/route 조합에 고정된 최소 `index.html` 생성 요청을 한 번씩 직접 보내고 실패 경로만 국소 수정한다.
|
||||
|
||||
### bench-lite
|
||||
|
||||
1. [[bench-lite-01] 초경량 Agent 모델 비교](phase/knowledge-tool-optimization-extension/milestones/thin-agent-model-comparison-benchmark.md)
|
||||
통과한 동일 경로를 복구·재개·자동 채점 없이 한 번씩 실행하고 최소 비교 표만 남긴다.
|
||||
- 선행 차단: `[bench-route-01]`
|
||||
|
||||
### route
|
||||
|
||||
3. [[route-03] Heavy Plan/Review 실행과 검증 MVP](phase/knowledge-tool-optimization-extension/milestones/knowledge-tool-validation-optimization.md)
|
||||
Hot Path의 lightweight Plan/Review를 장기 작업용 `heavy` mode로 확장해 `heavy-only` preset에서 재계획·검증·review/repair·resume 경계를 먼저 검증한다.
|
||||
- 선행 차단: `[bench-02]`
|
||||
|
||||
4. [[route-04] Execution Preset 하이브리드 Mode 라우팅](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-hybrid-request-execution-routing.md)
|
||||
cloud model advisory와 deterministic hard gate를 결합해 Edge가 외부 model에 매핑된 preset의 허용 mode 중 요청 난이도에 맞는 실행 경로를 고르고 route evidence를 축적한다.
|
||||
|
|
@ -17,11 +27,6 @@
|
|||
cloud-first route evidence가 품질·규모 gate를 통과하면 RAG local router를 shadow/canary로 검증해 운영 기본 경로로 점진 전환한다.
|
||||
- 선행 차단: `[observe-03]`, `[provider-02]`
|
||||
|
||||
### bench
|
||||
|
||||
2. [[bench-02] IOP 원샷 Agent 모델 비교 벤치마크](phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md)
|
||||
`[route-02]` 정식 smoke 뒤 동일 정적 웹 fixture로 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 조합을 각각 한 번 비교한다.
|
||||
|
||||
### output
|
||||
|
||||
1. [[output-01] OpenAI-compatible 출력 검증 필터](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-output-validation-filters.md)
|
||||
|
|
|
|||
|
|
@ -1,169 +0,0 @@
|
|||
# SDD: [bench-02] IOP 원샷 Agent 모델 비교 벤치마크
|
||||
|
||||
## 위치
|
||||
|
||||
- Milestone: [IOP 원샷 Agent 모델 비교 벤치마크](../../../phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md)
|
||||
- Phase: [PHASE.md](../../../phase/knowledge-tool-optimization-extension/PHASE.md)
|
||||
|
||||
## 상태
|
||||
|
||||
[승인됨]
|
||||
|
||||
## SDD 잠금
|
||||
|
||||
- 상태: 해제
|
||||
- 사용자 리뷰: 없음
|
||||
- 잠금 항목:
|
||||
- [x] [D01] 실제 비교는 `[route-02]` 정식 기능·필수 smoke와 `[bench-01]` pipeline 준비가 끝난 뒤 시작한다.
|
||||
- [x] [D02] 9개 scored 비교군은 모두 dev `../iop-s2` IOP runtime을 경유한다.
|
||||
- [x] [D03] 단독군은 Sonnet 5 최고, Gemini 3.6 Flash high, GPT-5.6 luna xhigh이며 Gemini/GPT는 Claude Code와 전용 caller(agy/Codex)를 각각 비교한다.
|
||||
- [x] [D04] 하이브리드는 Gemini 또는 GPT가 plan/review/repair를, ornith-fast가 work를 담당하고 각각 Claude Code와 전용 caller를 비교한다.
|
||||
- [x] [D05] 동일 이미지 2장과 vanilla HTML/CSS/JS 한 페이지 fixture를 clean workspace에 제공한다.
|
||||
- [x] [D06] 각 승인된 scored run의 repetitions는 cell별 1이며 clean workspace와 fresh caller session에서 사용자 작업 제출 1회부터 finish/complete 후 idle까지 사람 개입 없이 실행한다. caller launch 전 harness 결함으로 중단된 과거 run 뒤 사용자가 새 실행을 명시적으로 승인하면, 과거 run을 resume/retry하지 않고 새 run identity로 같은 immutable manifest를 한 번 실행할 수 있다.
|
||||
- [x] [D07] 시간은 첫 output, 첫 file write, model/stage별 작업, tool, queue와 전체 finish/idle을 clock/source와 함께 기록하고 중첩 구간이나 미관측 overhead를 임의 산술 분해하지 않는다.
|
||||
- [x] [D08] token은 input/output/reasoning/cached/total과 source를 model/stage별로 기록하고 미제공 값을 exact로 추정하지 않는다.
|
||||
- [x] [D09] 결과 identity를 가린 뒤 동일 100점 rubric으로 Codex가 채점하고 자동 검증과 수동 점수를 분리한다.
|
||||
- [x] [D10] scored failure는 보존하고 같은 run의 재실행은 명시적 retry의 새 attempt로만 기록하며 성공 결과만 골라 대표하지 않는다. 새 run 승인은 기존 실패 run/attempt를 대체하지 않는 별도 비교 cycle이며, 보고서에는 이전 실패 run과 새 run의 관계 및 한계를 함께 남긴다.
|
||||
- [x] [D11] 공식 `agy 1.1.12`는 Gemini API-key provider의 route별 `GOOGLE_GEMINI_BASE_URL`을 IOP Edge로 지정하고 `GEMINI_API_KEY`에는 upstream key가 아닌 IOP principal token을 넣는다. `--effort`와 비공식 custom model은 사용하지 않고 high effort는 IOP effective binding으로 검증한다.
|
||||
- [x] [D12] marked hybrid preset은 dev managed credential plane의 fresh projection, 고정 stage authorization과 sealed provider lease가 준비된 뒤에만 실행하며 legacy credential fallback을 허용하지 않는다.
|
||||
- [x] [D13] 배포 qualification의 5-cell direct 진단은 all-success가 아니라 terminal-evidence completeness를 판정한다. fresh `ready=5`, 정확히 5개의 fresh attempt, `unresolved=0`, `running=0`, `interrupted=0`, 모든 slot의 controller/product/harness/process/web-validation terminal evidence와 exhausted browser/CDP infrastructure block 없음이 필요하다. 제품 실패·provider rejection·caller failure 뒤 generated-missing·timeout은 보존할 benchmark 결과이며 암묵 재시도하지 않는다. Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. 이 unscored 진단은 D06/D10의 유일한 새 scored C01-C09 run identity를 소비하지 않는다.
|
||||
- [x] [D14] Milestone 완료에는 복구 후 승인된 새 C01-C09 full run에서 9개 cell 모두 `product=succeeded`, `harness=passed`, `process=exited/0`, 신뢰 가능한 생성물과 viewport screenshot을 남기고 익명 rubric 결과가 `scored=9`, `unscored=0`, `scoring_failed=0`, `blocked=0`이어야 한다. terminal failure를 보존했다는 사실만으로 Milestone이나 최종 벤치 task를 완료하지 않는다.
|
||||
- [x] [D15] 생성 파일·정적 안전·asset·render처럼 evaluator 입력의 존재와 신뢰성을 보장하는 gate는 hard eligibility로 유지한다. console·responsive·accessibility처럼 생성물의 품질을 평가하는 gate 실패는 신뢰 가능한 페이지가 존재하면 unscored 사유가 아니라 익명 rubric의 감점 evidence로 전달한다.
|
||||
|
||||
## 문제 / 비목표
|
||||
|
||||
- 문제: `[route-02]` 하이브리드 원샷의 실사용 가치와 overhead를 판단하려면 같은 IOP 경계, task fixture와 평가 기준에서 단독 모델·caller agent 조합과 속도·token·품질을 함께 비교해야 한다. 단일 성공 smoke만으로는 모델·agent·coordinator 차이를 설명할 수 없다.
|
||||
- 비목표:
|
||||
- `[route-02]` 완료 smoke를 대신하거나 benchmark 점수로 완료 상태를 소급 변경
|
||||
- 첫 보고서에서 통계적 다회 반복이나 장기/heavy 작업 평가
|
||||
- framework 설치·cache 성능 비교
|
||||
- model/provider 가격표를 billing-grade 비용으로 확정
|
||||
|
||||
## Source of Truth
|
||||
|
||||
| 영역 | 기준 | 메모 |
|
||||
|------|------|------|
|
||||
| Roadmap | [Milestone 문서](../../../phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md) | 9개 비교군과 완료 상태 원장 |
|
||||
| Pipeline | [bench-01 Milestone](../../../phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md)의 승인된 manifest/runner/report contract | 실행·측정·보고 구현 원본 |
|
||||
| Testbed | `../iop-s2` dev IOP runtime | 모든 scored model 호출의 IOP 경유 대상 |
|
||||
| Fixture | versioned prompt, 이미지 2장과 vanilla workspace checksum | 모든 cell의 동일 입력 기준 |
|
||||
| Evidence | `agent-test/runs/<run-id>/` | attempt별 timeline, usage, validation, screenshot와 score |
|
||||
| Report | `agent-test/dev/iop-one-shot-agent-comparison-<date>.md` | 현재 프로젝트의 사람이 읽는 비교 결과 |
|
||||
| API Contract | [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md) | caller ingress, stream/terminal과 usage 기준 |
|
||||
| Config Contract | [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md) | model route, preset, protocol profile과 credential 경계 |
|
||||
| User Decision | D01-D10 | 2026-08-06 확정 방향, 추가 사용자 결정 없음 |
|
||||
|
||||
## State Machine
|
||||
|
||||
| 상태 | 진입 조건 | 다음 상태 | 근거 |
|
||||
|------|-----------|-----------|------|
|
||||
| `blocked` | `[route-02]` smoke 또는 `[bench-01]` 완료 전 | `preflighting`, 종료 | active Milestone 상태와 pipeline evidence |
|
||||
| `preflighting` | 선행 조건 충족, execution-day caller/route/credential 점검 | `ready`, `blocked` | official caller transport, managed projection/lease와 redacted preflight matrix |
|
||||
| `ready` | fixture와 C01-C09 immutable manifest 확정 | `running`, `cancelled` | manifest/fixture/rubric digest |
|
||||
| `running` | seed 순서에 따라 각 cell에 사용자 작업 1회 제출 | `validating`, `failed`, `timed_out`, `cancelled` | cell/attempt event timeline |
|
||||
| `validating` | cell finish/complete 후 idle 확정 | `scoring`, `failed` | workspace, build/render/test evidence |
|
||||
| `scoring` | C01-C09 결과 identity 제거 완료 | `analyzing`, `failed` | 9개 blind mapping과 rubric worksheet |
|
||||
| `analyzing` | 자동 gate·시간·usage와 9개 score 완비 | `reported`, `failed` | `scored=9`, `unscored=0`, `scoring_failed=0`, `blocked=0` 및 comparison table |
|
||||
| `reported` | Markdown과 raw evidence 포인터 생성 | 종료 | report path와 digest |
|
||||
| `failed` | cell 실행·검증·채점·보고 실패 | `analyzing`, 종료 | 보존된 실패 attempt; 누락 없는 matrix |
|
||||
| `timed_out` | cell timeout | `analyzing`, 종료 | timeout/cancel/cleanup evidence |
|
||||
| `cancelled` | 명시 중단 | 종료 | 실행된 cell과 미실행 cell 상태 |
|
||||
|
||||
State invariant:
|
||||
|
||||
- C01-C09는 승인된 run마다 동일 fixture checksum, viewport, rubric version, fresh caller session, setup/cache policy와 repetitions=1을 사용한다. 새 run은 기존 실패 run과 다른 run identity를 가지며 기존 run을 resume/retry하거나 대표 결과에서 삭제하지 않는다.
|
||||
- execution order는 고정 seed로 생성해 보고서에 남기고 결과에 따라 재정렬하지 않는다.
|
||||
- preflight와 setup usage/time은 scored measurement에 합산하지 않지만 별도 기록한다.
|
||||
- 한 cell의 사용자 작업은 한 번 제출하며 사람의 feedback, manual edit, restart가 없다.
|
||||
- model/tool 호출 횟수는 제약이 아니라 측정 대상이며 finish event 뒤 idle까지가 wall-clock terminal이다.
|
||||
- 실패 cell도 report matrix에 남고 재실행 결과는 원래 attempt를 대체하지 않는다.
|
||||
- 배포 qualification의 5-cell direct 진단은 scored C01-C09 run과 별개다. terminal evidence가 완결된 제품 실패·provider rejection·timeout을 acceptance failure로 재해석하지 않고, Edge pre-ingress incompatibility 또는 최대 renderer 재시도 뒤 browser/CDP infrastructure block만 다음 단계 진입을 막는다.
|
||||
- 복구 후 final full run은 9개 cell 모두 성공 산출물과 익명 점수를 가져야 한다. 어느 한 cell이라도 product/harness/process/hard artifact gate가 실패하거나 `unscored`, `scoring_failed`, `blocked`이면 보고서는 진단 산출물로 보존하되 Milestone 완료로 판정하지 않는다.
|
||||
- 품질 gate 실패는 hard trust gate와 분리한다. 생성물이 신뢰 가능하면 console·responsive·accessibility 실패도 evaluator에 전달해 해당 rubric 항목에서 감점한다.
|
||||
|
||||
## Interface Contract
|
||||
|
||||
- 계약 원문: [Anthropic-Compatible Messages API](../../../../agent-contract/outer/anthropic-compatible-api.md), [OpenAI-Compatible API](../../../../agent-contract/outer/openai-compatible-api.md), [Gemini-Compatible API](../../../../agent-contract/outer/gemini-compatible-api.md), [Edge Config And Runtime Refresh](../../../../agent-contract/inner/edge-config-runtime-refresh.md)
|
||||
- 입력:
|
||||
- `fixture`: 동일 이미지 2장, one-page 요구사항, vanilla HTML/CSS/JS initial workspace와 checksum이다.
|
||||
- `cells`: C01-C09의 caller, IOP route/preset, expected model/stage와 effort binding이다.
|
||||
- `repetitions=1`, `session_policy=fresh`, `setup_cache_policy`: 초기 scored attempt 수, conversation/resume 격리와 공통 setup/cache 기준이다.
|
||||
- `environment=dev`, `testbed=../iop-s2`: 실제 IOP runtime 선택이다.
|
||||
- `completion`: caller별 finish/complete event와 idle 판정 규칙이다.
|
||||
- `agy`: `modelProvider=gemini`, route별 `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY=<IOP principal token>`과 official `stream-json` event다. API-key provider가 지원하지 않는 `--effort`는 전달하지 않는다.
|
||||
- 측정 출력:
|
||||
- timestamp: submitted, first output, first file write, model/stage start/end, tool start/end, finish, idle의 monotonic 값과 observation source다. overlap과 unavailable을 명시한다.
|
||||
- usage: call count, input/output/reasoning/cached/total token과 source다.
|
||||
- validation: requirement, build/serve, desktop/mobile, asset/console, responsive/accessibility 결과다.
|
||||
- score: rubric version, 항목별 점수/근거와 총점이며 identity mapping과 분리한다.
|
||||
- 100점 rubric:
|
||||
- 요구사항 충족 25, 시각 완성도 25, 반응형·접근성 15, 이미지 활용·디테일 10, 동작 안정성 10, 코드 품질 10, 자체 검증 완결성 5.
|
||||
- 금지:
|
||||
- IOP를 우회한 model 호출을 scored cell로 인정한다.
|
||||
- cell마다 prompt, asset, initial workspace나 viewport를 다르게 사용한다.
|
||||
- unavailable token을 0으로 기록하거나 estimated 값을 provider-reported와 합친다.
|
||||
- evaluator가 identity를 본 상태에서 점수를 조정하거나 결과를 수동 수정한다.
|
||||
|
||||
## Acceptance Scenarios
|
||||
|
||||
| ID | Milestone Task | Given | When | Then |
|
||||
|----|----------------|-------|------|------|
|
||||
| S01 | `fixture-lock` | 이미지 2장과 one-page benchmark brief | fixture 확정 | prompt/assets/workspace/viewports/rubric의 checksum과 version이 모든 cell에 동일하다. |
|
||||
| S02 | `route-readiness` | 5-cell direct 진단과 C01-C09 caller·dev IOP | execution-day qualification과 preflight | direct는 fresh `ready=5`, 정확히 5개 attempt, 모든 terminal evidence와 infrastructure block 없음으로 admission되고, 이후 C01-C09 auth/model/preset/effort/stream/finish/idle이 모두 확인되거나 exact blocker로 중단된다. |
|
||||
| S03 | `matrix-lock` | 선행 gate가 통과한 9개 cell | scored manifest 생성 | repetitions=1, 실행 순서 seed, fresh-session/setup-cache 정책, timeout과 expected binding이 immutable하게 기록된다. |
|
||||
| S04 | `claude-standalone` | C01 clean workspace | Claude Code 사용자 작업 1회 | IOP→Sonnet 최고 effort 결과와 complete/idle evidence가 생성된다. |
|
||||
| S05 | `gemini-standalone` | C02-C03 clean workspace | Claude Code와 agy 사용자 작업을 각각 1회 제출 | 두 caller 모두 IOP→Gemini high 결과와 caller별 timing/usage를 남긴다. |
|
||||
| S06 | `gpt-standalone` | C04-C05 clean workspace | Claude Code와 Codex 사용자 작업을 각각 1회 제출 | 두 caller 모두 IOP→GPT xhigh 결과와 caller별 timing/usage를 남긴다. |
|
||||
| S07 | `gemini-hybrid` | C06-C07 clean workspace | Claude Code와 agy 사용자 작업을 각각 1회 제출 | IOP Gemini plan→ornith work→Gemini review/repair의 stage evidence와 최종 결과를 남긴다. |
|
||||
| S08 | `gpt-hybrid` | C08-C09 clean workspace | Claude Code와 Codex 사용자 작업을 각각 1회 제출 | IOP GPT plan→ornith work→GPT review/repair의 stage evidence와 최종 결과를 남긴다. |
|
||||
| S09 | `objective-validation` | 복구 후 C01-C09 성공 workspace | 자동 웹 검증 | 각 cell의 동일 hard trust gate, 품질 gate와 desktop/mobile screenshot이 누락 없이 생성되며 hard gate는 모두 통과한다. |
|
||||
| S10 | `quality-scoring` | identity가 제거된 9개 신뢰 가능 결과 | Codex rubric 평가 | 9개 모두 항목별 점수/근거와 총점이 기록되고 `scored=9`, `unscored=0`, `scoring_failed=0`, `blocked=0`이다. 품질 gate 실패는 해당 항목의 감점 evidence다. |
|
||||
| S11 | `performance-usage` | 모든 attempt timeline/usage | 비교 집계 | 첫 output·첫 write·model/tool/queue/total 시간의 clock/source·overlap, 호출 수와 token/source가 cell·stage별 표가 된다. |
|
||||
| S12 | `benchmark-report` | S01-S11 evidence | 보고서 생성 | 조건·버전·9개 결과·속도·token·품질·실패·한계와 raw evidence 링크가 Markdown에 남는다. |
|
||||
| S13 | `agy-iop-compatibility` | official `agy 1.1.12`와 dev Edge | direct·hybrid route별 Gemini base URL로 실제 API-key 호출 | 두 호출 모두 `x-goog-api-key` IOP principal auth, Gemini-native request/tool/SSE, official `stream-json` finish/exit와 config-owned effective binding evidence를 남기고 upstream key 직접 호출이나 합성 event에 의존하지 않는다. |
|
||||
| S14 | `managed-credential-dev` | dev Control Plane·Edge·Node와 operator-owned security material | managed credential profile로 재기동하고 slot/route를 등록 | CA-signed mTLS·Edge HTTPS·fresh projection·sealed lease가 확인되고 legacy credential source나 cross-route fallback 없이 direct와 marked preset stage가 실행된다. |
|
||||
|
||||
## Evidence Map
|
||||
|
||||
| Scenario | Required Evidence | `agent-task` 연결 | 완료 Evidence 기대 |
|
||||
|----------|-------------------|------------------|---------------------------|
|
||||
| S01 | fixture prompt/assets/workspace/rubric digest | `agent-task/m-iop-one-shot-agent-model-comparison/fixture-lock/` | `fixture-lock` identical-input evidence |
|
||||
| S02 | redacted direct `ready=5`·5-slot terminal evidence·infrastructure 판정과 C01-C09 preflight matrix | `agent-task/m-iop-one-shot-agent-model-comparison/route-readiness/` | `route-readiness` direct admission과 auth/route/effort/terminal evidence |
|
||||
| S03 | immutable scored manifest와 order seed | `agent-task/m-iop-one-shot-agent-model-comparison/matrix-lock/` | `matrix-lock` 9-cell/repetitions=1 evidence |
|
||||
| S04 | C01 event/timing/usage/workspace evidence | `agent-task/m-iop-one-shot-agent-model-comparison/claude-standalone/` | `claude-standalone` one-submission/IOP evidence |
|
||||
| S05 | C02-C03 caller별 event/timing/usage/workspace evidence | `agent-task/m-iop-one-shot-agent-model-comparison/gemini-standalone/` | `gemini-standalone` two-caller evidence |
|
||||
| S06 | C04-C05 caller별 event/timing/usage/workspace evidence | `agent-task/m-iop-one-shot-agent-model-comparison/gpt-standalone/` | `gpt-standalone` two-caller evidence |
|
||||
| S07 | C06-C07 Gemini/ornith stage와 terminal evidence | `agent-task/m-iop-one-shot-agent-model-comparison/gemini-hybrid/` | `gemini-hybrid` two-caller stage evidence |
|
||||
| S08 | C08-C09 GPT/ornith stage와 terminal evidence | `agent-task/m-iop-one-shot-agent-model-comparison/gpt-hybrid/` | `gpt-hybrid` two-caller stage evidence |
|
||||
| S09 | build/render/viewport/asset/console/accessibility result와 screenshot | `agent-task/m-iop-one-shot-agent-model-comparison/objective-validation/` | `objective-validation` uniform gate evidence |
|
||||
| S10 | blind mapping 분리와 Codex rubric worksheet | `agent-task/m-iop-one-shot-agent-model-comparison/quality-scoring/` | `quality-scoring` 100-point evidence |
|
||||
| S11 | cell/stage별 normalized timeline, calls와 token-source table | `agent-task/m-iop-one-shot-agent-model-comparison/performance-usage/` | `performance-usage` speed/token evidence |
|
||||
| S12 | `agent-test/dev/` Markdown과 raw run links | `agent-task/m-iop-one-shot-agent-model-comparison/benchmark-report/` | `benchmark-report` complete comparison evidence |
|
||||
| S13 | official agy request-shape capture, Edge Gemini bridge tests, direct·hybrid live preflight와 sanitized lifecycle/usage | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `agy-iop-compatibility` official 1.1.12 IOP transport evidence |
|
||||
| S14 | dev config check, TLS/workload identity, projection generation, slot-route/lease attribution과 post-revoke no-fallback smoke | `agent-task/m-iop-one-shot-agent-model-comparison/06+05_comparison_runs/` | `managed-credential-dev` secure composition and hybrid admission evidence |
|
||||
|
||||
공통 완료 검증은 복구 후 승인된 새 C01-C09 full run에서 9개 모두 success terminal evidence, hard trust gate 통과, desktop/mobile screenshot, blind score와 timing/usage source를 가지며 최종 집계가 `scored=9`, `unscored=0`, `scoring_failed=0`, `blocked=0`인지 확인한다. 이전 실패 run은 원인·한계 evidence로 보존하지만 완료 근거를 대체하지 않는다. 필수 credential/model이 없으면 raw secret을 요구하거나 기록하지 않고 운영 절차로 등록을 요청한다.
|
||||
|
||||
## Cross-repo Dependencies
|
||||
|
||||
- 없음. 같은 IOP 프로젝트의 `[route-02]`와 `[bench-01]` 실행 순서는 [전역 마일스톤 실행 순서](../../../priority-queue.md)에서 관리한다.
|
||||
|
||||
## Drift Check
|
||||
|
||||
- [x] Milestone 기능 Task와 Acceptance Scenario가 일치한다.
|
||||
- [x] Evidence Map이 code-review/complete.log에서 검증 가능하다.
|
||||
- [x] agent-contract를 쓰는 경우 SDD에 계약 원문을 복제하지 않았다.
|
||||
- [x] 사용자 리뷰가 필요한 항목은 없고 확정된 D01-D13을 반영했다.
|
||||
|
||||
## 사용자 리뷰 이력
|
||||
|
||||
- 2026-08-06: 사용자가 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 비교군, Claude Code·agy·Codex caller, finish/idle 원샷, 초기 1회, dev `../iop-s2`, 동일 정적 웹 fixture와 시간·token·Codex 품질 평가를 확정했다.
|
||||
- 2026-08-12: 공식 `agy 1.1.12` API-key provider의 실제 Gemini-native 요청과 `stream-json` event를 확인했고, 사용자의 provider 직접 설정 지시에 따라 upstream key와 IOP principal token을 분리하며 dev managed credential plane까지 구성하는 D11-D12를 기술 보강했다.
|
||||
- 2026-08-13: 사용자가 terminal outcome과 dispatcher 환경 보완 뒤 다음 벤치까지 계속 실행하도록 승인했다. 이에 기존 실패 run을 보존하고 resume/retry하지 않은 채, 동일 immutable C01-C09 manifest로 repetitions=1인 새 scored run identity를 한 번 생성하는 D06/D10 경계를 확정했다.
|
||||
- 2026-08-13: 승인된 후속 packet에 따라 direct 배포 qualification을 terminal-evidence admission으로 분리했다. 제품 실패·provider rejection·timeout은 측정 결과로 보존하고, Edge pre-ingress incompatibility와 exhausted browser/CDP infrastructure block만 qualification을 막는 D13을 추가했다. D06/D10의 scored-run uniqueness는 유지한다.
|
||||
- 2026-08-13: 사용자가 2개 scored/7개 unscored 상태에서 종료하지 말고 9개 모두 측정되도록 원인을 해소하라고 명시했다. 이에 기존 실패 run은 보존하되 완료 조건을 복구 후 새 full run의 `9/9 scored`로 강화하고, 품질 gate와 hard trust gate를 분리하는 D14-D15를 추가했다.
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 표준선: 이 비교는 `[route-02]` 완료 이후의 추가 검증이며 정식 smoke의 일부나 대체 evidence가 아니다.
|
||||
- 후속 SDD: 없음
|
||||
17
agent-spec/archive/testing/agent-comparison-benchmark.log
Normal file
17
agent-spec/archive/testing/agent-comparison-benchmark.log
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
---
|
||||
spec_doc_type: archive-log
|
||||
spec_id: testing/agent-comparison-benchmark
|
||||
status: 폐기됨
|
||||
source_evidence:
|
||||
- type: user
|
||||
path: null
|
||||
notes: 2026-08-13 전용 하네스를 폐기하고 공식 caller별 수동 개별 검증을 우선하도록 결정
|
||||
- type: roadmap
|
||||
path: agent-roadmap/archive/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md
|
||||
notes: 폐기된 측정 Milestone과 경계 분리 근거
|
||||
---
|
||||
|
||||
# 폐기 기록: Agent 비교 벤치마크 파이프라인
|
||||
|
||||
전용 manifest runner, 재개·복구 저장소, 자동 웹 검증, 익명 채점과 보고 파이프라인은 2026-08-13 폐기했다.
|
||||
현재 IOP 호출부는 완료된 제품 기준선으로 유지한다. 구체적 결함이 재현되면 해당 코드와 로드맵 소유 영역에서 국소 처리하며, 전용 benchmark CLI나 전수 안정성 campaign은 현재 구현·완료 표면이 아니다.
|
||||
|
|
@ -27,7 +27,6 @@ AI agent가 작업 전에 읽는 지도이기도 하지만, 사람도 "지금
|
|||
- 출력 검증 런타임: staged response-start, evidence hold/release, filter arbitration, bounded recovery/rebuild, raw-free observation은 `runtime/stream-evidence-gate`에서 본다.
|
||||
- 외부 HTTP 입력: OpenAI-compatible 호출, Anthropic-compatible Messages 호출, managed principal route/slot binding, model-driven raw tunnel은 `input/openai-compatible-surface`, A2A JSON-RPC 호출은 `input/a2a-json-rpc-surface`에서 본다.
|
||||
- 운영 제어: Control Plane, credential HTTPS/host-local bootstrap, mTLS Edge enrollment, projection/lease flow, fleet/edge status, Flutter Client 상태 소비는 `control/control-plane-operations`에서 본다.
|
||||
- 테스트 도구: Claude Code, agy, Codex의 IOP 경유 비교를 manifest로 실행하고 격리·측정·웹 검증·익명 채점·보고하는 현재 harness는 `testing/agent-comparison-benchmark`에서 본다.
|
||||
|
||||
## 스펙 목록
|
||||
|
||||
|
|
@ -39,7 +38,6 @@ AI agent가 작업 전에 읽는 지도이기도 하지만, 사람도 "지금
|
|||
| `input/openai-compatible-surface` | 부분 | `/v1/models`, `/v1/chat/completions`, `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens`, `/anthropic/v1/models`, managed projection/slot routing, OpenAI-compatible auth/metadata/tool handling, Anthropic bearer/`X-Api-Key` auth, provider-pool native/bridge admission, safe slot attribution, marked single-request 내부 stage template과 caller-visible I/O 경계, and OpenAI-only usage metrics를 확인할 때 | `agent-spec/input/openai-compatible-surface.md` | `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/anthropic-compatible-api.md`, `apps/edge/internal/openai/chat_handler.go`, `apps/edge/internal/openai/anthropic_handler.go`, `apps/edge/internal/openai/anthropic_bridge.go`, `apps/edge/internal/openai/normalized_sse.go`, `apps/edge/internal/openai/usage_metrics.go` |
|
||||
| `input/a2a-json-rpc-surface` | 부분 | Edge A2A JSON-RPC, `message/send`, `tasks/get`, `tasks/cancel`, A2A task store와 bearer auth를 확인할 때 | `agent-spec/input/a2a-json-rpc-surface.md` | `agent-contract/outer/a2a-json-rpc-api.md`, `apps/edge/internal/input/a2a/server.go`, `apps/edge/internal/input/a2a/task_store.go` |
|
||||
| `control/control-plane-operations` | 부분 | credential HTTPS and host-local bootstrap, Control Plane-Edge mTLS projection/lease wire, Client-Control Plane wire, Control Plane HTTP Edge/fleet status view, Flutter Client status consumer를 확인할 때 | `agent-spec/control/control-plane-operations.md` | `agent-contract/inner/control-plane-edge-wire.md`, `agent-contract/inner/client-control-plane-wire.md`, `apps/control-plane/internal/wire/edge_server.go`, `apps/control-plane/internal/credentiallease/service.go` |
|
||||
| `testing/agent-comparison-benchmark` | 구현됨 | Claude Code, agy, Codex의 IOP 경유 benchmark manifest, preflight, run/resume, 격리, timing/usage, 웹 gate, 익명 채점과 Markdown report를 확인할 때 | `agent-spec/testing/agent-comparison-benchmark.md` | `scripts/agent_comparison_benchmark.py`, `scripts/agent_benchmark/attempts.py`, `scripts/agent_benchmark/scoring.py`, `scripts/agent_benchmark/reporting.py` |
|
||||
|
||||
## 작성 규칙
|
||||
|
||||
|
|
|
|||
|
|
@ -1,143 +0,0 @@
|
|||
---
|
||||
spec_doc_type: spec
|
||||
spec_id: testing/agent-comparison-benchmark
|
||||
status: 구현됨
|
||||
source_evidence:
|
||||
- type: code
|
||||
path: scripts/agent_comparison_benchmark.py
|
||||
notes: validate, preflight, run, resume, status, score, report 공개 CLI
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/manifest.py
|
||||
notes: manifest 정규화, 경로 격리, fixture와 입력 digest 검증
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/attempts.py
|
||||
notes: append-only run/attempt 저장과 preflight, 실행, 재개 lifecycle
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/connectivity.py
|
||||
notes: caller capability, requested/effective binding과 blocker 분류
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/measurement.py
|
||||
notes: source-aware timing과 usage 정규화
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/web_validation.py
|
||||
notes: 정적 웹 산출물과 desktop/mobile 자동 gate 검증
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/browser_cdp.py
|
||||
notes: Chromium/CDP 단일 시도 격리와 닫힌 transient class의 최대 3회 fresh 재시작
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/scoring.py
|
||||
notes: 익명화 입력, fresh evaluator와 scoring attempt 처리
|
||||
- type: code
|
||||
path: scripts/agent_benchmark/reporting.py
|
||||
notes: deterministic Markdown 보고서 생성
|
||||
- type: test
|
||||
path: scripts/agent_benchmark/connectivity_integration_test.py
|
||||
notes: 세 caller의 IOP binding, lifecycle, 격리와 실패 경계 통합 검증
|
||||
- type: test
|
||||
path: scripts/agent_benchmark/scoring_test.py
|
||||
notes: unscored, scoring_failed, 새 scoring attempt와 익명화 검증
|
||||
- type: test
|
||||
path: scripts/agent_benchmark/reporting_test.py
|
||||
notes: all-status, 동점과 raw evidence 포인터 보고 검증
|
||||
- type: test
|
||||
path: scripts/agent_benchmark/skill_contract_test.py
|
||||
notes: project-local skill과 공개 CLI 계약 검증
|
||||
- type: test
|
||||
path: scripts/agent_benchmark/browser_cdp_test.py
|
||||
notes: transient 재시도·소진·비재시도와 process/screenshot 정리 검증
|
||||
- type: contract
|
||||
path: agent-contract/outer/openai-compatible-api.md
|
||||
notes: agy와 Codex가 사용하는 IOP OpenAI-compatible ingress 계약
|
||||
- type: contract
|
||||
path: agent-contract/outer/anthropic-compatible-api.md
|
||||
notes: Claude Code가 사용하는 IOP Anthropic-compatible ingress 계약
|
||||
- type: contract
|
||||
path: agent-contract/inner/edge-config-runtime-refresh.md
|
||||
notes: model, route, execution preset과 protocol profile 설정 계약
|
||||
- type: roadmap
|
||||
path: agent-roadmap/archive/phase/knowledge-tool-optimization-extension/milestones/agent-comparison-benchmark-pipeline.md
|
||||
notes: bench-01 완료 범위와 evidence 집계
|
||||
- type: sdd
|
||||
path: agent-roadmap/archive/sdd/knowledge-tool-optimization-extension/agent-comparison-benchmark-pipeline/SDD.md
|
||||
notes: benchmark lifecycle, 실패 보존, 채점과 secret-safe evidence 결정
|
||||
---
|
||||
|
||||
# 스펙: Agent 비교 벤치마크 파이프라인
|
||||
|
||||
## 목적
|
||||
|
||||
Claude Code, agy, Codex가 IOP를 경유해 수행하는 동일 과업을 설정만 바꿔 반복 실행하고, 연결 상태부터 실행·검증·채점·보고까지 재현 가능한 evidence로 남기는 현재 benchmark harness를 설명한다.
|
||||
|
||||
## 기능 목록
|
||||
|
||||
| 기능 | 설명 |
|
||||
|------|------|
|
||||
| manifest 검증 | caller, IOP direct/preset route, model, effort, fixture, 반복 횟수, timeout, evaluator와 `agent-test/runs/<output-id>` 경로를 검증하고 canonical digest를 만든다. |
|
||||
| 연결 preflight | Claude Code, agy, Codex의 binary/config와 IOP endpoint·auth·model·effort·stream binding을 확인하고 `ready`, `registration_required`, `implementation_gap`으로 분류한다. |
|
||||
| 격리 실행과 재개 | 각 cell/repetition을 동일 checksum의 clean workspace와 fresh caller session에서 실행하며, 제품 결과·harness 정합성·process 종료를 독립 결과로 보존한다. top-level state는 `running`, `completed`, `timed_out`, `cancelled`, `interrupted`의 controller 상태만 나타내며, 실패한 attempt는 덮어쓰지 않는다. |
|
||||
| 측정과 evidence | 제출, 첫 출력, 첫 파일 쓰기, model/tool/queue, finish/idle 시간을 관측 source와 함께 정규화한다. token은 보고 주체와 미제공 상태를 보존하며 임의 추정값을 authoritative 값과 섞지 않는다. |
|
||||
| 웹 자동 검증 | product/harness 성공 여부와 무관하게 모든 terminal workspace에서 필수 HTML/CSS/JS와 로컬 이미지, 외부 asset 금지, desktop/mobile render, console/asset 오류, 반응형·접근성 gate와 screenshot을 확인한다. |
|
||||
| 익명 품질 채점 | 필수 자동 gate를 통과한 결과만 identity를 가린 뒤 manifest에 고정된 fresh evaluator로 100점 rubric을 평가한다. 부적격 결과는 `unscored`, 평가 실패는 `scoring_failed`로 남기며 retry는 새 scoring attempt id를 사용한다. |
|
||||
| 상태와 보고 | `validate`, `preflight`, `run`, `resume`, `status`, `score`, `report` CLI를 제공하고, controller/product/harness/process/artifact/scoring 축과 동점을 raw evidence 포인터와 함께 deterministic Markdown으로 만든다. |
|
||||
|
||||
## 범위
|
||||
|
||||
- 포함: benchmark manifest, caller adapter binding, run/attempt 저장, 격리 workspace, source-aware timing/usage, 정적 웹 gate, 익명 채점, Markdown 보고.
|
||||
- 제외: IOP runtime 자체 구현, provider credential 등록, 실제 9개 비교군 실행과 모델 우열 결론. 해당 실행과 결론은 `[bench-02]`가 소유한다.
|
||||
|
||||
## 주요 흐름
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Operator[사용자 또는 project skill] --> CLI[benchmark CLI]
|
||||
CLI --> Manifest[manifest 검증]
|
||||
Manifest --> Preflight[caller와 IOP preflight]
|
||||
Preflight --> Store[append-only run store]
|
||||
Store --> Attempt[clean workspace와 fresh session attempt]
|
||||
Attempt --> Evidence[timing, usage와 web evidence]
|
||||
Evidence --> Score[익명 evaluator scoring]
|
||||
Score --> Report[deterministic Markdown report]
|
||||
```
|
||||
|
||||
## 계약
|
||||
|
||||
- OpenAI-compatible caller ingress는 [OpenAI-Compatible API](../../agent-contract/outer/openai-compatible-api.md)를 따른다.
|
||||
- Claude Code ingress는 [Anthropic-Compatible Messages API](../../agent-contract/outer/anthropic-compatible-api.md)를 따른다.
|
||||
- model, direct/preset route와 protocol profile의 기준은 [Edge Config And Runtime Refresh](../../agent-contract/inner/edge-config-runtime-refresh.md)를 따른다.
|
||||
- 사용자-facing orchestration과 안전한 실행 순서는 [IOP Agent Comparison Benchmark skill](../../agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md)이 소유하고, 제품 호출과 evidence 생성은 deterministic Python pipeline이 소유한다.
|
||||
|
||||
## 설정/데이터/이벤트
|
||||
|
||||
- manifest의 matrix cell은 stable id, caller, IOP route kind, requested/effective model과 effort를 가진다. unsupported alias나 effort는 다른 값으로 대체하지 않고 fail-closed한다.
|
||||
- run state는 `agent-test/runs/<output-id>/<run-id>/` 아래에 격리되며 manifest digest가 다른 상태를 재개하지 않는다.
|
||||
- preflight는 scored attempt가 아니며, 실행 중 실패·timeout·cancel과 scoring 실패는 기존 attempt를 수정하지 않고 보존한다.
|
||||
- caller parser는 raw terminal 문자열 대신 `CallerEvent(finish|idle)`, `CallerTerminal(succeeded|failed)`와 typed metric만 반환한다. Claude result가 마지막 active assistant snapshot을 직접 완성하면 adapter가 typed finish와 idle을 함께 투영하고, assistant가 이미 finish를 냈으면 result는 idle만 투영한다. synthetic API error와 agy ERROR result는 `product=failed`, `harness=passed`가 될 수 있으며 parser malformed는 `product=unknown`, `harness=failed`로 구분한다.
|
||||
- durable lifecycle/measurement/attempt evidence는 `product`, `harness`, `process` 객체를 그대로 저장한다. `unresolved`은 수집/검증 완결성(모든 슬롯이 웹 검증 증거 보유)이며, `passed`는 전체 gate 성공으로 유지되고 retry/skip를 제어한다. scoring eligibility는 변경없으며, terminal failure는 `unscored` report row로 유지된다. run/resume exit 0은 `unresolved=0`을 요구하며, 독립 실패 축은 stdout에 남고 `score`로 분류된다.
|
||||
- 배포 qualification은 동일 clean source에서 5-cell direct manifest의 fresh preflight `ready=5`를 확인하고 정확히 5개의 fresh attempt를 한 번씩 실행한다. `unresolved=0`, `running=0`, `interrupted=0`, 모든 slot의 controller/product/harness/process/web-validation terminal evidence, exhausted browser/CDP infrastructure block 없음이 admission 조건이다. product failure, upstream HTTP rejection, caller failure 뒤 generated-missing과 timeout은 측정 결과로 보존하고 암묵 재시도하지 않는다. Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. admission 뒤 fresh C01-C09 preflight `ready=9`까지만 수행하고 hybrid 또는 scored C01-C09 실행은 후속 승인 전에는 할당하지 않는다.
|
||||
- Chromium/CDP renderer는 source/product validation을 재실행하지 않고, process start·handshake·socket loss의 닫힌 transient class에만 fresh browser/profile로 최대 3회 시도한다. 각 실패 시도는 process group과 부분 screenshot을 정리하며 세 번째 실패는 `artifact=blocked`로 유지한다.
|
||||
- lifecycle supervisor는 exit watcher와 출력 reader를 join한 뒤 하나의 child return code를 동결해 lifecycle result와 cleanup receipt가 동일한 exit/signal을 갖게 한다. 불일치 evidence는 resume에서 fail-closed한다.
|
||||
- raw credential과 private endpoint는 tracked manifest, event, log, screenshot과 report에 기록하지 않는다.
|
||||
- report는 run state의 canonical evidence에서 생성되며 성공하지 않은 결과를 0점으로 변환하거나 동점에 임의 순위를 부여하지 않는다.
|
||||
|
||||
## 검증
|
||||
|
||||
- `python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'` - 전체 benchmark unit/integration 계약이 통과한다.
|
||||
- `python3 -m unittest scripts.agent_benchmark.reporting_test scripts.agent_benchmark.skill_contract_test` - 공개 보고 CLI와 project-local skill 계약이 통과한다.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-manifest.example.json` - 표준 manifest가 유효하다.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-supported-direct.example.json` - 지원 direct route fixture가 유효하다.
|
||||
- `python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json` - direct preflight fixture가 유효하다.
|
||||
|
||||
## 한계와 주의사항
|
||||
|
||||
- example manifest와 fake adapter 테스트 통과는 실제 provider credential, endpoint 또는 preset의 live readiness를 의미하지 않는다. 실제 실행 전 redacted preflight가 필요하다.
|
||||
- 현재 living spec은 비교 파이프라인의 구현 상태만 다룬다. 9개 scored cell 실행, 비교 결과와 결론은 후속 `[bench-02]` evidence로 관리한다.
|
||||
- caller나 IOP가 제공하지 않은 usage는 `unavailable`로 남으며, 서로 다른 clock/source의 중첩 구간을 임의로 합산하지 않는다.
|
||||
- benchmark 결과는 허용된 `agent-test/runs/` 경계 안에만 생성한다.
|
||||
|
||||
## 변경 기록
|
||||
|
||||
- 2026-08-13: direct 배포 qualification을 all-success에서 terminal-evidence admission으로 분리했다. fresh `ready=5`, 정확히 5개 attempt와 완결 evidence는 요구하되 제품 실패·timeout은 결과로 보존하고, Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. Chromium/CDP 닫힌 transient class에는 최대 3회 fresh renderer 시도를 추가했다.
|
||||
- 2026-08-13: `unresolved`을 수집/검증 완결성으로 정의하고 `passed`(전체 gate 성공)와 분리했다. run/resume exit 0은 `unresolved=0`(모든 슬롯이 웹 검증 증거 보유)을 요구하며, 독립 실패 축은 stdout에 남고 `score`로 분류된다. 기존 attempt 바이트 변경 없음.
|
||||
- 2026-08-12: caller terminal을 closed typed observation으로 바꾸고 product/harness/process 결과, failure-inclusive artifact gate, 독립 CLI/report/scoring gate와 direct-first qualification을 구현했다.
|
||||
- 2026-08-12: official Claude result-direct/API-error 및 agy ERROR terminal을 lifecycle 계약에 맞게 분리하고, timeout cleanup result/receipt가 같은 child exit snapshot을 사용하도록 동기화했다.
|
||||
- 2026-08-12: `[bench-01]` 종료 감사에서 확인한 421개 benchmark test, manifest/CLI 계약과 구현 evidence를 기준으로 생성했다.
|
||||
|
|
@ -1,98 +0,0 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery plan=0 tag=REPAIR milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring -->
|
||||
|
||||
# Code Review Reference - REPAIR
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.** Complete the fixed checklist, record actual output, leave active files in place, and report ready. Do not decide the verdict, create follow-up plans, archive files, write `complete.log`, modify roadmap state, ask the user, or start the dependent final benchmark. A qualification failure must be recorded without retry; official review owns diagnosis and the next exact fix.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-13
|
||||
task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery, plan=0, tag=REPAIR
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`.
|
||||
- Preserved run: `run-20260813T081326Z-4e1ac5152c6c`, two scored/seven unscored; immutable diagnostic evidence, not completion.
|
||||
- Latest merged baseline: `origin/dev@6e2f4ff8`; deployed runtime was stale before this task.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Rerun safe deterministic checks and reconstruct the release/qualification evidence. PASS requires all six qualification paths to meet the plan's success/hard-gate oracle. If any path fails, do not PASS or let the final task start: collect the exact sanitized terminal/request-stage evidence, select one concrete root-cause fix with exact files/tests, and generate the required follow-up plan through the plan/finalize-routing flow.
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| REPAIR-1 Hard/quality scoring boundary | [ ] |
|
||||
| REPAIR-2 Five-minute manifests | [ ] |
|
||||
| REPAIR-3 Central dev deployment | [ ] |
|
||||
| REPAIR-4 Six-path qualification | [ ] |
|
||||
| REPAIR-5 Recovery report | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REPAIR-1] Separate hard artifact eligibility gates from quality-scoring gates and add fail-closed regression coverage.
|
||||
- [ ] [REPAIR-2] Set the compact benchmark and diagnostic per-cell ceiling to 300 seconds and lock both manifests in tests.
|
||||
- [ ] [REPAIR-3] Commit/push the exact feature, merge it into central dev, and complete the private dev-runtime deployment procedure with a safe restart and source/build identity proof.
|
||||
- [ ] [REPAIR-4] Run exactly one fresh non-scored six-path qualification and require all six caller/route paths to produce successful terminal evidence, generated files, screenshots, and all hard gates.
|
||||
- [ ] [REPAIR-5] Publish a sanitized deterministic recovery report without secrets or claims about the not-yet-run final benchmark.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** Implementing agents must not modify or check this section.
|
||||
|
||||
- [ ] Append one PASS/WARN/FAIL verdict with verified routing signals.
|
||||
- [ ] Re-run focused/full tests and validate both manifests.
|
||||
- [ ] Verify the scoring split remains fail-closed for hard evidence and quality evidence reaches the evaluator.
|
||||
- [ ] Verify exact central dev release/source/build/runtime identity and safe restart evidence.
|
||||
- [ ] Verify exactly one six-path qualification, no retry/resume/score/final run, six successes, 12 screenshots, and 24 passing hard gates.
|
||||
- [ ] On live failure, diagnose and select an exact fix before generating a follow-up; do not write `complete.log`.
|
||||
- [ ] On PASS, archive the pair, preserve milestone metadata, write `complete.log`, move the task directory, and leave the dependent task ready.
|
||||
- [ ] Reconcile `.gitignore`, work log, active parent, and no duplicate execution.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
_Implementer: replace with `None` or exact deviation, reason, and replacement command._
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
_Implementer: record the implemented hard/quality partition, manifest identity, release identity, and no-retry boundary._
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- D14 completion is not met here; this is recovery admission only.
|
||||
- D15 partition is exact: hard=`generated_files,static_safety,images,network`; quality=`console,responsive,accessibility`.
|
||||
- Runtime warnings alone are not caller failure; terminal product/harness/process and hard artifact evidence are authoritative.
|
||||
- No raw secret, credential, provider payload, or identity map enters tracked evidence.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### Focused and full deterministic tests
|
||||
|
||||
_Implementer: paste exact commands, output summaries, and exits._
|
||||
|
||||
### Central dev deployment
|
||||
|
||||
_Implementer: record sanitized feature/dev/release/tag SHA/tree, test counts, binary identities, restart/ports/nodes/providers/capacity results._
|
||||
|
||||
### Six-path qualification
|
||||
|
||||
_Implementer: record exact run id, preflight count, per-cell terminal/hard/quality/screenshot matrix, and no-retry audit._
|
||||
|
||||
### Diff and provenance audit
|
||||
|
||||
_Implementer: record manifest digests, contained report links, `git diff --check`, and status._
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING]** Fill every implementation-owned section and check the final checklist item. Leave review-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---|---|---|
|
||||
| Header, Overview, Archive Snapshot, Review instructions | Fixed | Do not modify |
|
||||
| Implementation completion/checklist | Implementer | Check status only |
|
||||
| Review-Only Checklist and verdict | Reviewer | Implementer must not modify |
|
||||
| Deviations, decisions, verification results | Implementer then reviewer | Record actual evidence |
|
||||
|
|
@ -1,267 +0,0 @@
|
|||
<!-- task=m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery plan=0 tag=REPAIR milestone-task=claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring -->
|
||||
|
||||
# Recover every caller path before the final nine-cell measurement
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement only the closed scoring/timeout/qualification packet below. Run every verification, fill the paired `CODE_REVIEW-cloud-G10.md` with actual output, keep the active files in place, and report ready for official review. Final verdict, follow-up planning, archive moves, `complete.log`, and roadmap synchronization belong only to official review. If the six-path qualification still fails, do not retry it or start the final benchmark; record the exact sanitized terminal evidence and resume condition so the reviewer can select the next concrete fix.
|
||||
|
||||
## Background
|
||||
|
||||
The preserved run `run-20260813T081326Z-4e1ac5152c6c` ended with two scored and seven unscored cells, so it is diagnostic evidence rather than milestone completion. C02 produced a trustworthy rendered page but accessibility alone made it ineligible; the other failures are concentrated in Claude and official agy caller paths against a stale dev runtime. This task first fixes the confirmed eligibility defect, adopts the user-approved 300-second per-cell ceiling, deploys the already-merged latest dev stream-liveness baseline, and runs one non-scored six-path qualification. It must not allocate the final C01-C09 run.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Predecessor: `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`; internal verdict PASS, but its final `2 scored/7 unscored` state is not the user-level completion condition.
|
||||
- Preserved evidence: C02 has `product=succeeded`, `harness=passed`, `process=exited/0`, both screenshots, and only `accessibility=false`; C01 has `parser_error`; C03 returned Gemini `400 INVALID_ARGUMENT` after a first tool turn; C04/C06/C08 returned Claude caller errors; C07 timed out.
|
||||
- Latest central baseline merged into this feature: `origin/dev@6e2f4ff8`, including the stream terminal-liveness recovery absent from the deployed `dev-974` runtime.
|
||||
- Do not modify, resume, retry, delete, or score the preserved run.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
- `agent-contract/outer/anthropic-compatible-api.md`
|
||||
- `agent-contract/outer/gemini-compatible-api.md`
|
||||
- `scripts/agent_benchmark/scoring.py`
|
||||
- `scripts/agent_benchmark/scoring_test.py`
|
||||
- `scripts/agent_benchmark/web_validation.py`
|
||||
- `scripts/agent_benchmark/agy_iop.py`
|
||||
- `scripts/agent_benchmark/agy_iop_test.py`
|
||||
- `scripts/agent_benchmark/claude_iop.py`
|
||||
- `scripts/agent_benchmark/claude_iop_test.py`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- `scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json`
|
||||
- `apps/edge/internal/openai/gemini_handler.go`
|
||||
- `apps/edge/internal/openai/gemini_bridge.go`
|
||||
- `apps/edge/internal/openai/gemini_handler_test.go`
|
||||
- `apps/edge/internal/openai/anthropic_stream.go`
|
||||
- `apps/edge/internal/openai/single_request_anthropic_stream.go`
|
||||
- `apps/edge/internal/openai/single_request_anthropic_stream_test.go`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`, `[승인됨]`, lock released.
|
||||
- First-line milestone task ids: `claude-standalone,gemini-standalone,gpt-standalone,gemini-hybrid,gpt-hybrid,objective-validation,quality-scoring`.
|
||||
- Target scenarios: S04-S10. Evidence Map rows require per-caller lifecycle/workspace evidence, uniform hard gates/screenshots, and anonymous scoring eligibility.
|
||||
- D14 makes 9/9 successful and scored the only final completion condition. D15 makes `generated_files`, `static_safety`, `images`, and `network` hard eligibility gates while `console`, `responsive`, and `accessibility` remain quality evidence. Those rows define REPAIR-1 and the six-path admission gate in REPAIR-4.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- No external handoff was supplied. Repository evidence and the preserved run were read directly.
|
||||
- Current local feature contains `origin/dev@6e2f4ff8`; live preflight is `ready=9`. The remote dev runner `/Users/toki/agent-work/iop-dev` was clean at `019500a5`, twelve commits behind `origin/dev`, with an approximately seven-hour-old Edge/Node runtime listening on 18082/18083/18084/19101.
|
||||
- External runner: `toki@toki-labs.com`, repo `/Users/toki/agent-work/iop-dev`, macOS/arm64. Before mutation, re-run the preflight in the project `dev-runtime-deploy` skill and require clean refs, exact fetched feature SHA, toolchain, runtime identity, and ports.
|
||||
- Use `agent-ops/skills/private/dev-runtime-deploy/SKILL.md` (or the project fallback when private is absent) for the complete feature-to-dev merge, release, tests, rebuild, safe restart, capacity smoke, finish, and atomic push. Do not shortcut its gates.
|
||||
- Constraint: qualification is append-only and non-scored; exactly one fresh run is allowed. Final C01-C09 execution belongs to the dependent task.
|
||||
- Gap: latest-dev live behavior has not yet been observed. Qualification failure is valid evidence, not authority for an implementation agent to choose an unplanned runtime fix. Official review must diagnose it and create a concrete follow-up plan.
|
||||
- Confidence: high for the scoring false positive and stale-runtime mismatch; medium for whether the latest baseline alone closes all six caller paths.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Existing scoring tests cover lifecycle and web failure eligibility, but not the D15 hard-versus-quality split; add regression cases.
|
||||
- Existing manifest tests pin `run_seconds=180`; update them to the approved 300 seconds and validate the diagnostic manifest identity.
|
||||
- Existing adapter/Edge unit tests cannot prove the deployed official Claude/agy process combination. The fresh six-path qualification supplies that external coverage.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No public symbol rename or removal.
|
||||
- Add private scoring gate constants only within `scripts/agent_benchmark/scoring.py`; `_eligibility` is the sole call site whose classification changes.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
- This is split subtask `16+15_all_cell_measurement_recovery`.
|
||||
- Predecessor `15+14_scored_benchmark_and_report` is satisfied by `agent-task/archive/2026/08/m-iop-one-shot-agent-model-comparison/15+14_scored_benchmark_and_report/complete.log`.
|
||||
- Stable output contract: a deployed exact dev release plus one six-path qualification where every cell has successful product/harness/process and all four hard gates. Dependent task `17+16_full_nine_cell_scored_benchmark` may start only after this task has archived PASS evidence.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- Included: scoring eligibility, manifest timeout/tests, one diagnostic manifest/report, central dev deployment, and six-path live qualification.
|
||||
- Excluded: preserved run mutation, final C01-C09 allocation/scoring/reporting, fixture prompt/assets/rubric changes, caller/model substitution, credential changes, and speculative Edge/adapter edits. A remaining live defect must be owned by a reviewer-selected follow-up packet.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; `finalizer=finalize-task-policy.sh`, mode `pair`.
|
||||
- Build and review closures are all true: scope/context/verification/evidence/ownership/decision are closed by D14-D15, the exact dev runner, and a one-run qualification oracle.
|
||||
- Build scores: scope 2, state 2, blast 2, evidence 2, verification 2 => G10; base/route basis `grade-boundary`, cloud, `worker/cloud/G10`.
|
||||
- Review scores: 2/2/2/2/2 => G10; `official-review`, cloud, `review/cloud/G10`.
|
||||
- `large_indivisible_context=false`; matched risks: `temporal_state`, `boundary_contract`, `structured_interpretation`, `variant_product`; count 4. `review_rework_count=0`, `evidence_integrity_failure=false`.
|
||||
- Canonical files: `PLAN-cloud-G10.md`, `CODE_REVIEW-cloud-G10.md`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REPAIR-1] Separate hard artifact eligibility gates from quality-scoring gates and add fail-closed regression coverage.
|
||||
- [ ] [REPAIR-2] Set the compact benchmark and diagnostic per-cell ceiling to 300 seconds and lock both manifests in tests.
|
||||
- [ ] [REPAIR-3] Commit/push the exact feature, merge it into central dev, and complete the private dev-runtime deployment procedure with a safe restart and source/build identity proof.
|
||||
- [ ] [REPAIR-4] Run exactly one fresh non-scored six-path qualification and require all six caller/route paths to produce successful terminal evidence, generated files, screenshots, and all hard gates.
|
||||
- [ ] [REPAIR-5] Publish a sanitized deterministic recovery report without secrets or claims about the not-yet-run final benchmark.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REPAIR-1] Split eligibility from quality evidence
|
||||
|
||||
**Problem:** `scripts/agent_benchmark/scoring.py:1056-1067` appends the aggregate `web_failed` reason and every failed web gate, so a trustworthy page with only accessibility/responsive/console findings is unscored even though those dimensions belong to the rubric.
|
||||
|
||||
**Solution:** Replace aggregate web-status rejection with explicit hard-gate evaluation. Keep record-shape validation for all ordered gates; reject missing/invalid evidence and failures of `generated_files`, `static_safety`, `images`, or `network`; allow failures of `console`, `responsive`, and `accessibility` to reach the unchanged anonymous evaluator input.
|
||||
|
||||
Before (`scoring.py:1056`):
|
||||
|
||||
```python
|
||||
if web.status != "passed":
|
||||
reasons.append(f"web_{web.status}")
|
||||
reasons.extend(f"gate_{item['id']}" for item in gates if not item["passed"])
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```python
|
||||
hard_failures = [item for item in gates if item["id"] in HARD_ELIGIBILITY_GATES and not item["passed"]]
|
||||
reasons.extend(f"gate_{item['id']}" for item in hard_failures)
|
||||
```
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/agent_benchmark/scoring.py`: define the closed hard/quality partition and use only hard failures for eligibility.
|
||||
- [ ] `scripts/agent_benchmark/scoring_test.py`: prove each hard failure remains unscored, each quality-only failure remains score-eligible, mixed failures reject, and malformed/missing web evidence still fails closed.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Add table-driven regression coverage in `scripts/agent_benchmark/scoring_test.py` using its existing web-validation fixture helpers. No production run is used.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test
|
||||
```
|
||||
|
||||
Expected: all tests pass; quality-only failures invoke the evaluator, hard failures do not.
|
||||
|
||||
### [REPAIR-2] Lock the five-minute compact ceiling and diagnostic matrix
|
||||
|
||||
**Problem:** `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json:10` and `scripts/agent_benchmark/manifest_test.py:207-280` still pin 180 seconds after the user approved a 300-second ceiling. There is no isolated manifest for testing only the six previously failing caller/route paths without consuming the final run.
|
||||
|
||||
**Solution:** Change only `timeout.run_seconds` to 300 in the immutable full manifest. Add a recovery qualification manifest with the same fixture checksum, viewport/rubric/session/cache policies and only C01, C03, C04, C06, C07, C08 equivalents, a distinct seed, and `output_root=agent-test/runs/bench-02-recovery`. Lock its exact matrix and non-scored purpose in manifest tests.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`: set `run_seconds=300`; change nothing else.
|
||||
- [ ] `scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json`: add the closed six-path diagnostic manifest.
|
||||
- [ ] `scripts/agent_benchmark/manifest_test.py`: update the full-manifest timeout oracle and add exact diagnostic identity/matrix assertions.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Use parser round-trip tests to prove only timeout differs in the final manifest and that qualification cannot be confused with the final output root or nine-cell seed.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.manifest_test
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
```
|
||||
|
||||
Expected: tests and both validations pass.
|
||||
|
||||
### [REPAIR-3] Deploy the exact central dev baseline safely
|
||||
|
||||
**Problem:** the live dev runtime predates `origin/dev@6e2f4ff8`, while this feature already includes it. Running qualification against that process would repeat stale stream-liveness behavior.
|
||||
|
||||
**Solution:** Use `commit-push` to publish this task's exact feature SHA. Then follow `agent-ops/skills/private/dev-runtime-deploy/SKILL.md` completely: fetch the exact feature on the declared runner, merge it into `dev`, use the commit-count release, pass sequential pre/post tests, rebuild all four binaries, restart Edge then nodes, verify ports/snapshots/routes/capacity, finish only after every release gate, and atomically push main/dev/tag. Preserve secrets and ignored runtime configuration.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: record only sanitized release/source/build/process/port and qualification evidence.
|
||||
- [ ] Run the exact private deploy skill without editing central common rules, dispatcher code, or agent runtime code.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
The deployment skill's full pre/post sequential Go tests, binary provenance, config checks, listener/node/provider checks, and capacity smokes are mandatory. A failure keeps the release unfinished and blocks REPAIR-4.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com 'cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev'
|
||||
```
|
||||
|
||||
Expected after successful finish: clean `dev`, local HEAD equals `origin/dev`, and the recorded deployed binary source equals the finished release tag tree.
|
||||
|
||||
### [REPAIR-4] Qualify all previously failing caller/route families once
|
||||
|
||||
**Problem:** unit/preflight success does not prove official Claude/agy execution against the rebuilt runtime. Reusing the final manifest would spend the only authorized final cycle before recovery is proven.
|
||||
|
||||
**Solution:** With `/tmp/iop-bench-13-env` and the declared dev environment, append one all-cell preflight for the six-path diagnostic manifest and execute it exactly once. Do not score, resume, or retry it. Require `ready=6`, exactly six attempts, `product_succeeded=6`, `harness_passed=6`, `process_exited=6`, exit code zero for each, two screenshots per cell, and all four hard gates true. Quality gates may be false and must be recorded as quality evidence.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: record run id, terminal counts, hard/quality gate matrix, and contained raw-run link.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
One live qualification run only. Any failure is preserved and handed to official review; no manual workspace edits, provider substitution, resume, retry, or final benchmark start.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py preflight --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py run --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
```
|
||||
|
||||
Expected: one new qualification run; `ready=6`; all six terminals and hard artifacts satisfy the criteria above.
|
||||
|
||||
### [REPAIR-5] Publish bounded recovery evidence
|
||||
|
||||
**Problem:** without a deterministic summary, a later task could mistake a diagnostic run or a warning-only log for final benchmark completion.
|
||||
|
||||
**Solution:** Write `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md` with exact source/release/run identities, six rows, hard/quality gates, failures if any, and the explicit statement that it is non-scored and cannot satisfy D14.
|
||||
|
||||
#### Modified Files and Checklist
|
||||
|
||||
- [ ] `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md`: publish sanitized evidence and no secrets.
|
||||
|
||||
#### Test Strategy
|
||||
|
||||
Reviewer reconstructs all six rows from the exact run and verifies every relative link remains inside its run root.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
git diff --check -- . ':(exclude)agent-task/archive/**'
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: no whitespace errors; only declared task changes before commit/final review.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Archived predecessor `15+14_scored_benchmark_and_report` is satisfied by the exact `complete.log` cited above.
|
||||
2. REPAIR-1 and REPAIR-2 run before any central deployment.
|
||||
3. REPAIR-3 must fully finish before REPAIR-4.
|
||||
4. REPAIR-4 is exactly once. REPAIR-5 records it. A failed qualification blocks task completion and the dependent final benchmark.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `scripts/agent_benchmark/scoring.py` | REPAIR-1 |
|
||||
| `scripts/agent_benchmark/scoring_test.py` | REPAIR-1 |
|
||||
| `scripts/agent_benchmark/manifest_test.py` | REPAIR-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json` | REPAIR-2 |
|
||||
| `scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json` | REPAIR-2, REPAIR-4 |
|
||||
| `agent-test/dev/iop-one-shot-agent-comparison-recovery-2026-08-13.md` | REPAIR-3..5 |
|
||||
| `agent-task/m-iop-one-shot-agent-model-comparison/16+15_all_cell_measurement_recovery/CODE_REVIEW-cloud-G10.md` | REPAIR-1..5 |
|
||||
|
||||
## Final Verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.scoring_test scripts.agent_benchmark.manifest_test
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json
|
||||
/bin/bash /tmp/iop-bench-13-env python3 scripts/agent_comparison_benchmark.py status --manifest scripts/fixtures/agent-comparison-benchmark-recovery-qualification.json --run-id <EXACT_QUALIFICATION_RUN_ID>
|
||||
git diff --check -- . ':(exclude)agent-task/archive/**'
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: deterministic suite passes; final manifest remains nine cells with a 300-second ceiling; diagnostic manifest remains six cells; the one exact qualification has six successful product/harness/process outcomes, two screenshots and four hard gates per cell; no final nine-cell run was allocated.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
37
agent-test/dev/iop-benchmark-route-minimal-html-smoke.md
Normal file
37
agent-test/dev/iop-benchmark-route-minimal-html-smoke.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# IOP 벤치 경로 최소 HTML 스모크
|
||||
|
||||
## 목적
|
||||
|
||||
벤치 대상 9개 caller/model/route 조합에 같은 최소 `index.html` 생성 요청을 한 번씩 직접 보내 호출 경로만 빠르게 확인한다. 전용 runner, manifest, 자동 retry, browser gate와 품질 채점은 사용하지 않는다.
|
||||
|
||||
## 고정 요청
|
||||
|
||||
빈 임시 workspace에 외부 asset과 JavaScript 없이 단일 `index.html`을 만든다. 문서에는 `<!doctype html>`, `<title>IOP Route Smoke</title>`, `<h1>IOP_ROUTE_SMOKE_OK</h1>`이 정확히 한 번씩 있어야 하며 파일 생성 뒤 종료한다.
|
||||
|
||||
## 사전 확인 — 2026-08-13
|
||||
|
||||
| 항목 | 결과 |
|
||||
|---|---|
|
||||
| Claude Code | 2.1.228 확인 |
|
||||
| agy | 1.1.12 확인 |
|
||||
| Codex | 0.147.0 확인 |
|
||||
| managed CA | 파일 존재 확인 |
|
||||
| 기존 benchmark principal token | 파일 존재, mode `0600`, 65 bytes 확인 |
|
||||
| public `/v1/models` | HTTP 401 |
|
||||
| active public Edge token hash 매핑 | 0건 |
|
||||
| direct Edge `:18086` | 현재 host에서 연결 불가 |
|
||||
| model 호출 | 시작하지 않음 |
|
||||
|
||||
## 현재 분류
|
||||
|
||||
- 소유 경계: 환경/credential
|
||||
- 근거: caller를 시작하기 전 공통 인증 preflight에서 HTTP 401이 발생했고, 현재 public Edge 활성 config의 `principal_tokens`에 test token hash가 없다.
|
||||
- IOP 제품 결함 판정: 아직 아님
|
||||
- caller/model 결함 판정: 아직 아님
|
||||
- benchmark script 결함 판정: 해당 없음. script를 사용하지 않았다.
|
||||
|
||||
## 재개 조건
|
||||
|
||||
기존 direct·execution-preset route를 볼 수 있는 유효한 dev-corp principal token을 operator-private 경로에 준비하거나 기존 test token을 active Edge에 안전하게 매핑·재시작한다. token 원문을 출력하지 않은 `/v1/models` 확인이 HTTP 200이고 `claude-sonnet-5`, `gemini-3.6-flash`, `gpt-5.6-luna`, `gemini-hybrid`, `gpt-hybrid`가 존재하면 9개 최소 호출을 각 1회 시작한다.
|
||||
|
||||
성공한 경로는 반복하지 않는다. 실패한 경로는 원인이 변경된 경우에만 해당 경로를 1회 재검증한다.
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
# IOP 원샷 Agent 모델 비교 결과 — 2026-08-13
|
||||
|
||||
## 결론
|
||||
|
||||
`run-20260813T081326Z-4e1ac5152c6c`은 C01-C09 모두에 대해 한 번씩 실행한 terminal evidence를 보존했다. 최종 상태는 `unresolved=0`, `scored=2`, `unscored=7`, `scoring_failed=0`, `blocked=0`이다.
|
||||
|
||||
자동 웹 gate를 모두 통과해 익명 품질 채점이 가능했던 C05 GPT 단독과 C09 GPT 하이브리드는 각각 93점으로 공동 1위다. 나머지 7개 결과는 0점이 아니라 `unscored`이며 순위를 부여하지 않는다. 따라서 이 1회 실행만으로 caller 또는 model 전체의 우열을 일반화할 수 없다.
|
||||
|
||||
- 결정적 원본 보고서: [run report](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/report.md)
|
||||
- 이전 `run-20260813T071816Z-755d136c3e2b`은 fixture/scoring 진단 run으로만 보존하며 이 결과나 순위에 포함하지 않았다.
|
||||
- 최종 run은 9개 cell마다 producer attempt가 정확히 1개다. producer caller를 retry하거나 새 run으로 교체하지 않았다.
|
||||
|
||||
## 실행 조건
|
||||
|
||||
| 항목 | 값 |
|
||||
|---|---|
|
||||
| environment | `dev` |
|
||||
| pipeline version | `2` |
|
||||
| manifest digest | `sha256:38e48ef35beaa6ecc1aee0460df443e0ccf101a89aa6b91ce1f4b198aa84022a` |
|
||||
| fixture | `product-card-v2` (`sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e`) |
|
||||
| rubric | `one-shot-agent-comparison-v1` |
|
||||
| session/cache | `fresh` / `isolated` |
|
||||
| evaluator | `codex/gpt-5.6-luna/xhigh` |
|
||||
| execution preflight | sequence 1, `ready=9` |
|
||||
| repetition / timeout | cell별 1회 / cell별 180초 |
|
||||
|
||||
## 9개 결과
|
||||
|
||||
표의 순서는 immutable seed가 정한 실제 실행 순서다.
|
||||
|
||||
| 순서 | cell | 구성 | controller | product | harness | process | 자동 웹 gate | 채점 | 점수 | 순위 |
|
||||
|---:|---|---|---|---|---|---|---|---|---:|---:|
|
||||
| 1 | C02 | Claude Code → Gemini 단독 | completed | succeeded | passed | exited | accessibility 실패 | unscored | — | — |
|
||||
| 2 | C05 | Codex → GPT 단독 | completed | succeeded | passed | exited | 통과 | scored | 93 | 1 |
|
||||
| 3 | C03 | agy → Gemini 단독 | completed | failed | passed | exited | 산출물 없음 | unscored | — | — |
|
||||
| 4 | C06 | Claude Code → Gemini 하이브리드 | completed | failed | passed | exited | 산출물 없음 | unscored | — | — |
|
||||
| 5 | C08 | Claude Code → GPT 하이브리드 | completed | failed | passed | exited | 산출물 없음 | unscored | — | — |
|
||||
| 6 | C09 | Codex → GPT 하이브리드 | completed | succeeded | passed | exited | 통과 | scored | 93 | 1 |
|
||||
| 7 | C01 | Claude Code → Claude 단독 | completed | unknown | failed | exited | 산출물 없음 | unscored | — | — |
|
||||
| 8 | C07 | agy → Gemini 하이브리드 | timed_out | unknown | failed | timed_out | 산출물 없음 | unscored | — | — |
|
||||
| 9 | C04 | Claude Code → GPT 단독 | completed | failed | passed | exited | 산출물 없음 | unscored | — | — |
|
||||
|
||||
집계는 controller `completed=8`, `timed_out=1`; product `succeeded=3`, `failed=4`, `unknown=2`; harness `passed=7`, `failed=2`; process `exited=8`, `timed_out=1`; artifact `passed=2`, `failed=7`이다.
|
||||
|
||||
## 익명 품질 점수
|
||||
|
||||
| 항목 | 최대 | C05 GPT 단독 | C09 GPT 하이브리드 |
|
||||
|---|---:|---:|---:|
|
||||
| 요구사항 충족 | 25 | 24 | 23 |
|
||||
| 시각 완성도 | 25 | 24 | 24 |
|
||||
| 반응형·접근성 | 15 | 13 | 13 |
|
||||
| 이미지 활용·디테일 | 10 | 10 | 10 |
|
||||
| 동작 안정성 | 10 | 9 | 9 |
|
||||
| 코드 품질 | 10 | 9 | 9 |
|
||||
| 자체 검증 완결성 | 5 | 4 | 5 |
|
||||
| 합계 | 100 | 93 | 93 |
|
||||
|
||||
자동 웹 gate는 채점 eligibility만 결정하며 점수에 합산되지 않았다. C05는 `score-000002`, C09는 provenance 경계 수정 후 새로 할당한 `score-000003`의 결과다.
|
||||
|
||||
## 시간 비교
|
||||
|
||||
모든 시간은 harness monotonic clock 기준이며, first write는 workspace poll로 관측했다. `first output`과 `first write`는 제출 시점부터의 경과 시간이다.
|
||||
|
||||
| cell | 전체 시간 | first output | first write | 관측 결과 |
|
||||
|---|---:|---:|---:|---|
|
||||
| C01 | 65.048초 | 1.285초 | 미관측 | parser error 뒤 product unknown |
|
||||
| C02 | 127.718초 | 1.363초 | 95.441초 | product 성공, accessibility gate 실패 |
|
||||
| C03 | 17.661초 | 0.662초 | 미관측 | caller error |
|
||||
| C04 | 12.139초 | 1.069초 | 미관측 | caller error |
|
||||
| C05 | 150.131초 | 0.791초 | 미제공 (`observer_unavailable`) | 통과·채점 |
|
||||
| C06 | 24.394초 | 1.368초 | 미관측 | caller error |
|
||||
| C07 | 180.284초 | 0.699초 | 미관측 | 180초 cell timeout |
|
||||
| C08 | 16.606초 | 1.477초 | 미관측 | caller error |
|
||||
| C09 | 179.132초 | 0.785초 | 77.213초 | 통과·채점 |
|
||||
|
||||
180초는 전체 9개 matrix의 제한이 아니라 각 cell의 실행 제한이다. cell들은 순차 실행되므로 전체 matrix가 3분 안에 끝났다고 해석할 수 없다. 위 시간은 cell별 attempt 관측치이며 preflight, cell 사이 overhead, scoring과 report 시간을 포함한 run 전체 wall clock으로 합산하지 않는다.
|
||||
|
||||
## 호출과 token 비교
|
||||
|
||||
caller가 제공한 값만 기록한다. 미제공 값을 0으로 바꾸거나 서로 다른 tokenizer의 값을 합산하지 않았다.
|
||||
|
||||
| cell | model calls | tool calls | input | cached input | cache write | output | reasoning | total |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| C02 | 12 | 미제공 | 182,484 | 97,822 | 0 | 3,701 | 미제공 | 미제공 |
|
||||
| C05 | 1 | 9 | 248,818 | 217,034 | 31,751 | 20,721 | 5,156 | 미제공 |
|
||||
| C09 | 1 | 7 | 247,278 | 218,156 | 28,649 | 12,134 | 4,317 | 미제공 |
|
||||
| C01, C03, C04, C06, C07, C08 | 미제공 | 미제공 | 미제공 | 미제공 | 미제공 | 미제공 | 미제공 | 미제공 |
|
||||
|
||||
C02의 caller-reported model duration은 116.081초, caller-reported total duration은 116.760초다. 이 구간은 harness 전체 시간과 clock/source가 다르고 중첩되므로 임의로 더하거나 overhead로 분해하지 않았다. 나머지 cell의 model/tool/queue duration은 미제공이다.
|
||||
|
||||
## 실패와 한계
|
||||
|
||||
- C02는 product가 성공하고 두 viewport screenshot도 생성했지만 accessibility gate가 실패해 unscored다.
|
||||
- C03, C04, C06, C08은 caller error와 non-zero exit 뒤 필수 산출물이 없어 unscored다.
|
||||
- C01은 parser error로 product가 unknown이고 harness가 실패했으며, C07은 180초 timeout으로 product가 unknown이다.
|
||||
- C05와 C09만 모든 자동 gate를 통과했다. 두 결과의 동점은 이 두 eligible artifact 사이의 1회 평가 결과일 뿐, 실패한 7개 조합의 품질이 0점이라는 의미가 아니다.
|
||||
- provider/caller가 제공하지 않은 timing·usage는 `미제공`으로 유지했다. stage별, queue, tool duration과 total token은 비교할 수 없다.
|
||||
- scored 결과는 각 1회뿐이므로 분산, 재현성, 비용 일반화나 통계적 유의성을 제공하지 않는다.
|
||||
|
||||
## Raw evidence
|
||||
|
||||
모든 링크는 preserved run 내부만 가리킨다.
|
||||
|
||||
| cell | 실행 | 측정 | 웹 검증 | 채점 | screenshot |
|
||||
|---|---|---|---|---|---|
|
||||
| C01 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c01-claude-sonnet-direct/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c01-claude-sonnet-direct/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c01-claude-sonnet-direct/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c01-claude-sonnet-direct/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C02 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/scoring/unscored.json) | [desktop](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/screenshot-desktop_1080.png), [mobile](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c02-claude-gemini-direct/repetition-0001/attempt-000001/screenshot-mobile_375.png) |
|
||||
| C03 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c03-agy-gemini-direct/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c03-agy-gemini-direct/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c03-agy-gemini-direct/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c03-agy-gemini-direct/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C04 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c04-claude-gpt-direct/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c04-claude-gpt-direct/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c04-claude-gpt-direct/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c04-claude-gpt-direct/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C05 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/web-validation.json) | [score-000002](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/scoring/score-000002/result.json) | [desktop](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/screenshot-desktop_1080.png), [mobile](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c05-codex-gpt-direct/repetition-0001/attempt-000001/screenshot-mobile_375.png) |
|
||||
| C06 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c06-claude-gemini-hybrid/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c06-claude-gemini-hybrid/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c06-claude-gemini-hybrid/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c06-claude-gemini-hybrid/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C07 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c07-agy-gemini-hybrid/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c07-agy-gemini-hybrid/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c07-agy-gemini-hybrid/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c07-agy-gemini-hybrid/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C08 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c08-claude-gpt-hybrid/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c08-claude-gpt-hybrid/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c08-claude-gpt-hybrid/repetition-0001/attempt-000001/web-validation.json) | [unscored](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c08-claude-gpt-hybrid/repetition-0001/attempt-000001/scoring/unscored.json) | — |
|
||||
| C09 | [attempt](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/attempt.json) | [measurement](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/attempt-measurement.json) | [web](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/web-validation.json) | [score-000003](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/scoring/score-000003/result.json) | [desktop](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/screenshot-desktop_1080.png), [mobile](../runs/bench-02/run-20260813T081326Z-4e1ac5152c6c/cells/c09-codex-gpt-hybrid/repetition-0001/attempt-000001/screenshot-mobile_375.png) |
|
||||
|
|
@ -70,7 +70,7 @@ model:
|
|||
aliases:
|
||||
"claude-sonnet-5":
|
||||
observed_at: "2026-08-10"
|
||||
status: active_edge_model_group_benchmark_target_observed
|
||||
status: active_edge_model_group_observed
|
||||
display_name: Claude Sonnet 5
|
||||
capacity_total: 1
|
||||
providers:
|
||||
|
|
@ -90,7 +90,7 @@ model:
|
|||
provider_snapshot: healthy_idle
|
||||
"gpt-5.6-luna":
|
||||
observed_at: "2026-08-10"
|
||||
status: active_edge_model_group_benchmark_target_observed
|
||||
status: active_edge_model_group_observed
|
||||
display_name: GPT-5.6 Luna
|
||||
capacity_total: 1
|
||||
providers:
|
||||
|
|
|
|||
|
|
@ -1,583 +0,0 @@
|
|||
# Agent Comparison Benchmark Dev Guide
|
||||
|
||||
이 문서는 IOP one-shot agent/model comparison benchmark의 현재 구현, dev 환경 구성, caller 연결 방식, managed credential 경계, 실행 절차와 장애 대응 기준을 한곳에 모은 운영 가이드다.
|
||||
|
||||
raw token, provider API key, private key, slot alias, lease id와 개인 endpoint는 이 문서에 기록하지 않는다. 실제 host, checkout, Node/provider endpoint와 최신 process 상태는 아래 source of truth에서 확인한다.
|
||||
|
||||
- benchmark manifest: `scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json`
|
||||
- public CLI: `scripts/agent_comparison_benchmark.py`
|
||||
- dev environment: `agent-test/dev/rules.md`
|
||||
- machine-readable dev inventory: `agent-test/inventory-dev.yaml`
|
||||
- Edge/Node verification: `agent-test/dev/edge-smoke.md`, `agent-test/dev/node-smoke.md`
|
||||
- API contracts: `agent-contract/outer/anthropic-compatible-api.md`, `agent-contract/outer/openai-compatible-api.md`, `agent-contract/outer/gemini-compatible-api.md`
|
||||
- benchmark SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md`
|
||||
|
||||
이 문서는 위 계약과 manifest를 설명할 뿐 덮어쓰지 않는다. 값이 다르면 manifest, contract, inventory, environment rule 순서로 최신 상태를 확인한다.
|
||||
|
||||
## 1. 현재 상태
|
||||
|
||||
2026-08-12 기준 상태는 다음과 같다.
|
||||
|
||||
| 영역 | 상태 |
|
||||
|---|---|
|
||||
| Gemini-native Edge ingress | 구현 완료. route-qualified `streamGenerateContent`, `x-goog-api-key` IOP principal auth, request/tool/SSE 변환을 지원한다. |
|
||||
| official agy adapter | 구현 완료. `agy 1.1.12`, Gemini API-key provider, real `init/step_update/result` JSONL을 사용한다. |
|
||||
| managed dev credential runtime | 구성 완료. Control Plane projection, credential slot/route, sealed lease, Edge HTTPS, CP↔Edge/Edge↔Node mTLS를 사용한다. |
|
||||
| caller readiness | 최근 완료된 readiness evidence에서 C01-C09 `ready=9`를 확인했다. 실제 실행일에는 fresh preflight가 다시 필요하다. |
|
||||
| controller recovery | control socket symlink와 caller launch 전 interruption 회귀 수정 및 테스트가 완료됐다. |
|
||||
| Codex production JSONL | `cache_write_input_tokens` 수용과 config-owned effective binding 정합화가 완료됐다. |
|
||||
| deterministic tests | focused 41 tests, 전체 benchmark 429 tests가 통과했다. 숫자는 당시 snapshot이며 현재 suite 결과는 fresh 실행으로 판단한다. |
|
||||
| C01-C09 scored execution | 아직 완료되지 않았다. 과거 incomplete/failed run은 append-only evidence로 보존하며 정상 결과로 간주하지 않는다. |
|
||||
| blind scoring/report | 유효한 9-cell execution이 생긴 뒤 수행하는 후속 단계다. |
|
||||
|
||||
과거 실행 실패는 새 run을 정당화하는 완료 evidence가 아니다. 이전 run tree를 수정하거나 old run을 암묵적으로 `resume`/retry하지 않는다. 새 scored execution은 그 실행을 명시적으로 소유하는 현재 plan과 사용자 권한이 있을 때만 한 번 수행한다. 이 문서 자체는 지속적 실행 승인을 부여하지 않는다.
|
||||
|
||||
## 2. 시스템 구성
|
||||
|
||||
```text
|
||||
Benchmark runner
|
||||
├─ Claude Code ── Anthropic Messages ───────────────┐
|
||||
├─ agy ───────── Gemini streamGenerateContent ─────┤
|
||||
└─ Codex ─────── OpenAI Responses ─────────────────┤
|
||||
v
|
||||
IOP Edge HTTPS
|
||||
│
|
||||
┌───────────────────────────┴───────────────────────────┐
|
||||
│ │
|
||||
direct model route execution preset
|
||||
│ selector/plan/work/review/repair
|
||||
└───────────────────────────┬───────────────────────────┘
|
||||
v
|
||||
managed credential lease
|
||||
Control Plane projection + Node sealed lease
|
||||
│
|
||||
v
|
||||
Node-owned provider
|
||||
```
|
||||
|
||||
핵심 보안 경계는 다음과 같다.
|
||||
|
||||
- benchmark caller는 하나의 IOP principal token으로 Edge만 인증한다.
|
||||
- caller token은 upstream provider credential이 아니다.
|
||||
- provider credential은 Control Plane에 등록된 slot과 Node 대상 sealed lease에서만 온다.
|
||||
- Edge는 provider credential을 복호화하지 않고 caller가 보낸 token을 upstream auth로 재사용하지 않는다.
|
||||
- managed mode는 legacy static principal/provider credential과 혼용하지 않는다.
|
||||
- config observation은 secret이 아닌 route/model/stage binding의 독립 증거다.
|
||||
|
||||
## 3. 환경 프로필
|
||||
|
||||
### 3.1 Benchmark runner
|
||||
|
||||
현재 검증된 runner class는 Linux/AArch64다. 명령은 repository root에서 실행한다.
|
||||
|
||||
필수 command:
|
||||
|
||||
```bash
|
||||
command -v python3
|
||||
command -v git
|
||||
command -v claude
|
||||
command -v agy
|
||||
command -v codex
|
||||
```
|
||||
|
||||
2026-08-12 확인 snapshot:
|
||||
|
||||
| Tool | 확인된 버전 | 정책 |
|
||||
|---|---|---|
|
||||
| Claude Code | `2.1.228` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`/`--help`를 확인한다. |
|
||||
| agy | `1.1.12` | adapter가 이 버전을 명시적으로 gate한다. 다른 버전은 재검증 전 fail closed한다. |
|
||||
| Codex CLI | `0.147.0` | 고정 버전으로 추정하지 않고 매 execution preflight에서 `--version`과 `exec --help`를 확인한다. |
|
||||
|
||||
### 3.2 Testbed
|
||||
|
||||
- path: `../iop-s2`
|
||||
- 현재 확인 branch: `dev`
|
||||
- 현재 확인 상태: clean
|
||||
- benchmark는 testbed를 read-only provenance로 취급한다.
|
||||
- caller별 workspace와 session은 run tree 아래에 새로 만들며 서로 공유하지 않는다.
|
||||
- testbed를 benchmark 결과로 수정하거나 결과 파일을 다시 복사하지 않는다.
|
||||
|
||||
fixture checksum과 source file 목록은 manifest가 고정한다. testbed HEAD와 clean 상태는 실행일에 다시 확인한다.
|
||||
|
||||
### 3.3 Dev runtime
|
||||
|
||||
현재 검증된 runtime class는 macOS/ARM64 remote dev runner다. exact SSH target과 checkout은 `agent-test/dev/rules.md` 및 `agent-test/inventory-dev.yaml`을 따른다.
|
||||
|
||||
benchmark 관련 runtime 역할:
|
||||
|
||||
| Port | 역할 |
|
||||
|---:|---|
|
||||
| `18082` | Edge artifact/bootstrap HTTP |
|
||||
| `18083` | managed Edge public HTTPS; Anthropic/OpenAI/Gemini caller ingress |
|
||||
| `18084` | native dev-runtime Edge↔Node TCP |
|
||||
| `19093` | Edge admin/config refresh |
|
||||
| `19101` | Edge metrics |
|
||||
|
||||
2026-08-12 read-only 확인에서 위 listener와 managed Edge process는 모두 active였고 remote checkout은 clean release 상태였다. exact commit, binary checksum, process id와 endpoint는 실행 evidence에만 기록하고 이 가이드에 고정하지 않는다.
|
||||
|
||||
최근 완료된 live readiness evidence는 4 connected Nodes와 8 healthy/available providers를 확인했다. 최신 Node/provider 세부와 접속 위치는 반드시 `agent-test/inventory-dev.yaml`에서 다시 확인한다.
|
||||
|
||||
이 benchmark는 compose dev stack의 Edge-Node TCP `19003`이 아니라 native dev-runtime provider pool의 `18084`를 사용한다. compose와 native profile은 포트, process와 판정 evidence가 서로 다르므로 한 실행에서 섞지 않는다. 현재 inventory가 가리키는 배포 산출물은 다음과 같다.
|
||||
|
||||
| Artifact | Path |
|
||||
|---|---|
|
||||
| native Edge config | `build/dev-runtime/edge.yaml` |
|
||||
| Edge binary | `build/dev-runtime/bin/edge` |
|
||||
| macOS Node binary | `build/dev-runtime/bin/iop-node` |
|
||||
| Linux ARM64 Node binary | `build/dev-runtime/bin/iop-node-linux-arm64` |
|
||||
| Windows AMD64 Node binary | `build/dev-runtime/bin/iop-node-windows-amd64.exe` |
|
||||
|
||||
모든 Edge/Node binary는 scored execution 전에 동일 source ref로 rebuild·redeploy·restart한다. `build/dev-runtime/**`의 runtime config와 untracked credential material은 원격 runner가 소유하며 tracked 문서나 testbed로 복사하지 않는다.
|
||||
|
||||
## 4. 보호 파일과 credential 역할
|
||||
|
||||
benchmark runner의 `token/` 아래에는 다음 파일이 준비돼 있다. 파일 존재와 mode만 확인하며 내용을 출력하지 않는다.
|
||||
|
||||
| Path | 역할 | 실행 시 사용 |
|
||||
|---|---|---|
|
||||
| `token/.iop-bench` | benchmark용 IOP principal token | preflight/run/score caller가 Edge를 인증할 때 사용 |
|
||||
| `token/iop-dev-ca.pem` | managed dev Edge HTTPS CA certificate | `SSL_CERT_FILE`, `NODE_EXTRA_CA_CERTS`로 전달 |
|
||||
| `token/.claude` | Claude provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
| `token/.gemini` | Gemini provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
| `token/.gpt` | GPT provider credential의 초기 provisioning source | provider slot 등록 시에만 사용; benchmark caller에 전달하지 않음 |
|
||||
|
||||
현재 secret source 파일은 `0600`, CA certificate는 `0644`로 확인됐다. CA certificate는 public trust material이지만 private key는 아니다.
|
||||
|
||||
안전 확인:
|
||||
|
||||
```bash
|
||||
for benchmark_secret_file in token/.iop-bench token/.claude token/.gemini token/.gpt; do
|
||||
test -f "$benchmark_secret_file"
|
||||
test "$(stat -c '%a' "$benchmark_secret_file")" = 600
|
||||
done
|
||||
test -f token/iop-dev-ca.pem
|
||||
```
|
||||
|
||||
macOS에서 동일 검사를 수행할 때는 BSD `stat` 문법을 사용한다. 어떤 경우에도 `cat`, `echo`, shell tracing(`set -x`)으로 secret 내용을 출력하지 않는다.
|
||||
|
||||
## 5. Provider와 managed credential 설정
|
||||
|
||||
현재 dev 구성은 다음 절차로 만들어졌다.
|
||||
|
||||
1. Control Plane credential plane용 CA, workload certificate, at-rest keyring과 lease issuer/recipient key를 operator-owned untracked 경로에 생성했다.
|
||||
2. Control Plane, Edge와 각 Node에 role/name-bound mTLS identity를 배치했다.
|
||||
3. Edge public ingress를 HTTPS로 구성하고 benchmark runner에 CA certificate만 전달했다.
|
||||
4. benchmark principal을 bootstrap하고 one-time token을 `token/.iop-bench`에 저장했다.
|
||||
5. `token/.claude`, `token/.gemini`, `token/.gpt`의 raw provider key를 credential HTTPS request body로 직접 등록했다. command argument, YAML, tracked docs나 task evidence에는 넣지 않았다.
|
||||
6. credential slot과 public route를 별도로 생성하고 principal projection에 direct route와 hybrid preset stage authorization을 연결했다.
|
||||
7. Control Plane → Edge → Nodes 순서로 bounded restart하고 fresh projection, sealed lease, route revision과 no-fallback 동작을 검증했다.
|
||||
|
||||
재구성이 필요하면 `docs/edge-local-dev-guide.md`의 “Managed credential plane and TLS startup”과 “Safe slot lifecycle”을 따른다. 실제 slot id, alias, revision과 lease id는 운영 상태이므로 이 문서에 복사하지 않는다.
|
||||
|
||||
## 6. Caller별 연결 방식
|
||||
|
||||
| Caller | Edge surface | Child 설정 | 중요한 제한 |
|
||||
|---|---|---|---|
|
||||
| Claude Code | Anthropic-compatible Messages | `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`; `--bare --print --verbose --output-format stream-json --no-session-persistence --permission-mode dontAsk --tools Read,Write,Edit --allowedTools Read,Write,Edit` | task는 stdin으로 한 번 제출한다. network/shell 도구 없이 격리 workspace 파일 작업만 허용한다. model/effort는 manifest 값을 그대로 전달한다. |
|
||||
| agy | Gemini-native `streamGenerateContent` | fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`; `--sandbox --output-format stream-json --model ... --print ...` | `agy 1.1.12`만 승인된다. API-key provider에 `--effort`를 전달하지 않으며 ambient user config를 읽지 않는다. |
|
||||
| Codex | OpenAI-compatible Responses | fresh session `HOME`, isolated `iop_benchmark` provider override, `IOP_BENCHMARK_API_KEY`; `exec --sandbox workspace-write --json --ephemeral --ignore-user-config --strict-config` | user config를 읽지 않고 격리 workspace에만 쓸 수 있다. current adapter effort는 `xhigh`; caller binding event가 없으면 admitted config binding을 사용하고, 보고된 mismatch는 거부한다. |
|
||||
|
||||
세 caller child에는 필요한 `PATH`와 CA 변수만 allowlist로 전달한다. parent의 caller/provider 설정이나 unrelated secret은 상속하지 않는다.
|
||||
|
||||
agy의 public base는 adapter가 cell별로 다음처럼 route-qualified 한다.
|
||||
|
||||
```text
|
||||
<edge-origin>/gemini/<route-id>
|
||||
```
|
||||
|
||||
`GEMINI_BASE_URL`, `AGY_PROVIDER`, `AGY_OPENAI_BASE_URL`, `AGY_OPENAI_API_KEY`는 이 benchmark transport가 아니다.
|
||||
|
||||
official agy planner가 Gemini `generationConfig.responseMimeType`과 `responseSchema` 또는 `responseJsonSchema`를 보내면 Edge는 이를 기존 Chat `response_format`으로 변환한다. function declaration은 official SDK가 사용하는 `parameters`와 `parametersJsonSchema` 표기 중 정확히 하나를 허용한다. 동의어 필드가 동시에 있거나 schema가 JSON object가 아니면 provider dispatch 전에 거부한다. 구조가 유효한 agy `result.status=ERROR`는 stream parser 오류로 바꾸지 않고 caller process 실패로 기록한다.
|
||||
|
||||
## 7. Benchmark 고정 설정
|
||||
|
||||
| 설정 | 값 |
|
||||
|---|---|
|
||||
| pipeline version | `2` |
|
||||
| environment | `dev` |
|
||||
| execution seed | `bench-02-c01-c09-v1` |
|
||||
| repetitions | `1` |
|
||||
| session policy | `fresh` |
|
||||
| setup cache policy | `isolated` |
|
||||
| run timeout | 300 seconds |
|
||||
| idle timeout | 30 seconds |
|
||||
| quiet window | 10 seconds |
|
||||
| cleanup grace | 5 seconds |
|
||||
| desktop viewport | `1920x1080` |
|
||||
| mobile viewport | `375x812` |
|
||||
| output root | `agent-test/runs/bench-02` |
|
||||
| rubric | `one-shot-agent-comparison-v1` |
|
||||
|
||||
execution seed로 결정되는 현재 slot 순서는 다음과 같다. 표의 C 번호 순서와 실제 실행 순서는 다르다.
|
||||
|
||||
1. C02 Claude→Gemini direct
|
||||
2. C05 Codex→GPT direct
|
||||
3. C03 agy→Gemini direct
|
||||
4. C06 Claude→Gemini hybrid
|
||||
5. C08 Claude→GPT hybrid
|
||||
6. C09 Codex→GPT hybrid
|
||||
7. C01 Claude→Sonnet direct
|
||||
8. C07 agy→Gemini hybrid
|
||||
9. C04 Claude→GPT direct
|
||||
|
||||
## 8. C01-C09 matrix와 route binding
|
||||
|
||||
| Cell | Caller | Route kind/id | Requested model/effort | Effective stage binding |
|
||||
|---|---|---|---|---|
|
||||
| C01 | Claude | direct / `claude-sonnet-5` | `claude-sonnet-5` / `max` | request=`claude-sonnet-5` max |
|
||||
| C02 | Claude | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
|
||||
| C03 | agy | direct / `gemini-3.6-flash` | `gemini-3.6-flash` / `high` | request=`gemini-3.6-flash` high |
|
||||
| C04 | Claude | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
|
||||
| C05 | Codex | direct / `gpt-5.6-luna` | `gpt-5.6-luna` / `xhigh` | request=`gpt-5.6-luna` xhigh |
|
||||
| C06 | Claude | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
|
||||
| C07 | agy | preset / `gemini-hybrid` | `gemini-hybrid` / `high` | selector/plan/review/repair=`gemini-3.6-flash` high; work=`ornith-fast` |
|
||||
| C08 | Claude | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
|
||||
| C09 | Codex | preset / `gpt-hybrid` | `gpt-hybrid` / `xhigh` | selector/plan/review/repair=`gpt-5.6-terra` high; work=`ornith-fast` |
|
||||
|
||||
hybrid preset은 caller가 stage를 따로 호출하는 구조가 아니다. 하나의 caller request 안에서 Edge가 selector/plan/work/review/repair를 소유한다.
|
||||
|
||||
## 9. Fixture와 결과 조건
|
||||
|
||||
공통 task는 fictional product “Lumen Atlas”의 responsive one-page landing page다.
|
||||
|
||||
- prompt: `scripts/fixtures/agent-comparison-benchmark/prompt.md`
|
||||
- copy: `scripts/fixtures/agent-comparison-benchmark/reference.txt`
|
||||
- images: `aurora-grid.svg`, `orbit-rings.svg`
|
||||
- 생성 파일: workspace root의 `index.html`, `styles.css`, `script.js` 정확히 세 개
|
||||
- 외부 asset, framework, package manager, build tool, analytics와 network dependency 금지
|
||||
- desktop/mobile responsive, semantic HTML, focus/contrast/accessibility 요구
|
||||
- 각 attempt는 fresh workspace/session에서 task를 한 번만 제출
|
||||
|
||||
fixture checksum은 manifest의 값이 유일한 기준이다. prompt나 asset을 변경하면 기존 run과 비교하지 말고 manifest/version/checksum을 함께 갱신하는 별도 작업으로 처리한다.
|
||||
|
||||
## 10. Process environment 준비
|
||||
|
||||
다음은 value를 출력하지 않는 process-local 예시다. `<edge-host>`를 문서에 실제 값으로 치환하지 말고 실행 환경에서만 주입한다.
|
||||
|
||||
Public live registry가 소비하는 environment contract는 다음과 같다.
|
||||
|
||||
| Variable | 값/의미 | Durable evidence |
|
||||
|---|---|---|
|
||||
| `IOP_BENCH_CLAUDE_BASE_URL` | managed Edge HTTPS origin | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_AGY_BASE_URL` | managed Edge HTTPS origin; adapter가 `/gemini/<route-id>`를 추가 | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_CODEX_BASE_URL` | managed Edge OpenAI-compatible `/v1` base | raw 값 금지; endpoint digest만 허용 |
|
||||
| `IOP_BENCH_CLAUDE_SECRET_ENV` | Claude가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_AGY_SECRET_ENV` | agy가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_CODEX_SECRET_ENV` | Codex가 사용할 secret-bearing variable 이름 | variable 이름만 허용 |
|
||||
| `IOP_BENCH_SHARED_TOKEN` | 이 가이드 예시의 secret-bearing variable | 값 기록 금지 |
|
||||
| `SSL_CERT_FILE` | dev Edge HTTPS CA certificate path | repository-relative file reference만 허용 |
|
||||
| `NODE_EXTRA_CA_CERTS` | Node.js caller용 동일 CA certificate path | repository-relative file reference만 허용 |
|
||||
| `IOP_BENCH_CONFIG_OBSERVATION_ENV` | config JSON을 보유한 variable 이름 | variable 이름만 허용 |
|
||||
| `BENCH_CONFIG` | schema v1 route/model/stage observation JSON | secret은 없지만 runtime과 일치하는 canonical digest만 evidence에 기록 |
|
||||
| `PATH` | caller binary resolution | resolved executable path/version만 preflight에서 확인 |
|
||||
|
||||
Child adapter가 내부적으로 만드는 값은 caller별로 격리된다.
|
||||
|
||||
- Claude: `ANTHROPIC_BASE_URL`, `ANTHROPIC_API_KEY`, restricted `Read,Write,Edit`, nonessential traffic/autoupdater disable flags와 CA variables
|
||||
- agy: fresh session `HOME`, `GOOGLE_GEMINI_BASE_URL`, `GEMINI_API_KEY`, `LANG=C.UTF-8`, `LC_ALL=C.UTF-8`, `TZ=UTC`와 CA variables
|
||||
- Codex: fresh session `HOME`, `IOP_BENCHMARK_API_KEY`, strict ephemeral provider override와 CA variables
|
||||
|
||||
이 child variable은 사용자가 별도로 준비할 값이 아니다. live registry가 위 public contract에서 파생하며, parent의 같은 이름 값을 그대로 신뢰하거나 상속하지 않는다.
|
||||
|
||||
```bash
|
||||
set -euo pipefail
|
||||
|
||||
benchmark_edge_origin="${BENCHMARK_EDGE_ORIGIN:?set BENCHMARK_EDGE_ORIGIN to the managed dev Edge HTTPS origin}"
|
||||
read -r IOP_BENCH_SHARED_TOKEN < token/.iop-bench
|
||||
export IOP_BENCH_SHARED_TOKEN
|
||||
|
||||
export IOP_BENCH_CLAUDE_BASE_URL="$benchmark_edge_origin"
|
||||
export IOP_BENCH_AGY_BASE_URL="$benchmark_edge_origin"
|
||||
export IOP_BENCH_CODEX_BASE_URL="$benchmark_edge_origin/v1"
|
||||
|
||||
export IOP_BENCH_CLAUDE_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_AGY_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
export IOP_BENCH_CODEX_SECRET_ENV=IOP_BENCH_SHARED_TOKEN
|
||||
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
|
||||
export IOP_BENCH_CONFIG_OBSERVATION_ENV=BENCH_CONFIG
|
||||
BENCH_CONFIG="$(python3 - <<'PY'
|
||||
import json
|
||||
|
||||
routes = [
|
||||
{"route_kind":"direct","route_id":"claude-sonnet-5","model":"claude-sonnet-5","bindings":[{"stage":"request","model":"claude-sonnet-5","effort":"max"}]},
|
||||
{"route_kind":"direct","route_id":"gemini-3.6-flash","model":"gemini-3.6-flash","bindings":[{"stage":"request","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"direct","route_id":"gpt-5.6-luna","model":"gpt-5.6-luna","bindings":[{"stage":"request","model":"gpt-5.6-luna","effort":"xhigh"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gemini-hybrid","model":"gemini-hybrid","bindings":[{"stage":"selector","model":"gemini-3.6-flash","effort":"high"},{"stage":"plan","model":"gemini-3.6-flash","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gemini-3.6-flash","effort":"high"},{"stage":"repair","model":"gemini-3.6-flash","effort":"high"}]},
|
||||
{"route_kind":"execution_preset","route_id":"gpt-hybrid","model":"gpt-hybrid","bindings":[{"stage":"selector","model":"gpt-5.6-terra","effort":"high"},{"stage":"plan","model":"gpt-5.6-terra","effort":"high"},{"stage":"work","model":"ornith-fast","effort":None},{"stage":"review","model":"gpt-5.6-terra","effort":"high"},{"stage":"repair","model":"gpt-5.6-terra","effort":"high"}]},
|
||||
]
|
||||
print(json.dumps({"schema_version":"1","routes":routes}, separators=(",",":")))
|
||||
PY
|
||||
)"
|
||||
export BENCH_CONFIG
|
||||
```
|
||||
|
||||
중요한 의미:
|
||||
|
||||
- `IOP_BENCH_*_SECRET_ENV`의 값은 secret이 아니라 실제 secret을 보유한 environment variable의 이름이다.
|
||||
- 세 caller는 같은 IOP principal을 사용하지만 서로 다른 protocol base를 받는다.
|
||||
- `BENCH_CONFIG`는 runtime에서 독립적으로 확인한 route/stage snapshot이어야 한다. manifest를 보고 임의 합성한 값을 live readiness evidence로 사용하면 안 된다.
|
||||
- endpoint와 config는 evidence에 raw 값 대신 digest identity로만 남는다.
|
||||
|
||||
작업 후에는 같은 shell에서 다음 변수를 제거한다.
|
||||
|
||||
```bash
|
||||
unset IOP_BENCH_SHARED_TOKEN
|
||||
unset IOP_BENCH_CLAUDE_BASE_URL IOP_BENCH_AGY_BASE_URL IOP_BENCH_CODEX_BASE_URL
|
||||
unset IOP_BENCH_CLAUDE_SECRET_ENV IOP_BENCH_AGY_SECRET_ENV IOP_BENCH_CODEX_SECRET_ENV
|
||||
unset IOP_BENCH_CONFIG_OBSERVATION_ENV BENCH_CONFIG
|
||||
unset SSL_CERT_FILE NODE_EXTRA_CA_CERTS
|
||||
unset BENCHMARK_EDGE_ORIGIN benchmark_edge_origin
|
||||
```
|
||||
|
||||
## 11. 실행 절차
|
||||
|
||||
manifest path는 모든 명령에서 동일하게 사용한다.
|
||||
|
||||
```bash
|
||||
benchmark_manifest=scripts/fixtures/agent-comparison-benchmark-iop-one-shot.json
|
||||
```
|
||||
|
||||
### 11.1 Source와 deterministic verification
|
||||
|
||||
```bash
|
||||
python3 -m unittest scripts.agent_benchmark.codex_iop_test scripts.agent_benchmark.connectivity_integration_test
|
||||
python3 -m unittest discover -s scripts/agent_benchmark -p '*_test.py'
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest "$benchmark_manifest"
|
||||
git diff --check
|
||||
```
|
||||
|
||||
필요하면 변경 범위에 따라 Go tests와 managed credential qualification도 실행한다.
|
||||
|
||||
```bash
|
||||
go test -count=1 ./...
|
||||
credential_smoke_parent="$(mktemp -d /tmp/iop-benchmark-credential.XXXXXX)"
|
||||
TMPDIR="$credential_smoke_parent" make test-credential-slot-smoke
|
||||
rmdir "$credential_smoke_parent"
|
||||
```
|
||||
|
||||
### 11.2 Direct-first qualification
|
||||
|
||||
동일 clean source ref를 모든 runtime binary에 배포하고 4/4 Node와 provider health를 확인한 다음, 먼저 기존 5-cell direct manifest를 사용한다.
|
||||
|
||||
```bash
|
||||
direct_manifest="scripts/fixtures/agent-comparison-benchmark-direct-preflight.example.json"
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest "$direct_manifest"
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest "$direct_manifest"
|
||||
```
|
||||
|
||||
direct canary는 unscored 진단이다. fresh preflight가 `ready=5`여야 하고, 정확히 5개의 fresh attempt를 한 번씩 할당하며, `unresolved=0`, `running=0`, `interrupted=0`과 함께 각 slot의 controller/product/harness/process/web-validation terminal evidence가 있어야 한다. 최대 3회 Chromium/CDP 재시도 뒤에도 `browser_*` 또는 `cdp_*` infrastructure block이 남거나 Edge pre-ingress incompatibility가 확인되면 qualification을 중단한다.
|
||||
|
||||
`product=failed`, upstream HTTP rejection, caller failure 뒤 `generated_missing`, timeout은 terminal evidence가 완결되면 측정 결과다. 이를 성공으로 바꾸거나 암묵 재시도하지 않으며, 그 자체로 다음 구현 cycle을 만들지 않는다. scoring eligibility는 각 독립 gate로 별도 판정한다.
|
||||
|
||||
### 11.3 Public nine-cell preflight
|
||||
|
||||
10절의 환경을 같은 shell에 준비한 뒤 실행한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest "$benchmark_manifest"
|
||||
```
|
||||
|
||||
성공 조건:
|
||||
|
||||
```text
|
||||
status=ready ready=9 registration_required=0 implementation_gap=0
|
||||
```
|
||||
|
||||
preflight는 다음을 함께 확인한다.
|
||||
|
||||
- caller binary/version/help
|
||||
- Edge model catalog
|
||||
- principal auth와 endpoint compatibility
|
||||
- direct/preset route 존재
|
||||
- requested model/effort
|
||||
- exact stage binding과 order
|
||||
- official agy transport capability
|
||||
|
||||
`registration_required` 또는 `implementation_gap`이면 즉시 중단한다. alias, model, effort, route나 caller를 대체하지 않는다. preflight-only run root는 evidence이므로 삭제하지 않는다.
|
||||
|
||||
terminal-evidence direct canary admission 뒤 fresh one-shot preflight가 `ready=9`인지 확인하고 멈춘다. 이 qualification 단계에서는 hybrid canary나 C01-C09 `run`을 호출하지 않는다.
|
||||
|
||||
### 11.4 Scored execution
|
||||
|
||||
fresh preflight와 명시적 실행 권한이 있는 현재 plan에서만 다음 명령을 한 번 호출한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest "$benchmark_manifest"
|
||||
```
|
||||
|
||||
- direct CLI stdout/stderr와 exit code를 보존한다.
|
||||
- CLI가 출력한 canonical `run-...` id만 이후 `status`, `score`, `report`에 사용한다.
|
||||
- command가 nonzero여도 같은 plan에서 `run`을 다시 호출하지 않는다.
|
||||
- CLI가 run id를 출력하지 않으면 임의 id나 성공 pointer를 만들지 않는다.
|
||||
- caller나 provider를 CLI 밖에서 별도로 호출해 scored result를 보충하지 않는다.
|
||||
|
||||
### 11.5 Status
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py status \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
현재 comparison execution 완료 조건:
|
||||
|
||||
- controller terminal 수 `completed + timed_out + cancelled + interrupted = 9`
|
||||
- `running = 0`
|
||||
- `interrupted = 0`
|
||||
- 각 cell/repetition에 retained terminal attempt가 존재
|
||||
- 모든 최신 attempt에 controller/product/harness/process/web-validation terminal evidence가 있고 `unresolved=0`
|
||||
|
||||
`completed`는 controller 종료만 뜻하며 product 성공을 뜻하지 않는다. product/harness/process/artifact 실패는 scoring eligibility와 최종 비교에서 각각 별도로 표시되고, terminal 실패를 성공 결과로 바꾸거나 암묵 재시도하지 않는다.
|
||||
|
||||
### 11.6 Blind scoring
|
||||
|
||||
유효한 execution run에 대해서만 수행한다.
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py score \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
evaluator는 Codex→`gpt-5.6-luna` xhigh direct route다. identity가 제거된 blind workspace만 보며 source cell identity mapping은 blind tree 밖에 유지한다.
|
||||
|
||||
product, harness, acceptable process 또는 artifact gate 실패는 각각의 reason을 가진 `unscored`이고 0점으로 바꾸지 않는다. `scoring_failed`도 명시적 `--retry-scoring-failed` 권한 없이 재시도하지 않는다.
|
||||
|
||||
### 11.7 Report
|
||||
|
||||
```bash
|
||||
python3 scripts/agent_comparison_benchmark.py report \
|
||||
--manifest "$benchmark_manifest" \
|
||||
--run-id "$benchmark_run_id"
|
||||
```
|
||||
|
||||
report는 run root의 immutable evidence를 읽어 idempotent `report.md`를 만든다. 기존 report 내용과 새 projection이 다르면 덮어쓰지 않고 실패한다.
|
||||
|
||||
## 12. 수집 evidence
|
||||
|
||||
각 attempt는 다음 범주의 evidence를 가진다.
|
||||
|
||||
| 범주 | 내용 |
|
||||
|---|---|
|
||||
| lifecycle | typed caller terminal, submission, first output, finish, idle, quiet와 독립 product/harness/process 결과 |
|
||||
| timeline | submitted, first output, first workspace write observation/mtime, total duration |
|
||||
| usage | input/output/reasoning/cache read/cache write/total tokens, model/tool calls와 duration |
|
||||
| workspace | fresh session identity, fixture checksum, testbed provenance, generated file tree |
|
||||
| web validation | product/harness 결과와 무관하게 모든 terminal workspace에서 생성되는 generated files, static safety, images, network, console, responsive, accessibility |
|
||||
| screenshots | desktop `1920x1080`, mobile `375x812` |
|
||||
| scoring | eligibility, blind allocation, rubric worksheet, score status |
|
||||
|
||||
caller가 보고하지 않은 metric은 0으로 만들지 않고 `unavailable`과 reason/source를 보존한다.
|
||||
|
||||
Codex current usage mapping:
|
||||
|
||||
| Caller JSONL field | Canonical metric |
|
||||
|---|---|
|
||||
| `input_tokens` | `input_tokens` |
|
||||
| `cached_input_tokens` | `cached_input_tokens` |
|
||||
| `cache_write_input_tokens` | `cache_write_tokens` |
|
||||
| `output_tokens` | `output_tokens` |
|
||||
| `reasoning_output_tokens` | `reasoning_tokens` |
|
||||
| `total_tokens` | `total_tokens` |
|
||||
|
||||
누락된 `total_tokens`는 하위 category 합으로 재구성하지 않는다.
|
||||
|
||||
## 13. 자동 gate와 100점 rubric
|
||||
|
||||
automatic web gates는 scoring eligibility만 결정하고 점수에 포함되지 않는다.
|
||||
|
||||
| Gate |
|
||||
|---|
|
||||
| generated files |
|
||||
| static safety |
|
||||
| local images |
|
||||
| no external network dependency |
|
||||
| console safety |
|
||||
| responsive layout |
|
||||
| accessibility |
|
||||
|
||||
quality rubric:
|
||||
|
||||
| Category | Max |
|
||||
|---|---:|
|
||||
| requirements fidelity | 25 |
|
||||
| visual completeness | 25 |
|
||||
| responsive accessibility | 15 |
|
||||
| image/detail usage | 10 |
|
||||
| behavior stability | 10 |
|
||||
| code quality | 10 |
|
||||
| self verification | 5 |
|
||||
| Total | 100 |
|
||||
|
||||
## 14. 실패 처리와 재개 원칙
|
||||
|
||||
| 상황 | 조치 |
|
||||
|---|---|
|
||||
| manifest invalid | source/manifest를 수정하고 validate부터 다시 시작한다. run을 만들지 않는다. |
|
||||
| preflight not ready | blocker를 해결하고 fresh preflight한다. attempt를 할당하지 않는다. |
|
||||
| caller launch 전 interruption | retained evidence를 보존한다. run tree를 직접 수정하지 않는다. |
|
||||
| lifecycle/parser failure | exact retained output으로 source 원인을 수정하고 deterministic regression을 추가한다. |
|
||||
| completed이지만 product/harness/process/artifact gate 실패 또는 timed_out/cancelled | 각 축 evidence로 보존한다. 암묵 retry하지 않는다. |
|
||||
| state가 `running`이지만 process가 없음 | manual JSON 수정/삭제/reconcile을 하지 않는다. reviewer evidence로 남기고 승인된 새 plan에서만 다음 상태를 결정한다. |
|
||||
| scoring_failed | 0점 처리하지 않는다. 명시적 retry 권한 없이는 중단한다. |
|
||||
| report unavailable | run evidence를 수정하거나 report를 수작업 생성하지 않는다. |
|
||||
|
||||
현재 milestone의 원칙은 old incomplete/failed run을 `resume --retry-failed`하지 않고 distinct fresh run을 만드는 것이다. 일반 CLI가 `resume`을 지원한다는 사실이 현재 benchmark에서 사용 권한을 뜻하지 않는다.
|
||||
|
||||
## 15. Secret-safe 기록 규칙
|
||||
|
||||
다음 값은 tracked docs, task review, run metadata, log, metric label과 command argument에 남기지 않는다.
|
||||
|
||||
- IOP principal token
|
||||
- raw provider credential
|
||||
- private key, at-rest key, issuer/recipient private key
|
||||
- slot alias와 lease id
|
||||
- credential-bearing URL
|
||||
- raw prompt/response, tool input/output
|
||||
- caller/provider session content
|
||||
|
||||
허용되는 내용:
|
||||
|
||||
- secret file의 상대 path와 존재/mode
|
||||
- route/model/stage 이름
|
||||
- safe credential slot reference/revision
|
||||
- hashed endpoint/config/spec identity
|
||||
- redacted lifecycle 상태, duration과 usage count
|
||||
- run id와 attempt identity가 필요한 controller evidence
|
||||
|
||||
incident evidence를 보존하기 전에 repository와 run output에서 secret 원문이 없는지 확인한다. 의심되는 artifact는 내용을 복사하지 말고 path와 redaction failure만 보고한다.
|
||||
|
||||
## 16. 실행 전 체크리스트
|
||||
|
||||
- [ ] 현재 manifest validation 통과
|
||||
- [ ] focused/full deterministic tests fresh PASS
|
||||
- [ ] benchmark runner와 `../iop-s2` provenance 확인
|
||||
- [ ] Claude/agy/Codex command와 current version/help 확인
|
||||
- [ ] `token/.iop-bench`, CA와 provider provisioning source 존재/mode 확인; 내용 출력 없음
|
||||
- [ ] dev runtime source/build identity, process와 listener 확인
|
||||
- [ ] managed projection, provider slot/route와 no-legacy-fallback 확인
|
||||
- [ ] config observation이 runtime route/stage와 정확히 일치
|
||||
- [ ] direct qualification `ready=5`, fresh attempt 5개, `unresolved=0`, `running=0`, `interrupted=0`, terminal evidence 5개와 infrastructure block 없음
|
||||
- [ ] public preflight `ready=9`
|
||||
- [ ] 현재 plan이 exactly one scored run을 소유하고 사용자 권한이 명확함
|
||||
- [ ] old run resume/retry/state edit 계획 없음
|
||||
- [ ] run 이후 status, scoring, report의 run id 전달 경로 준비
|
||||
|
||||
## 17. 관련 구현
|
||||
|
||||
- controller/state: `scripts/agent_benchmark/attempts.py`
|
||||
- manifest: `scripts/agent_benchmark/manifest.py`
|
||||
- lifecycle: `scripts/agent_benchmark/lifecycle.py`
|
||||
- workspace isolation: `scripts/agent_benchmark/workspace.py`
|
||||
- live routing/admission: `scripts/agent_benchmark/live_iop.py`
|
||||
- Claude adapter: `scripts/agent_benchmark/claude_iop.py`
|
||||
- agy adapter: `scripts/agent_benchmark/agy_iop.py`
|
||||
- Codex adapter: `scripts/agent_benchmark/codex_iop.py`
|
||||
- measurement: `scripts/agent_benchmark/measurement.py`
|
||||
- browser/web gate: `scripts/agent_benchmark/web_validation.py`
|
||||
- blind scoring: `scripts/agent_benchmark/scoring.py`
|
||||
- rubric: `scripts/agent_benchmark/rubric.py`
|
||||
- report: `scripts/agent_benchmark/reporting.py`
|
||||
|
|
@ -280,8 +280,8 @@ The deterministic Messages qualification succeeds alongside Chat: the Control Pl
|
|||
공식 `agy` 1.1.12 API-key provider는 upstream Gemini key가 아니라 관리형 IOP principal token을 사용한다. dev operator가 이미 발급한 token과 CA 파일을 보호된 `token/` 아래에 둔 경우 값을 명령행에 직접 쓰지 않고 다음처럼 읽는다.
|
||||
|
||||
```bash
|
||||
read -r IOP_BENCH_TOKEN < token/.iop-bench
|
||||
export GEMINI_API_KEY="$IOP_BENCH_TOKEN"
|
||||
read -r IOP_AGY_SMOKE_TOKEN < token/.iop-principal
|
||||
export GEMINI_API_KEY="$IOP_AGY_SMOKE_TOKEN"
|
||||
export SSL_CERT_FILE="$PWD/token/iop-dev-ca.pem"
|
||||
export NODE_EXTRA_CA_CERTS="$PWD/token/iop-dev-ca.pem"
|
||||
|
||||
|
|
@ -293,12 +293,10 @@ GOOGLE_GEMINI_BASE_URL="https://<edge-host>:<https-port>/gemini/<hybrid-preset-i
|
|||
agy --sandbox --output-format stream-json --model 'Gemini 3.6 Flash' \
|
||||
--print 'Inspect README.md and report only its first Markdown heading. Do not modify files.'
|
||||
|
||||
unset IOP_BENCH_TOKEN GEMINI_API_KEY SSL_CERT_FILE NODE_EXTRA_CA_CERTS
|
||||
unset IOP_AGY_SMOKE_TOKEN GEMINI_API_KEY SSL_CERT_FILE NODE_EXTRA_CA_CERTS
|
||||
```
|
||||
|
||||
`--effort`는 API-key provider 호출에 넣지 않는다. direct와 hybrid 모두 JSONL의 마지막 record가 `event=result`, 중첩 `result.status=SUCCESS` 한 건이어야 한다. hybrid는 plan/work/review가 포함되므로 direct보다 오래 걸릴 수 있으며, caller timeout을 이유로 같은 scored attempt를 재실행하지 않는다. Gemini ingress와 공식 event 구조의 상세 계약은 `agent-contract/outer/gemini-compatible-api.md`를 기준으로 한다.
|
||||
|
||||
benchmark 전체 preflight에서는 Claude/agy/Codex에 같은 IOP principal을 secret environment reference로 연결하고 `IOP_BENCH_CONFIG_OBSERVATION_ENV`가 가리키는 operator-owned route/binding observation을 함께 제공한다. 사설 CA 환경은 각 isolated caller child에 `SSL_CERT_FILE`과 `NODE_EXTRA_CA_CERTS`로 전달된다. preflight가 모든 cell을 `ready`로 판정하기 전에는 scored `run`을 시작하지 않는다.
|
||||
`--effort`는 API-key provider 호출에 넣지 않는다. direct와 hybrid 모두 JSONL의 마지막 record가 `event=result`, 중첩 `result.status=SUCCESS` 한 건이어야 한다. hybrid는 plan/work/review가 포함되므로 direct보다 오래 걸릴 수 있다. 한 경로가 실패하면 다른 경로를 묶어 재실행하지 않고 해당 IOP ingress, route binding, preset stage 또는 caller terminal을 분리해 확인한다. Gemini ingress와 공식 event 구조의 상세 계약은 `agent-contract/outer/gemini-compatible-api.md`를 기준으로 한다.
|
||||
|
||||
### Incident redaction check
|
||||
|
||||
|
|
|
|||
|
|
@ -1,190 +0,0 @@
|
|||
"""
|
||||
agent_benchmark - Closed manifest loader, validator, and workspace preparation.
|
||||
|
||||
This package is standard-library-only and exposes stable immutable
|
||||
manifest and workspace preparation APIs consuming only frozen inputs.
|
||||
All returned objects are frozen dataclasses.
|
||||
"""
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
AssetMapping,
|
||||
ExpectedBinding,
|
||||
Fixture,
|
||||
IopCell,
|
||||
Manifest,
|
||||
ManifestDigestError,
|
||||
ManifestError,
|
||||
ManifestPathError,
|
||||
ManifestValidationError,
|
||||
MatrixCell,
|
||||
Timeout,
|
||||
Viewport,
|
||||
digest_manifest_and_resolved_inputs,
|
||||
digest_workspace_inputs,
|
||||
load_manifest,
|
||||
validate_manifest_bytes,
|
||||
)
|
||||
from scripts.agent_benchmark.workspace import (
|
||||
AttemptIdentity,
|
||||
PreparedWorkspace,
|
||||
TestbedError,
|
||||
TestbedProvenance,
|
||||
WorkspaceChecksumError,
|
||||
WorkspaceError,
|
||||
WorkspacePathError,
|
||||
WorkspaceValidationError,
|
||||
inspect_testbed_provenance,
|
||||
prepare_workspace,
|
||||
validate_attempt_identity,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
COMPLETION_EXIT_AFTER_IDLE,
|
||||
COMPLETION_STOP_AFTER_IDLE,
|
||||
SUBMISSION_ARGV_TASK,
|
||||
SUBMISSION_STDIN_ONCE,
|
||||
CancellationToken,
|
||||
CaptureStream,
|
||||
InvocationResult,
|
||||
InvocationSpec,
|
||||
LifecycleError,
|
||||
LifecycleProtocolError,
|
||||
LifecycleRecoveryError,
|
||||
LifecycleValidationError,
|
||||
SupervisorLocator,
|
||||
TerminalOutcome,
|
||||
env_pairs,
|
||||
exact_value_redactor,
|
||||
read_locator,
|
||||
recover_invocation,
|
||||
run_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.connectivity import (
|
||||
CallerCapability,
|
||||
ConnectivityEvidenceError,
|
||||
ConnectivityIssue,
|
||||
ConnectivityResult,
|
||||
ConnectivityValidationError,
|
||||
EffectiveBinding,
|
||||
RequestedEffectiveBinding,
|
||||
canonical_evidence_bytes,
|
||||
classify_issues,
|
||||
make_result,
|
||||
read_evidence,
|
||||
validate_result,
|
||||
write_evidence,
|
||||
)
|
||||
from scripts.agent_benchmark.claude_iop import (
|
||||
ClaudeIopAdapter,
|
||||
ClaudeIopRuntime,
|
||||
claude_capability,
|
||||
)
|
||||
from scripts.agent_benchmark.agy_iop import (
|
||||
AgyCapability,
|
||||
AgyEventParser,
|
||||
AgyPreflightResult,
|
||||
AgyRuntimeInputs,
|
||||
AgyRuntimeObservation,
|
||||
inspect_agy_iop_capability,
|
||||
preflight_agy_iop,
|
||||
run_agy_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.codex_iop import (
|
||||
CodexInvocation,
|
||||
CodexInvocationResult,
|
||||
CodexJSONLParser,
|
||||
CodexRuntime,
|
||||
build_codex_invocation,
|
||||
codex_capability,
|
||||
run_codex_invocation,
|
||||
runtime_from_environment,
|
||||
)
|
||||
from scripts.agent_benchmark.attempts import (
|
||||
Attempt, AttemptError, AttemptStateError, CapabilityUnavailable, RunBusyError,
|
||||
PreflightAdapter, PreflightObservation, RunIdentity, RunPathError, RunStore,
|
||||
Slot, collect_preflight_observations, preflight_manifest, run_slots,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"Manifest",
|
||||
"Timeout",
|
||||
"Viewport",
|
||||
"AssetMapping",
|
||||
"Fixture",
|
||||
"IopCell",
|
||||
"ExpectedBinding",
|
||||
"MatrixCell",
|
||||
"ManifestError",
|
||||
"ManifestValidationError",
|
||||
"ManifestPathError",
|
||||
"ManifestDigestError",
|
||||
"load_manifest",
|
||||
"validate_manifest_bytes",
|
||||
"digest_workspace_inputs",
|
||||
"digest_manifest_and_resolved_inputs",
|
||||
"AttemptIdentity",
|
||||
"TestbedProvenance",
|
||||
"PreparedWorkspace",
|
||||
"WorkspaceError",
|
||||
"WorkspaceValidationError",
|
||||
"WorkspacePathError",
|
||||
"WorkspaceChecksumError",
|
||||
"TestbedError",
|
||||
"prepare_workspace",
|
||||
"inspect_testbed_provenance",
|
||||
"validate_attempt_identity",
|
||||
"SUBMISSION_ARGV_TASK",
|
||||
"SUBMISSION_STDIN_ONCE",
|
||||
"COMPLETION_EXIT_AFTER_IDLE",
|
||||
"COMPLETION_STOP_AFTER_IDLE",
|
||||
"InvocationSpec",
|
||||
"InvocationResult",
|
||||
"CaptureStream",
|
||||
"SupervisorLocator",
|
||||
"TerminalOutcome",
|
||||
"CancellationToken",
|
||||
"LifecycleError",
|
||||
"LifecycleValidationError",
|
||||
"LifecycleProtocolError",
|
||||
"LifecycleRecoveryError",
|
||||
"run_invocation",
|
||||
"recover_invocation",
|
||||
"read_locator",
|
||||
"env_pairs",
|
||||
"exact_value_redactor",
|
||||
"CallerCapability",
|
||||
"ConnectivityEvidenceError",
|
||||
"ConnectivityIssue",
|
||||
"ConnectivityResult",
|
||||
"ConnectivityValidationError",
|
||||
"EffectiveBinding",
|
||||
"RequestedEffectiveBinding",
|
||||
"canonical_evidence_bytes",
|
||||
"classify_issues",
|
||||
"make_result",
|
||||
"read_evidence",
|
||||
"validate_result",
|
||||
"write_evidence",
|
||||
"ClaudeIopAdapter",
|
||||
"ClaudeIopRuntime",
|
||||
"claude_capability",
|
||||
"AgyCapability",
|
||||
"AgyEventParser",
|
||||
"AgyPreflightResult",
|
||||
"AgyRuntimeInputs",
|
||||
"AgyRuntimeObservation",
|
||||
"inspect_agy_iop_capability",
|
||||
"preflight_agy_iop",
|
||||
"run_agy_invocation",
|
||||
"CodexInvocation",
|
||||
"CodexInvocationResult",
|
||||
"CodexJSONLParser",
|
||||
"CodexRuntime",
|
||||
"build_codex_invocation",
|
||||
"codex_capability",
|
||||
"run_codex_invocation",
|
||||
"runtime_from_environment",
|
||||
"Attempt", "AttemptError", "AttemptStateError", "CapabilityUnavailable",
|
||||
"RunBusyError", "PreflightAdapter", "PreflightObservation", "RunIdentity",
|
||||
"RunPathError", "RunStore", "Slot", "collect_preflight_observations",
|
||||
"preflight_manifest", "run_slots",
|
||||
]
|
||||
|
|
@ -1,567 +0,0 @@
|
|||
"""Closed agy-to-IOP adapter for the comparison benchmark.
|
||||
|
||||
The adapter deliberately recognises only a documented agy transport contract.
|
||||
In particular, it never inherits an ambient Gemini configuration: absent or
|
||||
unknown transport support is an implementation gap before a caller process is
|
||||
constructed. The module is standard-library-only and has no network calls.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from scripts.agent_benchmark.connectivity import (
|
||||
ISSUE_RESUME_CODES,
|
||||
CallerCapability,
|
||||
ConnectivityResult,
|
||||
ConnectivityIssue,
|
||||
RequestedEffectiveBinding,
|
||||
classify_issues,
|
||||
make_result,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CALLER_REASON_ERROR,
|
||||
CALLER_REASON_SUCCESS,
|
||||
CALLER_STATUS_FAILED,
|
||||
CALLER_STATUS_SUCCEEDED,
|
||||
COMPLETION_EXIT_AFTER_IDLE,
|
||||
SUBMISSION_ARGV_TASK,
|
||||
CallerEvent,
|
||||
CallerTerminal,
|
||||
InvocationResult,
|
||||
InvocationSpec,
|
||||
LifecycleMetricError,
|
||||
ParsedMetric,
|
||||
SupervisorLocator,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
env_pairs,
|
||||
exact_value_redactor,
|
||||
inherited_tls_ca_environment,
|
||||
run_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import PreparedWorkspace
|
||||
|
||||
|
||||
AGY_CALLER = "agy"
|
||||
AGY_KNOWN_VERSION = "1.1.12"
|
||||
AGY_ENDPOINT_ENV = "GOOGLE_GEMINI_BASE_URL"
|
||||
AGY_AUTH_ENV = "GEMINI_API_KEY"
|
||||
AGY_SETTINGS_RELATIVE_PATH = Path(".gemini/antigravity-cli/settings.json")
|
||||
_AGY_ISOLATED_SETTINGS = {
|
||||
"enableTelemetry": False,
|
||||
"modelProvider": "gemini",
|
||||
"toolPermission": "always-proceed",
|
||||
}
|
||||
AGY_MODEL_LABELS = {
|
||||
"gemini-3.6-flash": "Gemini 3.6 Flash",
|
||||
"gemini-hybrid": "Gemini 3.6 Flash",
|
||||
}
|
||||
_VERSION_RE = re.compile(r"(?:agy\s+)?(\d+\.\d+\.\d+)", re.IGNORECASE)
|
||||
_SAFE_EVENT_FIELDS = ("event", "state", "step_type", "status")
|
||||
_DOCUMENTED_OPTIONS = ("--print", "--output-format", "--dangerously-skip-permissions", "--model")
|
||||
_AGY_USAGE_METRICS = {
|
||||
"input_tokens": "input_tokens",
|
||||
"cache_read_tokens": "cached_input_tokens",
|
||||
"output_tokens": "output_tokens",
|
||||
"thinking_tokens": "reasoning_tokens",
|
||||
"total_tokens": "total_tokens",
|
||||
}
|
||||
_IDENTITY_RE = re.compile(r"sha256:[0-9a-f]{64}\Z")
|
||||
|
||||
|
||||
def _exact_token_present(text: str, token: str) -> bool:
|
||||
"""Match one documented help token, never a prefix or a suffix."""
|
||||
return re.search(r"(?<![A-Za-z0-9_-])" + re.escape(token) + r"(?![A-Za-z0-9_-])", text) is not None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgyRuntimeInputs:
|
||||
"""Private runtime values supplied by a preflight owner, never persisted."""
|
||||
|
||||
binary: str
|
||||
endpoint: str
|
||||
credential: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgyRuntimeObservation:
|
||||
"""Secret-free IOP configuration proof for exactly one benchmark cell."""
|
||||
|
||||
cell_id: str
|
||||
route_kind: str
|
||||
route_id: str
|
||||
endpoint_identity: str
|
||||
credential_identity: str
|
||||
config_identity: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _ValidatedAgyRuntime:
|
||||
"""Private launch values admitted only after IOP configuration validation."""
|
||||
|
||||
binary: str
|
||||
endpoint: str
|
||||
credential: str
|
||||
observation: AgyRuntimeObservation
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgyDocumentedCapabilities:
|
||||
"""Exact, known-version tokens parsed from public agy help output."""
|
||||
|
||||
options: tuple[str, ...]
|
||||
environment: tuple[str, ...]
|
||||
output_formats: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgyCapability:
|
||||
"""Versioned, documented capability observation; no caller is launched."""
|
||||
|
||||
version: str | None
|
||||
iop_transport_supported: bool
|
||||
endpoint_supported: bool
|
||||
auth_supported: bool
|
||||
protocol_supported: bool
|
||||
stream_supported: bool
|
||||
route_kinds: tuple[str, ...]
|
||||
efforts: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AgyPreflightResult:
|
||||
"""Closed outcome before invocation construction."""
|
||||
|
||||
capability: AgyCapability
|
||||
binding: RequestedEffectiveBinding
|
||||
issues: tuple[ConnectivityIssue, ...]
|
||||
status: str
|
||||
runtime: _ValidatedAgyRuntime | None
|
||||
|
||||
|
||||
class AgyAdapterError(Exception):
|
||||
"""Raised when a caller launch is requested without a proven transport."""
|
||||
|
||||
|
||||
def _issue(code: str) -> ConnectivityIssue:
|
||||
return ConnectivityIssue(code, ISSUE_RESUME_CODES[code])
|
||||
|
||||
|
||||
def _requested_binding(cell: MatrixCell) -> RequestedEffectiveBinding:
|
||||
return RequestedEffectiveBinding(
|
||||
cell.id,
|
||||
cell.caller,
|
||||
cell.iop.route_kind,
|
||||
cell.iop.route_id,
|
||||
cell.iop.request_model,
|
||||
cell.iop.requested_effort,
|
||||
)
|
||||
|
||||
|
||||
def parse_documented_agy_capabilities(help_output: str) -> AgyDocumentedCapabilities:
|
||||
"""Parse only complete documented tokens from a known agy help surface."""
|
||||
if not isinstance(help_output, str):
|
||||
return AgyDocumentedCapabilities((), (), ())
|
||||
return AgyDocumentedCapabilities(
|
||||
tuple(token for token in _DOCUMENTED_OPTIONS if _exact_token_present(help_output, token)),
|
||||
(),
|
||||
("stream-json",) if _exact_token_present(help_output, "stream-json") else (),
|
||||
)
|
||||
|
||||
|
||||
def inspect_agy_iop_capability(version_output: str, help_output: str) -> AgyCapability:
|
||||
"""Inspect only public, versioned help text for the closed IOP transport.
|
||||
|
||||
The known release is pinned because its Gemini provider environment is not
|
||||
printed by ``--help``. The public options and stream format still have to
|
||||
match exactly; a changed release fails closed until re-qualified.
|
||||
"""
|
||||
if not isinstance(version_output, str) or not isinstance(help_output, str):
|
||||
return AgyCapability(None, False, False, False, False, False, (), ())
|
||||
matched = _VERSION_RE.fullmatch(version_output.strip())
|
||||
version = matched.group(1) if matched else None
|
||||
known_version = version == AGY_KNOWN_VERSION
|
||||
documented = parse_documented_agy_capabilities(help_output)
|
||||
endpoint_supported = known_version
|
||||
auth_supported = known_version
|
||||
protocol_supported = known_version and all(
|
||||
option in documented.options for option in _DOCUMENTED_OPTIONS
|
||||
)
|
||||
stream_supported = "stream-json" in documented.output_formats
|
||||
supported = endpoint_supported and auth_supported and protocol_supported and stream_supported
|
||||
if not supported:
|
||||
return AgyCapability(
|
||||
version, False, endpoint_supported, auth_supported, protocol_supported, stream_supported, (), ()
|
||||
)
|
||||
return AgyCapability(
|
||||
version,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
("direct", "execution_preset"),
|
||||
("high", "low", "medium"),
|
||||
)
|
||||
|
||||
|
||||
def _runtime_identity(label: str, value: str) -> str:
|
||||
return "sha256:" + hashlib.sha256(
|
||||
b"agy-iop-runtime-v1\0" + label.encode("ascii") + b"\0" + value.encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _validate_config_owner_observation(
|
||||
cell: MatrixCell, observation: AgyRuntimeObservation
|
||||
) -> None:
|
||||
"""Validate an opaque observation supplied by the independent config owner."""
|
||||
if not isinstance(observation, AgyRuntimeObservation):
|
||||
raise AgyAdapterError("agy runtime observation is invalid")
|
||||
if (observation.cell_id, observation.route_kind, observation.route_id) != (
|
||||
cell.id, cell.iop.route_kind, cell.iop.route_id,
|
||||
):
|
||||
raise AgyAdapterError("agy IOP runtime observation mismatch")
|
||||
if not all(_IDENTITY_RE.fullmatch(value) for value in (
|
||||
observation.endpoint_identity,
|
||||
observation.credential_identity,
|
||||
observation.config_identity,
|
||||
)):
|
||||
raise AgyAdapterError("agy IOP runtime observation identity is invalid")
|
||||
|
||||
|
||||
def validate_agy_iop_runtime(
|
||||
cell: MatrixCell, runtime: AgyRuntimeInputs, observation: AgyRuntimeObservation
|
||||
) -> _ValidatedAgyRuntime:
|
||||
"""Admit runtime-only launch values only when IOP identity proof is exact."""
|
||||
if not isinstance(cell, MatrixCell) or cell.caller != AGY_CALLER:
|
||||
raise AgyAdapterError("agy runtime validation requires an agy matrix cell")
|
||||
if not isinstance(runtime, AgyRuntimeInputs) or not isinstance(observation, AgyRuntimeObservation):
|
||||
raise AgyAdapterError("agy runtime inputs are invalid")
|
||||
if not all(isinstance(value, str) and value for value in (runtime.binary, runtime.endpoint, runtime.credential)):
|
||||
raise AgyAdapterError("agy runtime values are unavailable")
|
||||
endpoint = urlsplit(runtime.endpoint)
|
||||
route_path = f"/gemini/{cell.iop.route_id}"
|
||||
if (
|
||||
endpoint.scheme != "https"
|
||||
or not endpoint.netloc
|
||||
or endpoint.query
|
||||
or endpoint.fragment
|
||||
or endpoint.path.rstrip("/") != route_path
|
||||
):
|
||||
raise AgyAdapterError("agy IOP endpoint is invalid")
|
||||
_validate_config_owner_observation(cell, observation)
|
||||
if observation.endpoint_identity != _runtime_identity("endpoint", runtime.endpoint):
|
||||
raise AgyAdapterError("agy IOP runtime observation mismatch")
|
||||
if observation.credential_identity != _runtime_identity("credential", runtime.credential):
|
||||
raise AgyAdapterError("agy IOP runtime observation mismatch")
|
||||
return _ValidatedAgyRuntime(runtime.binary, runtime.endpoint, runtime.credential, observation)
|
||||
|
||||
|
||||
def preflight_agy_iop(
|
||||
cell: MatrixCell,
|
||||
capability: AgyCapability,
|
||||
runtime: AgyRuntimeInputs,
|
||||
observation: AgyRuntimeObservation,
|
||||
) -> AgyPreflightResult:
|
||||
"""Classify only registration and implementation gaps without launching agy."""
|
||||
if not isinstance(cell, MatrixCell) or cell.caller != AGY_CALLER:
|
||||
raise AgyAdapterError("agy preflight requires an agy matrix cell")
|
||||
if not isinstance(capability, AgyCapability) or not isinstance(runtime, AgyRuntimeInputs):
|
||||
raise AgyAdapterError("agy preflight inputs are invalid")
|
||||
|
||||
issues: list[ConnectivityIssue] = []
|
||||
validated_runtime: _ValidatedAgyRuntime | None = None
|
||||
if runtime.credential:
|
||||
try:
|
||||
validated_runtime = validate_agy_iop_runtime(cell, runtime, observation)
|
||||
except AgyAdapterError:
|
||||
issues.append(_issue("endpoint_incompatible"))
|
||||
if not runtime.credential:
|
||||
issues.append(_issue("credential_missing"))
|
||||
if cell.iop.request_model not in AGY_MODEL_LABELS:
|
||||
issues.append(_issue("model_missing"))
|
||||
if not runtime.endpoint:
|
||||
issues.append(_issue("endpoint_incompatible"))
|
||||
if not capability.endpoint_supported:
|
||||
issues.append(_issue("endpoint_incompatible"))
|
||||
if not capability.auth_supported:
|
||||
issues.append(_issue("auth_incompatible"))
|
||||
if not capability.protocol_supported:
|
||||
issues.append(_issue("protocol_incompatible"))
|
||||
if not capability.stream_supported:
|
||||
issues.append(_issue("stream_incompatible"))
|
||||
if capability.iop_transport_supported and cell.iop.route_kind not in capability.route_kinds:
|
||||
issues.append(_issue("protocol_incompatible"))
|
||||
elif capability.iop_transport_supported and cell.iop.requested_effort not in capability.efforts:
|
||||
issues.append(_issue("effort_unsupported"))
|
||||
|
||||
# Preserve connectivity.py's canonical issue order without leaking values.
|
||||
unique = {item.code: item for item in issues}
|
||||
ordered = tuple(
|
||||
unique[code]
|
||||
for code in (
|
||||
"credential_missing", "model_missing", "route_missing", "effort_unsupported",
|
||||
"endpoint_incompatible", "auth_incompatible", "protocol_incompatible", "stream_incompatible",
|
||||
)
|
||||
if code in unique
|
||||
)
|
||||
status = classify_issues(ordered)
|
||||
return AgyPreflightResult(capability, _requested_binding(cell), ordered, status, validated_runtime)
|
||||
|
||||
|
||||
def build_agy_invocation(
|
||||
cell: MatrixCell,
|
||||
prepared: PreparedWorkspace,
|
||||
task_payload: bytes,
|
||||
timeout: Timeout,
|
||||
preflight: AgyPreflightResult,
|
||||
) -> InvocationSpec:
|
||||
"""Build one isolated official agy print invocation after a ready preflight."""
|
||||
if preflight.status != "ready" or not preflight.capability.iop_transport_supported or preflight.runtime is None:
|
||||
raise AgyAdapterError("agy IOP transport is not proven")
|
||||
runtime = preflight.runtime
|
||||
if not runtime.binary or not Path(runtime.binary).is_file():
|
||||
raise AgyAdapterError("agy binary is unavailable")
|
||||
if (
|
||||
not isinstance(prepared, PreparedWorkspace)
|
||||
or not Path(prepared.workspace_dir).is_dir()
|
||||
or not Path(prepared.session_dir).is_dir()
|
||||
or not Path(prepared.attempt_root).is_dir()
|
||||
):
|
||||
raise AgyAdapterError("prepared workspace is unavailable")
|
||||
if not isinstance(task_payload, bytes) or not task_payload:
|
||||
raise AgyAdapterError("agy task payload is unavailable")
|
||||
try:
|
||||
task_text = task_payload.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise AgyAdapterError("agy task payload must be UTF-8") from exc
|
||||
|
||||
_stage_agy_provider_settings(prepared.session_dir)
|
||||
|
||||
# The child receives a minimal environment and explicit IOP-only provider
|
||||
# settings. No parent agy/Gemini config or session variable is inherited.
|
||||
environment = {
|
||||
"PATH": os.environ.get("PATH", "/usr/bin:/bin"),
|
||||
"LANG": "C.UTF-8",
|
||||
"LC_ALL": "C.UTF-8",
|
||||
"TZ": "UTC",
|
||||
"HOME": prepared.session_dir,
|
||||
AGY_ENDPOINT_ENV: runtime.endpoint,
|
||||
AGY_AUTH_ENV: runtime.credential,
|
||||
}
|
||||
environment.update(inherited_tls_ca_environment())
|
||||
return InvocationSpec(
|
||||
argv=(
|
||||
runtime.binary,
|
||||
"--dangerously-skip-permissions",
|
||||
"--output-format", "stream-json",
|
||||
"--model", AGY_MODEL_LABELS[cell.iop.request_model],
|
||||
"--print", task_text,
|
||||
),
|
||||
cwd=prepared.workspace_dir,
|
||||
env=env_pairs(environment),
|
||||
env_allowlist=(AGY_ENDPOINT_ENV, AGY_AUTH_ENV, *TLS_CA_ENV_KEYS),
|
||||
submission_mode=SUBMISSION_ARGV_TASK,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=timeout,
|
||||
evidence_dir=prepared.attempt_root,
|
||||
control_dir=str(Path(prepared.attempt_root) / "agy-control"),
|
||||
)
|
||||
|
||||
|
||||
def _stage_agy_provider_settings(session_dir: str) -> None:
|
||||
"""Create the exact secret-free provider selector in one isolated HOME."""
|
||||
try:
|
||||
session = Path(session_dir).resolve(strict=True)
|
||||
if not session.is_dir():
|
||||
raise OSError("session is not a directory")
|
||||
settings = session / AGY_SETTINGS_RELATIVE_PATH
|
||||
settings.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
if settings.parent.resolve(strict=True) != session / AGY_SETTINGS_RELATIVE_PATH.parent:
|
||||
raise OSError("settings directory escapes session")
|
||||
payload = (
|
||||
json.dumps(_AGY_ISOLATED_SETTINGS, sort_keys=True, separators=(",", ":")) + "\n"
|
||||
).encode("utf-8")
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
descriptor = os.open(settings, flags, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as handle:
|
||||
handle.write(payload)
|
||||
except (OSError, ValueError) as exc:
|
||||
raise AgyAdapterError("agy isolated provider settings are unavailable") from exc
|
||||
|
||||
|
||||
def _safe_identifier(value: Any) -> str | None:
|
||||
return value if isinstance(value, str) and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:+-]{0,63}", value) else None
|
||||
|
||||
|
||||
def redact_agy_event(raw_line: str, sensitive_values: tuple[str, ...] = ()) -> str:
|
||||
"""Return a canonical allowlisted event projection, never caller content."""
|
||||
try:
|
||||
parsed = json.loads(raw_line)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
return '{"event":"unparseable"}'
|
||||
if not isinstance(parsed, dict):
|
||||
return '{"event":"unparseable"}'
|
||||
event = _safe_identifier(parsed.get("event"))
|
||||
if event is None or event in sensitive_values:
|
||||
return '{"event":"unparseable"}'
|
||||
safe: dict[str, str] = {"event": event}
|
||||
payload = parsed.get(event)
|
||||
if not isinstance(payload, dict):
|
||||
payload = {}
|
||||
for field in _SAFE_EVENT_FIELDS[1:]:
|
||||
value = _safe_identifier(payload.get(field))
|
||||
if value is not None and value not in sensitive_values:
|
||||
safe[field] = value
|
||||
return json.dumps(safe, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
|
||||
|
||||
|
||||
class AgyEventParser:
|
||||
"""Parse the official 1.1.12 stream while trusting binding only from config."""
|
||||
|
||||
def __init__(self, cell: MatrixCell, admitted_binding: RequestedEffectiveBinding) -> None:
|
||||
if not isinstance(admitted_binding, RequestedEffectiveBinding):
|
||||
raise AgyAdapterError("agy admitted binding is invalid")
|
||||
if admitted_binding.cell_id != cell.id or admitted_binding.caller != cell.caller:
|
||||
raise AgyAdapterError("agy admitted binding mismatch")
|
||||
self._cell = cell
|
||||
self._admitted_binding = admitted_binding
|
||||
self._init_seen = False
|
||||
self._result_seen = False
|
||||
self._latest_usage: dict[str, Any] | None = None
|
||||
|
||||
def __call__(self, stream: str, raw_line: str) -> Any:
|
||||
return self.parse(stream, raw_line)
|
||||
|
||||
def parse(self, stream: str, raw_line: str) -> Any:
|
||||
if stream != "stdout":
|
||||
return None
|
||||
try:
|
||||
item = json.loads(raw_line)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
if not isinstance(item, dict):
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
event = item.get("event")
|
||||
payload = item.get(event) if isinstance(event, str) else None
|
||||
if not isinstance(payload, dict):
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
if event == "init":
|
||||
if self._init_seen or self._result_seen:
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
self._init_seen = True
|
||||
return None
|
||||
if event == "step_update":
|
||||
if not self._init_seen or self._result_seen:
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
usage = payload.get("usage")
|
||||
if usage is not None:
|
||||
if self._usage_metrics(usage) is None:
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
self._latest_usage = usage
|
||||
return None
|
||||
if event != "result" or not self._init_seen or self._result_seen:
|
||||
raise AgyAdapterError("malformed agy event")
|
||||
self._result_seen = True
|
||||
if payload.get("status") != "SUCCESS":
|
||||
if payload.get("status") != "ERROR":
|
||||
raise AgyAdapterError("malformed agy result status")
|
||||
return (
|
||||
CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR),
|
||||
CallerEvent("finish"),
|
||||
CallerEvent("idle"),
|
||||
)
|
||||
metrics: list[ParsedMetric] = []
|
||||
usage = payload.get("usage", self._latest_usage)
|
||||
if usage is not None:
|
||||
parsed_usage = self._usage_metrics(usage)
|
||||
if parsed_usage is None:
|
||||
raise AgyAdapterError("malformed agy usage")
|
||||
metrics.extend(parsed_usage)
|
||||
try:
|
||||
if "duration_seconds" in payload:
|
||||
metrics.append(duration_metric(
|
||||
"total_duration", payload["duration_seconds"], reported_unit="s",
|
||||
model=self._cell.iop.request_model,
|
||||
))
|
||||
if "num_turns" in payload:
|
||||
metrics.append(count_metric(
|
||||
"model_calls", payload["num_turns"], model=self._cell.iop.request_model,
|
||||
))
|
||||
except LifecycleMetricError:
|
||||
raise AgyAdapterError("malformed agy metric")
|
||||
return tuple(metrics) + (
|
||||
CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS),
|
||||
CallerEvent("finish"),
|
||||
CallerEvent("idle"),
|
||||
)
|
||||
|
||||
def _usage_metrics(self, usage: Any) -> tuple[ParsedMetric, ...] | None:
|
||||
if not isinstance(usage, dict) or not set(usage) <= set(_AGY_USAGE_METRICS):
|
||||
return None
|
||||
metrics: list[ParsedMetric] = []
|
||||
try:
|
||||
for wire_name, metric_name in _AGY_USAGE_METRICS.items():
|
||||
if wire_name in usage:
|
||||
metrics.append(count_metric(
|
||||
metric_name, usage[wire_name], model=self._cell.iop.request_model,
|
||||
))
|
||||
except LifecycleMetricError:
|
||||
return None
|
||||
return tuple(metrics)
|
||||
|
||||
def observed_result(self, capability: AgyCapability, lifecycle: InvocationResult) -> ConnectivityResult:
|
||||
requested = _requested_binding(self._cell)
|
||||
closed_gap = (_issue("stream_incompatible"),)
|
||||
caller_capability = CallerCapability(AGY_CALLER, capability.route_kinds, capability.efforts)
|
||||
if (
|
||||
not isinstance(lifecycle, InvocationResult)
|
||||
or lifecycle.product.status != CALLER_STATUS_SUCCEEDED
|
||||
or lifecycle.harness.status != "passed"
|
||||
or not lifecycle.harness.ordered_terminal
|
||||
or not self._result_seen
|
||||
):
|
||||
return make_result(self._cell, caller_capability, requested, closed_gap)
|
||||
try:
|
||||
return make_result(self._cell, caller_capability, self._admitted_binding)
|
||||
except Exception:
|
||||
return make_result(self._cell, caller_capability, requested, closed_gap)
|
||||
|
||||
|
||||
def run_agy_invocation(
|
||||
spec: InvocationSpec,
|
||||
parser: AgyEventParser,
|
||||
preflight: AgyPreflightResult,
|
||||
on_started: Callable[[SupervisorLocator], None],
|
||||
) -> InvocationResult:
|
||||
"""Run a prepared agy call with structural output redaction only."""
|
||||
if (
|
||||
not isinstance(preflight, AgyPreflightResult)
|
||||
or preflight.status != "ready"
|
||||
or not preflight.capability.iop_transport_supported
|
||||
or preflight.runtime is None
|
||||
):
|
||||
raise AgyAdapterError("agy IOP transport is not proven")
|
||||
runtime = preflight.runtime
|
||||
sensitive = (runtime.endpoint, runtime.credential)
|
||||
structural = lambda line: redact_agy_event(line, sensitive)
|
||||
# exact replacement is retained as a final defence for non-JSON stderr.
|
||||
exact = exact_value_redactor(sensitive)
|
||||
return run_invocation(
|
||||
spec,
|
||||
parse_event=parser,
|
||||
on_started=on_started,
|
||||
redact=lambda line: structural(exact(line)),
|
||||
)
|
||||
|
|
@ -1,303 +0,0 @@
|
|||
"""Credential-free tests for the official agy 1.1.12 IOP adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.agy_iop import (
|
||||
AGY_AUTH_ENV,
|
||||
AGY_ENDPOINT_ENV,
|
||||
AGY_KNOWN_VERSION,
|
||||
AGY_SETTINGS_RELATIVE_PATH,
|
||||
AgyAdapterError,
|
||||
AgyEventParser,
|
||||
AgyRuntimeInputs,
|
||||
AgyRuntimeObservation,
|
||||
_runtime_identity,
|
||||
build_agy_invocation,
|
||||
inspect_agy_iop_capability,
|
||||
preflight_agy_iop,
|
||||
redact_agy_event,
|
||||
run_agy_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.connectivity import EffectiveBinding, RequestedEffectiveBinding
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
REASON_MALFORMED_EVENT,
|
||||
REASON_NONZERO_EXIT,
|
||||
SUBMISSION_ARGV_TASK,
|
||||
InvocationSpec,
|
||||
env_pairs,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import AttemptIdentity, PreparedWorkspace, TestbedProvenance
|
||||
|
||||
|
||||
def _help() -> str:
|
||||
return "--print --output-format stream-json --dangerously-skip-permissions --model --effort"
|
||||
|
||||
|
||||
def _cell() -> MatrixCell:
|
||||
return MatrixCell(
|
||||
"agy-direct", "agy",
|
||||
IopCell("gemini-3.6-flash", "high", "direct", "agy-direct", (
|
||||
ExpectedBinding("request", "gemini-3.6-flash", "high"),
|
||||
)),
|
||||
)
|
||||
|
||||
|
||||
def _binding() -> RequestedEffectiveBinding:
|
||||
return RequestedEffectiveBinding(
|
||||
"agy-direct", "agy", "direct", "agy-direct", "gemini-3.6-flash", "high",
|
||||
"direct", "agy-direct", "gemini-3.6-flash", "high",
|
||||
(EffectiveBinding("request", "gemini-3.6-flash", "high"),),
|
||||
)
|
||||
|
||||
|
||||
class AgyIopTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.temp.name)
|
||||
self.workspace = self.root / "workspace"
|
||||
self.workspace.mkdir()
|
||||
self.session = self.root / "session"
|
||||
self.session.mkdir()
|
||||
self.runtime = AgyRuntimeInputs(
|
||||
sys.executable, "https://private.invalid/gemini/agy-direct", "iop_secret_123456789"
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def _observation(self, runtime: AgyRuntimeInputs | None = None) -> AgyRuntimeObservation:
|
||||
value = runtime or self.runtime
|
||||
return AgyRuntimeObservation(
|
||||
"agy-direct", "direct", "agy-direct",
|
||||
_runtime_identity("endpoint", value.endpoint),
|
||||
_runtime_identity("credential", value.credential),
|
||||
"sha256:" + "c" * 64,
|
||||
)
|
||||
|
||||
def _prepared(self) -> PreparedWorkspace:
|
||||
return PreparedWorkspace(
|
||||
AttemptIdentity("run", "agy-direct", 1, 1), str(self.root), str(self.workspace),
|
||||
str(self.session), "fresh-session", True, "sha256:" + "0" * 64,
|
||||
"isolated", TestbedProvenance("/testbed", "main", "0" * 40, "sha256:" + "1" * 64, True),
|
||||
"2026-01-01T00:00:00+00:00",
|
||||
)
|
||||
|
||||
def _preflight(self, runtime: AgyRuntimeInputs | None = None):
|
||||
value = runtime or self.runtime
|
||||
return preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("agy 1.1.12", _help()), value,
|
||||
self._observation(value),
|
||||
)
|
||||
|
||||
def _run_lines(self, lines: list[str], parser: AgyEventParser, *, exit_code: int = 0):
|
||||
evidence = self.root / f"evidence-{len(list(self.root.glob('evidence-*')))}"
|
||||
evidence.mkdir()
|
||||
source = (
|
||||
"import sys; lines=" + repr(lines)
|
||||
+ "; [print(line) for line in lines]; raise SystemExit(" + repr(exit_code) + ")"
|
||||
)
|
||||
spec = InvocationSpec(
|
||||
argv=(sys.executable, "-u", "-c", source), cwd=str(self.root),
|
||||
env=env_pairs({"PATH": os.environ.get("PATH", "/usr/bin:/bin")}),
|
||||
submission_mode="stdin_once", completion_mode="exit_after_idle",
|
||||
timeout=Timeout(5, 1, 1, 1), evidence_dir=str(evidence), task_payload=b"task",
|
||||
)
|
||||
return run_agy_invocation(spec, parser, self._preflight(), lambda _: None)
|
||||
|
||||
def test_official_public_surface_is_pinned_without_invented_environment(self) -> None:
|
||||
capability = inspect_agy_iop_capability("1.1.12", _help())
|
||||
self.assertEqual(capability.version, AGY_KNOWN_VERSION)
|
||||
self.assertTrue(capability.iop_transport_supported)
|
||||
self.assertTrue(capability.endpoint_supported)
|
||||
self.assertTrue(capability.auth_supported)
|
||||
self.assertFalse(inspect_agy_iop_capability("1.1.11", _help()).iop_transport_supported)
|
||||
self.assertFalse(inspect_agy_iop_capability("1.1.12", _help().replace("stream-json", "json")).stream_supported)
|
||||
missing_permission = inspect_agy_iop_capability(
|
||||
"1.1.12", _help().replace("--dangerously-skip-permissions", "")
|
||||
)
|
||||
self.assertFalse(missing_permission.protocol_supported)
|
||||
self.assertFalse(missing_permission.iop_transport_supported)
|
||||
|
||||
def test_build_uses_official_gemini_api_key_transport(self) -> None:
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = build_agy_invocation(_cell(), self._prepared(), b"one task", Timeout(5, 1, 1, 1), self._preflight())
|
||||
environment = dict(spec.env)
|
||||
self.assertEqual(environment[AGY_ENDPOINT_ENV], self.runtime.endpoint)
|
||||
self.assertEqual(environment[AGY_AUTH_ENV], self.runtime.credential)
|
||||
self.assertEqual(environment["HOME"], str(self.session))
|
||||
self.assertNotIn("AGY_PROVIDER", environment)
|
||||
self.assertNotIn("AGY_OPENAI_BASE_URL", environment)
|
||||
self.assertNotIn("AGY_OPENAI_API_KEY", environment)
|
||||
self.assertNotIn("--sandbox", spec.argv)
|
||||
self.assertEqual(spec.argv.count("--dangerously-skip-permissions"), 1)
|
||||
self.assertNotIn("--effort", spec.argv)
|
||||
self.assertEqual(spec.argv[spec.argv.index("--model") + 1], "Gemini 3.6 Flash")
|
||||
self.assertEqual(spec.argv[-2:], ("--print", "one task"))
|
||||
self.assertEqual(spec.submission_mode, SUBMISSION_ARGV_TASK)
|
||||
self.assertEqual(spec.task_payload, b"")
|
||||
self.assertEqual(environment["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(environment["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(spec.env_allowlist, (AGY_ENDPOINT_ENV, AGY_AUTH_ENV, "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS"))
|
||||
settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH
|
||||
self.assertEqual(json.loads(settings_path.read_text(encoding="utf-8")), {
|
||||
"enableTelemetry": False,
|
||||
"modelProvider": "gemini",
|
||||
"toolPermission": "always-proceed",
|
||||
})
|
||||
self.assertEqual(settings_path.stat().st_mode & 0o777, 0o600)
|
||||
settings_text = settings_path.read_text(encoding="utf-8")
|
||||
self.assertNotIn(self.runtime.endpoint, settings_text)
|
||||
self.assertNotIn(self.runtime.credential, settings_text)
|
||||
|
||||
def test_build_rejects_preexisting_isolated_settings(self) -> None:
|
||||
settings_path = self.session / AGY_SETTINGS_RELATIVE_PATH
|
||||
settings_path.parent.mkdir(parents=True)
|
||||
settings_path.write_text('{"modelProvider":"other"}\n', encoding="utf-8")
|
||||
with self.assertRaisesRegex(AgyAdapterError, "isolated provider settings"):
|
||||
build_agy_invocation(
|
||||
_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), self._preflight()
|
||||
)
|
||||
|
||||
def test_build_rejects_missing_prepared_workspace_or_session(self) -> None:
|
||||
prepared = self._prepared()
|
||||
for field in ("workspace_dir", "session_dir", "attempt_root"):
|
||||
missing = prepared.__class__(**{
|
||||
**prepared.__dict__, field: str(self.root / f"missing-{field}"),
|
||||
})
|
||||
with self.subTest(field=field):
|
||||
with self.assertRaisesRegex(AgyAdapterError, "prepared workspace"):
|
||||
build_agy_invocation(
|
||||
_cell(), missing, b"task", Timeout(5, 1, 1, 1), self._preflight()
|
||||
)
|
||||
|
||||
def test_build_rejects_non_utf8_task_for_print_argument(self) -> None:
|
||||
with self.assertRaisesRegex(AgyAdapterError, "must be UTF-8"):
|
||||
build_agy_invocation(
|
||||
_cell(), self._prepared(), b"\xff", Timeout(5, 1, 1, 1), self._preflight()
|
||||
)
|
||||
|
||||
def test_route_qualified_https_runtime_is_required(self) -> None:
|
||||
for endpoint in (
|
||||
"http://private.invalid/gemini/agy-direct",
|
||||
"https://private.invalid/v1",
|
||||
"https://private.invalid/gemini/other",
|
||||
):
|
||||
runtime = replace(self.runtime, endpoint=endpoint)
|
||||
result = self._preflight(runtime)
|
||||
self.assertEqual(result.status, "implementation_gap")
|
||||
self.assertEqual([issue.code for issue in result.issues], ["endpoint_incompatible"])
|
||||
missing = replace(self.runtime, credential="")
|
||||
result = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("1.1.12", _help()), missing,
|
||||
self._observation(self.runtime),
|
||||
)
|
||||
self.assertEqual(result.status, "registration_required")
|
||||
self.assertEqual([issue.code for issue in result.issues], ["credential_missing"])
|
||||
|
||||
def test_unknown_model_and_unvalidated_observation_fail_closed(self) -> None:
|
||||
unsupported = replace(_cell(), iop=replace(_cell().iop, request_model="gemini-unknown"))
|
||||
result = preflight_agy_iop(
|
||||
unsupported, inspect_agy_iop_capability("1.1.12", _help()), self.runtime,
|
||||
self._observation(),
|
||||
)
|
||||
self.assertIn("model_missing", [issue.code for issue in result.issues])
|
||||
mismatched = replace(self._observation(), endpoint_identity="sha256:" + "d" * 64)
|
||||
result = preflight_agy_iop(
|
||||
_cell(), inspect_agy_iop_capability("1.1.12", _help()), self.runtime, mismatched,
|
||||
)
|
||||
self.assertIsNone(result.runtime)
|
||||
with self.assertRaises(AgyAdapterError):
|
||||
build_agy_invocation(_cell(), self._prepared(), b"task", Timeout(5, 1, 1, 1), result)
|
||||
|
||||
def test_official_fixture_completes_and_preserves_metrics(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
fixture = Path("scripts/fixtures/agent-comparison-benchmark/agy-iop-stream.jsonl")
|
||||
result = self._run_lines(fixture.read_text(encoding="utf-8").splitlines(), parser)
|
||||
self.assertTrue(result.product.status == "succeeded")
|
||||
self.assertTrue(result.harness.ordered_terminal)
|
||||
metrics = {metric.name: metric for metric in result.metrics}
|
||||
self.assertEqual(
|
||||
set(metrics),
|
||||
{"total_duration", "model_calls", "input_tokens", "cached_input_tokens", "output_tokens", "reasoning_tokens", "total_tokens"},
|
||||
)
|
||||
self.assertEqual(metrics["total_duration"].value, 12_000_000)
|
||||
self.assertEqual(metrics["model_calls"].value, 1)
|
||||
self.assertEqual(metrics["total_tokens"].value, 12)
|
||||
capability = inspect_agy_iop_capability("1.1.12", _help())
|
||||
self.assertEqual(parser.observed_result(capability, result).binding, _binding())
|
||||
|
||||
def test_result_must_follow_init_be_unique_and_successful(self) -> None:
|
||||
result = {
|
||||
"event": "result", "result": {
|
||||
"status": "SUCCESS", "duration_seconds": 0.1, "num_turns": 1,
|
||||
"usage": {"input_tokens": 1}, "response": "secret content",
|
||||
},
|
||||
}
|
||||
for lines, reason in (
|
||||
([json.dumps(result)], REASON_MALFORMED_EVENT),
|
||||
([json.dumps({"event": "init", "init": {}}), json.dumps(result), json.dumps(result)], REASON_MALFORMED_EVENT),
|
||||
):
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
invocation = self._run_lines(lines, parser)
|
||||
self.assertFalse(invocation.product.status == "succeeded")
|
||||
self.assertEqual(invocation.harness.reason, "parser_error")
|
||||
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
error_result = json.dumps({
|
||||
"event": "result", "result": {"status": "ERROR", "response": "private"},
|
||||
})
|
||||
invocation = self._run_lines([
|
||||
json.dumps({"event": "init", "init": {}}), error_result,
|
||||
], parser, exit_code=1)
|
||||
self.assertFalse(invocation.product.status == "succeeded")
|
||||
self.assertEqual(invocation.product.status, "failed")
|
||||
self.assertEqual(invocation.harness.status, "passed")
|
||||
self.assertEqual(invocation.process.exit_code, 1)
|
||||
|
||||
def test_latest_step_usage_is_used_only_when_result_omits_usage(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
lines = [
|
||||
json.dumps({"event": "init", "conversation_id": "c", "init": {"model": "Gemini 3.6 Flash"}}),
|
||||
json.dumps({"event": "step_update", "step_update": {"state": "DONE", "step_index": 0, "step_type": "agent_response", "text_delta": "private", "usage": {"input_tokens": 2, "output_tokens": 1, "total_tokens": 3}}}),
|
||||
json.dumps({"event": "result", "result": {"status": "SUCCESS", "duration_seconds": 0.2, "num_turns": 1, "response": "private"}}),
|
||||
]
|
||||
invocation = self._run_lines(lines, parser)
|
||||
self.assertTrue(invocation.product.status == "succeeded")
|
||||
self.assertEqual({m.name: m.value for m in invocation.metrics}["total_tokens"], 3)
|
||||
|
||||
def test_malformed_usage_fails_without_partial_metric(self) -> None:
|
||||
parser = AgyEventParser(_cell(), _binding())
|
||||
invocation = self._run_lines([
|
||||
'{"event":"init","init":{}}',
|
||||
'{"event":"result","result":{"status":"SUCCESS","usage":{"input_tokens":"1"}}}',
|
||||
], parser)
|
||||
self.assertFalse(invocation.product.status == "succeeded")
|
||||
self.assertEqual(invocation.harness.reason, "parser_error")
|
||||
self.assertEqual(invocation.metrics, ())
|
||||
|
||||
def test_structural_redaction_excludes_response_tools_endpoint_and_secret(self) -> None:
|
||||
raw = json.dumps({
|
||||
"event": "result", "result": {
|
||||
"status": "SUCCESS", "response": "raw prompt",
|
||||
"tool_input": {"secret": "x"}, "endpoint": self.runtime.endpoint,
|
||||
"token": self.runtime.credential,
|
||||
},
|
||||
})
|
||||
redacted = redact_agy_event(raw, (self.runtime.endpoint, self.runtime.credential))
|
||||
self.assertEqual(redacted, '{"event":"result","status":"SUCCESS"}')
|
||||
for forbidden in ("raw prompt", "tool_input", self.runtime.endpoint, self.runtime.credential):
|
||||
self.assertNotIn(forbidden, redacted)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,870 +0,0 @@
|
|||
"""Contained loopback Chromium rendering for benchmark web evidence.
|
||||
|
||||
Only regular files beneath one workspace are served. The CDP client is a
|
||||
small, bounded RFC6455 implementation and every browser is launched in an
|
||||
owned process group which is reaped before its temporary profile is removed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import stat
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote_to_bytes, urlparse
|
||||
from urllib.request import urlopen
|
||||
|
||||
from scripts.agent_benchmark.manifest import VIEWPORT_ID_RE
|
||||
|
||||
|
||||
MAX_HANDSHAKE_BYTES = 16 * 1024
|
||||
MAX_MESSAGE_BYTES = 8 * 1024 * 1024
|
||||
MAX_STATIC_BYTES = 32 * 1024 * 1024
|
||||
MAX_PENDING_FIRE_COMMANDS = 4096
|
||||
WEBSOCKET_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
|
||||
|
||||
|
||||
class BrowserError(Exception):
|
||||
"""A redacted renderer/protocol failure safe for durable evidence."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ViewportObservation:
|
||||
id: str
|
||||
width: int
|
||||
height: int
|
||||
screenshot: str
|
||||
screenshot_digest: str
|
||||
screenshot_size: int
|
||||
image_facts: tuple[dict, ...]
|
||||
layout: dict
|
||||
accessibility: dict
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RenderObservation:
|
||||
browser: str
|
||||
origin: str
|
||||
requests: tuple[dict, ...]
|
||||
console: tuple[dict, ...]
|
||||
viewports: tuple[ViewportObservation, ...]
|
||||
|
||||
|
||||
def _digest(data: bytes) -> str:
|
||||
return "sha256:" + hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
def _decode_request_path(raw: str) -> tuple[tuple[str, ...], str]:
|
||||
try:
|
||||
decoded = unquote_to_bytes(raw).decode("utf-8", "strict")
|
||||
except (UnicodeDecodeError, ValueError) as exc:
|
||||
raise FileNotFoundError from exc
|
||||
if not decoded.startswith("/") or "\x00" in decoded or "\\" in decoded:
|
||||
raise FileNotFoundError
|
||||
parts = tuple(decoded[1:].split("/"))
|
||||
if not parts or any(part in ("", ".", "..") for part in parts):
|
||||
raise FileNotFoundError
|
||||
return parts, decoded
|
||||
|
||||
|
||||
def _read_contained_regular_no_follow(root: Path, raw: str) -> tuple[bytes, str]:
|
||||
"""Open every path component relative to ``root`` without following links."""
|
||||
parts, decoded = _decode_request_path(raw)
|
||||
directory_flags = os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
directory_flags |= os.O_NOFOLLOW
|
||||
file_flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
file_flags |= os.O_NOFOLLOW
|
||||
descriptors: list[int] = []
|
||||
try:
|
||||
current = os.open(root, directory_flags)
|
||||
descriptors.append(current)
|
||||
for component in parts[:-1]:
|
||||
current = os.open(component, directory_flags, dir_fd=current)
|
||||
descriptors.append(current)
|
||||
fd = os.open(parts[-1], file_flags, dir_fd=current)
|
||||
descriptors.append(fd)
|
||||
info = os.fstat(fd)
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_size > MAX_STATIC_BYTES:
|
||||
raise FileNotFoundError
|
||||
chunks: list[bytes] = []
|
||||
remaining = info.st_size
|
||||
while remaining:
|
||||
chunk = os.read(fd, min(remaining, 1024 * 1024))
|
||||
if not chunk:
|
||||
raise FileNotFoundError
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
# Refuse a file that grew beyond the bounded snapshot while being read.
|
||||
if os.read(fd, 1):
|
||||
raise FileNotFoundError
|
||||
return b"".join(chunks), Path(decoded).suffix.lower()
|
||||
except (OSError, ValueError) as exc:
|
||||
raise FileNotFoundError from exc
|
||||
finally:
|
||||
for descriptor in reversed(descriptors):
|
||||
try:
|
||||
os.close(descriptor)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
class _StaticServer(ThreadingHTTPServer):
|
||||
daemon_threads = True
|
||||
allow_reuse_address = True
|
||||
|
||||
def __init__(self, root: Path):
|
||||
if root.is_symlink() or not root.is_dir():
|
||||
raise BrowserError("workspace_unavailable")
|
||||
self.root = root.resolve()
|
||||
self.requests: list[dict] = []
|
||||
self._requests_lock = threading.Lock()
|
||||
super().__init__(("127.0.0.1", 0), _StaticHandler)
|
||||
|
||||
def record_request(self, record: dict) -> None:
|
||||
with self._requests_lock:
|
||||
self.requests.append(record)
|
||||
|
||||
def request_snapshot(self) -> tuple[dict, ...]:
|
||||
with self._requests_lock:
|
||||
return tuple(dict(item) for item in self.requests)
|
||||
|
||||
|
||||
class _StaticHandler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *_args):
|
||||
return
|
||||
|
||||
def do_GET(self):
|
||||
server: _StaticServer = self.server # type: ignore[assignment]
|
||||
raw = self.path.split("?", 1)[0].split("#", 1)[0]
|
||||
if raw == "/favicon.ico":
|
||||
server.record_request(
|
||||
{"kind": "local", "path": raw, "allowed": True, "status": 204}
|
||||
)
|
||||
self.send_response(204)
|
||||
self.end_headers()
|
||||
return
|
||||
if raw == "/":
|
||||
raw = "/index.html"
|
||||
try:
|
||||
data, suffix = _read_contained_regular_no_follow(server.root, raw)
|
||||
except FileNotFoundError:
|
||||
server.record_request(
|
||||
{"kind": "local", "path": raw, "allowed": False, "status": 404}
|
||||
)
|
||||
self.send_error(404)
|
||||
return
|
||||
server.record_request(
|
||||
{"kind": "local", "path": raw, "allowed": True, "status": 200}
|
||||
)
|
||||
content_type = {
|
||||
".html": "text/html; charset=utf-8",
|
||||
".css": "text/css; charset=utf-8",
|
||||
".js": "application/javascript; charset=utf-8",
|
||||
".svg": "image/svg+xml",
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
}.get(suffix, "application/octet-stream")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", content_type)
|
||||
self.send_header("Content-Length", str(len(data)))
|
||||
self.end_headers()
|
||||
self.wfile.write(data)
|
||||
|
||||
|
||||
class _CDP:
|
||||
"""Bounded RFC6455 client with strict response/event correlation."""
|
||||
|
||||
def __init__(self, url: str, deadline: float):
|
||||
parsed = urlparse(url)
|
||||
try:
|
||||
address = ipaddress.ip_address(parsed.hostname or "")
|
||||
except ValueError as exc:
|
||||
raise BrowserError("CDP endpoint is not loopback") from exc
|
||||
if parsed.scheme != "ws" or not address.is_loopback or parsed.port is None:
|
||||
raise BrowserError("CDP endpoint is not loopback")
|
||||
self.deadline = deadline
|
||||
self.seq = 0
|
||||
self.events: list[dict] = []
|
||||
self.event_handler = None
|
||||
self._fire_ids: set[int] = set()
|
||||
self._buffer = bytearray()
|
||||
self._closed = False
|
||||
timeout = self._remaining()
|
||||
try:
|
||||
self.sock = socket.create_connection(
|
||||
(parsed.hostname, parsed.port), timeout
|
||||
)
|
||||
key = base64.b64encode(secrets.token_bytes(16)).decode("ascii")
|
||||
request = (
|
||||
f"GET {parsed.path or '/'} HTTP/1.1\r\n"
|
||||
f"Host: {parsed.hostname}:{parsed.port}\r\n"
|
||||
"Upgrade: websocket\r\nConnection: Upgrade\r\n"
|
||||
f"Sec-WebSocket-Key: {key}\r\n"
|
||||
"Sec-WebSocket-Version: 13\r\n\r\n"
|
||||
).encode("ascii")
|
||||
self._sendall(request)
|
||||
self._read_http(key)
|
||||
except Exception:
|
||||
sock = getattr(self, "sock", None)
|
||||
if sock is not None:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
def _remaining(self) -> float:
|
||||
remaining = self.deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise BrowserError("CDP deadline expired")
|
||||
return remaining
|
||||
|
||||
def _read_http(self, key: str) -> None:
|
||||
data = bytearray()
|
||||
while b"\r\n\r\n" not in data:
|
||||
if len(data) >= MAX_HANDSHAKE_BYTES:
|
||||
raise BrowserError("oversized CDP handshake")
|
||||
try:
|
||||
self.sock.settimeout(self._remaining())
|
||||
chunk = self.sock.recv(4096)
|
||||
except (OSError, socket.timeout) as exc:
|
||||
raise BrowserError("CDP handshake failed") from exc
|
||||
if not chunk:
|
||||
raise BrowserError("CDP handshake closed")
|
||||
data.extend(chunk)
|
||||
if len(data) > MAX_HANDSHAKE_BYTES:
|
||||
raise BrowserError("oversized CDP handshake")
|
||||
header, tail = bytes(data).split(b"\r\n\r\n", 1)
|
||||
lines = header.split(b"\r\n")
|
||||
status_parts = lines[0].split(b" ", 2)
|
||||
if len(status_parts) < 2 or status_parts[0] != b"HTTP/1.1" or status_parts[1] != b"101":
|
||||
raise BrowserError("CDP WebSocket upgrade failed")
|
||||
headers: dict[str, str] = {}
|
||||
for line in lines[1:]:
|
||||
if b":" not in line:
|
||||
raise BrowserError("CDP WebSocket headers are invalid")
|
||||
name, value = line.split(b":", 1)
|
||||
try:
|
||||
headers[name.decode("ascii").strip().lower()] = value.decode(
|
||||
"ascii"
|
||||
).strip()
|
||||
except UnicodeDecodeError as exc:
|
||||
raise BrowserError("CDP WebSocket headers are invalid") from exc
|
||||
expected = base64.b64encode(
|
||||
hashlib.sha1((key + WEBSOCKET_GUID).encode("ascii")).digest()
|
||||
).decode("ascii")
|
||||
if (
|
||||
headers.get("upgrade", "").lower() != "websocket"
|
||||
or "upgrade"
|
||||
not in {part.strip().lower() for part in headers.get("connection", "").split(",")}
|
||||
or headers.get("sec-websocket-accept") != expected
|
||||
):
|
||||
raise BrowserError("CDP WebSocket handshake is invalid")
|
||||
self._buffer.extend(tail)
|
||||
|
||||
def _exact(self, size: int) -> bytes:
|
||||
data = bytearray()
|
||||
if self._buffer:
|
||||
take = min(size, len(self._buffer))
|
||||
data.extend(self._buffer[:take])
|
||||
del self._buffer[:take]
|
||||
while len(data) < size:
|
||||
try:
|
||||
self.sock.settimeout(self._remaining())
|
||||
chunk = self.sock.recv(size - len(data))
|
||||
except socket.timeout as exc:
|
||||
raise BrowserError("CDP deadline expired") from exc
|
||||
except OSError as exc:
|
||||
raise BrowserError("CDP socket read failed") from exc
|
||||
if not chunk:
|
||||
raise BrowserError("CDP socket closed")
|
||||
data.extend(chunk)
|
||||
return bytes(data)
|
||||
|
||||
def _frame(self) -> tuple[bool, int, bytes]:
|
||||
first, second = self._exact(2)
|
||||
fin = bool(first & 0x80)
|
||||
if first & 0x70:
|
||||
raise BrowserError("reserved CDP frame bits")
|
||||
opcode = first & 0x0F
|
||||
masked = bool(second & 0x80)
|
||||
if masked:
|
||||
raise BrowserError("masked CDP server frame")
|
||||
length = second & 0x7F
|
||||
if length == 126:
|
||||
length = struct.unpack("!H", self._exact(2))[0]
|
||||
if length < 126:
|
||||
raise BrowserError("non-canonical CDP frame length")
|
||||
elif length == 127:
|
||||
encoded = self._exact(8)
|
||||
if encoded[0] & 0x80:
|
||||
raise BrowserError("invalid CDP frame length")
|
||||
length = struct.unpack("!Q", encoded)[0]
|
||||
if length <= 65535:
|
||||
raise BrowserError("non-canonical CDP frame length")
|
||||
if length > MAX_MESSAGE_BYTES:
|
||||
raise BrowserError("oversized CDP frame")
|
||||
if opcode >= 8 and (not fin or length > 125):
|
||||
raise BrowserError("invalid CDP control frame")
|
||||
return fin, opcode, self._exact(length)
|
||||
|
||||
def receive_message(self) -> dict:
|
||||
fragments = bytearray()
|
||||
fragmented = False
|
||||
while True:
|
||||
fin, opcode, payload = self._frame()
|
||||
if opcode == 8:
|
||||
if len(payload) == 1:
|
||||
raise BrowserError("invalid CDP close frame")
|
||||
try:
|
||||
self._send(payload, opcode=8)
|
||||
except BrowserError:
|
||||
pass
|
||||
raise BrowserError("CDP socket closed")
|
||||
if opcode == 9:
|
||||
self._send(payload, opcode=10)
|
||||
continue
|
||||
if opcode == 10:
|
||||
continue
|
||||
if opcode == 2:
|
||||
raise BrowserError("binary CDP message")
|
||||
if opcode == 1:
|
||||
if fragmented:
|
||||
raise BrowserError("interleaved CDP data frame")
|
||||
fragments.extend(payload)
|
||||
fragmented = not fin
|
||||
elif opcode == 0:
|
||||
if not fragmented:
|
||||
raise BrowserError("unexpected CDP continuation")
|
||||
fragments.extend(payload)
|
||||
fragmented = not fin
|
||||
else:
|
||||
raise BrowserError("unsupported CDP frame")
|
||||
if len(fragments) > MAX_MESSAGE_BYTES:
|
||||
raise BrowserError("oversized CDP message")
|
||||
if fragmented:
|
||||
continue
|
||||
try:
|
||||
message = json.loads(fragments.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise BrowserError("malformed CDP message") from exc
|
||||
if not isinstance(message, dict):
|
||||
raise BrowserError("malformed CDP message")
|
||||
return message
|
||||
|
||||
def _sendall(self, data: bytes) -> None:
|
||||
try:
|
||||
self.sock.settimeout(self._remaining())
|
||||
self.sock.sendall(data)
|
||||
except socket.timeout as exc:
|
||||
raise BrowserError("CDP deadline expired") from exc
|
||||
except OSError as exc:
|
||||
raise BrowserError("CDP socket write failed") from exc
|
||||
|
||||
def _send(self, data: bytes, *, opcode: int = 1) -> None:
|
||||
if len(data) > MAX_MESSAGE_BYTES:
|
||||
raise BrowserError("oversized CDP client message")
|
||||
mask = secrets.token_bytes(4)
|
||||
size = len(data)
|
||||
if size < 126:
|
||||
header = bytes((0x80 | opcode, 0x80 | size))
|
||||
elif size <= 65535:
|
||||
header = bytes((0x80 | opcode, 0x80 | 126)) + struct.pack("!H", size)
|
||||
else:
|
||||
header = bytes((0x80 | opcode, 0x80 | 127)) + struct.pack("!Q", size)
|
||||
masked = bytes(value ^ mask[index % 4] for index, value in enumerate(data))
|
||||
self._sendall(header + mask + masked)
|
||||
|
||||
@staticmethod
|
||||
def _command(ident: int, method: str, params: dict | None) -> bytes:
|
||||
return json.dumps(
|
||||
{"id": ident, "method": method, "params": params or {}},
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
|
||||
def call(self, method: str, params: dict | None = None) -> dict:
|
||||
self.seq += 1
|
||||
ident = self.seq
|
||||
self._send(self._command(ident, method, params))
|
||||
while True:
|
||||
message = self.receive_message()
|
||||
response_id = message.get("id")
|
||||
if response_id is not None:
|
||||
if isinstance(response_id, bool) or not isinstance(response_id, int):
|
||||
raise BrowserError("invalid CDP response id")
|
||||
if response_id in self._fire_ids:
|
||||
self._fire_ids.remove(response_id)
|
||||
continue
|
||||
if response_id != ident:
|
||||
raise BrowserError("uncorrelated CDP response")
|
||||
if "error" in message:
|
||||
raise BrowserError(f"CDP {method} failed")
|
||||
result = message.get("result", {})
|
||||
if not isinstance(result, dict):
|
||||
raise BrowserError("invalid CDP result")
|
||||
return result
|
||||
if not isinstance(message.get("method"), str) or not isinstance(
|
||||
message.get("params", {}), dict
|
||||
):
|
||||
raise BrowserError("invalid CDP event")
|
||||
self.events.append(message)
|
||||
if self.event_handler is not None:
|
||||
self.event_handler(message)
|
||||
|
||||
def fire(self, method: str, params: dict | None = None) -> None:
|
||||
"""Send an event-callback command without entering a nested receive loop."""
|
||||
if len(self._fire_ids) >= MAX_PENDING_FIRE_COMMANDS:
|
||||
raise BrowserError("too many pending CDP commands")
|
||||
self.seq += 1
|
||||
self._fire_ids.add(self.seq)
|
||||
self._send(self._command(self.seq, method, params))
|
||||
|
||||
def close(self) -> None:
|
||||
if self._closed:
|
||||
return
|
||||
self._closed = True
|
||||
try:
|
||||
self.sock.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
self.sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
_OBSERVATION_SCRIPT = r"""(() => {
|
||||
const rect = e => { const r=e.getBoundingClientRect(); return {x:r.x,y:r.y,width:r.width,height:r.height,right:r.right,bottom:r.bottom}; };
|
||||
const visible = e => { const r=e.getBoundingClientRect(),s=getComputedStyle(e); return r.width>0&&r.height>0&&s.display!=='none'&&s.visibility!=='hidden'&&Number(s.opacity)>0; };
|
||||
const controls=[...document.querySelectorAll('a[href],button,input,select,textarea,[tabindex]')].filter(e=>!e.disabled&&visible(e));
|
||||
const clipped=[...document.querySelectorAll('img,a[href],button,input,select,textarea,[tabindex]')].filter(visible).filter(e=>{const r=e.getBoundingClientRect();return r.left<0||r.right>innerWidth;}).length;
|
||||
let overlaps=0; for(let i=0;i<controls.length;i++)for(let j=i+1;j<controls.length;j++){let a=controls[i].getBoundingClientRect(),b=controls[j].getBoundingClientRect();if(Math.min(a.right,b.right)>Math.max(a.left,b.left)&&Math.min(a.bottom,b.bottom)>Math.max(a.top,b.top))overlaps++;}
|
||||
const name=e=>(e.getAttribute('aria-label')||e.getAttribute('alt')||e.getAttribute('title')||e.value||e.textContent||'').trim();
|
||||
const rgba=s=>{let m=s.match(/[\d.]+/g)||[];return m.slice(0,4).map(Number);};
|
||||
const lum=c=>{c=c/255;return c<=.03928?c/12.92:Math.pow((c+.055)/1.055,2.4);};
|
||||
const contrast=e=>{let s=getComputedStyle(e),f=rgba(s.color),b=rgba(s.backgroundColor),p=e.parentElement;while((b.length<3||(b.length>=4&&b[3]===0))&&p){b=rgba(getComputedStyle(p).backgroundColor);p=p.parentElement;}if(f.length<3||b.length<3)return 0;let a=.2126*lum(f[0])+.7152*lum(f[1])+.0722*lum(f[2]),z=.2126*lum(b[0])+.7152*lum(b[1])+.0722*lum(b[2]);return (Math.max(a,z)+.05)/(Math.min(a,z)+.05);};
|
||||
const focusStyle=e=>{let s=getComputedStyle(e);return {outline_style:s.outlineStyle,outline_width:s.outlineWidth,outline_color:s.outlineColor,outline_offset:s.outlineOffset,box_shadow:s.boxShadow,background_color:s.backgroundColor,background_image:s.backgroundImage,border_top:[s.borderTopWidth,s.borderTopStyle,s.borderTopColor],border_right:[s.borderRightWidth,s.borderRightStyle,s.borderRightColor],border_bottom:[s.borderBottomWidth,s.borderBottomStyle,s.borderBottomColor],border_left:[s.borderLeftWidth,s.borderLeftStyle,s.borderLeftColor]};};
|
||||
const changed=(before,after,keys)=>keys.some(key=>before[key]!==after[key]);
|
||||
const colorProbe=document.createElement('canvas').getContext('2d');
|
||||
const paintAlpha=value=>{if(!colorProbe)return 0;colorProbe.fillStyle='rgba(0,0,0,0)';colorProbe.fillStyle=value;let normalized=colorProbe.fillStyle,m=normalized.match(/^rgba?\(([^)]+)\)$/i);if(m){let parts=m[1].replace(/\//g,' ').split(/[,\s]+/).filter(Boolean);return normalized.toLowerCase().startsWith('rgba')?Number(parts[3]||0):1;}if(/^#[0-9a-f]{6}$/i.test(normalized))return 1;if(/^#[0-9a-f]{8}$/i.test(normalized))return parseInt(normalized.slice(7,9),16)/255;return 0;};
|
||||
const splitShadows=value=>{let result=[],start=0,depth=0;for(let i=0;i<value.length;i++){if(value[i]==='(')depth++;else if(value[i]===')')depth--;else if(value[i]===','&&depth===0){result.push(value.slice(start,i));start=i+1;}}result.push(value.slice(start));return result;};
|
||||
const paintedOutlineChanged=(before,after)=>(parseFloat(after.outline_width)||0)>0&&after.outline_style!=='none'&&after.outline_style!=='hidden'&&paintAlpha(after.outline_color)>0&&changed(before,after,['outline_style','outline_width','outline_color','outline_offset']);
|
||||
const paintedShadowChanged=(before,after)=>before.box_shadow!==after.box_shadow&&after.box_shadow!=='none'&&splitShadows(after.box_shadow).some(shadow=>{let color=(shadow.match(/rgba?\([^)]*\)|#[0-9a-f]{3,8}\b/ig)||[])[0]||'transparent';return paintAlpha(color)>0&&(shadow.match(/-?\d+(?:\.\d+)?px/g)||[]).some(value=>Math.abs(parseFloat(value))>0);});
|
||||
const paintedBorderChanged=(before,after)=>['border_top','border_right','border_bottom','border_left'].some(key=>{let current=after[key],prior=before[key];return (parseFloat(current[0])||0)>0&¤t[1]!=='none'&¤t[1]!=='hidden'&&paintAlpha(current[2])>0&¤t.some((value,index)=>value!==prior[index]);});
|
||||
const paintedBackgroundColorChanged=(before,after)=>before.background_color!==after.background_color&&paintAlpha(after.background_color)>0;
|
||||
const unfocus=()=>{let active=document.activeElement;if(active&&typeof active.blur==='function')active.blur();};
|
||||
const focus=controls.map(e=>{unfocus();e.blur();let before=focusStyle(e);e.focus();let after=focusStyle(e),focused=document.activeElement===e;let indicator=paintedOutlineChanged(before,after)||paintedShadowChanged(before,after)||paintedBorderChanged(before,after)||paintedBackgroundColorChanged(before,after);return {name:!!name(e),tab_index:e.tabIndex,focused,focus_visible:focused&&indicator,contrast:contrast(e)};});
|
||||
const headings=[...document.querySelectorAll('h1,h2,h3,h4,h5,h6')].map(e=>Number(e.tagName.slice(1)));
|
||||
let heading_progression=true; for(let i=1;i<headings.length;i++)if(headings[i]>headings[i-1]+1)heading_progression=false;
|
||||
return {
|
||||
images:[...document.images].map(e=>({src:e.getAttribute('src')||'',alt:e.alt||'',complete:e.complete,natural_width:e.naturalWidth,natural_height:e.naturalHeight,visible:visible(e),rect:rect(e)})),
|
||||
layout:{scroll_width:document.documentElement.scrollWidth,client_width:document.documentElement.clientWidth,clipped,overlaps},
|
||||
accessibility:{h1_count:document.querySelectorAll('h1').length,headings,heading_progression,main_count:document.querySelectorAll('main').length,landmarks:document.querySelectorAll('main,nav,header,footer,[role="main"],[role="navigation"],[role="banner"],[role="contentinfo"]').length,controls:focus}
|
||||
}; })()"""
|
||||
|
||||
|
||||
def _write_new(path: Path, data: bytes) -> None:
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_CLOEXEC
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
try:
|
||||
fd = os.open(path, flags, 0o600)
|
||||
except OSError as exc:
|
||||
raise BrowserError("screenshot_collision") from exc
|
||||
try:
|
||||
with os.fdopen(fd, "wb", closefd=True) as handle:
|
||||
handle.write(data)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
except Exception:
|
||||
try:
|
||||
path.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def _terminate_owned_process_group(process: subprocess.Popen, grace: float = 3.0) -> None:
|
||||
"""Terminate then kill the session led by ``process`` and reap its leader."""
|
||||
pgid = process.pid
|
||||
if pgid == os.getpgrp():
|
||||
raise BrowserError("browser process group ownership is invalid")
|
||||
try:
|
||||
os.killpg(pgid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
except OSError as exc:
|
||||
raise BrowserError("browser process cleanup failed") from exc
|
||||
try:
|
||||
process.wait(timeout=grace)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
except OSError as exc:
|
||||
raise BrowserError("browser process cleanup failed") from exc
|
||||
try:
|
||||
process.wait(timeout=grace)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise BrowserError("browser process cleanup failed") from exc
|
||||
# The leader can exit before a descendant. Kill any surviving member and
|
||||
# give the kernel a bounded moment to reap it under its own parent.
|
||||
try:
|
||||
os.killpg(pgid, 0)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
except OSError:
|
||||
return
|
||||
try:
|
||||
os.killpg(pgid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
end = time.monotonic() + grace
|
||||
while time.monotonic() < end:
|
||||
try:
|
||||
os.killpg(pgid, 0)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
except OSError:
|
||||
return
|
||||
time.sleep(0.02)
|
||||
raise BrowserError("browser process cleanup failed")
|
||||
|
||||
|
||||
def _console_projection(events: list[dict]) -> tuple[dict, ...]:
|
||||
projected: list[dict] = []
|
||||
for event in events:
|
||||
method = event.get("method")
|
||||
params = event.get("params", {})
|
||||
if method == "Runtime.consoleAPICalled":
|
||||
projected.append({"kind": "console", "level": str(params.get("type", "unknown"))[:32]})
|
||||
elif method == "Runtime.exceptionThrown":
|
||||
projected.append({"kind": "exception", "level": "error"})
|
||||
elif method == "Log.entryAdded":
|
||||
entry = params.get("entry", {})
|
||||
projected.append({"kind": "log", "level": str(entry.get("level", "unknown"))[:32]})
|
||||
return tuple(projected)
|
||||
|
||||
|
||||
class BrowserRenderer:
|
||||
def __init__(self, browser_binary: str = "chromium"):
|
||||
self.browser_binary = browser_binary
|
||||
|
||||
def render(
|
||||
self,
|
||||
*,
|
||||
workspace_root: str | Path,
|
||||
output_root: str | Path,
|
||||
viewports,
|
||||
timeout_seconds: int,
|
||||
) -> RenderObservation:
|
||||
raw_root, raw_out = Path(workspace_root), Path(output_root)
|
||||
if raw_root.is_symlink() or not raw_root.is_dir():
|
||||
raise BrowserError("workspace_unavailable")
|
||||
if raw_out.is_symlink() or not raw_out.is_dir():
|
||||
raise BrowserError("output_unavailable")
|
||||
root, out = raw_root.resolve(), raw_out.resolve()
|
||||
viewport_list = tuple(viewports)
|
||||
if not viewport_list:
|
||||
raise BrowserError("viewport_unavailable")
|
||||
for viewport in viewport_list:
|
||||
if (
|
||||
not isinstance(getattr(viewport, "id", None), str)
|
||||
or not VIEWPORT_ID_RE.fullmatch(viewport.id)
|
||||
or isinstance(getattr(viewport, "width", None), bool)
|
||||
or not isinstance(viewport.width, int)
|
||||
or not 1 <= viewport.width <= 10000
|
||||
or isinstance(getattr(viewport, "height", None), bool)
|
||||
or not isinstance(viewport.height, int)
|
||||
or not 1 <= viewport.height <= 10000
|
||||
):
|
||||
raise BrowserError("viewport_unavailable")
|
||||
targets = [out / f"screenshot-{viewport.id}.png" for viewport in viewport_list]
|
||||
if len(set(targets)) != len(targets) or any(
|
||||
target.exists() or target.is_symlink() for target in targets
|
||||
):
|
||||
raise BrowserError("screenshot_collision")
|
||||
binary = self.browser_binary
|
||||
if os.sep not in binary:
|
||||
binary = shutil.which(binary) or ""
|
||||
if not binary:
|
||||
raise BrowserError("browser_unavailable")
|
||||
transient = {
|
||||
"browser_start_failed",
|
||||
"browser_cdp_unavailable",
|
||||
"CDP handshake failed",
|
||||
"CDP handshake closed",
|
||||
"CDP socket read failed",
|
||||
"CDP socket closed",
|
||||
"CDP socket write failed",
|
||||
}
|
||||
for attempt in range(3):
|
||||
try:
|
||||
return self._render_once(
|
||||
root=root,
|
||||
out=out,
|
||||
viewport_list=viewport_list,
|
||||
targets=targets,
|
||||
binary=binary,
|
||||
timeout_seconds=timeout_seconds,
|
||||
)
|
||||
except BrowserError as exc:
|
||||
if str(exc) not in transient or attempt == 2:
|
||||
raise
|
||||
self._remove_partial_screenshots(out, viewport_list)
|
||||
raise BrowserError("browser_restart_exhausted")
|
||||
|
||||
@staticmethod
|
||||
def _remove_partial_screenshots(output_root: Path, viewports) -> None:
|
||||
for viewport in viewports:
|
||||
target = output_root / f"screenshot-{getattr(viewport, 'id', '')}.png"
|
||||
try:
|
||||
if target.is_file() and not target.is_symlink():
|
||||
target.unlink()
|
||||
elif target.exists() or target.is_symlink():
|
||||
raise BrowserError("screenshot_cleanup_failed")
|
||||
except OSError as exc:
|
||||
raise BrowserError("screenshot_cleanup_failed") from exc
|
||||
|
||||
def _render_once(
|
||||
self,
|
||||
*,
|
||||
root: Path,
|
||||
out: Path,
|
||||
viewport_list,
|
||||
targets: list[Path],
|
||||
binary: str,
|
||||
timeout_seconds: int,
|
||||
) -> RenderObservation:
|
||||
deadline = time.monotonic() + max(3, timeout_seconds)
|
||||
server = _StaticServer(root)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
profile = tempfile.TemporaryDirectory(
|
||||
prefix="iop-browser-", ignore_cleanup_errors=True
|
||||
)
|
||||
process: subprocess.Popen | None = None
|
||||
cdp: _CDP | None = None
|
||||
cleanup_error: Exception | None = None
|
||||
completed = False
|
||||
try:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as port_socket:
|
||||
port_socket.bind(("127.0.0.1", 0))
|
||||
port = int(port_socket.getsockname()[1])
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
[
|
||||
binary,
|
||||
"--headless",
|
||||
"--no-sandbox",
|
||||
"--disable-gpu",
|
||||
"--disable-extensions",
|
||||
"--disable-background-networking",
|
||||
"--no-first-run",
|
||||
"--remote-allow-origins=*",
|
||||
"--remote-debugging-address=127.0.0.1",
|
||||
f"--remote-debugging-port={port}",
|
||||
f"--user-data-dir={profile.name}",
|
||||
"about:blank",
|
||||
],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
start_new_session=True,
|
||||
)
|
||||
except OSError as exc:
|
||||
raise BrowserError("browser_start_failed") from exc
|
||||
target = None
|
||||
while time.monotonic() < deadline:
|
||||
if process.poll() is not None:
|
||||
raise BrowserError("browser_start_failed")
|
||||
try:
|
||||
with urlopen(
|
||||
f"http://127.0.0.1:{port}/json", timeout=0.5
|
||||
) as response:
|
||||
candidates = json.load(response)
|
||||
if not isinstance(candidates, list):
|
||||
raise ValueError("invalid target list")
|
||||
target = next(
|
||||
(
|
||||
item
|
||||
for item in candidates
|
||||
if isinstance(item, dict) and item.get("type") == "page"
|
||||
),
|
||||
None,
|
||||
)
|
||||
if target:
|
||||
break
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
time.sleep(0.05)
|
||||
if not target or not isinstance(target.get("webSocketDebuggerUrl"), str):
|
||||
raise BrowserError("browser_cdp_unavailable")
|
||||
cdp = _CDP(target["webSocketDebuggerUrl"], deadline)
|
||||
for method in (
|
||||
"Page.enable",
|
||||
"Runtime.enable",
|
||||
"Log.enable",
|
||||
"Accessibility.enable",
|
||||
):
|
||||
cdp.call(method)
|
||||
cdp.call("Page.bringToFront")
|
||||
version = str(cdp.call("Browser.getVersion").get("product", "unknown"))[:128]
|
||||
origin = f"http://127.0.0.1:{server.server_address[1]}"
|
||||
denied: list[dict] = []
|
||||
|
||||
def intercept(event):
|
||||
if event.get("method") != "Fetch.requestPaused":
|
||||
return
|
||||
params = event.get("params", {})
|
||||
request = params.get("request", {})
|
||||
url = request.get("url", "")
|
||||
request_id = params.get("requestId")
|
||||
if not isinstance(request_id, str) or not request_id:
|
||||
raise BrowserError("browser request identity is invalid")
|
||||
if isinstance(url, str) and url.startswith(origin + "/"):
|
||||
cdp.fire(
|
||||
"Fetch.continueRequest", {"requestId": request_id}
|
||||
)
|
||||
else:
|
||||
encoded = str(url).encode("utf-8", "replace")
|
||||
denied.append(
|
||||
{
|
||||
"kind": "external",
|
||||
"url_digest": _digest(encoded),
|
||||
"allowed": False,
|
||||
"status": 0,
|
||||
}
|
||||
)
|
||||
cdp.fire(
|
||||
"Fetch.failRequest",
|
||||
{
|
||||
"requestId": request_id,
|
||||
"errorReason": "BlockedByClient",
|
||||
},
|
||||
)
|
||||
|
||||
cdp.event_handler = intercept
|
||||
cdp.call(
|
||||
"Fetch.enable",
|
||||
{"patterns": [{"urlPattern": "*", "requestStage": "Request"}]},
|
||||
)
|
||||
results: list[ViewportObservation] = []
|
||||
for viewport, screenshot in zip(viewport_list, targets):
|
||||
cdp.call(
|
||||
"Emulation.setDeviceMetricsOverride",
|
||||
{
|
||||
"width": viewport.width,
|
||||
"height": viewport.height,
|
||||
"deviceScaleFactor": 1,
|
||||
"mobile": viewport.width < 600,
|
||||
},
|
||||
)
|
||||
cdp.call("Page.navigate", {"url": origin + "/index.html"})
|
||||
cdp.call(
|
||||
"Runtime.evaluate",
|
||||
{
|
||||
"expression": "new Promise((resolve,reject)=>{const done=()=>requestAnimationFrame(()=>requestAnimationFrame(resolve));if(document.readyState==='complete')done();else window.addEventListener('load',done,{once:true});setTimeout(()=>reject(new Error('load timeout')),5000);})",
|
||||
"awaitPromise": True,
|
||||
"returnByValue": True,
|
||||
},
|
||||
)
|
||||
facts = cdp.call(
|
||||
"Runtime.evaluate",
|
||||
{"expression": _OBSERVATION_SCRIPT, "returnByValue": True},
|
||||
)
|
||||
parsed = facts.get("result", {}).get("value", {})
|
||||
if not isinstance(parsed, dict) or set(parsed) != {
|
||||
"images",
|
||||
"layout",
|
||||
"accessibility",
|
||||
}:
|
||||
raise BrowserError("browser_observation_invalid")
|
||||
ax_nodes = cdp.call("Accessibility.getFullAXTree").get("nodes", [])
|
||||
if not isinstance(ax_nodes, list):
|
||||
raise BrowserError("browser_accessibility_invalid")
|
||||
ax = {
|
||||
"nodes": len(ax_nodes),
|
||||
"non_ignored": sum(
|
||||
1 for node in ax_nodes if isinstance(node, dict) and not node.get("ignored", False)
|
||||
),
|
||||
"named": sum(
|
||||
1
|
||||
for node in ax_nodes
|
||||
if isinstance(node, dict)
|
||||
and isinstance(node.get("name"), dict)
|
||||
and bool(node["name"].get("value"))
|
||||
),
|
||||
}
|
||||
accessibility = dict(parsed["accessibility"])
|
||||
accessibility["ax"] = ax
|
||||
try:
|
||||
png = base64.b64decode(
|
||||
cdp.call("Page.captureScreenshot", {"format": "png"})["data"],
|
||||
validate=True,
|
||||
)
|
||||
except (KeyError, ValueError, TypeError) as exc:
|
||||
raise BrowserError("browser_screenshot_invalid") from exc
|
||||
if not png.startswith(b"\x89PNG\r\n\x1a\n"):
|
||||
raise BrowserError("browser_screenshot_invalid")
|
||||
_write_new(screenshot, png)
|
||||
results.append(
|
||||
ViewportObservation(
|
||||
viewport.id,
|
||||
viewport.width,
|
||||
viewport.height,
|
||||
screenshot.name,
|
||||
_digest(png),
|
||||
len(png),
|
||||
tuple(parsed["images"]),
|
||||
parsed["layout"],
|
||||
accessibility,
|
||||
)
|
||||
)
|
||||
observation = RenderObservation(
|
||||
version,
|
||||
origin,
|
||||
tuple([*server.request_snapshot(), *denied]),
|
||||
_console_projection(cdp.events),
|
||||
tuple(results),
|
||||
)
|
||||
completed = True
|
||||
return observation
|
||||
finally:
|
||||
if cdp is not None:
|
||||
cdp.close()
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(2)
|
||||
if process is not None:
|
||||
try:
|
||||
_terminate_owned_process_group(process)
|
||||
except Exception as exc: # preserve cleanup failure after body success
|
||||
cleanup_error = exc
|
||||
if not completed:
|
||||
for target in targets:
|
||||
try:
|
||||
if target.is_file() and not target.is_symlink():
|
||||
target.unlink()
|
||||
except OSError:
|
||||
cleanup_error = cleanup_error or BrowserError(
|
||||
"screenshot_cleanup_failed"
|
||||
)
|
||||
profile.cleanup()
|
||||
if cleanup_error is not None:
|
||||
raise cleanup_error
|
||||
|
|
@ -1,591 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import socket
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
|
||||
from scripts.agent_benchmark.browser_cdp import (
|
||||
MAX_MESSAGE_BYTES,
|
||||
WEBSOCKET_GUID,
|
||||
BrowserError,
|
||||
BrowserRenderer,
|
||||
_CDP,
|
||||
_StaticServer,
|
||||
_terminate_owned_process_group,
|
||||
)
|
||||
from scripts.agent_benchmark.web_validation import _runtime_gates
|
||||
|
||||
|
||||
def _frame(payload: bytes, *, opcode: int = 1, fin: bool = True, masked: bool = False) -> bytes:
|
||||
first = (0x80 if fin else 0) | opcode
|
||||
size = len(payload)
|
||||
if size < 126:
|
||||
header = bytes((first, (0x80 if masked else 0) | size))
|
||||
elif size <= 65535:
|
||||
header = bytes((first, (0x80 if masked else 0) | 126)) + struct.pack("!H", size)
|
||||
else:
|
||||
header = bytes((first, (0x80 if masked else 0) | 127)) + struct.pack("!Q", size)
|
||||
if not masked:
|
||||
return header + payload
|
||||
mask = b"mask"
|
||||
return header + mask + bytes(value ^ mask[index % 4] for index, value in enumerate(payload))
|
||||
|
||||
|
||||
def _exact(connection: socket.socket, size: int) -> bytes:
|
||||
chunks = bytearray()
|
||||
while len(chunks) < size:
|
||||
chunk = connection.recv(size - len(chunks))
|
||||
if not chunk:
|
||||
raise EOFError
|
||||
chunks.extend(chunk)
|
||||
return bytes(chunks)
|
||||
|
||||
|
||||
def _client_frame(connection: socket.socket) -> tuple[int, bytes]:
|
||||
first, second = _exact(connection, 2)
|
||||
size = second & 0x7F
|
||||
if size == 126:
|
||||
size = struct.unpack("!H", _exact(connection, 2))[0]
|
||||
elif size == 127:
|
||||
size = struct.unpack("!Q", _exact(connection, 8))[0]
|
||||
if not second & 0x80:
|
||||
raise AssertionError("client frame was not masked")
|
||||
mask = _exact(connection, 4)
|
||||
payload = _exact(connection, size)
|
||||
return first & 0x0F, bytes(
|
||||
value ^ mask[index % 4] for index, value in enumerate(payload)
|
||||
)
|
||||
|
||||
|
||||
class _WebSocketFixture:
|
||||
def __init__(self, script, *, valid_accept: bool = True):
|
||||
self.script = script
|
||||
self.valid_accept = valid_accept
|
||||
self.error: BaseException | None = None
|
||||
self.listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
self.listener.bind(("127.0.0.1", 0))
|
||||
self.listener.listen(1)
|
||||
self.thread = threading.Thread(target=self._serve, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
@property
|
||||
def url(self) -> str:
|
||||
return f"ws://127.0.0.1:{self.listener.getsockname()[1]}/devtools/page/1"
|
||||
|
||||
def _serve(self) -> None:
|
||||
try:
|
||||
connection, _ = self.listener.accept()
|
||||
with connection:
|
||||
request = bytearray()
|
||||
while b"\r\n\r\n" not in request:
|
||||
request.extend(connection.recv(4096))
|
||||
key = ""
|
||||
for line in bytes(request).split(b"\r\n"):
|
||||
if line.lower().startswith(b"sec-websocket-key:"):
|
||||
key = line.split(b":", 1)[1].strip().decode("ascii")
|
||||
accept = base64.b64encode(
|
||||
hashlib.sha1((key + WEBSOCKET_GUID).encode("ascii")).digest()
|
||||
).decode("ascii")
|
||||
if not self.valid_accept:
|
||||
accept = "invalid"
|
||||
connection.sendall(
|
||||
(
|
||||
"HTTP/1.1 101 Switching Protocols\r\n"
|
||||
"Upgrade: websocket\r\nConnection: keep-alive, Upgrade\r\n"
|
||||
f"Sec-WebSocket-Accept: {accept}\r\n\r\n"
|
||||
).encode("ascii")
|
||||
)
|
||||
if self.valid_accept:
|
||||
self.script(connection)
|
||||
except BaseException as exc: # surfaced by close()
|
||||
self.error = exc
|
||||
finally:
|
||||
self.listener.close()
|
||||
|
||||
def close(self) -> None:
|
||||
self.thread.join(2)
|
||||
if self.thread.is_alive():
|
||||
self.listener.close()
|
||||
self.thread.join(2)
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
|
||||
|
||||
class BrowserProtocolTest(unittest.TestCase):
|
||||
def test_loopback_handler_rejects_escape_and_all_symlinks(self):
|
||||
with tempfile.TemporaryDirectory() as raw:
|
||||
root = Path(raw)
|
||||
(root / "nested").mkdir()
|
||||
(root / "index.html").write_text("ok", encoding="utf-8")
|
||||
(root / "nested" / "ok.txt").write_text("nested", encoding="utf-8")
|
||||
outside = root.parent / f"browser-cdp-outside-{root.name}.txt"
|
||||
outside.write_text("no", encoding="utf-8")
|
||||
(root / "internal-link").symlink_to(root / "index.html")
|
||||
(root / "outside-link").symlink_to(outside)
|
||||
(root / "nested-link").symlink_to(root / "nested", target_is_directory=True)
|
||||
server = _StaticServer(root)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
base = f"http://127.0.0.1:{server.server_address[1]}"
|
||||
with urlopen(base + "/index.html") as response:
|
||||
self.assertEqual(response.read(), b"ok")
|
||||
for path in (
|
||||
"/internal-link",
|
||||
"/outside-link",
|
||||
"/nested-link/ok.txt",
|
||||
"/%2e%2e/index.html",
|
||||
"/nested/%2e%2e/index.html",
|
||||
):
|
||||
with self.subTest(path=path), self.assertRaises(HTTPError):
|
||||
urlopen(base + path)
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
thread.join(2)
|
||||
outside.unlink(missing_ok=True)
|
||||
|
||||
def test_fragment_ping_event_and_response_correlation(self):
|
||||
def script(connection: socket.socket) -> None:
|
||||
opcode, payload = _client_frame(connection)
|
||||
command = json.loads(payload)
|
||||
self.assertEqual(opcode, 1)
|
||||
connection.sendall(_frame(b'{"method":"Page.ready","params":{"ok":true}}'))
|
||||
connection.sendall(_frame(b"ping", opcode=9))
|
||||
response = json.dumps({"id": command["id"], "result": {"ok": True}}).encode()
|
||||
split = len(response) // 2
|
||||
connection.sendall(_frame(response[:split], fin=False))
|
||||
connection.sendall(_frame(response[split:], opcode=0))
|
||||
pong, pong_payload = _client_frame(connection)
|
||||
self.assertEqual((pong, pong_payload), (10, b"ping"))
|
||||
|
||||
server = _WebSocketFixture(script)
|
||||
client = _CDP(server.url, time.monotonic() + 2)
|
||||
try:
|
||||
self.assertEqual(client.call("Page.enable"), {"ok": True})
|
||||
self.assertEqual(client.events[0]["method"], "Page.ready")
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_fire_and_call_responses_can_be_interleaved(self):
|
||||
def script(connection: socket.socket) -> None:
|
||||
_, first = _client_frame(connection)
|
||||
_, second = _client_frame(connection)
|
||||
fire, call = json.loads(first), json.loads(second)
|
||||
connection.sendall(_frame(json.dumps({"id": fire["id"], "result": {}}).encode()))
|
||||
connection.sendall(_frame(b'{"method":"Fetch.paused","params":{}}'))
|
||||
connection.sendall(_frame(json.dumps({"id": call["id"], "result": {"done": 1}}).encode()))
|
||||
|
||||
server = _WebSocketFixture(script)
|
||||
client = _CDP(server.url, time.monotonic() + 2)
|
||||
try:
|
||||
client.fire("Fetch.continueRequest", {"requestId": "request"})
|
||||
self.assertEqual(client.call("Runtime.evaluate"), {"done": 1})
|
||||
self.assertEqual(client.events[0]["method"], "Fetch.paused")
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_extended_frame_lengths_are_canonical_and_bounded(self):
|
||||
for size in (200, 70_000):
|
||||
with self.subTest(size=size):
|
||||
value = "x" * size
|
||||
|
||||
def script(connection: socket.socket, value=value) -> None:
|
||||
_, payload = _client_frame(connection)
|
||||
command = json.loads(payload)
|
||||
response = json.dumps(
|
||||
{"id": command["id"], "result": {"value": value}},
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
connection.sendall(_frame(response))
|
||||
|
||||
server = _WebSocketFixture(script)
|
||||
client = _CDP(server.url, time.monotonic() + 2)
|
||||
try:
|
||||
self.assertEqual(
|
||||
client.call("Runtime.evaluate")["value"], value
|
||||
)
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_handshake_and_malformed_frame_matrix_fail_closed(self):
|
||||
bad_accept = _WebSocketFixture(lambda _connection: None, valid_accept=False)
|
||||
with self.assertRaises(BrowserError):
|
||||
_CDP(bad_accept.url, time.monotonic() + 1)
|
||||
bad_accept.close()
|
||||
|
||||
cases = {
|
||||
"masked": _frame(b"{}", masked=True),
|
||||
"binary": _frame(b"{}", opcode=2),
|
||||
"continuation": _frame(b"{}", opcode=0),
|
||||
"malformed-json": _frame(b"{"),
|
||||
"close": _frame(b"", opcode=8),
|
||||
"oversized": bytes((0x81, 127)) + struct.pack("!Q", MAX_MESSAGE_BYTES + 1),
|
||||
"noncanonical": bytes((0x81, 126)) + struct.pack("!H", 1) + b"x",
|
||||
}
|
||||
for name, response in cases.items():
|
||||
with self.subTest(name=name):
|
||||
def script(connection: socket.socket, response=response) -> None:
|
||||
_client_frame(connection)
|
||||
connection.sendall(response)
|
||||
|
||||
server = _WebSocketFixture(script)
|
||||
client = _CDP(server.url, time.monotonic() + 1)
|
||||
try:
|
||||
with self.assertRaises(BrowserError):
|
||||
client.call("Runtime.evaluate")
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_uncorrelated_response_and_deadline_fail_closed(self):
|
||||
def wrong_id(connection: socket.socket) -> None:
|
||||
_, payload = _client_frame(connection)
|
||||
ident = json.loads(payload)["id"]
|
||||
connection.sendall(_frame(json.dumps({"id": ident + 1, "result": {}}).encode()))
|
||||
|
||||
server = _WebSocketFixture(wrong_id)
|
||||
client = _CDP(server.url, time.monotonic() + 1)
|
||||
try:
|
||||
with self.assertRaises(BrowserError):
|
||||
client.call("Runtime.evaluate")
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def no_response(connection: socket.socket) -> None:
|
||||
_client_frame(connection)
|
||||
time.sleep(0.2)
|
||||
|
||||
server = _WebSocketFixture(no_response)
|
||||
client = _CDP(server.url, time.monotonic() + 0.05)
|
||||
try:
|
||||
with self.assertRaises(BrowserError):
|
||||
client.call("Runtime.evaluate")
|
||||
finally:
|
||||
client.close()
|
||||
server.close()
|
||||
|
||||
def test_owned_process_group_is_terminated_and_reaped(self):
|
||||
source = (
|
||||
"import signal,subprocess,sys,time;"
|
||||
"signal.signal(signal.SIGTERM,signal.SIG_IGN);"
|
||||
"p=subprocess.Popen([sys.executable,'-c',"
|
||||
"'import signal,time;signal.signal(signal.SIGTERM,signal.SIG_IGN);time.sleep(60)']);"
|
||||
"print(p.pid,flush=True);time.sleep(60)"
|
||||
)
|
||||
process = subprocess.Popen(
|
||||
[sys.executable, "-c", source],
|
||||
stdout=subprocess.PIPE,
|
||||
text=True,
|
||||
start_new_session=True,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
child_pid = int(process.stdout.readline().strip())
|
||||
try:
|
||||
_terminate_owned_process_group(process, grace=1)
|
||||
self.assertIsNotNone(process.poll())
|
||||
with self.assertRaises(ProcessLookupError):
|
||||
os.killpg(process.pid, 0)
|
||||
with self.assertRaises(ProcessLookupError):
|
||||
os.kill(child_pid, 0)
|
||||
finally:
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
process.wait(timeout=2)
|
||||
process.stdout.close()
|
||||
|
||||
def test_renderer_error_reaps_its_owned_process_group(self):
|
||||
with tempfile.TemporaryDirectory(dir=Path.cwd()) as raw:
|
||||
root = Path(raw)
|
||||
(root / "index.html").write_text("<main>unused</main>", encoding="utf-8")
|
||||
pid_path = root / "browser-pids"
|
||||
binary = root / "fake-browser"
|
||||
binary.write_text(
|
||||
"#!/usr/bin/env python3\n"
|
||||
"import os, signal, subprocess, sys, time\n"
|
||||
"signal.signal(signal.SIGTERM, signal.SIG_IGN)\n"
|
||||
"child = subprocess.Popen([sys.executable, '-c', "
|
||||
"'import signal,time;signal.signal(signal.SIGTERM,signal.SIG_IGN);time.sleep(60)'])\n"
|
||||
f"with open({str(pid_path)!r}, 'a', encoding='ascii') as handle:\n"
|
||||
" handle.write(f'{os.getpid()} {child.pid}\\n')\n"
|
||||
" handle.flush()\n"
|
||||
" os.fsync(handle.fileno())\n"
|
||||
"time.sleep(60)\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
binary.chmod(0o700)
|
||||
pids: list[int] = []
|
||||
try:
|
||||
with self.assertRaisesRegex(BrowserError, "browser_cdp_unavailable"):
|
||||
BrowserRenderer(str(binary)).render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=(
|
||||
SimpleNamespace(
|
||||
id="mobile.small+wide", width=375, height=700
|
||||
),
|
||||
),
|
||||
timeout_seconds=1,
|
||||
)
|
||||
pids = [int(value) for value in pid_path.read_text().split()]
|
||||
self.assertEqual(len(pids), 6)
|
||||
for pid in pids:
|
||||
with self.subTest(pid=pid), self.assertRaises(ProcessLookupError):
|
||||
os.kill(pid, 0)
|
||||
finally:
|
||||
for pid in pids:
|
||||
try:
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
|
||||
def test_renderer_restarts_only_closed_transient_failures(self):
|
||||
with tempfile.TemporaryDirectory() as raw:
|
||||
root = Path(raw)
|
||||
(root / "index.html").write_text("<main>unused</main>", encoding="utf-8")
|
||||
viewport = SimpleNamespace(id="desktop", width=800, height=600)
|
||||
renderer = BrowserRenderer(sys.executable)
|
||||
calls = 0
|
||||
expected = object()
|
||||
|
||||
def transient_then_success(**_kwargs):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
screenshot = root / "screenshot-desktop.png"
|
||||
if calls < 3:
|
||||
screenshot.write_bytes(b"partial")
|
||||
raise BrowserError("CDP socket closed")
|
||||
self.assertFalse(screenshot.exists())
|
||||
return expected
|
||||
|
||||
renderer._render_once = transient_then_success
|
||||
self.assertIs(
|
||||
renderer.render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=(viewport,),
|
||||
timeout_seconds=1,
|
||||
),
|
||||
expected,
|
||||
)
|
||||
self.assertEqual(calls, 3)
|
||||
|
||||
for error in ("CDP socket closed", "screenshot_collision"):
|
||||
with self.subTest(error=error):
|
||||
calls = 0
|
||||
|
||||
def fail(**_kwargs):
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
raise BrowserError(error)
|
||||
|
||||
renderer._render_once = fail
|
||||
with self.assertRaisesRegex(BrowserError, error):
|
||||
renderer.render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=(viewport,),
|
||||
timeout_seconds=1,
|
||||
)
|
||||
self.assertEqual(calls, 3 if error == "CDP socket closed" else 1)
|
||||
|
||||
|
||||
class _Counter(BaseHTTPRequestHandler):
|
||||
count = 0
|
||||
|
||||
def do_GET(self):
|
||||
type(self).count += 1
|
||||
self.send_response(200)
|
||||
self.end_headers()
|
||||
|
||||
def log_message(self, *_args):
|
||||
pass
|
||||
|
||||
|
||||
class BrowserIntegrationTest(unittest.TestCase):
|
||||
_VIEWPORTS = (
|
||||
SimpleNamespace(id="desktop", width=900, height=700),
|
||||
SimpleNamespace(id="mobile.small+wide", width=375, height=700),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _valid_page(
|
||||
root: Path,
|
||||
extra_image: str = "",
|
||||
*,
|
||||
focus_css: str = "",
|
||||
autofocus: bool = False,
|
||||
) -> None:
|
||||
(root / "assets").mkdir()
|
||||
for name in ("a.svg", "b.svg"):
|
||||
(root / "assets" / name).write_text(
|
||||
"<svg xmlns='http://www.w3.org/2000/svg' width='80' height='60'/>",
|
||||
encoding="utf-8",
|
||||
)
|
||||
focus_attribute = " autofocus" if autofocus else ""
|
||||
(root / "index.html").write_text(
|
||||
"<link rel='stylesheet' href='styles.css'>"
|
||||
"<main><h1>Ready</h1><img src='assets/a.svg' alt='A'>"
|
||||
"<img src='assets/b.svg' alt='B'>"
|
||||
f"{extra_image}<a href='#x'{focus_attribute}>go</a></main>"
|
||||
"<script src='script.js'></script>",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(root / "styles.css").write_text(
|
||||
f"body{{color:#111;background:#fff}}{focus_css}", encoding="utf-8"
|
||||
)
|
||||
(root / "script.js").write_text("document.body.dataset.ready='1'", encoding="utf-8")
|
||||
|
||||
def test_valid_page_emits_complete_two_viewport_observations(self):
|
||||
with tempfile.TemporaryDirectory() as raw:
|
||||
root = Path(raw)
|
||||
self._valid_page(root)
|
||||
render = BrowserRenderer().render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=self._VIEWPORTS,
|
||||
timeout_seconds=20,
|
||||
)
|
||||
self.assertEqual(
|
||||
[view.id for view in render.viewports],
|
||||
["desktop", "mobile.small+wide"],
|
||||
)
|
||||
self.assertTrue(all((root / view.screenshot).stat().st_size > 0 for view in render.viewports))
|
||||
self.assertTrue(all(len(view.image_facts) == 2 for view in render.viewports))
|
||||
self.assertFalse([item for item in render.requests if item["kind"] == "external"])
|
||||
|
||||
def test_denied_page_dispatches_no_external_request(self):
|
||||
_Counter.count = 0
|
||||
with tempfile.TemporaryDirectory() as raw:
|
||||
root = Path(raw)
|
||||
counter = ThreadingHTTPServer(("127.0.0.1", 0), _Counter)
|
||||
thread = threading.Thread(target=counter.serve_forever, daemon=True)
|
||||
thread.start()
|
||||
try:
|
||||
self._valid_page(
|
||||
root,
|
||||
f"<img src='http://127.0.0.1:{counter.server_address[1]}/leak' alt='blocked'>",
|
||||
)
|
||||
render = BrowserRenderer().render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=self._VIEWPORTS,
|
||||
timeout_seconds=20,
|
||||
)
|
||||
self.assertEqual(_Counter.count, 0)
|
||||
self.assertTrue(any(item["kind"] == "external" for item in render.requests))
|
||||
finally:
|
||||
counter.shutdown()
|
||||
counter.server_close()
|
||||
thread.join(2)
|
||||
|
||||
def test_focus_visibility_uses_computed_indicator(self):
|
||||
manifest = SimpleNamespace(
|
||||
fixture=SimpleNamespace(
|
||||
assets=tuple(
|
||||
SimpleNamespace(workspace_path=f"assets/{name}.svg")
|
||||
for name in ("a", "b")
|
||||
)
|
||||
),
|
||||
viewports=self._VIEWPORTS,
|
||||
)
|
||||
cases = (
|
||||
("suppressed", "a:focus{outline:none;box-shadow:none}", False, False),
|
||||
(
|
||||
"transparent-outline",
|
||||
"a:focus{outline:4px solid rgba(0,85,255,0);box-shadow:none}",
|
||||
False,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"transparent-shadow",
|
||||
"a:focus{outline:none;box-shadow:0 0 0 4px rgba(0,85,255,0)}",
|
||||
False,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"transparent-border",
|
||||
"a:focus{outline:none;box-shadow:none;border:4px solid rgba(0,85,255,0)}",
|
||||
False,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"transparent-gradient",
|
||||
"a:focus{outline:none;box-shadow:none;background-image:linear-gradient(rgba(0,85,255,0),rgba(0,85,255,0))}",
|
||||
False,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"autofocus-visible-outline",
|
||||
"a:focus{outline:4px solid #05f;box-shadow:none}",
|
||||
True,
|
||||
True,
|
||||
),
|
||||
(
|
||||
"visible-outline",
|
||||
"a:focus{outline:4px solid #05f;box-shadow:none}",
|
||||
True,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"visible-shadow",
|
||||
"a:focus{outline:none;box-shadow:0 0 0 4px #05f}",
|
||||
True,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"visible-border",
|
||||
"a:focus{outline:none;box-shadow:none;border:4px solid #05f}",
|
||||
True,
|
||||
False,
|
||||
),
|
||||
(
|
||||
"visible-background",
|
||||
"a:focus{outline:none;box-shadow:none;background-color:#8cf}",
|
||||
True,
|
||||
False,
|
||||
),
|
||||
)
|
||||
for label, focus_css, expected, autofocus in cases:
|
||||
with self.subTest(case=label), tempfile.TemporaryDirectory() as raw:
|
||||
root = Path(raw)
|
||||
self._valid_page(
|
||||
root, focus_css=focus_css, autofocus=autofocus
|
||||
)
|
||||
render = BrowserRenderer().render(
|
||||
workspace_root=root,
|
||||
output_root=root,
|
||||
viewports=self._VIEWPORTS,
|
||||
timeout_seconds=20,
|
||||
)
|
||||
observed = [
|
||||
control["focus_visible"]
|
||||
for viewport in render.viewports
|
||||
for control in viewport.accessibility["controls"]
|
||||
]
|
||||
self.assertEqual(observed, [expected, expected])
|
||||
if not expected:
|
||||
self.assertFalse(
|
||||
_runtime_gates(manifest, render)["accessibility"]["passed"]
|
||||
)
|
||||
|
|
@ -1,535 +0,0 @@
|
|||
"""Secret-safe Claude Code adapter for the IOP benchmark lifecycle.
|
||||
|
||||
The generic lifecycle deliberately does not know Claude's command line or its
|
||||
JSONL protocol. This module converts one immutable benchmark cell and one
|
||||
prepared workspace into that closed boundary. It has no network dependency;
|
||||
tests use a local fake executable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
from scripts.agent_benchmark.connectivity import (
|
||||
CallerCapability,
|
||||
ConnectivityResult,
|
||||
EffectiveBinding,
|
||||
RequestedEffectiveBinding,
|
||||
make_result,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CALLER_REASON_ERROR,
|
||||
CALLER_REASON_SUCCESS,
|
||||
CALLER_STATUS_FAILED,
|
||||
CALLER_STATUS_SUCCEEDED,
|
||||
COMPLETION_EXIT_AFTER_IDLE,
|
||||
SUBMISSION_STDIN_ONCE,
|
||||
CallerEvent,
|
||||
CallerTerminal,
|
||||
InvocationSpec,
|
||||
LifecycleMetricError,
|
||||
ParsedMetric,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
exact_value_redactor,
|
||||
inherited_tls_ca_environment,
|
||||
is_reported_number,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import PreparedWorkspace
|
||||
|
||||
|
||||
REDACTED = "[redacted]"
|
||||
CLAUDE_ROUTE_KINDS = ("direct", "execution_preset")
|
||||
# This is lexical order, as required by the closed connectivity capability
|
||||
# tuple. The cell's requested effort is still passed through unchanged.
|
||||
CLAUDE_EFFORTS = ("high", "low", "max", "medium", "xhigh")
|
||||
_STRUCTURAL_SECRET_KEYS = frozenset(
|
||||
{
|
||||
"content", "text", "input", "arguments", "tool_input", "prompt", "query",
|
||||
"result", "error", "errors", "error_message",
|
||||
}
|
||||
)
|
||||
# The exact Claude result fields this adapter is allowed to observe. Anything
|
||||
# outside these allowlists never becomes timing or usage evidence, and
|
||||
# ``duration_api_ms`` is marked as overlapping because it is reported inside
|
||||
# the same wall-clock window as ``duration_ms``.
|
||||
_CLAUDE_DURATION_FIELDS = (
|
||||
("duration_ms", "total_duration", False),
|
||||
("duration_api_ms", "model_duration", True),
|
||||
)
|
||||
_CLAUDE_USAGE_FIELDS = {
|
||||
"input_tokens": "input_tokens",
|
||||
"output_tokens": "output_tokens",
|
||||
"cache_read_input_tokens": "cached_input_tokens",
|
||||
"cache_creation_input_tokens": "cache_write_tokens",
|
||||
}
|
||||
_CLAUDE_USAGE_BOOKKEEPING_FIELDS = {
|
||||
"cache_creation", "inference_geo", "iterations", "output_tokens_details",
|
||||
"server_tool_use", "service_tier", "speed",
|
||||
}
|
||||
_CLAUDE_USAGE_NESTED_COUNTS = {
|
||||
"cache_creation": {"ephemeral_1h_input_tokens", "ephemeral_5m_input_tokens"},
|
||||
"output_tokens_details": {"thinking_tokens"},
|
||||
"server_tool_use": {"web_fetch_requests", "web_search_requests"},
|
||||
}
|
||||
|
||||
|
||||
class ClaudeIopError(Exception):
|
||||
"""Base exception for configuration and protocol failures."""
|
||||
|
||||
|
||||
class ClaudeIopValidationError(ClaudeIopError):
|
||||
"""Raised when a runtime input or preflight shape is inadmissible."""
|
||||
|
||||
|
||||
class ClaudeIopProtocolError(ClaudeIopError):
|
||||
"""Raised for a malformed or contradictory claimed Claude terminal."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ClaudeIopRuntime:
|
||||
"""Runtime-only IOP inputs. Values are never written to durable evidence."""
|
||||
|
||||
binary: str
|
||||
base_url: str
|
||||
api_key: str
|
||||
|
||||
|
||||
def _require_string(value: Any, name: str) -> str:
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ClaudeIopValidationError(f"invalid {name}")
|
||||
return value
|
||||
|
||||
|
||||
def _exact_object(raw_line: str) -> dict[str, Any]:
|
||||
if not isinstance(raw_line, str):
|
||||
raise ClaudeIopProtocolError("invalid Claude JSONL line")
|
||||
try:
|
||||
value = json.loads(raw_line)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ClaudeIopProtocolError("invalid Claude JSONL line") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise ClaudeIopProtocolError("invalid Claude JSONL object")
|
||||
return value
|
||||
|
||||
|
||||
def _required_string(data: dict[str, Any], name: str) -> str:
|
||||
value = data.get(name)
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ClaudeIopProtocolError(f"missing Claude {name}")
|
||||
return value
|
||||
|
||||
|
||||
def _structural_redact(
|
||||
value: Any,
|
||||
sensitive_values: tuple[str, ...],
|
||||
key: str = "",
|
||||
*,
|
||||
redact_terminal_message: bool = False,
|
||||
) -> Any:
|
||||
if key in _STRUCTURAL_SECRET_KEYS:
|
||||
return REDACTED
|
||||
if redact_terminal_message and key == "message":
|
||||
return REDACTED
|
||||
if isinstance(value, str):
|
||||
for sensitive in sensitive_values:
|
||||
if sensitive:
|
||||
value = value.replace(sensitive, REDACTED)
|
||||
return value
|
||||
if isinstance(value, list):
|
||||
return [
|
||||
_structural_redact(
|
||||
item, sensitive_values, redact_terminal_message=redact_terminal_message
|
||||
)
|
||||
for item in value
|
||||
]
|
||||
if isinstance(value, dict):
|
||||
return {
|
||||
str(name): _structural_redact(
|
||||
item,
|
||||
sensitive_values,
|
||||
str(name),
|
||||
redact_terminal_message=redact_terminal_message,
|
||||
)
|
||||
for name, item in value.items()
|
||||
}
|
||||
return value
|
||||
|
||||
|
||||
def redact_claude_event(raw_line: str, sensitive_values: tuple[str, ...]) -> str:
|
||||
"""Return canonical JSON without task/tool content or runtime secrets.
|
||||
|
||||
A malformed line is represented by a fixed marker so error reporting cannot
|
||||
accidentally retain the raw malformed payload.
|
||||
"""
|
||||
try:
|
||||
event = _exact_object(raw_line)
|
||||
except ClaudeIopProtocolError:
|
||||
return '{"type":"invalid_claude_json"}'
|
||||
redacted = _structural_redact(
|
||||
event,
|
||||
sensitive_values,
|
||||
redact_terminal_message=event.get("type") in ("error", "result"),
|
||||
)
|
||||
return json.dumps(redacted, sort_keys=True, separators=(",", ":"), ensure_ascii=True)
|
||||
|
||||
|
||||
def parse_preflight_binding(raw_line: str, cell: MatrixCell) -> RequestedEffectiveBinding:
|
||||
"""Parse an explicit, secret-free IOP binding observation from JSONL.
|
||||
|
||||
The adapter never derives effective values from the requested cell. A
|
||||
caller must provide all requested/effective scalars and every stage, or the
|
||||
connectivity contract rejects the observation as incomplete.
|
||||
"""
|
||||
event = _exact_object(raw_line)
|
||||
if event.get("type") != "system" or event.get("subtype") != "iop_binding":
|
||||
raise ClaudeIopProtocolError("invalid Claude preflight event")
|
||||
binding = event.get("binding")
|
||||
if not isinstance(binding, dict) or set(binding) != {
|
||||
"cell_id", "caller", "requested_route_kind", "requested_route_id",
|
||||
"requested_model", "requested_effort", "effective_route_kind",
|
||||
"effective_route_id", "effective_model", "effective_effort", "effective_bindings",
|
||||
}:
|
||||
raise ClaudeIopProtocolError("invalid Claude preflight binding")
|
||||
raw_stages = binding["effective_bindings"]
|
||||
if not isinstance(raw_stages, list):
|
||||
raise ClaudeIopProtocolError("invalid Claude preflight stages")
|
||||
stages: list[EffectiveBinding] = []
|
||||
for item in raw_stages:
|
||||
if not isinstance(item, dict) or set(item) != {"stage", "model", "effort"}:
|
||||
raise ClaudeIopProtocolError("invalid Claude preflight stage")
|
||||
stages.append(EffectiveBinding(item["stage"], item["model"], item["effort"]))
|
||||
try:
|
||||
result = RequestedEffectiveBinding(
|
||||
binding["cell_id"], binding["caller"], binding["requested_route_kind"],
|
||||
binding["requested_route_id"], binding["requested_model"],
|
||||
binding["requested_effort"], binding["effective_route_kind"],
|
||||
binding["effective_route_id"], binding["effective_model"],
|
||||
binding["effective_effort"], tuple(stages),
|
||||
)
|
||||
# Validation is intentionally delegated to the one shared contract.
|
||||
make_result(cell, claude_capability(), result)
|
||||
except Exception as exc:
|
||||
if isinstance(exc, ClaudeIopProtocolError):
|
||||
raise
|
||||
raise ClaudeIopProtocolError("invalid Claude preflight binding") from exc
|
||||
return result
|
||||
|
||||
|
||||
class ClaudeStreamParser:
|
||||
"""Parse one fresh Claude Code stream without inventing terminal bindings."""
|
||||
|
||||
def __init__(self, cell: MatrixCell, session_id: str) -> None:
|
||||
if not isinstance(cell, MatrixCell) or cell.caller != "claude":
|
||||
raise ClaudeIopValidationError("Claude adapter requires a Claude matrix cell")
|
||||
self.cell = cell
|
||||
# The prepared workspace identity proves that this is a fresh caller
|
||||
# invocation. Claude Code emits its own UUID in system/init, so it
|
||||
# must be derived from that event rather than compared to this local
|
||||
# opaque label.
|
||||
self.prepared_session_id = _require_string(session_id, "session_id")
|
||||
self.claude_session_id: str | None = None
|
||||
self._phase = "await_init"
|
||||
self._assistant_messages = 0
|
||||
self._active_message_id: str | None = None
|
||||
self._completed_message_ids: set[str] = set()
|
||||
self._continuation_pending = False
|
||||
self._api_error_seen = False
|
||||
|
||||
def _require_bound_session(self, event: dict[str, Any]) -> None:
|
||||
if self.claude_session_id is None:
|
||||
raise ClaudeIopProtocolError("missing Claude init")
|
||||
if _required_string(event, "session_id") != self.claude_session_id:
|
||||
raise ClaudeIopProtocolError("Claude session binding mismatch")
|
||||
|
||||
def _consume_init(self, event: dict[str, Any]) -> None:
|
||||
if self._phase != "await_init":
|
||||
raise ClaudeIopProtocolError("duplicate or out-of-order Claude init")
|
||||
if _required_string(event, "model") != self.cell.iop.request_model:
|
||||
raise ClaudeIopProtocolError("Claude model binding mismatch")
|
||||
self.claude_session_id = _required_string(event, "session_id")
|
||||
self._phase = "await_assistant"
|
||||
|
||||
def _consume_assistant(self, event: dict[str, Any]) -> CallerEvent | None:
|
||||
if self._phase != "await_assistant":
|
||||
raise ClaudeIopProtocolError("duplicate or out-of-order Claude assistant")
|
||||
self._require_bound_session(event)
|
||||
message = event.get("message")
|
||||
if not isinstance(message, dict):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant event")
|
||||
if event.get("is_api_error_message") is True:
|
||||
if message.get("model") != "<synthetic>" or message.get("stop_reason") != "stop_sequence":
|
||||
raise ClaudeIopProtocolError("invalid Claude API error event")
|
||||
self._complete_active_message()
|
||||
self._api_error_seen = True
|
||||
self._phase = "await_error_result"
|
||||
return None
|
||||
if _required_string(message, "model") != self.cell.iop.request_model:
|
||||
raise ClaudeIopProtocolError("Claude model binding mismatch")
|
||||
stop_reason = message.get("stop_reason")
|
||||
if stop_reason not in (None, "tool_use", "end_turn"):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant stop reason")
|
||||
message_id = message.get("id")
|
||||
if message_id is not None and (not isinstance(message_id, str) or not message_id):
|
||||
raise ClaudeIopProtocolError("invalid Claude assistant message id")
|
||||
if self._continuation_pending:
|
||||
if message_id != self._active_message_id:
|
||||
self._complete_active_message()
|
||||
self._continuation_pending = False
|
||||
if stop_reason is None:
|
||||
if message_id is None:
|
||||
raise ClaudeIopProtocolError("unbound Claude assistant snapshot")
|
||||
if message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
if self._active_message_id not in (None, message_id):
|
||||
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
|
||||
self._active_message_id = message_id
|
||||
return None
|
||||
|
||||
if message_id is None:
|
||||
# Older fixture-shaped output did not include a message id. It is
|
||||
# admissible only for the single final assistant event.
|
||||
if stop_reason != "end_turn" or self._active_message_id is not None:
|
||||
raise ClaudeIopProtocolError("unbound Claude assistant terminal")
|
||||
self._assistant_messages += 1
|
||||
else:
|
||||
if message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
if self._active_message_id not in (None, message_id):
|
||||
raise ClaudeIopProtocolError("overlapping Claude assistant messages")
|
||||
self._active_message_id = message_id
|
||||
self._complete_active_message()
|
||||
if stop_reason == "tool_use":
|
||||
self._phase = "await_tool_result"
|
||||
return None
|
||||
self._phase = "await_result"
|
||||
return CallerEvent("finish")
|
||||
|
||||
def _consume_user(self, event: dict[str, Any]) -> None:
|
||||
if self._phase not in ("await_assistant", "await_tool_result"):
|
||||
raise ClaudeIopProtocolError("out-of-order Claude user event")
|
||||
self._require_bound_session(event)
|
||||
if self._active_message_id is not None:
|
||||
# Claude Code may publish cumulative snapshots with the same
|
||||
# assistant message id on both sides of one or more tool results.
|
||||
# The next assistant snapshot (or the result terminal) determines
|
||||
# whether this message continues or a new model call begins.
|
||||
self._continuation_pending = True
|
||||
elif self._phase != "await_tool_result":
|
||||
raise ClaudeIopProtocolError("unexpected Claude user event")
|
||||
self._phase = "await_assistant"
|
||||
|
||||
def _complete_active_message(self) -> None:
|
||||
if self._active_message_id is None:
|
||||
return
|
||||
if self._active_message_id in self._completed_message_ids:
|
||||
raise ClaudeIopProtocolError("duplicate Claude assistant message")
|
||||
self._completed_message_ids.add(self._active_message_id)
|
||||
self._active_message_id = None
|
||||
self._continuation_pending = False
|
||||
self._assistant_messages += 1
|
||||
|
||||
def _consume_result(self, event: dict[str, Any]) -> tuple[Any, ...] | None:
|
||||
if self._phase == "await_error_result":
|
||||
self._require_bound_session(event)
|
||||
if (
|
||||
not self._api_error_seen
|
||||
or event.get("subtype") != "success"
|
||||
or event.get("is_error") is not True
|
||||
or event.get("terminal_reason") != "api_error"
|
||||
):
|
||||
raise ClaudeIopProtocolError("invalid Claude API error terminal")
|
||||
self._phase = "complete"
|
||||
return (
|
||||
CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR),
|
||||
CallerEvent("finish"),
|
||||
CallerEvent("idle"),
|
||||
)
|
||||
result_completes_active_message = False
|
||||
if self._phase == "await_assistant" and self._active_message_id is not None:
|
||||
if self._continuation_pending:
|
||||
raise ClaudeIopProtocolError("Claude result followed an unresolved tool result")
|
||||
self._complete_active_message()
|
||||
result_completes_active_message = True
|
||||
elif self._phase != "await_result":
|
||||
raise ClaudeIopProtocolError("duplicate or out-of-order Claude result")
|
||||
self._require_bound_session(event)
|
||||
if event.get("subtype") != "success" or event.get("is_error") is True:
|
||||
raise ClaudeIopProtocolError("invalid Claude result terminal")
|
||||
self._phase = "complete"
|
||||
terminal = (
|
||||
(CallerEvent("finish"), CallerEvent("idle"))
|
||||
if result_completes_active_message
|
||||
else (CallerEvent("idle"),)
|
||||
)
|
||||
return (
|
||||
*self._observations(event),
|
||||
CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS),
|
||||
*terminal,
|
||||
)
|
||||
|
||||
def _observations(self, event: dict[str, Any]) -> tuple[ParsedMetric, ...]:
|
||||
"""Convert only allowlisted reported Claude values into observations."""
|
||||
model = self.cell.iop.request_model
|
||||
observations: list[ParsedMetric] = []
|
||||
try:
|
||||
for field, name, overlap in _CLAUDE_DURATION_FIELDS:
|
||||
if field in event:
|
||||
if not is_reported_number(event[field]):
|
||||
raise ClaudeIopProtocolError("invalid Claude duration observation")
|
||||
observations.append(duration_metric(
|
||||
name, event[field], reported_unit="ms",
|
||||
model=model, overlap=overlap,
|
||||
))
|
||||
observations.append(count_metric(
|
||||
"model_calls", self._assistant_messages, model=model
|
||||
))
|
||||
observations.extend(self._usage_observations(event, model))
|
||||
except LifecycleMetricError as exc:
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation") from exc
|
||||
return tuple(observations)
|
||||
|
||||
@staticmethod
|
||||
def _usage_observations(event: dict[str, Any], model: str) -> list[ParsedMetric]:
|
||||
usage = event.get("usage")
|
||||
if usage is None:
|
||||
return []
|
||||
allowed = set(_CLAUDE_USAGE_FIELDS) | _CLAUDE_USAGE_BOOKKEEPING_FIELDS
|
||||
if not isinstance(usage, dict) or not set(usage) <= allowed:
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation")
|
||||
ClaudeStreamParser._validate_usage_bookkeeping(usage)
|
||||
return [
|
||||
count_metric(_CLAUDE_USAGE_FIELDS[field], value, model=model)
|
||||
for field, value in sorted(usage.items()) if field in _CLAUDE_USAGE_FIELDS
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def _validate_usage_bookkeeping(usage: dict[str, Any]) -> None:
|
||||
for field, nested_fields in _CLAUDE_USAGE_NESTED_COUNTS.items():
|
||||
if field not in usage:
|
||||
continue
|
||||
value = usage[field]
|
||||
if not isinstance(value, dict) or not set(value) <= nested_fields:
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation")
|
||||
if any(not is_reported_number(item) for item in value.values()):
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation")
|
||||
if "iterations" in usage and usage["iterations"] != []:
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation")
|
||||
for field in ("inference_geo", "service_tier", "speed"):
|
||||
if field in usage and (
|
||||
not isinstance(usage[field], str) or len(usage[field]) > 64
|
||||
):
|
||||
raise ClaudeIopProtocolError("invalid Claude usage observation")
|
||||
|
||||
def __call__(self, stream: str, raw_line: str) -> Any:
|
||||
if stream != "stdout":
|
||||
return None
|
||||
event = _exact_object(raw_line)
|
||||
event_type = event.get("type")
|
||||
if event_type == "system" and event.get("subtype") == "init":
|
||||
self._consume_init(event)
|
||||
return None
|
||||
if event_type == "assistant":
|
||||
return self._consume_assistant(event)
|
||||
if event_type == "user":
|
||||
self._consume_user(event)
|
||||
return None
|
||||
if event_type == "result":
|
||||
return self._consume_result(event)
|
||||
# Informational events are deliberately ignored only after they have
|
||||
# passed exact JSON-object decoding above.
|
||||
return None
|
||||
|
||||
|
||||
def claude_capability() -> CallerCapability:
|
||||
return CallerCapability("claude", CLAUDE_ROUTE_KINDS, CLAUDE_EFFORTS)
|
||||
|
||||
|
||||
def resolve_claude_binary(binary: str) -> str:
|
||||
"""Resolve one executable before lifecycle process creation."""
|
||||
candidate = _require_string(binary, "Claude binary")
|
||||
path = Path(candidate)
|
||||
resolved = str(path.resolve()) if path.parent != Path(".") else shutil.which(candidate)
|
||||
if not resolved or not Path(resolved).is_file() or not os.access(resolved, os.X_OK):
|
||||
raise ClaudeIopValidationError("Claude binary is unavailable")
|
||||
return str(Path(resolved).resolve())
|
||||
|
||||
|
||||
class ClaudeIopAdapter:
|
||||
"""Build safe Claude invocations and parse their IOP-bound stream evidence."""
|
||||
|
||||
capability = claude_capability()
|
||||
|
||||
def __init__(self, cell: MatrixCell, workspace: PreparedWorkspace, runtime: ClaudeIopRuntime) -> None:
|
||||
if not isinstance(cell, MatrixCell) or cell.caller != "claude":
|
||||
raise ClaudeIopValidationError("Claude adapter requires a Claude matrix cell")
|
||||
if not isinstance(workspace, PreparedWorkspace) or not workspace.session_is_fresh:
|
||||
raise ClaudeIopValidationError("Claude adapter requires a fresh prepared workspace")
|
||||
if not isinstance(runtime, ClaudeIopRuntime):
|
||||
raise ClaudeIopValidationError("invalid Claude runtime")
|
||||
self.cell = cell
|
||||
self.workspace = workspace
|
||||
self.runtime = runtime
|
||||
self.binary = resolve_claude_binary(runtime.binary)
|
||||
self.base_url = _require_string(runtime.base_url, "IOP base URL")
|
||||
self.api_key = _require_string(runtime.api_key, "IOP API key")
|
||||
|
||||
def preflight(self, raw_line: str) -> ConnectivityResult:
|
||||
return make_result(self.cell, self.capability, parse_preflight_binding(raw_line, self.cell))
|
||||
|
||||
def parser(self) -> ClaudeStreamParser:
|
||||
return ClaudeStreamParser(self.cell, self.workspace.session_id)
|
||||
|
||||
def redactor(self, task: str) -> Callable[[str], str]:
|
||||
_require_string(task, "task")
|
||||
structural_values = (task, self.base_url, self.api_key)
|
||||
exact = exact_value_redactor(structural_values)
|
||||
|
||||
def _redact(line: str) -> str:
|
||||
return exact(redact_claude_event(line, structural_values))
|
||||
|
||||
return _redact
|
||||
|
||||
def invocation(self, task: str, evidence_dir: str | Path, timeout: Timeout) -> InvocationSpec:
|
||||
_require_string(task, "task")
|
||||
if not isinstance(timeout, Timeout):
|
||||
raise ClaudeIopValidationError("invalid invocation timeout")
|
||||
evidence_path = _require_string(str(evidence_dir), "evidence directory")
|
||||
cwd = Path(self.workspace.workspace_dir)
|
||||
if not cwd.is_dir():
|
||||
raise ClaudeIopValidationError("prepared workspace is unavailable")
|
||||
env = (
|
||||
("PATH", os.environ.get("PATH", "/usr/bin:/bin")),
|
||||
("ANTHROPIC_BASE_URL", self.base_url),
|
||||
("ANTHROPIC_API_KEY", self.api_key),
|
||||
("CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "1"),
|
||||
("CLAUDE_CODE_DISABLE_AUTOUPDATER", "1"),
|
||||
) + inherited_tls_ca_environment()
|
||||
return InvocationSpec(
|
||||
argv=(
|
||||
self.binary, "--bare", "--print", "--verbose",
|
||||
"--input-format", "text", "--output-format", "stream-json",
|
||||
"--model", self.cell.iop.request_model, "--effort", self.cell.iop.requested_effort,
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk",
|
||||
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
|
||||
),
|
||||
cwd=str(cwd),
|
||||
env=env,
|
||||
env_allowlist=(
|
||||
"ANTHROPIC_BASE_URL", "ANTHROPIC_API_KEY",
|
||||
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", "CLAUDE_CODE_DISABLE_AUTOUPDATER",
|
||||
*TLS_CA_ENV_KEYS,
|
||||
),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=timeout,
|
||||
evidence_dir=evidence_path,
|
||||
task_payload=task.encode("utf-8"),
|
||||
)
|
||||
|
|
@ -1,484 +0,0 @@
|
|||
"""Hermetic contract tests for the Claude Code IOP benchmark adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.claude_iop import (
|
||||
ClaudeIopAdapter,
|
||||
ClaudeIopProtocolError,
|
||||
ClaudeIopRuntime,
|
||||
ClaudeIopValidationError,
|
||||
ClaudeStreamParser,
|
||||
parse_preflight_binding,
|
||||
redact_claude_event,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CallerEvent, CallerTerminal, ParsedMetric,
|
||||
REASON_PARSER_ERROR, REASON_SUCCESS, run_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import (
|
||||
AttemptIdentity,
|
||||
PreparedWorkspace,
|
||||
TestbedProvenance,
|
||||
)
|
||||
|
||||
|
||||
SENTINELS = ("prompt-secret-sentinel", "https://private.iop.invalid", "api-secret-sentinel")
|
||||
ARBITRARY_SENTINELS = (
|
||||
"tool-secret-sentinel", "result-secret-sentinel", "error-secret-sentinel",
|
||||
)
|
||||
|
||||
|
||||
def _cell(route_kind: str = "direct") -> MatrixCell:
|
||||
bindings = (ExpectedBinding("request", "claude-sonnet", "high"),)
|
||||
if route_kind == "execution_preset":
|
||||
bindings = (
|
||||
ExpectedBinding("selector", "claude-sonnet", "high"),
|
||||
ExpectedBinding("plan", "claude-sonnet", "high"),
|
||||
ExpectedBinding("work", "claude-sonnet", "high"),
|
||||
ExpectedBinding("review", "claude-sonnet", "high"),
|
||||
)
|
||||
return MatrixCell("claude-direct", "claude", IopCell(
|
||||
"claude-sonnet", "high", route_kind, "iop-route", bindings,
|
||||
))
|
||||
|
||||
|
||||
def _workspace(root: Path, session_id: str = "session-fixture") -> PreparedWorkspace:
|
||||
workspace = root / "workspace"
|
||||
workspace.mkdir()
|
||||
return PreparedWorkspace(
|
||||
AttemptIdentity("run-20260102T030405Z-abcdef123456", "claude-direct", 1, 1),
|
||||
str(root), str(workspace), str(root / "session"), session_id, True,
|
||||
"sha256:" + "0" * 64, "isolated",
|
||||
TestbedProvenance("/testbed", "main", "0" * 40, "sha256:" + "1" * 64, True),
|
||||
"2026-01-02T03:04:05Z",
|
||||
)
|
||||
|
||||
|
||||
def _binding_event(cell: MatrixCell) -> str:
|
||||
return json.dumps({
|
||||
"type": "system", "subtype": "iop_binding", "binding": {
|
||||
"cell_id": cell.id, "caller": cell.caller,
|
||||
"requested_route_kind": cell.iop.route_kind, "requested_route_id": cell.iop.route_id,
|
||||
"requested_model": cell.iop.request_model, "requested_effort": cell.iop.requested_effort,
|
||||
"effective_route_kind": cell.iop.route_kind, "effective_route_id": cell.iop.route_id,
|
||||
"effective_model": cell.iop.request_model, "effective_effort": cell.iop.requested_effort,
|
||||
"effective_bindings": [
|
||||
{"stage": item.stage, "model": item.model, "effort": item.effort}
|
||||
for item in cell.iop.expected_bindings
|
||||
],
|
||||
},
|
||||
})
|
||||
|
||||
|
||||
class ClaudeIopTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
# The lifecycle intentionally requires an executable fake CLI. Some
|
||||
# CI hosts mount /tmp noexec, so keep this short-lived directory under
|
||||
# the repository worktree instead.
|
||||
self.temp = tempfile.TemporaryDirectory(dir=Path.cwd(), prefix=".claude-iop-test-")
|
||||
self.root = Path(self.temp.name)
|
||||
self.cell = _cell()
|
||||
self.workspace = _workspace(self.root)
|
||||
self.binary = self.root / "claude"
|
||||
self.binary.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
|
||||
self.binary.chmod(0o700)
|
||||
self.runtime = ClaudeIopRuntime(str(self.binary), SENTINELS[1], SENTINELS[2])
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def _adapter(self, *, cell: MatrixCell | None = None) -> ClaudeIopAdapter:
|
||||
return ClaudeIopAdapter(cell or self.cell, self.workspace, self.runtime)
|
||||
|
||||
def _fixture_lines(self) -> list[str]:
|
||||
fixture = Path("scripts/fixtures/agent-comparison-benchmark/claude-iop-stream.jsonl")
|
||||
return fixture.read_text(encoding="utf-8").splitlines()
|
||||
|
||||
def _run_fake(self, lines: list[str], *, exit_code: int = 0):
|
||||
self.binary.write_text(textwrap.dedent(f"""\
|
||||
#!/usr/bin/env python3
|
||||
import os, sys
|
||||
assert sys.stdin.read() == {SENTINELS[0]!r}
|
||||
assert os.environ["ANTHROPIC_BASE_URL"] == {SENTINELS[1]!r}
|
||||
assert os.environ["ANTHROPIC_API_KEY"] == {SENTINELS[2]!r}
|
||||
for line in {lines!r}:
|
||||
print(line)
|
||||
raise SystemExit({exit_code!r})
|
||||
"""), encoding="utf-8")
|
||||
evidence = self.root / f"evidence-{len(tuple(self.root.glob('evidence-*')))}"
|
||||
evidence.mkdir()
|
||||
adapter = self._adapter()
|
||||
result = run_invocation(
|
||||
adapter.invocation(SENTINELS[0], evidence, Timeout(5, 1, 1, 1)),
|
||||
parse_event=adapter.parser(), redact=adapter.redactor(SENTINELS[0]),
|
||||
on_started=lambda _: None,
|
||||
)
|
||||
durable = "\n".join(
|
||||
path.read_text(encoding="utf-8") for path in sorted(evidence.iterdir())
|
||||
)
|
||||
return result, durable
|
||||
|
||||
def test_exact_iop_only_invocation_and_fresh_workspace(self) -> None:
|
||||
adapter = self._adapter()
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = adapter.invocation(SENTINELS[0], self.root / "evidence", Timeout(5, 1, 1, 1))
|
||||
self.assertEqual(spec.cwd, self.workspace.workspace_dir)
|
||||
self.assertEqual(spec.submission_mode, "stdin_once")
|
||||
self.assertEqual(spec.task_payload, SENTINELS[0].encode())
|
||||
self.assertEqual(spec.argv[1:], (
|
||||
"--bare", "--print", "--verbose", "--input-format", "text",
|
||||
"--output-format", "stream-json", "--model", "claude-sonnet", "--effort", "high",
|
||||
"--no-session-persistence", "--permission-mode", "dontAsk",
|
||||
"--tools", "Read,Write,Edit", "--allowedTools", "Read,Write,Edit",
|
||||
))
|
||||
env = dict(spec.env)
|
||||
self.assertEqual(env["ANTHROPIC_BASE_URL"], SENTINELS[1])
|
||||
self.assertEqual(env["ANTHROPIC_API_KEY"], SENTINELS[2])
|
||||
self.assertEqual(env["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(env["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertNotIn("ANTHROPIC_AUTH_TOKEN", env)
|
||||
self.assertNotIn("CLAUDE_CONFIG_DIR", env)
|
||||
(self.root / "other").mkdir()
|
||||
not_fresh = _workspace(self.root / "other", "session-other")
|
||||
object.__setattr__(not_fresh, "session_is_fresh", False)
|
||||
with self.assertRaises(ClaudeIopValidationError):
|
||||
ClaudeIopAdapter(self.cell, not_fresh, self.runtime)
|
||||
|
||||
def test_direct_and_preset_preflight_are_exact_without_substitution(self) -> None:
|
||||
for route_kind in ("direct", "execution_preset"):
|
||||
cell = _cell(route_kind)
|
||||
event = _binding_event(cell)
|
||||
adapter = self._adapter(cell=cell)
|
||||
self.assertEqual(adapter.preflight(event).status, "ready")
|
||||
mutated = json.loads(event)
|
||||
mutated["binding"]["effective_model"] = "fallback"
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
parse_preflight_binding(json.dumps(mutated), cell)
|
||||
incomplete = json.loads(_binding_event(self.cell))
|
||||
del incomplete["binding"]["effective_effort"]
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
self._adapter().preflight(json.dumps(incomplete))
|
||||
|
||||
def test_runtime_requires_available_binary_and_complete_iop_config(self) -> None:
|
||||
missing_binary = ClaudeIopRuntime(
|
||||
str(self.root / "missing-claude"), SENTINELS[1], SENTINELS[2]
|
||||
)
|
||||
with self.assertRaises(ClaudeIopValidationError):
|
||||
ClaudeIopAdapter(self.cell, self.workspace, missing_binary)
|
||||
for base_url, api_key in (("", SENTINELS[2]), (SENTINELS[1], "")):
|
||||
with self.assertRaises(ClaudeIopValidationError):
|
||||
ClaudeIopAdapter(
|
||||
self.cell, self.workspace,
|
||||
ClaudeIopRuntime(str(self.binary), base_url, api_key),
|
||||
)
|
||||
|
||||
def test_fixture_uses_production_shaped_ordered_terminal_evidence(self) -> None:
|
||||
lines = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
parsed = [parser("stdout", line) for line in lines]
|
||||
self.assertEqual(parsed[:2], [None, CallerEvent("finish")])
|
||||
self.assertEqual(parsed[2][-1], CallerEvent("idle"))
|
||||
|
||||
malformed = json.loads(lines[1])
|
||||
del malformed["message"]["model"]
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
parser("stdout", lines[0])
|
||||
parser("stdout", json.dumps(malformed))
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
parser("stdout", "not-json")
|
||||
|
||||
def test_parser_rejects_missing_duplicate_mismatched_and_out_of_order_evidence(self) -> None:
|
||||
init, assistant, result = self._fixture_lines()
|
||||
wrong_session = json.loads(assistant)
|
||||
wrong_session["session_id"] = "other-claude-session"
|
||||
wrong_model = json.loads(assistant)
|
||||
wrong_model["message"]["model"] = "fallback-model"
|
||||
missing_session = json.loads(result)
|
||||
del missing_session["session_id"]
|
||||
missing_result = [init, assistant]
|
||||
cases = {
|
||||
"assistant-before-init": [assistant],
|
||||
"result-before-assistant": [init, result],
|
||||
"duplicate-init": [init, init],
|
||||
"duplicate-assistant": [init, assistant, assistant],
|
||||
"session-mismatch": [init, json.dumps(wrong_session)],
|
||||
"nested-model-mismatch": [init, json.dumps(wrong_model)],
|
||||
"missing-result-session": [init, assistant, json.dumps(missing_session)],
|
||||
}
|
||||
for name, events in cases.items():
|
||||
with self.subTest(name=name):
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
for event in events:
|
||||
parser("stdout", event)
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
self.assertEqual([parser("stdout", event) for event in missing_result], [None, CallerEvent("finish")])
|
||||
|
||||
def test_parser_accepts_partial_snapshots_and_tool_result_cycles(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
partial = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-tool-1", "model": "claude-sonnet", "stop_reason": None,
|
||||
"content": [{"type": "tool_use", "id": "tool-1"}]},
|
||||
})
|
||||
user = json.dumps({
|
||||
"type": "user", "session_id": "claude-session-fixture",
|
||||
"message": {"content": [{"type": "tool_result", "tool_use_id": "tool-1"}]},
|
||||
})
|
||||
final = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-final", "model": "claude-sonnet",
|
||||
"stop_reason": "end_turn", "content": []},
|
||||
})
|
||||
self.assertEqual(
|
||||
[parser("stdout", line) for line in (init, partial, partial, user, final)],
|
||||
[None, None, None, None, CallerEvent("finish")],
|
||||
)
|
||||
observations = parser("stdout", result)
|
||||
metrics = {
|
||||
metric.name: metric.value for metric in observations
|
||||
if isinstance(metric, ParsedMetric)
|
||||
}
|
||||
self.assertEqual(metrics["model_calls"], 2)
|
||||
|
||||
def test_parser_accepts_cumulative_message_ids_and_direct_result(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
partial = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-cumulative", "model": "claude-sonnet",
|
||||
"stop_reason": None, "content": []},
|
||||
})
|
||||
user = json.dumps({
|
||||
"type": "user", "session_id": "claude-session-fixture",
|
||||
"message": {"content": []},
|
||||
})
|
||||
terminal = json.loads(result)
|
||||
terminal["usage"].update({
|
||||
"cache_creation": {"ephemeral_1h_input_tokens": 0, "ephemeral_5m_input_tokens": 0},
|
||||
"inference_geo": "", "iterations": [],
|
||||
"output_tokens_details": {"thinking_tokens": 0},
|
||||
"server_tool_use": {"web_fetch_requests": 0, "web_search_requests": 0},
|
||||
"service_tier": "standard", "speed": "standard",
|
||||
})
|
||||
parsed = [
|
||||
parser("stdout", line)
|
||||
for line in (init, partial, partial, user, partial, user, user, partial)
|
||||
]
|
||||
self.assertEqual(parsed, [None] * len(parsed))
|
||||
observations = parser("stdout", json.dumps(terminal))
|
||||
self.assertEqual(observations[-2:], (CallerEvent("finish"), CallerEvent("idle")))
|
||||
self.assertEqual(
|
||||
{metric.name: metric.value for metric in observations if isinstance(metric, ParsedMetric)}["model_calls"], 1
|
||||
)
|
||||
|
||||
def test_parser_classifies_synthetic_api_error_without_parser_failure(self) -> None:
|
||||
init, _, _ = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
synthetic = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"is_api_error_message": True,
|
||||
"message": {"id": "synthetic", "model": "<synthetic>",
|
||||
"stop_reason": "stop_sequence", "content": []},
|
||||
})
|
||||
terminal = json.dumps({
|
||||
"type": "result", "session_id": "claude-session-fixture",
|
||||
"subtype": "success", "is_error": True, "terminal_reason": "api_error",
|
||||
})
|
||||
self.assertEqual(
|
||||
[parser("stdout", line) for line in (init, synthetic, terminal)],
|
||||
[None, None, (
|
||||
CallerTerminal("failed", "caller_error"),
|
||||
CallerEvent("finish"), CallerEvent("idle"),
|
||||
)],
|
||||
)
|
||||
|
||||
outcome, _ = self._run_fake([init, synthetic, terminal], exit_code=1)
|
||||
self.assertFalse(outcome.product.status == "succeeded")
|
||||
self.assertEqual(outcome.product.status, "failed")
|
||||
self.assertEqual(outcome.harness.status, "passed")
|
||||
|
||||
def test_lifecycle_accepts_result_direct_active_snapshot(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
partial = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-direct", "model": "claude-sonnet",
|
||||
"stop_reason": None, "content": []},
|
||||
})
|
||||
outcome, _ = self._run_fake([init, partial, result])
|
||||
self.assertTrue(outcome.product.status == "succeeded", outcome)
|
||||
kinds = [event.kind for event in outcome.events]
|
||||
self.assertLess(kinds.index("finish"), kinds.index("idle"))
|
||||
|
||||
def test_parser_accepts_explicit_tool_use_terminal_once(self) -> None:
|
||||
init, _, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
tool_use = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-tool-1", "model": "claude-sonnet",
|
||||
"stop_reason": "tool_use", "content": []},
|
||||
})
|
||||
user = json.dumps({
|
||||
"type": "user", "session_id": "claude-session-fixture", "message": {"content": []},
|
||||
})
|
||||
final = json.dumps({
|
||||
"type": "assistant", "session_id": "claude-session-fixture",
|
||||
"message": {"id": "msg-final", "model": "claude-sonnet",
|
||||
"stop_reason": "end_turn", "content": []},
|
||||
})
|
||||
self.assertEqual(
|
||||
[parser("stdout", line) for line in (init, tool_use, user, final)],
|
||||
[None, None, None, CallerEvent("finish")],
|
||||
)
|
||||
self.assertEqual(
|
||||
{metric.name: metric.value for metric in parser("stdout", result) if isinstance(metric, ParsedMetric)}["model_calls"],
|
||||
2,
|
||||
)
|
||||
duplicate = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
duplicate("stdout", init)
|
||||
duplicate("stdout", tool_use)
|
||||
duplicate("stdout", user)
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
duplicate("stdout", tool_use)
|
||||
|
||||
def test_reported_result_values_become_bound_normalized_observations(self) -> None:
|
||||
init, assistant, result = self._fixture_lines()
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
parser("stdout", init)
|
||||
parser("stdout", assistant)
|
||||
parsed = parser("stdout", result)
|
||||
self.assertEqual(parsed[-1], CallerEvent("idle"))
|
||||
observed = {metric.name: metric for metric in parsed if isinstance(metric, ParsedMetric)}
|
||||
self.assertEqual(observed["total_duration"].value, 1234 * 10 ** 6)
|
||||
self.assertFalse(observed["total_duration"].overlap)
|
||||
# The reported API duration is inside the reported total, so it is
|
||||
# published as an overlapping interval and never subtracted from it.
|
||||
self.assertEqual(observed["model_duration"].value, 1000 * 10 ** 6)
|
||||
self.assertTrue(observed["model_duration"].overlap)
|
||||
self.assertEqual(observed["model_calls"].value, 1)
|
||||
self.assertEqual(observed["input_tokens"].value, 11)
|
||||
self.assertEqual(observed["output_tokens"].value, 22)
|
||||
self.assertEqual(observed["cached_input_tokens"].value, 5)
|
||||
# The fixture omits cache creation, so that category stays unreported
|
||||
# rather than being reported as zero.
|
||||
self.assertNotIn("cache_write_tokens", observed)
|
||||
for metric in parsed:
|
||||
if not isinstance(metric, ParsedMetric):
|
||||
continue
|
||||
self.assertEqual(metric.model, "claude-sonnet")
|
||||
self.assertEqual(metric.source, "caller_output")
|
||||
self.assertEqual(observed["input_tokens"].clock, "none")
|
||||
self.assertEqual(observed["total_duration"].clock, "caller_reported")
|
||||
|
||||
def test_unknown_negative_boolean_and_fractional_usage_fails_closed(self) -> None:
|
||||
init, assistant, result = self._fixture_lines()
|
||||
base = json.loads(result)
|
||||
cases = {
|
||||
"unknown-usage-key": {"usage": {"input_tokens": 1, "web_search_requests": 2}},
|
||||
"fractional-token": {"usage": {"input_tokens": 1.5}},
|
||||
"boolean-token": {"usage": {"input_tokens": True}},
|
||||
"negative-token": {"usage": {"input_tokens": -1}},
|
||||
"string-duration": {"duration_ms": "1234"},
|
||||
"boolean-duration": {"duration_ms": True},
|
||||
"negative-duration": {"duration_ms": -5},
|
||||
"non-object-usage": {"usage": [1, 2]},
|
||||
}
|
||||
for name, override in cases.items():
|
||||
with self.subTest(name=name):
|
||||
parser = ClaudeStreamParser(self.cell, "session-fixture")
|
||||
parser("stdout", init)
|
||||
parser("stdout", assistant)
|
||||
with self.assertRaises(ClaudeIopProtocolError):
|
||||
parser("stdout", json.dumps({**base, **override}))
|
||||
|
||||
def test_structural_redaction_never_retains_sensitive_content(self) -> None:
|
||||
raw = json.dumps({
|
||||
"type": "result", "result": "result-secret-sentinel",
|
||||
"message": "error-secret-sentinel",
|
||||
"content": "prompt-secret-sentinel",
|
||||
"tool_input": {"arguments": "tool-secret-sentinel"},
|
||||
"diagnostic": "https://private.iop.invalid api-secret-sentinel",
|
||||
})
|
||||
redacted = redact_claude_event(raw, SENTINELS)
|
||||
error_redacted = redact_claude_event(json.dumps({
|
||||
"type": "error", "message": "error-secret-sentinel",
|
||||
"error": {"detail": "tool-secret-sentinel"},
|
||||
}), SENTINELS)
|
||||
for sentinel in (*SENTINELS, *ARBITRARY_SENTINELS):
|
||||
self.assertNotIn(sentinel, redacted)
|
||||
self.assertNotIn(sentinel, error_redacted)
|
||||
self.assertIn("[redacted]", redacted)
|
||||
self.assertEqual(redact_claude_event("raw prompt-secret-sentinel", SENTINELS),
|
||||
'{"type":"invalid_claude_json"}')
|
||||
|
||||
def test_fake_cli_runs_once_and_durable_evidence_is_redacted(self) -> None:
|
||||
init, assistant, result = (json.loads(line) for line in self._fixture_lines())
|
||||
assistant["message"]["content"] = SENTINELS[0]
|
||||
assistant["tool_input"] = {"arguments": ARBITRARY_SENTINELS[0]}
|
||||
result["result"] = ARBITRARY_SENTINELS[1]
|
||||
diagnostic = {"type": "system", "subtype": "notice", "error": ARBITRARY_SENTINELS[2]}
|
||||
outcome, durable = self._run_fake([
|
||||
json.dumps(init), json.dumps(diagnostic), json.dumps(assistant), json.dumps(result),
|
||||
])
|
||||
result = outcome
|
||||
self.assertTrue(result.product.status == "succeeded", result)
|
||||
self.assertEqual(result.harness.reason, REASON_SUCCESS)
|
||||
self.assertTrue(result.submitted)
|
||||
self.assertTrue(result.harness.ordered_terminal)
|
||||
for sentinel in (*SENTINELS, *ARBITRARY_SENTINELS):
|
||||
self.assertNotIn(sentinel, durable)
|
||||
self.assertEqual(
|
||||
[metric.name for metric in result.metrics],
|
||||
[
|
||||
"total_duration", "model_duration", "model_calls",
|
||||
"cached_input_tokens", "input_tokens", "output_tokens",
|
||||
],
|
||||
)
|
||||
# Numeric observations survive the content redactors that removed every
|
||||
# sentinel above, and the durable journal keeps them in one closed shape.
|
||||
self.assertIn('"kind": "metric:total_duration"', durable)
|
||||
self.assertIn('\\"value\\":1234000000', durable)
|
||||
|
||||
def test_lifecycle_rejects_boundary_violations(self) -> None:
|
||||
init, assistant, result = self._fixture_lines()
|
||||
bad_model = json.loads(assistant)
|
||||
bad_model["message"]["model"] = "fallback-model"
|
||||
cases = {
|
||||
"missing-init": [assistant, result],
|
||||
"missing-result": [init, assistant],
|
||||
"duplicate-init": [init, init, assistant, result],
|
||||
"out-of-order-result": [init, result, assistant],
|
||||
"mismatched-model": [init, json.dumps(bad_model), result],
|
||||
"malformed-json": [init, "not-json", assistant, result],
|
||||
}
|
||||
for name, lines in cases.items():
|
||||
with self.subTest(name=name):
|
||||
outcome, durable = self._run_fake(lines)
|
||||
self.assertFalse(outcome.product.status == "succeeded", outcome)
|
||||
for sentinel in (*SENTINELS, *ARBITRARY_SENTINELS):
|
||||
self.assertNotIn(sentinel, durable)
|
||||
|
||||
def test_metric_prefixed_malformed_output_is_redacted_before_durable_capture(self) -> None:
|
||||
metric_sentinel = "metric:" + SENTINELS[0]
|
||||
outcome, durable = self._run_fake([metric_sentinel])
|
||||
self.assertFalse(outcome.product.status == "succeeded", outcome)
|
||||
self.assertEqual(outcome.harness.reason, REASON_PARSER_ERROR)
|
||||
self.assertTrue(outcome.submitted)
|
||||
self.assertIn("invalid_claude_json", durable)
|
||||
for sentinel in (*SENTINELS, metric_sentinel, SENTINELS[0]):
|
||||
self.assertNotIn(sentinel, durable)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,504 +0,0 @@
|
|||
"""Isolated, secret-safe Codex ``exec`` adapter for IOP benchmark cells.
|
||||
|
||||
The adapter deliberately owns no public runner command. It converts a frozen
|
||||
manifest cell and a caller-supplied runtime into one bounded lifecycle
|
||||
invocation. The only executable it starts is a small bridge in the lifecycle
|
||||
owned process group; that bridge starts ``codex exec``, forwards its JSONL, and
|
||||
emits an idle marker only after the child has exited and both output streams
|
||||
reached EOF.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Mapping
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
# The bridge is intentionally invoked by absolute script path from an isolated
|
||||
# workspace. Make the repository package importable without inheriting an
|
||||
# ambient PYTHONPATH.
|
||||
if __package__ in (None, ""):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CALLER_REASON_ERROR,
|
||||
CALLER_REASON_SUCCESS,
|
||||
CALLER_STATUS_FAILED,
|
||||
CALLER_STATUS_SUCCEEDED,
|
||||
COMPLETION_EXIT_AFTER_IDLE,
|
||||
SUBMISSION_STDIN_ONCE,
|
||||
CallerEvent,
|
||||
CallerTerminal,
|
||||
InvocationResult,
|
||||
InvocationSpec,
|
||||
LifecycleMetricError,
|
||||
LifecycleValidationError,
|
||||
ParsedMetric,
|
||||
SupervisorLocator,
|
||||
TLS_CA_ENV_KEYS,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
env_pairs,
|
||||
exact_value_redactor,
|
||||
inherited_tls_ca_environment,
|
||||
is_reported_number,
|
||||
run_invocation,
|
||||
)
|
||||
from scripts.agent_benchmark.connectivity import (
|
||||
ISSUE_RESUME_CODES,
|
||||
CallerCapability,
|
||||
ConnectivityIssue,
|
||||
ConnectivityResult,
|
||||
RequestedEffectiveBinding,
|
||||
make_result,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import MatrixCell, Timeout, TOKEN_RE
|
||||
from scripts.agent_benchmark.workspace import PreparedWorkspace
|
||||
|
||||
|
||||
PROVIDER_ID = "iop_benchmark"
|
||||
SECRET_ENV_KEY = "IOP_BENCHMARK_API_KEY"
|
||||
BASE_URL_ENV_KEY = "IOP_BENCHMARK_BASE_URL"
|
||||
SUPPORTED_EFFORTS = ("xhigh",)
|
||||
_BRIDGE_IDLE_TYPE = "adapter.idle"
|
||||
_BRIDGE_ID = "codex_iop"
|
||||
_SENSITIVE_KEYS = frozenset({
|
||||
"api_key", "authorization", "base_url", "command", "content", "endpoint",
|
||||
"input", "instructions", "message", "output", "prompt", "secret", "text",
|
||||
"token", "tool_input", "tool_output", "url",
|
||||
})
|
||||
_SAFE_STRING_KEYS = frozenset({
|
||||
"adapter", "effort", "model", "nonce", "reasoning_effort", "route_id",
|
||||
"route_kind", "stage", "status", "type",
|
||||
})
|
||||
# The exact reported Codex usage keys this adapter observes. A tool interval is
|
||||
# consumed only when the completed item explicitly pairs one bounded duration
|
||||
# with one call identifier; nothing is derived from event arrival order.
|
||||
_CODEX_USAGE_FIELDS = {
|
||||
"input_tokens": "input_tokens",
|
||||
"cached_input_tokens": "cached_input_tokens",
|
||||
"cache_write_input_tokens": "cache_write_tokens",
|
||||
"output_tokens": "output_tokens",
|
||||
"reasoning_output_tokens": "reasoning_tokens",
|
||||
"total_tokens": "total_tokens",
|
||||
}
|
||||
_CODEX_ITEM_COMPLETED = "item.completed"
|
||||
_CODEX_CALL_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.:-]{0,63}$")
|
||||
_CODEX_TOOL_ITEM_TYPES = frozenset({
|
||||
"command_execution", "file_change", "mcp_tool_call", "web_search",
|
||||
})
|
||||
|
||||
|
||||
class CodexIOPError(Exception):
|
||||
"""Base class for closed Codex adapter validation failures."""
|
||||
|
||||
|
||||
class CodexRuntimeError(CodexIOPError):
|
||||
"""The caller-supplied runtime is absent, malformed, or unsafe."""
|
||||
|
||||
|
||||
class CodexJSONLError(CodexIOPError):
|
||||
"""A Codex JSONL event cannot safely satisfy the adapter protocol."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CodexRuntime:
|
||||
"""The two runtime-only values needed to reach the IOP Responses surface."""
|
||||
|
||||
base_url: str
|
||||
api_key: str
|
||||
path: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CodexInvocation:
|
||||
"""One immutable spec plus its parser and capture redactor."""
|
||||
|
||||
spec: InvocationSpec
|
||||
parser: "CodexJSONLParser"
|
||||
redact: Callable[[str], str]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CodexInvocationResult:
|
||||
"""Lifecycle result and independently observed effective binding, if any."""
|
||||
|
||||
lifecycle: InvocationResult
|
||||
effective_binding: tuple[str, str, str, str] | None
|
||||
|
||||
|
||||
def runtime_from_environment(environment: Mapping[str, str]) -> CodexRuntime:
|
||||
"""Read exactly the named IOP runtime values without consulting ambient env."""
|
||||
if not isinstance(environment, Mapping):
|
||||
raise CodexRuntimeError("invalid runtime environment")
|
||||
required = {BASE_URL_ENV_KEY, SECRET_ENV_KEY, "PATH"}
|
||||
if set(environment) != required or not all(isinstance(key, str) for key in environment):
|
||||
raise CodexRuntimeError("runtime environment has unsupported keys")
|
||||
values = {key: environment[key] for key in required}
|
||||
if not all(isinstance(value, str) and value for value in values.values()):
|
||||
raise CodexRuntimeError("runtime environment is incomplete")
|
||||
parsed = urlsplit(values[BASE_URL_ENV_KEY])
|
||||
if parsed.scheme not in ("http", "https") or not parsed.netloc or parsed.query or parsed.fragment:
|
||||
raise CodexRuntimeError("invalid IOP base URL")
|
||||
return CodexRuntime(values[BASE_URL_ENV_KEY], values[SECRET_ENV_KEY], values["PATH"])
|
||||
|
||||
|
||||
def _require_cell(cell: MatrixCell) -> None:
|
||||
if not isinstance(cell, MatrixCell) or cell.caller != "codex":
|
||||
raise CodexRuntimeError("Codex adapter requires a Codex matrix cell")
|
||||
if cell.iop.requested_effort not in SUPPORTED_EFFORTS:
|
||||
raise CodexRuntimeError("unsupported Codex reasoning effort")
|
||||
if not TOKEN_RE.fullmatch(cell.iop.request_model):
|
||||
raise CodexRuntimeError("invalid requested model")
|
||||
|
||||
|
||||
def codex_capability() -> CallerCapability:
|
||||
"""The closed local capability claimed by this adapter implementation."""
|
||||
return CallerCapability("codex", ("direct", "execution_preset"), SUPPORTED_EFFORTS)
|
||||
|
||||
|
||||
def _require_prepared(prepared: PreparedWorkspace) -> None:
|
||||
if not isinstance(prepared, PreparedWorkspace) or not prepared.session_is_fresh:
|
||||
raise CodexRuntimeError("Codex invocation requires a fresh prepared workspace")
|
||||
if not Path(prepared.workspace_dir).is_dir() or not Path(prepared.session_dir).is_dir():
|
||||
raise CodexRuntimeError("prepared workspace is unavailable")
|
||||
if not Path(prepared.attempt_root).is_dir():
|
||||
raise CodexRuntimeError("prepared attempt root is unavailable")
|
||||
|
||||
|
||||
def _toml_string(value: str) -> str:
|
||||
if not isinstance(value, str) or any(ord(char) < 0x20 for char in value):
|
||||
raise CodexRuntimeError("invalid provider configuration value")
|
||||
# JSON strings are valid TOML basic strings and avoid hand-built quoting.
|
||||
return json.dumps(value, ensure_ascii=True)
|
||||
|
||||
|
||||
def _provider_overrides(cell: MatrixCell, runtime: CodexRuntime) -> tuple[str, ...]:
|
||||
return (
|
||||
f"model_provider={_toml_string(PROVIDER_ID)}",
|
||||
f"model_providers.{PROVIDER_ID}.name={_toml_string('IOP Benchmark')}",
|
||||
f"model_providers.{PROVIDER_ID}.base_url={_toml_string(runtime.base_url)}",
|
||||
f"model_providers.{PROVIDER_ID}.env_key={_toml_string(SECRET_ENV_KEY)}",
|
||||
f"shell_environment_policy.filters.{SECRET_ENV_KEY}={_toml_string('exclude')}",
|
||||
f"model_providers.{PROVIDER_ID}.wire_api={_toml_string('responses')}",
|
||||
f"model_reasoning_effort={_toml_string(cell.iop.requested_effort)}",
|
||||
)
|
||||
|
||||
|
||||
def build_codex_spec(
|
||||
cell: MatrixCell,
|
||||
prepared: PreparedWorkspace,
|
||||
runtime: CodexRuntime,
|
||||
task_payload: bytes,
|
||||
timeout: Timeout,
|
||||
*,
|
||||
codex_executable: str | tuple[str, ...] = "codex",
|
||||
) -> InvocationSpec:
|
||||
"""Build one isolated Codex invocation without reading user configuration.
|
||||
|
||||
The secret remains only in the child environment for provider
|
||||
authentication. ``shell_environment_policy.filters`` explicitly excludes
|
||||
the secret from shell tool environments and snapshots so it never
|
||||
serializes into durable evidence. The base URL is an ephemeral process
|
||||
argument required by Codex's provider override; neither is serialized
|
||||
into fixture, capture, lifecycle result, or durable config.
|
||||
"""
|
||||
_require_cell(cell)
|
||||
_require_prepared(prepared)
|
||||
if not isinstance(runtime, CodexRuntime):
|
||||
raise CodexRuntimeError("invalid Codex runtime")
|
||||
if not isinstance(task_payload, bytes) or not task_payload:
|
||||
raise CodexRuntimeError("Codex task payload is required")
|
||||
try:
|
||||
task_payload.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise CodexRuntimeError("Codex task payload must be UTF-8") from exc
|
||||
if not isinstance(timeout, Timeout):
|
||||
raise CodexRuntimeError("invalid timeout")
|
||||
if isinstance(codex_executable, str):
|
||||
executable_argv = (codex_executable,)
|
||||
elif isinstance(codex_executable, tuple) and codex_executable and all(
|
||||
isinstance(item, str) and item for item in codex_executable
|
||||
):
|
||||
# Test-only seam for a non-executable fixture script; normal callers
|
||||
# always receive the exact single-token ``codex`` command above.
|
||||
executable_argv = codex_executable
|
||||
else:
|
||||
raise CodexRuntimeError("invalid Codex executable")
|
||||
|
||||
codex_argv: list[str] = [
|
||||
*executable_argv, "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
|
||||
"--strict-config", "--skip-git-repo-check", "-C", prepared.workspace_dir,
|
||||
"-m", cell.iop.request_model,
|
||||
]
|
||||
for override in _provider_overrides(cell, runtime):
|
||||
codex_argv.extend(("-c", override))
|
||||
codex_argv.append("-")
|
||||
return InvocationSpec(
|
||||
# The lifecycle changes cwd to the isolated workspace, so use this
|
||||
# module's absolute script path rather than relying on repository
|
||||
# import resolution in the child bridge.
|
||||
argv=(sys.executable, str(Path(__file__).resolve()), "--bridge", "--", *codex_argv),
|
||||
cwd=prepared.workspace_dir,
|
||||
env=env_pairs({
|
||||
"PATH": runtime.path,
|
||||
"HOME": prepared.session_dir,
|
||||
SECRET_ENV_KEY: runtime.api_key,
|
||||
**dict(inherited_tls_ca_environment()),
|
||||
}),
|
||||
env_allowlist=(SECRET_ENV_KEY, *TLS_CA_ENV_KEYS),
|
||||
submission_mode=SUBMISSION_STDIN_ONCE,
|
||||
completion_mode=COMPLETION_EXIT_AFTER_IDLE,
|
||||
timeout=timeout,
|
||||
evidence_dir=prepared.attempt_root,
|
||||
task_payload=task_payload,
|
||||
control_dir=str(Path(prepared.attempt_root) / "codex-control"),
|
||||
)
|
||||
|
||||
|
||||
def redact_codex_jsonl(text: str, secrets: tuple[str, ...]) -> str:
|
||||
"""Remove JSON content and runtime values before any line becomes evidence."""
|
||||
exact = exact_value_redactor(secrets)
|
||||
try:
|
||||
value = json.loads(text)
|
||||
except (TypeError, json.JSONDecodeError):
|
||||
return "[redacted]"
|
||||
|
||||
def redact(value: Any, key: str | None = None) -> Any:
|
||||
if key is not None and key.lower() in _SENSITIVE_KEYS:
|
||||
return "[redacted]"
|
||||
if isinstance(value, dict):
|
||||
return {str(item_key): redact(item_value, str(item_key)) for item_key, item_value in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [redact(item) for item in value]
|
||||
if isinstance(value, str):
|
||||
return exact(value) if key in _SAFE_STRING_KEYS else "[redacted]"
|
||||
return value
|
||||
|
||||
return json.dumps(redact(value), ensure_ascii=True, separators=(",", ":"))
|
||||
|
||||
|
||||
class CodexJSONLParser:
|
||||
"""Parse only terminal evidence and optional explicit IOP effective binding."""
|
||||
|
||||
def __init__(self, cell: MatrixCell, idle_nonce: str) -> None:
|
||||
_require_cell(cell)
|
||||
if not isinstance(idle_nonce, str) or len(idle_nonce) < 16:
|
||||
raise CodexRuntimeError("invalid bridge idle nonce")
|
||||
self._cell = cell
|
||||
self._idle_nonce = idle_nonce
|
||||
self._effective_binding: tuple[str, str, str, str] | None = None
|
||||
self._turns = 0
|
||||
self._tool_calls: set[str] = set()
|
||||
|
||||
@property
|
||||
def effective_binding(self) -> tuple[str, str, str, str] | None:
|
||||
return self._effective_binding
|
||||
|
||||
def connectivity_result(self) -> ConnectivityResult:
|
||||
"""Classify absent evidence without synthesizing unreported stage bindings."""
|
||||
iop = self._cell.iop
|
||||
if self._effective_binding is not None:
|
||||
# Codex's optional scalar observation proves neither the complete
|
||||
# stage list nor its exact order. Never manufacture that missing
|
||||
# contract evidence from the manifest just to produce ``ready``.
|
||||
raise CodexJSONLError("effective stage binding is unavailable")
|
||||
binding = RequestedEffectiveBinding(
|
||||
self._cell.id, self._cell.caller, iop.route_kind, iop.route_id,
|
||||
iop.request_model, iop.requested_effort,
|
||||
None, None, None, None, (),
|
||||
)
|
||||
issues: tuple[ConnectivityIssue, ...] = (
|
||||
ConnectivityIssue("stream_incompatible", ISSUE_RESUME_CODES["stream_incompatible"]),
|
||||
)
|
||||
return make_result(self._cell, codex_capability(), binding, issues)
|
||||
|
||||
def parse(self, stream: str, line: str) -> Any:
|
||||
if stream != "stdout":
|
||||
return None
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except (TypeError, json.JSONDecodeError) as exc:
|
||||
raise CodexJSONLError("malformed Codex JSONL") from exc
|
||||
if not isinstance(record, dict) or not isinstance(record.get("type"), str):
|
||||
raise CodexJSONLError("malformed Codex JSONL")
|
||||
self._observe_effective_binding(record)
|
||||
if record["type"] == "turn.completed":
|
||||
status = record.get("status")
|
||||
if status is not None and status not in ("completed", "success"):
|
||||
return (
|
||||
CallerTerminal(CALLER_STATUS_FAILED, CALLER_REASON_ERROR),
|
||||
CallerEvent("finish"),
|
||||
)
|
||||
self._turns += 1
|
||||
return (
|
||||
*self._turn_observations(record),
|
||||
CallerTerminal(CALLER_STATUS_SUCCEEDED, CALLER_REASON_SUCCESS),
|
||||
CallerEvent("finish"),
|
||||
)
|
||||
if record["type"] == _CODEX_ITEM_COMPLETED:
|
||||
return self._tool_interval(record)
|
||||
if record["type"] == _BRIDGE_IDLE_TYPE:
|
||||
if record != {"type": _BRIDGE_IDLE_TYPE, "adapter": _BRIDGE_ID, "nonce": self._idle_nonce, "child_exit": 0}:
|
||||
raise CodexJSONLError("unverified bridge idle marker")
|
||||
return CallerEvent("idle")
|
||||
return None
|
||||
|
||||
def _turn_observations(self, record: dict[str, Any]) -> tuple[ParsedMetric, ...]:
|
||||
"""Observe reported turn usage plus the turn and tool call counts."""
|
||||
model = self._cell.iop.request_model
|
||||
usage = record.get("usage")
|
||||
if usage is not None and (
|
||||
not isinstance(usage, dict) or not set(usage) <= set(_CODEX_USAGE_FIELDS)
|
||||
):
|
||||
raise CodexJSONLError("invalid Codex usage observation")
|
||||
try:
|
||||
observations = [
|
||||
count_metric(_CODEX_USAGE_FIELDS[field], value, model=model)
|
||||
for field, value in sorted((usage or {}).items())
|
||||
]
|
||||
observations.append(count_metric("model_calls", self._turns, model=model))
|
||||
observations.append(
|
||||
count_metric("tool_calls", len(self._tool_calls), model=model)
|
||||
)
|
||||
except LifecycleMetricError as exc:
|
||||
raise CodexJSONLError("invalid Codex usage observation") from exc
|
||||
return tuple(observations)
|
||||
|
||||
def _tool_interval(self, record: dict[str, Any]) -> ParsedMetric | None:
|
||||
"""Count a completed tool and optionally observe its reported interval."""
|
||||
item = record.get("item")
|
||||
if not isinstance(item, dict) or item.get("type") not in _CODEX_TOOL_ITEM_TYPES:
|
||||
return None
|
||||
call_id = item.get("id")
|
||||
if not isinstance(call_id, str) or _CODEX_CALL_ID_RE.fullmatch(call_id) is None:
|
||||
raise CodexJSONLError("unpaired Codex tool completion")
|
||||
if call_id in self._tool_calls:
|
||||
raise CodexJSONLError("duplicate Codex tool completion")
|
||||
if "duration_ms" not in item:
|
||||
self._tool_calls.add(call_id)
|
||||
return None
|
||||
if not is_reported_number(item["duration_ms"]):
|
||||
raise CodexJSONLError("invalid Codex tool interval")
|
||||
try:
|
||||
observation = duration_metric(
|
||||
"tool_duration", item["duration_ms"], reported_unit="ms",
|
||||
model=self._cell.iop.request_model, call_id=call_id, overlap=True,
|
||||
)
|
||||
except LifecycleMetricError as exc:
|
||||
raise CodexJSONLError("invalid Codex tool interval") from exc
|
||||
self._tool_calls.add(call_id)
|
||||
return observation
|
||||
|
||||
def _observe_effective_binding(self, record: dict[str, Any]) -> None:
|
||||
observed = record.get("iop_effective_binding")
|
||||
if observed is None:
|
||||
return
|
||||
if not isinstance(observed, dict) or set(observed) != {"route_kind", "route_id", "model", "effort"}:
|
||||
raise CodexJSONLError("invalid effective binding observation")
|
||||
values = tuple(observed[key] for key in ("route_kind", "route_id", "model", "effort"))
|
||||
if not all(isinstance(value, str) and TOKEN_RE.fullmatch(value) for value in values):
|
||||
raise CodexJSONLError("invalid effective binding observation")
|
||||
expected = (
|
||||
self._cell.iop.route_kind,
|
||||
self._cell.iop.route_id,
|
||||
self._cell.iop.request_model,
|
||||
self._cell.iop.requested_effort,
|
||||
)
|
||||
if values != expected:
|
||||
raise CodexJSONLError("effective binding substitution")
|
||||
if self._effective_binding is not None:
|
||||
raise CodexJSONLError("duplicate effective binding observation")
|
||||
self._effective_binding = values # type: ignore[assignment]
|
||||
|
||||
|
||||
def build_codex_invocation(
|
||||
cell: MatrixCell,
|
||||
prepared: PreparedWorkspace,
|
||||
runtime: CodexRuntime,
|
||||
task_payload: bytes,
|
||||
timeout: Timeout,
|
||||
*,
|
||||
codex_executable: str | tuple[str, ...] = "codex",
|
||||
) -> CodexInvocation:
|
||||
"""Pair the closed invocation spec with its parser and structural redactor."""
|
||||
nonce = secrets.token_hex(16)
|
||||
spec = build_codex_spec(cell, prepared, runtime, task_payload, timeout, codex_executable=codex_executable)
|
||||
parser = CodexJSONLParser(cell, nonce)
|
||||
# The nonce must reach the bridge but not Codex. Put it before the bridge
|
||||
# delimiter so the bridge removes it before starting the child.
|
||||
argv = (*spec.argv[:3], f"--idle-nonce={nonce}", *spec.argv[3:])
|
||||
task_text = task_payload.decode("utf-8")
|
||||
return CodexInvocation(spec=InvocationSpec(**{**spec.__dict__, "argv": argv}), parser=parser,
|
||||
redact=lambda line: redact_codex_jsonl(line, (runtime.base_url, runtime.api_key, task_text)))
|
||||
|
||||
|
||||
def run_codex_invocation(
|
||||
invocation: CodexInvocation,
|
||||
on_started: Callable[[SupervisorLocator], None],
|
||||
) -> CodexInvocationResult:
|
||||
"""Run exactly one prepared Codex invocation through the generic lifecycle."""
|
||||
if not isinstance(invocation, CodexInvocation) or not callable(on_started):
|
||||
raise CodexRuntimeError("invalid Codex invocation")
|
||||
result = run_invocation(invocation.spec, parse_event=invocation.parser.parse,
|
||||
on_started=on_started, redact=invocation.redact)
|
||||
return CodexInvocationResult(result, invocation.parser.effective_binding)
|
||||
|
||||
|
||||
def _forward_stream(stream: Any, destination: Any) -> None:
|
||||
try:
|
||||
for line in iter(stream.readline, b""):
|
||||
destination.buffer.write(line)
|
||||
destination.buffer.flush()
|
||||
finally:
|
||||
stream.close()
|
||||
|
||||
|
||||
def _bridge(argv: list[str]) -> int:
|
||||
"""Own a child Codex process and emit idle only after exit plus both EOFs."""
|
||||
if not argv or not argv[0].startswith("--idle-nonce="):
|
||||
return 64
|
||||
nonce = argv.pop(0).partition("=")[2]
|
||||
if not nonce or not argv or argv.pop(0) != "--":
|
||||
return 64
|
||||
try:
|
||||
child = subprocess.Popen(argv, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
|
||||
except OSError:
|
||||
return 127
|
||||
assert child.stdin is not None and child.stdout is not None and child.stderr is not None
|
||||
payload = sys.stdin.buffer.read()
|
||||
try:
|
||||
child.stdin.write(payload)
|
||||
child.stdin.close()
|
||||
except OSError:
|
||||
child.terminate()
|
||||
stdout_thread = threading.Thread(target=_forward_stream, args=(child.stdout, sys.stdout), daemon=True)
|
||||
stderr_thread = threading.Thread(target=_forward_stream, args=(child.stderr, sys.stderr), daemon=True)
|
||||
stdout_thread.start()
|
||||
stderr_thread.start()
|
||||
exit_code = child.wait()
|
||||
stdout_thread.join()
|
||||
stderr_thread.join()
|
||||
if exit_code == 0:
|
||||
sys.stdout.write(json.dumps({"type": _BRIDGE_IDLE_TYPE, "adapter": _BRIDGE_ID, "nonce": nonce, "child_exit": 0}, separators=(",", ":")) + "\n")
|
||||
sys.stdout.flush()
|
||||
return exit_code
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
values = list(sys.argv[1:] if argv is None else argv)
|
||||
if not values or values.pop(0) != "--bridge":
|
||||
return 64
|
||||
return _bridge(values)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -1,294 +0,0 @@
|
|||
"""Credential-free tests for the isolated Codex IOP benchmark adapter."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from scripts.agent_benchmark.codex_iop import (
|
||||
BASE_URL_ENV_KEY,
|
||||
SECRET_ENV_KEY,
|
||||
CodexJSONLError,
|
||||
CodexJSONLParser,
|
||||
CodexRuntimeError,
|
||||
build_codex_invocation,
|
||||
build_codex_spec,
|
||||
redact_codex_jsonl,
|
||||
run_codex_invocation,
|
||||
runtime_from_environment,
|
||||
)
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CallerEvent,
|
||||
ParsedMetric,
|
||||
REASON_DUPLICATE_EVENT,
|
||||
REASON_NONZERO_EXIT,
|
||||
REASON_PARSER_ERROR,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCell, Timeout
|
||||
from scripts.agent_benchmark.workspace import AttemptIdentity, PreparedWorkspace, TestbedProvenance
|
||||
|
||||
|
||||
_ENDPOINT = "https://iop.private.example/v1"
|
||||
_SECRET = "iop_test_secret_123456789"
|
||||
_PROMPT = b"private benchmark prompt must not persist"
|
||||
|
||||
|
||||
def _cell(*, effort: str = "xhigh") -> MatrixCell:
|
||||
return MatrixCell(
|
||||
"codex-gpt", "codex",
|
||||
IopCell("gpt-5.6-luna", effort, "direct", "iop-gpt", (
|
||||
ExpectedBinding("request", "gpt-5.6-luna", effort),
|
||||
)),
|
||||
)
|
||||
|
||||
|
||||
class CodexIOPTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = Path(self.tmp.name)
|
||||
self.workspace = self.root / "workspace"
|
||||
self.workspace.mkdir()
|
||||
self.session = self.root / "session"
|
||||
self.session.mkdir()
|
||||
self.evidence = self.root / "attempt"
|
||||
self.evidence.mkdir()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.tmp.cleanup()
|
||||
|
||||
def _runtime(self):
|
||||
return runtime_from_environment({
|
||||
BASE_URL_ENV_KEY: _ENDPOINT,
|
||||
SECRET_ENV_KEY: _SECRET,
|
||||
"PATH": os.environ.get("PATH", "/usr/bin:/bin"),
|
||||
})
|
||||
|
||||
def _prepared(self) -> PreparedWorkspace:
|
||||
return PreparedWorkspace(
|
||||
identity=AttemptIdentity("run-0001", "codex-gpt", 1, 1),
|
||||
attempt_root=str(self.evidence), workspace_dir=str(self.workspace),
|
||||
session_dir=str(self.session), session_id="fresh-session", session_is_fresh=True,
|
||||
workspace_checksum="sha256:" + "0" * 64, setup_cache_policy="isolated",
|
||||
testbed_provenance=TestbedProvenance("../iop-s2", "main", "0" * 40, "0" * 64, True),
|
||||
prepared_at="2026-08-10T00:00:00+00:00",
|
||||
)
|
||||
|
||||
def _timeout(self) -> Timeout:
|
||||
return Timeout(5, 2, 1, 1)
|
||||
|
||||
def _fake_codex(self, records: list[object], exit_code: int = 0) -> str:
|
||||
path = self.root / f"fake-codex-{len(list(self.root.glob('fake-codex-*')))}.py"
|
||||
lines = [
|
||||
"#!/usr/bin/env python3", "import json, sys", "task = sys.stdin.read()",
|
||||
]
|
||||
for record in records:
|
||||
if record == "TASK":
|
||||
lines.append("print(json.dumps({'type': 'turn.completed', 'status': 'completed', 'content': task}), flush=True)")
|
||||
elif isinstance(record, str):
|
||||
lines.append(f"print({record!r}, flush=True)")
|
||||
else:
|
||||
lines.append(f"print(json.dumps({record!r}), flush=True)")
|
||||
lines.append(f"raise SystemExit({exit_code})")
|
||||
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
return str(path)
|
||||
|
||||
def test_exact_isolated_responses_spec_uses_one_stdin_submission(self) -> None:
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
|
||||
argv = list(spec.argv)
|
||||
codex = argv[argv.index("--") + 1:]
|
||||
self.assertFalse((self.workspace / ".git").exists())
|
||||
self.assertEqual(codex.count("--skip-git-repo-check"), 1)
|
||||
self.assertEqual(codex[:14], [
|
||||
"codex", "exec", "--sandbox", "workspace-write", "--json", "--ephemeral", "--ignore-user-config",
|
||||
"--strict-config", "--skip-git-repo-check", "-C", str(self.workspace),
|
||||
"-m", "gpt-5.6-luna", "-c",
|
||||
])
|
||||
self.assertEqual(codex[-1], "-")
|
||||
overrides = [codex[index + 1] for index, item in enumerate(codex[:-1]) if item == "-c"]
|
||||
self.assertEqual(overrides, [
|
||||
'model_provider="iop_benchmark"',
|
||||
'model_providers.iop_benchmark.name="IOP Benchmark"',
|
||||
f'model_providers.iop_benchmark.base_url="{_ENDPOINT}"',
|
||||
'model_providers.iop_benchmark.env_key="IOP_BENCHMARK_API_KEY"',
|
||||
'shell_environment_policy.filters.IOP_BENCHMARK_API_KEY="exclude"',
|
||||
'model_providers.iop_benchmark.wire_api="responses"',
|
||||
'model_reasoning_effort="xhigh"',
|
||||
])
|
||||
self.assertEqual(spec.submission_mode, "stdin_once")
|
||||
self.assertEqual(spec.completion_mode, "exit_after_idle")
|
||||
self.assertEqual(dict(spec.env)[SECRET_ENV_KEY], _SECRET)
|
||||
self.assertEqual(dict(spec.env)["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(dict(spec.env)["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertNotIn(BASE_URL_ENV_KEY, dict(spec.env))
|
||||
self.assertNotIn("OPENAI_API_KEY", dict(spec.env))
|
||||
|
||||
def test_provider_secret_is_explicitly_excluded_from_shell_environment(self) -> None:
|
||||
"""Regression: the provider env_key injects the secret into the child
|
||||
environment for authentication, but the same secret is explicitly
|
||||
excluded from shell tool environments and snapshots.
|
||||
|
||||
Provider authentication, shell exclusion, and the exact invocation
|
||||
spec are one invariant; both must hold together.
|
||||
"""
|
||||
with patch.dict(os.environ, {"SSL_CERT_FILE": "/operator/dev-ca.pem", "NODE_EXTRA_CA_CERTS": "/operator/dev-ca.pem"}):
|
||||
spec = build_codex_spec(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout())
|
||||
env = dict(spec.env)
|
||||
# Provider authentication: the secret must still reach the child process.
|
||||
self.assertEqual(env[SECRET_ENV_KEY], _SECRET)
|
||||
# Shell exclusion: the exact override must appear in the invocation.
|
||||
argv = list(spec.argv)
|
||||
codex = argv[argv.index("--") + 1:]
|
||||
overrides = {codex[index + 1] for index, item in enumerate(codex[:-1]) if item == "-c"}
|
||||
self.assertIn(
|
||||
f'shell_environment_policy.filters.IOP_BENCHMARK_API_KEY="exclude"',
|
||||
overrides,
|
||||
"the benchmark secret must be explicitly excluded from shell environments",
|
||||
)
|
||||
# TLS variables, isolated HOME, and no leaking caller keys.
|
||||
self.assertEqual(env["SSL_CERT_FILE"], "/operator/dev-ca.pem")
|
||||
self.assertEqual(env["NODE_EXTRA_CA_CERTS"], "/operator/dev-ca.pem")
|
||||
self.assertNotIn(BASE_URL_ENV_KEY, env)
|
||||
self.assertNotIn("OPENAI_API_KEY", env)
|
||||
|
||||
def test_runtime_and_effort_are_closed(self) -> None:
|
||||
with self.assertRaises(CodexRuntimeError):
|
||||
runtime_from_environment({BASE_URL_ENV_KEY: _ENDPOINT, SECRET_ENV_KEY: _SECRET})
|
||||
with self.assertRaises(CodexRuntimeError):
|
||||
runtime_from_environment({BASE_URL_ENV_KEY: _ENDPOINT, SECRET_ENV_KEY: _SECRET, "PATH": "/bin", "EXTRA": "x"})
|
||||
with self.assertRaises(CodexRuntimeError):
|
||||
build_codex_spec(_cell(effort="high"), self._prepared(), self._runtime(), _PROMPT, self._timeout())
|
||||
|
||||
def test_fixture_finish_then_verified_idle_and_structural_redaction(self) -> None:
|
||||
parser = CodexJSONLParser(_cell(), "fixture-nonce-0001")
|
||||
fixture = Path("scripts/fixtures/agent-comparison-benchmark/codex-iop-stream.jsonl")
|
||||
events = [parser.parse("stdout", line) for line in fixture.read_text(encoding="utf-8").splitlines()]
|
||||
self.assertEqual([events[0], events[-1]], [None, CallerEvent("idle")])
|
||||
tool = events[1]
|
||||
self.assertEqual((tool.name, tool.value, tool.call_id), ("tool_duration", 7_250_000, "call-1"))
|
||||
# A tool interval is reported inside the turn, so it is published as an
|
||||
# overlapping interval instead of a subtractable slice.
|
||||
self.assertTrue(tool.overlap)
|
||||
self.assertEqual(events[2][-1], CallerEvent("finish"))
|
||||
turn = {metric.name: metric.value for metric in events[2] if isinstance(metric, ParsedMetric)}
|
||||
self.assertEqual(turn, {
|
||||
"cache_write_tokens": 6, "cached_input_tokens": 8,
|
||||
"input_tokens": 31, "output_tokens": 12, "reasoning_tokens": 4,
|
||||
"model_calls": 1, "tool_calls": 1,
|
||||
})
|
||||
# The fixture omits the provider total, so it is never reconstructed
|
||||
# from the reported categories.
|
||||
self.assertNotIn("total_tokens", turn)
|
||||
redacted = redact_codex_jsonl(
|
||||
json.dumps({"type": "turn.completed", "content": _PROMPT.decode(), "endpoint": _ENDPOINT, "token": _SECRET}),
|
||||
(_ENDPOINT, _SECRET, _PROMPT.decode()),
|
||||
)
|
||||
self.assertNotIn(_PROMPT.decode(), redacted)
|
||||
self.assertNotIn(_ENDPOINT, redacted)
|
||||
self.assertNotIn(_SECRET, redacted)
|
||||
self.assertIn("[redacted]", redacted)
|
||||
|
||||
def test_bridge_proves_finish_then_idle_after_child_exit(self) -> None:
|
||||
fake = self._fake_codex(["TASK"])
|
||||
invocation = build_codex_invocation(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake))
|
||||
result = run_codex_invocation(invocation, lambda _: None)
|
||||
self.assertTrue(result.lifecycle.product.status == "succeeded")
|
||||
self.assertEqual([event.kind for event in result.lifecycle.events], [
|
||||
"submitted", "first_output", "metric:model_calls", "metric:tool_calls",
|
||||
"caller_terminal", "finish", "idle", "exited", "quiet",
|
||||
])
|
||||
capture = result.lifecycle.stdout.text
|
||||
self.assertNotIn(_PROMPT.decode(), capture)
|
||||
self.assertNotIn(_ENDPOINT, capture)
|
||||
self.assertNotIn(_SECRET, capture)
|
||||
self.assertIsNone(result.effective_binding)
|
||||
|
||||
def test_child_failure_never_synthesizes_idle(self) -> None:
|
||||
fake = self._fake_codex([{"type": "turn.completed", "status": "completed"}], exit_code=7)
|
||||
result = run_codex_invocation(build_codex_invocation(_cell(), self._prepared(), self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake)), lambda _: None)
|
||||
self.assertFalse(result.lifecycle.product.status == "succeeded")
|
||||
self.assertEqual(result.lifecycle.harness.reason, REASON_NONZERO_EXIT)
|
||||
self.assertNotIn("idle", [event.kind for event in result.lifecycle.events])
|
||||
|
||||
def test_duplicate_malformed_and_unverified_idle_fail_closed(self) -> None:
|
||||
cases = (
|
||||
([{"type": "turn.completed", "status": "completed"}, {"type": "turn.completed", "status": "completed"}], REASON_DUPLICATE_EVENT),
|
||||
(["not-json"], REASON_PARSER_ERROR),
|
||||
([{"type": "adapter.idle", "adapter": "codex_iop", "nonce": "not-the-bridge-nonce", "child_exit": 0}], REASON_PARSER_ERROR),
|
||||
)
|
||||
for index, (records, reason) in enumerate(cases):
|
||||
with self.subTest(reason=reason):
|
||||
evidence = self.root / f"attempt-{reason}-{index}"
|
||||
evidence.mkdir()
|
||||
prepared = self._prepared().__class__(**{**self._prepared().__dict__, "attempt_root": str(evidence)})
|
||||
fake = self._fake_codex(records)
|
||||
result = run_codex_invocation(build_codex_invocation(_cell(), prepared, self._runtime(), _PROMPT, self._timeout(), codex_executable=(sys.executable, fake)), lambda _: None)
|
||||
self.assertFalse(result.lifecycle.product.status == "succeeded")
|
||||
self.assertEqual(result.lifecycle.harness.reason, reason)
|
||||
|
||||
def test_tool_intervals_require_one_explicit_unique_pairing(self) -> None:
|
||||
parser = CodexJSONLParser(_cell(), "0123456789abcdef")
|
||||
# A non-tool item carries neither a tool count nor an interval.
|
||||
self.assertIsNone(parser.parse("stdout", json.dumps(
|
||||
{"type": "item.completed", "item": {"type": "agent_message", "text": "x"}}
|
||||
)))
|
||||
# Real Codex command completions omit duration; they must still count.
|
||||
self.assertIsNone(parser.parse("stdout", json.dumps(
|
||||
{"type": "item.completed", "item": {"id": "call-0", "type": "command_execution"}}
|
||||
)))
|
||||
paired = json.dumps({"type": "item.completed", "item": {
|
||||
"id": "call-1", "type": "command_execution", "duration_ms": 3,
|
||||
}})
|
||||
self.assertEqual(parser.parse("stdout", paired).call_id, "call-1")
|
||||
for name, item in {
|
||||
"duplicate-call": {"id": "call-1", "type": "command_execution", "duration_ms": 4},
|
||||
"unpaired-duration": {"type": "command_execution", "duration_ms": 4},
|
||||
"unsafe-call-id": {"id": "call 1", "type": "command_execution", "duration_ms": 4},
|
||||
"string-duration": {"id": "call-2", "type": "command_execution", "duration_ms": "4"},
|
||||
"negative-duration": {"id": "call-3", "type": "command_execution", "duration_ms": -4},
|
||||
}.items():
|
||||
with self.subTest(name=name):
|
||||
with self.assertRaises(CodexJSONLError):
|
||||
parser.parse("stdout", json.dumps({"type": "item.completed", "item": item}))
|
||||
turn = parser.parse("stdout", json.dumps({"type": "turn.completed"}))
|
||||
counts = {metric.name: metric.value for metric in turn if isinstance(metric, ParsedMetric)}
|
||||
self.assertEqual(counts, {"model_calls": 1, "tool_calls": 2})
|
||||
|
||||
def test_unknown_or_fractional_turn_usage_fails_closed(self) -> None:
|
||||
for usage in (
|
||||
{"unknown_tokens": 1},
|
||||
{"input_tokens": 1.5},
|
||||
{"input_tokens": True},
|
||||
{"input_tokens": -1},
|
||||
):
|
||||
with self.subTest(usage=usage):
|
||||
parser = CodexJSONLParser(_cell(), "0123456789abcdef")
|
||||
with self.assertRaises(CodexJSONLError):
|
||||
parser.parse("stdout", json.dumps({
|
||||
"type": "turn.completed", "status": "completed", "usage": usage,
|
||||
}))
|
||||
|
||||
def test_effective_binding_is_optional_but_any_observation_is_exact(self) -> None:
|
||||
parser = CodexJSONLParser(_cell(), "0123456789abcdef")
|
||||
self.assertIsNone(parser.effective_binding)
|
||||
self.assertEqual(parser.connectivity_result().status, "implementation_gap")
|
||||
parser.parse("stdout", json.dumps({"type": "thread.started", "iop_effective_binding": {
|
||||
"route_kind": "direct", "route_id": "iop-gpt", "model": "gpt-5.6-luna", "effort": "xhigh",
|
||||
}}))
|
||||
self.assertEqual(parser.effective_binding, ("direct", "iop-gpt", "gpt-5.6-luna", "xhigh"))
|
||||
with self.assertRaises(CodexJSONLError):
|
||||
parser.connectivity_result()
|
||||
mismatch = CodexJSONLParser(_cell(), "0123456789abcdef")
|
||||
with self.assertRaises(CodexJSONLError):
|
||||
mismatch.parse("stdout", json.dumps({"type": "thread.started", "iop_effective_binding": {
|
||||
"route_kind": "direct", "route_id": "iop-gpt", "model": "gpt-alias", "effort": "xhigh",
|
||||
}}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,679 +0,0 @@
|
|||
"""Closed, secret-safe connectivity preflight contracts for benchmark cells.
|
||||
|
||||
This module deliberately has no caller, provider, or network dependency. Caller
|
||||
adapters provide typed observations, while this boundary proves that those
|
||||
observations exactly match one immutable manifest cell before they can be stored.
|
||||
A blocked caller may report that it observed no effective binding at all, but it
|
||||
can never report a partial or manifest-derived synthetic one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Protocol
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
CALLER_ENUM,
|
||||
ROUTE_KIND_ENUM,
|
||||
STAGE_ENUM,
|
||||
STAGE_RANK,
|
||||
ExpectedBinding,
|
||||
MatrixCell,
|
||||
TOKEN_RE,
|
||||
)
|
||||
|
||||
|
||||
SCHEMA_VERSION = "1"
|
||||
IDENTITY_RE = re.compile(r"^sha256:[0-9a-f]{64}$")
|
||||
MAX_EVIDENCE_BYTES = 65536
|
||||
|
||||
# Canonical issue order: registration blockers first, then implementation gaps.
|
||||
ISSUE_CODE_ORDER = (
|
||||
"credential_missing",
|
||||
"model_missing",
|
||||
"route_missing",
|
||||
"effort_unsupported",
|
||||
"endpoint_incompatible",
|
||||
"auth_incompatible",
|
||||
"protocol_incompatible",
|
||||
"stream_incompatible",
|
||||
)
|
||||
REGISTRATION_ISSUE_CODES = frozenset(ISSUE_CODE_ORDER[:4])
|
||||
IMPLEMENTATION_ISSUE_CODES = frozenset(ISSUE_CODE_ORDER[4:])
|
||||
ISSUE_CODES = REGISTRATION_ISSUE_CODES | IMPLEMENTATION_ISSUE_CODES
|
||||
ISSUE_RANK: dict[str, int] = {code: rank for rank, code in enumerate(ISSUE_CODE_ORDER)}
|
||||
# The only resume vocabulary; callers can never attach their own text.
|
||||
ISSUE_RESUME_CODES: dict[str, str] = {
|
||||
"credential_missing": "register_credential",
|
||||
"model_missing": "register_model",
|
||||
"route_missing": "register_route",
|
||||
"effort_unsupported": "register_effort_support",
|
||||
"endpoint_incompatible": "implement_endpoint_adapter",
|
||||
"auth_incompatible": "implement_auth_adapter",
|
||||
"protocol_incompatible": "implement_protocol_adapter",
|
||||
"stream_incompatible": "implement_stream_adapter",
|
||||
}
|
||||
RESUME_CODES = frozenset(ISSUE_RESUME_CODES.values())
|
||||
RESULT_STATUSES = ("ready", "registration_required", "implementation_gap")
|
||||
EFFECTIVE_FIELDS = (
|
||||
"effective_route_kind",
|
||||
"effective_route_id",
|
||||
"effective_model",
|
||||
"effective_effort",
|
||||
)
|
||||
|
||||
|
||||
class ConnectivityError(Exception):
|
||||
"""Base class for closed connectivity contract failures."""
|
||||
|
||||
|
||||
class ConnectivityValidationError(ConnectivityError):
|
||||
"""Raised when a caller observation or evidence value is not admissible."""
|
||||
|
||||
|
||||
class ConnectivityEvidenceError(ConnectivityError):
|
||||
"""Raised when durable evidence cannot be safely written or read."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CallerCapability:
|
||||
"""Static public capability claimed by one benchmark caller adapter."""
|
||||
|
||||
caller: str
|
||||
route_kinds: tuple[str, ...]
|
||||
efforts: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EffectiveBinding:
|
||||
"""One observed stage binding, including its exact effective effort."""
|
||||
|
||||
stage: str
|
||||
model: str
|
||||
effort: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RequestedEffectiveBinding:
|
||||
"""Requested identity plus one all-or-none observed effective binding.
|
||||
|
||||
The requested fields are always required. The effective group is either
|
||||
fully absent, which only a blocked result may report, or complete and exact.
|
||||
"""
|
||||
|
||||
cell_id: str
|
||||
caller: str
|
||||
requested_route_kind: str
|
||||
requested_route_id: str
|
||||
requested_model: str
|
||||
requested_effort: str
|
||||
effective_route_kind: str | None = None
|
||||
effective_route_id: str | None = None
|
||||
effective_model: str | None = None
|
||||
effective_effort: str | None = None
|
||||
effective_bindings: tuple[EffectiveBinding, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConnectivityIssue:
|
||||
"""Closed blocker code paired with its one permitted resume code."""
|
||||
|
||||
code: str
|
||||
resume_code: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConnectivityResult:
|
||||
"""Validated immutable result for exactly one caller/cell preflight."""
|
||||
|
||||
capability: CallerCapability
|
||||
binding: RequestedEffectiveBinding
|
||||
issues: tuple[ConnectivityIssue, ...]
|
||||
status: str
|
||||
|
||||
|
||||
class CallerPreflight(Protocol):
|
||||
"""Adapter boundary; implementations must not return raw caller output."""
|
||||
|
||||
capability: CallerCapability
|
||||
|
||||
def preflight(
|
||||
self, cell: MatrixCell
|
||||
) -> tuple[RequestedEffectiveBinding, tuple[ConnectivityIssue, ...]]:
|
||||
"""Return typed, secret-free observation for ``cell``."""
|
||||
|
||||
|
||||
def _fail(message: str) -> None:
|
||||
# Messages contain only fixed contract field names, never input values.
|
||||
raise ConnectivityValidationError(message)
|
||||
|
||||
|
||||
def _require_identifier(value: Any, field_name: str) -> str:
|
||||
if not isinstance(value, str) or not TOKEN_RE.fullmatch(value):
|
||||
_fail(f"invalid {field_name}")
|
||||
return value
|
||||
|
||||
|
||||
def _require_identity(value: Any, field_name: str) -> str:
|
||||
if not isinstance(value, str) or not IDENTITY_RE.fullmatch(value):
|
||||
raise ConnectivityEvidenceError(f"invalid {field_name}")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_closed_tuple(
|
||||
values: Any,
|
||||
field_name: str,
|
||||
*,
|
||||
allowed: tuple[str, ...] | None = None,
|
||||
) -> None:
|
||||
"""Check shape, item type, membership and uniqueness before canonical order.
|
||||
|
||||
Ordering is checked last so unknown or unhashable entries can never reach
|
||||
``set`` or an enum index lookup and escape as a built-in exception.
|
||||
"""
|
||||
if not isinstance(values, tuple) or not values:
|
||||
_fail(f"invalid {field_name}")
|
||||
for item in values:
|
||||
if not isinstance(item, str):
|
||||
_fail(f"invalid {field_name} item")
|
||||
if allowed is None:
|
||||
_require_identifier(item, f"{field_name} item")
|
||||
elif item not in allowed:
|
||||
_fail(f"invalid {field_name} item")
|
||||
if len(set(values)) != len(values):
|
||||
_fail(f"duplicate {field_name} item")
|
||||
if tuple(sorted(values, key=allowed.index if allowed is not None else None)) != values:
|
||||
_fail(f"non-canonical {field_name}")
|
||||
|
||||
|
||||
def _validate_capability(capability: CallerCapability) -> None:
|
||||
if not isinstance(capability, CallerCapability):
|
||||
_fail("invalid caller capability")
|
||||
if capability.caller not in CALLER_ENUM:
|
||||
_fail("invalid capability caller")
|
||||
_validate_closed_tuple(
|
||||
capability.route_kinds, "capability route_kinds", allowed=ROUTE_KIND_ENUM
|
||||
)
|
||||
_validate_closed_tuple(capability.efforts, "capability efforts")
|
||||
|
||||
|
||||
def _validate_binding_shape(binding: RequestedEffectiveBinding) -> bool:
|
||||
"""Validate requested fields, then the all-or-none effective group.
|
||||
|
||||
Returns whether the caller reported an effective observation at all.
|
||||
"""
|
||||
if not isinstance(binding, RequestedEffectiveBinding):
|
||||
_fail("invalid requested/effective binding")
|
||||
_require_identifier(binding.cell_id, "binding cell_id")
|
||||
if binding.caller not in CALLER_ENUM:
|
||||
_fail("invalid binding caller")
|
||||
if binding.requested_route_kind not in ROUTE_KIND_ENUM:
|
||||
_fail("invalid requested route_kind")
|
||||
for name in ("requested_route_id", "requested_model", "requested_effort"):
|
||||
_require_identifier(getattr(binding, name), f"binding {name}")
|
||||
|
||||
if not isinstance(binding.effective_bindings, tuple):
|
||||
_fail("invalid effective_bindings")
|
||||
observed = tuple(getattr(binding, name) is not None for name in EFFECTIVE_FIELDS)
|
||||
if any(observed) != all(observed):
|
||||
_fail("partial effective observation")
|
||||
if not any(observed):
|
||||
if binding.effective_bindings:
|
||||
_fail("partial effective observation")
|
||||
return False
|
||||
|
||||
if binding.effective_route_kind not in ROUTE_KIND_ENUM:
|
||||
_fail("invalid effective route_kind")
|
||||
for name in ("effective_route_id", "effective_model", "effective_effort"):
|
||||
_require_identifier(getattr(binding, name), f"binding {name}")
|
||||
if not binding.effective_bindings:
|
||||
_fail("partial effective observation")
|
||||
stages: list[str] = []
|
||||
for item in binding.effective_bindings:
|
||||
if not isinstance(item, EffectiveBinding):
|
||||
_fail("invalid effective binding item")
|
||||
if item.stage not in STAGE_ENUM:
|
||||
_fail("invalid effective binding stage")
|
||||
_require_identifier(item.model, "effective binding model")
|
||||
if item.effort is not None:
|
||||
_require_identifier(item.effort, "effective binding effort")
|
||||
stages.append(item.stage)
|
||||
if len(set(stages)) != len(stages):
|
||||
_fail("duplicate effective binding stage")
|
||||
if stages != sorted(stages, key=STAGE_RANK.__getitem__):
|
||||
_fail("non-canonical effective binding order")
|
||||
return True
|
||||
|
||||
|
||||
def _expected_as_effective(expected: ExpectedBinding) -> EffectiveBinding:
|
||||
return EffectiveBinding(expected.stage, expected.model, expected.effort)
|
||||
|
||||
|
||||
def _cell_binding(cell: MatrixCell) -> tuple[str, str, str, str]:
|
||||
return (
|
||||
cell.iop.route_kind,
|
||||
cell.iop.route_id,
|
||||
cell.iop.request_model,
|
||||
cell.iop.requested_effort,
|
||||
)
|
||||
|
||||
|
||||
def validate_requested_binding(
|
||||
cell: MatrixCell,
|
||||
capability: CallerCapability,
|
||||
binding: RequestedEffectiveBinding,
|
||||
) -> None:
|
||||
"""Fail closed unless caller identity and requested route/model/effort are exact."""
|
||||
if not isinstance(cell, MatrixCell):
|
||||
_fail("invalid matrix cell")
|
||||
_validate_capability(capability)
|
||||
_validate_binding_shape(binding)
|
||||
|
||||
if capability.caller != cell.caller or binding.caller != cell.caller:
|
||||
_fail("caller identity mismatch")
|
||||
if binding.cell_id != cell.id:
|
||||
_fail("cell identity mismatch")
|
||||
if cell.iop.route_kind not in capability.route_kinds:
|
||||
_fail("unsupported route_kind")
|
||||
if cell.iop.requested_effort not in capability.efforts:
|
||||
_fail("unsupported effort")
|
||||
|
||||
requested = (
|
||||
binding.requested_route_kind,
|
||||
binding.requested_route_id,
|
||||
binding.requested_model,
|
||||
binding.requested_effort,
|
||||
)
|
||||
if requested != _cell_binding(cell):
|
||||
_fail("requested binding mismatch")
|
||||
|
||||
|
||||
def validate_effective_binding(
|
||||
cell: MatrixCell,
|
||||
binding: RequestedEffectiveBinding,
|
||||
*,
|
||||
required: bool,
|
||||
) -> None:
|
||||
"""Require the observation only for ``ready``; any present group must be exact."""
|
||||
if not isinstance(cell, MatrixCell):
|
||||
_fail("invalid matrix cell")
|
||||
if not _validate_binding_shape(binding):
|
||||
if required:
|
||||
_fail("missing effective observation")
|
||||
return
|
||||
|
||||
effective = (
|
||||
binding.effective_route_kind,
|
||||
binding.effective_route_id,
|
||||
binding.effective_model,
|
||||
binding.effective_effort,
|
||||
)
|
||||
if effective != _cell_binding(cell):
|
||||
_fail("effective binding substitution")
|
||||
expected_stages = tuple(_expected_as_effective(item) for item in cell.iop.expected_bindings)
|
||||
if binding.effective_bindings != expected_stages:
|
||||
_fail("effective stage binding mismatch")
|
||||
|
||||
|
||||
def validate_binding(
|
||||
cell: MatrixCell,
|
||||
capability: CallerCapability,
|
||||
binding: RequestedEffectiveBinding,
|
||||
*,
|
||||
require_effective: bool = True,
|
||||
) -> None:
|
||||
"""Validate the requested phase and then the effective phase of one binding."""
|
||||
validate_requested_binding(cell, capability, binding)
|
||||
validate_effective_binding(cell, binding, required=require_effective)
|
||||
|
||||
|
||||
def _validate_issue(issue: ConnectivityIssue) -> None:
|
||||
if not isinstance(issue, ConnectivityIssue):
|
||||
_fail("invalid issue")
|
||||
if not isinstance(issue.code, str) or issue.code not in ISSUE_CODES:
|
||||
_fail("invalid issue code")
|
||||
if not isinstance(issue.resume_code, str) or issue.resume_code != ISSUE_RESUME_CODES[issue.code]:
|
||||
_fail("invalid issue resume_code")
|
||||
|
||||
|
||||
def classify_issues(issues: tuple[ConnectivityIssue, ...]) -> str:
|
||||
"""Return the only permitted status, with implementation gaps taking precedence."""
|
||||
if not isinstance(issues, tuple):
|
||||
_fail("issues must be a tuple")
|
||||
codes: list[str] = []
|
||||
for issue in issues:
|
||||
_validate_issue(issue)
|
||||
codes.append(issue.code)
|
||||
if len(set(codes)) != len(codes):
|
||||
_fail("duplicate issue code")
|
||||
if codes != sorted(codes, key=ISSUE_RANK.__getitem__):
|
||||
_fail("non-canonical issue order")
|
||||
if any(code in IMPLEMENTATION_ISSUE_CODES for code in codes):
|
||||
return "implementation_gap"
|
||||
if any(code in REGISTRATION_ISSUE_CODES for code in codes):
|
||||
return "registration_required"
|
||||
return "ready"
|
||||
|
||||
|
||||
def make_result(
|
||||
cell: MatrixCell,
|
||||
capability: CallerCapability,
|
||||
binding: RequestedEffectiveBinding,
|
||||
issues: tuple[ConnectivityIssue, ...] = (),
|
||||
) -> ConnectivityResult:
|
||||
"""Construct a result only after phase-exact no-substitution validation."""
|
||||
status = classify_issues(issues)
|
||||
validate_requested_binding(cell, capability, binding)
|
||||
validate_effective_binding(cell, binding, required=status == "ready")
|
||||
return ConnectivityResult(capability, binding, issues, status)
|
||||
|
||||
|
||||
def validate_result(cell: MatrixCell, result: ConnectivityResult) -> None:
|
||||
"""Revalidate a received result before it is consumed or persisted."""
|
||||
if not isinstance(result, ConnectivityResult):
|
||||
_fail("invalid connectivity result")
|
||||
status = classify_issues(result.issues)
|
||||
if result.status not in RESULT_STATUSES or result.status != status:
|
||||
_fail("result status mismatch")
|
||||
validate_requested_binding(cell, result.capability, result.binding)
|
||||
validate_effective_binding(cell, result.binding, required=status == "ready")
|
||||
|
||||
|
||||
def _binding_dict(binding: RequestedEffectiveBinding) -> dict[str, Any]:
|
||||
return {
|
||||
"cell_id": binding.cell_id,
|
||||
"caller": binding.caller,
|
||||
"requested_route_kind": binding.requested_route_kind,
|
||||
"requested_route_id": binding.requested_route_id,
|
||||
"requested_model": binding.requested_model,
|
||||
"requested_effort": binding.requested_effort,
|
||||
"effective_route_kind": binding.effective_route_kind,
|
||||
"effective_route_id": binding.effective_route_id,
|
||||
"effective_model": binding.effective_model,
|
||||
"effective_effort": binding.effective_effort,
|
||||
"effective_bindings": [
|
||||
{"stage": item.stage, "model": item.model, "effort": item.effort}
|
||||
for item in binding.effective_bindings
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def canonical_evidence_bytes(
|
||||
cell: MatrixCell,
|
||||
result: ConnectivityResult,
|
||||
endpoint_identity: str,
|
||||
config_identity: str,
|
||||
) -> bytes:
|
||||
"""Return deterministic schema-closed evidence containing no raw endpoint data."""
|
||||
validate_result(cell, result)
|
||||
_require_identity(endpoint_identity, "endpoint_identity")
|
||||
_require_identity(config_identity, "config_identity")
|
||||
payload = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"cell": {"id": cell.id, "caller": cell.caller},
|
||||
"status": result.status,
|
||||
"binding": _binding_dict(result.binding),
|
||||
"issues": [
|
||||
{"code": issue.code, "resume_code": issue.resume_code}
|
||||
for issue in result.issues
|
||||
],
|
||||
"endpoint_identity": endpoint_identity,
|
||||
"config_identity": config_identity,
|
||||
}
|
||||
return json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("ascii") + b"\n"
|
||||
|
||||
|
||||
def _safe_relative_path(relative_path: str | Path) -> Path:
|
||||
if not isinstance(relative_path, (str, Path)):
|
||||
raise ConnectivityEvidenceError("invalid evidence relative path")
|
||||
path = Path(relative_path)
|
||||
if (
|
||||
path.is_absolute()
|
||||
or not path.parts
|
||||
or any(part in ("", ".", "..") for part in path.parts)
|
||||
or path.suffix != ".json"
|
||||
):
|
||||
raise ConnectivityEvidenceError("invalid evidence relative path")
|
||||
return path
|
||||
|
||||
|
||||
def _require_nofollow_support() -> None:
|
||||
if (
|
||||
not hasattr(os, "O_NOFOLLOW")
|
||||
or not hasattr(os, "O_DIRECTORY")
|
||||
or os.open not in os.supports_dir_fd
|
||||
or os.mkdir not in os.supports_dir_fd
|
||||
):
|
||||
raise ConnectivityEvidenceError("evidence no-follow traversal unsupported")
|
||||
|
||||
|
||||
def _open_directory(name: str, dir_fd: int | None, message: str) -> int:
|
||||
flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
|
||||
try:
|
||||
return os.open(name, flags, dir_fd=dir_fd)
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError(message) from exc
|
||||
|
||||
|
||||
def _open_root(root: str | Path) -> int:
|
||||
"""Open the evidence root by descending one no-follow component at a time."""
|
||||
if not isinstance(root, (str, Path)):
|
||||
raise ConnectivityEvidenceError("invalid evidence root")
|
||||
path = Path(root)
|
||||
parts = list(path.parts)
|
||||
if path.is_absolute():
|
||||
descriptor = _open_directory(parts[0], None, "invalid evidence root")
|
||||
parts = parts[1:]
|
||||
else:
|
||||
descriptor = _open_directory(".", None, "invalid evidence root")
|
||||
try:
|
||||
for part in parts:
|
||||
if part in ("", ".", ".."):
|
||||
raise ConnectivityEvidenceError("invalid evidence root")
|
||||
child = _open_directory(part, descriptor, "invalid evidence root")
|
||||
os.close(descriptor)
|
||||
descriptor = child
|
||||
except BaseException:
|
||||
os.close(descriptor)
|
||||
raise
|
||||
return descriptor
|
||||
|
||||
|
||||
def _open_evidence_parent(
|
||||
root: str | Path, relative_path: str | Path, *, create: bool
|
||||
) -> tuple[int, str]:
|
||||
"""Return a descriptor for the verified parent directory and the final name."""
|
||||
relative = _safe_relative_path(relative_path)
|
||||
_require_nofollow_support()
|
||||
descriptor = _open_root(root)
|
||||
try:
|
||||
for part in relative.parts[:-1]:
|
||||
if create:
|
||||
try:
|
||||
os.mkdir(part, 0o700, dir_fd=descriptor)
|
||||
except FileExistsError:
|
||||
pass
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError("unsafe evidence parent") from exc
|
||||
child = _open_directory(part, descriptor, "unsafe evidence parent")
|
||||
os.close(descriptor)
|
||||
descriptor = child
|
||||
except BaseException:
|
||||
os.close(descriptor)
|
||||
raise
|
||||
return descriptor, relative.parts[-1]
|
||||
|
||||
|
||||
def write_evidence(
|
||||
root: str | Path,
|
||||
relative_path: str | Path,
|
||||
cell: MatrixCell,
|
||||
result: ConnectivityResult,
|
||||
endpoint_identity: str,
|
||||
config_identity: str,
|
||||
) -> None:
|
||||
"""Atomically create evidence once; any existing or symlinked target is rejected."""
|
||||
payload = canonical_evidence_bytes(cell, result, endpoint_identity, config_identity)
|
||||
parent, name = _open_evidence_parent(root, relative_path, create=True)
|
||||
try:
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW
|
||||
try:
|
||||
descriptor = os.open(name, flags, 0o600, dir_fd=parent)
|
||||
except FileExistsError as exc:
|
||||
raise ConnectivityEvidenceError("evidence target already exists") from exc
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError("evidence write rejected") from exc
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb") as stream:
|
||||
stream.write(payload)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError("evidence write rejected") from exc
|
||||
finally:
|
||||
os.close(parent)
|
||||
|
||||
|
||||
def _read_bounded(descriptor: int) -> bytes:
|
||||
"""Read at most one byte past the cap so oversized input fails closed."""
|
||||
chunks: list[bytes] = []
|
||||
remaining = MAX_EVIDENCE_BYTES + 1
|
||||
try:
|
||||
while remaining > 0:
|
||||
chunk = os.read(descriptor, remaining)
|
||||
if not chunk:
|
||||
break
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError("invalid evidence bytes") from exc
|
||||
raw = b"".join(chunks)
|
||||
if len(raw) > MAX_EVIDENCE_BYTES:
|
||||
raise ConnectivityEvidenceError("evidence target too large")
|
||||
return raw
|
||||
|
||||
|
||||
def _read_evidence_bytes(root: str | Path, relative_path: str | Path) -> bytes:
|
||||
parent, name = _open_evidence_parent(root, relative_path, create=False)
|
||||
try:
|
||||
try:
|
||||
descriptor = os.open(
|
||||
name, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK, dir_fd=parent
|
||||
)
|
||||
except OSError as exc:
|
||||
raise ConnectivityEvidenceError("unsafe evidence target") from exc
|
||||
try:
|
||||
info = os.fstat(descriptor)
|
||||
if not stat.S_ISREG(info.st_mode):
|
||||
raise ConnectivityEvidenceError("unsafe evidence target")
|
||||
if info.st_size > MAX_EVIDENCE_BYTES:
|
||||
raise ConnectivityEvidenceError("evidence target too large")
|
||||
return _read_bounded(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
finally:
|
||||
os.close(parent)
|
||||
|
||||
|
||||
def _no_duplicate_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ConnectivityEvidenceError("duplicate evidence field")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _read_binding(data: Any) -> RequestedEffectiveBinding:
|
||||
if not isinstance(data, dict) or set(data) != {
|
||||
"cell_id", "caller", "requested_route_kind", "requested_route_id",
|
||||
"requested_model", "requested_effort", "effective_route_kind",
|
||||
"effective_route_id", "effective_model", "effective_effort", "effective_bindings",
|
||||
}:
|
||||
raise ConnectivityEvidenceError("invalid evidence binding")
|
||||
stages = data["effective_bindings"]
|
||||
if not isinstance(stages, list):
|
||||
raise ConnectivityEvidenceError("invalid evidence binding")
|
||||
bindings: list[EffectiveBinding] = []
|
||||
for item in stages:
|
||||
if not isinstance(item, dict) or set(item) != {"stage", "model", "effort"}:
|
||||
raise ConnectivityEvidenceError("invalid evidence binding")
|
||||
bindings.append(EffectiveBinding(item["stage"], item["model"], item["effort"]))
|
||||
try:
|
||||
binding = RequestedEffectiveBinding(
|
||||
data["cell_id"], data["caller"], data["requested_route_kind"],
|
||||
data["requested_route_id"], data["requested_model"], data["requested_effort"],
|
||||
data["effective_route_kind"], data["effective_route_id"],
|
||||
data["effective_model"], data["effective_effort"], tuple(bindings),
|
||||
)
|
||||
_validate_binding_shape(binding)
|
||||
except ConnectivityValidationError as exc:
|
||||
raise ConnectivityEvidenceError("invalid evidence binding") from exc
|
||||
return binding
|
||||
|
||||
|
||||
def _validate_evidence_payload(payload: dict[str, Any], cell: MatrixCell) -> None:
|
||||
if not isinstance(cell, MatrixCell):
|
||||
raise ConnectivityEvidenceError("invalid matrix cell")
|
||||
if not isinstance(payload.get("cell"), dict) or set(payload["cell"]) != {"id", "caller"}:
|
||||
raise ConnectivityEvidenceError("invalid evidence cell")
|
||||
binding = _read_binding(payload["binding"])
|
||||
if payload["cell"] != {"id": binding.cell_id, "caller": binding.caller}:
|
||||
raise ConnectivityEvidenceError("evidence cell mismatch")
|
||||
if binding.cell_id != cell.id or binding.caller != cell.caller:
|
||||
raise ConnectivityEvidenceError("evidence cell mismatch")
|
||||
requested = (
|
||||
binding.requested_route_kind,
|
||||
binding.requested_route_id,
|
||||
binding.requested_model,
|
||||
binding.requested_effort,
|
||||
)
|
||||
if requested != _cell_binding(cell):
|
||||
raise ConnectivityEvidenceError("requested binding mismatch")
|
||||
try:
|
||||
validate_effective_binding(cell, binding, required=payload["status"] == "ready")
|
||||
except ConnectivityValidationError as exc:
|
||||
raise ConnectivityEvidenceError("invalid evidence binding") from exc
|
||||
issues_raw = payload["issues"]
|
||||
if not isinstance(issues_raw, list):
|
||||
raise ConnectivityEvidenceError("invalid evidence issues")
|
||||
try:
|
||||
issues = tuple(
|
||||
ConnectivityIssue(item["code"], item["resume_code"])
|
||||
for item in issues_raw
|
||||
if isinstance(item, dict) and set(item) == {"code", "resume_code"}
|
||||
)
|
||||
if len(issues) != len(issues_raw) or classify_issues(issues) != payload["status"]:
|
||||
raise ConnectivityValidationError("result status mismatch")
|
||||
except (ConnectivityValidationError, KeyError, TypeError) as exc:
|
||||
raise ConnectivityEvidenceError("invalid evidence issues") from exc
|
||||
|
||||
|
||||
def read_evidence(
|
||||
root: str | Path, relative_path: str | Path, cell: MatrixCell
|
||||
) -> dict[str, Any]:
|
||||
"""Read only canonical, schema-closed evidence; corruption fails closed."""
|
||||
raw = _read_evidence_bytes(root, relative_path)
|
||||
try:
|
||||
parsed = json.loads(raw.decode("ascii"), object_pairs_hook=_no_duplicate_object)
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise ConnectivityEvidenceError("invalid evidence bytes") from exc
|
||||
if not isinstance(parsed, dict) or set(parsed) != {
|
||||
"schema_version", "cell", "status", "binding", "issues",
|
||||
"endpoint_identity", "config_identity",
|
||||
}:
|
||||
raise ConnectivityEvidenceError("invalid evidence schema")
|
||||
if parsed.get("schema_version") != SCHEMA_VERSION or parsed.get("status") not in RESULT_STATUSES:
|
||||
raise ConnectivityEvidenceError("invalid evidence schema")
|
||||
_require_identity(parsed.get("endpoint_identity"), "endpoint_identity")
|
||||
_require_identity(parsed.get("config_identity"), "config_identity")
|
||||
_validate_evidence_payload(parsed, cell)
|
||||
canonical = json.dumps(parsed, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("ascii") + b"\n"
|
||||
if raw != canonical:
|
||||
raise ConnectivityEvidenceError("non-canonical evidence")
|
||||
return parsed
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,600 +0,0 @@
|
|||
"""Deterministic tests for the closed benchmark connectivity contract."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import dataclasses
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from dataclasses import FrozenInstanceError
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.agent_benchmark.connectivity import (
|
||||
ISSUE_RESUME_CODES,
|
||||
MAX_EVIDENCE_BYTES,
|
||||
CallerCapability,
|
||||
ConnectivityEvidenceError,
|
||||
ConnectivityIssue,
|
||||
ConnectivityResult,
|
||||
ConnectivityValidationError,
|
||||
EffectiveBinding,
|
||||
RequestedEffectiveBinding,
|
||||
canonical_evidence_bytes,
|
||||
classify_issues,
|
||||
make_result,
|
||||
read_evidence,
|
||||
validate_binding,
|
||||
validate_result,
|
||||
write_evidence,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import ExpectedBinding, IopCell, MatrixCell
|
||||
|
||||
EFFECTIVE_SCALARS = (
|
||||
"effective_route_kind",
|
||||
"effective_route_id",
|
||||
"effective_model",
|
||||
"effective_effort",
|
||||
)
|
||||
|
||||
|
||||
def _identity(label: str) -> str:
|
||||
return "sha256:" + hashlib.sha256(label.encode("ascii")).hexdigest()
|
||||
|
||||
|
||||
def _cell(route_kind: str = "direct") -> MatrixCell:
|
||||
if route_kind == "direct":
|
||||
bindings = (ExpectedBinding("request", "gpt", "xhigh"),)
|
||||
else:
|
||||
bindings = (
|
||||
ExpectedBinding("selector", "gpt", "xhigh"),
|
||||
ExpectedBinding("plan", "gpt", "high"),
|
||||
ExpectedBinding("work", "gpt", None),
|
||||
ExpectedBinding("review", "gpt", "high"),
|
||||
)
|
||||
return MatrixCell("cell", "codex", IopCell("gpt", "xhigh", route_kind, "route", bindings))
|
||||
|
||||
|
||||
def _capability() -> CallerCapability:
|
||||
return CallerCapability("codex", ("direct", "execution_preset"), ("high", "xhigh"))
|
||||
|
||||
|
||||
def _requested_only(cell: MatrixCell) -> RequestedEffectiveBinding:
|
||||
"""Blocked observation: requested identity only, no effective group at all."""
|
||||
return RequestedEffectiveBinding(
|
||||
cell.id, cell.caller, cell.iop.route_kind, cell.iop.route_id,
|
||||
cell.iop.request_model, cell.iop.requested_effort,
|
||||
)
|
||||
|
||||
|
||||
def _binding(cell: MatrixCell) -> RequestedEffectiveBinding:
|
||||
return RequestedEffectiveBinding(
|
||||
cell.id, cell.caller, cell.iop.route_kind, cell.iop.route_id,
|
||||
cell.iop.request_model, cell.iop.requested_effort,
|
||||
cell.iop.route_kind, cell.iop.route_id,
|
||||
cell.iop.request_model, cell.iop.requested_effort,
|
||||
tuple(EffectiveBinding(item.stage, item.model, item.effort) for item in cell.iop.expected_bindings),
|
||||
)
|
||||
|
||||
|
||||
def _issue(code: str) -> ConnectivityIssue:
|
||||
return ConnectivityIssue(code, ISSUE_RESUME_CODES[code])
|
||||
|
||||
|
||||
class ConnectivityContractTest(unittest.TestCase):
|
||||
def test_direct_and_preset_exact_contracts_are_frozen(self):
|
||||
for route_kind in ("direct", "execution_preset"):
|
||||
cell = _cell(route_kind)
|
||||
result = make_result(cell, _capability(), _binding(cell))
|
||||
self.assertEqual(result.status, "ready")
|
||||
self.assertEqual(result.binding.effective_bindings, tuple(
|
||||
EffectiveBinding(item.stage, item.model, item.effort)
|
||||
for item in cell.iop.expected_bindings
|
||||
))
|
||||
with self.assertRaises(FrozenInstanceError):
|
||||
result.status = "implementation_gap" # type: ignore[misc]
|
||||
|
||||
def test_every_requested_or_effective_substitution_fails_closed(self):
|
||||
cell = _cell()
|
||||
binding = _binding(cell)
|
||||
substitutions = (
|
||||
{"cell_id": "other"}, {"caller": "agy"},
|
||||
{"requested_route_kind": "execution_preset"}, {"requested_route_id": "other"},
|
||||
{"requested_model": "alias"}, {"requested_effort": "high"},
|
||||
{"effective_route_kind": "execution_preset"}, {"effective_route_id": "other"},
|
||||
{"effective_model": "alias"}, {"effective_effort": "high"},
|
||||
)
|
||||
for replacement in substitutions:
|
||||
with self.subTest(replacement=replacement):
|
||||
mutated = RequestedEffectiveBinding(**{**binding.__dict__, **replacement})
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
validate_binding(cell, _capability(), mutated)
|
||||
|
||||
def test_missing_extra_and_reordered_stage_bindings_fail_closed(self):
|
||||
cell = _cell("execution_preset")
|
||||
binding = _binding(cell)
|
||||
cases = (
|
||||
binding.effective_bindings[:-1],
|
||||
binding.effective_bindings + (EffectiveBinding("repair", "gpt", "high"),),
|
||||
tuple(reversed(binding.effective_bindings)),
|
||||
binding.effective_bindings[:-1] + (EffectiveBinding("review", "alias", "high"),),
|
||||
)
|
||||
for stages in cases:
|
||||
with self.subTest(stages=stages):
|
||||
observed = RequestedEffectiveBinding(**{**binding.__dict__, "effective_bindings": stages})
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
validate_binding(cell, _capability(), observed)
|
||||
|
||||
def test_capability_is_closed_and_requires_requested_effort(self):
|
||||
cell = _cell()
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
validate_binding(cell, CallerCapability("codex", ("direct",), ("high",)), _binding(cell))
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
validate_binding(cell, CallerCapability("codex", ("execution_preset", "direct"), ("high", "xhigh")), _binding(cell))
|
||||
|
||||
def test_malformed_capability_entries_raise_closed_error(self):
|
||||
cell = _cell()
|
||||
binding = _binding(cell)
|
||||
capabilities = (
|
||||
CallerCapability("codex", ("unknown", "direct"), ("high", "xhigh")),
|
||||
CallerCapability("codex", ("direct", ["execution_preset"]), ("high", "xhigh")),
|
||||
CallerCapability("codex", ("direct", "direct"), ("high", "xhigh")),
|
||||
CallerCapability("codex", (), ("high", "xhigh")),
|
||||
CallerCapability("codex", ["direct"], ("high", "xhigh")),
|
||||
CallerCapability("codex", ("direct",), ("high", 3)),
|
||||
CallerCapability("codex", ("direct",), (["xhigh"],)),
|
||||
CallerCapability("codex", ("direct",), ("xhigh", "high")),
|
||||
CallerCapability("codex", ("direct",), ("Xhigh",)),
|
||||
CallerCapability("unknown", ("direct",), ("xhigh",)),
|
||||
"codex",
|
||||
)
|
||||
for capability in capabilities:
|
||||
with self.subTest(capability=repr(capability)):
|
||||
with self.assertRaises(ConnectivityValidationError) as caught:
|
||||
validate_binding(cell, capability, binding)
|
||||
self.assertNotIsInstance(caught.exception, (ValueError, TypeError, KeyError))
|
||||
|
||||
def test_classifier_is_closed_and_implementation_gap_has_precedence(self):
|
||||
registration = (_issue("credential_missing"),)
|
||||
implementation = (_issue("stream_incompatible"),)
|
||||
self.assertEqual(classify_issues(()), "ready")
|
||||
self.assertEqual(classify_issues(registration), "registration_required")
|
||||
self.assertEqual(classify_issues(implementation), "implementation_gap")
|
||||
self.assertEqual(classify_issues(registration + implementation), "implementation_gap")
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues((ConnectivityIssue("unknown", "register_credential"),))
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues([_issue("credential_missing")])
|
||||
|
||||
def test_issue_resume_pairs_are_closed_and_canonically_ordered(self):
|
||||
for code, resume_code in ISSUE_RESUME_CODES.items():
|
||||
with self.subTest(code=code):
|
||||
self.assertIn(
|
||||
classify_issues((ConnectivityIssue(code, resume_code),)),
|
||||
("registration_required", "implementation_gap"),
|
||||
)
|
||||
for other_code, other_resume in ISSUE_RESUME_CODES.items():
|
||||
if other_code == code:
|
||||
continue
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues((ConnectivityIssue(code, other_resume),))
|
||||
for text in ("register credential", "sk-live-0000", "", "register_credential ", None):
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues((ConnectivityIssue(code, text),))
|
||||
canonical = (_issue("model_missing"), _issue("stream_incompatible"))
|
||||
self.assertEqual(classify_issues(canonical), "implementation_gap")
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues(tuple(reversed(canonical)))
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
classify_issues(canonical + canonical[:1])
|
||||
|
||||
def test_malformed_issue_entries_raise_closed_error(self):
|
||||
cell = _cell()
|
||||
capability = _capability()
|
||||
binding = _binding(cell)
|
||||
valid_resume = ISSUE_RESUME_CODES["credential_missing"]
|
||||
other_resume = ISSUE_RESUME_CODES["model_missing"]
|
||||
bad_code_entries = (
|
||||
ConnectivityIssue(["credential_missing"], valid_resume),
|
||||
ConnectivityIssue({"code": "credential_missing"}, valid_resume),
|
||||
ConnectivityIssue(42, valid_resume),
|
||||
ConnectivityIssue(None, valid_resume),
|
||||
ConnectivityIssue("unknown", valid_resume),
|
||||
)
|
||||
bad_resume_entries = (
|
||||
ConnectivityIssue("credential_missing", ["register_credential"]),
|
||||
ConnectivityIssue("credential_missing", {"resume_code": "register_credential"}),
|
||||
ConnectivityIssue("credential_missing", 42),
|
||||
ConnectivityIssue("credential_missing", None),
|
||||
ConnectivityIssue("credential_missing", "register credential"),
|
||||
ConnectivityIssue("credential_missing", "sk-live-0000"),
|
||||
ConnectivityIssue("credential_missing", ""),
|
||||
ConnectivityIssue("credential_missing", other_resume),
|
||||
)
|
||||
non_issue_objects = (
|
||||
{"code": "credential_missing", "resume_code": valid_resume},
|
||||
"credential_missing",
|
||||
42,
|
||||
None,
|
||||
("credential_missing", valid_resume),
|
||||
)
|
||||
expected_message = {
|
||||
"code": "invalid issue code",
|
||||
"resume": "invalid issue resume_code",
|
||||
"object": "invalid issue",
|
||||
}
|
||||
forbidden_raw_values = (
|
||||
"sk-live-0000", "register credential", "credential_missing",
|
||||
"register_credential", "register_model", "42", "None",
|
||||
)
|
||||
labelled = (
|
||||
*(("code", item) for item in bad_code_entries),
|
||||
*(("resume", item) for item in bad_resume_entries),
|
||||
*(("object", item) for item in non_issue_objects),
|
||||
)
|
||||
for label, entry in labelled:
|
||||
with self.subTest(label=label, entry=repr(entry)):
|
||||
issues = (entry,)
|
||||
with self.assertRaises(ConnectivityValidationError) as classifier_caught:
|
||||
classify_issues(issues)
|
||||
self.assertNotIsInstance(
|
||||
classifier_caught.exception,
|
||||
(TypeError, KeyError, ValueError, AttributeError),
|
||||
)
|
||||
self.assertEqual(str(classifier_caught.exception), expected_message[label])
|
||||
for token in forbidden_raw_values:
|
||||
self.assertNotIn(token, str(classifier_caught.exception))
|
||||
self.assertNotIn(repr(entry), str(classifier_caught.exception))
|
||||
with self.assertRaises(ConnectivityValidationError) as result_caught:
|
||||
make_result(cell, capability, binding, issues)
|
||||
self.assertNotIsInstance(
|
||||
result_caught.exception,
|
||||
(TypeError, KeyError, ValueError, AttributeError),
|
||||
)
|
||||
self.assertEqual(str(result_caught.exception), expected_message[label])
|
||||
for token in forbidden_raw_values:
|
||||
self.assertNotIn(token, str(result_caught.exception))
|
||||
self.assertNotIn(repr(entry), str(result_caught.exception))
|
||||
|
||||
def test_ready_requires_complete_exact_effective_observation(self):
|
||||
cell = _cell("execution_preset")
|
||||
binding = _binding(cell)
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
make_result(cell, _capability(), _requested_only(cell))
|
||||
partials = [{name: None} for name in EFFECTIVE_SCALARS]
|
||||
partials.append({"effective_bindings": ()})
|
||||
partials.append({name: None for name in EFFECTIVE_SCALARS})
|
||||
partials.append({"effective_model": "alias"})
|
||||
partials.append({"effective_bindings": binding.effective_bindings[:-1]})
|
||||
for replacement in partials:
|
||||
with self.subTest(replacement=replacement):
|
||||
mutated = RequestedEffectiveBinding(**{**binding.__dict__, **replacement})
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
make_result(cell, _capability(), mutated)
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
make_result(cell, _capability(), mutated, (_issue("model_missing"),))
|
||||
|
||||
def test_blocked_results_stay_blocked_and_cannot_be_forged_ready(self):
|
||||
cell = _cell()
|
||||
issues = (_issue("credential_missing"),)
|
||||
blocked = make_result(cell, _capability(), _requested_only(cell), issues)
|
||||
self.assertEqual(blocked.status, "registration_required")
|
||||
self.assertIsNone(blocked.binding.effective_model)
|
||||
self.assertEqual(blocked.binding.effective_bindings, ())
|
||||
validate_result(cell, blocked)
|
||||
forgeries = (
|
||||
ConnectivityResult(_capability(), _requested_only(cell), issues, "ready"),
|
||||
ConnectivityResult(_capability(), _binding(cell), issues, "ready"),
|
||||
ConnectivityResult(_capability(), _binding(cell), (), "blocked"),
|
||||
ConnectivityResult(_capability(), _requested_only(cell), (), "ready"),
|
||||
)
|
||||
for forged in forgeries:
|
||||
with self.subTest(status=forged.status, issues=len(forged.issues)):
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
validate_result(cell, forged)
|
||||
|
||||
def test_no_contract_field_accepts_opaque_caller_text(self):
|
||||
self.assertEqual(
|
||||
{field.name for field in dataclasses.fields(ConnectivityIssue)},
|
||||
{"code", "resume_code"},
|
||||
)
|
||||
cell = _cell()
|
||||
binding = _binding(cell)
|
||||
opaque = "sk-live-0000000000000000"
|
||||
secret = "Authorization: Bearer should-not-appear"
|
||||
for text in (secret, opaque, "https://private.invalid"):
|
||||
with self.subTest(text=text):
|
||||
with self.assertRaises(ConnectivityValidationError) as caught:
|
||||
make_result(cell, _capability(), binding, (ConnectivityIssue("credential_missing", text),))
|
||||
self.assertNotIn(text, str(caught.exception))
|
||||
for replacement in (
|
||||
{"requested_model": opaque}, {"effective_model": opaque},
|
||||
{"effective_route_id": "private-endpoint"},
|
||||
):
|
||||
with self.subTest(replacement=replacement):
|
||||
mutated = RequestedEffectiveBinding(**{**binding.__dict__, **replacement})
|
||||
with self.assertRaises(ConnectivityValidationError) as caught:
|
||||
make_result(cell, _capability(), mutated)
|
||||
self.assertNotIn(opaque, str(caught.exception))
|
||||
self.assertNotIn("private-endpoint", str(caught.exception))
|
||||
|
||||
|
||||
class ConnectivityEvidenceTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(dir="/tmp", prefix="connectivity-")
|
||||
self.base = Path(os.path.realpath(self.temp.name))
|
||||
self.root = self.base / "evidence"
|
||||
self.root.mkdir()
|
||||
self.cell = _cell()
|
||||
self.result = make_result(self.cell, _capability(), _binding(self.cell))
|
||||
self.endpoint_identity = _identity("endpoint")
|
||||
self.config_identity = _identity("config")
|
||||
|
||||
def tearDown(self):
|
||||
self.temp.cleanup()
|
||||
|
||||
def _write(self, root, relative_path, cell=None, result=None):
|
||||
write_evidence(
|
||||
root, relative_path, cell or self.cell, result or self.result,
|
||||
self.endpoint_identity, self.config_identity,
|
||||
)
|
||||
|
||||
def test_canonical_evidence_is_deterministic_and_secret_safe(self):
|
||||
one = canonical_evidence_bytes(self.cell, self.result, self.endpoint_identity, self.config_identity)
|
||||
two = canonical_evidence_bytes(self.cell, self.result, self.endpoint_identity, self.config_identity)
|
||||
self.assertEqual(one, two)
|
||||
self.assertIn(self.endpoint_identity.encode(), one)
|
||||
for forbidden in (b"Authorization", b"Bearer", b"https://", b"prompt", b"tool"):
|
||||
self.assertNotIn(forbidden, one)
|
||||
|
||||
def test_blocked_results_omit_effective_observations_and_round_trip(self):
|
||||
cases = (
|
||||
("direct", _issue("model_missing"), "registration_required"),
|
||||
("execution_preset", _issue("endpoint_incompatible"), "implementation_gap"),
|
||||
)
|
||||
for index, (route_kind, issue, status) in enumerate(cases):
|
||||
with self.subTest(route_kind=route_kind):
|
||||
cell = _cell(route_kind)
|
||||
result = make_result(cell, _capability(), _requested_only(cell), (issue,))
|
||||
self.assertEqual(result.status, status)
|
||||
raw = canonical_evidence_bytes(cell, result, self.endpoint_identity, self.config_identity)
|
||||
for absent in EFFECTIVE_SCALARS:
|
||||
self.assertIn(f'"{absent}":null'.encode("ascii"), raw)
|
||||
self.assertIn(b'"effective_bindings":[]', raw)
|
||||
name = f"blocked-{index}.json"
|
||||
self._write(self.root, name, cell, result)
|
||||
parsed = read_evidence(self.root, name, cell)
|
||||
self.assertEqual(parsed["status"], status)
|
||||
self.assertIsNone(parsed["binding"]["effective_model"])
|
||||
self.assertEqual(parsed["binding"]["effective_bindings"], [])
|
||||
self.assertEqual(
|
||||
parsed["issues"], [{"code": issue.code, "resume_code": issue.resume_code}]
|
||||
)
|
||||
forged = (
|
||||
(self.root / name).read_text(encoding="ascii")
|
||||
.replace(f'"status":"{status}"', '"status":"ready"')
|
||||
.replace(f'{{"code":"{issue.code}","resume_code":"{issue.resume_code}"}}', "")
|
||||
)
|
||||
(self.root / f"forged-{index}.json").write_text(forged, encoding="ascii")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, f"forged-{index}.json", cell)
|
||||
|
||||
def test_write_is_no_overwrite_and_read_rejects_corruption(self):
|
||||
self._write(self.root, "nested/preflight.json")
|
||||
raw = (self.root / "nested/preflight.json").read_bytes()
|
||||
self.assertEqual(read_evidence(self.root, "nested/preflight.json", self.cell)["status"], "ready")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, "nested/preflight.json")
|
||||
(self.root / "nested/preflight.json").write_bytes(raw + b" ")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "nested/preflight.json", self.cell)
|
||||
|
||||
def test_reader_rejects_canonical_schema_drift(self):
|
||||
self._write(self.root, "preflight.json")
|
||||
raw = (self.root / "preflight.json").read_text(encoding="ascii")
|
||||
(self.root / "preflight.json").write_text(raw.replace('"status":"ready"', '"status":"registration_required"'), encoding="ascii")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "preflight.json", self.cell)
|
||||
|
||||
def test_reader_rejects_canonical_binding_semantic_substitution(self):
|
||||
def _reseat(parsed: dict) -> bytes:
|
||||
return (
|
||||
json.dumps(
|
||||
parsed, sort_keys=True, separators=(",", ":"), ensure_ascii=True
|
||||
).encode("ascii")
|
||||
+ b"\n"
|
||||
)
|
||||
|
||||
def _write_mutant(cell, result, name, mutate):
|
||||
self._write(self.root, f"{name}.json", cell, result)
|
||||
baseline = read_evidence(self.root, f"{name}.json", cell)
|
||||
self.assertEqual(baseline["status"], result.status)
|
||||
original = (self.root / f"{name}.json").read_text(encoding="ascii")
|
||||
parsed = json.loads(original)
|
||||
mutant = copy.deepcopy(parsed)
|
||||
mutate(mutant)
|
||||
mutant_name = f"mutant-{name}.json"
|
||||
(self.root / mutant_name).write_bytes(_reseat(mutant))
|
||||
return mutant_name
|
||||
|
||||
cases: list[tuple] = []
|
||||
|
||||
direct_cell = _cell("direct")
|
||||
direct_result = make_result(direct_cell, _capability(), _binding(direct_cell))
|
||||
direct_scalars = (
|
||||
("requested_route_kind", "execution_preset"),
|
||||
("requested_route_id", "route2"),
|
||||
("requested_model", "gpth"),
|
||||
("requested_effort", "high"),
|
||||
("effective_route_kind", "execution_preset"),
|
||||
("effective_route_id", "route2"),
|
||||
("effective_model", "gpth"),
|
||||
("effective_effort", "high"),
|
||||
)
|
||||
for field, value in direct_scalars:
|
||||
cases.append((
|
||||
"direct-ready-scalar", direct_cell, direct_result, f"direct-{field}",
|
||||
lambda p, f=field, v=value: p["binding"].__setitem__(f, v),
|
||||
))
|
||||
cases.append((
|
||||
"direct-ready-stage", direct_cell, direct_result, "direct-stage",
|
||||
lambda p: p["binding"]["effective_bindings"].__setitem__(
|
||||
0, {"stage": "request", "model": "gpth", "effort": "xhigh"}
|
||||
),
|
||||
))
|
||||
|
||||
preset_cell = _cell("execution_preset")
|
||||
preset_result = make_result(preset_cell, _capability(), _binding(preset_cell))
|
||||
preset_scalars = (
|
||||
("requested_route_kind", "direct"),
|
||||
("requested_route_id", "route2"),
|
||||
("requested_model", "gpth"),
|
||||
("requested_effort", "high"),
|
||||
("effective_route_kind", "direct"),
|
||||
("effective_route_id", "route2"),
|
||||
("effective_model", "gpth"),
|
||||
("effective_effort", "high"),
|
||||
)
|
||||
for field, value in preset_scalars:
|
||||
cases.append((
|
||||
"preset-ready-scalar", preset_cell, preset_result, f"preset-{field}",
|
||||
lambda p, f=field, v=value: p["binding"].__setitem__(f, v),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-model", preset_cell, preset_result, "preset-stage-model",
|
||||
lambda p: p["binding"]["effective_bindings"].__setitem__(
|
||||
0, {"stage": "selector", "model": "gpth", "effort": "xhigh"}
|
||||
),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-effort", preset_cell, preset_result, "preset-stage-effort",
|
||||
lambda p: p["binding"]["effective_bindings"].__setitem__(
|
||||
1, {"stage": "plan", "model": "gpt", "effort": "xhigh"}
|
||||
),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-set", preset_cell, preset_result, "preset-stage-set",
|
||||
lambda p: p["binding"]["effective_bindings"].__setitem__(
|
||||
0, {"stage": "request", "model": "gpt", "effort": "xhigh"}
|
||||
),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-order", preset_cell, preset_result, "preset-stage-order",
|
||||
lambda p: p["binding"]["effective_bindings"].reverse(),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-extra", preset_cell, preset_result, "preset-stage-extra",
|
||||
lambda p: p["binding"]["effective_bindings"].append(
|
||||
{"stage": "repair", "model": "gpt", "effort": "high"}
|
||||
),
|
||||
))
|
||||
cases.append((
|
||||
"preset-ready-stage-missing", preset_cell, preset_result, "preset-stage-missing",
|
||||
lambda p: p["binding"]["effective_bindings"].pop(),
|
||||
))
|
||||
|
||||
blocked_cell = _cell("execution_preset")
|
||||
blocked_result = make_result(
|
||||
blocked_cell, _capability(), _binding(blocked_cell), (_issue("credential_missing"),)
|
||||
)
|
||||
self.assertEqual(blocked_result.status, "registration_required")
|
||||
cases.append((
|
||||
"blocked-observed-stage-model", blocked_cell, blocked_result, "blocked-stage-model",
|
||||
lambda p: p["binding"]["effective_bindings"].__setitem__(
|
||||
0, {"stage": "selector", "model": "gpth", "effort": "xhigh"}
|
||||
),
|
||||
))
|
||||
cases.append((
|
||||
"blocked-observed-stage-order", blocked_cell, blocked_result, "blocked-stage-order",
|
||||
lambda p: p["binding"]["effective_bindings"].reverse(),
|
||||
))
|
||||
|
||||
for label, cell, result, name, mutate in cases:
|
||||
with self.subTest(label=label, name=name):
|
||||
mutant_path = _write_mutant(cell, result, name, mutate)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, mutant_path, cell)
|
||||
|
||||
def test_reader_rejects_noncanonical_issue_order(self):
|
||||
cell = _cell()
|
||||
issues = (_issue("model_missing"), _issue("stream_incompatible"))
|
||||
result = make_result(cell, _capability(), _requested_only(cell), issues)
|
||||
self._write(self.root, "ordered.json", cell, result)
|
||||
raw = (self.root / "ordered.json").read_text(encoding="ascii")
|
||||
first = '{"code":"model_missing","resume_code":"register_model"}'
|
||||
second = '{"code":"stream_incompatible","resume_code":"implement_stream_adapter"}'
|
||||
self.assertIn(f"{first},{second}", raw)
|
||||
(self.root / "swapped.json").write_text(raw.replace(f"{first},{second}", f"{second},{first}"), encoding="ascii")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "swapped.json", cell)
|
||||
with self.assertRaises(ConnectivityValidationError):
|
||||
make_result(cell, _capability(), _requested_only(cell), tuple(reversed(issues)))
|
||||
|
||||
def test_symlinked_roots_parents_and_targets_are_rejected(self):
|
||||
outside = self.base / "outside"
|
||||
outside.mkdir()
|
||||
real_root = self.base / "real" / "evidence"
|
||||
real_root.mkdir(parents=True)
|
||||
(self.base / "link").symlink_to(self.base / "real", target_is_directory=True)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.base / "link" / "evidence", "preflight.json")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.base / "link" / "evidence", "preflight.json", self.cell)
|
||||
self.assertFalse((real_root / "preflight.json").exists())
|
||||
self._write(real_root, "preflight.json")
|
||||
self.assertEqual(read_evidence(real_root, "preflight.json", self.cell)["status"], "ready")
|
||||
|
||||
(self.base / "root-link").symlink_to(self.root, target_is_directory=True)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.base / "root-link", "preflight.json")
|
||||
|
||||
(self.root / "nested").symlink_to(outside, target_is_directory=True)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, "nested/preflight.json")
|
||||
self.assertFalse((outside / "preflight.json").exists())
|
||||
|
||||
(self.root / "link.json").symlink_to(outside / "escape.json")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, "link.json")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "link.json", self.cell)
|
||||
self.assertFalse((outside / "escape.json").exists())
|
||||
|
||||
def test_oversized_directory_and_non_regular_targets_are_rejected(self):
|
||||
self._write(self.root, "small.json")
|
||||
self.assertEqual(read_evidence(self.root, "small.json", self.cell)["status"], "ready")
|
||||
(self.root / "big.json").write_bytes(b"{" + b" " * (MAX_EVIDENCE_BYTES + 16) + b"}")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "big.json", self.cell)
|
||||
(self.root / "dir.json").mkdir()
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "dir.json", self.cell)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, "dir.json")
|
||||
if hasattr(os, "mkfifo"):
|
||||
os.mkfifo(self.root / "fifo.json")
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, "fifo.json", self.cell)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, "fifo.json")
|
||||
|
||||
def test_private_identity_and_escaping_paths_are_rejected(self):
|
||||
for identity in ("https://private.invalid", "sha256:" + "z" * 64, "", "sha256:abcd", None):
|
||||
with self.subTest(identity=identity):
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
canonical_evidence_bytes(self.cell, self.result, identity, self.config_identity)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
canonical_evidence_bytes(self.cell, self.result, self.endpoint_identity, identity)
|
||||
for relative in ("../escape.json", "/abs.json", "nested/../escape.json", "preflight.txt", "preflight", "", ".json"):
|
||||
with self.subTest(relative=relative):
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
self._write(self.root, relative)
|
||||
with self.assertRaises(ConnectivityEvidenceError):
|
||||
read_evidence(self.root, relative, self.cell)
|
||||
self.assertFalse((self.base / "escape.json").exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,807 +0,0 @@
|
|||
"""
|
||||
Closed manifest loader, validator, and digest calculator.
|
||||
|
||||
Standard-library-only. Returns frozen dataclasses; never mutable raw dicts.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import posixpath
|
||||
import re
|
||||
import struct
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
PIPELINE_VERSION = "2"
|
||||
ENVIRONMENT = "dev"
|
||||
TESTBED_REQUIRED = "../iop-s2"
|
||||
SESSION_POLICY = "fresh"
|
||||
SETUP_CACHE_POLICY = "isolated"
|
||||
DEFAULT_REPETITIONS = 1
|
||||
RUBRIC_VERSION = "landing-quality-v1"
|
||||
ONE_SHOT_RUBRIC_VERSION = "one-shot-agent-comparison-v1"
|
||||
RUBRIC_VERSIONS = (RUBRIC_VERSION, ONE_SHOT_RUBRIC_VERSION)
|
||||
|
||||
CALLER_ENUM = ("claude", "agy", "codex")
|
||||
ROUTE_KIND_ENUM = ("direct", "execution_preset")
|
||||
STAGE_ENUM = ("request", "selector", "plan", "work", "review", "repair")
|
||||
# Canonical rank for sorting bindings: lower rank = earlier position
|
||||
STAGE_RANK: dict[str, int] = {s: i for i, s in enumerate(STAGE_ENUM)}
|
||||
|
||||
CELL_ID_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
|
||||
TOKEN_RE = re.compile(r"^[a-z0-9][a-z0-9_.+-]{0,31}$")
|
||||
VIEWPORT_ID_RE = re.compile(r"^[a-z0-9][a-z0-9_.+-]{0,31}$")
|
||||
CHECKSUM_RE = re.compile(r"^sha256:[0-9a-f]{64}$")
|
||||
|
||||
WORKSPACE_PREFIX = b"IOP-BENCH-WORKSPACE\x00"
|
||||
MANIFEST_PREFIX = b"IOP-BENCH-MANIFEST\x00"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frozen data containers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Timeout:
|
||||
run_seconds: int
|
||||
idle_seconds: int
|
||||
quiet_seconds: int
|
||||
cleanup_grace_seconds: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Viewport:
|
||||
id: str
|
||||
width: int
|
||||
height: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AssetMapping:
|
||||
source: str
|
||||
workspace_path: str
|
||||
content: bytes = field(default=b"", repr=False)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpectedBinding:
|
||||
stage: str
|
||||
model: str
|
||||
effort: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IopCell:
|
||||
request_model: str
|
||||
requested_effort: str
|
||||
route_kind: str
|
||||
route_id: str
|
||||
expected_bindings: tuple[ExpectedBinding, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MatrixCell:
|
||||
id: str
|
||||
caller: str
|
||||
iop: IopCell
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Evaluator:
|
||||
caller: str
|
||||
iop: IopCell
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Fixture:
|
||||
version: str
|
||||
prompt: str
|
||||
assets: tuple[AssetMapping, ...]
|
||||
checksum: str
|
||||
prompt_content: bytes = field(default=b"", repr=False)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Manifest:
|
||||
pipeline_version: str
|
||||
environment: str
|
||||
testbed: str
|
||||
repetitions: int
|
||||
session_policy: str
|
||||
setup_cache_policy: str
|
||||
timeout: Timeout
|
||||
viewports: tuple[Viewport, ...]
|
||||
rubric_version: str
|
||||
evaluator: Evaluator
|
||||
output_root: str
|
||||
fixture: Fixture
|
||||
matrix: tuple[MatrixCell, ...]
|
||||
digest: str
|
||||
execution_order_seed: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Errors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class ManifestError(Exception):
|
||||
"""Base error for manifest validation failures."""
|
||||
|
||||
|
||||
class ManifestValidationError(ManifestError):
|
||||
"""Raised when the manifest JSON fails schema validation."""
|
||||
|
||||
|
||||
class ManifestPathError(ManifestError):
|
||||
"""Raised when a path rule is violated (escape, symlink, collision, etc.)."""
|
||||
|
||||
|
||||
class ManifestDigestError(ManifestError):
|
||||
"""Raised when a declared digest does not match the computed value."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Internal helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _require_bool(cond: bool, msg: str) -> None:
|
||||
if not cond:
|
||||
raise ManifestValidationError(msg)
|
||||
|
||||
|
||||
def _require_str(value: Any, field_name: str) -> str:
|
||||
if not isinstance(value, str):
|
||||
raise ManifestValidationError(f"field '{field_name}' must be a string")
|
||||
return value
|
||||
|
||||
|
||||
def _require_int(value: Any, field_name: str) -> int:
|
||||
if isinstance(value, bool) or not isinstance(value, int):
|
||||
raise ManifestValidationError(f"field '{field_name}' must be an integer")
|
||||
return value
|
||||
|
||||
|
||||
def _require_object(value: Any, field_name: str) -> dict[str, Any]:
|
||||
if not isinstance(value, dict):
|
||||
raise ManifestValidationError(f"field '{field_name}' must be an object")
|
||||
return value
|
||||
|
||||
|
||||
def _require_array(value: Any, field_name: str) -> list[Any]:
|
||||
if not isinstance(value, list):
|
||||
raise ManifestValidationError(f"field '{field_name}' must be an array")
|
||||
return value
|
||||
|
||||
|
||||
def _require_enum(value: Any, field_name: str, allowed: tuple[str, ...]) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if s not in allowed:
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' must be one of {allowed}"
|
||||
)
|
||||
return s
|
||||
|
||||
|
||||
def _require_pattern(value: Any, field_name: str, pattern: re.Pattern) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if pattern.fullmatch(s) is None:
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' does not match required pattern"
|
||||
)
|
||||
return s
|
||||
|
||||
|
||||
def _require_positive_int(value: Any, field_name: str) -> int:
|
||||
val = _require_int(value, field_name)
|
||||
if val <= 0:
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' must be a positive integer"
|
||||
)
|
||||
return val
|
||||
|
||||
|
||||
def _require_bounded_int(value: Any, field_name: str, lo: int, hi: int) -> int:
|
||||
val = _require_int(value, field_name)
|
||||
if not (lo <= val <= hi):
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' must be between {lo} and {hi}"
|
||||
)
|
||||
return val
|
||||
|
||||
|
||||
def _require_sha256(value: Any, field_name: str) -> str:
|
||||
s = _require_str(value, field_name)
|
||||
if not CHECKSUM_RE.match(s):
|
||||
raise ManifestValidationError(
|
||||
f"field '{field_name}' must match sha256:<64 hex chars>"
|
||||
)
|
||||
return s
|
||||
|
||||
|
||||
def _normalize_posix_relative_path(path_str: Any, context: str) -> str:
|
||||
"""Validate and return a normalized POSIX relative path.
|
||||
|
||||
Rejects non-string, empty, absolute, colon, backslash, non-normal,
|
||||
or escaping paths ('.', '..', starting with '../').
|
||||
"""
|
||||
if not isinstance(path_str, str) or not path_str:
|
||||
raise ManifestPathError(f"field '{context}' must be a non-empty string path")
|
||||
if "\\" in path_str or ":" in path_str:
|
||||
raise ManifestPathError(f"field '{context}' contains invalid path characters")
|
||||
if path_str.startswith("/"):
|
||||
raise ManifestPathError(f"field '{context}' must be a relative path")
|
||||
|
||||
norm = posixpath.normpath(path_str)
|
||||
if path_str != norm:
|
||||
raise ManifestPathError(f"field '{context}' is not in canonical relative form")
|
||||
if norm in (".", "..") or norm.startswith("../"):
|
||||
raise ManifestPathError(f"field '{context}' escapes root or is empty")
|
||||
return norm
|
||||
|
||||
|
||||
def _require_regular_file(path: Path, context: str) -> None:
|
||||
"""Require that path exists and is a regular file (no symlinks)."""
|
||||
if not path.exists():
|
||||
raise ManifestPathError(f"field '{context}' target file does not exist")
|
||||
if path.is_symlink():
|
||||
raise ManifestPathError(f"field '{context}' target must be a regular file, not a symlink")
|
||||
if not path.is_file():
|
||||
raise ManifestPathError(f"field '{context}' target is not a regular file")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Digest computation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def digest_workspace_inputs(assets: Iterable[AssetMapping]) -> str:
|
||||
"""Compute the declared fixture checksum from asset workspace paths and content.
|
||||
|
||||
sha256(b"IOP-BENCH-WORKSPACE\x00" + length-framed assets sorted by
|
||||
workspace_path, each asset framed as:
|
||||
uint64 BE len(workspace_path) + workspace_path UTF-8 bytes +
|
||||
uint64 BE len(file_content) + file_content bytes
|
||||
"""
|
||||
sorted_assets = sorted(assets, key=lambda a: a.workspace_path)
|
||||
data = bytearray(WORKSPACE_PREFIX)
|
||||
for asset in sorted_assets:
|
||||
wp_bytes = asset.workspace_path.encode("utf-8")
|
||||
data += struct.pack(">Q", len(wp_bytes)) + wp_bytes
|
||||
data += struct.pack(">Q", len(asset.content)) + asset.content
|
||||
return "sha256:" + hashlib.sha256(bytes(data)).hexdigest()
|
||||
|
||||
|
||||
def digest_manifest_and_resolved_inputs(manifest: Manifest) -> str:
|
||||
"""Compute the self-contained manifest digest.
|
||||
|
||||
sha256(b"IOP-BENCH-MANIFEST\x00" + canonical JSON + length-framed
|
||||
prompt path/content + length-framed every asset source/destination/content).
|
||||
"""
|
||||
prompt_content = manifest.fixture.prompt_content
|
||||
|
||||
canonical = json.dumps(
|
||||
_manifest_to_dict(manifest),
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=False,
|
||||
).encode("utf-8")
|
||||
|
||||
data = bytearray(MANIFEST_PREFIX)
|
||||
data += struct.pack(">Q", len(canonical)) + canonical
|
||||
|
||||
prompt_path_bytes = manifest.fixture.prompt.encode("utf-8")
|
||||
data += struct.pack(">Q", len(prompt_path_bytes)) + prompt_path_bytes
|
||||
data += struct.pack(">Q", len(prompt_content)) + prompt_content
|
||||
|
||||
for asset in sorted(manifest.fixture.assets, key=lambda a: a.workspace_path):
|
||||
src_bytes = asset.source.encode("utf-8")
|
||||
data += struct.pack(">Q", len(src_bytes)) + src_bytes
|
||||
wp_bytes = asset.workspace_path.encode("utf-8")
|
||||
data += struct.pack(">Q", len(wp_bytes)) + wp_bytes
|
||||
data += struct.pack(">Q", len(asset.content)) + asset.content
|
||||
|
||||
return "sha256:" + hashlib.sha256(bytes(data)).hexdigest()
|
||||
|
||||
|
||||
def _manifest_to_dict(manifest: Manifest) -> dict[str, Any]:
|
||||
"""Convert a Manifest to a plain dict for canonical JSON serialization."""
|
||||
canonical = {
|
||||
"pipeline_version": manifest.pipeline_version,
|
||||
"environment": manifest.environment,
|
||||
"testbed": manifest.testbed,
|
||||
"repetitions": manifest.repetitions,
|
||||
"session_policy": manifest.session_policy,
|
||||
"setup_cache_policy": manifest.setup_cache_policy,
|
||||
"timeout": {
|
||||
"run_seconds": manifest.timeout.run_seconds,
|
||||
"idle_seconds": manifest.timeout.idle_seconds,
|
||||
"quiet_seconds": manifest.timeout.quiet_seconds,
|
||||
"cleanup_grace_seconds": manifest.timeout.cleanup_grace_seconds,
|
||||
},
|
||||
"viewports": [
|
||||
{"id": v.id, "width": v.width, "height": v.height}
|
||||
for v in manifest.viewports
|
||||
],
|
||||
"rubric_version": manifest.rubric_version,
|
||||
"evaluator": {
|
||||
"caller": manifest.evaluator.caller,
|
||||
"iop": {
|
||||
"request_model": manifest.evaluator.iop.request_model,
|
||||
"requested_effort": manifest.evaluator.iop.requested_effort,
|
||||
"route_kind": manifest.evaluator.iop.route_kind,
|
||||
"route_id": manifest.evaluator.iop.route_id,
|
||||
"expected_bindings": [
|
||||
{
|
||||
"stage": binding.stage,
|
||||
"model": binding.model,
|
||||
**({"effort": binding.effort} if binding.effort else {}),
|
||||
}
|
||||
for binding in manifest.evaluator.iop.expected_bindings
|
||||
],
|
||||
},
|
||||
},
|
||||
"output_root": manifest.output_root,
|
||||
"fixture": {
|
||||
"version": manifest.fixture.version,
|
||||
"prompt": manifest.fixture.prompt,
|
||||
"assets": [
|
||||
{"source": a.source, "workspace_path": a.workspace_path}
|
||||
for a in manifest.fixture.assets
|
||||
],
|
||||
"checksum": manifest.fixture.checksum,
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": c.id,
|
||||
"caller": c.caller,
|
||||
"iop": {
|
||||
"request_model": c.iop.request_model,
|
||||
"requested_effort": c.iop.requested_effort,
|
||||
"route_kind": c.iop.route_kind,
|
||||
"route_id": c.iop.route_id,
|
||||
"expected_bindings": [
|
||||
{
|
||||
"stage": b.stage,
|
||||
"model": b.model,
|
||||
**({"effort": b.effort} if b.effort else {}),
|
||||
}
|
||||
for b in c.iop.expected_bindings
|
||||
],
|
||||
},
|
||||
}
|
||||
for c in manifest.matrix
|
||||
],
|
||||
}
|
||||
# Pipeline-v2 manifests without an explicit seed keep their legacy
|
||||
# canonical bytes and digest. An explicit seed is part of the contract.
|
||||
if manifest.execution_order_seed is not None:
|
||||
canonical["execution_order_seed"] = manifest.execution_order_seed
|
||||
return canonical
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Validation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _validate_timeout(data: dict[str, Any]) -> Timeout:
|
||||
obj = _require_object(data, "timeout")
|
||||
expected_keys = {"run_seconds", "idle_seconds", "quiet_seconds", "cleanup_grace_seconds"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError("timeout has invalid schema")
|
||||
return Timeout(
|
||||
run_seconds=_require_bounded_int(obj["run_seconds"], "timeout.run_seconds", 1, 86400),
|
||||
idle_seconds=_require_bounded_int(obj["idle_seconds"], "timeout.idle_seconds", 1, 600),
|
||||
quiet_seconds=_require_bounded_int(obj["quiet_seconds"], "timeout.quiet_seconds", 1, 60),
|
||||
cleanup_grace_seconds=_require_bounded_int(
|
||||
obj["cleanup_grace_seconds"], "timeout.cleanup_grace_seconds", 1, 60
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _validate_viewports(data: list[Any]) -> tuple[Viewport, ...]:
|
||||
arr = _require_array(data, "viewports")
|
||||
if len(arr) < 1:
|
||||
raise ManifestValidationError("viewports must have at least 1 item")
|
||||
seen_ids: set[str] = set()
|
||||
viewports: list[Viewport] = []
|
||||
for i, item in enumerate(arr):
|
||||
obj = _require_object(item, f"viewports[{i}]")
|
||||
expected_keys = {"id", "width", "height"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError(f"viewports[{i}] has invalid schema")
|
||||
vid = _require_pattern(obj["id"], f"viewports[{i}].id", VIEWPORT_ID_RE)
|
||||
if vid in seen_ids:
|
||||
raise ManifestValidationError(f"viewports[{i}].id is duplicate")
|
||||
seen_ids.add(vid)
|
||||
w = _require_bounded_int(obj["width"], f"viewports[{i}].width", 1, 8192)
|
||||
h = _require_bounded_int(obj["height"], f"viewports[{i}].height", 1, 8192)
|
||||
viewports.append(Viewport(id=vid, width=w, height=h))
|
||||
return tuple(viewports)
|
||||
|
||||
|
||||
def _validate_output_root(data: Any, repo_root: Path) -> str:
|
||||
s = _require_str(data, "output_root")
|
||||
if "\\" in s or ":" in s:
|
||||
raise ManifestPathError("field 'output_root' contains invalid path characters")
|
||||
norm = posixpath.normpath(s)
|
||||
if s != norm:
|
||||
raise ManifestPathError("field 'output_root' is not in canonical form")
|
||||
if not s.startswith("agent-test/runs/"):
|
||||
raise ManifestValidationError(
|
||||
"field 'output_root' must be relative to agent-test/runs/"
|
||||
)
|
||||
tail = s[len("agent-test/runs/"):]
|
||||
if not tail or "/" in tail:
|
||||
raise ManifestValidationError(
|
||||
"field 'output_root' must be a single non-empty segment after agent-test/runs/"
|
||||
)
|
||||
|
||||
runs_dir = (repo_root / "agent-test" / "runs").resolve()
|
||||
output_path = (repo_root / s).resolve()
|
||||
try:
|
||||
output_path.relative_to(runs_dir)
|
||||
except ValueError:
|
||||
raise ManifestPathError("field 'output_root' resolves outside agent-test/runs")
|
||||
return s
|
||||
|
||||
|
||||
def _validate_asset_mapping(
|
||||
data: dict[str, Any], i: int
|
||||
) -> tuple[str, str]:
|
||||
obj = _require_object(data, f"assets[{i}]")
|
||||
expected_keys = {"source", "workspace_path"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError(f"assets[{i}] has invalid schema")
|
||||
source = _normalize_posix_relative_path(obj["source"], f"assets[{i}].source")
|
||||
workspace_path = _normalize_posix_relative_path(
|
||||
obj["workspace_path"], f"assets[{i}].workspace_path"
|
||||
)
|
||||
return source, workspace_path
|
||||
|
||||
|
||||
def _validate_fixture(
|
||||
data: dict[str, Any], repo_root: Path
|
||||
) -> Fixture:
|
||||
obj = _require_object(data, "fixture")
|
||||
expected_keys = {"version", "prompt", "assets", "checksum"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError("fixture has invalid schema")
|
||||
|
||||
version = _require_pattern(obj["version"], "fixture.version", TOKEN_RE)
|
||||
prompt = _normalize_posix_relative_path(obj["prompt"], "fixture.prompt")
|
||||
|
||||
assets_raw = _require_array(obj["assets"], "fixture.assets")
|
||||
if len(assets_raw) < 1:
|
||||
raise ManifestValidationError("fixture.assets must have at least 1 item")
|
||||
|
||||
assets_list: list[AssetMapping] = []
|
||||
workspace_destinations: set[str] = set()
|
||||
for i, item in enumerate(assets_raw):
|
||||
source, workspace_path = _validate_asset_mapping(item, i)
|
||||
if workspace_path in workspace_destinations:
|
||||
raise ManifestPathError(
|
||||
"fixture.assets workspace_path is duplicate"
|
||||
)
|
||||
workspace_destinations.add(workspace_path)
|
||||
|
||||
src_path = repo_root / source
|
||||
_require_regular_file(src_path, f"fixture asset source ({i})")
|
||||
try:
|
||||
src_path.resolve().relative_to(repo_root.resolve())
|
||||
except ValueError:
|
||||
raise ManifestPathError("fixture asset source resolves outside repo root")
|
||||
|
||||
content = src_path.read_bytes()
|
||||
assets_list.append(AssetMapping(source=source, workspace_path=workspace_path, content=content))
|
||||
|
||||
assets_list.sort(key=lambda a: a.workspace_path)
|
||||
|
||||
# Validate prompt file
|
||||
prompt_path = repo_root / prompt
|
||||
_require_regular_file(prompt_path, "fixture prompt")
|
||||
try:
|
||||
prompt_path.resolve().relative_to(repo_root.resolve())
|
||||
except ValueError:
|
||||
raise ManifestPathError("fixture prompt resolves outside repo root")
|
||||
prompt_content = prompt_path.read_bytes()
|
||||
|
||||
checksum = _require_sha256(obj["checksum"], "fixture.checksum")
|
||||
computed = digest_workspace_inputs(assets_list)
|
||||
if checksum != computed:
|
||||
raise ManifestDigestError("fixture.checksum mismatch")
|
||||
|
||||
return Fixture(
|
||||
version=version,
|
||||
prompt=prompt,
|
||||
assets=tuple(assets_list),
|
||||
checksum=checksum,
|
||||
prompt_content=prompt_content,
|
||||
)
|
||||
|
||||
|
||||
def _validate_expected_binding(
|
||||
data: dict[str, Any], i: int
|
||||
) -> ExpectedBinding:
|
||||
obj = _require_object(data, f"expected_bindings[{i}]")
|
||||
expected_keys = {"stage", "model"}
|
||||
if not expected_keys.issubset(set(obj.keys())):
|
||||
raise ManifestValidationError(f"expected_bindings[{i}] missing required keys")
|
||||
extra = set(obj.keys()) - {"stage", "model", "effort"}
|
||||
if extra:
|
||||
raise ManifestValidationError(f"expected_bindings[{i}] has unexpected keys")
|
||||
stage = _require_enum(obj["stage"], f"expected_bindings[{i}].stage", STAGE_ENUM)
|
||||
model = _require_pattern(obj["model"], f"expected_bindings[{i}].model", TOKEN_RE)
|
||||
effort = None
|
||||
if "effort" in obj:
|
||||
effort = _require_pattern(obj["effort"], f"expected_bindings[{i}].effort", TOKEN_RE)
|
||||
return ExpectedBinding(stage=stage, model=model, effort=effort)
|
||||
|
||||
|
||||
def _validate_iop_cell(data: dict[str, Any]) -> IopCell:
|
||||
obj = _require_object(data, "iop")
|
||||
expected_keys = {"request_model", "requested_effort", "route_kind", "route_id", "expected_bindings"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError("iop has invalid schema")
|
||||
|
||||
request_model = _require_pattern(obj["request_model"], "iop.request_model", TOKEN_RE)
|
||||
requested_effort = _require_pattern(obj["requested_effort"], "iop.requested_effort", TOKEN_RE)
|
||||
route_kind = _require_enum(obj["route_kind"], "iop.route_kind", ROUTE_KIND_ENUM)
|
||||
route_id = _require_pattern(obj["route_id"], "iop.route_id", TOKEN_RE)
|
||||
|
||||
bindings_raw = _require_array(obj["expected_bindings"], "iop.expected_bindings")
|
||||
if len(bindings_raw) < 1:
|
||||
raise ManifestValidationError("iop.expected_bindings must have at least 1 item")
|
||||
|
||||
bindings: list[ExpectedBinding] = []
|
||||
seen_stages: set[str] = set()
|
||||
for i, item in enumerate(bindings_raw):
|
||||
b = _validate_expected_binding(item, i)
|
||||
if b.stage in seen_stages:
|
||||
raise ManifestValidationError("iop.expected_bindings contains duplicate stage")
|
||||
seen_stages.add(b.stage)
|
||||
bindings.append(b)
|
||||
|
||||
# Sort bindings by canonical stage rank
|
||||
bindings.sort(key=lambda b: STAGE_RANK[b.stage])
|
||||
|
||||
# Validate direct vs execution_preset constraints
|
||||
if route_kind == "direct":
|
||||
if set(b.stage for b in bindings) != {"request"}:
|
||||
raise ManifestValidationError(
|
||||
"direct route requires exactly one binding with stage=request"
|
||||
)
|
||||
elif route_kind == "execution_preset":
|
||||
required_stages = {"selector", "plan", "work", "review"}
|
||||
allowed_stages = required_stages | {"repair"}
|
||||
actual_stages = set(b.stage for b in bindings)
|
||||
if actual_stages not in (required_stages, allowed_stages):
|
||||
raise ManifestValidationError(
|
||||
"execution_preset has an invalid stage set"
|
||||
)
|
||||
|
||||
return IopCell(
|
||||
request_model=request_model,
|
||||
requested_effort=requested_effort,
|
||||
route_kind=route_kind,
|
||||
route_id=route_id,
|
||||
expected_bindings=tuple(bindings),
|
||||
)
|
||||
|
||||
|
||||
def _validate_cell(data: dict[str, Any], i: int) -> MatrixCell:
|
||||
obj = _require_object(data, f"matrix[{i}]")
|
||||
expected_keys = {"id", "caller", "iop"}
|
||||
if set(obj.keys()) != expected_keys:
|
||||
raise ManifestValidationError(f"matrix[{i}] has invalid schema")
|
||||
|
||||
cell_id = _require_pattern(obj["id"], f"matrix[{i}].id", CELL_ID_RE)
|
||||
caller = _require_enum(obj["caller"], f"matrix[{i}].caller", CALLER_ENUM)
|
||||
iop = _validate_iop_cell(obj["iop"])
|
||||
return MatrixCell(id=cell_id, caller=caller, iop=iop)
|
||||
|
||||
|
||||
def _validate_evaluator(data: dict[str, Any]) -> Evaluator:
|
||||
obj = _require_object(data, "evaluator")
|
||||
if set(obj) != {"caller", "iop"}:
|
||||
raise ManifestValidationError("evaluator has invalid schema")
|
||||
caller = _require_enum(obj["caller"], "evaluator.caller", ("codex",))
|
||||
return Evaluator(caller=caller, iop=_validate_iop_cell(obj["iop"]))
|
||||
|
||||
|
||||
def _execution_order_key(seed: str, cell: MatrixCell) -> tuple[bytes, str]:
|
||||
rank = hashlib.sha256(
|
||||
b"iop-benchmark-order-v1\0"
|
||||
+ seed.encode("ascii")
|
||||
+ b"\0"
|
||||
+ cell.id.encode("ascii")
|
||||
).digest()
|
||||
return rank, cell.id
|
||||
|
||||
|
||||
def _validate_matrix(
|
||||
data: list[Any], execution_order_seed: str | None
|
||||
) -> tuple[MatrixCell, ...]:
|
||||
arr = _require_array(data, "matrix")
|
||||
if len(arr) < 1:
|
||||
raise ManifestValidationError("matrix must have at least 1 item")
|
||||
|
||||
cells: list[MatrixCell] = []
|
||||
seen_ids: set[str] = set()
|
||||
for i, item in enumerate(arr):
|
||||
cell = _validate_cell(item, i)
|
||||
if cell.id in seen_ids:
|
||||
raise ManifestValidationError("matrix contains duplicate cell id")
|
||||
seen_ids.add(cell.id)
|
||||
cells.append(cell)
|
||||
|
||||
# No seed preserves the pipeline-v2 cell-id order. An explicit seed is
|
||||
# stable across JSON input permutations and platforms.
|
||||
cells.sort(
|
||||
key=(lambda cell: cell.id)
|
||||
if execution_order_seed is None
|
||||
else (lambda cell: _execution_order_key(execution_order_seed, cell))
|
||||
)
|
||||
return tuple(cells)
|
||||
|
||||
|
||||
def _default_repo_root(manifest_path: Path) -> Path:
|
||||
"""Walk up from manifest_path to find the repository root.
|
||||
|
||||
Looks for a directory containing ``Makefile`` or ``.git``.
|
||||
Falls back to the manifest file's parent directory.
|
||||
"""
|
||||
candidate = manifest_path.resolve()
|
||||
for _ in range(20): # safety limit
|
||||
if (candidate / "Makefile").is_file() or (candidate / ".git").exists():
|
||||
return candidate
|
||||
parent = candidate.parent
|
||||
if parent == candidate:
|
||||
break
|
||||
candidate = parent
|
||||
return manifest_path.resolve().parent
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_manifest(path: str | Path, repo_root: str | Path | None = None) -> Manifest:
|
||||
"""Load and validate a benchmark manifest JSON file.
|
||||
|
||||
Args:
|
||||
path: Path to the manifest JSON file.
|
||||
repo_root: Repository root for resolving relative paths.
|
||||
|
||||
Returns:
|
||||
A frozen Manifest object with computed digest.
|
||||
|
||||
Raises:
|
||||
ManifestValidationError: If the JSON fails schema validation.
|
||||
ManifestPathError: If a path rule is violated.
|
||||
ManifestDigestError: If a declared digest does not match.
|
||||
"""
|
||||
path = Path(path)
|
||||
if repo_root is None:
|
||||
repo_root = _default_repo_root(path)
|
||||
repo_root = Path(repo_root).resolve()
|
||||
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
except FileNotFoundError:
|
||||
raise ManifestValidationError("manifest file not found")
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
raise ManifestValidationError("invalid JSON format")
|
||||
|
||||
if not isinstance(raw, dict):
|
||||
raise ManifestValidationError("manifest must be a JSON object")
|
||||
|
||||
# Top-level field validation
|
||||
expected_top = {
|
||||
"pipeline_version", "environment", "testbed", "fixture", "matrix",
|
||||
"session_policy", "setup_cache_policy", "timeout", "viewports",
|
||||
"rubric_version", "evaluator", "output_root",
|
||||
}
|
||||
optional_top = {"repetitions", "execution_order_seed"}
|
||||
declared_keys = set(raw.keys())
|
||||
required_present = expected_top.issubset(declared_keys)
|
||||
extra = declared_keys - expected_top - optional_top
|
||||
if not required_present:
|
||||
raise ManifestValidationError("manifest missing required top-level fields")
|
||||
if extra:
|
||||
raise ManifestValidationError("manifest has unexpected top-level fields")
|
||||
|
||||
pipeline_version = _require_enum(raw["pipeline_version"], "pipeline_version", (PIPELINE_VERSION,))
|
||||
environment = _require_enum(raw["environment"], "environment", (ENVIRONMENT,))
|
||||
testbed = _require_enum(raw["testbed"], "testbed", (TESTBED_REQUIRED,))
|
||||
|
||||
repetitions = DEFAULT_REPETITIONS
|
||||
if "repetitions" in raw:
|
||||
repetitions = _require_positive_int(raw["repetitions"], "repetitions")
|
||||
|
||||
execution_order_seed: str | None = None
|
||||
if "execution_order_seed" in raw:
|
||||
execution_order_seed = _require_pattern(
|
||||
raw["execution_order_seed"], "execution_order_seed", CELL_ID_RE
|
||||
)
|
||||
|
||||
session_policy = _require_enum(raw["session_policy"], "session_policy", (SESSION_POLICY,))
|
||||
setup_cache_policy = _require_enum(
|
||||
raw["setup_cache_policy"], "setup_cache_policy", (SETUP_CACHE_POLICY,)
|
||||
)
|
||||
timeout = _validate_timeout(raw["timeout"])
|
||||
viewports = _validate_viewports(raw["viewports"])
|
||||
rubric_version = _require_enum(
|
||||
raw["rubric_version"], "rubric_version", RUBRIC_VERSIONS
|
||||
)
|
||||
evaluator = _validate_evaluator(raw["evaluator"])
|
||||
output_root = _validate_output_root(raw["output_root"], repo_root)
|
||||
|
||||
fixture = _validate_fixture(raw["fixture"], repo_root)
|
||||
matrix = _validate_matrix(raw["matrix"], execution_order_seed)
|
||||
|
||||
# Compute manifest digest during load
|
||||
dummy_manifest = Manifest(
|
||||
pipeline_version=pipeline_version,
|
||||
environment=environment,
|
||||
testbed=testbed,
|
||||
repetitions=repetitions,
|
||||
session_policy=session_policy,
|
||||
setup_cache_policy=setup_cache_policy,
|
||||
timeout=timeout,
|
||||
viewports=viewports,
|
||||
rubric_version=rubric_version,
|
||||
evaluator=evaluator,
|
||||
output_root=output_root,
|
||||
fixture=fixture,
|
||||
matrix=matrix,
|
||||
digest="",
|
||||
execution_order_seed=execution_order_seed,
|
||||
)
|
||||
computed_digest = digest_manifest_and_resolved_inputs(dummy_manifest)
|
||||
|
||||
return Manifest(
|
||||
pipeline_version=pipeline_version,
|
||||
environment=environment,
|
||||
testbed=testbed,
|
||||
repetitions=repetitions,
|
||||
session_policy=session_policy,
|
||||
setup_cache_policy=setup_cache_policy,
|
||||
timeout=timeout,
|
||||
viewports=viewports,
|
||||
rubric_version=rubric_version,
|
||||
evaluator=evaluator,
|
||||
output_root=output_root,
|
||||
fixture=fixture,
|
||||
matrix=matrix,
|
||||
digest=computed_digest,
|
||||
execution_order_seed=execution_order_seed,
|
||||
)
|
||||
|
||||
|
||||
def validate_manifest_bytes(
|
||||
data: bytes,
|
||||
path_hint: str = "<bytes>",
|
||||
repo_root: str | Path | None = None,
|
||||
) -> Manifest:
|
||||
"""Validate manifest JSON bytes without writing to persistent disk."""
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="wb", suffix=".json", delete=False
|
||||
) as tmp:
|
||||
tmp.write(data)
|
||||
tmp_path = tmp.name
|
||||
try:
|
||||
return load_manifest(tmp_path, repo_root=repo_root)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp_path)
|
||||
except OSError:
|
||||
pass
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,926 +0,0 @@
|
|||
"""Source-aware timing and usage evidence for exactly one benchmark attempt.
|
||||
|
||||
This module owns three things and nothing else: the closed measurement schema,
|
||||
a bounded workspace write observer, and the no-clobber sidecar publisher and
|
||||
strict loader for ``attempt-measurement.json``.
|
||||
|
||||
Every required value is either an ``observed`` value carrying its unit, clock
|
||||
and source, or an explicit ``unavailable`` value carrying the reason it could
|
||||
not be observed. Nothing here decomposes, sums, subtracts or reconstructs a
|
||||
value the caller or the harness did not report: a missing provider total stays
|
||||
unavailable, overlapping intervals stay overlapping, and a filesystem
|
||||
modification time is never presented as proof of the first write.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Mapping, Optional
|
||||
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CLOCK_FILESYSTEM_MTIME,
|
||||
CLOCK_HARNESS_MONOTONIC,
|
||||
CLOCK_NONE,
|
||||
EVENT_FIRST_OUTPUT,
|
||||
EVENT_SUBMITTED,
|
||||
METRIC_CLOCKS,
|
||||
METRIC_NAMES,
|
||||
METRIC_PREFIX,
|
||||
METRIC_SOURCES,
|
||||
METRIC_UNITS,
|
||||
SOURCE_HARNESS,
|
||||
SOURCE_WORKSPACE_POLL,
|
||||
UNIT_NANOSECONDS,
|
||||
InvocationResult,
|
||||
HarnessOutcome,
|
||||
LifecycleMetricError,
|
||||
LifecycleValidationError,
|
||||
ParsedMetric,
|
||||
ProcessOutcome,
|
||||
ProductOutcome,
|
||||
metric_record,
|
||||
publish_bytes_no_replace,
|
||||
validate_metric,
|
||||
)
|
||||
|
||||
MEASUREMENT_FILENAME = "attempt-measurement.json"
|
||||
MEASUREMENT_VERSION = 2
|
||||
MEASUREMENT_RECORD = "attempt_measurement"
|
||||
|
||||
STATUS_OBSERVED = "observed"
|
||||
STATUS_UNAVAILABLE = "unavailable"
|
||||
|
||||
REASON_NOT_REPORTED = "not_reported"
|
||||
REASON_NOT_OBSERVED = "not_observed"
|
||||
REASON_AMBIGUOUS_TOTAL = "ambiguous_total"
|
||||
REASON_OBSERVER_UNAVAILABLE = "observer_unavailable"
|
||||
UNAVAILABLE_REASONS = (
|
||||
REASON_NOT_REPORTED, REASON_NOT_OBSERVED, REASON_AMBIGUOUS_TOTAL,
|
||||
REASON_OBSERVER_UNAVAILABLE,
|
||||
)
|
||||
|
||||
TIMELINE_NAMES = (
|
||||
"submitted_at", "first_output_at", "first_write_observed_at",
|
||||
"first_write_mtime", "total_duration",
|
||||
)
|
||||
OBSERVATION_UNITS = tuple(sorted(set(METRIC_UNITS.values())))
|
||||
|
||||
DURATION_NS_PER_SECOND = 10 ** 9
|
||||
# The sampling cadence matches the lifecycle controller's own poll interval and
|
||||
# is published as the observation precision. Each sample is bounded by the
|
||||
# entry and depth caps below, and sampling ends at the first observed write.
|
||||
OBSERVER_INTERVAL_SECONDS = 0.02
|
||||
OBSERVER_JOIN_SECONDS = 10.0
|
||||
OBSERVER_MAX_ENTRIES = 4096
|
||||
OBSERVER_MAX_DEPTH = 16
|
||||
MAX_OBSERVATION_RECORDS = 1000
|
||||
DIGEST_PREFIX = "sha256:"
|
||||
_PATH_DIGEST_DOMAIN = b"iop-benchmark-workspace-path-v1\0"
|
||||
|
||||
|
||||
class MeasurementError(Exception):
|
||||
"""Raised when measurement evidence cannot be produced or trusted."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Observation:
|
||||
"""One required value that is either observed or explicitly unavailable."""
|
||||
|
||||
status: str
|
||||
value: Optional[int]
|
||||
unit: str
|
||||
clock: str
|
||||
source: str
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorkspaceWriteObservation:
|
||||
"""The bounded observer's report about the first observed workspace write."""
|
||||
|
||||
observed: bool
|
||||
monotonic_ns: Optional[int]
|
||||
mtime_ns: Optional[int]
|
||||
path_digest: str
|
||||
precision_ns: int
|
||||
samples: int
|
||||
reason: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorkspaceScan:
|
||||
"""One closed workspace snapshot; incomplete snapshots are never compared."""
|
||||
|
||||
files: dict[str, tuple[int, int, int]]
|
||||
status: str
|
||||
|
||||
@property
|
||||
def complete(self) -> bool:
|
||||
return self.status == "complete"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AttemptMeasurement:
|
||||
"""One immutable measurement record bound to exactly one attempt."""
|
||||
|
||||
run_id: str
|
||||
cell_id: str
|
||||
repetition: int
|
||||
attempt: int
|
||||
caller: str
|
||||
spec_digest: str
|
||||
product: ProductOutcome
|
||||
harness: HarnessOutcome
|
||||
process: ProcessOutcome
|
||||
timeline: dict[str, Observation]
|
||||
usage: dict[str, Observation]
|
||||
observer: WorkspaceWriteObservation
|
||||
observations: tuple[ParsedMetric, ...]
|
||||
|
||||
|
||||
def observed(value: int, unit: str, clock: str, source: str) -> Observation:
|
||||
"""Build one observed value with its exact unit, clock and source."""
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value < 0:
|
||||
raise MeasurementError("observed value must be a non-negative integer")
|
||||
return Observation(STATUS_OBSERVED, value, unit, clock, source, "")
|
||||
|
||||
|
||||
def unavailable(reason: str, source: str) -> Observation:
|
||||
"""Build one explicitly unavailable value; never substitute a zero."""
|
||||
if reason not in UNAVAILABLE_REASONS:
|
||||
raise MeasurementError("unavailable reason is not a closed value")
|
||||
return Observation(STATUS_UNAVAILABLE, None, "", "", source, reason)
|
||||
|
||||
|
||||
def observation_record(observation: Observation) -> dict[str, Any]:
|
||||
"""Return the canonical projection of one observed/unavailable value."""
|
||||
if observation.status == STATUS_OBSERVED:
|
||||
return {
|
||||
"status": STATUS_OBSERVED,
|
||||
"value": observation.value,
|
||||
"unit": observation.unit,
|
||||
"clock": observation.clock,
|
||||
"source": observation.source,
|
||||
}
|
||||
if observation.status != STATUS_UNAVAILABLE:
|
||||
raise MeasurementError("observation status is not a closed value")
|
||||
return {
|
||||
"status": STATUS_UNAVAILABLE,
|
||||
"value": None,
|
||||
"reason": observation.reason,
|
||||
"source": observation.source,
|
||||
}
|
||||
|
||||
|
||||
def _observation_from_record(raw: Any) -> Observation:
|
||||
if not isinstance(raw, dict):
|
||||
raise MeasurementError("observation is invalid")
|
||||
status = raw.get("status")
|
||||
if status == STATUS_OBSERVED:
|
||||
if set(raw) != {"status", "value", "unit", "clock", "source"}:
|
||||
raise MeasurementError("observed value schema is invalid")
|
||||
value, unit, clock = raw["value"], raw["unit"], raw["clock"]
|
||||
if (
|
||||
isinstance(value, bool) or not isinstance(value, int) or value < 0
|
||||
or unit not in OBSERVATION_UNITS
|
||||
or clock not in METRIC_CLOCKS or raw["source"] not in METRIC_SOURCES
|
||||
):
|
||||
raise MeasurementError("observed value is invalid")
|
||||
return Observation(STATUS_OBSERVED, value, unit, clock, raw["source"], "")
|
||||
if status == STATUS_UNAVAILABLE:
|
||||
if set(raw) != {"status", "value", "reason", "source"}:
|
||||
raise MeasurementError("unavailable value schema is invalid")
|
||||
if (
|
||||
raw["value"] is not None
|
||||
or raw["reason"] not in UNAVAILABLE_REASONS
|
||||
or raw["source"] not in METRIC_SOURCES
|
||||
):
|
||||
raise MeasurementError("unavailable value is invalid")
|
||||
return Observation(STATUS_UNAVAILABLE, None, "", "", raw["source"], raw["reason"])
|
||||
raise MeasurementError("observation status is not a closed value")
|
||||
|
||||
|
||||
def _is_digest(value: Any) -> bool:
|
||||
"""True for one exact lowercase sha256 identity string."""
|
||||
if not isinstance(value, str) or not value.startswith(DIGEST_PREFIX):
|
||||
return False
|
||||
body = value[len(DIGEST_PREFIX):]
|
||||
return len(body) == 64 and all(char in "0123456789abcdef" for char in body)
|
||||
|
||||
|
||||
def path_digest(relative_path: str) -> str:
|
||||
"""Digest one workspace-relative path so no caller-chosen name persists."""
|
||||
return DIGEST_PREFIX + hashlib.sha256(
|
||||
_PATH_DIGEST_DOMAIN + os.fsencode(relative_path)
|
||||
).hexdigest()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Bounded workspace write observer
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _scan_workspace(root: Path) -> WorkspaceScan:
|
||||
"""Snapshot contained regular files without following any link.
|
||||
|
||||
Every directory entry consumes one shared budget, including directories,
|
||||
links and non-regular files. A cap, depth or I/O boundary returns an
|
||||
incomplete result rather than a partial snapshot that could be compared.
|
||||
"""
|
||||
found: dict[str, tuple[int, int, int]] = {}
|
||||
pending: deque[tuple[Path, int]] = deque([(root, 0)])
|
||||
consumed = 0
|
||||
while pending:
|
||||
current, depth = pending.popleft()
|
||||
try:
|
||||
with os.scandir(current) as scan:
|
||||
for entry in scan:
|
||||
# Conservatively report exhaustion as soon as the bounded
|
||||
# budget has been consumed. This avoids reading one more
|
||||
# entry merely to distinguish an exactly-full directory.
|
||||
if consumed >= OBSERVER_MAX_ENTRIES:
|
||||
return WorkspaceScan(found, "exhausted")
|
||||
consumed += 1
|
||||
try:
|
||||
if entry.is_symlink():
|
||||
continue
|
||||
if entry.is_dir(follow_symlinks=False):
|
||||
if depth >= OBSERVER_MAX_DEPTH:
|
||||
return WorkspaceScan(found, "exhausted")
|
||||
pending.append((Path(entry.path), depth + 1))
|
||||
continue
|
||||
info = entry.stat(follow_symlinks=False)
|
||||
if not stat.S_ISREG(info.st_mode):
|
||||
continue
|
||||
relative = os.path.relpath(entry.path, root)
|
||||
except (OSError, ValueError):
|
||||
return WorkspaceScan(found, "unavailable")
|
||||
found[relative] = (info.st_mtime_ns, info.st_size, info.st_ino)
|
||||
except OSError:
|
||||
return WorkspaceScan(found, "unavailable")
|
||||
return WorkspaceScan(found, "complete")
|
||||
|
||||
|
||||
class WorkspaceWriteObserver:
|
||||
"""Sample one workspace at a bounded interval and keep the first write seen.
|
||||
|
||||
The observer starts before the caller is invoked so that its baseline is
|
||||
older than any caller write. It reports its own harness observation time,
|
||||
the filesystem modification time it read, its source and its polling
|
||||
precision. It never claims that a terminal snapshot proves the first write.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
root: str | Path,
|
||||
*,
|
||||
interval_seconds: float = OBSERVER_INTERVAL_SECONDS,
|
||||
clock: Callable[[], int] = time.monotonic_ns,
|
||||
) -> None:
|
||||
self.root = Path(root)
|
||||
if interval_seconds <= 0:
|
||||
raise MeasurementError("observer interval must be positive")
|
||||
self.interval_seconds = float(interval_seconds)
|
||||
self._clock = clock
|
||||
self._stop = threading.Event()
|
||||
self._thread: Optional[threading.Thread] = None
|
||||
self._baseline: dict[str, tuple[int, int, int]] = {}
|
||||
self._baseline_complete = False
|
||||
self._samples = 0
|
||||
self._first: Optional[tuple[int, int, str]] = None
|
||||
self._unavailable_reason = ""
|
||||
self._started = False
|
||||
self._stopped = False
|
||||
self._final_observation: Optional[WorkspaceWriteObservation] = None
|
||||
|
||||
@property
|
||||
def precision_ns(self) -> int:
|
||||
return int(self.interval_seconds * DURATION_NS_PER_SECOND)
|
||||
|
||||
@property
|
||||
def stopped(self) -> bool:
|
||||
"""True once the sampling thread has been joined and is gone."""
|
||||
return self._stopped
|
||||
|
||||
def start(self) -> None:
|
||||
"""Take the immutable baseline, then start the sampling thread."""
|
||||
if self._started:
|
||||
raise MeasurementError("observer has already started")
|
||||
if not self.root.is_dir() or self.root.is_symlink():
|
||||
raise MeasurementError("observer root must be an existing directory")
|
||||
self._started = True
|
||||
baseline = _scan_workspace(self.root)
|
||||
if not baseline.complete:
|
||||
self._unavailable_reason = REASON_OBSERVER_UNAVAILABLE
|
||||
return
|
||||
self._baseline = baseline.files
|
||||
self._baseline_complete = True
|
||||
self._thread = threading.Thread(target=self._sample_until_stopped, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def _sample_until_stopped(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
if self._sample_once():
|
||||
return
|
||||
self._stop.wait(self.interval_seconds)
|
||||
|
||||
def _sample_once(self) -> bool:
|
||||
"""Return True once the first created or changed file has been seen."""
|
||||
current = _scan_workspace(self.root)
|
||||
self._samples += 1
|
||||
if not current.complete:
|
||||
self._unavailable_reason = REASON_OBSERVER_UNAVAILABLE
|
||||
return True
|
||||
changed = sorted(
|
||||
(relative, info) for relative, info in current.files.items()
|
||||
if self._baseline.get(relative) != info
|
||||
)
|
||||
if not changed:
|
||||
return False
|
||||
relative, info = changed[0]
|
||||
# A detection instant is sampled only after the complete snapshot has
|
||||
# found the change; it never labels scan work as observation time.
|
||||
now = self._clock()
|
||||
self._first = (now, info[0], relative)
|
||||
return True
|
||||
|
||||
def stop(self) -> WorkspaceWriteObservation:
|
||||
"""Stop and join the sampling thread, then freeze the observation.
|
||||
|
||||
Cleanup never raises, so it is safe on every terminal path; a thread
|
||||
that refuses to leave is reported through :attr:`stopped` instead. A
|
||||
fully stopped result is frozen and returned unchanged by every later
|
||||
call, so a file written after shutdown can never become the
|
||||
invocation's first write. A join that times out is not frozen so a
|
||||
later call can retry cleanup once the sampler has exited.
|
||||
"""
|
||||
if self._final_observation is not None:
|
||||
return self._final_observation
|
||||
self._stop.set()
|
||||
thread = self._thread
|
||||
if thread is None:
|
||||
self._stopped = self._started
|
||||
else:
|
||||
thread.join(OBSERVER_JOIN_SECONDS)
|
||||
self._stopped = not thread.is_alive()
|
||||
if self._stopped:
|
||||
self._thread = None
|
||||
if (
|
||||
self._stopped
|
||||
and self._baseline_complete
|
||||
and self._first is None
|
||||
and not self._unavailable_reason
|
||||
):
|
||||
# The joined sampler cannot race this final bounded scan. It closes
|
||||
# the interval between its final poll and caller cleanup.
|
||||
self._sample_once()
|
||||
if self._first is None:
|
||||
observation = WorkspaceWriteObservation(
|
||||
False, None, None, "", self.precision_ns, self._samples,
|
||||
self._unavailable_reason or REASON_NOT_OBSERVED,
|
||||
)
|
||||
else:
|
||||
monotonic_ns, mtime_ns, relative = self._first
|
||||
observation = WorkspaceWriteObservation(
|
||||
True, monotonic_ns, mtime_ns, path_digest(relative),
|
||||
self.precision_ns, self._samples,
|
||||
)
|
||||
if self._stopped:
|
||||
self._final_observation = observation
|
||||
return observation
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Measurement construction
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _event_instant(result: InvocationResult, kind: str) -> Observation:
|
||||
for event in result.events:
|
||||
if event.kind == kind:
|
||||
return observed(
|
||||
event.monotonic_ns, UNIT_NANOSECONDS,
|
||||
CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS,
|
||||
)
|
||||
return unavailable(REASON_NOT_OBSERVED, SOURCE_HARNESS)
|
||||
|
||||
|
||||
def _write_instant(value: Any, clock: str, observation: WorkspaceWriteObservation) -> Observation:
|
||||
"""Report one observer value, or say plainly that it was not observed."""
|
||||
if (not observation.observed or isinstance(value, bool)
|
||||
or not isinstance(value, int) or value < 0):
|
||||
return unavailable(
|
||||
observation.reason or REASON_NOT_OBSERVED, SOURCE_WORKSPACE_POLL
|
||||
)
|
||||
return observed(value, UNIT_NANOSECONDS, clock, SOURCE_WORKSPACE_POLL)
|
||||
|
||||
|
||||
def _timeline(
|
||||
result: InvocationResult, observation: WorkspaceWriteObservation
|
||||
) -> dict[str, Observation]:
|
||||
"""Build the timeline without comparing values across clock domains."""
|
||||
duration = result.duration_ns
|
||||
return {
|
||||
"submitted_at": _event_instant(result, EVENT_SUBMITTED),
|
||||
"first_output_at": _event_instant(result, EVENT_FIRST_OUTPUT),
|
||||
"first_write_observed_at": _write_instant(
|
||||
observation.monotonic_ns, CLOCK_HARNESS_MONOTONIC, observation
|
||||
),
|
||||
# The filesystem clock is reported beside the harness clock and never
|
||||
# subtracted from it; a caller of this record cannot mix the two.
|
||||
"first_write_mtime": _write_instant(
|
||||
observation.mtime_ns, CLOCK_FILESYSTEM_MTIME, observation
|
||||
),
|
||||
"total_duration": (
|
||||
observed(duration, UNIT_NANOSECONDS, CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS)
|
||||
if isinstance(duration, int) and not isinstance(duration, bool) and duration >= 0
|
||||
else unavailable(REASON_NOT_OBSERVED, SOURCE_HARNESS)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _usage(metrics: tuple[ParsedMetric, ...]) -> dict[str, Observation]:
|
||||
"""Project only whole caller totals; never sum or split labelled intervals.
|
||||
|
||||
A stage or call label marks one part of a larger report, so only unstaged,
|
||||
uncalled observations can be a total. Two totals with identical labels are
|
||||
a contradiction and fail closed; two totals bound to different models are
|
||||
ambiguous and stay unavailable rather than being merged.
|
||||
"""
|
||||
totals: dict[str, list[ParsedMetric]] = {}
|
||||
for metric in metrics:
|
||||
if metric.stage or metric.call_id:
|
||||
continue
|
||||
candidates = totals.setdefault(metric.name, [])
|
||||
if any(item.model == metric.model for item in candidates):
|
||||
raise MeasurementError("caller reported a duplicate total")
|
||||
candidates.append(metric)
|
||||
usage: dict[str, Observation] = {}
|
||||
for name in METRIC_NAMES:
|
||||
candidates = totals.get(name, [])
|
||||
if len(candidates) == 1:
|
||||
metric = candidates[0]
|
||||
usage[name] = observed(metric.value, metric.unit, metric.clock, metric.source)
|
||||
elif candidates:
|
||||
usage[name] = unavailable(REASON_AMBIGUOUS_TOTAL, candidates[0].source)
|
||||
else:
|
||||
usage[name] = unavailable(REASON_NOT_REPORTED, SOURCE_HARNESS)
|
||||
return usage
|
||||
|
||||
|
||||
def build_measurement(
|
||||
*,
|
||||
run_id: str,
|
||||
cell_id: str,
|
||||
repetition: int,
|
||||
attempt: int,
|
||||
caller: str,
|
||||
result: InvocationResult,
|
||||
observation: WorkspaceWriteObservation,
|
||||
) -> AttemptMeasurement:
|
||||
"""Join lifecycle events, caller observations and the observer into one record."""
|
||||
if not isinstance(result, InvocationResult):
|
||||
raise MeasurementError("invocation result is invalid")
|
||||
if not isinstance(observation, WorkspaceWriteObservation):
|
||||
raise MeasurementError("workspace observation is invalid")
|
||||
if any(not isinstance(text, str) or not text for text in (run_id, cell_id, caller)):
|
||||
raise MeasurementError("measurement identity is invalid")
|
||||
if any(
|
||||
isinstance(number, bool) or not isinstance(number, int) or number < 1
|
||||
for number in (repetition, attempt)
|
||||
):
|
||||
raise MeasurementError("measurement identity is invalid")
|
||||
if not _is_digest(result.spec_digest):
|
||||
raise MeasurementError("measurement invocation identity is invalid")
|
||||
metrics = tuple(result.metrics)
|
||||
if len(metrics) > MAX_OBSERVATION_RECORDS:
|
||||
raise MeasurementError("observation count exceeds the bounded record")
|
||||
try:
|
||||
for metric in metrics:
|
||||
validate_metric(metric)
|
||||
except LifecycleMetricError as exc:
|
||||
raise MeasurementError("caller observation is invalid") from exc
|
||||
published = [
|
||||
event.kind for event in result.events
|
||||
if event.kind.startswith(METRIC_PREFIX)
|
||||
and event.kind[len(METRIC_PREFIX):] in METRIC_UNITS
|
||||
]
|
||||
if published != [METRIC_PREFIX + metric.name for metric in metrics]:
|
||||
raise MeasurementError("observation set does not match published events")
|
||||
return AttemptMeasurement(
|
||||
run_id=run_id,
|
||||
cell_id=cell_id,
|
||||
repetition=repetition,
|
||||
attempt=attempt,
|
||||
caller=caller,
|
||||
spec_digest=result.spec_digest,
|
||||
product=result.product,
|
||||
harness=result.harness,
|
||||
process=result.process,
|
||||
timeline=_timeline(result, observation),
|
||||
usage=_usage(metrics),
|
||||
observer=observation,
|
||||
observations=metrics,
|
||||
)
|
||||
|
||||
|
||||
def build_recovery_measurement(
|
||||
*,
|
||||
run_id: str,
|
||||
cell_id: str,
|
||||
repetition: int,
|
||||
attempt: int,
|
||||
caller: str,
|
||||
spec_digest: str,
|
||||
product: ProductOutcome,
|
||||
harness: HarnessOutcome,
|
||||
process: ProcessOutcome,
|
||||
) -> AttemptMeasurement:
|
||||
"""Build closed unavailable evidence for an authenticated receipt-only terminal.
|
||||
|
||||
A controller can disappear after its supervisor has durably cleaned up the
|
||||
caller but before it publishes lifecycle, observer, or measurement output.
|
||||
This constructor records only that absence; it never infers values from the
|
||||
cleanup receipt or the workspace.
|
||||
"""
|
||||
if any(not isinstance(text, str) or not text for text in (
|
||||
run_id, cell_id, caller, spec_digest,
|
||||
)) or not _is_digest(spec_digest):
|
||||
raise MeasurementError("measurement identity is invalid")
|
||||
if any(
|
||||
isinstance(number, bool) or not isinstance(number, int) or number < 1
|
||||
for number in (repetition, attempt)
|
||||
):
|
||||
raise MeasurementError("measurement identity is invalid")
|
||||
if not all(isinstance(value, expected) for value, expected in (
|
||||
(product, ProductOutcome),
|
||||
(harness, HarnessOutcome),
|
||||
(process, ProcessOutcome),
|
||||
)):
|
||||
raise MeasurementError("recovery outcomes are invalid")
|
||||
timeline = {
|
||||
name: unavailable(
|
||||
REASON_NOT_OBSERVED,
|
||||
SOURCE_WORKSPACE_POLL if name.startswith("first_write") else SOURCE_HARNESS,
|
||||
)
|
||||
for name in TIMELINE_NAMES
|
||||
}
|
||||
return AttemptMeasurement(
|
||||
run_id=run_id,
|
||||
cell_id=cell_id,
|
||||
repetition=repetition,
|
||||
attempt=attempt,
|
||||
caller=caller,
|
||||
spec_digest=spec_digest,
|
||||
product=product,
|
||||
harness=harness,
|
||||
process=process,
|
||||
timeline=timeline,
|
||||
usage={
|
||||
name: unavailable(REASON_NOT_REPORTED, SOURCE_HARNESS)
|
||||
for name in METRIC_NAMES
|
||||
},
|
||||
observer=WorkspaceWriteObservation(
|
||||
observed=False,
|
||||
monotonic_ns=None,
|
||||
mtime_ns=None,
|
||||
path_digest="",
|
||||
precision_ns=int(OBSERVER_INTERVAL_SECONDS * DURATION_NS_PER_SECOND),
|
||||
samples=0,
|
||||
reason=REASON_OBSERVER_UNAVAILABLE,
|
||||
),
|
||||
observations=(),
|
||||
)
|
||||
|
||||
|
||||
def measurement_record(measurement: AttemptMeasurement) -> dict[str, Any]:
|
||||
"""Return the canonical durable projection of one measurement."""
|
||||
if not isinstance(measurement, AttemptMeasurement):
|
||||
raise MeasurementError("measurement is invalid")
|
||||
observer = measurement.observer
|
||||
return {
|
||||
"record": MEASUREMENT_RECORD,
|
||||
"measurement_version": MEASUREMENT_VERSION,
|
||||
"attempt": {
|
||||
"run_id": measurement.run_id,
|
||||
"cell_id": measurement.cell_id,
|
||||
"repetition": measurement.repetition,
|
||||
"attempt": measurement.attempt,
|
||||
},
|
||||
"caller": measurement.caller,
|
||||
"spec_digest": measurement.spec_digest,
|
||||
"product": {
|
||||
"status": measurement.product.status,
|
||||
"reason": measurement.product.reason,
|
||||
},
|
||||
"harness": {
|
||||
"status": measurement.harness.status,
|
||||
"reason": measurement.harness.reason,
|
||||
"ordered_terminal": measurement.harness.ordered_terminal,
|
||||
"cleanup_complete": measurement.harness.cleanup_complete,
|
||||
},
|
||||
"process": {
|
||||
"status": measurement.process.status,
|
||||
"exit_code": measurement.process.exit_code,
|
||||
"signal": measurement.process.signal,
|
||||
},
|
||||
"timeline": {
|
||||
name: observation_record(measurement.timeline[name])
|
||||
for name in TIMELINE_NAMES
|
||||
},
|
||||
"usage": {
|
||||
name: observation_record(measurement.usage[name]) for name in METRIC_NAMES
|
||||
},
|
||||
"observer": {
|
||||
"source": SOURCE_WORKSPACE_POLL,
|
||||
"status": STATUS_OBSERVED if observer.observed else STATUS_UNAVAILABLE,
|
||||
"path_digest": observer.path_digest,
|
||||
"precision_ns": observer.precision_ns,
|
||||
"samples": observer.samples,
|
||||
"reason": "" if observer.observed else (
|
||||
observer.reason or REASON_NOT_OBSERVED
|
||||
),
|
||||
},
|
||||
"observations": [metric_record(metric) for metric in measurement.observations],
|
||||
}
|
||||
|
||||
|
||||
def measurement_bytes(measurement: AttemptMeasurement) -> bytes:
|
||||
"""Serialize one measurement into canonical, sorted, ASCII bytes."""
|
||||
return json.dumps(
|
||||
measurement_record(measurement), sort_keys=True, separators=(",", ":"),
|
||||
ensure_ascii=True,
|
||||
).encode("ascii") + b"\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Durable publication and strict loading
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def measurement_path(attempt_root: str | Path) -> Path:
|
||||
return Path(attempt_root) / MEASUREMENT_FILENAME
|
||||
|
||||
|
||||
def publish_measurement(
|
||||
attempt_root: str | Path, measurement: AttemptMeasurement
|
||||
) -> Path:
|
||||
"""Publish the sidecar once; a collision never mutates the prior bytes."""
|
||||
path = measurement_path(attempt_root)
|
||||
data = measurement_bytes(measurement)
|
||||
try:
|
||||
publish_bytes_no_replace(path, data)
|
||||
except OSError as exc:
|
||||
raise MeasurementError("measurement publication refused an existing target") from exc
|
||||
except Exception as exc: # lifecycle publication failure is never silent
|
||||
raise MeasurementError("measurement publication failed") from exc
|
||||
return path
|
||||
|
||||
|
||||
def _read_regular_bytes(path: Path) -> bytes:
|
||||
"""Read one durable file without following links or trusting its type."""
|
||||
try:
|
||||
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
||||
except OSError as exc:
|
||||
raise MeasurementError("measurement is unavailable") from exc
|
||||
try:
|
||||
if not stat.S_ISREG(os.fstat(fd).st_mode):
|
||||
raise MeasurementError("measurement must be a regular file")
|
||||
chunks: list[bytes] = []
|
||||
while True:
|
||||
chunk = os.read(fd, 1 << 20)
|
||||
if not chunk:
|
||||
return b"".join(chunks)
|
||||
chunks.append(chunk)
|
||||
finally:
|
||||
os.close(fd)
|
||||
|
||||
|
||||
def _metric_from_record(raw: Any) -> ParsedMetric:
|
||||
fields = {
|
||||
"name", "value", "unit", "clock", "source", "stage", "model", "call_id",
|
||||
"overlap",
|
||||
}
|
||||
if not isinstance(raw, dict) or set(raw) != fields:
|
||||
raise MeasurementError("observation schema is invalid")
|
||||
try:
|
||||
return validate_metric(ParsedMetric(
|
||||
raw["name"], raw["value"], raw["unit"], raw["clock"], raw["source"],
|
||||
raw["stage"], raw["model"], raw["call_id"], raw["overlap"],
|
||||
))
|
||||
except (LifecycleMetricError, TypeError) as exc:
|
||||
raise MeasurementError("observation is invalid") from exc
|
||||
|
||||
|
||||
def _observer_from_record(raw: Any) -> WorkspaceWriteObservation:
|
||||
fields = {"source", "status", "path_digest", "precision_ns", "samples", "reason"}
|
||||
if not isinstance(raw, dict) or set(raw) != fields:
|
||||
raise MeasurementError("observer schema is invalid")
|
||||
precision, samples = raw["precision_ns"], raw["samples"]
|
||||
if (
|
||||
raw["source"] != SOURCE_WORKSPACE_POLL
|
||||
or raw["status"] not in (STATUS_OBSERVED, STATUS_UNAVAILABLE)
|
||||
or isinstance(precision, bool) or not isinstance(precision, int) or precision <= 0
|
||||
or isinstance(samples, bool) or not isinstance(samples, int) or samples < 0
|
||||
or not isinstance(raw["path_digest"], str)
|
||||
or not isinstance(raw["reason"], str)
|
||||
):
|
||||
raise MeasurementError("observer record is invalid")
|
||||
seen = raw["status"] == STATUS_OBSERVED
|
||||
digest = raw["path_digest"]
|
||||
if (seen != bool(digest) or (digest and not _is_digest(digest))
|
||||
or (seen and raw["reason"])
|
||||
or (not seen and raw["reason"] not in UNAVAILABLE_REASONS)):
|
||||
raise MeasurementError("observer record is invalid")
|
||||
return WorkspaceWriteObservation(
|
||||
seen, None, None, digest, precision, samples, raw["reason"]
|
||||
)
|
||||
|
||||
|
||||
def _identity_from_record(raw: Any) -> tuple[str, str, int, int]:
|
||||
if not isinstance(raw, dict) or set(raw) != {
|
||||
"run_id", "cell_id", "repetition", "attempt"
|
||||
}:
|
||||
raise MeasurementError("attempt identity schema is invalid")
|
||||
repetition, attempt = raw["repetition"], raw["attempt"]
|
||||
if (
|
||||
not isinstance(raw["run_id"], str) or not raw["run_id"]
|
||||
or not isinstance(raw["cell_id"], str) or not raw["cell_id"]
|
||||
or isinstance(repetition, bool) or not isinstance(repetition, int) or repetition < 1
|
||||
or isinstance(attempt, bool) or not isinstance(attempt, int) or attempt < 1
|
||||
):
|
||||
raise MeasurementError("attempt identity is invalid")
|
||||
return raw["run_id"], raw["cell_id"], repetition, attempt
|
||||
|
||||
|
||||
def _observation_map(raw: Any, names: tuple[str, ...], label: str) -> dict[str, Observation]:
|
||||
if not isinstance(raw, dict) or set(raw) != set(names):
|
||||
raise MeasurementError(f"{label} schema is invalid")
|
||||
return {name: _observation_from_record(raw[name]) for name in names}
|
||||
|
||||
|
||||
def load_measurement(attempt_root: str | Path) -> AttemptMeasurement:
|
||||
"""Load one sidecar and revalidate every closed field before use."""
|
||||
raw_bytes = _read_regular_bytes(measurement_path(attempt_root))
|
||||
try:
|
||||
record = json.loads(raw_bytes.decode("ascii"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise MeasurementError("measurement is not canonical JSON") from exc
|
||||
fields = {
|
||||
"record", "measurement_version", "attempt", "caller", "spec_digest",
|
||||
"product", "harness", "process", "timeline", "usage", "observer",
|
||||
"observations",
|
||||
}
|
||||
if not isinstance(record, dict) or set(record) != fields:
|
||||
raise MeasurementError("measurement schema is invalid")
|
||||
if (
|
||||
record["record"] != MEASUREMENT_RECORD
|
||||
or record["measurement_version"] != MEASUREMENT_VERSION
|
||||
or not isinstance(record["caller"], str) or not record["caller"]
|
||||
or not isinstance(record["spec_digest"], str) or not _is_digest(record["spec_digest"])
|
||||
or not isinstance(record["observations"], list)
|
||||
):
|
||||
raise MeasurementError("measurement identity is invalid")
|
||||
run_id, cell_id, repetition, attempt = _identity_from_record(record["attempt"])
|
||||
try:
|
||||
product = ProductOutcome(**record["product"])
|
||||
harness = HarnessOutcome(**record["harness"])
|
||||
process = ProcessOutcome(**record["process"])
|
||||
except (TypeError, LifecycleValidationError, ValueError) as exc:
|
||||
raise MeasurementError("measurement outcomes are invalid") from exc
|
||||
measurement = AttemptMeasurement(
|
||||
run_id=run_id,
|
||||
cell_id=cell_id,
|
||||
repetition=repetition,
|
||||
attempt=attempt,
|
||||
caller=record["caller"],
|
||||
spec_digest=record["spec_digest"],
|
||||
product=product,
|
||||
harness=harness,
|
||||
process=process,
|
||||
timeline=_observation_map(record["timeline"], TIMELINE_NAMES, "timeline"),
|
||||
usage=_observation_map(record["usage"], METRIC_NAMES, "usage"),
|
||||
observer=_observer_from_record(record["observer"]),
|
||||
observations=tuple(_metric_from_record(item) for item in record["observations"]),
|
||||
)
|
||||
if len(measurement.observations) > MAX_OBSERVATION_RECORDS:
|
||||
raise MeasurementError("observation count exceeds the bounded record")
|
||||
if measurement_record(measurement) != record or measurement_bytes(measurement) != raw_bytes:
|
||||
raise MeasurementError("measurement is non-canonical")
|
||||
_require_derived_coherence(measurement)
|
||||
return measurement
|
||||
|
||||
|
||||
def _require_derived_coherence(measurement: AttemptMeasurement) -> None:
|
||||
"""Refuse any usage or write value that its own observations do not support."""
|
||||
if measurement.usage != _usage(measurement.observations):
|
||||
raise MeasurementError("usage does not match the recorded observations")
|
||||
semantics = {
|
||||
"submitted_at": (CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS),
|
||||
"first_output_at": (CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS),
|
||||
"first_write_observed_at": (CLOCK_HARNESS_MONOTONIC, SOURCE_WORKSPACE_POLL),
|
||||
"first_write_mtime": (CLOCK_FILESYSTEM_MTIME, SOURCE_WORKSPACE_POLL),
|
||||
"total_duration": (CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS),
|
||||
}
|
||||
for name, value in measurement.timeline.items():
|
||||
clock, source = semantics[name]
|
||||
if value.source != source:
|
||||
raise MeasurementError("timeline source does not match its evidence")
|
||||
if value.status == STATUS_OBSERVED and (
|
||||
value.unit != UNIT_NANOSECONDS or value.clock != clock
|
||||
):
|
||||
raise MeasurementError("timeline clock does not match its evidence")
|
||||
seen = measurement.observer.observed
|
||||
write = measurement.timeline["first_write_observed_at"].status
|
||||
mtime = measurement.timeline["first_write_mtime"].status
|
||||
if ((not seen and STATUS_OBSERVED in (write, mtime))
|
||||
or (seen and write != STATUS_OBSERVED)
|
||||
or (seen and mtime != STATUS_OBSERVED)):
|
||||
raise MeasurementError("workspace write does not match the observer")
|
||||
|
||||
|
||||
def validate_measurement_lifecycle_binding(
|
||||
measurement: AttemptMeasurement, lifecycle: Mapping[str, Any]
|
||||
) -> None:
|
||||
"""Bind a sidecar to the immutable lifecycle metric and timeline evidence.
|
||||
|
||||
The lifecycle result/journal validation belongs to ``RunStore``. This
|
||||
helper consumes that already-validated terminal projection and refuses a
|
||||
coherent sidecar rewrite whose observations or derived values no longer
|
||||
correspond to its immutable event stream.
|
||||
"""
|
||||
if not isinstance(lifecycle, Mapping):
|
||||
raise MeasurementError("lifecycle evidence is invalid")
|
||||
expected_outcomes = {
|
||||
"product": {
|
||||
"status": measurement.product.status,
|
||||
"reason": measurement.product.reason,
|
||||
},
|
||||
"harness": {
|
||||
"status": measurement.harness.status,
|
||||
"reason": measurement.harness.reason,
|
||||
"ordered_terminal": measurement.harness.ordered_terminal,
|
||||
"cleanup_complete": measurement.harness.cleanup_complete,
|
||||
},
|
||||
"process": {
|
||||
"status": measurement.process.status,
|
||||
"exit_code": measurement.process.exit_code,
|
||||
"signal": measurement.process.signal,
|
||||
},
|
||||
}
|
||||
if any(lifecycle.get(name) != value for name, value in expected_outcomes.items()):
|
||||
raise MeasurementError("measurement outcomes do not match lifecycle evidence")
|
||||
events = lifecycle.get("events")
|
||||
if not isinstance(events, list):
|
||||
raise MeasurementError("lifecycle events are invalid")
|
||||
|
||||
metric_events: list[ParsedMetric] = []
|
||||
instants: dict[str, int] = {}
|
||||
for event in events:
|
||||
if not isinstance(event, Mapping):
|
||||
raise MeasurementError("lifecycle event is invalid")
|
||||
kind, source = event.get("kind"), event.get("source")
|
||||
if kind in (EVENT_SUBMITTED, EVENT_FIRST_OUTPUT):
|
||||
value = event.get("monotonic_ns")
|
||||
if source != SOURCE_HARNESS or isinstance(value, bool) or not isinstance(value, int):
|
||||
raise MeasurementError("lifecycle timeline evidence is invalid")
|
||||
if kind in instants:
|
||||
raise MeasurementError("lifecycle timeline evidence is ambiguous")
|
||||
instants[kind] = value
|
||||
if not isinstance(kind, str) or not kind.startswith(METRIC_PREFIX):
|
||||
continue
|
||||
name = kind[len(METRIC_PREFIX):]
|
||||
try:
|
||||
raw = json.loads(str(event.get("detail", "")))
|
||||
metric = _metric_from_record(raw)
|
||||
except (json.JSONDecodeError, MeasurementError) as exc:
|
||||
raise MeasurementError("lifecycle metric evidence is invalid") from exc
|
||||
if metric.name != name or source != metric.source:
|
||||
raise MeasurementError("lifecycle metric evidence is invalid")
|
||||
metric_events.append(metric)
|
||||
|
||||
if tuple(metric_events) != measurement.observations:
|
||||
raise MeasurementError("measurement observations do not match lifecycle evidence")
|
||||
required_instants = {
|
||||
"submitted_at": EVENT_SUBMITTED,
|
||||
"first_output_at": EVENT_FIRST_OUTPUT,
|
||||
}
|
||||
for timeline_name, event_kind in required_instants.items():
|
||||
timeline = measurement.timeline[timeline_name]
|
||||
event_value = instants.get(event_kind)
|
||||
if event_value is None:
|
||||
if timeline.status != STATUS_UNAVAILABLE:
|
||||
raise MeasurementError("timeline claims a missing lifecycle event")
|
||||
elif timeline.status != STATUS_OBSERVED or timeline.value != event_value:
|
||||
raise MeasurementError("timeline does not match lifecycle evidence")
|
||||
duration = lifecycle.get("duration_ns")
|
||||
total = measurement.timeline["total_duration"]
|
||||
if isinstance(duration, bool) or not isinstance(duration, int) or duration < 0:
|
||||
raise MeasurementError("lifecycle duration is invalid")
|
||||
if total.status != STATUS_OBSERVED or total.value != duration:
|
||||
raise MeasurementError("timeline duration does not match lifecycle evidence")
|
||||
|
|
@ -1,686 +0,0 @@
|
|||
"""Credential-free tests for source-aware timing and usage evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import unittest
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import scripts.agent_benchmark.measurement as measurement_module
|
||||
|
||||
from scripts.agent_benchmark.lifecycle import (
|
||||
CLOCK_CALLER_REPORTED,
|
||||
CLOCK_HARNESS_MONOTONIC,
|
||||
CLOCK_NONE,
|
||||
EVENT_FIRST_OUTPUT,
|
||||
EVENT_SUBMITTED,
|
||||
METRIC_PREFIX,
|
||||
SOURCE_CALLER_OUTPUT,
|
||||
SOURCE_HARNESS,
|
||||
SOURCE_WORKSPACE_POLL,
|
||||
UNIT_NANOSECONDS,
|
||||
CaptureStream,
|
||||
ProductOutcome,
|
||||
HarnessOutcome,
|
||||
ProcessOutcome,
|
||||
InvocationResult,
|
||||
LifecycleMetricError,
|
||||
LifecycleEvent,
|
||||
ParsedMetric,
|
||||
count_metric,
|
||||
duration_metric,
|
||||
normalize_count,
|
||||
normalize_duration_ns,
|
||||
validate_metric,
|
||||
)
|
||||
from scripts.agent_benchmark.measurement import (
|
||||
MEASUREMENT_FILENAME,
|
||||
MeasurementError,
|
||||
WorkspaceWriteObservation,
|
||||
WorkspaceWriteObserver,
|
||||
WorkspaceScan,
|
||||
OBSERVER_MAX_ENTRIES,
|
||||
REASON_NOT_OBSERVED,
|
||||
REASON_OBSERVER_UNAVAILABLE,
|
||||
_scan_workspace,
|
||||
build_measurement,
|
||||
load_measurement,
|
||||
measurement_bytes,
|
||||
measurement_record,
|
||||
observation_record,
|
||||
observed,
|
||||
path_digest,
|
||||
publish_measurement,
|
||||
unavailable,
|
||||
)
|
||||
|
||||
|
||||
def _event(kind: str, monotonic_ns: int, source: str = SOURCE_HARNESS) -> LifecycleEvent:
|
||||
return LifecycleEvent(kind, source, "", monotonic_ns, 0, "2026-08-11T00:00:00+00:00", "")
|
||||
|
||||
|
||||
def _capture(stream: str) -> CaptureStream:
|
||||
return CaptureStream(stream, "", 0, 0, False)
|
||||
|
||||
|
||||
def _result(
|
||||
metrics: tuple[ParsedMetric, ...] = (),
|
||||
*,
|
||||
events: tuple[LifecycleEvent, ...] | None = None,
|
||||
duration_ns: int = 5_000,
|
||||
terminal_reason: str = "success",
|
||||
) -> InvocationResult:
|
||||
"""Build one frozen lifecycle projection with matching metric events."""
|
||||
if events is None:
|
||||
events = (_event(EVENT_SUBMITTED, 1_000), _event(EVENT_FIRST_OUTPUT, 2_000))
|
||||
published = tuple(
|
||||
_event(METRIC_PREFIX + metric.name, 3_000, metric.source) for metric in metrics
|
||||
)
|
||||
return InvocationResult(
|
||||
product=ProductOutcome(
|
||||
"succeeded" if terminal_reason == "success" else "unknown",
|
||||
"caller_success" if terminal_reason == "success" else "unavailable",
|
||||
),
|
||||
harness=HarnessOutcome(
|
||||
"passed" if terminal_reason == "success" else "failed",
|
||||
terminal_reason,
|
||||
True,
|
||||
True,
|
||||
),
|
||||
process=ProcessOutcome("exited", 0, None),
|
||||
submitted=True,
|
||||
process_group_alive=False,
|
||||
events=events + published,
|
||||
stdout=_capture("stdout"),
|
||||
stderr=_capture("stderr"),
|
||||
journal_path="",
|
||||
result_path="",
|
||||
locator=None,
|
||||
spec_digest="sha256:" + "a" * 64,
|
||||
started_at="2026-08-11T00:00:00+00:00",
|
||||
ended_at="2026-08-11T00:00:01+00:00",
|
||||
duration_ns=duration_ns,
|
||||
metrics=metrics,
|
||||
)
|
||||
|
||||
|
||||
def _observation() -> WorkspaceWriteObservation:
|
||||
"""One frozen observer report used by the record-level tests."""
|
||||
return WorkspaceWriteObservation(
|
||||
True, 4_000, 1_700_000_000_000_000_000, path_digest("out.txt"), 10_000_000, 3
|
||||
)
|
||||
|
||||
|
||||
class MetricContractTest(unittest.TestCase):
|
||||
def test_duration_decimals_normalize_losslessly_to_nanoseconds(self) -> None:
|
||||
for value, unit, expected in (
|
||||
(12, "ms", 12_000_000),
|
||||
(12.5, "ms", 12_500_000),
|
||||
("0.000001", "ms", 1),
|
||||
(Decimal("1.5"), "s", 1_500_000_000),
|
||||
(7, "us", 7_000),
|
||||
(0, "ms", 0),
|
||||
(9, "ns", 9),
|
||||
):
|
||||
with self.subTest(value=value, unit=unit):
|
||||
self.assertEqual(normalize_duration_ns(value, unit), expected)
|
||||
|
||||
def test_unrepresentable_and_non_numeric_durations_fail_closed(self) -> None:
|
||||
for value, unit in (
|
||||
(0.0000001, "ms"), # 0.1 ns cannot be represented without invention
|
||||
(Decimal("0.5"), "ns"),
|
||||
(-1, "ms"),
|
||||
(True, "ms"),
|
||||
(float("inf"), "ms"),
|
||||
("nan", "ms"),
|
||||
("not-a-number", "ms"),
|
||||
(None, "ms"),
|
||||
(12, "minutes"),
|
||||
):
|
||||
with self.subTest(value=value, unit=unit):
|
||||
with self.assertRaises(LifecycleMetricError):
|
||||
normalize_duration_ns(value, unit)
|
||||
|
||||
def test_counts_admit_only_non_negative_integers(self) -> None:
|
||||
self.assertEqual(normalize_count(0), 0)
|
||||
self.assertEqual(normalize_count(41), 41)
|
||||
for value in (1.5, 2.0, True, -1, "3", None, Decimal("4")):
|
||||
with self.subTest(value=value):
|
||||
with self.assertRaises(LifecycleMetricError):
|
||||
normalize_count(value)
|
||||
|
||||
def test_metric_vocabulary_clock_and_labels_are_closed(self) -> None:
|
||||
rejected = (
|
||||
ParsedMetric("unknown_metric", 1, UNIT_NANOSECONDS, CLOCK_CALLER_REPORTED, SOURCE_CALLER_OUTPUT),
|
||||
ParsedMetric("total_duration", 1, "tokens", CLOCK_CALLER_REPORTED, SOURCE_CALLER_OUTPUT),
|
||||
ParsedMetric("total_duration", 1, UNIT_NANOSECONDS, CLOCK_NONE, SOURCE_CALLER_OUTPUT),
|
||||
ParsedMetric("input_tokens", 1, "tokens", CLOCK_CALLER_REPORTED, SOURCE_CALLER_OUTPUT),
|
||||
ParsedMetric("input_tokens", 1, "tokens", CLOCK_NONE, SOURCE_CALLER_OUTPUT, overlap=True),
|
||||
ParsedMetric("input_tokens", 1, "tokens", CLOCK_NONE, "invented_source"),
|
||||
ParsedMetric("input_tokens", -1, "tokens", CLOCK_NONE, SOURCE_CALLER_OUTPUT),
|
||||
ParsedMetric("input_tokens", 1, "tokens", CLOCK_NONE, SOURCE_CALLER_OUTPUT, model="two words"),
|
||||
ParsedMetric("input_tokens", 1, "tokens", CLOCK_NONE, SOURCE_CALLER_OUTPUT, call_id="sk-abcdefgh12345"),
|
||||
"metric:total_duration",
|
||||
)
|
||||
for metric in rejected:
|
||||
with self.subTest(metric=metric):
|
||||
with self.assertRaises(LifecycleMetricError):
|
||||
validate_metric(metric)
|
||||
accepted = duration_metric(
|
||||
"tool_duration", 3, model="claude-sonnet", call_id="call-1", overlap=True
|
||||
)
|
||||
self.assertEqual(accepted.value, 3_000_000)
|
||||
self.assertEqual(accepted.clock, CLOCK_CALLER_REPORTED)
|
||||
|
||||
def test_observed_and_unavailable_projections_stay_distinct(self) -> None:
|
||||
self.assertEqual(
|
||||
observation_record(observed(7, UNIT_NANOSECONDS, CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS)),
|
||||
{
|
||||
"status": "observed", "value": 7, "unit": UNIT_NANOSECONDS,
|
||||
"clock": CLOCK_HARNESS_MONOTONIC, "source": SOURCE_HARNESS,
|
||||
},
|
||||
)
|
||||
# An unavailable value is explicitly null; it is never a zero.
|
||||
self.assertEqual(
|
||||
observation_record(unavailable("not_reported", SOURCE_CALLER_OUTPUT)),
|
||||
{
|
||||
"status": "unavailable", "value": None, "reason": "not_reported",
|
||||
"source": SOURCE_CALLER_OUTPUT,
|
||||
},
|
||||
)
|
||||
with self.assertRaises(MeasurementError):
|
||||
unavailable("because", SOURCE_HARNESS)
|
||||
with self.assertRaises(MeasurementError):
|
||||
observed(-1, UNIT_NANOSECONDS, CLOCK_HARNESS_MONOTONIC, SOURCE_HARNESS)
|
||||
|
||||
|
||||
class MeasurementRecordTest(unittest.TestCase):
|
||||
def _measurement(self, metrics: tuple[ParsedMetric, ...], **kwargs: Any):
|
||||
return build_measurement(
|
||||
run_id="run-20260811T000000Z-0123456789ab",
|
||||
cell_id="claude-direct",
|
||||
repetition=1,
|
||||
attempt=1,
|
||||
caller="claude",
|
||||
result=_result(metrics, **kwargs),
|
||||
observation=_observation(),
|
||||
)
|
||||
|
||||
def test_reported_totals_are_preserved_without_any_synthesis(self) -> None:
|
||||
metrics = (
|
||||
duration_metric("total_duration", 1000, model="claude-sonnet"),
|
||||
duration_metric("model_duration", 400, model="claude-sonnet", overlap=True),
|
||||
count_metric("input_tokens", 11, model="claude-sonnet"),
|
||||
count_metric("output_tokens", 22, model="claude-sonnet"),
|
||||
)
|
||||
usage = self._measurement(metrics).usage
|
||||
self.assertEqual(usage["total_duration"].value, 1_000_000_000)
|
||||
self.assertEqual(usage["model_duration"].value, 400_000_000)
|
||||
self.assertEqual(usage["input_tokens"].value, 11)
|
||||
# Nothing is added and nothing is subtracted: the unreported provider
|
||||
# total and the unreported queue time both stay unavailable.
|
||||
for name in ("total_tokens", "queue_duration", "tool_duration", "model_calls"):
|
||||
self.assertEqual(usage[name].status, "unavailable")
|
||||
self.assertIsNone(usage[name].value)
|
||||
self.assertEqual(usage[name].reason, "not_reported")
|
||||
|
||||
def test_overlapping_and_labelled_intervals_never_become_totals(self) -> None:
|
||||
metrics = (
|
||||
duration_metric("tool_duration", 30, call_id="call-1", overlap=True),
|
||||
duration_metric("tool_duration", 70, call_id="call-2", overlap=True),
|
||||
)
|
||||
measurement = self._measurement(metrics)
|
||||
self.assertEqual(measurement.usage["tool_duration"].status, "unavailable")
|
||||
self.assertEqual(
|
||||
[(item.call_id, item.value, item.overlap) for item in measurement.observations],
|
||||
[("call-1", 30_000_000, True), ("call-2", 70_000_000, True)],
|
||||
)
|
||||
|
||||
def test_duplicate_total_fails_closed(self) -> None:
|
||||
metrics = (
|
||||
duration_metric("total_duration", 10, model="claude-sonnet"),
|
||||
duration_metric("total_duration", 20, model="claude-sonnet"),
|
||||
)
|
||||
with self.assertRaises(MeasurementError):
|
||||
self._measurement(metrics)
|
||||
|
||||
def test_two_model_totals_stay_ambiguous_instead_of_merging(self) -> None:
|
||||
metrics = (
|
||||
duration_metric("total_duration", 10, model="claude-sonnet"),
|
||||
duration_metric("total_duration", 20, model="gemini-2.0-flash"),
|
||||
)
|
||||
measurement = self._measurement(metrics)
|
||||
self.assertEqual(measurement.usage["total_duration"].status, "unavailable")
|
||||
self.assertEqual(measurement.usage["total_duration"].reason, "ambiguous_total")
|
||||
self.assertEqual(
|
||||
[(item.model, item.value) for item in measurement.observations],
|
||||
[("claude-sonnet", 10_000_000), ("gemini-2.0-flash", 20_000_000)],
|
||||
)
|
||||
|
||||
def test_timeline_keeps_every_clock_domain_separate(self) -> None:
|
||||
measurement = self._measurement(())
|
||||
timeline = measurement.timeline
|
||||
self.assertEqual(timeline["submitted_at"].value, 1_000)
|
||||
self.assertEqual(timeline["submitted_at"].clock, CLOCK_HARNESS_MONOTONIC)
|
||||
self.assertEqual(timeline["first_output_at"].value, 2_000)
|
||||
self.assertEqual(timeline["first_output_at"].source, SOURCE_HARNESS)
|
||||
# The observer's own clock and the filesystem clock are reported as two
|
||||
# separate values, so nothing can subtract one from the other.
|
||||
self.assertEqual(timeline["first_write_observed_at"].clock, CLOCK_HARNESS_MONOTONIC)
|
||||
self.assertEqual(timeline["first_write_observed_at"].source, SOURCE_WORKSPACE_POLL)
|
||||
self.assertEqual(timeline["first_write_mtime"].clock, "filesystem_mtime")
|
||||
self.assertEqual(timeline["total_duration"].value, 5_000)
|
||||
|
||||
def test_missing_first_output_is_unavailable_rather_than_zero(self) -> None:
|
||||
measurement = self._measurement((), events=(_event(EVENT_SUBMITTED, 1_000),))
|
||||
self.assertEqual(measurement.timeline["first_output_at"].status, "unavailable")
|
||||
self.assertIsNone(measurement.timeline["first_output_at"].value)
|
||||
self.assertEqual(measurement.timeline["first_output_at"].reason, "not_observed")
|
||||
|
||||
def test_incomplete_identity_is_refused_before_any_publication(self) -> None:
|
||||
base = {
|
||||
"run_id": "run-20260811T000000Z-0123456789ab", "cell_id": "claude-direct",
|
||||
"repetition": 1, "attempt": 1, "caller": "claude",
|
||||
"result": _result(), "observation": _observation(),
|
||||
}
|
||||
for override in (
|
||||
{"run_id": ""}, {"cell_id": None}, {"caller": ""},
|
||||
{"repetition": 0}, {"attempt": True},
|
||||
{"result": object()},
|
||||
{"observation": None},
|
||||
):
|
||||
with self.subTest(override=tuple(override)):
|
||||
with self.assertRaises(MeasurementError):
|
||||
build_measurement(**{**base, **override})
|
||||
|
||||
def test_observation_set_must_match_published_events(self) -> None:
|
||||
result = _result((count_metric("input_tokens", 1),))
|
||||
forged = ParsedMetric("output_tokens", 5, "tokens", CLOCK_NONE, SOURCE_CALLER_OUTPUT)
|
||||
with self.assertRaises(MeasurementError):
|
||||
build_measurement(
|
||||
run_id="run-20260811T000000Z-0123456789ab", cell_id="c", repetition=1,
|
||||
attempt=1, caller="claude",
|
||||
result=InvocationResult(**{**result.__dict__, "metrics": (*result.metrics, forged)}),
|
||||
observation=_observation(),
|
||||
)
|
||||
|
||||
|
||||
class WorkspaceObserverTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temp = tempfile.TemporaryDirectory(dir="/tmp", prefix="measurement-")
|
||||
self.root = Path(self.temp.name)
|
||||
self.workspace = self.root / "workspace"
|
||||
self.workspace.mkdir()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def _write(self, name: str, text: str) -> Path:
|
||||
path = self.workspace / name
|
||||
path.write_text(text, encoding="utf-8")
|
||||
return path
|
||||
|
||||
@staticmethod
|
||||
def _wait_observed(observer: WorkspaceWriteObserver) -> None:
|
||||
deadline = time.monotonic() + 10
|
||||
while observer._first is None and time.monotonic() < deadline:
|
||||
time.sleep(0.005)
|
||||
|
||||
def test_first_observed_write_is_reported_with_clock_source_and_precision(self) -> None:
|
||||
observer = WorkspaceWriteObserver(
|
||||
self.workspace, interval_seconds=0.005,
|
||||
)
|
||||
observer.start()
|
||||
self._write("out.txt", "x")
|
||||
self._wait_observed(observer)
|
||||
observation = observer.stop()
|
||||
self.assertTrue(observer.stopped)
|
||||
self.assertTrue(observation.observed)
|
||||
self.assertEqual(observation.path_digest, path_digest("out.txt"))
|
||||
self.assertEqual(observation.precision_ns, 5_000_000)
|
||||
self.assertGreaterEqual(observation.samples, 2)
|
||||
self.assertEqual(
|
||||
observation.mtime_ns, (self.workspace / "out.txt").stat().st_mtime_ns
|
||||
)
|
||||
|
||||
def test_first_observed_order_survives_a_later_final_mtime(self) -> None:
|
||||
observer = WorkspaceWriteObserver(
|
||||
self.workspace, interval_seconds=0.005,
|
||||
)
|
||||
observer.start()
|
||||
self._write("b.txt", "1")
|
||||
self._wait_observed(observer)
|
||||
observation = observer.stop()
|
||||
# After the observation is frozen, a second file is created and the
|
||||
# first file is rewritten, so the final snapshot now orders a.txt
|
||||
# before b.txt. The observer still reports the write it actually saw.
|
||||
later = self._write("a.txt", "2")
|
||||
os.utime(later, ns=(2_000_000_000_000_000_000, 2_000_000_000_000_000_000))
|
||||
rewritten = self._write("b.txt", "3")
|
||||
os.utime(rewritten, ns=(3_000_000_000_000_000_000, 3_000_000_000_000_000_000))
|
||||
snapshot = sorted(
|
||||
(item.stat().st_mtime_ns, item.name) for item in self.workspace.iterdir()
|
||||
)
|
||||
self.assertEqual(snapshot[0][1], "a.txt")
|
||||
self.assertEqual(observation.path_digest, path_digest("b.txt"))
|
||||
|
||||
def test_exhausted_scan_is_unavailable_and_never_compared(self) -> None:
|
||||
for index in range(OBSERVER_MAX_ENTRIES + 1):
|
||||
(self.workspace / f"empty-{index:04d}").mkdir()
|
||||
baseline = _scan_workspace(self.workspace)
|
||||
self.assertEqual(baseline.status, "exhausted")
|
||||
self.assertLessEqual(len(baseline.files), OBSERVER_MAX_ENTRIES)
|
||||
observer = WorkspaceWriteObserver(self.workspace, interval_seconds=0.005)
|
||||
observer.start()
|
||||
observation = observer.stop()
|
||||
self.assertFalse(observation.observed)
|
||||
self.assertEqual(observation.reason, REASON_OBSERVER_UNAVAILABLE)
|
||||
self.assertTrue(observer.stopped)
|
||||
|
||||
def test_detection_clock_runs_after_the_complete_scan(self) -> None:
|
||||
order: list[str] = []
|
||||
observer = WorkspaceWriteObserver(
|
||||
self.workspace, interval_seconds=0.005,
|
||||
clock=lambda: (order.append("clock") or 123),
|
||||
)
|
||||
observer._baseline = {}
|
||||
original = _scan_workspace
|
||||
|
||||
def scan(_root: Path) -> WorkspaceScan:
|
||||
order.append("scan")
|
||||
return WorkspaceScan({"out.txt": (1, 1, 1)}, "complete")
|
||||
|
||||
try:
|
||||
import scripts.agent_benchmark.measurement as measurement_module
|
||||
measurement_module._scan_workspace = scan
|
||||
self.assertTrue(observer._sample_once())
|
||||
finally:
|
||||
measurement_module._scan_workspace = original
|
||||
self.assertEqual(order, ["scan", "clock"])
|
||||
|
||||
def test_symlink_and_non_regular_entries_are_never_observed(self) -> None:
|
||||
outside = self.root / "outside.txt"
|
||||
outside.write_text("outside", encoding="utf-8")
|
||||
os.symlink(outside, self.workspace / "link.txt")
|
||||
os.mkfifo(self.workspace / "pipe")
|
||||
observer = WorkspaceWriteObserver(self.workspace, interval_seconds=0.005)
|
||||
observer.start()
|
||||
outside.write_text("changed outside", encoding="utf-8")
|
||||
observation = observer.stop()
|
||||
self.assertFalse(observation.observed)
|
||||
self.assertEqual(observation.path_digest, "")
|
||||
|
||||
def test_no_write_is_unavailable_and_leaves_no_thread(self) -> None:
|
||||
before = set(threading.enumerate())
|
||||
observer = WorkspaceWriteObserver(self.workspace, interval_seconds=0.005)
|
||||
observer.start()
|
||||
observation = observer.stop()
|
||||
self.assertFalse(observation.observed)
|
||||
self.assertIsNone(observation.monotonic_ns)
|
||||
self.assertTrue(observer.stopped)
|
||||
self.assertEqual(set(threading.enumerate()) - before, set())
|
||||
with self.assertRaises(MeasurementError):
|
||||
observer.start()
|
||||
|
||||
def test_stop_closes_the_final_interval_with_one_scan_after_sampler_exit(self) -> None:
|
||||
calls: list[str] = []
|
||||
sampled = threading.Event()
|
||||
snapshots = iter((
|
||||
WorkspaceScan({}, "complete"),
|
||||
WorkspaceScan({}, "complete"),
|
||||
WorkspaceScan({"out.txt": (1, 1, 1)}, "complete"),
|
||||
))
|
||||
original = measurement_module._scan_workspace
|
||||
|
||||
def scan(_root: Path) -> WorkspaceScan:
|
||||
calls.append("scan")
|
||||
result = next(snapshots)
|
||||
if len(calls) == 2:
|
||||
sampled.set()
|
||||
return result
|
||||
|
||||
try:
|
||||
measurement_module._scan_workspace = scan
|
||||
observer = WorkspaceWriteObserver(
|
||||
self.workspace, interval_seconds=60,
|
||||
clock=lambda: (calls.append("clock") or 123),
|
||||
)
|
||||
observer.start()
|
||||
self.assertTrue(sampled.wait(5))
|
||||
observation = observer.stop()
|
||||
finally:
|
||||
measurement_module._scan_workspace = original
|
||||
self.assertTrue(observer.stopped)
|
||||
self.assertTrue(observation.observed)
|
||||
self.assertEqual(observation.samples, 2)
|
||||
self.assertEqual(calls, ["scan", "scan", "scan", "clock"])
|
||||
|
||||
def test_final_exhausted_scan_is_unavailable_and_thread_is_cleaned_up(self) -> None:
|
||||
sampled = threading.Event()
|
||||
calls = 0
|
||||
snapshots = iter((
|
||||
WorkspaceScan({}, "complete"),
|
||||
WorkspaceScan({}, "complete"),
|
||||
WorkspaceScan({}, "exhausted"),
|
||||
))
|
||||
original = measurement_module._scan_workspace
|
||||
|
||||
def scan(_root: Path) -> WorkspaceScan:
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
result = next(snapshots)
|
||||
if calls == 2:
|
||||
sampled.set()
|
||||
return result
|
||||
|
||||
try:
|
||||
measurement_module._scan_workspace = scan
|
||||
observer = WorkspaceWriteObserver(self.workspace, interval_seconds=60)
|
||||
observer.start()
|
||||
self.assertTrue(sampled.wait(5))
|
||||
observation = observer.stop()
|
||||
finally:
|
||||
measurement_module._scan_workspace = original
|
||||
self.assertTrue(observer.stopped)
|
||||
self.assertFalse(observation.observed)
|
||||
self.assertEqual(observation.reason, REASON_OBSERVER_UNAVAILABLE)
|
||||
self.assertEqual(observation.samples, 2)
|
||||
self.assertEqual(calls, 3)
|
||||
|
||||
def test_successful_stop_is_idempotent_and_excludes_post_stop_writes(self) -> None:
|
||||
sampled = threading.Event()
|
||||
calls = 0
|
||||
original = measurement_module._scan_workspace
|
||||
|
||||
def scan(root: Path) -> WorkspaceScan:
|
||||
nonlocal calls
|
||||
calls += 1
|
||||
result = original(root)
|
||||
# Signal once the background sampler has completed its own empty
|
||||
# scan so the main thread stops a started, no-write observer.
|
||||
if calls == 2:
|
||||
sampled.set()
|
||||
return result
|
||||
|
||||
try:
|
||||
measurement_module._scan_workspace = scan
|
||||
observer = WorkspaceWriteObserver(self.workspace, interval_seconds=60)
|
||||
observer.start()
|
||||
self.assertTrue(sampled.wait(5))
|
||||
first = observer.stop()
|
||||
self._write("post-stop.txt", "after shutdown")
|
||||
second = observer.stop()
|
||||
finally:
|
||||
measurement_module._scan_workspace = original
|
||||
self.assertTrue(observer.stopped)
|
||||
self.assertIs(second, first)
|
||||
self.assertFalse(first.observed)
|
||||
self.assertEqual(first.reason, REASON_NOT_OBSERVED)
|
||||
self.assertEqual(second.samples, first.samples)
|
||||
# Baseline, the background sampler's empty scan, and the one final scan
|
||||
# account for every scan; the cached second stop performs no fourth scan.
|
||||
self.assertEqual(calls, 3)
|
||||
self.assertIsNone(observer._thread)
|
||||
|
||||
def test_observer_requires_a_real_directory_and_positive_interval(self) -> None:
|
||||
with self.assertRaises(MeasurementError):
|
||||
WorkspaceWriteObserver(self.workspace, interval_seconds=0)
|
||||
with self.assertRaises(MeasurementError):
|
||||
WorkspaceWriteObserver(self.root / "missing").start()
|
||||
os.symlink(self.workspace, self.root / "alias")
|
||||
with self.assertRaises(MeasurementError):
|
||||
WorkspaceWriteObserver(self.root / "alias").start()
|
||||
|
||||
|
||||
class MeasurementSidecarTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temp = tempfile.TemporaryDirectory(dir="/tmp", prefix="measurement-")
|
||||
self.root = Path(self.temp.name)
|
||||
self.measurement = build_measurement(
|
||||
run_id="run-20260811T000000Z-0123456789ab",
|
||||
cell_id="claude-direct",
|
||||
repetition=1,
|
||||
attempt=1,
|
||||
caller="claude",
|
||||
result=_result((
|
||||
duration_metric("total_duration", 1000, model="claude-sonnet"),
|
||||
count_metric("input_tokens", 11, model="claude-sonnet"),
|
||||
)),
|
||||
observation=_observation(),
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.temp.cleanup()
|
||||
|
||||
def test_publish_then_load_round_trips_every_closed_field(self) -> None:
|
||||
path = publish_measurement(self.root, self.measurement)
|
||||
self.assertEqual(path.name, MEASUREMENT_FILENAME)
|
||||
loaded = load_measurement(self.root)
|
||||
self.assertEqual(measurement_record(loaded), measurement_record(self.measurement))
|
||||
self.assertEqual(loaded.caller, "claude")
|
||||
self.assertEqual(loaded.usage["input_tokens"].value, 11)
|
||||
self.assertEqual(loaded.observations[0].name, "total_duration")
|
||||
self.assertEqual(loaded.observer.path_digest, path_digest("out.txt"))
|
||||
self.assertEqual(path.stat().st_mode & 0o777, 0o600)
|
||||
|
||||
def test_publication_is_no_clobber_and_preserves_prior_bytes(self) -> None:
|
||||
prior = b'{"record":"prior"}\n'
|
||||
(self.root / MEASUREMENT_FILENAME).write_bytes(prior)
|
||||
with self.assertRaises(MeasurementError):
|
||||
publish_measurement(self.root, self.measurement)
|
||||
self.assertEqual((self.root / MEASUREMENT_FILENAME).read_bytes(), prior)
|
||||
|
||||
def test_tampered_and_non_canonical_records_fail_closed(self) -> None:
|
||||
record = measurement_record(self.measurement)
|
||||
cases = {
|
||||
"unknown-field": {**record, "extra": 1},
|
||||
"wrong-version": {**record, "measurement_version": 1},
|
||||
"forged-digest": {**record, "spec_digest": "sha256:not-a-digest"},
|
||||
"zeroed-unavailable": {
|
||||
**record,
|
||||
"usage": {
|
||||
**record["usage"],
|
||||
"total_tokens": {
|
||||
"status": "unavailable", "value": 0,
|
||||
"reason": "not_reported", "source": "harness",
|
||||
},
|
||||
},
|
||||
},
|
||||
"invented-clock": {
|
||||
**record,
|
||||
"timeline": {
|
||||
**record["timeline"],
|
||||
"submitted_at": {
|
||||
**record["timeline"]["submitted_at"], "clock": "wall_clock",
|
||||
},
|
||||
},
|
||||
},
|
||||
"non-temporal-instant": {
|
||||
**record,
|
||||
"timeline": {
|
||||
**record["timeline"],
|
||||
"submitted_at": {
|
||||
**record["timeline"]["submitted_at"], "clock": "none",
|
||||
},
|
||||
},
|
||||
},
|
||||
"wrong-timeline-source": {
|
||||
**record,
|
||||
"timeline": {
|
||||
**record["timeline"],
|
||||
"first_write_observed_at": {
|
||||
**record["timeline"]["first_write_observed_at"],
|
||||
"source": SOURCE_HARNESS,
|
||||
},
|
||||
},
|
||||
},
|
||||
"invented-usage-total": {
|
||||
**record,
|
||||
"usage": {
|
||||
**record["usage"],
|
||||
"total_tokens": {
|
||||
"status": "observed", "value": 11, "unit": "tokens",
|
||||
"clock": "none", "source": "caller_output",
|
||||
},
|
||||
},
|
||||
},
|
||||
"observer-contradiction": {
|
||||
**record,
|
||||
"observer": {
|
||||
**record["observer"], "status": "unavailable", "path_digest": "",
|
||||
},
|
||||
},
|
||||
"unbound-observation": {
|
||||
**record,
|
||||
"observations": [
|
||||
{**record["observations"][0], "name": "unknown_metric"}
|
||||
],
|
||||
},
|
||||
}
|
||||
for name, payload in cases.items():
|
||||
with self.subTest(name=name):
|
||||
target = self.root / name
|
||||
target.mkdir()
|
||||
(target / MEASUREMENT_FILENAME).write_bytes(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode() + b"\n"
|
||||
)
|
||||
with self.assertRaises(MeasurementError):
|
||||
load_measurement(target)
|
||||
|
||||
def test_reordered_bytes_are_rejected_as_non_canonical(self) -> None:
|
||||
(self.root / MEASUREMENT_FILENAME).write_bytes(
|
||||
json.dumps(measurement_record(self.measurement), indent=2).encode() + b"\n"
|
||||
)
|
||||
with self.assertRaises(MeasurementError):
|
||||
load_measurement(self.root)
|
||||
|
||||
def test_missing_symlinked_and_non_regular_sidecars_fail_closed(self) -> None:
|
||||
with self.assertRaises(MeasurementError):
|
||||
load_measurement(self.root)
|
||||
publish_measurement(self.root, self.measurement)
|
||||
aliased = self.root / "aliased"
|
||||
aliased.mkdir()
|
||||
os.symlink(self.root / MEASUREMENT_FILENAME, aliased / MEASUREMENT_FILENAME)
|
||||
with self.assertRaises(MeasurementError):
|
||||
load_measurement(aliased)
|
||||
piped = self.root / "piped"
|
||||
piped.mkdir()
|
||||
os.mkfifo(piped / MEASUREMENT_FILENAME)
|
||||
with self.assertRaises(MeasurementError):
|
||||
load_measurement(piped)
|
||||
|
||||
def test_canonical_bytes_are_stable_and_ascii(self) -> None:
|
||||
first = measurement_bytes(self.measurement)
|
||||
self.assertEqual(first, measurement_bytes(self.measurement))
|
||||
first.decode("ascii")
|
||||
self.assertTrue(first.endswith(b"\n"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,544 +0,0 @@
|
|||
"""Deterministic, fail-closed Markdown reporting for benchmark evidence.
|
||||
|
||||
This module is intentionally a reader of the immutable run tree. It does not
|
||||
retry execution or scoring, derive values that producers did not record, or
|
||||
write anything except the idempotent ``report.md`` artifact after every input
|
||||
has passed its owning strict loader.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from scripts.agent_benchmark import scoring as _scoring
|
||||
from scripts.agent_benchmark.attempts import (
|
||||
Attempt,
|
||||
AttemptStateError,
|
||||
RunIdentity,
|
||||
RunStore,
|
||||
_connectivity_result_from_payload,
|
||||
)
|
||||
from scripts.agent_benchmark.manifest import Manifest, MatrixCell
|
||||
from scripts.agent_benchmark.measurement import (
|
||||
METRIC_NAMES,
|
||||
TIMELINE_NAMES,
|
||||
AttemptMeasurement,
|
||||
MeasurementError,
|
||||
Observation,
|
||||
load_measurement,
|
||||
)
|
||||
from scripts.agent_benchmark.web_validation import (
|
||||
WebValidation,
|
||||
WebValidationError,
|
||||
load_web_validation,
|
||||
)
|
||||
from scripts.agent_benchmark.rubric import validate_worksheet
|
||||
|
||||
REPORT_FILENAME = "report.md"
|
||||
_MAX_REPORT_BYTES = 2 * 1024 * 1024
|
||||
|
||||
|
||||
class ReportError(Exception):
|
||||
"""The immutable evidence cannot safely be projected into a report."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CategoryProjection:
|
||||
id: str
|
||||
score: int
|
||||
max_score: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ScoreProjection:
|
||||
status: str
|
||||
categories: tuple[CategoryProjection, ...]
|
||||
total: int | None
|
||||
rank: int | None
|
||||
reasons: tuple[str, ...]
|
||||
score_id: str | None
|
||||
evaluator: tuple[str, str, str, str] | None
|
||||
raw_paths: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AttemptProjection:
|
||||
cell: MatrixCell
|
||||
attempt: Attempt
|
||||
measurement: AttemptMeasurement | None
|
||||
web: WebValidation | None
|
||||
score: ScoreProjection
|
||||
raw_paths: tuple[str, ...]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReportProjection:
|
||||
run: RunIdentity
|
||||
manifest: Manifest
|
||||
preflights: tuple[dict[str, Any], ...]
|
||||
attempts: tuple[AttemptProjection, ...]
|
||||
|
||||
|
||||
def _markdown(value: object) -> str:
|
||||
"""Escape one table cell without allowing a value to alter Markdown shape."""
|
||||
return str(value).replace("\\", "\\\\").replace("|", "\\|").replace(
|
||||
"\r", " ").replace("\n", " ")
|
||||
|
||||
|
||||
def _relative_path(value: str) -> str:
|
||||
if not isinstance(value, str) or not value or "\\" in value or ":" in value:
|
||||
raise ReportError("report evidence path is invalid")
|
||||
path = PurePosixPath(value)
|
||||
if path.is_absolute() or str(path) != value or any(
|
||||
part in ("", ".", "..") for part in path.parts
|
||||
):
|
||||
raise ReportError("report evidence path is invalid")
|
||||
return value
|
||||
|
||||
|
||||
def _regular_under(root: Path, relative: str) -> None:
|
||||
"""Require a contained regular file and reject every symlink component."""
|
||||
relative = _relative_path(relative)
|
||||
current = root
|
||||
try:
|
||||
root_info = os.lstat(root)
|
||||
except OSError as exc:
|
||||
raise ReportError("report run root is unavailable") from exc
|
||||
if not stat.S_ISDIR(root_info.st_mode) or stat.S_ISLNK(root_info.st_mode):
|
||||
raise ReportError("report run root is invalid")
|
||||
for component in PurePosixPath(relative).parts:
|
||||
current = current / component
|
||||
try:
|
||||
info = os.lstat(current)
|
||||
except OSError as exc:
|
||||
raise ReportError("report evidence is unavailable") from exc
|
||||
if stat.S_ISLNK(info.st_mode):
|
||||
raise ReportError("report evidence path is invalid")
|
||||
if not stat.S_ISREG(info.st_mode):
|
||||
raise ReportError("report evidence must be a regular file")
|
||||
try:
|
||||
current.resolve().relative_to(root.resolve())
|
||||
except ValueError as exc:
|
||||
raise ReportError("report evidence escapes run root") from exc
|
||||
|
||||
|
||||
def _raw_link(path: str) -> str:
|
||||
path = _relative_path(path)
|
||||
return f"[raw](<{path.replace('>', '%3E')}>)"
|
||||
|
||||
|
||||
def _attempt_label(item: AttemptProjection) -> str:
|
||||
"""Render the complete durable identity used by detailed report rows."""
|
||||
identity = item.attempt.identity
|
||||
return f"{item.cell.id}/r{identity.repetition}/a{identity.attempt}"
|
||||
|
||||
|
||||
def _observation(value: Observation) -> str:
|
||||
if value.status == "observed":
|
||||
return (
|
||||
f"{value.value} {value.unit}; clock={value.clock}; "
|
||||
f"source={value.source}"
|
||||
)
|
||||
return f"unavailable; reason={value.reason}; source={value.source}"
|
||||
|
||||
|
||||
def _observation_group(values: dict[str, Observation]) -> str:
|
||||
"""Keep every named value while coalescing identical unavailable causes."""
|
||||
groups: dict[tuple[str, str], list[str]] = {}
|
||||
rendered: list[str] = []
|
||||
for name, value in values.items():
|
||||
if value.status == "observed":
|
||||
rendered.append(f"{name}={_observation(value)}")
|
||||
else:
|
||||
groups.setdefault((value.reason, value.source), []).append(name)
|
||||
for (reason, source), names in groups.items():
|
||||
rendered.append(
|
||||
f"{','.join(names)}=unavailable; reason={reason}; source={source}"
|
||||
)
|
||||
return "; ".join(rendered)
|
||||
|
||||
|
||||
def _read_canonical_json(path: Path, label: str) -> dict[str, Any]:
|
||||
"""Use scoring's no-follow reader and reject non-canonical JSON records."""
|
||||
try:
|
||||
value = _scoring._load_canonical(path, label)
|
||||
except Exception as exc:
|
||||
raise ReportError("report scoring evidence is unavailable") from exc
|
||||
if not isinstance(value, dict):
|
||||
raise ReportError("report scoring evidence is invalid")
|
||||
return value
|
||||
|
||||
|
||||
def _score_projection(
|
||||
run: RunIdentity, manifest: Manifest, attempt: Attempt, *, blocked: bool
|
||||
) -> ScoreProjection:
|
||||
"""Strictly load the terminal score state without allocating or recovering."""
|
||||
root = Path(attempt.root)
|
||||
raw_paths: list[str] = []
|
||||
try:
|
||||
score_root = _scoring._score_root(attempt, create=False)
|
||||
score_dirs = _scoring._score_dirs(score_root)
|
||||
is_unscored = _scoring._validate_unscored(run, manifest, attempt)
|
||||
if is_unscored:
|
||||
if score_dirs:
|
||||
raise ReportError("report scoring state is invalid")
|
||||
raw_paths.append(
|
||||
f"cells/{attempt.identity.cell_id}/repetition-"
|
||||
f"{attempt.identity.repetition:04d}/attempt-"
|
||||
f"{attempt.identity.attempt:06d}/scoring/unscored.json"
|
||||
)
|
||||
record = _read_canonical_json(root / "scoring" / "unscored.json", "unscored evidence")
|
||||
reasons = record.get("reasons")
|
||||
if not isinstance(reasons, list) or not all(isinstance(item, str) for item in reasons):
|
||||
raise ReportError("report unscored evidence is invalid")
|
||||
return ScoreProjection(
|
||||
"unscored", (), None, None, tuple(reasons), None, None,
|
||||
tuple(raw_paths),
|
||||
)
|
||||
|
||||
if not score_dirs:
|
||||
return ScoreProjection(
|
||||
"blocked" if blocked else "unavailable", (), None, None,
|
||||
("evaluator_preflight_blocked",) if blocked else ("not_recorded",),
|
||||
None, None, (),
|
||||
)
|
||||
|
||||
statuses: list[str] = []
|
||||
allocations: list[dict[str, Any]] = []
|
||||
for score_root in score_dirs:
|
||||
status = _scoring._result_status(score_root, run, manifest, attempt)
|
||||
if status is None:
|
||||
raise ReportError("report scoring result is incomplete")
|
||||
statuses.append(status)
|
||||
allocations.append(
|
||||
_scoring._validate_allocation(
|
||||
score_root / "allocation.json", run, manifest, attempt, score_root.name
|
||||
)
|
||||
)
|
||||
prefix = (
|
||||
f"cells/{attempt.identity.cell_id}/repetition-"
|
||||
f"{attempt.identity.repetition:04d}/attempt-"
|
||||
f"{attempt.identity.attempt:06d}/scoring/{score_root.name}"
|
||||
)
|
||||
raw_paths.extend((f"{prefix}/allocation.json", f"{prefix}/result.json"))
|
||||
|
||||
if "scored" in statuses:
|
||||
if statuses[-1] != "scored" or statuses.count("scored") != 1:
|
||||
raise ReportError("report scoring state is invalid")
|
||||
score_root = score_dirs[-1]
|
||||
result = _read_canonical_json(score_root / "result.json", "scoring result")
|
||||
try:
|
||||
worksheet = validate_worksheet(result.get("worksheet"))
|
||||
except Exception as exc:
|
||||
raise ReportError("report worksheet is invalid") from exc
|
||||
categories = tuple(
|
||||
CategoryProjection(item.id, item.score, item.max_score)
|
||||
for item in worksheet.categories
|
||||
)
|
||||
if not categories:
|
||||
raise ReportError("report worksheet is invalid")
|
||||
evaluator = allocations[-1]["evaluator"]
|
||||
binding = (
|
||||
evaluator["caller"], evaluator["route_id"],
|
||||
evaluator["request_model"], evaluator["requested_effort"],
|
||||
)
|
||||
return ScoreProjection(
|
||||
"scored", categories, worksheet.total, None, (), score_root.name,
|
||||
binding, tuple(raw_paths),
|
||||
)
|
||||
|
||||
if any(status != "scoring_failed" for status in statuses):
|
||||
raise ReportError("report scoring state is invalid")
|
||||
result = _read_canonical_json(score_dirs[-1] / "result.json", "scoring result")
|
||||
reason = result.get("reason")
|
||||
if not isinstance(reason, str) or not reason:
|
||||
raise ReportError("report scoring failure is invalid")
|
||||
evaluator = allocations[-1]["evaluator"]
|
||||
binding = (
|
||||
evaluator["caller"], evaluator["route_id"],
|
||||
evaluator["request_model"], evaluator["requested_effort"],
|
||||
)
|
||||
return ScoreProjection(
|
||||
"scoring_failed", (), None, None, (reason,), score_dirs[-1].name,
|
||||
binding, tuple(raw_paths),
|
||||
)
|
||||
except ReportError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ReportError("report scoring evidence is invalid") from exc
|
||||
|
||||
|
||||
def _scoring_preflight_blocked(run: RunIdentity, manifest: Manifest) -> bool:
|
||||
"""Validate evaluator preflights and report whether the latest one blocked."""
|
||||
root = Path(run.root) / "scoring-preflight"
|
||||
if not root.exists() and not root.is_symlink():
|
||||
return False
|
||||
try:
|
||||
if root.is_symlink() or not root.is_dir():
|
||||
raise ReportError("report scoring preflight is invalid")
|
||||
evaluator = _scoring._evaluator_cell(manifest)
|
||||
statuses: list[str] = []
|
||||
for expected, path in enumerate(sorted(root.iterdir()), start=1):
|
||||
if path.name != f"preflight-{expected:06d}.json":
|
||||
raise ReportError("report scoring preflight sequence is invalid")
|
||||
payload = _read_canonical_json(path, "evaluator preflight")
|
||||
result, _, _ = _connectivity_result_from_payload(payload, evaluator)
|
||||
statuses.append(result.status)
|
||||
return bool(statuses and statuses[-1] != "ready")
|
||||
except ReportError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise ReportError("report scoring preflight is invalid") from exc
|
||||
|
||||
|
||||
def _rank(attempts: list[AttemptProjection]) -> tuple[AttemptProjection, ...]:
|
||||
"""Assign competition ranks without using display order as a tie-breaker."""
|
||||
scored = sorted(
|
||||
(item for item in attempts if item.score.status == "scored"),
|
||||
key=lambda item: -int(item.score.total),
|
||||
)
|
||||
ranks: dict[tuple[str, int, int], int] = {}
|
||||
previous: int | None = None
|
||||
for index, item in enumerate(scored, start=1):
|
||||
total = int(item.score.total)
|
||||
if total != previous:
|
||||
rank = index
|
||||
previous = total
|
||||
ranks[(item.attempt.identity.cell_id, item.attempt.identity.repetition, item.attempt.identity.attempt)] = rank
|
||||
result: list[AttemptProjection] = []
|
||||
for item in attempts:
|
||||
key = (item.attempt.identity.cell_id, item.attempt.identity.repetition, item.attempt.identity.attempt)
|
||||
score = item.score
|
||||
result.append(AttemptProjection(
|
||||
item.cell, item.attempt, item.measurement, item.web,
|
||||
ScoreProjection(
|
||||
score.status, score.categories, score.total, ranks.get(key),
|
||||
score.reasons, score.score_id, score.evaluator, score.raw_paths,
|
||||
),
|
||||
item.raw_paths,
|
||||
))
|
||||
return tuple(result)
|
||||
|
||||
|
||||
def project_report(store: RunStore, run: RunIdentity, manifest: Manifest) -> ReportProjection:
|
||||
"""Read every report input in stable manifest/slot/attempt order."""
|
||||
try:
|
||||
bound = store.open(manifest, run.run_id)
|
||||
if bound != run:
|
||||
raise ReportError("report run identity is invalid")
|
||||
preflights = store.preflights(bound, manifest)
|
||||
blocked = _scoring_preflight_blocked(bound, manifest)
|
||||
cells = {cell.id: cell for cell in manifest.matrix}
|
||||
rows: list[AttemptProjection] = []
|
||||
for attempt in store.execution_attempts(bound, manifest):
|
||||
cell = cells.get(attempt.identity.cell_id)
|
||||
if cell is None:
|
||||
raise ReportError("report attempt cell is invalid")
|
||||
measurement = None
|
||||
web = None
|
||||
raw_paths = [
|
||||
f"cells/{attempt.identity.cell_id}/repetition-"
|
||||
f"{attempt.identity.repetition:04d}/attempt-"
|
||||
f"{attempt.identity.attempt:06d}/attempt.json",
|
||||
]
|
||||
if attempt.state != "running":
|
||||
measurement = load_measurement(attempt.root)
|
||||
if (
|
||||
measurement.run_id != bound.run_id
|
||||
or measurement.cell_id != attempt.identity.cell_id
|
||||
or measurement.repetition != attempt.identity.repetition
|
||||
or measurement.attempt != attempt.identity.attempt
|
||||
or measurement.caller != cell.caller
|
||||
):
|
||||
raise ReportError("report measurement identity is invalid")
|
||||
web = load_web_validation(attempt.root, manifest=manifest)
|
||||
if web.record["attempt"] != {
|
||||
"run_id": bound.run_id, "cell_id": attempt.identity.cell_id,
|
||||
"repetition": attempt.identity.repetition, "attempt": attempt.identity.attempt,
|
||||
}:
|
||||
raise ReportError("report web evidence identity is invalid")
|
||||
raw_paths.extend((
|
||||
f"cells/{attempt.identity.cell_id}/repetition-{attempt.identity.repetition:04d}/attempt-{attempt.identity.attempt:06d}/attempt-measurement.json",
|
||||
f"cells/{attempt.identity.cell_id}/repetition-{attempt.identity.repetition:04d}/attempt-{attempt.identity.attempt:06d}/web-validation.json",
|
||||
))
|
||||
score = _score_projection(bound, manifest, attempt, blocked=blocked)
|
||||
rows.append(AttemptProjection(cell, attempt, measurement, web, score, tuple(raw_paths)))
|
||||
return ReportProjection(bound, manifest, preflights, _rank(rows))
|
||||
except ReportError:
|
||||
raise
|
||||
except (AttemptStateError, MeasurementError, WebValidationError) as exc:
|
||||
raise ReportError("report evidence is invalid") from exc
|
||||
except Exception as exc:
|
||||
raise ReportError("report evidence is unavailable") from exc
|
||||
|
||||
|
||||
def render_report(projection: ReportProjection) -> bytes:
|
||||
"""Render one fixed-order UTF-8/LF Markdown projection."""
|
||||
root = Path(projection.run.root)
|
||||
raw_paths = {"manifest.json", "run.json"}
|
||||
scoring_preflight = root / "scoring-preflight"
|
||||
if scoring_preflight.exists() or scoring_preflight.is_symlink():
|
||||
# project_report already performed the schema validation; retain the
|
||||
# exact immutable preflight bytes as the blocked/ready score pointer.
|
||||
for expected, path in enumerate(sorted(scoring_preflight.iterdir()), start=1):
|
||||
if path.name != f"preflight-{expected:06d}.json":
|
||||
raise ReportError("report scoring preflight sequence is invalid")
|
||||
raw_paths.add(f"scoring-preflight/{path.name}")
|
||||
lines = [
|
||||
"# Agent comparison benchmark report",
|
||||
"",
|
||||
"## Run identity",
|
||||
"",
|
||||
"| field | value |",
|
||||
"|---|---|",
|
||||
f"| run_id | {_markdown(projection.run.run_id)} |",
|
||||
f"| manifest_digest | {_markdown(projection.run.manifest_digest)} |",
|
||||
f"| pipeline_version | {_markdown(projection.manifest.pipeline_version)} |",
|
||||
"",
|
||||
"## Immutable conditions",
|
||||
"",
|
||||
"| field | value |",
|
||||
"|---|---|",
|
||||
f"| environment | {_markdown(projection.manifest.environment)} |",
|
||||
f"| fixture | {_markdown(projection.manifest.fixture.version)} ({_markdown(projection.manifest.fixture.checksum)}) |",
|
||||
f"| rubric | {_markdown(projection.manifest.rubric_version)} |",
|
||||
f"| session_policy | {_markdown(projection.manifest.session_policy)} |",
|
||||
f"| setup_cache_policy | {_markdown(projection.manifest.setup_cache_policy)} |",
|
||||
f"| evaluator | {_markdown(projection.manifest.evaluator.caller)}/{_markdown(projection.manifest.evaluator.iop.request_model)}/{_markdown(projection.manifest.evaluator.iop.requested_effort)} |",
|
||||
"",
|
||||
"## Execution preflight",
|
||||
"",
|
||||
"| sequence | status | results |",
|
||||
"|---:|---|---:|",
|
||||
]
|
||||
if projection.preflights:
|
||||
for item in projection.preflights:
|
||||
path = f"preflight/preflight-{item['sequence']:06d}.json"
|
||||
raw_paths.add(path)
|
||||
lines.append(f"| {item['sequence']} | {_markdown(item['status'])} | {len(item['results'])} |")
|
||||
else:
|
||||
lines.append("| — | unavailable | 0 |")
|
||||
|
||||
lines.extend(("", "## Attempt outcomes", "", "| cell | repetition | attempt | controller | product | harness | process | artifact | scoring | total | rank |", "|---|---:|---:|---|---|---|---|---|---|---:|---:|"))
|
||||
if not projection.attempts:
|
||||
lines.append("| — | — | — | blocked | unavailable | unavailable | unavailable | unavailable | unavailable | — | — |")
|
||||
for item in projection.attempts:
|
||||
product = "unavailable" if item.measurement is None else item.measurement.product.status
|
||||
harness = "unavailable" if item.measurement is None else item.measurement.harness.status
|
||||
process = "unavailable" if item.measurement is None else item.measurement.process.status
|
||||
artifact = "unavailable" if item.web is None else item.web.status
|
||||
total = "—" if item.score.total is None else str(item.score.total)
|
||||
rank = "—" if item.score.rank is None else str(item.score.rank)
|
||||
lines.append(
|
||||
f"| {_markdown(item.cell.id)} | {item.attempt.identity.repetition} | {item.attempt.identity.attempt} | "
|
||||
f"{_markdown(item.attempt.state)} | {_markdown(product)} | {_markdown(harness)} | "
|
||||
f"{_markdown(process)} | {_markdown(artifact)} | "
|
||||
f"{_markdown(item.score.status)} | {total} | {rank} |"
|
||||
)
|
||||
raw_paths.update(item.raw_paths)
|
||||
raw_paths.update(item.score.raw_paths)
|
||||
|
||||
lines.extend((
|
||||
"", "## Quality score breakdown", "",
|
||||
"| cell/repetition/attempt | category | score | max |",
|
||||
"|---|---|---:|---:|",
|
||||
))
|
||||
wrote_categories = False
|
||||
for item in projection.attempts:
|
||||
label = _attempt_label(item)
|
||||
for category in item.score.categories:
|
||||
lines.append(
|
||||
f"| {_markdown(label)} | {_markdown(category.id)} | "
|
||||
f"{category.score} | {category.max_score} |"
|
||||
)
|
||||
wrote_categories = True
|
||||
if not wrote_categories:
|
||||
lines.append("| — | unavailable | — | — |")
|
||||
|
||||
lines.extend(("", "## Timing and token evidence", "", "| cell/repetition/attempt | time observations | token observations |", "|---|---|---|"))
|
||||
wrote_metrics = False
|
||||
for item in projection.attempts:
|
||||
if item.measurement is None:
|
||||
continue
|
||||
label = _attempt_label(item)
|
||||
timeline = _observation_group(item.measurement.timeline)
|
||||
usage = _observation_group(item.measurement.usage)
|
||||
lines.append(f"| {_markdown(label)} | {_markdown(timeline)} | {_markdown(usage)} |")
|
||||
wrote_metrics = True
|
||||
if not wrote_metrics:
|
||||
lines.append("| — | unavailable; reason=not_recorded | unavailable; reason=not_recorded |")
|
||||
|
||||
lines.extend(("", "## Web validation and scoring provenance", "", "| cell/repetition/attempt | web gates | screenshots | score_id | evaluator | scoring condition |", "|---|---|---|---|---|---|"))
|
||||
if not projection.attempts:
|
||||
lines.append("| — | unavailable | unavailable | — | — | blocked |")
|
||||
for item in projection.attempts:
|
||||
gates, shots = "unavailable", "unavailable"
|
||||
if item.web is not None:
|
||||
gates = ", ".join(
|
||||
f"{gate['id']}={'pass' if gate['passed'] else 'fail'}"
|
||||
for gate in item.web.record["gates"]
|
||||
)
|
||||
shots = ", ".join(
|
||||
str(view["screenshot"]["file"])
|
||||
for view in item.web.record["viewports"]
|
||||
) or "unavailable"
|
||||
evaluator = "unavailable" if item.score.evaluator is None else "/".join(item.score.evaluator)
|
||||
condition = ", ".join(item.score.reasons) or "recorded"
|
||||
lines.append(
|
||||
f"| {_markdown(_attempt_label(item))} | {_markdown(gates)} | {_markdown(shots)} | "
|
||||
f"{_markdown(item.score.score_id or '—')} | {_markdown(evaluator)} | {_markdown(condition)} |"
|
||||
)
|
||||
|
||||
lines.extend(("", "## Limitations", "", "- Values marked `unavailable` retain the producing source and reason; they are not inferred as zero.", "- Automatic web gates establish eligibility only and contribute no quality points.", "- Equal scored totals share a competition rank; unscored and scoring-failed attempts do not receive a rank.", "", "## Raw evidence index", "", "| contained pointer |", "|---|"))
|
||||
for path in sorted(raw_paths):
|
||||
_regular_under(root, path)
|
||||
lines.append(f"| {_raw_link(path)} |")
|
||||
lines.append("")
|
||||
return "\n".join(lines).encode("utf-8")
|
||||
|
||||
|
||||
def publish_report(store: RunStore, run: RunIdentity, manifest: Manifest) -> Path:
|
||||
"""Render and idempotently publish the sole run-owned report artifact."""
|
||||
data = render_report(project_report(store, run, manifest))
|
||||
if len(data) > _MAX_REPORT_BYTES:
|
||||
raise ReportError("report exceeds the bounded artifact size")
|
||||
root = Path(run.root)
|
||||
path = root / REPORT_FILENAME
|
||||
try:
|
||||
info = os.lstat(path)
|
||||
except FileNotFoundError:
|
||||
info = None
|
||||
except OSError as exc:
|
||||
raise ReportError("report publication is unavailable") from exc
|
||||
if info is not None:
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||
raise ReportError("report publication target is invalid")
|
||||
try:
|
||||
existing = path.read_bytes()
|
||||
except OSError as exc:
|
||||
raise ReportError("report publication is unavailable") from exc
|
||||
if existing != data:
|
||||
raise ReportError("report publication refused an existing target")
|
||||
return path
|
||||
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_CLOEXEC
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
try:
|
||||
descriptor = os.open(path, flags, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as handle:
|
||||
handle.write(data)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
directory = os.open(root, os.O_RDONLY | os.O_DIRECTORY | os.O_CLOEXEC)
|
||||
try:
|
||||
os.fsync(directory)
|
||||
finally:
|
||||
os.close(directory)
|
||||
except OSError as exc:
|
||||
raise ReportError("report publication failed") from exc
|
||||
return path
|
||||
|
|
@ -1,303 +0,0 @@
|
|||
"""Regression coverage for the strict benchmark report projection."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import io
|
||||
import tempfile
|
||||
import unittest
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
import scripts.agent_comparison_benchmark as cli_module
|
||||
from scripts.agent_benchmark import scoring as scoring_module
|
||||
from scripts.agent_benchmark.reporting import ReportError, project_report, publish_report, render_report
|
||||
from scripts.agent_benchmark import scoring_test
|
||||
|
||||
|
||||
class ReportingTest(unittest.TestCase):
|
||||
"""Use the S13 synthetic run builder to cover all S14 terminal states."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self.fixture = Path(
|
||||
"scripts/fixtures/agent-comparison-benchmark-report.expected.md"
|
||||
)
|
||||
self.harness = scoring_test.ScoringTest()
|
||||
self.harness.setUp()
|
||||
self.addCleanup(self.harness.doCleanups)
|
||||
|
||||
def _all_status_run(self) -> None:
|
||||
# The RunStore retains every immutable attempt. Scoring sees the latest
|
||||
# slot attempt on each invocation, leaving a representative S14 history.
|
||||
self.harness._attempt()
|
||||
scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(),
|
||||
)
|
||||
self.harness._attempt()
|
||||
scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(),
|
||||
)
|
||||
self.harness._attempt("failed")
|
||||
scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(),
|
||||
)
|
||||
self.harness._attempt()
|
||||
scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(modes=["malformed"]),
|
||||
)
|
||||
self.harness._attempt()
|
||||
summary = scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(blocked=True),
|
||||
)
|
||||
self.assertEqual(summary.blocked, 1)
|
||||
|
||||
def test_all_status_tie_projection_matches_golden(self) -> None:
|
||||
self._all_status_run()
|
||||
projection = project_report(
|
||||
self.harness.store, self.harness.run, self.harness.manifest
|
||||
)
|
||||
self.assertEqual(
|
||||
[item.score.status for item in projection.attempts],
|
||||
["scored", "scored", "unscored", "scoring_failed", "blocked"],
|
||||
)
|
||||
self.assertEqual(
|
||||
[item.score.rank for item in projection.attempts], [1, 1, None, None, None]
|
||||
)
|
||||
self.assertEqual(
|
||||
[
|
||||
tuple((category.id, category.score, category.max_score)
|
||||
for category in item.score.categories)
|
||||
for item in projection.attempts
|
||||
],
|
||||
[
|
||||
(
|
||||
("task_fidelity", 24, 25),
|
||||
("visual_hierarchy", 25, 25),
|
||||
("responsive_composition", 20, 20),
|
||||
("typography_readability", 15, 15),
|
||||
("polish_consistency", 15, 15),
|
||||
),
|
||||
(
|
||||
("task_fidelity", 24, 25),
|
||||
("visual_hierarchy", 25, 25),
|
||||
("responsive_composition", 20, 20),
|
||||
("typography_readability", 15, 15),
|
||||
("polish_consistency", 15, 15),
|
||||
),
|
||||
(),
|
||||
(),
|
||||
(),
|
||||
],
|
||||
)
|
||||
self.assertEqual(
|
||||
render_report(projection), self.fixture.read_bytes()
|
||||
)
|
||||
|
||||
def test_detail_tables_include_repetition_in_attempt_label(self) -> None:
|
||||
self._all_status_run()
|
||||
projection = project_report(
|
||||
self.harness.store, self.harness.run, self.harness.manifest
|
||||
)
|
||||
first = projection.attempts[0]
|
||||
repeated = replace(
|
||||
first,
|
||||
attempt=replace(
|
||||
first.attempt,
|
||||
identity=replace(first.attempt.identity, repetition=2),
|
||||
),
|
||||
)
|
||||
rendered = render_report(
|
||||
replace(projection, attempts=(first, repeated))
|
||||
).decode("utf-8")
|
||||
|
||||
for section, next_section in (
|
||||
("## Quality score breakdown", "## Timing and token evidence"),
|
||||
("## Timing and token evidence", "## Web validation and scoring provenance"),
|
||||
("## Web validation and scoring provenance", "## Limitations"),
|
||||
):
|
||||
table = rendered.split(section, 1)[1].split(next_section, 1)[0]
|
||||
self.assertIn("| cell-sentinel/r1/a1 |", table)
|
||||
self.assertIn("| cell-sentinel/r2/a1 |", table)
|
||||
|
||||
def test_unscored_and_score_directory_conflict_is_rejected(self) -> None:
|
||||
attempt = self.harness._attempt("failed")
|
||||
scoring_module.score_run(
|
||||
self.harness.store, self.harness.run, self.harness.manifest,
|
||||
adapter=scoring_test.FakeScoringAdapter(),
|
||||
)
|
||||
(Path(attempt.root) / "scoring" / "score-000001").mkdir()
|
||||
with self.assertRaisesRegex(ReportError, "report scoring state is invalid"):
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest)
|
||||
self.assertFalse((Path(self.harness.run.root) / "report.md").exists())
|
||||
|
||||
def test_publication_is_idempotent_and_refuses_replacement(self) -> None:
|
||||
self._all_status_run()
|
||||
path = publish_report(
|
||||
self.harness.store, self.harness.run, self.harness.manifest
|
||||
)
|
||||
expected = self.fixture.read_bytes()
|
||||
self.assertEqual(path.read_bytes(), expected)
|
||||
self.assertEqual(
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest),
|
||||
path,
|
||||
)
|
||||
path.write_bytes(expected + b"changed\n")
|
||||
with self.assertRaisesRegex(ReportError, "refused an existing target"):
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest)
|
||||
self.assertEqual(path.read_bytes(), expected + b"changed\n")
|
||||
|
||||
def test_corrupt_required_measurement_creates_no_report(self) -> None:
|
||||
attempt = self.harness._attempt()
|
||||
measurement = Path(attempt.root) / "attempt-measurement.json"
|
||||
measurement.write_bytes(b"{}\n")
|
||||
with self.assertRaises(ReportError):
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest)
|
||||
self.assertFalse((Path(self.harness.run.root) / "report.md").exists())
|
||||
|
||||
def test_symlinked_required_evidence_is_not_a_contained_raw_link(self) -> None:
|
||||
attempt = self.harness._attempt()
|
||||
measurement = Path(attempt.root) / "attempt-measurement.json"
|
||||
measurement.unlink()
|
||||
measurement.symlink_to(Path(self.harness.run.root) / "manifest.json")
|
||||
with self.assertRaises(ReportError):
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest)
|
||||
self.assertFalse((Path(self.harness.run.root) / "report.md").exists())
|
||||
|
||||
def test_report_target_symlink_is_rejected(self) -> None:
|
||||
self.harness._attempt()
|
||||
path = Path(self.harness.run.root) / "report.md"
|
||||
path.symlink_to("manifest.json")
|
||||
with self.assertRaisesRegex(ReportError, "publication target is invalid"):
|
||||
publish_report(self.harness.store, self.harness.run, self.harness.manifest)
|
||||
|
||||
def test_markdown_cell_escaping_is_stable(self) -> None:
|
||||
from scripts.agent_benchmark.reporting import _markdown
|
||||
|
||||
self.assertEqual(_markdown("line|next\\tail\nlast"), "line\\|next\\\\tail last")
|
||||
|
||||
|
||||
class ReportCliTest(unittest.TestCase):
|
||||
"""Deterministic boundary coverage for the public ``report`` CLI handler."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
self.manifest_path = Path(self._tmp.name) / "manifest.json"
|
||||
self.manifest_bytes = b"{\"version\": \"example\"}\n"
|
||||
self.manifest_path.write_bytes(self.manifest_bytes)
|
||||
self.report_path = cli_module._REPO_ROOT / "runs" / "example" / "report.md"
|
||||
self.report_rel = "runs/example/report.md"
|
||||
|
||||
def _namespace(self, run_id: str = "run-1") -> argparse.Namespace:
|
||||
return argparse.Namespace(manifest=str(self.manifest_path), run_id=run_id)
|
||||
|
||||
def _patch_boundaries(self) -> dict[str, mock.Mock]:
|
||||
stack = contextlib.ExitStack()
|
||||
self.addCleanup(stack.close)
|
||||
load_manifest = stack.enter_context(
|
||||
mock.patch.object(cli_module, "load_manifest", return_value=mock.sentinel.manifest)
|
||||
)
|
||||
run_store_cls = stack.enter_context(mock.patch.object(cli_module, "RunStore"))
|
||||
store = run_store_cls.return_value
|
||||
store.open.return_value = mock.sentinel.run
|
||||
publish_report_mock = stack.enter_context(
|
||||
mock.patch.object(
|
||||
cli_module, "publish_report", return_value=self.report_path
|
||||
)
|
||||
)
|
||||
build_registry = stack.enter_context(
|
||||
mock.patch.object(cli_module, "build_adapter_registry")
|
||||
)
|
||||
return {
|
||||
"load_manifest": load_manifest,
|
||||
"RunStore": run_store_cls,
|
||||
"store": store,
|
||||
"publish_report": publish_report_mock,
|
||||
"build_adapter_registry": build_registry,
|
||||
}
|
||||
|
||||
def _invoke(self, run_id: str = "run-1") -> tuple[int, str, str]:
|
||||
out = io.StringIO()
|
||||
err = io.StringIO()
|
||||
with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
|
||||
code = cli_module._cmd_report(self._namespace(run_id=run_id))
|
||||
return code, out.getvalue(), err.getvalue()
|
||||
|
||||
def test_success_prints_repo_relative_path_and_exits_zero(self) -> None:
|
||||
mocks = self._patch_boundaries()
|
||||
code, out, err = self._invoke(run_id="run-1")
|
||||
|
||||
self.assertEqual(code, cli_module.EXIT_VALID)
|
||||
mocks["load_manifest"].assert_called_once_with(
|
||||
self.manifest_path, repo_root=cli_module._REPO_ROOT
|
||||
)
|
||||
mocks["RunStore"].assert_called_once_with(cli_module._REPO_ROOT)
|
||||
mocks["store"].open.assert_called_once_with(
|
||||
mock.sentinel.manifest, "run-1", self.manifest_bytes
|
||||
)
|
||||
mocks["publish_report"].assert_called_once_with(
|
||||
mocks["store"], mock.sentinel.run, mock.sentinel.manifest
|
||||
)
|
||||
self.assertEqual(out, f"ok: report run_id=run-1 path={self.report_rel}\n")
|
||||
self.assertEqual(err, "")
|
||||
mocks["build_adapter_registry"].assert_not_called()
|
||||
|
||||
def test_typed_failure_emits_only_closed_line_and_exits_invalid(self) -> None:
|
||||
mocks = self._patch_boundaries()
|
||||
mocks["publish_report"].side_effect = ReportError("projection failed")
|
||||
|
||||
code, out, err = self._invoke(run_id="run-9")
|
||||
|
||||
self.assertEqual(code, cli_module.EXIT_INVALID)
|
||||
self.assertEqual(out, "")
|
||||
self.assertEqual(err, "error: benchmark report is unavailable\n")
|
||||
mocks["build_adapter_registry"].assert_not_called()
|
||||
|
||||
def test_generic_failure_emits_only_closed_line_and_exits_invalid(self) -> None:
|
||||
mocks = self._patch_boundaries()
|
||||
mocks["publish_report"].side_effect = ValueError("unexpected boundary")
|
||||
|
||||
code, out, err = self._invoke(run_id="run-9")
|
||||
|
||||
self.assertEqual(code, cli_module.EXIT_INVALID)
|
||||
self.assertEqual(out, "")
|
||||
self.assertEqual(err, "error: benchmark report is unavailable\n")
|
||||
mocks["build_adapter_registry"].assert_not_called()
|
||||
|
||||
def test_two_successful_calls_delegate_idempotently_to_reporter(self) -> None:
|
||||
mocks = self._patch_boundaries()
|
||||
|
||||
for _ in range(2):
|
||||
code, out, err = self._invoke(run_id="run-1")
|
||||
self.assertEqual(code, cli_module.EXIT_VALID)
|
||||
self.assertEqual(out, f"ok: report run_id=run-1 path={self.report_rel}\n")
|
||||
self.assertEqual(err, "")
|
||||
|
||||
# Each CLI call delegates exactly once to the strict reporter boundary;
|
||||
# the reporter itself remains the sole idempotent publication surface.
|
||||
self.assertEqual(mocks["publish_report"].call_count, 2)
|
||||
self.assertEqual(mocks["store"].open.call_count, 2)
|
||||
mocks["build_adapter_registry"].assert_not_called()
|
||||
|
||||
def test_adapter_registry_is_never_constructed(self) -> None:
|
||||
mocks = self._patch_boundaries()
|
||||
|
||||
code, _, _ = self._invoke(run_id="run-1")
|
||||
self.assertEqual(code, cli_module.EXIT_VALID)
|
||||
|
||||
mocks["publish_report"].side_effect = ReportError("publication failed")
|
||||
code, _, _ = self._invoke(run_id="run-1")
|
||||
self.assertEqual(code, cli_module.EXIT_INVALID)
|
||||
|
||||
mocks["build_adapter_registry"].assert_not_called()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,197 +0,0 @@
|
|||
"""Strict versioned benchmark worksheet contract.
|
||||
|
||||
Automatic web gates establish scoring eligibility; they are intentionally not
|
||||
represented in this 100-point worksheet and can never contribute points.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from types import MappingProxyType
|
||||
from typing import Any, Mapping
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
ONE_SHOT_RUBRIC_VERSION,
|
||||
RUBRIC_VERSION,
|
||||
)
|
||||
|
||||
|
||||
MAX_WORKSHEET_BYTES = 64 * 1024
|
||||
MAX_EVIDENCE_CHARS = 4096
|
||||
RUBRIC_CATEGORIES = (
|
||||
("task_fidelity", 25),
|
||||
("visual_hierarchy", 25),
|
||||
("responsive_composition", 20),
|
||||
("typography_readability", 15),
|
||||
("polish_consistency", 15),
|
||||
)
|
||||
ONE_SHOT_RUBRIC_CATEGORIES = (
|
||||
("requirements_fidelity", 25),
|
||||
("visual_completeness", 25),
|
||||
("responsive_accessibility", 15),
|
||||
("image_detail_usage", 10),
|
||||
("behavior_stability", 10),
|
||||
("code_quality", 10),
|
||||
("self_verification", 5),
|
||||
)
|
||||
RUBRIC_CATEGORIES_BY_VERSION: Mapping[str, tuple[tuple[str, int], ...]] = (
|
||||
MappingProxyType(
|
||||
{
|
||||
RUBRIC_VERSION: RUBRIC_CATEGORIES,
|
||||
ONE_SHOT_RUBRIC_VERSION: ONE_SHOT_RUBRIC_CATEGORIES,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class RubricError(Exception):
|
||||
"""A worksheet is missing, malformed, non-canonical, or out of bounds."""
|
||||
|
||||
|
||||
def rubric_categories(version: str) -> tuple[tuple[str, int], ...]:
|
||||
"""Return the immutable ordered category table for a known rubric version."""
|
||||
if not isinstance(version, str):
|
||||
raise RubricError("rubric version is invalid")
|
||||
try:
|
||||
return RUBRIC_CATEGORIES_BY_VERSION[version]
|
||||
except KeyError as exc:
|
||||
raise RubricError("rubric version is invalid") from exc
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CategoryScore:
|
||||
id: str
|
||||
max_score: int
|
||||
score: int
|
||||
evidence: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Worksheet:
|
||||
rubric_version: str
|
||||
categories: tuple[CategoryScore, ...]
|
||||
total: int
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"rubric_version": self.rubric_version,
|
||||
"categories": [
|
||||
{
|
||||
"id": item.id,
|
||||
"max_score": item.max_score,
|
||||
"score": item.score,
|
||||
"evidence": item.evidence,
|
||||
}
|
||||
for item in self.categories
|
||||
],
|
||||
"total": self.total,
|
||||
}
|
||||
|
||||
|
||||
def canonical_worksheet_bytes(worksheet: Worksheet) -> bytes:
|
||||
if not isinstance(worksheet, Worksheet):
|
||||
raise RubricError("worksheet object is invalid")
|
||||
return (
|
||||
json.dumps(
|
||||
worksheet.as_dict(),
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
ensure_ascii=True,
|
||||
).encode("ascii")
|
||||
+ b"\n"
|
||||
)
|
||||
|
||||
|
||||
def validate_worksheet(
|
||||
value: Any, expected_version: str | None = None
|
||||
) -> Worksheet:
|
||||
if not isinstance(value, dict) or set(value) != {
|
||||
"rubric_version", "categories", "total",
|
||||
}:
|
||||
raise RubricError("worksheet schema is invalid")
|
||||
rubric_version = value["rubric_version"]
|
||||
expected_categories = rubric_categories(rubric_version)
|
||||
if expected_version is not None:
|
||||
rubric_categories(expected_version)
|
||||
if expected_version is not None and rubric_version != expected_version:
|
||||
raise RubricError("worksheet rubric version is invalid")
|
||||
raw_categories = value["categories"]
|
||||
if not isinstance(raw_categories, list) or len(raw_categories) != len(
|
||||
expected_categories
|
||||
):
|
||||
raise RubricError("worksheet categories are invalid")
|
||||
|
||||
categories: list[CategoryScore] = []
|
||||
for raw, (expected_id, expected_max) in zip(
|
||||
raw_categories, expected_categories
|
||||
):
|
||||
if not isinstance(raw, dict) or set(raw) != {
|
||||
"id", "max_score", "score", "evidence",
|
||||
}:
|
||||
raise RubricError("worksheet category schema is invalid")
|
||||
score = raw["score"]
|
||||
evidence = raw["evidence"]
|
||||
if (
|
||||
raw["id"] != expected_id
|
||||
or raw["max_score"] != expected_max
|
||||
or isinstance(score, bool)
|
||||
or not isinstance(score, int)
|
||||
or not 0 <= score <= expected_max
|
||||
or not isinstance(evidence, str)
|
||||
or not evidence.strip()
|
||||
or len(evidence) > MAX_EVIDENCE_CHARS
|
||||
or any(ord(char) < 0x20 and char not in "\n\t" for char in evidence)
|
||||
):
|
||||
raise RubricError("worksheet category is invalid")
|
||||
categories.append(
|
||||
CategoryScore(expected_id, expected_max, score, evidence)
|
||||
)
|
||||
|
||||
total = value["total"]
|
||||
expected_total = sum(item.score for item in categories)
|
||||
if (
|
||||
isinstance(total, bool)
|
||||
or not isinstance(total, int)
|
||||
or total != expected_total
|
||||
or not 0 <= total <= 100
|
||||
):
|
||||
raise RubricError("worksheet total is invalid")
|
||||
return Worksheet(rubric_version, tuple(categories), total)
|
||||
|
||||
|
||||
def load_worksheet(
|
||||
path: str | Path, expected_version: str | None = None
|
||||
) -> Worksheet:
|
||||
target = Path(path)
|
||||
flags = os.O_RDONLY | os.O_CLOEXEC | os.O_NONBLOCK
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
try:
|
||||
fd = os.open(target, flags)
|
||||
except OSError as exc:
|
||||
raise RubricError("worksheet is unavailable") from exc
|
||||
try:
|
||||
info = os.fstat(fd)
|
||||
if not stat.S_ISREG(info.st_mode) or info.st_size > MAX_WORKSHEET_BYTES:
|
||||
raise RubricError("worksheet must be a bounded regular file")
|
||||
raw = bytearray()
|
||||
while len(raw) < info.st_size:
|
||||
chunk = os.read(fd, info.st_size - len(raw))
|
||||
if not chunk:
|
||||
raise RubricError("worksheet changed while reading")
|
||||
raw.extend(chunk)
|
||||
if os.read(fd, 1):
|
||||
raise RubricError("worksheet changed while reading")
|
||||
except OSError as exc:
|
||||
raise RubricError("worksheet is unavailable") from exc
|
||||
finally:
|
||||
os.close(fd)
|
||||
try:
|
||||
value = json.loads(bytes(raw).decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise RubricError("worksheet JSON is invalid") from exc
|
||||
return validate_worksheet(value, expected_version=expected_version)
|
||||
|
|
@ -1,157 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
ONE_SHOT_RUBRIC_VERSION,
|
||||
RUBRIC_VERSION,
|
||||
)
|
||||
from scripts.agent_benchmark.rubric import (
|
||||
ONE_SHOT_RUBRIC_CATEGORIES,
|
||||
RUBRIC_CATEGORIES,
|
||||
RUBRIC_CATEGORIES_BY_VERSION,
|
||||
RubricError,
|
||||
canonical_worksheet_bytes,
|
||||
load_worksheet,
|
||||
rubric_categories,
|
||||
validate_worksheet,
|
||||
)
|
||||
|
||||
|
||||
def _worksheet(rubric_version: str = RUBRIC_VERSION) -> dict:
|
||||
categories = [
|
||||
{
|
||||
"id": ident,
|
||||
"max_score": maximum,
|
||||
"score": maximum,
|
||||
"evidence": f"Evidence for {ident}.",
|
||||
}
|
||||
for ident, maximum in rubric_categories(rubric_version)
|
||||
]
|
||||
return {
|
||||
"rubric_version": rubric_version,
|
||||
"categories": categories,
|
||||
"total": 100,
|
||||
}
|
||||
|
||||
|
||||
class RubricTest(unittest.TestCase):
|
||||
def test_exact_categories_and_total_are_accepted(self):
|
||||
worksheet = validate_worksheet(_worksheet())
|
||||
self.assertEqual(worksheet.total, 100)
|
||||
self.assertEqual(
|
||||
[(item.id, item.max_score) for item in worksheet.categories],
|
||||
list(RUBRIC_CATEGORIES),
|
||||
)
|
||||
self.assertEqual(
|
||||
validate_worksheet(json.loads(canonical_worksheet_bytes(worksheet))),
|
||||
worksheet,
|
||||
)
|
||||
self.assertEqual(
|
||||
hashlib.sha256(canonical_worksheet_bytes(worksheet)).hexdigest(),
|
||||
"233727170226ab30409657007751c13616a16c83ca75700e15a2c8dce56a96ee",
|
||||
)
|
||||
|
||||
def test_one_shot_rubric_exact_categories_and_total_are_accepted(self):
|
||||
expected = (
|
||||
("requirements_fidelity", 25),
|
||||
("visual_completeness", 25),
|
||||
("responsive_accessibility", 15),
|
||||
("image_detail_usage", 10),
|
||||
("behavior_stability", 10),
|
||||
("code_quality", 10),
|
||||
("self_verification", 5),
|
||||
)
|
||||
self.assertEqual(ONE_SHOT_RUBRIC_CATEGORIES, expected)
|
||||
self.assertEqual(rubric_categories(ONE_SHOT_RUBRIC_VERSION), expected)
|
||||
self.assertEqual(tuple(RUBRIC_CATEGORIES_BY_VERSION), (
|
||||
RUBRIC_VERSION,
|
||||
ONE_SHOT_RUBRIC_VERSION,
|
||||
))
|
||||
self.assertEqual(sum(maximum for _, maximum in RUBRIC_CATEGORIES), 100)
|
||||
self.assertEqual(sum(maximum for _, maximum in expected), 100)
|
||||
|
||||
worksheet = validate_worksheet(
|
||||
_worksheet(ONE_SHOT_RUBRIC_VERSION),
|
||||
expected_version=ONE_SHOT_RUBRIC_VERSION,
|
||||
)
|
||||
self.assertEqual(worksheet.rubric_version, ONE_SHOT_RUBRIC_VERSION)
|
||||
self.assertEqual(worksheet.total, 100)
|
||||
self.assertEqual(
|
||||
[(item.id, item.max_score) for item in worksheet.categories],
|
||||
list(expected),
|
||||
)
|
||||
|
||||
def test_unknown_and_cross_version_worksheets_are_rejected(self):
|
||||
with self.assertRaises(RubricError):
|
||||
rubric_categories("unknown-rubric-v1")
|
||||
with self.assertRaises(RubricError):
|
||||
validate_worksheet(_worksheet(), expected_version=ONE_SHOT_RUBRIC_VERSION)
|
||||
with self.assertRaises(RubricError):
|
||||
validate_worksheet(
|
||||
_worksheet(ONE_SHOT_RUBRIC_VERSION),
|
||||
expected_version=RUBRIC_VERSION,
|
||||
)
|
||||
unknown = _worksheet()
|
||||
unknown["rubric_version"] = "unknown-rubric-v1"
|
||||
with self.assertRaises(RubricError):
|
||||
validate_worksheet(unknown)
|
||||
|
||||
def test_cross_version_worksheet_is_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path = Path(temp) / "worksheet.json"
|
||||
path.write_text(json.dumps(_worksheet()), encoding="utf-8")
|
||||
with self.assertRaises(RubricError):
|
||||
load_worksheet(path, expected_version=ONE_SHOT_RUBRIC_VERSION)
|
||||
|
||||
def test_missing_extra_reordered_and_out_of_range_values_fail(self):
|
||||
cases = []
|
||||
missing = _worksheet()
|
||||
missing["categories"] = missing["categories"][:-1]
|
||||
cases.append(missing)
|
||||
extra = _worksheet()
|
||||
extra["automatic_gate_points"] = 1
|
||||
cases.append(extra)
|
||||
reordered = _worksheet()
|
||||
reordered["categories"] = list(reversed(reordered["categories"]))
|
||||
cases.append(reordered)
|
||||
too_high = _worksheet()
|
||||
too_high["categories"][0]["score"] = 26
|
||||
too_high["total"] = 101
|
||||
cases.append(too_high)
|
||||
bad_total = _worksheet()
|
||||
bad_total["total"] = 0
|
||||
cases.append(bad_total)
|
||||
empty_evidence = _worksheet()
|
||||
empty_evidence["categories"][0]["evidence"] = " "
|
||||
cases.append(empty_evidence)
|
||||
for value in cases:
|
||||
with self.subTest(value=value):
|
||||
with self.assertRaises(RubricError):
|
||||
validate_worksheet(value)
|
||||
|
||||
def test_loader_refuses_nonregular_and_malformed_files(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
valid = root / "worksheet.json"
|
||||
valid.write_bytes(json.dumps(_worksheet()).encode())
|
||||
self.assertEqual(load_worksheet(valid).total, 100)
|
||||
|
||||
malformed = root / "malformed.json"
|
||||
malformed.write_bytes(b"{}")
|
||||
with self.assertRaises(RubricError):
|
||||
load_worksheet(malformed)
|
||||
|
||||
link = root / "link.json"
|
||||
os.symlink(valid, link)
|
||||
with self.assertRaises(RubricError):
|
||||
load_worksheet(link)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
File diff suppressed because it is too large
Load diff
|
|
@ -1,580 +0,0 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest import mock
|
||||
|
||||
from scripts.agent_benchmark import web_validation as web_validation_module
|
||||
from scripts.agent_benchmark.browser_cdp import (
|
||||
BrowserError,
|
||||
RenderObservation,
|
||||
ViewportObservation,
|
||||
)
|
||||
from scripts.agent_benchmark.web_validation import (
|
||||
WEB_GATES,
|
||||
WebValidationError,
|
||||
build_web_validation,
|
||||
load_web_validation,
|
||||
publish_web_validation,
|
||||
validate_web_attempt,
|
||||
)
|
||||
|
||||
|
||||
def _digest(data: bytes) -> str:
|
||||
return "sha256:" + hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
class WebValidationTest(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.temporary = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temporary.cleanup)
|
||||
self.attempt = Path(self.temporary.name) / "attempt"
|
||||
self.attempt.mkdir()
|
||||
self.workspace = self.attempt / "workspace"
|
||||
self.workspace.mkdir()
|
||||
(self.workspace / "assets").mkdir()
|
||||
(self.workspace / "brief").mkdir()
|
||||
self.asset_content = {
|
||||
"assets/a.svg": b"<svg xmlns='http://www.w3.org/2000/svg' width='20' height='20'/>",
|
||||
"assets/b.svg": b"<svg xmlns='http://www.w3.org/2000/svg' width='20' height='20'/>",
|
||||
"brief/reference.txt": b"reference\n",
|
||||
}
|
||||
for relative, content in self.asset_content.items():
|
||||
(self.workspace / relative).write_bytes(content)
|
||||
(self.workspace / "index.html").write_text(
|
||||
"<main><h1>Ready</h1><img src='assets/a.svg' alt='A'>"
|
||||
"<img src='assets/b.svg' alt='B'><a href='#x'>go</a>"
|
||||
"<script src='script.js'></script></main>",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(self.workspace / "styles.css").write_text(
|
||||
"body{color:#111;background:#fff}img{width:20px}"
|
||||
"a:focus{outline:2px solid #05f}",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(self.workspace / "script.js").write_text(
|
||||
"document.body.dataset.ready='1';", encoding="utf-8"
|
||||
)
|
||||
(self.attempt / "attempt-measurement.json").write_bytes(b"measurement\n")
|
||||
|
||||
def _manifest(self):
|
||||
assets = tuple(
|
||||
SimpleNamespace(workspace_path=path, content=content)
|
||||
for path, content in sorted(self.asset_content.items())
|
||||
)
|
||||
return SimpleNamespace(
|
||||
digest="sha256:" + "a" * 64,
|
||||
fixture=SimpleNamespace(
|
||||
checksum="sha256:" + "b" * 64,
|
||||
assets=assets,
|
||||
),
|
||||
viewports=(
|
||||
SimpleNamespace(id="desktop", width=800, height=600),
|
||||
SimpleNamespace(id="mobile", width=375, height=600),
|
||||
),
|
||||
timeout=SimpleNamespace(run_seconds=20),
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _measurement(reason: str = "success"):
|
||||
return SimpleNamespace(
|
||||
run_id="run",
|
||||
cell_id="cell",
|
||||
repetition=1,
|
||||
attempt=1,
|
||||
)
|
||||
|
||||
def _view(self, ident: str, width: int, *, suffix: str = "") -> ViewportObservation:
|
||||
screenshot = f"screenshot-{ident}.png"
|
||||
png = b"\x89PNG\r\n\x1a\n" + ident.encode("ascii") + suffix.encode("ascii")
|
||||
(self.attempt / screenshot).write_bytes(png)
|
||||
images = tuple(
|
||||
{
|
||||
"src": path,
|
||||
"alt": path,
|
||||
"complete": True,
|
||||
"natural_width": 20,
|
||||
"natural_height": 20,
|
||||
"visible": True,
|
||||
"rect": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"width": 20,
|
||||
"height": 20,
|
||||
"right": 20,
|
||||
"bottom": 20,
|
||||
},
|
||||
}
|
||||
for path in ("assets/a.svg", "assets/b.svg")
|
||||
)
|
||||
accessibility = {
|
||||
"h1_count": 1,
|
||||
"headings": [1],
|
||||
"heading_progression": True,
|
||||
"main_count": 1,
|
||||
"landmarks": 1,
|
||||
"controls": [
|
||||
{
|
||||
"name": True,
|
||||
"tab_index": 0,
|
||||
"focused": True,
|
||||
"focus_visible": True,
|
||||
"contrast": 7.0,
|
||||
}
|
||||
],
|
||||
"ax": {"nodes": 4, "non_ignored": 3, "named": 2},
|
||||
}
|
||||
return ViewportObservation(
|
||||
ident,
|
||||
width,
|
||||
600,
|
||||
screenshot,
|
||||
_digest(png),
|
||||
len(png),
|
||||
images,
|
||||
{
|
||||
"scroll_width": width,
|
||||
"client_width": width,
|
||||
"clipped": 0,
|
||||
"overlaps": 0,
|
||||
},
|
||||
accessibility,
|
||||
)
|
||||
|
||||
def _render(self) -> RenderObservation:
|
||||
return RenderObservation(
|
||||
"Chromium/Test",
|
||||
"http://127.0.0.1:12345",
|
||||
(
|
||||
{"kind": "local", "path": "/index.html", "allowed": True, "status": 200},
|
||||
{"kind": "local", "path": "/assets/a.svg", "allowed": True, "status": 200},
|
||||
{"kind": "local", "path": "/assets/b.svg", "allowed": True, "status": 200},
|
||||
),
|
||||
(),
|
||||
(self._view("desktop", 800), self._view("mobile", 375)),
|
||||
)
|
||||
|
||||
def _build(self, render=None, reason: str = "success"):
|
||||
return build_web_validation(
|
||||
self._manifest(),
|
||||
self.workspace,
|
||||
self._measurement(reason),
|
||||
self._render() if render is None else render,
|
||||
)
|
||||
|
||||
def _publish_valid(self):
|
||||
record = self._build()
|
||||
publish_web_validation(self.attempt, record)
|
||||
return record
|
||||
|
||||
def _rewrite_record(self, mutate) -> None:
|
||||
path = self.attempt / "web-validation.json"
|
||||
record = json.loads(path.read_text(encoding="ascii"))
|
||||
mutate(record)
|
||||
path.write_text(
|
||||
json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n",
|
||||
encoding="ascii",
|
||||
)
|
||||
|
||||
def _reproject_runtime_gates(self, record) -> None:
|
||||
manifest = self._manifest()
|
||||
render = SimpleNamespace(
|
||||
requests=tuple(record["requests"]),
|
||||
console=tuple(record["console"]),
|
||||
viewports=tuple(
|
||||
SimpleNamespace(
|
||||
id=item["id"],
|
||||
width=item["width"],
|
||||
height=item["height"],
|
||||
image_facts=tuple(item["images"]),
|
||||
layout=item["layout"],
|
||||
accessibility=item["accessibility"],
|
||||
)
|
||||
for item in record["viewports"]
|
||||
),
|
||||
)
|
||||
gates = web_validation_module._runtime_gates(manifest, render)
|
||||
generated = web_validation_module._generated_gate(record["workspace"])
|
||||
gates["generated_files"] = generated
|
||||
gates["static_safety"] = web_validation_module._static_gate(
|
||||
self.workspace, generated, manifest
|
||||
)
|
||||
record["screenshots"] = [
|
||||
{"id": item["id"], **item["screenshot"]}
|
||||
for item in record["viewports"]
|
||||
]
|
||||
record["gates"] = [gates[ident] for ident in WEB_GATES]
|
||||
passed = all(item["passed"] for item in record["gates"])
|
||||
record["status"] = "passed" if passed else "failed"
|
||||
record["reason"] = "" if passed else next(
|
||||
item["reason"] for item in record["gates"] if not item["passed"]
|
||||
)
|
||||
|
||||
def test_valid_record_binds_complete_evidence_and_is_immutable(self):
|
||||
record = self._publish_valid()
|
||||
loaded = load_web_validation(self.attempt, manifest=self._manifest())
|
||||
self.assertEqual(loaded.status, "passed")
|
||||
self.assertEqual([item["id"] for item in loaded.record["gates"]], list(WEB_GATES))
|
||||
self.assertEqual(len(loaded.record["workspace"]["inputs"]), 3)
|
||||
self.assertEqual(len(loaded.record["viewports"]), 2)
|
||||
self.assertEqual(len(loaded.record["screenshots"]), 2)
|
||||
self.assertEqual(record.record, loaded.record)
|
||||
before = (self.attempt / "web-validation.json").read_bytes()
|
||||
with self.assertRaises(WebValidationError):
|
||||
publish_web_validation(self.attempt, record)
|
||||
self.assertEqual((self.attempt / "web-validation.json").read_bytes(), before)
|
||||
|
||||
def test_static_workspace_fault_matrix_is_recorded_failed(self):
|
||||
cases = {
|
||||
"fixture_mutation": lambda: (self.workspace / "assets/a.svg").write_bytes(b"changed"),
|
||||
"fixture_symlink": self._replace_fixture_with_symlink,
|
||||
"extra_nested": lambda: (self.workspace / "assets/extra.svg").write_text("extra"),
|
||||
"external_reference": lambda: (self.workspace / "index.html").write_text(
|
||||
"<main><h1>x</h1><img src='https://example.invalid/a.png' alt='x'></main>"
|
||||
),
|
||||
}
|
||||
for name, mutate in cases.items():
|
||||
with self.subTest(name=name):
|
||||
self.tearDown()
|
||||
self.setUp()
|
||||
mutate()
|
||||
record = self._build()
|
||||
self.assertEqual(record.status, "failed")
|
||||
self.assertFalse(all(item["passed"] for item in record.record["gates"]))
|
||||
|
||||
def _replace_fixture_with_symlink(self) -> None:
|
||||
target = self.workspace / "assets/a.svg"
|
||||
target.unlink()
|
||||
target.symlink_to(self.workspace / "assets/b.svg")
|
||||
|
||||
def test_runtime_gate_one_fault_matrix(self):
|
||||
cases = {}
|
||||
render = self._render()
|
||||
image_views = list(render.viewports)
|
||||
image_facts = [dict(item) for item in image_views[0].image_facts]
|
||||
image_facts[0]["complete"] = False
|
||||
image_views[0] = SimpleNamespace(**{
|
||||
**image_views[0].__dict__, "image_facts": tuple(image_facts)
|
||||
})
|
||||
cases["images"] = SimpleNamespace(**{**render.__dict__, "viewports": tuple(image_views)})
|
||||
|
||||
render = self._render()
|
||||
cases["network"] = SimpleNamespace(**{
|
||||
**render.__dict__,
|
||||
"requests": (*render.requests, {"kind": "external", "url_digest": _digest(b"x"), "allowed": False, "status": 0}),
|
||||
})
|
||||
render = self._render()
|
||||
cases["console"] = SimpleNamespace(**{
|
||||
**render.__dict__, "console": ({"kind": "exception", "level": "error"},)
|
||||
})
|
||||
render = self._render()
|
||||
views = list(render.viewports)
|
||||
views[0] = SimpleNamespace(**{
|
||||
**views[0].__dict__,
|
||||
"layout": {"scroll_width": 801, "client_width": 800, "clipped": 1, "overlaps": 0},
|
||||
})
|
||||
cases["responsive"] = SimpleNamespace(**{**render.__dict__, "viewports": tuple(views)})
|
||||
render = self._render()
|
||||
views = list(render.viewports)
|
||||
accessibility = copy.deepcopy(views[0].accessibility)
|
||||
accessibility["controls"][0]["focus_visible"] = False
|
||||
views[0] = SimpleNamespace(**{**views[0].__dict__, "accessibility": accessibility})
|
||||
cases["accessibility"] = SimpleNamespace(**{**render.__dict__, "viewports": tuple(views)})
|
||||
|
||||
for gate, faulty in cases.items():
|
||||
with self.subTest(gate=gate):
|
||||
record = self._build(faulty)
|
||||
self.assertEqual(record.status, "failed")
|
||||
gates = {item["id"]: item for item in record.record["gates"]}
|
||||
self.assertFalse(gates[gate]["passed"])
|
||||
|
||||
def test_lifecycle_non_success_still_validates_workspace(self):
|
||||
for reason in ("nonzero_exit", "timed_out", "cancelled", "controller_lost"):
|
||||
with self.subTest(reason=reason):
|
||||
record = build_web_validation(
|
||||
self._manifest(), self.workspace, self._measurement(reason), None
|
||||
)
|
||||
self.assertEqual(record.status, "failed")
|
||||
self.assertEqual(record.record["reason"], "render_not_run")
|
||||
gates = {item["id"]: item for item in record.record["gates"]}
|
||||
self.assertTrue(gates["generated_files"]["passed"])
|
||||
self.assertTrue(gates["static_safety"]["passed"])
|
||||
self.assertFalse(gates["images"]["passed"])
|
||||
self.assertFalse(gates["responsive"]["passed"])
|
||||
|
||||
def test_browser_discovery_or_start_failure_is_blocked(self):
|
||||
prepared = SimpleNamespace(
|
||||
workspace_dir=str(self.workspace), attempt_root=str(self.attempt)
|
||||
)
|
||||
for error in (FileNotFoundError("missing"), OSError("start")):
|
||||
with self.subTest(error=type(error).__name__), mock.patch(
|
||||
"scripts.agent_benchmark.web_validation.BrowserRenderer.render",
|
||||
side_effect=error,
|
||||
):
|
||||
record = validate_web_attempt(
|
||||
self._manifest(),
|
||||
self.attempt,
|
||||
prepared,
|
||||
self._measurement(),
|
||||
)
|
||||
self.assertEqual(record.status, "blocked")
|
||||
self.assertFalse(record.record["screenshots"])
|
||||
|
||||
with mock.patch(
|
||||
"scripts.agent_benchmark.web_validation.BrowserRenderer.render",
|
||||
side_effect=BrowserError("screenshot_collision"),
|
||||
), self.assertRaises(WebValidationError):
|
||||
validate_web_attempt(
|
||||
self._manifest(),
|
||||
self.attempt,
|
||||
prepared,
|
||||
self._measurement(),
|
||||
)
|
||||
|
||||
def test_missing_generated_files_are_failed_not_not_run(self):
|
||||
(self.workspace / "index.html").unlink()
|
||||
record = build_web_validation(
|
||||
self._manifest(), self.workspace, self._measurement(), None
|
||||
)
|
||||
self.assertEqual(record.status, "failed")
|
||||
self.assertFalse(record.record["gates"][0]["passed"])
|
||||
|
||||
def test_status_gate_contradiction_and_unknown_nested_field_are_rejected(self):
|
||||
self._publish_valid()
|
||||
self._rewrite_record(
|
||||
lambda record: record["gates"][0].update(
|
||||
{"passed": False, "reason": "generated_missing"}
|
||||
)
|
||||
)
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
(self.attempt / "web-validation.json").unlink()
|
||||
self._publish_valid()
|
||||
self._rewrite_record(lambda record: record["browser"].update({"unknown": 1}))
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
(self.attempt / "web-validation.json").unlink()
|
||||
self._publish_valid()
|
||||
self._rewrite_record(
|
||||
lambda record: record["viewports"][0]["images"][0].update(
|
||||
{"complete": False}
|
||||
)
|
||||
)
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
(self.attempt / "web-validation.json").unlink()
|
||||
self._publish_valid()
|
||||
self._rewrite_record(
|
||||
lambda record: record["gates"][0]["evidence"].append("invented")
|
||||
)
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
def test_screenshot_corruption_missing_nonregular_and_symlink_are_rejected(self):
|
||||
variants = ("corrupt", "missing", "directory", "symlink", "extra")
|
||||
for variant in variants:
|
||||
with self.subTest(variant=variant):
|
||||
self.tearDown()
|
||||
self.setUp()
|
||||
record = self._publish_valid()
|
||||
target = self.attempt / record.record["screenshots"][0]["file"]
|
||||
if variant == "corrupt":
|
||||
target.write_bytes(b"not-png")
|
||||
elif variant == "extra":
|
||||
(self.attempt / "screenshot-extra.png").write_bytes(
|
||||
b"\x89PNG\r\n\x1a\nextra"
|
||||
)
|
||||
else:
|
||||
target.unlink()
|
||||
if variant == "directory":
|
||||
target.mkdir()
|
||||
elif variant == "symlink":
|
||||
target.symlink_to(self.attempt / record.record["screenshots"][1]["file"])
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
def test_fixture_generated_extra_and_measurement_changes_are_rejected(self):
|
||||
variants = ("fixture", "generated", "extra", "measurement")
|
||||
for variant in variants:
|
||||
with self.subTest(variant=variant):
|
||||
self.tearDown()
|
||||
self.setUp()
|
||||
self._publish_valid()
|
||||
if variant == "fixture":
|
||||
(self.workspace / "assets/a.svg").write_bytes(b"changed")
|
||||
elif variant == "generated":
|
||||
(self.workspace / "script.js").write_text("changed")
|
||||
elif variant == "extra":
|
||||
(self.workspace / "extra.txt").write_text("extra")
|
||||
else:
|
||||
(self.attempt / "attempt-measurement.json").write_bytes(b"changed")
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt)
|
||||
|
||||
def test_manifest_fixture_and_viewport_binding_are_strict(self):
|
||||
self._publish_valid()
|
||||
manifest = self._manifest()
|
||||
manifest.fixture.checksum = "sha256:" + "c" * 64
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt, manifest=manifest)
|
||||
|
||||
manifest = self._manifest()
|
||||
manifest.viewports = tuple(reversed(manifest.viewports))
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt, manifest=manifest)
|
||||
|
||||
def test_observed_records_require_exact_manifest_viewports(self):
|
||||
def failed_render():
|
||||
render = self._render()
|
||||
return SimpleNamespace(
|
||||
**{
|
||||
**render.__dict__,
|
||||
"console": ({"kind": "exception", "level": "error"},),
|
||||
}
|
||||
)
|
||||
|
||||
valid_failed = self._build(failed_render())
|
||||
self.assertEqual(valid_failed.status, "failed")
|
||||
publish_web_validation(self.attempt, valid_failed)
|
||||
loaded = load_web_validation(self.attempt, manifest=self._manifest())
|
||||
self.assertEqual(loaded.status, "failed")
|
||||
self.assertEqual(len(loaded.record["viewports"]), 2)
|
||||
|
||||
def missing(record):
|
||||
removed = record["viewports"].pop()
|
||||
(self.attempt / removed["screenshot"]["file"]).unlink()
|
||||
|
||||
def empty(record):
|
||||
for item in record["viewports"]:
|
||||
(self.attempt / item["screenshot"]["file"]).unlink()
|
||||
record["viewports"] = []
|
||||
|
||||
def duplicate(record):
|
||||
removed = record["viewports"][1]
|
||||
(self.attempt / removed["screenshot"]["file"]).unlink()
|
||||
record["viewports"] = [
|
||||
copy.deepcopy(record["viewports"][0]),
|
||||
copy.deepcopy(record["viewports"][0]),
|
||||
]
|
||||
|
||||
def reordered(record):
|
||||
record["viewports"].reverse()
|
||||
|
||||
def foreign(record):
|
||||
item = record["viewports"][1]
|
||||
old = self.attempt / item["screenshot"]["file"]
|
||||
item["id"] = "foreign"
|
||||
item["screenshot"]["file"] = "screenshot-foreign.png"
|
||||
old.rename(self.attempt / item["screenshot"]["file"])
|
||||
|
||||
for name, mutate in (
|
||||
("missing", missing),
|
||||
("empty", empty),
|
||||
("duplicate", duplicate),
|
||||
("reordered", reordered),
|
||||
("foreign", foreign),
|
||||
):
|
||||
with self.subTest(case=name):
|
||||
self.tearDown()
|
||||
self.setUp()
|
||||
record = self._build(failed_render())
|
||||
publish_web_validation(self.attempt, record)
|
||||
path = self.attempt / "web-validation.json"
|
||||
raw = json.loads(path.read_text(encoding="ascii"))
|
||||
mutate(raw)
|
||||
self._reproject_runtime_gates(raw)
|
||||
path.write_text(
|
||||
json.dumps(raw, sort_keys=True, separators=(",", ":")) + "\n",
|
||||
encoding="ascii",
|
||||
)
|
||||
before = path.read_bytes()
|
||||
with self.assertRaises(WebValidationError):
|
||||
load_web_validation(self.attempt, manifest=self._manifest())
|
||||
self.assertEqual(path.read_bytes(), before)
|
||||
|
||||
def test_manifest_viewport_id_grammar_round_trips(self):
|
||||
manifest = self._manifest()
|
||||
manifest.viewports = (
|
||||
SimpleNamespace(id="mobile.small+wide", width=375, height=600),
|
||||
)
|
||||
render = RenderObservation(
|
||||
"Chromium/Test",
|
||||
"http://127.0.0.1:12345",
|
||||
(
|
||||
{
|
||||
"kind": "local",
|
||||
"path": "/index.html",
|
||||
"allowed": True,
|
||||
"status": 200,
|
||||
},
|
||||
{
|
||||
"kind": "local",
|
||||
"path": "/assets/a.svg",
|
||||
"allowed": True,
|
||||
"status": 200,
|
||||
},
|
||||
{
|
||||
"kind": "local",
|
||||
"path": "/assets/b.svg",
|
||||
"allowed": True,
|
||||
"status": 200,
|
||||
},
|
||||
),
|
||||
(),
|
||||
(self._view("mobile.small+wide", 375),),
|
||||
)
|
||||
record = build_web_validation(
|
||||
manifest,
|
||||
self.workspace,
|
||||
self._measurement(),
|
||||
render,
|
||||
)
|
||||
self.assertEqual(record.status, "passed")
|
||||
publish_web_validation(self.attempt, record)
|
||||
loaded = load_web_validation(self.attempt, manifest=manifest)
|
||||
self.assertEqual(
|
||||
[item["id"] for item in loaded.record["viewports"]],
|
||||
["mobile.small+wide"],
|
||||
)
|
||||
|
||||
|
||||
def test_negative_tab_index_is_failed_evidence_not_schema_error(self):
|
||||
"""Signed negative tab_index must serialize as failed evidence, not raise."""
|
||||
render = self._render()
|
||||
views = list(render.viewports)
|
||||
# Modify both viewports to have tab_index=-1
|
||||
for i, view in enumerate(views):
|
||||
accessibility = copy.deepcopy(view.accessibility)
|
||||
accessibility["controls"][0]["tab_index"] = -1
|
||||
views[i] = SimpleNamespace(**{
|
||||
**view.__dict__, "accessibility": accessibility
|
||||
})
|
||||
render = SimpleNamespace(**{
|
||||
**render.__dict__, "viewports": tuple(views)
|
||||
})
|
||||
record = self._build(render)
|
||||
self.assertEqual(record.status, "failed")
|
||||
gates = {item["id"]: item for item in record.record["gates"]}
|
||||
self.assertFalse(gates["accessibility"]["passed"])
|
||||
self.assertIn("accessibility_failed", record.record["reason"])
|
||||
for vp in record.record["viewports"]:
|
||||
for ctrl in vp["accessibility"]["controls"]:
|
||||
self.assertEqual(ctrl["tab_index"], -1)
|
||||
publish_web_validation(self.attempt, record)
|
||||
loaded = load_web_validation(self.attempt, manifest=self._manifest())
|
||||
self.assertEqual(loaded.status, "failed")
|
||||
for vp in loaded.record["viewports"]:
|
||||
for ctrl in vp["accessibility"]["controls"]:
|
||||
self.assertEqual(ctrl["tab_index"], -1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,680 +0,0 @@
|
|||
"""
|
||||
workspace.py - Fixture-seeded workspace and fresh caller-session materialization.
|
||||
|
||||
Provides deterministic, standard-library-only workspace preparation beneath an
|
||||
already allocated empty attempt root.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import posixpath
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
AssetMapping,
|
||||
Manifest,
|
||||
digest_workspace_inputs,
|
||||
)
|
||||
|
||||
RUN_ID_RE = re.compile(r"^run-\d{8}T\d{6}Z-[0-9a-f]{12}$")
|
||||
CELL_ID_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{0,63}$")
|
||||
|
||||
|
||||
class WorkspaceError(Exception):
|
||||
"""Base exception for workspace preparation failures."""
|
||||
|
||||
|
||||
class WorkspaceValidationError(WorkspaceError):
|
||||
"""Raised when identity or manifest configuration is invalid for workspace preparation."""
|
||||
|
||||
|
||||
class WorkspacePathError(WorkspaceError):
|
||||
"""Raised when an attempt root or asset path violates containment, symlink, or collision rules."""
|
||||
|
||||
|
||||
class WorkspaceChecksumError(WorkspaceError):
|
||||
"""Raised when initial fixture or materialized workspace checksum fails verification."""
|
||||
|
||||
|
||||
class TestbedError(WorkspaceError):
|
||||
"""Raised when the testbed provenance check fails or testbed is dirty/mutated."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AttemptIdentity:
|
||||
run_id: str
|
||||
cell_id: str
|
||||
repetition: int
|
||||
attempt: int
|
||||
|
||||
|
||||
class _PreparationOwnership:
|
||||
"""Tracks filesystem paths created and owned by a workspace preparation transaction."""
|
||||
|
||||
def __init__(self, attempt_root: Path) -> None:
|
||||
self.attempt_root = attempt_root.resolve()
|
||||
self.created_paths: list[Path] = []
|
||||
|
||||
def register(self, path: Path) -> None:
|
||||
resolved = path.resolve()
|
||||
if resolved not in self.created_paths:
|
||||
self.created_paths.append(resolved)
|
||||
|
||||
def unregister(self, path: Path) -> None:
|
||||
resolved = path.resolve()
|
||||
if resolved in self.created_paths:
|
||||
self.created_paths.remove(resolved)
|
||||
|
||||
def rollback(self) -> None:
|
||||
"""Remove all registered paths created by this invocation in reverse order."""
|
||||
for path in reversed(self.created_paths):
|
||||
if not path.exists() and not path.is_symlink():
|
||||
continue
|
||||
try:
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path, ignore_errors=True)
|
||||
else:
|
||||
path.unlink(missing_ok=True)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _destinations_conflict(path1: str, path2: str) -> bool:
|
||||
"""Return True if path1 and path2 conflict as identical or ancestor/descendant paths."""
|
||||
if path1 == path2:
|
||||
return True
|
||||
return path2.startswith(path1 + "/") or path1.startswith(path2 + "/")
|
||||
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TestbedProvenance:
|
||||
path: str
|
||||
branch: str
|
||||
head: str
|
||||
status_digest: str
|
||||
clean: bool
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PreparedWorkspace:
|
||||
identity: AttemptIdentity
|
||||
attempt_root: str
|
||||
workspace_dir: str
|
||||
session_dir: str
|
||||
session_id: str
|
||||
session_is_fresh: bool
|
||||
workspace_checksum: str
|
||||
setup_cache_policy: str
|
||||
testbed_provenance: TestbedProvenance
|
||||
prepared_at: str
|
||||
|
||||
|
||||
def _find_repo_root(start_path: Path) -> Path:
|
||||
"""Find repository root by searching upward for Makefile or .git."""
|
||||
candidate = start_path.resolve()
|
||||
for _ in range(20):
|
||||
if (candidate / "Makefile").is_file() or (candidate / ".git").exists():
|
||||
return candidate
|
||||
parent = candidate.parent
|
||||
if parent == candidate:
|
||||
break
|
||||
candidate = parent
|
||||
return Path.cwd().resolve()
|
||||
|
||||
|
||||
def validate_attempt_identity(
|
||||
identity: AttemptIdentity, manifest: Manifest | None = None
|
||||
) -> None:
|
||||
"""Validate format and bounds of an AttemptIdentity.
|
||||
|
||||
Raises:
|
||||
WorkspaceValidationError: If any field fails format or boundary checks.
|
||||
"""
|
||||
if not isinstance(identity, AttemptIdentity):
|
||||
raise WorkspaceValidationError("identity must be an AttemptIdentity instance")
|
||||
|
||||
if not isinstance(identity.run_id, str) or not RUN_ID_RE.match(identity.run_id):
|
||||
raise WorkspaceValidationError(f"invalid run_id format '{identity.run_id}'")
|
||||
|
||||
if not isinstance(identity.cell_id, str) or not CELL_ID_RE.match(identity.cell_id):
|
||||
raise WorkspaceValidationError(f"invalid cell_id format '{identity.cell_id}'")
|
||||
|
||||
if (
|
||||
isinstance(identity.repetition, bool)
|
||||
or not isinstance(identity.repetition, int)
|
||||
or identity.repetition < 1
|
||||
):
|
||||
raise WorkspaceValidationError("repetition must be a positive integer >= 1")
|
||||
|
||||
if (
|
||||
isinstance(identity.attempt, bool)
|
||||
or not isinstance(identity.attempt, int)
|
||||
or identity.attempt < 1
|
||||
):
|
||||
raise WorkspaceValidationError("attempt must be a positive integer >= 1")
|
||||
|
||||
if manifest is not None:
|
||||
valid_cell_ids = {cell.id for cell in manifest.matrix}
|
||||
if identity.cell_id not in valid_cell_ids:
|
||||
raise WorkspaceValidationError(
|
||||
f"cell_id '{identity.cell_id}' not found in manifest matrix"
|
||||
)
|
||||
if identity.repetition > manifest.repetitions:
|
||||
raise WorkspaceValidationError(
|
||||
f"repetition {identity.repetition} exceeds manifest repetitions ({manifest.repetitions})"
|
||||
)
|
||||
|
||||
|
||||
def inspect_testbed_provenance(testbed_path: str | Path) -> TestbedProvenance:
|
||||
"""Capture Git provenance of the runtime testbed directory.
|
||||
|
||||
Requires a clean Git working copy. Never modifies the repository.
|
||||
|
||||
Raises:
|
||||
TestbedError: If directory does not exist, is not a git repo, or is dirty.
|
||||
"""
|
||||
path = Path(testbed_path).resolve()
|
||||
if not path.exists() or not path.is_dir():
|
||||
raise TestbedError(
|
||||
f"testbed directory '{path}' does not exist or is not a directory"
|
||||
)
|
||||
|
||||
try:
|
||||
proc_status = subprocess.run(
|
||||
["git", "status", "--porcelain=v1", "--untracked-files=all"],
|
||||
cwd=path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise TestbedError(f"failed to run git status in testbed '{path}': {exc}")
|
||||
|
||||
if proc_status.returncode != 0:
|
||||
raise TestbedError(f"git status returned non-zero exit code in testbed '{path}'")
|
||||
|
||||
status_output = proc_status.stdout.strip()
|
||||
if status_output:
|
||||
raise TestbedError(f"testbed repository '{path}' is dirty:\n{status_output}")
|
||||
|
||||
try:
|
||||
proc_branch = subprocess.run(
|
||||
["git", "branch", "--show-current"],
|
||||
cwd=path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
branch = proc_branch.stdout.strip()
|
||||
if not branch:
|
||||
proc_head_ref = subprocess.run(
|
||||
["git", "rev-parse", "--abbrev-ref", "HEAD"],
|
||||
cwd=path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
branch = proc_head_ref.stdout.strip()
|
||||
except Exception as exc:
|
||||
raise TestbedError(f"failed to inspect git branch in testbed '{path}': {exc}")
|
||||
|
||||
try:
|
||||
proc_head = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
cwd=path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise TestbedError(f"failed to inspect git HEAD in testbed '{path}': {exc}")
|
||||
|
||||
if proc_head.returncode != 0:
|
||||
raise TestbedError(f"git rev-parse HEAD failed in testbed '{path}'")
|
||||
|
||||
head = proc_head.stdout.strip()
|
||||
status_raw = f"branch:{branch}\nhead:{head}\nstatus:{status_output}".encode("utf-8")
|
||||
status_digest = "sha256:" + hashlib.sha256(status_raw).hexdigest()
|
||||
|
||||
return TestbedProvenance(
|
||||
path=str(path),
|
||||
branch=branch,
|
||||
head=head,
|
||||
status_digest=status_digest,
|
||||
clean=True,
|
||||
)
|
||||
|
||||
|
||||
def prepare_workspace(
|
||||
manifest: Manifest,
|
||||
attempt_root: str | Path,
|
||||
identity: AttemptIdentity,
|
||||
repo_root: str | Path | None = None,
|
||||
) -> PreparedWorkspace:
|
||||
"""Materialize clean workspace and session under an allocated empty attempt root.
|
||||
|
||||
Args:
|
||||
manifest: Validated benchmark manifest object.
|
||||
attempt_root: Path to an exclusively allocated empty attempt root directory.
|
||||
identity: Frozen AttemptIdentity specifying run/cell/repetition/attempt.
|
||||
repo_root: Optional repository root path for resolving testbed/assets.
|
||||
|
||||
Returns:
|
||||
PreparedWorkspace dataclass containing materialized metadata.
|
||||
|
||||
Raises:
|
||||
WorkspaceValidationError: If identity or manifest schema/policy checks fail.
|
||||
WorkspacePathError: If path containment, symlink, or root canonicality fails.
|
||||
WorkspaceChecksumError: If fixture or workspace content digest verification fails.
|
||||
TestbedError: If testbed provenance inspection fails or testbed is dirty/modified.
|
||||
"""
|
||||
attempt_root_input = Path(attempt_root)
|
||||
|
||||
if repo_root is None:
|
||||
repo_root_path = _find_repo_root(attempt_root_input)
|
||||
else:
|
||||
repo_root_path = Path(repo_root).resolve()
|
||||
|
||||
# Phase 1: Validate complete plan before any visible publication.
|
||||
validated_inputs = _validate_complete_plan(
|
||||
manifest, attempt_root_input, identity, repo_root_path
|
||||
)
|
||||
|
||||
resolved_attempt_root = validated_inputs["resolved_attempt_root"]
|
||||
owned = _PreparationOwnership(resolved_attempt_root)
|
||||
|
||||
# Phase 2: Stage preparation beneath the validated empty attempt root in a private owned directory.
|
||||
# Phase 3: Verify staged content (checksum, postflight).
|
||||
# Phase 4: Publish metadata and prepared.json.
|
||||
# All three phases are bounded; any exception triggers rollback of function-owned artifacts.
|
||||
try:
|
||||
staging = _stage_preparation(validated_inputs, owned)
|
||||
_verify_staged_preparation(staging, manifest)
|
||||
prepared = _publish_preparation(staging, owned)
|
||||
except Exception:
|
||||
_rollback_owned_preparation(owned)
|
||||
raise
|
||||
|
||||
return prepared
|
||||
|
||||
|
||||
def _validate_complete_plan(
|
||||
manifest: Manifest,
|
||||
attempt_root_input: Path,
|
||||
identity: AttemptIdentity,
|
||||
repo_root_path: Path,
|
||||
) -> dict[str, Any]:
|
||||
"""Validate the entire preparation plan without creating any files.
|
||||
|
||||
Returns a dict of resolved paths and validated state for staging.
|
||||
"""
|
||||
# 1. Validate identity against manifest
|
||||
validate_attempt_identity(identity, manifest)
|
||||
|
||||
if manifest.session_policy != "fresh":
|
||||
raise WorkspaceValidationError(
|
||||
f"unsupported session_policy '{manifest.session_policy}', expected 'fresh'"
|
||||
)
|
||||
if manifest.setup_cache_policy != "isolated":
|
||||
raise WorkspaceValidationError(
|
||||
f"unsupported setup_cache_policy '{manifest.setup_cache_policy}', expected 'isolated'"
|
||||
)
|
||||
|
||||
# 2. Path & Symlink validation on attempt_root
|
||||
raw_path = attempt_root_input
|
||||
if raw_path.is_symlink():
|
||||
raise WorkspacePathError("attempt_root cannot be a symlink")
|
||||
|
||||
# Walk up parent segments to ensure no symlinks in path
|
||||
curr = raw_path
|
||||
while curr != curr.parent:
|
||||
if curr.is_symlink():
|
||||
raise WorkspacePathError(f"path segment '{curr}' in attempt_root is a symlink")
|
||||
if curr == repo_root_path:
|
||||
break
|
||||
curr = curr.parent
|
||||
|
||||
resolved_attempt_root = raw_path.resolve()
|
||||
|
||||
# Verify expected canonical relative path
|
||||
rep_segment = f"repetition-{identity.repetition:04d}"
|
||||
att_segment = f"attempt-{identity.attempt:06d}"
|
||||
expected_rel = (
|
||||
Path(manifest.output_root)
|
||||
/ identity.run_id
|
||||
/ "cells"
|
||||
/ identity.cell_id
|
||||
/ rep_segment
|
||||
/ att_segment
|
||||
)
|
||||
expected_abs = (repo_root_path / expected_rel).resolve()
|
||||
|
||||
if resolved_attempt_root != expected_abs:
|
||||
raise WorkspacePathError(
|
||||
f"attempt_root '{resolved_attempt_root}' does not match expected canonical path '{expected_abs}'"
|
||||
)
|
||||
|
||||
if not resolved_attempt_root.exists() or not resolved_attempt_root.is_dir():
|
||||
raise WorkspacePathError(
|
||||
f"attempt_root '{resolved_attempt_root}' does not exist or is not a directory"
|
||||
)
|
||||
|
||||
# Verify attempt_root is empty
|
||||
if list(resolved_attempt_root.iterdir()):
|
||||
raise WorkspacePathError(f"attempt_root '{resolved_attempt_root}' is not empty")
|
||||
|
||||
# 3. Testbed preflight provenance check (must not be beneath attempt_root)
|
||||
testbed_path = (repo_root_path / manifest.testbed).resolve()
|
||||
try:
|
||||
testbed_path.relative_to(resolved_attempt_root)
|
||||
raise WorkspacePathError("testbed cannot be located beneath attempt_root")
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
testbed_before = inspect_testbed_provenance(testbed_path)
|
||||
|
||||
# 4. Check fixture checksum before copying
|
||||
computed_fixture_checksum = digest_workspace_inputs(manifest.fixture.assets)
|
||||
if computed_fixture_checksum != manifest.fixture.checksum:
|
||||
raise WorkspaceChecksumError(
|
||||
f"declared fixture checksum '{manifest.fixture.checksum}' does not match computed fixture asset digest '{computed_fixture_checksum}'"
|
||||
)
|
||||
|
||||
# 5. Validate all asset sources exist, aren't symlinks, are within repo root
|
||||
# and check for destination collisions before any creation.
|
||||
workspace_dir = resolved_attempt_root / "workspace"
|
||||
if workspace_dir.exists():
|
||||
raise WorkspacePathError("workspace child directory already exists")
|
||||
|
||||
asset_validations: list[dict[str, Any]] = []
|
||||
workspace_destinations: set[str] = set()
|
||||
for asset in manifest.fixture.assets:
|
||||
raw_src_path = repo_root_path / asset.source
|
||||
if raw_src_path.is_symlink():
|
||||
raise WorkspacePathError(f"asset source '{asset.source}' cannot be a symlink")
|
||||
src_path = raw_src_path.resolve()
|
||||
if src_path.is_symlink():
|
||||
raise WorkspacePathError(f"asset source '{asset.source}' cannot be a symlink")
|
||||
if not src_path.exists() or not src_path.is_file():
|
||||
raise WorkspacePathError(
|
||||
f"asset source '{asset.source}' does not exist or is not a regular file"
|
||||
)
|
||||
try:
|
||||
src_path.relative_to(repo_root_path)
|
||||
except ValueError:
|
||||
raise WorkspacePathError(f"asset source '{asset.source}' escapes repository root")
|
||||
|
||||
wp = asset.workspace_path
|
||||
if "\\" in wp or ":" in wp:
|
||||
raise WorkspacePathError(f"asset workspace_path '{wp}' contains invalid characters")
|
||||
wp_norm = posixpath.normpath(wp)
|
||||
if wp != wp_norm or wp.startswith("/") or wp.startswith(".."):
|
||||
raise WorkspacePathError(
|
||||
f"asset workspace_path '{wp}' is not in canonical posix relative form"
|
||||
)
|
||||
|
||||
target_file = workspace_dir / wp_norm
|
||||
try:
|
||||
target_file.relative_to(workspace_dir)
|
||||
except ValueError:
|
||||
raise WorkspacePathError(f"asset workspace_path '{wp}' escapes workspace directory")
|
||||
|
||||
# Check for destination collisions before copying
|
||||
if target_file.exists():
|
||||
raise WorkspacePathError(
|
||||
f"asset destination '{wp}' already exists in workspace"
|
||||
)
|
||||
|
||||
# Check parent path for symlinks
|
||||
parent_check = target_file.parent
|
||||
while parent_check != workspace_dir:
|
||||
if parent_check.is_symlink():
|
||||
raise WorkspacePathError(
|
||||
f"symlink detected in asset target path '{parent_check}'"
|
||||
)
|
||||
parent_check = parent_check.parent
|
||||
|
||||
for existing in workspace_destinations:
|
||||
if _destinations_conflict(existing, wp_norm):
|
||||
raise WorkspacePathError(
|
||||
f"asset workspace_path '{wp_norm}' conflicts with '{existing}'"
|
||||
)
|
||||
workspace_destinations.add(wp_norm)
|
||||
|
||||
asset_validations.append({
|
||||
"source": asset.source,
|
||||
"src_path": src_path,
|
||||
"workspace_path": wp_norm,
|
||||
"target_path": target_file,
|
||||
"content": asset.content,
|
||||
})
|
||||
|
||||
# 6. Verify prompt is not copied unless declared as an asset
|
||||
prompt_declared = any(
|
||||
a.source == manifest.fixture.prompt or a.workspace_path == manifest.fixture.prompt
|
||||
for a in manifest.fixture.assets
|
||||
)
|
||||
if not prompt_declared:
|
||||
prompt_in_ws = workspace_dir / manifest.fixture.prompt
|
||||
if prompt_in_ws.exists():
|
||||
raise WorkspacePathError(
|
||||
"prompt file materialized in workspace without being declared as an asset"
|
||||
)
|
||||
|
||||
return {
|
||||
"resolved_attempt_root": resolved_attempt_root,
|
||||
"workspace_dir": workspace_dir,
|
||||
"testbed_path": testbed_path,
|
||||
"testbed_before": testbed_before,
|
||||
"asset_validations": asset_validations,
|
||||
"prompt_declared": prompt_declared,
|
||||
"identity": identity,
|
||||
}
|
||||
|
||||
|
||||
def _stage_preparation(
|
||||
validated_inputs: dict[str, Any], owned: _PreparationOwnership
|
||||
) -> dict[str, Any]:
|
||||
"""Create workspace/ and session/ directories and copy assets in private staging.
|
||||
|
||||
This is the staging phase. Any failure here triggers rollback of owned entries.
|
||||
"""
|
||||
resolved_attempt_root = validated_inputs["resolved_attempt_root"]
|
||||
|
||||
staging_dir = resolved_attempt_root / f".staging-{secrets.token_hex(6)}"
|
||||
staging_dir.mkdir(parents=False, exist_ok=False)
|
||||
owned.register(staging_dir)
|
||||
|
||||
staging_workspace_dir = staging_dir / "workspace"
|
||||
staging_workspace_dir.mkdir(parents=False, exist_ok=False)
|
||||
|
||||
materialized_assets: list[AssetMapping] = []
|
||||
for av in validated_inputs["asset_validations"]:
|
||||
src_path = av["src_path"]
|
||||
target_file = staging_workspace_dir / av["workspace_path"]
|
||||
content = src_path.read_bytes()
|
||||
if av["content"] and av["content"] != content:
|
||||
raise WorkspaceChecksumError(
|
||||
f"asset content for '{av['source']}' does not match resolved file content"
|
||||
)
|
||||
target_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
target_file.write_bytes(content)
|
||||
materialized_assets.append(
|
||||
AssetMapping(source=av["source"], workspace_path=av["workspace_path"], content=content)
|
||||
)
|
||||
|
||||
staging_session_dir = staging_dir / "session"
|
||||
staging_session_dir.mkdir(parents=False, exist_ok=False)
|
||||
|
||||
return {
|
||||
"resolved_attempt_root": resolved_attempt_root,
|
||||
"staging_dir": staging_dir,
|
||||
"workspace_dir": staging_workspace_dir,
|
||||
"session_dir": staging_session_dir,
|
||||
"materialized_assets": materialized_assets,
|
||||
"validated_inputs": validated_inputs,
|
||||
}
|
||||
|
||||
|
||||
def _verify_staged_preparation(
|
||||
staging: dict[str, Any], manifest: Manifest
|
||||
) -> None:
|
||||
"""Verify staged workspace checksum and testbed postflight."""
|
||||
workspace_dir = staging["workspace_dir"]
|
||||
testbed_path = staging["validated_inputs"]["testbed_path"]
|
||||
testbed_before = staging["validated_inputs"]["testbed_before"]
|
||||
|
||||
# Recompute workspace checksum after materialization
|
||||
actual_assets: list[AssetMapping] = []
|
||||
for root, _, files in os.walk(workspace_dir):
|
||||
for f in files:
|
||||
fp = Path(root) / f
|
||||
if fp.is_symlink():
|
||||
raise WorkspacePathError(f"symlink created in workspace directory '{fp}'")
|
||||
rel_wp = fp.relative_to(workspace_dir).as_posix()
|
||||
actual_assets.append(
|
||||
AssetMapping(source="", workspace_path=rel_wp, content=fp.read_bytes())
|
||||
)
|
||||
|
||||
computed_workspace_checksum = digest_workspace_inputs(actual_assets)
|
||||
if computed_workspace_checksum != manifest.fixture.checksum:
|
||||
raise WorkspaceChecksumError(
|
||||
f"materialized workspace checksum '{computed_workspace_checksum}' does not match declared fixture checksum '{manifest.fixture.checksum}'"
|
||||
)
|
||||
|
||||
# Postflight testbed check
|
||||
testbed_after = inspect_testbed_provenance(testbed_path)
|
||||
if testbed_before != testbed_after:
|
||||
raise TestbedError("testbed repository state was modified during workspace preparation")
|
||||
|
||||
# Store testbed_after and workspace_checksum in staging for publish phase
|
||||
staging["testbed_after"] = testbed_after
|
||||
staging["workspace_checksum"] = computed_workspace_checksum
|
||||
|
||||
|
||||
def _publish_owned_directory(
|
||||
staging_dir: Path,
|
||||
final_dir: Path,
|
||||
owned: _PreparationOwnership,
|
||||
collision_message: str,
|
||||
) -> None:
|
||||
"""Exclusively create final_dir, register ownership, and move staged contents into it."""
|
||||
try:
|
||||
final_dir.mkdir(exist_ok=False)
|
||||
except (FileExistsError, OSError):
|
||||
raise WorkspacePathError(collision_message)
|
||||
|
||||
owned.register(final_dir)
|
||||
|
||||
for item in staging_dir.iterdir():
|
||||
item.rename(final_dir / item.name)
|
||||
|
||||
|
||||
def _publish_preparation(
|
||||
staging: dict[str, Any], owned: _PreparationOwnership
|
||||
) -> PreparedWorkspace:
|
||||
"""Generate session ID, build metadata, publish final entries, and write prepared.json."""
|
||||
validated_inputs = staging["validated_inputs"]
|
||||
identity = validated_inputs["identity"]
|
||||
|
||||
token = secrets.token_hex(6)
|
||||
session_id = (
|
||||
f"session-{identity.run_id}-{identity.cell_id}-"
|
||||
f"rep{identity.repetition:04d}-att{identity.attempt:06d}-{token}"
|
||||
)
|
||||
session_is_fresh = True
|
||||
|
||||
staging_dir = staging["staging_dir"]
|
||||
staging_workspace_dir = staging["workspace_dir"]
|
||||
staging_session_dir = staging["session_dir"]
|
||||
resolved_attempt_root = staging["resolved_attempt_root"]
|
||||
testbed_after = staging["testbed_after"]
|
||||
workspace_checksum = staging["workspace_checksum"]
|
||||
|
||||
final_workspace_dir = resolved_attempt_root / "workspace"
|
||||
_publish_owned_directory(
|
||||
staging_workspace_dir,
|
||||
final_workspace_dir,
|
||||
owned,
|
||||
"workspace child directory already exists in attempt_root",
|
||||
)
|
||||
|
||||
final_session_dir = resolved_attempt_root / "session"
|
||||
_publish_owned_directory(
|
||||
staging_session_dir,
|
||||
final_session_dir,
|
||||
owned,
|
||||
"session child directory already exists in attempt_root",
|
||||
)
|
||||
|
||||
if staging_dir.exists():
|
||||
shutil.rmtree(staging_dir, ignore_errors=True)
|
||||
owned.unregister(staging_dir)
|
||||
|
||||
prepared_at = datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
prepared = PreparedWorkspace(
|
||||
identity=identity,
|
||||
attempt_root=str(resolved_attempt_root),
|
||||
workspace_dir=str(final_workspace_dir),
|
||||
session_dir=str(final_session_dir),
|
||||
session_id=session_id,
|
||||
session_is_fresh=session_is_fresh,
|
||||
workspace_checksum=workspace_checksum,
|
||||
setup_cache_policy="isolated",
|
||||
testbed_provenance=testbed_after,
|
||||
prepared_at=prepared_at,
|
||||
)
|
||||
|
||||
prepared_data = {
|
||||
"identity": {
|
||||
"run_id": identity.run_id,
|
||||
"cell_id": identity.cell_id,
|
||||
"repetition": identity.repetition,
|
||||
"attempt": identity.attempt,
|
||||
},
|
||||
"attempt_root": prepared.attempt_root,
|
||||
"workspace_dir": prepared.workspace_dir,
|
||||
"session_dir": prepared.session_dir,
|
||||
"session_id": prepared.session_id,
|
||||
"session_is_fresh": prepared.session_is_fresh,
|
||||
"workspace_checksum": prepared.workspace_checksum,
|
||||
"setup_cache_policy": prepared.setup_cache_policy,
|
||||
"testbed_provenance": {
|
||||
"path": prepared.testbed_provenance.path,
|
||||
"branch": prepared.testbed_provenance.branch,
|
||||
"head": prepared.testbed_provenance.head,
|
||||
"status_digest": prepared.testbed_provenance.status_digest,
|
||||
"clean": prepared.testbed_provenance.clean,
|
||||
},
|
||||
"prepared_at": prepared.prepared_at,
|
||||
}
|
||||
|
||||
prepared_json_path = resolved_attempt_root / "prepared.json"
|
||||
if prepared_json_path.exists() or prepared_json_path.is_symlink():
|
||||
raise WorkspacePathError("prepared.json already exists in attempt_root")
|
||||
|
||||
try:
|
||||
with prepared_json_path.open("x", encoding="utf-8") as f:
|
||||
json.dump(prepared_data, f, indent=2)
|
||||
except FileExistsError:
|
||||
raise WorkspacePathError("prepared.json already exists in attempt_root")
|
||||
|
||||
owned.register(prepared_json_path)
|
||||
|
||||
return prepared
|
||||
|
||||
|
||||
def _rollback_owned_preparation(owned: _PreparationOwnership) -> None:
|
||||
"""Remove paths explicitly registered as created by this preparation transaction."""
|
||||
owned.rollback()
|
||||
|
|
@ -1,758 +0,0 @@
|
|||
"""
|
||||
workspace_test.py - Comprehensive tests for workspace materialization and isolation.
|
||||
|
||||
Covers exact run/cell/repetition/attempt grammar, path containment, symlink/collision rejection,
|
||||
asset mapping, prompt exclusion, checksum verification, session freshness, testbed provenance,
|
||||
and cross-attempt isolation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from scripts.agent_benchmark.manifest import (
|
||||
AssetMapping,
|
||||
Fixture,
|
||||
IopCell,
|
||||
Manifest,
|
||||
MatrixCell,
|
||||
Timeout,
|
||||
Viewport,
|
||||
digest_manifest_and_resolved_inputs,
|
||||
digest_workspace_inputs,
|
||||
load_manifest,
|
||||
)
|
||||
from scripts.agent_benchmark.workspace import (
|
||||
AttemptIdentity,
|
||||
PreparedWorkspace,
|
||||
TestbedError,
|
||||
TestbedProvenance,
|
||||
WorkspaceChecksumError,
|
||||
WorkspaceError,
|
||||
WorkspacePathError,
|
||||
WorkspaceValidationError,
|
||||
inspect_testbed_provenance,
|
||||
prepare_workspace,
|
||||
validate_attempt_identity,
|
||||
)
|
||||
|
||||
|
||||
class BaseWorkspaceTest(unittest.TestCase):
|
||||
"""Base test class providing temporary Git repositories and testbed fixtures."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self.tmp_dir_obj = tempfile.TemporaryDirectory()
|
||||
self.tmp_dir = Path(self.tmp_dir_obj.name).resolve()
|
||||
|
||||
self.repo_root = self.tmp_dir / "repo"
|
||||
self.repo_root.mkdir()
|
||||
|
||||
# Initialize git repo in repo_root
|
||||
subprocess.run(["git", "init"], cwd=self.repo_root, capture_output=True, check=True)
|
||||
subprocess.run(
|
||||
["git", "config", "user.name", "Test"], cwd=self.repo_root, capture_output=True, check=True
|
||||
)
|
||||
subprocess.run(
|
||||
["git", "config", "user.email", "test@example.com"],
|
||||
cwd=self.repo_root,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
# Initialize git repo in testbed (iop-s2)
|
||||
self.testbed_dir = self.tmp_dir / "iop-s2"
|
||||
self.testbed_dir.mkdir()
|
||||
(self.testbed_dir / "README.md").write_text("testbed content", encoding="utf-8")
|
||||
subprocess.run(["git", "init"], cwd=self.testbed_dir, capture_output=True, check=True)
|
||||
subprocess.run(
|
||||
["git", "config", "user.name", "Test"], cwd=self.testbed_dir, capture_output=True, check=True
|
||||
)
|
||||
subprocess.run(
|
||||
["git", "config", "user.email", "test@example.com"],
|
||||
cwd=self.testbed_dir,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
subprocess.run(["git", "add", "."], cwd=self.testbed_dir, capture_output=True, check=True)
|
||||
subprocess.run(
|
||||
["git", "commit", "-m", "initial testbed commit"],
|
||||
cwd=self.testbed_dir,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
# Create fixture files under repo_root
|
||||
self.fixture_dir = self.repo_root / "scripts" / "fixtures" / "bench"
|
||||
self.fixture_dir.mkdir(parents=True)
|
||||
|
||||
self.prompt_rel = "scripts/fixtures/bench/prompt.md"
|
||||
self.prompt_file = self.repo_root / self.prompt_rel
|
||||
self.prompt_content = b"# Test Prompt\nDo task.\n"
|
||||
self.prompt_file.write_bytes(self.prompt_content)
|
||||
|
||||
self.ref_rel = "scripts/fixtures/bench/ref.txt"
|
||||
self.ref_file = self.repo_root / self.ref_rel
|
||||
self.ref_content = b"Reference data content\n"
|
||||
self.ref_file.write_bytes(self.ref_content)
|
||||
|
||||
subprocess.run(["git", "add", "."], cwd=self.repo_root, capture_output=True, check=True)
|
||||
subprocess.run(
|
||||
["git", "commit", "-m", "add fixture files"],
|
||||
cwd=self.repo_root,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
|
||||
# Assets list (initially only ref.txt is an asset, prompt.md is separate prompt file)
|
||||
self.assets = [
|
||||
AssetMapping(source=self.ref_rel, workspace_path="data/ref.txt", content=self.ref_content)
|
||||
]
|
||||
self.fixture_checksum = digest_workspace_inputs(self.assets)
|
||||
|
||||
self.run_id = "run-20260809T161730Z-0123456789ab"
|
||||
self.output_root_rel = "agent-test/runs/bench-01"
|
||||
|
||||
self.manifest_raw = {
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"repetitions": 2,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5,
|
||||
},
|
||||
"viewports": [{"id": "desktop_1080", "width": 1920, "height": 1080}],
|
||||
"rubric_version": "landing-quality-v1",
|
||||
"evaluator": {"caller": "codex", "iop": {"request_model": "judge", "requested_effort": "high", "route_kind": "direct", "route_id": "judge", "expected_bindings": [{"stage": "request", "model": "judge", "effort": "high"}]}},
|
||||
"output_root": self.output_root_rel,
|
||||
"fixture": {
|
||||
"version": "v1.0",
|
||||
"prompt": self.prompt_rel,
|
||||
"assets": [{"source": self.ref_rel, "workspace_path": "data/ref.txt"}],
|
||||
"checksum": self.fixture_checksum,
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "cell-1",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-4-20250514",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-direct",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-4-20250514", "effort": "high"}
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
"id": "cell-2",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-2.0-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "agy-direct",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-2.0-flash", "effort": "high"}
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
self.manifest_file = self.repo_root / "manifest.json"
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
self.manifest = load_manifest(self.manifest_file, repo_root=self.repo_root)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.tmp_dir_obj.cleanup()
|
||||
|
||||
def make_attempt_root(
|
||||
self, cell_id: str = "cell-1", repetition: int = 1, attempt: int = 1
|
||||
) -> tuple[AttemptIdentity, Path]:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id=cell_id, repetition=repetition, attempt=attempt
|
||||
)
|
||||
rep_segment = f"repetition-{repetition:04d}"
|
||||
att_segment = f"attempt-{attempt:06d}"
|
||||
attempt_root = (
|
||||
self.repo_root
|
||||
/ self.output_root_rel
|
||||
/ self.run_id
|
||||
/ "cells"
|
||||
/ cell_id
|
||||
/ rep_segment
|
||||
/ att_segment
|
||||
)
|
||||
attempt_root.mkdir(parents=True, exist_ok=True)
|
||||
return identity, attempt_root
|
||||
|
||||
|
||||
class TestAttemptIdentityValidation(BaseWorkspaceTest):
|
||||
"""Tests for AttemptIdentity format and boundary validation."""
|
||||
|
||||
def test_valid_identity(self) -> None:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=1, attempt=1
|
||||
)
|
||||
validate_attempt_identity(identity, self.manifest)
|
||||
|
||||
def test_invalid_run_id_rejected(self) -> None:
|
||||
bad_run_ids = [
|
||||
"invalid_run_id",
|
||||
"run-20260809161730Z-0123456789ab", # missing T
|
||||
"run-20260809T161730Z-0123456789aG", # uppercase G
|
||||
"run-20260809T161730Z-short",
|
||||
]
|
||||
for run_id in bad_run_ids:
|
||||
identity = AttemptIdentity(run_id=run_id, cell_id="cell-1", repetition=1, attempt=1)
|
||||
with self.assertRaises(WorkspaceValidationError):
|
||||
validate_attempt_identity(identity, self.manifest)
|
||||
|
||||
def test_invalid_cell_id_rejected(self) -> None:
|
||||
# Invalid format or missing from manifest
|
||||
identity_bad_fmt = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="-invalid", repetition=1, attempt=1
|
||||
)
|
||||
with self.assertRaises(WorkspaceValidationError):
|
||||
validate_attempt_identity(identity_bad_fmt, self.manifest)
|
||||
|
||||
identity_missing = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-nonexistent", repetition=1, attempt=1
|
||||
)
|
||||
with self.assertRaises(WorkspaceValidationError):
|
||||
validate_attempt_identity(identity_missing, self.manifest)
|
||||
|
||||
def test_invalid_repetition_rejected(self) -> None:
|
||||
# Non-positive or exceeding manifest repetitions (2)
|
||||
for rep in [0, -1, 3, True]:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=rep, attempt=1
|
||||
)
|
||||
with self.assertRaises(WorkspaceValidationError):
|
||||
validate_attempt_identity(identity, self.manifest)
|
||||
|
||||
def test_invalid_attempt_rejected(self) -> None:
|
||||
for att in [0, -1, True]:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=1, attempt=att
|
||||
)
|
||||
with self.assertRaises(WorkspaceValidationError):
|
||||
validate_attempt_identity(identity, self.manifest)
|
||||
|
||||
|
||||
class TestAttemptRootPathRules(BaseWorkspaceTest):
|
||||
"""Tests path rules for attempt_root."""
|
||||
|
||||
def test_attempt_root_does_not_exist(self) -> None:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=1, attempt=1
|
||||
)
|
||||
non_existent = (
|
||||
self.repo_root
|
||||
/ self.output_root_rel
|
||||
/ self.run_id
|
||||
/ "cells"
|
||||
/ "cell-1"
|
||||
/ "repetition-0001"
|
||||
/ "attempt-000001"
|
||||
)
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, non_existent, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
def test_attempt_root_is_file(self) -> None:
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
attempt_root.rmdir()
|
||||
attempt_root.write_text("file instead of dir", encoding="utf-8")
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
def test_attempt_root_is_symlink(self) -> None:
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
attempt_root.rmdir()
|
||||
real_target = self.tmp_dir / "real_target"
|
||||
real_target.mkdir()
|
||||
attempt_root.symlink_to(real_target)
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
def test_attempt_root_parent_is_symlink(self) -> None:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=1, attempt=1
|
||||
)
|
||||
parent_dir = (
|
||||
self.repo_root
|
||||
/ self.output_root_rel
|
||||
/ self.run_id
|
||||
/ "cells"
|
||||
/ "cell-1"
|
||||
/ "repetition-0001"
|
||||
)
|
||||
parent_dir.mkdir(parents=True, exist_ok=True)
|
||||
real_parent = self.tmp_dir / "real_parent"
|
||||
real_parent.mkdir()
|
||||
attempt_root_real = real_parent / "attempt-000001"
|
||||
attempt_root_real.mkdir()
|
||||
|
||||
symlink_att = parent_dir / "attempt-000001"
|
||||
symlink_att.symlink_to(attempt_root_real)
|
||||
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, symlink_att, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
def test_attempt_root_not_empty(self) -> None:
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
(attempt_root / "existing.txt").write_text("existing", encoding="utf-8")
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
def test_attempt_root_canonical_path_mismatch(self) -> None:
|
||||
identity = AttemptIdentity(
|
||||
run_id=self.run_id, cell_id="cell-1", repetition=1, attempt=1
|
||||
)
|
||||
wrong_path = (
|
||||
self.repo_root
|
||||
/ self.output_root_rel
|
||||
/ self.run_id
|
||||
/ "cells"
|
||||
/ "cell-1"
|
||||
/ "repetition-0002" # mismatched repetition
|
||||
/ "attempt-000001"
|
||||
)
|
||||
wrong_path.mkdir(parents=True, exist_ok=True)
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(self.manifest, wrong_path, identity, repo_root=self.repo_root)
|
||||
|
||||
def test_exclusive_child_collision(self) -> None:
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
(attempt_root / "workspace").mkdir()
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
|
||||
class TestWorkspaceMaterialization(BaseWorkspaceTest):
|
||||
"""Tests for workspace asset copying, prompt exclusion, checksums, and metadata."""
|
||||
|
||||
def test_successful_workspace_preparation(self) -> None:
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
prepared = prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
self.assertEqual(prepared.identity, identity)
|
||||
self.assertEqual(prepared.workspace_checksum, self.manifest.fixture.checksum)
|
||||
self.assertTrue(prepared.session_is_fresh)
|
||||
self.assertEqual(prepared.setup_cache_policy, "isolated")
|
||||
|
||||
# Verify directories exist
|
||||
ws_path = Path(prepared.workspace_dir)
|
||||
session_path = Path(prepared.session_dir)
|
||||
self.assertTrue(ws_path.is_dir())
|
||||
self.assertTrue(session_path.is_dir())
|
||||
|
||||
# Verify session directory is empty
|
||||
self.assertEqual(list(session_path.iterdir()), [])
|
||||
|
||||
# Verify asset materialization
|
||||
asset_file = ws_path / "data" / "ref.txt"
|
||||
self.assertTrue(asset_file.is_file())
|
||||
self.assertEqual(asset_file.read_bytes(), self.ref_content)
|
||||
|
||||
# Verify prepared.json content
|
||||
prep_json = Path(prepared.attempt_root) / "prepared.json"
|
||||
self.assertTrue(prep_json.is_file())
|
||||
data = json.loads(prep_json.read_text(encoding="utf-8"))
|
||||
self.assertEqual(data["session_id"], prepared.session_id)
|
||||
self.assertEqual(data["workspace_checksum"], prepared.workspace_checksum)
|
||||
|
||||
def test_prompt_exclusion_when_not_declared(self) -> None:
|
||||
# Prompt file exists in fixture.prompt, but is not listed in fixture.assets
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
prepared = prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
ws_path = Path(prepared.workspace_dir)
|
||||
|
||||
# Prompt should not exist in workspace
|
||||
prompt_ws = ws_path / self.prompt_rel
|
||||
self.assertFalse(prompt_ws.exists())
|
||||
|
||||
def test_prompt_included_when_declared_as_asset(self) -> None:
|
||||
# Update manifest to declare prompt.md as an asset too
|
||||
new_assets = [
|
||||
{"source": self.ref_rel, "workspace_path": "data/ref.txt"},
|
||||
{"source": self.prompt_rel, "workspace_path": "prompt.md"},
|
||||
]
|
||||
asset_objs = [
|
||||
AssetMapping(source=self.ref_rel, workspace_path="data/ref.txt", content=self.ref_content),
|
||||
AssetMapping(source=self.prompt_rel, workspace_path="prompt.md", content=self.prompt_content),
|
||||
]
|
||||
checksum = digest_workspace_inputs(asset_objs)
|
||||
self.manifest_raw["fixture"]["assets"] = new_assets
|
||||
self.manifest_raw["fixture"]["checksum"] = checksum
|
||||
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
manifest = load_manifest(self.manifest_file, repo_root=self.repo_root)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
prepared = prepare_workspace(manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
ws_path = Path(prepared.workspace_dir)
|
||||
|
||||
prompt_ws = ws_path / "prompt.md"
|
||||
self.assertTrue(prompt_ws.is_file())
|
||||
self.assertEqual(prompt_ws.read_bytes(), self.prompt_content)
|
||||
|
||||
def test_fixture_checksum_mismatch_rejected(self) -> None:
|
||||
# Corrupt declared checksum in manifest
|
||||
self.manifest_raw["fixture"]["checksum"] = "sha256:" + "0" * 64
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
|
||||
# Manually create manifest with corrupt checksum to bypass load_manifest validation
|
||||
dummy_manifest = Manifest(
|
||||
pipeline_version=self.manifest.pipeline_version,
|
||||
environment=self.manifest.environment,
|
||||
testbed=self.manifest.testbed,
|
||||
repetitions=self.manifest.repetitions,
|
||||
session_policy=self.manifest.session_policy,
|
||||
setup_cache_policy=self.manifest.setup_cache_policy,
|
||||
timeout=self.manifest.timeout,
|
||||
viewports=self.manifest.viewports,
|
||||
rubric_version=self.manifest.rubric_version,
|
||||
evaluator=self.manifest.evaluator,
|
||||
output_root=self.manifest.output_root,
|
||||
fixture=Fixture(
|
||||
version=self.manifest.fixture.version,
|
||||
prompt=self.manifest.fixture.prompt,
|
||||
assets=self.manifest.fixture.assets,
|
||||
checksum="sha256:" + "0" * 64,
|
||||
prompt_content=self.manifest.fixture.prompt_content,
|
||||
),
|
||||
matrix=self.manifest.matrix,
|
||||
digest=self.manifest.digest,
|
||||
)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
with self.assertRaises(WorkspaceChecksumError):
|
||||
prepare_workspace(dummy_manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
def test_symlink_asset_source_rejected(self) -> None:
|
||||
symlink_source = self.repo_root / "scripts" / "fixtures" / "bench" / "symlink.txt"
|
||||
symlink_source.symlink_to(self.ref_file)
|
||||
|
||||
symlink_rel = "scripts/fixtures/bench/symlink.txt"
|
||||
self.manifest_raw["fixture"]["assets"] = [
|
||||
{"source": symlink_rel, "workspace_path": "data/symlink.txt"}
|
||||
]
|
||||
asset_objs = [
|
||||
AssetMapping(source=symlink_rel, workspace_path="data/symlink.txt", content=self.ref_content)
|
||||
]
|
||||
self.manifest_raw["fixture"]["checksum"] = digest_workspace_inputs(asset_objs)
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
|
||||
# Create manifest directly to bypass load_manifest symlink check
|
||||
dummy_manifest = Manifest(
|
||||
pipeline_version=self.manifest.pipeline_version,
|
||||
environment=self.manifest.environment,
|
||||
testbed=self.manifest.testbed,
|
||||
repetitions=self.manifest.repetitions,
|
||||
session_policy=self.manifest.session_policy,
|
||||
setup_cache_policy=self.manifest.setup_cache_policy,
|
||||
timeout=self.manifest.timeout,
|
||||
viewports=self.manifest.viewports,
|
||||
rubric_version=self.manifest.rubric_version,
|
||||
evaluator=self.manifest.evaluator,
|
||||
output_root=self.manifest.output_root,
|
||||
fixture=Fixture(
|
||||
version=self.manifest.fixture.version,
|
||||
prompt=self.manifest.fixture.prompt,
|
||||
assets=(AssetMapping(source=symlink_rel, workspace_path="data/symlink.txt", content=self.ref_content),),
|
||||
checksum=self.manifest_raw["fixture"]["checksum"],
|
||||
prompt_content=self.manifest.fixture.prompt_content,
|
||||
),
|
||||
matrix=self.manifest.matrix,
|
||||
digest=self.manifest.digest,
|
||||
)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(dummy_manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
def test_escaping_workspace_path_rejected(self) -> None:
|
||||
bad_assets = [
|
||||
{"source": self.ref_rel, "workspace_path": "../data/ref.txt"},
|
||||
]
|
||||
self.manifest_raw["fixture"]["assets"] = bad_assets
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
|
||||
with self.assertRaises(Exception):
|
||||
load_manifest(self.manifest_file, repo_root=self.repo_root)
|
||||
|
||||
def test_source_drift_failure_leaves_attempt_root_empty_and_retryable(self) -> None:
|
||||
"""R1: Mutate a fixture source after manifest load, prove rollback and retry."""
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
|
||||
# Capture original ref.txt content
|
||||
original_ref_content = self.ref_file.read_bytes()
|
||||
|
||||
# Mutate the source file after manifest load (simulates source drift)
|
||||
self.ref_file.write_bytes(b"corrupted reference data\n")
|
||||
|
||||
with self.assertRaises(WorkspaceChecksumError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
# Assert the attempt root is empty (rollback succeeded)
|
||||
self.assertEqual(list(attempt_root.iterdir()), [])
|
||||
|
||||
# Restore the original source
|
||||
self.ref_file.write_bytes(original_ref_content)
|
||||
|
||||
# Prove the same attempt root can succeed on retry
|
||||
# (need to recreate it since rollback removed it)
|
||||
identity2, attempt_root2 = self.make_attempt_root()
|
||||
prepared = prepare_workspace(
|
||||
self.manifest, attempt_root2, identity2, repo_root=self.repo_root
|
||||
)
|
||||
self.assertEqual(prepared.identity, identity2)
|
||||
self.assertTrue(Path(prepared.workspace_dir).is_dir())
|
||||
self.assertTrue(Path(prepared.session_dir).is_dir())
|
||||
|
||||
def test_postflight_failure_leaves_attempt_root_empty(self) -> None:
|
||||
"""R1: Deterministic mocked postflight failure proves rollback of all owned entries."""
|
||||
import unittest.mock
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
|
||||
# Mock inspect_testbed_provenance to succeed on first call (preflight) but fail on second (postflight)
|
||||
original_inspect = inspect_testbed_provenance
|
||||
call_count = {"n": 0}
|
||||
|
||||
def mock_inspect(path):
|
||||
call_count["n"] += 1
|
||||
if call_count["n"] == 1:
|
||||
return original_inspect(path) # preflight succeeds
|
||||
# postflight raises TestbedError
|
||||
from scripts.agent_benchmark.workspace import TestbedError as TE
|
||||
raise TE("mocked postflight: testbed modified")
|
||||
|
||||
with unittest.mock.patch(
|
||||
"scripts.agent_benchmark.workspace.inspect_testbed_provenance",
|
||||
side_effect=mock_inspect,
|
||||
):
|
||||
with self.assertRaises(TestbedError):
|
||||
prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
# Assert the attempt root is empty (rollback succeeded)
|
||||
self.assertEqual(list(attempt_root.iterdir()), [])
|
||||
|
||||
# Assert prepared.json does not exist
|
||||
prepared_json = attempt_root / "prepared.json"
|
||||
self.assertFalse(prepared_json.exists())
|
||||
|
||||
def test_ancestor_destination_collision_rejected_before_mutation(self) -> None:
|
||||
"""R1: Asset destinations with ancestor/file conflict are rejected before mutation."""
|
||||
new_assets = [
|
||||
{"source": self.ref_rel, "workspace_path": "data"},
|
||||
{"source": self.prompt_rel, "workspace_path": "data/prompt.md"},
|
||||
]
|
||||
asset_objs = [
|
||||
AssetMapping(source=self.ref_rel, workspace_path="data", content=self.ref_content),
|
||||
AssetMapping(source=self.prompt_rel, workspace_path="data/prompt.md", content=self.prompt_content),
|
||||
]
|
||||
checksum = digest_workspace_inputs(asset_objs)
|
||||
self.manifest_raw["fixture"]["assets"] = new_assets
|
||||
self.manifest_raw["fixture"]["checksum"] = checksum
|
||||
self.manifest_file.write_text(json.dumps(self.manifest_raw, indent=2), encoding="utf-8")
|
||||
manifest = load_manifest(self.manifest_file, repo_root=self.repo_root)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
# Assert attempt_root remains completely empty (no staging, workspace, or session created)
|
||||
self.assertEqual(list(attempt_root.iterdir()), [])
|
||||
|
||||
def test_concurrent_collision_preserves_unrelated_entries(self) -> None:
|
||||
"""R1: Concurrent collision content not created by this preparation is preserved on rollback."""
|
||||
import unittest.mock
|
||||
import scripts.agent_benchmark.workspace
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
|
||||
def mock_verify_with_collision(staging, manifest):
|
||||
# Simulate a caller/external collision creating workspace/caller-owned.txt after validation
|
||||
caller_ws = attempt_root / "workspace"
|
||||
caller_ws.mkdir(exist_ok=True)
|
||||
sentinel = caller_ws / "caller-owned.txt"
|
||||
sentinel.write_text("caller owned content", encoding="utf-8")
|
||||
raise TestbedError("mocked failure during preparation")
|
||||
|
||||
with unittest.mock.patch(
|
||||
"scripts.agent_benchmark.workspace._verify_staged_preparation",
|
||||
side_effect=mock_verify_with_collision,
|
||||
):
|
||||
with self.assertRaises(TestbedError):
|
||||
prepare_workspace(self.manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
# Assert caller-owned collision file was preserved on rollback
|
||||
sentinel = attempt_root / "workspace" / "caller-owned.txt"
|
||||
self.assertTrue(sentinel.is_file())
|
||||
self.assertEqual(sentinel.read_text(encoding="utf-8"), "caller owned content")
|
||||
|
||||
def test_empty_publication_collision_preserves_unrelated_directory(self) -> None:
|
||||
"""R1: Empty concurrent collision directory created before final publication is preserved on rollback."""
|
||||
import unittest.mock
|
||||
import scripts.agent_benchmark.workspace
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
|
||||
empty_inode: int | None = None
|
||||
orig_publish = scripts.agent_benchmark.workspace._publish_owned_directory
|
||||
|
||||
def mock_publish_with_empty_collision(staging_dir, final_dir, owned, collision_message):
|
||||
nonlocal empty_inode
|
||||
if final_dir.name == "workspace":
|
||||
final_dir.mkdir(exist_ok=False)
|
||||
empty_inode = final_dir.stat().st_ino
|
||||
return orig_publish(staging_dir, final_dir, owned, collision_message)
|
||||
|
||||
with unittest.mock.patch(
|
||||
"scripts.agent_benchmark.workspace._publish_owned_directory",
|
||||
side_effect=mock_publish_with_empty_collision,
|
||||
):
|
||||
with self.assertRaises(WorkspacePathError):
|
||||
prepare_workspace(self.manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
caller_ws = attempt_root / "workspace"
|
||||
self.assertTrue(caller_ws.is_dir())
|
||||
self.assertIsNotNone(empty_inode)
|
||||
self.assertEqual(caller_ws.stat().st_ino, empty_inode)
|
||||
self.assertEqual(list(caller_ws.iterdir()), [])
|
||||
self.assertFalse((attempt_root / "session").exists())
|
||||
self.assertFalse((attempt_root / "prepared.json").exists())
|
||||
self.assertEqual(list(attempt_root.iterdir()), [caller_ws])
|
||||
|
||||
|
||||
class TestTestbedProvenanceAndNonMutation(BaseWorkspaceTest):
|
||||
"""Tests for runtime testbed provenance checking and non-mutation."""
|
||||
|
||||
def test_clean_testbed_provenance(self) -> None:
|
||||
prov = inspect_testbed_provenance(self.testbed_dir)
|
||||
self.assertTrue(prov.clean)
|
||||
self.assertTrue(prov.status_digest.startswith("sha256:"))
|
||||
self.assertTrue(len(prov.head) > 0)
|
||||
|
||||
def test_dirty_testbed_rejected(self) -> None:
|
||||
# Create an uncommitted file in testbed_dir
|
||||
(self.testbed_dir / "dirty.txt").write_text("dirty", encoding="utf-8")
|
||||
|
||||
with self.assertRaises(TestbedError):
|
||||
inspect_testbed_provenance(self.testbed_dir)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
with self.assertRaises(TestbedError):
|
||||
prepare_workspace(self.manifest, attempt_root, identity, repo_root=self.repo_root)
|
||||
|
||||
def test_testbed_unaffected_by_preparation(self) -> None:
|
||||
prov_before = inspect_testbed_provenance(self.testbed_dir)
|
||||
|
||||
identity, attempt_root = self.make_attempt_root()
|
||||
prepared = prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
|
||||
prov_after = inspect_testbed_provenance(self.testbed_dir)
|
||||
self.assertEqual(prov_before, prov_after)
|
||||
|
||||
# Assert no file from testbed_dir was copied into workspace_dir
|
||||
ws_files = list(Path(prepared.workspace_dir).rglob("*"))
|
||||
ws_rel_paths = {f.name for f in ws_files}
|
||||
self.assertNotIn("README.md", ws_rel_paths) # testbed README.md is not in workspace
|
||||
|
||||
|
||||
class TestCrossAttemptIsolation(BaseWorkspaceTest):
|
||||
"""API-2: Prove cross-attempt isolation and source integrity."""
|
||||
|
||||
def test_cross_attempt_isolation_and_source_integrity(self) -> None:
|
||||
# Prepare 4 attempt roots (2 cells x 2 repetitions)
|
||||
testbed_before = inspect_testbed_provenance(self.testbed_dir)
|
||||
|
||||
attempts_config = [
|
||||
("cell-1", 1, 1),
|
||||
("cell-1", 2, 1),
|
||||
("cell-2", 1, 1),
|
||||
("cell-2", 2, 1),
|
||||
]
|
||||
|
||||
prepared_list: list[PreparedWorkspace] = []
|
||||
for cell_id, rep, att in attempts_config:
|
||||
identity, attempt_root = self.make_attempt_root(
|
||||
cell_id=cell_id, repetition=rep, attempt=att
|
||||
)
|
||||
prep = prepare_workspace(
|
||||
self.manifest, attempt_root, identity, repo_root=self.repo_root
|
||||
)
|
||||
prepared_list.append(prep)
|
||||
|
||||
# 1. Assert four distinct workspace/session identities
|
||||
session_ids = {p.session_id for p in prepared_list}
|
||||
self.assertEqual(len(session_ids), 4)
|
||||
|
||||
attempt_roots = {p.attempt_root for p in prepared_list}
|
||||
self.assertEqual(len(attempt_roots), 4)
|
||||
|
||||
# 2. Assert identical initial workspace digests matching manifest.fixture.checksum
|
||||
workspace_checksums = {p.workspace_checksum for p in prepared_list}
|
||||
self.assertEqual(workspace_checksums, {self.manifest.fixture.checksum})
|
||||
|
||||
# 3. Mutate one session (session #1) and prove peer sessions remain empty
|
||||
session1_path = Path(prepared_list[0].session_dir)
|
||||
sentinel = session1_path / "history-sentinel"
|
||||
sentinel.write_text("owned", encoding="utf-8")
|
||||
self.assertTrue(sentinel.is_file())
|
||||
|
||||
for peer in prepared_list[1:]:
|
||||
self.assertEqual(list(Path(peer.session_dir).iterdir()), [])
|
||||
|
||||
# 4. Capture testbed branch/HEAD/status digest before and after and assert equality
|
||||
testbed_after = inspect_testbed_provenance(self.testbed_dir)
|
||||
self.assertEqual(testbed_before, testbed_after)
|
||||
|
||||
# 5. Assert no path under testbed appears beneath any workspace
|
||||
testbed_files = {f.name for f in self.testbed_dir.rglob("*") if f.is_file()}
|
||||
for prep in prepared_list:
|
||||
ws_files = {f.name for f in Path(prep.workspace_dir).rglob("*") if f.is_file()}
|
||||
# Intersection of testbed files and workspace files should be empty
|
||||
# (testbed contains README.md; workspace contains data/ref.txt)
|
||||
self.assertEqual(testbed_files.intersection(ws_files), set())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -1,319 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Public CLI for the agent comparison benchmark manifest.
|
||||
|
||||
Usage:
|
||||
python3 scripts/agent_comparison_benchmark.py validate --manifest PATH
|
||||
python3 scripts/agent_comparison_benchmark.py preflight --manifest PATH
|
||||
python3 scripts/agent_comparison_benchmark.py run --manifest PATH
|
||||
python3 scripts/agent_comparison_benchmark.py resume --manifest PATH --run-id RUN_ID
|
||||
python3 scripts/agent_comparison_benchmark.py status --manifest PATH --run-id RUN_ID
|
||||
python3 scripts/agent_comparison_benchmark.py score --manifest PATH --run-id RUN_ID
|
||||
|
||||
Exits:
|
||||
0 - manifest is valid or every matrix preflight cell is ready
|
||||
64 - usage error (missing args, bad flags)
|
||||
69 - validation/state failed or preflight is blocked
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from collections.abc import Mapping
|
||||
|
||||
# Ensure the repo root is on sys.path for imports.
|
||||
_REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
if str(_REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_REPO_ROOT))
|
||||
|
||||
from scripts.agent_benchmark.manifest import CALLER_ENUM, ManifestError, load_manifest
|
||||
from scripts.agent_benchmark.attempts import (
|
||||
AttemptStateError,
|
||||
CapabilityUnavailable,
|
||||
ExecutionAdapter,
|
||||
RunStore,
|
||||
preflight_manifest,
|
||||
run_slots,
|
||||
)
|
||||
from scripts.agent_benchmark.live_iop import (
|
||||
build_live_adapter_registry,
|
||||
build_live_scoring_adapter,
|
||||
)
|
||||
from scripts.agent_benchmark.scoring import ScoringError, score_run
|
||||
from scripts.agent_benchmark.reporting import ReportError, publish_report
|
||||
from scripts.agent_benchmark.workspace import prepare_workspace
|
||||
|
||||
EXIT_VALID = 0
|
||||
EXIT_USAGE = 64
|
||||
EXIT_INVALID = 69
|
||||
|
||||
|
||||
def build_adapter_registry(
|
||||
environment: Mapping[str, str] | None = None,
|
||||
) -> dict[str, ExecutionAdapter]:
|
||||
"""Build the exact three-caller registry from explicit live inputs only."""
|
||||
registry = build_live_adapter_registry(
|
||||
os.environ if environment is None else environment
|
||||
)
|
||||
if tuple(registry) != CALLER_ENUM:
|
||||
raise RuntimeError("caller adapter registry is invalid")
|
||||
return registry
|
||||
|
||||
|
||||
class _SanitizedArgumentParser(argparse.ArgumentParser):
|
||||
def error(self, message: str) -> None:
|
||||
print("error: invalid usage", file=sys.stderr)
|
||||
sys.exit(EXIT_USAGE)
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
parser = _SanitizedArgumentParser(
|
||||
prog="agent_comparison_benchmark",
|
||||
description="Agent comparison benchmark manifest tools.",
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
p_validate = sub.add_parser(
|
||||
"validate",
|
||||
help="Validate a benchmark manifest JSON file.",
|
||||
)
|
||||
p_validate.add_argument(
|
||||
"--manifest",
|
||||
required=True,
|
||||
help="Path to the manifest JSON file.",
|
||||
)
|
||||
for command in ("preflight", "run", "resume", "status", "score", "report"):
|
||||
entry = sub.add_parser(command, help=f"Safely {command} benchmark state.")
|
||||
entry.add_argument("--manifest", required=True, help="Path to the manifest JSON file.")
|
||||
if command in {"resume", "status", "score", "report"}:
|
||||
entry.add_argument("--run-id", required=True, help="Harness-generated run id.")
|
||||
if command == "resume":
|
||||
entry.add_argument("--retry-failed", action="store_true")
|
||||
if command == "score":
|
||||
entry.add_argument("--retry-scoring-failed", action="store_true")
|
||||
return parser
|
||||
|
||||
|
||||
def _cmd_validate(args: argparse.Namespace) -> int:
|
||||
manifest_path = Path(args.manifest)
|
||||
if not manifest_path.is_file():
|
||||
print("error: manifest is unavailable", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
try:
|
||||
load_manifest(manifest_path)
|
||||
except ManifestError as exc:
|
||||
print(f"error: {exc}", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
except Exception:
|
||||
print("error: manifest validation failed", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
print("ok: manifest is valid")
|
||||
return EXIT_VALID
|
||||
|
||||
|
||||
def _cmd_state(args: argparse.Namespace) -> int:
|
||||
try:
|
||||
manifest_path = Path(args.manifest)
|
||||
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
|
||||
raw = manifest_path.read_bytes()
|
||||
store = RunStore(_REPO_ROOT)
|
||||
if args.command == "run":
|
||||
run = store.create(manifest, raw)
|
||||
else:
|
||||
run = store.open(manifest, args.run_id, raw)
|
||||
if args.command == "status":
|
||||
status = store.status(run, manifest)
|
||||
attempts = status["attempts"]
|
||||
outcomes = status["outcomes"]
|
||||
attempt_summary = " ".join(
|
||||
f"{state}={attempts[state]}"
|
||||
for state in (
|
||||
"completed", "timed_out", "cancelled", "interrupted", "running"
|
||||
)
|
||||
)
|
||||
axes = " ".join(
|
||||
f"{axis}_{name}={outcomes[axis][name]}"
|
||||
for axis in ("product", "harness", "process", "artifact")
|
||||
for name in outcomes[axis]
|
||||
)
|
||||
print(
|
||||
f"ok: status run_id={run.run_id} "
|
||||
f"unresolved={outcomes['unresolved']} {attempt_summary} {axes}"
|
||||
)
|
||||
return EXIT_VALID
|
||||
|
||||
completed = run_slots(
|
||||
store,
|
||||
run,
|
||||
manifest,
|
||||
adapters=build_adapter_registry(),
|
||||
prepare=lambda bound_manifest, attempt: prepare_workspace(
|
||||
bound_manifest,
|
||||
attempt.root,
|
||||
attempt.identity,
|
||||
repo_root=_REPO_ROOT,
|
||||
),
|
||||
retry_failed=bool(getattr(args, "retry_failed", False)),
|
||||
)
|
||||
status = store.status(run, manifest)
|
||||
preflight = status["preflight"]
|
||||
preflight_summary = (
|
||||
f"run_id={run.run_id} status={preflight['latest_status']} "
|
||||
f"ready={preflight['ready']} "
|
||||
f"registration_required={preflight['registration_required']} "
|
||||
f"implementation_gap={preflight['implementation_gap']}"
|
||||
)
|
||||
if preflight["latest_status"] != "ready":
|
||||
print("error: preflight blocked " + preflight_summary, file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
attempts = status["attempts"]
|
||||
attempt_summary = " ".join(
|
||||
f"{state}={attempts[state]}"
|
||||
for state in (
|
||||
"completed", "timed_out", "cancelled", "interrupted", "running"
|
||||
)
|
||||
)
|
||||
outcomes = status["outcomes"]
|
||||
unresolved = outcomes["unresolved"]
|
||||
axes = " ".join(
|
||||
f"{prefix}_{name}={counts[name]}"
|
||||
for prefix, counts in (
|
||||
("product", outcomes["product"]),
|
||||
("harness", outcomes["harness"]),
|
||||
("process", outcomes["process"]),
|
||||
("artifact", outcomes["artifact"]),
|
||||
)
|
||||
for name in counts
|
||||
)
|
||||
summary = (
|
||||
f"run_id={run.run_id} executed={len(completed)} "
|
||||
f"unresolved={unresolved} {attempt_summary} {axes}"
|
||||
)
|
||||
if unresolved:
|
||||
print("error: benchmark execution failed " + summary, file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
print(f"ok: {args.command} " + summary)
|
||||
return EXIT_VALID
|
||||
except CapabilityUnavailable:
|
||||
print("error: capability unavailable", file=sys.stderr)
|
||||
except Exception:
|
||||
print("error: benchmark state is unavailable", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
|
||||
def _cmd_preflight(args: argparse.Namespace) -> int:
|
||||
try:
|
||||
manifest_path = Path(args.manifest)
|
||||
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
|
||||
raw = manifest_path.read_bytes()
|
||||
run, record = preflight_manifest(
|
||||
RunStore(_REPO_ROOT),
|
||||
manifest,
|
||||
raw,
|
||||
adapters=build_adapter_registry(),
|
||||
)
|
||||
counts = {status: 0 for status in ("ready", "registration_required", "implementation_gap")}
|
||||
for result in record["results"]:
|
||||
counts[result["status"]] += 1
|
||||
summary = (
|
||||
f"run_id={run.run_id} status={record['status']} "
|
||||
f"ready={counts['ready']} "
|
||||
f"registration_required={counts['registration_required']} "
|
||||
f"implementation_gap={counts['implementation_gap']}"
|
||||
)
|
||||
if record["status"] == "ready":
|
||||
print("ok: preflight " + summary)
|
||||
return EXIT_VALID
|
||||
print("error: preflight blocked " + summary, file=sys.stderr)
|
||||
except CapabilityUnavailable:
|
||||
print("error: capability unavailable", file=sys.stderr)
|
||||
except Exception:
|
||||
print("error: benchmark preflight is unavailable", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
|
||||
def _cmd_score(args: argparse.Namespace) -> int:
|
||||
run_id = str(args.run_id)
|
||||
try:
|
||||
manifest_path = Path(args.manifest)
|
||||
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
|
||||
raw = manifest_path.read_bytes()
|
||||
store = RunStore(_REPO_ROOT)
|
||||
run = store.open(manifest, run_id, raw)
|
||||
summary = score_run(
|
||||
store,
|
||||
run,
|
||||
manifest,
|
||||
adapter=build_live_scoring_adapter(os.environ),
|
||||
retry_scoring_failed=bool(args.retry_scoring_failed),
|
||||
)
|
||||
counts = (
|
||||
f"run_id={summary.run_id} scored={summary.scored} "
|
||||
f"unscored={summary.unscored} "
|
||||
f"scoring_failed={summary.scoring_failed} blocked={summary.blocked}"
|
||||
)
|
||||
if summary.scoring_failed or summary.blocked:
|
||||
print("error: benchmark scoring failed " + counts, file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
print("ok: score " + counts)
|
||||
return EXIT_VALID
|
||||
except (ManifestError, ScoringError, AttemptStateError, OSError):
|
||||
print(
|
||||
f"error: benchmark scoring is unavailable run_id={run_id}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
except Exception:
|
||||
print(
|
||||
f"error: benchmark scoring is unavailable run_id={run_id}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return EXIT_INVALID
|
||||
|
||||
|
||||
def _cmd_report(args: argparse.Namespace) -> int:
|
||||
run_id = str(args.run_id)
|
||||
try:
|
||||
manifest_path = Path(args.manifest)
|
||||
manifest = load_manifest(manifest_path, repo_root=_REPO_ROOT)
|
||||
raw = manifest_path.read_bytes()
|
||||
store = RunStore(_REPO_ROOT)
|
||||
run = store.open(manifest, run_id, raw)
|
||||
path = publish_report(store, run, manifest)
|
||||
rel = str(path.relative_to(_REPO_ROOT))
|
||||
print(f"ok: report run_id={run_id} path={rel}")
|
||||
return EXIT_VALID
|
||||
except (ManifestError, ReportError, AttemptStateError, OSError):
|
||||
print("error: benchmark report is unavailable", file=sys.stderr)
|
||||
except Exception:
|
||||
print("error: benchmark report is unavailable", file=sys.stderr)
|
||||
return EXIT_INVALID
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = _build_parser()
|
||||
try:
|
||||
args = parser.parse_args(argv)
|
||||
except SystemExit as exc:
|
||||
return exc.code if isinstance(exc.code, int) else EXIT_USAGE
|
||||
|
||||
if args.command == "validate":
|
||||
return _cmd_validate(args)
|
||||
if args.command == "preflight":
|
||||
return _cmd_preflight(args)
|
||||
if args.command == "score":
|
||||
return _cmd_score(args)
|
||||
if args.command == "report":
|
||||
return _cmd_report(args)
|
||||
if args.command in {"run", "resume", "status"}:
|
||||
return _cmd_state(args)
|
||||
|
||||
return EXIT_USAGE
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "landing-quality-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-01-direct-preflight",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "claude-sonnet-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "max",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-sonnet-5",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "claude-gemini-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "claude-gpt-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "agy-gemini-direct",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "codex-gpt-direct",
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,178 +0,0 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-c01-c09-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-02",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "c01-claude-sonnet-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "max",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-sonnet-5",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c02-claude-gemini-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c03-agy-gemini-direct",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c04-claude-gpt-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c05-codex-gpt-direct",
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c06-claude-gemini-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c07-agy-gemini-hybrid",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c08-claude-gpt-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-hybrid",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gpt-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c09-codex-gpt-hybrid",
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-hybrid",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gpt-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,92 +0,0 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "landing-quality-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-01",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "claude-generic-preset",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "claude-generic",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "claude-sonnet-5"},
|
||||
{"stage": "plan", "model": "claude-sonnet-5"},
|
||||
{"stage": "work", "model": "claude-sonnet-5"},
|
||||
{"stage": "review", "model": "claude-sonnet-5"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "agy-generic-preset",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "agy-generic",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash"},
|
||||
{"stage": "work", "model": "gemini-3.6-flash"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "codex-generic-preset",
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "codex-generic",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gpt-5.6-luna"},
|
||||
{"stage": "plan", "model": "gpt-5.6-luna"},
|
||||
{"stage": "work", "model": "gpt-5.6-luna"},
|
||||
{"stage": "review", "model": "gpt-5.6-luna"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,240 +0,0 @@
|
|||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://iop.local/schemas/agent-comparison-benchmark-manifest.schema.json",
|
||||
"title": "Agent comparison benchmark manifest",
|
||||
"description": "Closed Draft 2020-12 schema for the benchmark manifest. Every object rejects unknown members. The loader accepts only the subset used by this packet.",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"pipeline_version",
|
||||
"environment",
|
||||
"testbed",
|
||||
"fixture",
|
||||
"matrix",
|
||||
"session_policy",
|
||||
"setup_cache_policy",
|
||||
"timeout",
|
||||
"viewports",
|
||||
"rubric_version",
|
||||
"evaluator",
|
||||
"output_root"
|
||||
],
|
||||
"properties": {
|
||||
"pipeline_version": { "const": "2" },
|
||||
"environment": { "const": "dev" },
|
||||
"testbed": { "const": "../iop-s2" },
|
||||
"repetitions": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"default": 1
|
||||
},
|
||||
"execution_order_seed": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-z0-9][a-z0-9_-]{0,63}$"
|
||||
},
|
||||
"session_policy": { "const": "fresh" },
|
||||
"setup_cache_policy": { "const": "isolated" },
|
||||
"timeout": { "$ref": "#/$defs/timeout" },
|
||||
"viewports": { "$ref": "#/$defs/viewports" },
|
||||
"rubric_version": {
|
||||
"enum": ["landing-quality-v1", "one-shot-agent-comparison-v1"]
|
||||
},
|
||||
"evaluator": { "$ref": "#/$defs/evaluator" },
|
||||
"output_root": { "$ref": "#/$defs/output_root" },
|
||||
"fixture": { "$ref": "#/$defs/fixture" },
|
||||
"matrix": { "$ref": "#/$defs/matrix" }
|
||||
},
|
||||
"$defs": {
|
||||
"bounded_token": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-z0-9][a-z0-9_.+-]{0,31}$"
|
||||
},
|
||||
"timeout": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["run_seconds", "idle_seconds", "quiet_seconds", "cleanup_grace_seconds"],
|
||||
"properties": {
|
||||
"run_seconds": { "type": "integer", "minimum": 1, "maximum": 86400 },
|
||||
"idle_seconds": { "type": "integer", "minimum": 1, "maximum": 600 },
|
||||
"quiet_seconds": { "type": "integer", "minimum": 1, "maximum": 60 },
|
||||
"cleanup_grace_seconds": { "type": "integer", "minimum": 1, "maximum": 60 }
|
||||
}
|
||||
},
|
||||
"viewport_record": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "width", "height"],
|
||||
"properties": {
|
||||
"id": { "$ref": "#/$defs/bounded_token" },
|
||||
"width": { "type": "integer", "minimum": 1, "maximum": 8192 },
|
||||
"height": { "type": "integer", "minimum": 1, "maximum": 8192 }
|
||||
}
|
||||
},
|
||||
"viewports": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": { "$ref": "#/$defs/viewport_record" },
|
||||
"uniqueItems": true
|
||||
},
|
||||
"output_root": {
|
||||
"type": "string",
|
||||
"pattern": "^agent-test/runs/[^/]+$"
|
||||
},
|
||||
"asset_mapping": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source", "workspace_path"],
|
||||
"properties": {
|
||||
"source": {
|
||||
"type": "string",
|
||||
"pattern": "^[^/][^:]*$"
|
||||
},
|
||||
"workspace_path": {
|
||||
"type": "string",
|
||||
"pattern": "^[^/][^:]*$"
|
||||
}
|
||||
}
|
||||
},
|
||||
"fixture": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["version", "prompt", "assets", "checksum"],
|
||||
"properties": {
|
||||
"version": { "$ref": "#/$defs/bounded_token" },
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"pattern": "^[^/][^:]*$"
|
||||
},
|
||||
"assets": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": { "$ref": "#/$defs/asset_mapping" }
|
||||
},
|
||||
"checksum": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
}
|
||||
}
|
||||
},
|
||||
"binding": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["stage", "model"],
|
||||
"properties": {
|
||||
"stage": {
|
||||
"type": "string",
|
||||
"enum": ["request", "selector", "plan", "work", "review", "repair"]
|
||||
},
|
||||
"model": { "$ref": "#/$defs/bounded_token" },
|
||||
"effort": { "$ref": "#/$defs/bounded_token" }
|
||||
}
|
||||
},
|
||||
"expected_binding": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["stage", "model"],
|
||||
"properties": {
|
||||
"stage": {
|
||||
"type": "string",
|
||||
"enum": ["request", "selector", "plan", "work", "review", "repair"]
|
||||
},
|
||||
"model": { "$ref": "#/$defs/bounded_token" },
|
||||
"effort": { "$ref": "#/$defs/bounded_token" }
|
||||
}
|
||||
},
|
||||
"iop_cell": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["request_model", "requested_effort", "route_kind", "route_id", "expected_bindings"],
|
||||
"properties": {
|
||||
"request_model": { "$ref": "#/$defs/bounded_token" },
|
||||
"requested_effort": { "$ref": "#/$defs/bounded_token" },
|
||||
"route_kind": { "type": "string", "enum": ["direct", "execution_preset"] },
|
||||
"route_id": { "$ref": "#/$defs/bounded_token" },
|
||||
"expected_bindings": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": { "$ref": "#/$defs/expected_binding" },
|
||||
"uniqueItems": true
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{
|
||||
"if": {
|
||||
"properties": { "route_kind": { "const": "direct" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"expected_bindings": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"maxItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"stage": { "const": "request" }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"if": {
|
||||
"properties": { "route_kind": { "const": "execution_preset" } }
|
||||
},
|
||||
"then": {
|
||||
"properties": {
|
||||
"expected_bindings": {
|
||||
"type": "array",
|
||||
"minItems": 4,
|
||||
"maxItems": 5,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"stage": { "enum": ["selector", "plan", "work", "review", "repair"] }
|
||||
}
|
||||
},
|
||||
"allOf": [
|
||||
{ "contains": { "properties": { "stage": { "const": "selector" } }, "required": ["stage"] }, "minContains": 1, "maxContains": 1 },
|
||||
{ "contains": { "properties": { "stage": { "const": "plan" } }, "required": ["stage"] }, "minContains": 1, "maxContains": 1 },
|
||||
{ "contains": { "properties": { "stage": { "const": "work" } }, "required": ["stage"] }, "minContains": 1, "maxContains": 1 },
|
||||
{ "contains": { "properties": { "stage": { "const": "review" } }, "required": ["stage"] }, "minContains": 1, "maxContains": 1 },
|
||||
{ "contains": { "properties": { "stage": { "const": "repair" } }, "required": ["stage"] }, "minContains": 0, "maxContains": 1 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"evaluator": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["caller", "iop"],
|
||||
"properties": {
|
||||
"caller": { "const": "codex" },
|
||||
"iop": { "$ref": "#/$defs/iop_cell" }
|
||||
}
|
||||
},
|
||||
"cell": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "caller", "iop"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-z0-9][a-z0-9_-]{0,63}$"
|
||||
},
|
||||
"caller": { "type": "string", "enum": ["claude", "agy", "codex"] },
|
||||
"iop": { "$ref": "#/$defs/iop_cell" }
|
||||
}
|
||||
},
|
||||
"matrix": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": { "$ref": "#/$defs/cell" },
|
||||
"uniqueItems": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,135 +0,0 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"execution_order_seed": "bench-02-recovery-qualification-v1",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "one-shot-agent-comparison-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-02-recovery",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "c01-claude-sonnet-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "max",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-sonnet-5",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c03-agy-gemini-direct",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c04-claude-gpt-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c06-claude-gemini-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c07-agy-gemini-hybrid",
|
||||
"caller": "agy",
|
||||
"iop": {
|
||||
"request_model": "gemini-hybrid",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gemini-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "plan", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gemini-3.6-flash", "effort": "high"},
|
||||
{"stage": "repair", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "c08-claude-gpt-hybrid",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-hybrid",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "execution_preset",
|
||||
"route_id": "gpt-hybrid",
|
||||
"expected_bindings": [
|
||||
{"stage": "selector", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "plan", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "work", "model": "ornith-fast"},
|
||||
{"stage": "review", "model": "gpt-5.6-terra", "effort": "high"},
|
||||
{"stage": "repair", "model": "gpt-5.6-terra", "effort": "high"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
# Agent comparison benchmark report
|
||||
|
||||
## Run identity
|
||||
|
||||
| field | value |
|
||||
|---|---|
|
||||
| run_id | run-20260811T010203Z-123456abcdef |
|
||||
| manifest_digest | sha256:39715007db41882abcfe5a3fd5f8e3cc4bcadce0ef31c17df4f3347825184359 |
|
||||
| pipeline_version | 2 |
|
||||
|
||||
## Immutable conditions
|
||||
|
||||
| field | value |
|
||||
|---|---|
|
||||
| environment | dev |
|
||||
| fixture | landing-v1 (sha256:4b5c9dcfe799d21f86a4462a2ada35275d72d5fe919b2adb069eeb3f4ac72fcb) |
|
||||
| rubric | landing-quality-v1 |
|
||||
| session_policy | fresh |
|
||||
| setup_cache_policy | isolated |
|
||||
| evaluator | codex/judge-model/xhigh |
|
||||
|
||||
## Execution preflight
|
||||
|
||||
| sequence | status | results |
|
||||
|---:|---|---:|
|
||||
| — | unavailable | 0 |
|
||||
|
||||
## Attempt outcomes
|
||||
|
||||
| cell | repetition | attempt | controller | product | harness | process | artifact | scoring | total | rank |
|
||||
|---|---:|---:|---|---|---|---|---|---|---:|---:|
|
||||
| cell-sentinel | 1 | 1 | completed | succeeded | passed | exited | passed | scored | 99 | 1 |
|
||||
| cell-sentinel | 1 | 2 | completed | succeeded | passed | exited | passed | scored | 99 | 1 |
|
||||
| cell-sentinel | 1 | 3 | completed | failed | passed | exited | failed | unscored | — | — |
|
||||
| cell-sentinel | 1 | 4 | completed | succeeded | passed | exited | passed | scoring_failed | — | — |
|
||||
| cell-sentinel | 1 | 5 | completed | succeeded | passed | exited | passed | blocked | — | — |
|
||||
|
||||
## Quality score breakdown
|
||||
|
||||
| cell/repetition/attempt | category | score | max |
|
||||
|---|---|---:|---:|
|
||||
| cell-sentinel/r1/a1 | task_fidelity | 24 | 25 |
|
||||
| cell-sentinel/r1/a1 | visual_hierarchy | 25 | 25 |
|
||||
| cell-sentinel/r1/a1 | responsive_composition | 20 | 20 |
|
||||
| cell-sentinel/r1/a1 | typography_readability | 15 | 15 |
|
||||
| cell-sentinel/r1/a1 | polish_consistency | 15 | 15 |
|
||||
| cell-sentinel/r1/a2 | task_fidelity | 24 | 25 |
|
||||
| cell-sentinel/r1/a2 | visual_hierarchy | 25 | 25 |
|
||||
| cell-sentinel/r1/a2 | responsive_composition | 20 | 20 |
|
||||
| cell-sentinel/r1/a2 | typography_readability | 15 | 15 |
|
||||
| cell-sentinel/r1/a2 | polish_consistency | 15 | 15 |
|
||||
|
||||
## Timing and token evidence
|
||||
|
||||
| cell/repetition/attempt | time observations | token observations |
|
||||
|---|---|---|
|
||||
| cell-sentinel/r1/a1 | total_duration=1 ns; clock=harness_monotonic; source=harness; submitted_at,first_output_at=unavailable; reason=not_observed; source=harness; first_write_observed_at,first_write_mtime=unavailable; reason=not_observed; source=workspace_poll | cache_write_tokens,cached_input_tokens,input_tokens,model_calls,model_duration,output_tokens,queue_duration,reasoning_tokens,tool_calls,tool_duration,total_duration,total_tokens=unavailable; reason=not_reported; source=harness |
|
||||
| cell-sentinel/r1/a2 | total_duration=1 ns; clock=harness_monotonic; source=harness; submitted_at,first_output_at=unavailable; reason=not_observed; source=harness; first_write_observed_at,first_write_mtime=unavailable; reason=not_observed; source=workspace_poll | cache_write_tokens,cached_input_tokens,input_tokens,model_calls,model_duration,output_tokens,queue_duration,reasoning_tokens,tool_calls,tool_duration,total_duration,total_tokens=unavailable; reason=not_reported; source=harness |
|
||||
| cell-sentinel/r1/a3 | total_duration=1 ns; clock=harness_monotonic; source=harness; submitted_at,first_output_at=unavailable; reason=not_observed; source=harness; first_write_observed_at,first_write_mtime=unavailable; reason=not_observed; source=workspace_poll | cache_write_tokens,cached_input_tokens,input_tokens,model_calls,model_duration,output_tokens,queue_duration,reasoning_tokens,tool_calls,tool_duration,total_duration,total_tokens=unavailable; reason=not_reported; source=harness |
|
||||
| cell-sentinel/r1/a4 | total_duration=1 ns; clock=harness_monotonic; source=harness; submitted_at,first_output_at=unavailable; reason=not_observed; source=harness; first_write_observed_at,first_write_mtime=unavailable; reason=not_observed; source=workspace_poll | cache_write_tokens,cached_input_tokens,input_tokens,model_calls,model_duration,output_tokens,queue_duration,reasoning_tokens,tool_calls,tool_duration,total_duration,total_tokens=unavailable; reason=not_reported; source=harness |
|
||||
| cell-sentinel/r1/a5 | total_duration=1 ns; clock=harness_monotonic; source=harness; submitted_at,first_output_at=unavailable; reason=not_observed; source=harness; first_write_observed_at,first_write_mtime=unavailable; reason=not_observed; source=workspace_poll | cache_write_tokens,cached_input_tokens,input_tokens,model_calls,model_duration,output_tokens,queue_duration,reasoning_tokens,tool_calls,tool_duration,total_duration,total_tokens=unavailable; reason=not_reported; source=harness |
|
||||
|
||||
## Web validation and scoring provenance
|
||||
|
||||
| cell/repetition/attempt | web gates | screenshots | score_id | evaluator | scoring condition |
|
||||
|---|---|---|---|---|---|
|
||||
| cell-sentinel/r1/a1 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded |
|
||||
| cell-sentinel/r1/a2 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | recorded |
|
||||
| cell-sentinel/r1/a3 | generated_files=pass, static_safety=pass, images=fail, network=fail, console=fail, responsive=fail, accessibility=fail | unavailable | — | unavailable | product_failed, process_nonzero_exit, gate_images, gate_network |
|
||||
| cell-sentinel/r1/a4 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | score-000001 | codex/judge-route/judge-model/xhigh | invalid_worksheet |
|
||||
| cell-sentinel/r1/a5 | generated_files=pass, static_safety=pass, images=pass, network=pass, console=pass, responsive=pass, accessibility=pass | screenshot-desktop.png, screenshot-mobile.png | — | unavailable | evaluator_preflight_blocked |
|
||||
|
||||
## Limitations
|
||||
|
||||
- Values marked `unavailable` retain the producing source and reason; they are not inferred as zero.
|
||||
- Automatic web gates establish eligibility only and contribute no quality points.
|
||||
- Equal scored totals share a competition rank; unscored and scoring-failed attempts do not receive a rank.
|
||||
|
||||
## Raw evidence index
|
||||
|
||||
| contained pointer |
|
||||
|---|
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000001/attempt-measurement.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000001/attempt.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000001/scoring/score-000001/allocation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000001/scoring/score-000001/result.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000001/web-validation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000002/attempt-measurement.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000002/attempt.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000002/scoring/score-000001/allocation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000002/scoring/score-000001/result.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000002/web-validation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000003/attempt-measurement.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000003/attempt.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000003/scoring/unscored.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000003/web-validation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000004/attempt-measurement.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000004/attempt.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000004/scoring/score-000001/allocation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000004/scoring/score-000001/result.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000004/web-validation.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000005/attempt-measurement.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000005/attempt.json>) |
|
||||
| [raw](<cells/cell-sentinel/repetition-0001/attempt-000005/web-validation.json>) |
|
||||
| [raw](<manifest.json>) |
|
||||
| [raw](<run.json>) |
|
||||
| [raw](<scoring-preflight/preflight-000001.json>) |
|
||||
| [raw](<scoring-preflight/preflight-000002.json>) |
|
||||
| [raw](<scoring-preflight/preflight-000003.json>) |
|
||||
| [raw](<scoring-preflight/preflight-000004.json>) |
|
||||
|
|
@ -1,96 +0,0 @@
|
|||
{
|
||||
"pipeline_version": "2",
|
||||
"environment": "dev",
|
||||
"testbed": "../iop-s2",
|
||||
"repetitions": 1,
|
||||
"session_policy": "fresh",
|
||||
"setup_cache_policy": "isolated",
|
||||
"timeout": {
|
||||
"run_seconds": 300,
|
||||
"idle_seconds": 30,
|
||||
"quiet_seconds": 10,
|
||||
"cleanup_grace_seconds": 5
|
||||
},
|
||||
"viewports": [
|
||||
{"id": "desktop_1080", "width": 1920, "height": 1080},
|
||||
{"id": "mobile_375", "width": 375, "height": 812}
|
||||
],
|
||||
"rubric_version": "landing-quality-v1",
|
||||
"evaluator": {
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
"output_root": "agent-test/runs/bench-01-supported-direct",
|
||||
"fixture": {
|
||||
"version": "product-card-v2",
|
||||
"prompt": "scripts/fixtures/agent-comparison-benchmark/prompt.md",
|
||||
"assets": [
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/reference.txt", "workspace_path": "brief/reference.txt"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/aurora-grid.svg", "workspace_path": "assets/aurora-grid.svg"},
|
||||
{"source": "scripts/fixtures/agent-comparison-benchmark/orbit-rings.svg", "workspace_path": "assets/orbit-rings.svg"}
|
||||
],
|
||||
"checksum": "sha256:fb16198fd4c3576f880f047ed7de54dddc160b0f70c2ba55b435cf078c61828e"
|
||||
},
|
||||
"matrix": [
|
||||
{
|
||||
"id": "claude-sonnet-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "claude-sonnet-5",
|
||||
"requested_effort": "max",
|
||||
"route_kind": "direct",
|
||||
"route_id": "claude-sonnet-5",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "claude-sonnet-5", "effort": "max"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "claude-gemini-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gemini-3.6-flash",
|
||||
"requested_effort": "high",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gemini-3.6-flash",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gemini-3.6-flash", "effort": "high"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "claude-gpt-direct",
|
||||
"caller": "claude",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "codex-gpt-direct",
|
||||
"caller": "codex",
|
||||
"iop": {
|
||||
"request_model": "gpt-5.6-luna",
|
||||
"requested_effort": "xhigh",
|
||||
"route_kind": "direct",
|
||||
"route_id": "gpt-5.6-luna",
|
||||
"expected_bindings": [
|
||||
{"stage": "request", "model": "gpt-5.6-luna", "effort": "xhigh"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
{"conversation_id":"fixture-conversation","event":"init","init":{"cwd":"/workspace","model":"Gemini 3.6 Flash","permission_mode":"sandbox","tools":[]}}
|
||||
{"event":"step_update","step_update":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"state":"DONE","step_index":0,"step_type":"agent_response","text_delta":"fixture content must be redacted","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}}
|
||||
{"event":"result","result":{"conversation_id":"fixture-conversation","duration_seconds":0.012,"num_turns":1,"response":"fixture content must be redacted","status":"SUCCESS","usage":{"cache_read_tokens":1,"input_tokens":5,"output_tokens":4,"thinking_tokens":2,"total_tokens":12}}}
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 800" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Aurora service grid</title>
|
||||
<desc id="desc">Abstract luminous nodes connected across a dark blue operational grid.</desc>
|
||||
<defs>
|
||||
<linearGradient id="background" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0" stop-color="#08172e"/>
|
||||
<stop offset="1" stop-color="#163764"/>
|
||||
</linearGradient>
|
||||
<radialGradient id="glow">
|
||||
<stop offset="0" stop-color="#7ef9d6" stop-opacity="0.95"/>
|
||||
<stop offset="1" stop-color="#42a5ff" stop-opacity="0"/>
|
||||
</radialGradient>
|
||||
<pattern id="grid" width="80" height="80" patternUnits="userSpaceOnUse">
|
||||
<path d="M80 0H0V80" fill="none" stroke="#b9d8ff" stroke-opacity="0.14"/>
|
||||
</pattern>
|
||||
</defs>
|
||||
<rect width="1200" height="800" rx="48" fill="url(#background)"/>
|
||||
<rect width="1200" height="800" rx="48" fill="url(#grid)"/>
|
||||
<g fill="none" stroke="#8bd6ff" stroke-width="4" stroke-opacity="0.65">
|
||||
<path d="M180 520 390 300 610 430 830 210 1030 390"/>
|
||||
<path d="M260 170 390 300 520 155 830 210 940 610"/>
|
||||
<path d="M180 520 480 650 610 430 940 610"/>
|
||||
</g>
|
||||
<g fill="url(#glow)">
|
||||
<circle cx="180" cy="520" r="120"/>
|
||||
<circle cx="390" cy="300" r="140"/>
|
||||
<circle cx="610" cy="430" r="150"/>
|
||||
<circle cx="830" cy="210" r="125"/>
|
||||
<circle cx="940" cy="610" r="135"/>
|
||||
</g>
|
||||
<g fill="#d9fff3" stroke="#0b263e" stroke-width="8">
|
||||
<circle cx="180" cy="520" r="18"/>
|
||||
<circle cx="390" cy="300" r="24"/>
|
||||
<circle cx="610" cy="430" r="28"/>
|
||||
<circle cx="830" cy="210" r="21"/>
|
||||
<circle cx="940" cy="610" r="23"/>
|
||||
<circle cx="260" cy="170" r="14"/>
|
||||
<circle cx="520" cy="155" r="16"/>
|
||||
<circle cx="1030" cy="390" r="17"/>
|
||||
<circle cx="480" cy="650" r="15"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 1.8 KiB |
|
|
@ -1,3 +0,0 @@
|
|||
{"type":"system","subtype":"init","session_id":"claude-session-fixture","model":"claude-sonnet","cwd":"[redacted]"}
|
||||
{"type":"assistant","session_id":"claude-session-fixture","message":{"model":"claude-sonnet","stop_reason":"end_turn","content":[{"type":"text","text":"[redacted]"}]}}
|
||||
{"type":"result","subtype":"success","session_id":"claude-session-fixture","duration_ms":1234,"duration_api_ms":1000,"usage":{"input_tokens":11,"output_tokens":22,"cache_read_input_tokens":5},"result":"[redacted]"}
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
{"type":"thread.started","thread_id":"public-fixture"}
|
||||
{"type":"item.completed","item":{"id":"call-1","type":"command_execution","duration_ms":7.25}}
|
||||
{"type":"turn.completed","status":"completed","usage":{"input_tokens":31,"cached_input_tokens":8,"cache_write_input_tokens":6,"output_tokens":12,"reasoning_output_tokens":4},"item":{"content":"public fixture content"}}
|
||||
{"type":"adapter.idle","adapter":"codex_iop","nonce":"fixture-nonce-0001","child_exit":0}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" role="img" aria-labelledby="title desc">
|
||||
<title id="title">Layered orbit rings</title>
|
||||
<desc id="desc">Abstract rings and markers arranged around a bright central operating point.</desc>
|
||||
<defs>
|
||||
<linearGradient id="background" x1="0" y1="1" x2="1" y2="0">
|
||||
<stop offset="0" stop-color="#eef8ff"/>
|
||||
<stop offset="1" stop-color="#dffcf2"/>
|
||||
</linearGradient>
|
||||
<linearGradient id="ring" x1="0" y1="0" x2="1" y2="1">
|
||||
<stop offset="0" stop-color="#1769aa"/>
|
||||
<stop offset="1" stop-color="#00a77b"/>
|
||||
</linearGradient>
|
||||
<radialGradient id="core">
|
||||
<stop offset="0" stop-color="#ffffff"/>
|
||||
<stop offset="0.55" stop-color="#79f2cb"/>
|
||||
<stop offset="1" stop-color="#1686d9"/>
|
||||
</radialGradient>
|
||||
</defs>
|
||||
<rect width="1000" height="1000" rx="72" fill="url(#background)"/>
|
||||
<g transform="translate(500 500)" fill="none" stroke="url(#ring)">
|
||||
<ellipse rx="350" ry="150" stroke-width="18" transform="rotate(-18)"/>
|
||||
<ellipse rx="350" ry="150" stroke-width="10" stroke-opacity="0.55" transform="rotate(42)"/>
|
||||
<ellipse rx="350" ry="150" stroke-width="6" stroke-opacity="0.35" transform="rotate(102)"/>
|
||||
<circle r="245" stroke-width="3" stroke-dasharray="8 18" stroke-opacity="0.42"/>
|
||||
</g>
|
||||
<circle cx="500" cy="500" r="112" fill="url(#core)"/>
|
||||
<g fill="#0a3157" stroke="#ffffff" stroke-width="12">
|
||||
<circle cx="170" cy="390" r="30"/>
|
||||
<circle cx="710" cy="245" r="25"/>
|
||||
<circle cx="820" cy="610" r="34"/>
|
||||
<circle cx="340" cy="780" r="27"/>
|
||||
</g>
|
||||
<g fill="#ffffff" opacity="0.8">
|
||||
<circle cx="470" cy="462" r="18"/>
|
||||
<circle cx="535" cy="520" r="12"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 1.7 KiB |
|
|
@ -1,14 +0,0 @@
|
|||
Build a polished, responsive product card for the fictional product described in `brief/reference.txt`.
|
||||
|
||||
Requirements:
|
||||
|
||||
- Create exactly these implementation files at the workspace root: `index.html`, `styles.css`, and `script.js`.
|
||||
- Use both provided local images, `assets/aurora-grid.svg` and `assets/orbit-rings.svg`, as visible `<img>` content with meaningful `alt` text.
|
||||
- Compose one compact card with a small product header, one hero area, and one primary CTA using the supplied copy. Do not add feature, workflow, testimonial, pricing, or footer sections.
|
||||
- Provide one tiny keyboard-accessible toggle in `script.js` for the supplied status detail; the detail must remain readable when JavaScript is unavailable.
|
||||
- Support the supplied desktop and mobile viewports without horizontal overflow, clipped primary content, or overlapping controls.
|
||||
- Use semantic HTML, visible focus states, sufficient text/background contrast, a logical heading order, and labels for interactive controls.
|
||||
- Do not use external network assets, frameworks, package managers, build tools, inline data URLs, or generated replacements for the provided images.
|
||||
- Do not add credentials, private endpoints, analytics, trackers, or production data.
|
||||
|
||||
Finish only after all three required files exist and the page can be served as static files from the workspace root.
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
PRODUCT: Lumen Atlas
|
||||
EYEBROW: Operational clarity, at a glance
|
||||
HEADLINE: Turn scattered signals into a shared operating picture.
|
||||
SUMMARY: Lumen Atlas brings service health, ownership, and live operational context into one focused workspace so teams can decide with confidence.
|
||||
PRIMARY CTA: Explore the workspace
|
||||
STATUS TOGGLE: Show live context
|
||||
STATUS DETAIL: Ownership, service health, and recent changes stay connected in one calm view.
|
||||
Loading…
Reference in a new issue