diff --git a/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md b/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md index 3f14163f..6b08cce8 100644 --- a/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md +++ b/agent-ops/skills/project/iop-agent-comparison-benchmark/SKILL.md @@ -180,7 +180,7 @@ stderr: - Ready execution binds one exact cell and immutable attempt identity to one fresh workspace/session and one task submission. - Product, harness, process, and artifact are independent gates. Controller state `completed` only means the invocation controller reached a terminal state. - Resolution (`unresolved=0`) requires every manifest slot to have complete terminal evidence (web validation present); it does not require every gate to pass. Failed gates remain visible as independent failure counts and are classified by `score` as `unscored`. -- Release qualification runs the five-cell direct manifest as one unscored canary and requires all four gates for all five cells before a fresh nine-cell preflight; it does not allocate hybrid or scored execution. +- Release qualification runs the five-cell direct manifest as one unscored canary and requires fresh `ready=5`, exactly five fresh attempts, `unresolved=0`, `running=0`, `interrupted=0`, terminal controller/product/harness/process/web-validation evidence for every slot, and no exhausted browser/CDP infrastructure block before a fresh nine-cell preflight. Product failure, upstream HTTP rejection, generated-missing after caller failure, and timeout remain measured outcomes and do not trigger an implicit retry. Edge pre-ingress incompatibility or an exhausted browser/CDP infrastructure block stops qualification; this step does not allocate hybrid or scored execution. - Scoring copies only anonymous generated files, two local images, and screenshots into an opaque run-owned blind tree; the identity mapping remains outside that tree. - Scoring records `unscored`, `scored`, and `scoring_failed` append-only, and a retry always allocates a fresh score id/session. - The internal workspace API (`RunStore`, `Manifest`, etc.) is not a user command. Do not expose it. @@ -189,7 +189,7 @@ stderr: - Stop immediately on a preflight `registration_required` or `implementation_gap` result. Do not substitute an alias, change an effort, or continue to attempt allocation. - Stop immediately on a run/resume preflight blocker without allocating an attempt or invoking another execution path. -- Stop after a retained execution failure unless the user explicitly requests resume with `--retry-failed`. +- Do not resume or retry a retained execution failure unless the user explicitly requests resume with `--retry-failed`. A release-qualification run with complete terminal evidence may continue to the fresh nine-cell preflight under the Safety rules without retrying or reinterpreting the failed result. - Stop after `scoring_failed` unless the user explicitly requests score with `--retry-scoring-failed`. - Stop immediately and report the unavailable-state line for report requests that cannot project the deterministic report. - Stop immediately if the manifest path is missing or the file is not readable. diff --git a/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md b/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md index ee7a7d8f..22ee4182 100644 --- a/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md +++ b/agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/iop-one-shot-agent-model-comparison.md @@ -105,7 +105,7 @@ - 관련 경로: `agent-test/dev/`, `agent-test/runs/`, `../iop-s2` - 표준선: preflight는 scored attempt와 분리하고, scored 실행이 시작된 뒤의 실패는 결과로 보존하며 재실행이 필요하면 새 attempt로 기록한다. - 표준선: IOP credential/model route가 없으면 안전한 등록을 요청하고, alias/effort를 임의 대체하지 않는다. -- 현재 차단: 없음. 2026-08-12의 첫 scored run은 첫 slot caller launch 전 control socket 등록 결함으로 중단됐고 해당 run/attempt evidence는 그대로 보존한다. socket 등록, pre-registration reconcile, terminal outcome projection과 caller 환경 결함은 회귀 테스트로 보완됐다. 2026-08-13 사용자가 다음 벤치까지 계속 실행하도록 승인했으므로, 기존 run을 resume하거나 `--retry-failed`로 재사용하지 않고 동일 immutable C01-C09 manifest의 새 run identity를 repetitions=1로 한 번 실행한다. 새 보고서는 이전 실패 run을 삭제하거나 성공 결과로 대체하지 않고 관계와 한계를 명시한다. +- 현재 차단: packet 14의 fresh 5-cell direct 진단과 배포 qualification 진행 중. retained direct run은 `unresolved=0`인 terminal evidence지만 all-success는 아니며 기존 run을 resume/retry/수정하지 않는다. 새 qualification은 fresh `ready=5`, 정확히 5개 attempt, 모든 controller/product/harness/process/web-validation terminal evidence와 exhausted browser/CDP infrastructure block 없음으로 판정하고 제품 실패·provider rejection·timeout은 benchmark 결과로 보존한다. 이 진단이 통과하면 fresh C01-C09 `ready=9`를 확인하고, 후속 packet에서 기존 run과 다른 identity로 repetitions=1 scored run을 한 번 실행한다. 비교 Task 체크 상태는 scored evidence가 생길 때까지 변경하지 않는다. - 실행 순서와 차단 관계: [전역 마일스톤 실행 순서](../../../priority-queue.md) - 관련 Milestone: [[bench-01] Agent 비교 벤치마크 파이프라인 준비](agent-comparison-benchmark-pipeline.md), [[route-02] IOP 단일 요청 Agent 실행](../../../archive/phase/knowledge-tool-optimization-extension/milestones/iop-owned-single-request-agent-execution.md) - 확인 필요: 없음 diff --git a/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md b/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md index e285cc32..290a312b 100644 --- a/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md +++ b/agent-roadmap/sdd/knowledge-tool-optimization-extension/iop-one-shot-agent-model-comparison/SDD.md @@ -26,6 +26,7 @@ - [x] [D10] scored failure는 보존하고 같은 run의 재실행은 명시적 retry의 새 attempt로만 기록하며 성공 결과만 골라 대표하지 않는다. 새 run 승인은 기존 실패 run/attempt를 대체하지 않는 별도 비교 cycle이며, 보고서에는 이전 실패 run과 새 run의 관계 및 한계를 함께 남긴다. - [x] [D11] 공식 `agy 1.1.12`는 Gemini API-key provider의 route별 `GOOGLE_GEMINI_BASE_URL`을 IOP Edge로 지정하고 `GEMINI_API_KEY`에는 upstream key가 아닌 IOP principal token을 넣는다. `--effort`와 비공식 custom model은 사용하지 않고 high effort는 IOP effective binding으로 검증한다. - [x] [D12] marked hybrid preset은 dev managed credential plane의 fresh projection, 고정 stage authorization과 sealed provider lease가 준비된 뒤에만 실행하며 legacy credential fallback을 허용하지 않는다. + - [x] [D13] 배포 qualification의 5-cell direct 진단은 all-success가 아니라 terminal-evidence completeness를 판정한다. fresh `ready=5`, 정확히 5개의 fresh attempt, `unresolved=0`, `running=0`, `interrupted=0`, 모든 slot의 controller/product/harness/process/web-validation terminal evidence와 exhausted browser/CDP infrastructure block 없음이 필요하다. 제품 실패·provider rejection·caller failure 뒤 generated-missing·timeout은 보존할 benchmark 결과이며 암묵 재시도하지 않는다. Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. 이 unscored 진단은 D06/D10의 유일한 새 scored C01-C09 run identity를 소비하지 않는다. ## 문제 / 비목표 @@ -74,6 +75,7 @@ State invariant: - 한 cell의 사용자 작업은 한 번 제출하며 사람의 feedback, manual edit, restart가 없다. - model/tool 호출 횟수는 제약이 아니라 측정 대상이며 finish event 뒤 idle까지가 wall-clock terminal이다. - 실패 cell도 report matrix에 남고 재실행 결과는 원래 attempt를 대체하지 않는다. +- 배포 qualification의 5-cell direct 진단은 scored C01-C09 run과 별개다. terminal evidence가 완결된 제품 실패·provider rejection·timeout을 acceptance failure로 재해석하지 않고, Edge pre-ingress incompatibility 또는 최대 renderer 재시도 뒤 browser/CDP infrastructure block만 다음 단계 진입을 막는다. ## Interface Contract @@ -103,7 +105,7 @@ State invariant: | ID | Milestone Task | Given | When | Then | |----|----------------|-------|------|------| | S01 | `fixture-lock` | 이미지 2장과 one-page benchmark brief | fixture 확정 | prompt/assets/workspace/viewports/rubric의 checksum과 version이 모든 cell에 동일하다. | -| S02 | `route-readiness` | C01-C09 caller와 dev IOP | execution-day preflight | auth, model/preset, effort, stream/finish/idle이 모두 확인되거나 exact blocker로 중단된다. | +| S02 | `route-readiness` | 5-cell direct 진단과 C01-C09 caller·dev IOP | execution-day qualification과 preflight | direct는 fresh `ready=5`, 정확히 5개 attempt, 모든 terminal evidence와 infrastructure block 없음으로 admission되고, 이후 C01-C09 auth/model/preset/effort/stream/finish/idle이 모두 확인되거나 exact blocker로 중단된다. | | S03 | `matrix-lock` | 선행 gate가 통과한 9개 cell | scored manifest 생성 | repetitions=1, 실행 순서 seed, fresh-session/setup-cache 정책, timeout과 expected binding이 immutable하게 기록된다. | | S04 | `claude-standalone` | C01 clean workspace | Claude Code 사용자 작업 1회 | IOP→Sonnet 최고 effort 결과와 complete/idle evidence가 생성된다. | | S05 | `gemini-standalone` | C02-C03 clean workspace | Claude Code와 agy 사용자 작업을 각각 1회 제출 | 두 caller 모두 IOP→Gemini high 결과와 caller별 timing/usage를 남긴다. | @@ -122,7 +124,7 @@ State invariant: | Scenario | Required Evidence | `agent-task` 연결 | 완료 Evidence 기대 | |----------|-------------------|------------------|---------------------------| | S01 | fixture prompt/assets/workspace/rubric digest | `agent-task/m-iop-one-shot-agent-model-comparison/fixture-lock/` | `fixture-lock` identical-input evidence | -| S02 | redacted C01-C09 preflight matrix | `agent-task/m-iop-one-shot-agent-model-comparison/route-readiness/` | `route-readiness` auth/route/effort/terminal evidence | +| S02 | redacted direct `ready=5`·5-slot terminal evidence·infrastructure 판정과 C01-C09 preflight matrix | `agent-task/m-iop-one-shot-agent-model-comparison/route-readiness/` | `route-readiness` direct admission과 auth/route/effort/terminal evidence | | S03 | immutable scored manifest와 order seed | `agent-task/m-iop-one-shot-agent-model-comparison/matrix-lock/` | `matrix-lock` 9-cell/repetitions=1 evidence | | S04 | C01 event/timing/usage/workspace evidence | `agent-task/m-iop-one-shot-agent-model-comparison/claude-standalone/` | `claude-standalone` one-submission/IOP evidence | | S05 | C02-C03 caller별 event/timing/usage/workspace evidence | `agent-task/m-iop-one-shot-agent-model-comparison/gemini-standalone/` | `gemini-standalone` two-caller evidence | @@ -147,13 +149,14 @@ State invariant: - [x] Milestone 기능 Task와 Acceptance Scenario가 일치한다. - [x] Evidence Map이 code-review/complete.log에서 검증 가능하다. - [x] agent-contract를 쓰는 경우 SDD에 계약 원문을 복제하지 않았다. -- [x] 사용자 리뷰가 필요한 항목은 없고 확정된 D01-D10을 반영했다. +- [x] 사용자 리뷰가 필요한 항목은 없고 확정된 D01-D13을 반영했다. ## 사용자 리뷰 이력 - 2026-08-06: 사용자가 Sonnet/Gemini/GPT 단독과 Gemini/GPT 하이브리드의 9개 IOP 경유 비교군, Claude Code·agy·Codex caller, finish/idle 원샷, 초기 1회, dev `../iop-s2`, 동일 정적 웹 fixture와 시간·token·Codex 품질 평가를 확정했다. - 2026-08-12: 공식 `agy 1.1.12` API-key provider의 실제 Gemini-native 요청과 `stream-json` event를 확인했고, 사용자의 provider 직접 설정 지시에 따라 upstream key와 IOP principal token을 분리하며 dev managed credential plane까지 구성하는 D11-D12를 기술 보강했다. - 2026-08-13: 사용자가 terminal outcome과 dispatcher 환경 보완 뒤 다음 벤치까지 계속 실행하도록 승인했다. 이에 기존 실패 run을 보존하고 resume/retry하지 않은 채, 동일 immutable C01-C09 manifest로 repetitions=1인 새 scored run identity를 한 번 생성하는 D06/D10 경계를 확정했다. +- 2026-08-13: 승인된 후속 packet에 따라 direct 배포 qualification을 terminal-evidence admission으로 분리했다. 제품 실패·provider rejection·timeout은 측정 결과로 보존하고, Edge pre-ingress incompatibility와 exhausted browser/CDP infrastructure block만 qualification을 막는 D13을 추가했다. D06/D10의 scored-run uniqueness는 유지한다. ## 작업 컨텍스트 diff --git a/agent-spec/testing/agent-comparison-benchmark.md b/agent-spec/testing/agent-comparison-benchmark.md index 4975831a..d6c8a83e 100644 --- a/agent-spec/testing/agent-comparison-benchmark.md +++ b/agent-spec/testing/agent-comparison-benchmark.md @@ -21,6 +21,9 @@ source_evidence: - type: code path: scripts/agent_benchmark/web_validation.py notes: 정적 웹 산출물과 desktop/mobile 자동 gate 검증 + - type: code + path: scripts/agent_benchmark/browser_cdp.py + notes: Chromium/CDP 단일 시도 격리와 닫힌 transient class의 최대 3회 fresh 재시작 - type: code path: scripts/agent_benchmark/scoring.py notes: 익명화 입력, fresh evaluator와 scoring attempt 처리 @@ -39,6 +42,9 @@ source_evidence: - type: test path: scripts/agent_benchmark/skill_contract_test.py notes: project-local skill과 공개 CLI 계약 검증 + - type: test + path: scripts/agent_benchmark/browser_cdp_test.py + notes: transient 재시도·소진·비재시도와 process/screenshot 정리 검증 - type: contract path: agent-contract/outer/openai-compatible-api.md notes: agy와 Codex가 사용하는 IOP OpenAI-compatible ingress 계약 @@ -107,7 +113,8 @@ flowchart LR - preflight는 scored attempt가 아니며, 실행 중 실패·timeout·cancel과 scoring 실패는 기존 attempt를 수정하지 않고 보존한다. - caller parser는 raw terminal 문자열 대신 `CallerEvent(finish|idle)`, `CallerTerminal(succeeded|failed)`와 typed metric만 반환한다. Claude result가 마지막 active assistant snapshot을 직접 완성하면 adapter가 typed finish와 idle을 함께 투영하고, assistant가 이미 finish를 냈으면 result는 idle만 투영한다. synthetic API error와 agy ERROR result는 `product=failed`, `harness=passed`가 될 수 있으며 parser malformed는 `product=unknown`, `harness=failed`로 구분한다. - durable lifecycle/measurement/attempt evidence는 `product`, `harness`, `process` 객체를 그대로 저장한다. `unresolved`은 수집/검증 완결성(모든 슬롯이 웹 검증 증거 보유)이며, `passed`는 전체 gate 성공으로 유지되고 retry/skip를 제어한다. scoring eligibility는 변경없으며, terminal failure는 `unscored` report row로 유지된다. run/resume exit 0은 `unresolved=0`을 요구하며, 독립 실패 축은 stdout에 남고 `score`로 분류된다. -- 배포 qualification은 동일 clean source에서 5-cell direct manifest를 unscored canary로 한 번 실행해 네 gate 5/5를 확인한 뒤 fresh C01-C09 preflight `ready=9`까지만 수행한다. hybrid 또는 scored C01-C09 실행은 후속 승인 전에는 할당하지 않는다. +- 배포 qualification은 동일 clean source에서 5-cell direct manifest의 fresh preflight `ready=5`를 확인하고 정확히 5개의 fresh attempt를 한 번씩 실행한다. `unresolved=0`, `running=0`, `interrupted=0`, 모든 slot의 controller/product/harness/process/web-validation terminal evidence, exhausted browser/CDP infrastructure block 없음이 admission 조건이다. product failure, upstream HTTP rejection, caller failure 뒤 generated-missing과 timeout은 측정 결과로 보존하고 암묵 재시도하지 않는다. Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. admission 뒤 fresh C01-C09 preflight `ready=9`까지만 수행하고 hybrid 또는 scored C01-C09 실행은 후속 승인 전에는 할당하지 않는다. +- Chromium/CDP renderer는 source/product validation을 재실행하지 않고, process start·handshake·socket loss의 닫힌 transient class에만 fresh browser/profile로 최대 3회 시도한다. 각 실패 시도는 process group과 부분 screenshot을 정리하며 세 번째 실패는 `artifact=blocked`로 유지한다. - lifecycle supervisor는 exit watcher와 출력 reader를 join한 뒤 하나의 child return code를 동결해 lifecycle result와 cleanup receipt가 동일한 exit/signal을 갖게 한다. 불일치 evidence는 resume에서 fail-closed한다. - raw credential과 private endpoint는 tracked manifest, event, log, screenshot과 report에 기록하지 않는다. - report는 run state의 canonical evidence에서 생성되며 성공하지 않은 결과를 0점으로 변환하거나 동점에 임의 순위를 부여하지 않는다. @@ -129,6 +136,7 @@ flowchart LR ## 변경 기록 +- 2026-08-13: direct 배포 qualification을 all-success에서 terminal-evidence admission으로 분리했다. fresh `ready=5`, 정확히 5개 attempt와 완결 evidence는 요구하되 제품 실패·timeout은 결과로 보존하고, Edge pre-ingress incompatibility 또는 exhausted browser/CDP infrastructure block만 qualification을 막는다. Chromium/CDP 닫힌 transient class에는 최대 3회 fresh renderer 시도를 추가했다. - 2026-08-13: `unresolved`을 수집/검증 완결성으로 정의하고 `passed`(전체 gate 성공)와 분리했다. run/resume exit 0은 `unresolved=0`(모든 슬롯이 웹 검증 증거 보유)을 요구하며, 독립 실패 축은 stdout에 남고 `score`로 분류된다. 기존 attempt 바이트 변경 없음. - 2026-08-12: caller terminal을 closed typed observation으로 바꾸고 product/harness/process 결과, failure-inclusive artifact gate, 독립 CLI/report/scoring gate와 direct-first qualification을 구현했다. - 2026-08-12: official Claude result-direct/API-error 및 agy ERROR terminal을 lifecycle 계약에 맞게 분리하고, timeout cleanup result/receipt가 같은 child exit snapshot을 사용하도록 동기화했다. diff --git a/apps/edge/internal/openai/anthropic_bridge_test.go b/apps/edge/internal/openai/anthropic_bridge_test.go index 975e237f..28819c66 100644 --- a/apps/edge/internal/openai/anthropic_bridge_test.go +++ b/apps/edge/internal/openai/anthropic_bridge_test.go @@ -9,6 +9,9 @@ import ( "strings" "testing" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + edgeservice "iop/apps/edge/internal/service" "iop/packages/go/config" iop "iop/proto/gen/iop" @@ -574,6 +577,39 @@ func TestAnthropicChatBridgeProviderError(t *testing.T) { } } +func TestAnthropicChatBridgeProviderRejectionObservationIsClassificationOnly(t *testing.T) { + core, observed := observer.New(zap.InfoLevel) + secret := "secret-provider-message-must-not-be-logged" + candidate := anthropicTestCandidate(t, "openai") + candidate.ActualModel = "served-chat" + fake := &providerFakeRunService{ + poolDispatchPath: string(edgeservice.ProviderPoolPathTunnel), + poolSelectedCandidate: candidate, + tunnelFrames: anthropicTunnelFrames( + http.StatusBadRequest, + "application/json", + []byte(`{"error":{"type":"invalid_request_error","message":"`+secret+`"}}`), + ), + } + srv := NewServer(config.EdgeOpenAIConf{}, fake, zap.New(core)) + srv.SetModelCatalog([]config.ModelCatalogEntry{{ID: "claude-route", Providers: map[string]string{"chat": "served-chat"}}}) + w := serveAnthropicRequest(srv, "/v1/messages", `{"model":"claude-route","max_tokens":16,"messages":[{"role":"user","content":"hello"}]}`) + if w.Code != http.StatusBadRequest { + t.Fatalf("status=%d body=%s", w.Code, w.Body.String()) + } + entries := observed.FilterMessage(anthropicChatBridgeRejectionLogMessage).All() + if len(entries) != 1 { + t.Fatalf("rejection observations=%d logs=%+v", len(entries), observed.All()) + } + context := entries[0].ContextMap() + if len(context) != 4 || context["surface"] != "messages" || context["bridge"] != "chat" || context["rejection_class"] != "provider_http" || context["http_status"] != int64(http.StatusBadRequest) { + t.Fatalf("unsafe or incomplete observation: %+v", context) + } + if strings.Contains(fmt.Sprint(context), secret) { + t.Fatalf("secret leaked to observation: %+v", context) + } +} + func TestAnthropicChatBridgeStreamFragmentationOrderAndTerminal(t *testing.T) { fixture := mustReadAnthropicFixture(t, "chat_bridge_stream.sse") parts := bytes.SplitN(fixture, []byte("---ANTHROPIC-OUTPUT---\n"), 2) diff --git a/apps/edge/internal/openai/anthropic_stream.go b/apps/edge/internal/openai/anthropic_stream.go index c2a4f18d..f24905ba 100644 --- a/apps/edge/internal/openai/anthropic_stream.go +++ b/apps/edge/internal/openai/anthropic_stream.go @@ -11,11 +11,25 @@ import ( "sync" "time" + "go.uber.org/zap" + edgeservice "iop/apps/edge/internal/service" "iop/packages/go/streamgate" iop "iop/proto/gen/iop" ) +const anthropicChatBridgeRejectionLogMessage = "edge_anthropic_chat_bridge_rejection" + +func (s *Server) observeAnthropicChatBridgeRejection(status int) { + s.logger.Info( + anthropicChatBridgeRejectionLogMessage, + zap.String("surface", "messages"), + zap.String("bridge", "chat"), + zap.String("rejection_class", "provider_http"), + zap.Int("http_status", status), + ) +} + type openAIChatStreamChunk struct { ID string `json:"id"` Model string `json:"model"` @@ -1132,6 +1146,9 @@ func (s *Server) writeAnthropicChatBridgeResponse(w http.ResponseWriter, r *http if status == 0 { status = http.StatusOK } + if status >= http.StatusBadRequest { + s.observeAnthropicChatBridgeRejection(status) + } for key, value := range frame.GetHeaders() { headers[key] = value } diff --git a/apps/edge/internal/openai/gemini_bridge.go b/apps/edge/internal/openai/gemini_bridge.go index c5aacef9..cad4cdb5 100644 --- a/apps/edge/internal/openai/gemini_bridge.go +++ b/apps/edge/internal/openai/gemini_bridge.go @@ -26,6 +26,8 @@ func newGeminiBridgeResponseWriter(target http.ResponseWriter, model string) *ge func (w *geminiBridgeResponseWriter) Header() http.Header { return w.header } +func (w *geminiBridgeResponseWriter) Status() int { return w.status } + func (w *geminiBridgeResponseWriter) WriteHeader(status int) { if w.status == 0 { w.status = status @@ -111,6 +113,7 @@ func geminiSafeErrorMessage(status int) string { } type geminiBridgeToolState struct { + id string name string thoughtSignature string arguments strings.Builder @@ -211,6 +214,12 @@ func (s *geminiBridgeStream) consumeSSEEvent(event []byte) error { if delta.Function.Name != "" { state.name = delta.Function.Name } + if delta.ID != "" { + if !geminiToolCallID.MatchString(delta.ID) || (state.id != "" && state.id != delta.ID) { + return fmt.Errorf("tool call id is invalid") + } + state.id = delta.ID + } if delta.ExtraContent.Google != nil && delta.ExtraContent.Google.ThoughtSignature != "" { state.thoughtSignature = delta.ExtraContent.Google.ThoughtSignature } @@ -254,6 +263,7 @@ func (s *geminiBridgeStream) emitTools() error { } sort.Ints(indices) parts := make([]any, 0, len(indices)) + seenIDs := make(map[string]struct{}) for _, index := range indices { state := s.tools[index] if !geminiPathToken.MatchString(state.name) { @@ -263,7 +273,15 @@ func (s *geminiBridgeStream) emitTools() error { if json.Unmarshal([]byte(state.arguments.String()), &args) != nil { return fmt.Errorf("tool arguments are invalid") } - part := map[string]any{"functionCall": map[string]any{"name": state.name, "args": args}} + call := map[string]any{"name": state.name, "args": args} + if state.id != "" { + if _, duplicate := seenIDs[state.id]; duplicate { + return fmt.Errorf("duplicate tool call id") + } + seenIDs[state.id] = struct{}{} + call["id"] = state.id + } + part := map[string]any{"functionCall": call} if state.thoughtSignature != "" { part["thoughtSignature"] = state.thoughtSignature } diff --git a/apps/edge/internal/openai/gemini_handler.go b/apps/edge/internal/openai/gemini_handler.go index 2a57c496..76725b24 100644 --- a/apps/edge/internal/openai/gemini_handler.go +++ b/apps/edge/internal/openai/gemini_handler.go @@ -8,9 +8,21 @@ import ( "net/http" "regexp" "strings" + + "go.uber.org/zap" ) var geminiPathToken = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*$`) +var geminiToolCallID = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`) + +const geminiRejectionLogMessage = "edge_gemini_rejection" + +type geminiRejectionClass string + +const ( + geminiRejectionPreIngress geminiRejectionClass = "pre_ingress" + geminiRejectionProviderHTTP geminiRejectionClass = "provider_http" +) func isGeminiRequest(r *http.Request) bool { return r != nil && strings.HasPrefix(r.URL.Path, geminiPathPrefix) @@ -22,25 +34,40 @@ func writeGeminiError(w http.ResponseWriter, status int, code, message string) { }}) } +func (s *Server) observeGeminiRejection(class geminiRejectionClass, status int) { + s.logger.Info( + geminiRejectionLogMessage, + zap.String("surface", "gemini"), + zap.String("bridge", "chat"), + zap.String("rejection_class", string(class)), + zap.Int("http_status", status), + ) +} + +func (s *Server) writeGeminiPreIngressError(w http.ResponseWriter, status int, code, message string) { + s.observeGeminiRejection(geminiRejectionPreIngress, status) + writeGeminiError(w, status, code, message) +} + func (s *Server) handleGeminiStreamGenerateContent(w http.ResponseWriter, r *http.Request) { if r.Method != http.MethodPost { - writeGeminiError(w, http.StatusMethodNotAllowed, "INVALID_ARGUMENT", "method not allowed") + s.writeGeminiPreIngressError(w, http.StatusMethodNotAllowed, "INVALID_ARGUMENT", "method not allowed") return } routeID, callerModel, err := parseGeminiStreamPath(r) if err != nil { - writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request path is invalid") + s.writeGeminiPreIngressError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request path is invalid") return } defer r.Body.Close() body, err := readOpenAIIngressBody(w, r, s.maxIngressSnapshotBytes()) if err != nil { - writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") + s.writeGeminiPreIngressError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") return } chatBody, err := prepareGeminiChatBridge(body, routeID) if err != nil { - writeGeminiError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") + s.writeGeminiPreIngressError(w, http.StatusBadRequest, "INVALID_ARGUMENT", "request body is invalid") return } @@ -58,6 +85,9 @@ func (s *Server) handleGeminiStreamGenerateContent(w http.ResponseWriter, r *htt bridge := newGeminiBridgeResponseWriter(w, callerModel) s.handleChatCompletions(bridge, internal) + if bridge.Status() >= http.StatusBadRequest { + s.observeGeminiRejection(geminiRejectionProviderHTTP, bridge.Status()) + } bridge.Finish() } @@ -113,7 +143,7 @@ func prepareGeminiChatBridge(body []byte, routeID string) ([]byte, error) { } messages = append(messages, map[string]any{"role": "system", "content": text}) } - pendingCalls := make(map[string][]string) + pendingCalls := newGeminiPendingCalls() for contentIndex, content := range req.Contents { converted, err := geminiContentToChat(content, contentIndex, pendingCalls) if err != nil { @@ -312,7 +342,72 @@ func geminiTextOnly(content geminiContent) (string, error) { return strings.Join(texts, "\n"), nil } -func geminiContentToChat(content geminiContent, contentIndex int, pending map[string][]string) ([]map[string]any, error) { +type geminiPendingCall struct { + id string + name string + explicit bool +} + +type geminiPendingCalls struct { + byID map[string]geminiPendingCall + byName map[string][]string +} + +func newGeminiPendingCalls() *geminiPendingCalls { + return &geminiPendingCalls{byID: make(map[string]geminiPendingCall), byName: make(map[string][]string)} +} + +func (p *geminiPendingCalls) add(name, id string, explicit bool) error { + if _, duplicate := p.byID[id]; duplicate { + return fmt.Errorf("duplicate functionCall id") + } + p.byID[id] = geminiPendingCall{id: id, name: name, explicit: explicit} + p.byName[name] = append(p.byName[name], id) + return nil +} + +func (p *geminiPendingCalls) consume(name, id string) (string, error) { + if id != "" { + if !geminiToolCallID.MatchString(id) { + return "", fmt.Errorf("functionResponse id is invalid") + } + call, ok := p.byID[id] + if !ok || call.name != name { + return "", fmt.Errorf("functionResponse has no matching call") + } + p.remove(call) + return id, nil + } + ids := p.byName[name] + if len(ids) == 0 { + return "", fmt.Errorf("functionResponse has no matching call") + } + call, ok := p.byID[ids[0]] + if !ok || call.explicit { + return "", fmt.Errorf("functionResponse id is required") + } + p.remove(call) + return call.id, nil +} + +func (p *geminiPendingCalls) remove(call geminiPendingCall) { + delete(p.byID, call.id) + ids := p.byName[call.name] + for index, id := range ids { + if id != call.id { + continue + } + ids = append(ids[:index], ids[index+1:]...) + break + } + if len(ids) == 0 { + delete(p.byName, call.name) + } else { + p.byName[call.name] = ids + } +} + +func geminiContentToChat(content geminiContent, contentIndex int, pending *geminiPendingCalls) ([]map[string]any, error) { role := strings.ToLower(strings.TrimSpace(content.Role)) if role != "user" && role != "model" { return nil, fmt.Errorf("content role is invalid") @@ -356,8 +451,16 @@ func geminiContentToChat(content geminiContent, contentIndex int, pending map[st if json.Unmarshal(part.FunctionCall.Args, &args) != nil { return nil, fmt.Errorf("functionCall args are invalid") } - callID := fmt.Sprintf("gemini_call_%d_%d", contentIndex, partIndex) - pending[part.FunctionCall.Name] = append(pending[part.FunctionCall.Name], callID) + callID := part.FunctionCall.ID + explicitID := callID != "" + if !explicitID { + callID = fmt.Sprintf("gemini_call_%d_%d", contentIndex, partIndex) + } else if !geminiToolCallID.MatchString(callID) { + return nil, fmt.Errorf("functionCall id is invalid") + } + if err := pending.add(part.FunctionCall.Name, callID, explicitID); err != nil { + return nil, err + } call := map[string]any{"id": callID, "type": "function", "function": map[string]any{"name": part.FunctionCall.Name, "arguments": string(part.FunctionCall.Args)}} if part.ThoughtSignature != "" { call["extra_content"] = openAIChatThoughtSignature(part.ThoughtSignature) @@ -369,13 +472,14 @@ func geminiContentToChat(content geminiContent, contentIndex int, pending map[st if role != "user" || !geminiPathToken.MatchString(response.Name) { return nil, fmt.Errorf("functionResponse is invalid") } - ids := pending[response.Name] - if len(ids) == 0 || !json.Valid(response.Response) { + if !json.Valid(response.Response) { return nil, fmt.Errorf("functionResponse has no matching call") } - callID := ids[0] - pending[response.Name] = ids[1:] - toolMessages = append(toolMessages, map[string]any{"role": "tool", "tool_call_id": callID, "tool_name": response.Name, "content": string(response.Response)}) + callID, err := pending.consume(response.Name, response.ID) + if err != nil { + return nil, err + } + toolMessages = append(toolMessages, map[string]any{"role": "tool", "tool_call_id": callID, "content": string(response.Response)}) } if role == "model" { message := map[string]any{"role": "assistant", "content": strings.Join(texts, "\n")} diff --git a/apps/edge/internal/openai/gemini_handler_test.go b/apps/edge/internal/openai/gemini_handler_test.go index ed8b2757..db54de4d 100644 --- a/apps/edge/internal/openai/gemini_handler_test.go +++ b/apps/edge/internal/openai/gemini_handler_test.go @@ -2,11 +2,16 @@ package openai import ( "encoding/json" + "fmt" "net/http" "net/http/httptest" "strings" "testing" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + edgeservice "iop/apps/edge/internal/service" "iop/packages/go/config" iop "iop/proto/gen/iop" ) @@ -113,6 +118,101 @@ func TestGeminiRequestBridgePreservesToolsAndThoughtSignature(t *testing.T) { t.Fatalf("missing %s in %s", want, text) } } + if strings.Contains(text, `"tool_name"`) { + t.Fatalf("nonstandard tool result field reached Chat bridge: %s", text) + } +} + +func TestGeminiRequestBridgeMatchesExplicitToolCallIDsOutOfOrder(t *testing.T) { + converted, err := prepareGeminiChatBridge([]byte(`{ + "contents":[ + {"role":"model","parts":[ + {"functionCall":{"id":"call_first","name":"lookup","args":{"q":"first"}}}, + {"functionCall":{"id":"call_second","name":"lookup","args":{"q":"second"}}} + ]}, + {"role":"user","parts":[ + {"functionResponse":{"id":"call_second","name":"lookup","response":{"value":2}}}, + {"functionResponse":{"id":"call_first","name":"lookup","response":{"value":1}}} + ]} + ] + }`), "gemini-route") + if err != nil { + t.Fatal(err) + } + var body struct { + Messages []map[string]any `json:"messages"` + } + if err := json.Unmarshal(converted, &body); err != nil { + t.Fatal(err) + } + if len(body.Messages) != 3 { + t.Fatalf("messages=%+v", body.Messages) + } + if body.Messages[1]["tool_call_id"] != "call_second" || body.Messages[2]["tool_call_id"] != "call_first" { + t.Fatalf("explicit response order was not preserved: %+v", body.Messages) + } + for _, message := range body.Messages[1:] { + if _, exists := message["tool_name"]; exists { + t.Fatalf("nonstandard tool_name field survived: %+v", message) + } + } +} + +func TestGeminiRequestBridgeKeepsIDLessFIFOCompatibility(t *testing.T) { + converted, err := prepareGeminiChatBridge([]byte(`{ + "contents":[ + {"role":"model","parts":[ + {"functionCall":{"name":"lookup","args":{"q":"first"}}}, + {"functionCall":{"name":"lookup","args":{"q":"second"}}} + ]}, + {"role":"user","parts":[ + {"functionResponse":{"name":"lookup","response":{"value":1}}}, + {"functionResponse":{"name":"lookup","response":{"value":2}}} + ]} + ] + }`), "gemini-route") + if err != nil { + t.Fatal(err) + } + var body struct { + Messages []map[string]any `json:"messages"` + } + if err := json.Unmarshal(converted, &body); err != nil { + t.Fatal(err) + } + if len(body.Messages) != 3 || body.Messages[1]["tool_call_id"] != "gemini_call_0_0" || body.Messages[2]["tool_call_id"] != "gemini_call_0_1" { + t.Fatalf("ID-less FIFO changed: %+v", body.Messages) + } +} + +func TestGeminiRequestBridgeRejectsInvalidToolCallIdentity(t *testing.T) { + for _, tc := range []struct { + name string + body string + }{ + { + name: "duplicate id", + body: `{"contents":[{"role":"model","parts":[{"functionCall":{"id":"call_1","name":"a","args":{}}},{"functionCall":{"id":"call_1","name":"b","args":{}}}]}]}`, + }, + { + name: "mismatched name", + body: `{"contents":[{"role":"model","parts":[{"functionCall":{"id":"call_1","name":"a","args":{}}}]},{"role":"user","parts":[{"functionResponse":{"id":"call_1","name":"b","response":{}}}]}]}`, + }, + { + name: "missing response id for explicit call", + body: `{"contents":[{"role":"model","parts":[{"functionCall":{"id":"call_1","name":"a","args":{}}}]},{"role":"user","parts":[{"functionResponse":{"name":"a","response":{}}}]}]}`, + }, + { + name: "invalid id", + body: `{"contents":[{"role":"model","parts":[{"functionCall":{"id":"call id","name":"a","args":{}}}]}]}`, + }, + } { + t.Run(tc.name, func(t *testing.T) { + if _, err := prepareGeminiChatBridge([]byte(tc.body), "gemini-route"); err == nil { + t.Fatal("invalid identity must fail") + } + }) + } } func TestGeminiRequestBridgeUsesProviderNativeThinkingEnvelope(t *testing.T) { @@ -186,7 +286,7 @@ func TestGeminiRequestBridgeRejectsConflictingStructuredSchemas(t *testing.T) { func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) { w := httptest.NewRecorder() stream := newGeminiBridgeStream(w, "gemini-3.6-flash") - input := "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\"}}]}}]}\n\n" + + input := "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_provider_1\",\"function\":{\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\"}}]}}]}\n\n" + "data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"function\":{\"arguments\":\"\\\"x\\\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\n" + "data: [DONE]\n\n" if err := stream.Feed([]byte(input)); err != nil { @@ -195,6 +295,9 @@ func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) { if strings.Count(w.Body.String(), `"functionCall"`) != 1 || !strings.Contains(w.Body.String(), `"finishReason":"STOP"`) { t.Fatalf("tool terminal mismatch: %s", w.Body.String()) } + if !strings.Contains(w.Body.String(), `"id":"call_provider_1"`) { + t.Fatalf("provider tool call id was not preserved: %s", w.Body.String()) + } usageWriter := httptest.NewRecorder() usageStream := newGeminiBridgeStream(usageWriter, "m") if err := usageStream.Feed([]byte("data: {\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":2,\"total_tokens\":5,\"prompt_tokens_details\":{\"cached_tokens\":1},\"completion_tokens_details\":{\"reasoning_tokens\":1}}}\n\ndata: [DONE]\n\n")); err != nil { @@ -214,3 +317,41 @@ func TestGeminiStreamBridgeEmitsBoundedToolOnce(t *testing.T) { t.Fatal("oversize tool arguments must fail") } } + +func TestGeminiRejectionObservationIsClassificationOnly(t *testing.T) { + core, observed := observer.New(zap.InfoLevel) + secret := "secret-fixture-must-not-be-logged" + srv := NewServer(config.EdgeOpenAIConf{BearerToken: "principal"}, &fakeRunService{}, zap.New(core)) + req := httptest.NewRequest(http.MethodPost, "/gemini/r/v1beta/models/m:streamGenerateContent?alt=sse", strings.NewReader(`{"contents":"`+secret+`"}`)) + req.Header.Set("X-Goog-Api-Key", "principal") + srv.routes().ServeHTTP(httptest.NewRecorder(), req) + + candidate := anthropicTestCandidate(t, "openai") + candidate.ActualModel = "served-chat" + providerBody := []byte(`{"error":{"message":"` + secret + `"}}`) + fake := &providerFakeRunService{ + poolDispatchPath: string(edgeservice.ProviderPoolPathTunnel), + poolSelectedCandidate: candidate, + tunnelFrames: anthropicTunnelFrames(http.StatusBadRequest, "application/json", providerBody), + } + providerServer := NewServer(config.EdgeOpenAIConf{BearerToken: "principal"}, fake, zap.New(core)) + providerServer.SetModelCatalog([]config.ModelCatalogEntry{{ID: "r", Providers: map[string]string{"chat": "served-chat"}}}) + providerReq := httptest.NewRequest(http.MethodPost, "/gemini/r/v1beta/models/m:streamGenerateContent?alt=sse", strings.NewReader(`{"contents":[{"role":"user","parts":[{"text":"hello"}]}]}`)) + providerReq.Header.Set("X-Goog-Api-Key", "principal") + providerServer.routes().ServeHTTP(httptest.NewRecorder(), providerReq) + + entries := observed.FilterMessage(geminiRejectionLogMessage).All() + if len(entries) != 2 { + t.Fatalf("rejection observations=%d logs=%+v", len(entries), observed.All()) + } + wantClasses := []string{string(geminiRejectionPreIngress), string(geminiRejectionProviderHTTP)} + for index, entry := range entries { + context := entry.ContextMap() + if len(context) != 4 || context["surface"] != "gemini" || context["bridge"] != "chat" || context["rejection_class"] != wantClasses[index] || context["http_status"] != int64(http.StatusBadRequest) { + t.Fatalf("unsafe or incomplete observation: %+v", context) + } + if strings.Contains(fmt.Sprint(context), secret) { + t.Fatalf("secret leaked to observation: %+v", context) + } + } +} diff --git a/apps/edge/internal/openai/gemini_types.go b/apps/edge/internal/openai/gemini_types.go index b21bdbc1..d701f77e 100644 --- a/apps/edge/internal/openai/gemini_types.go +++ b/apps/edge/internal/openai/gemini_types.go @@ -29,11 +29,13 @@ type geminiPart struct { } type geminiFunctionCall struct { + ID string `json:"id,omitempty"` Name string `json:"name"` Args json.RawMessage `json:"args"` } type geminiFunctionResponse struct { + ID string `json:"id,omitempty"` Name string `json:"name"` Response json.RawMessage `json:"response"` } diff --git a/docs/agent-comparison-benchmark-dev-guide.md b/docs/agent-comparison-benchmark-dev-guide.md index 45161186..73e46098 100644 --- a/docs/agent-comparison-benchmark-dev-guide.md +++ b/docs/agent-comparison-benchmark-dev-guide.md @@ -368,7 +368,9 @@ python3 scripts/agent_comparison_benchmark.py preflight --manifest "$direct_mani python3 scripts/agent_comparison_benchmark.py run --manifest "$direct_manifest" ``` -direct canary는 unscored이며 각 최신 slot의 `product=succeeded`, `harness=passed`, `process=exited`/exit 0/no signal, `artifact=passed`가 모두 5/5여야 한다. `running`, `interrupted`, `unresolved`는 0이어야 한다. 실패 시 hybrid 또는 9-cell scored run을 할당하지 않는다. +direct canary는 unscored 진단이다. fresh preflight가 `ready=5`여야 하고, 정확히 5개의 fresh attempt를 한 번씩 할당하며, `unresolved=0`, `running=0`, `interrupted=0`과 함께 각 slot의 controller/product/harness/process/web-validation terminal evidence가 있어야 한다. 최대 3회 Chromium/CDP 재시도 뒤에도 `browser_*` 또는 `cdp_*` infrastructure block이 남거나 Edge pre-ingress incompatibility가 확인되면 qualification을 중단한다. + +`product=failed`, upstream HTTP rejection, caller failure 뒤 `generated_missing`, timeout은 terminal evidence가 완결되면 측정 결과다. 이를 성공으로 바꾸거나 암묵 재시도하지 않으며, 그 자체로 다음 구현 cycle을 만들지 않는다. scoring eligibility는 각 독립 gate로 별도 판정한다. ### 11.3 Public nine-cell preflight @@ -396,7 +398,7 @@ preflight는 다음을 함께 확인한다. `registration_required` 또는 `implementation_gap`이면 즉시 중단한다. alias, model, effort, route나 caller를 대체하지 않는다. preflight-only run root는 evidence이므로 삭제하지 않는다. -direct canary 5/5 뒤 fresh one-shot preflight가 `ready=9`인지 확인하고 멈춘다. 이 qualification 단계에서는 hybrid canary나 C01-C09 `run`을 호출하지 않는다. +terminal-evidence direct canary admission 뒤 fresh one-shot preflight가 `ready=9`인지 확인하고 멈춘다. 이 qualification 단계에서는 hybrid canary나 C01-C09 `run`을 호출하지 않는다. ### 11.4 Scored execution @@ -426,9 +428,9 @@ python3 scripts/agent_comparison_benchmark.py status \ - `running = 0` - `interrupted = 0` - 각 cell/repetition에 retained terminal attempt가 존재 -- 최신 attempt에서 `product_succeeded=9`, `harness_passed=9`, `process_exited=9`, `artifact_passed=9`, `unresolved=0` +- 모든 최신 attempt에 controller/product/harness/process/web-validation terminal evidence가 있고 `unresolved=0` -`completed`는 controller 종료만 뜻하며 product 성공을 뜻하지 않는다. product/harness/process/artifact 실패는 scoring eligibility와 최종 비교에서 각각 별도로 표시된다. +`completed`는 controller 종료만 뜻하며 product 성공을 뜻하지 않는다. product/harness/process/artifact 실패는 scoring eligibility와 최종 비교에서 각각 별도로 표시되고, terminal 실패를 성공 결과로 바꾸거나 암묵 재시도하지 않는다. ### 11.6 Blind scoring @@ -558,6 +560,7 @@ incident evidence를 보존하기 전에 repository와 run output에서 secret - [ ] dev runtime source/build identity, process와 listener 확인 - [ ] managed projection, provider slot/route와 no-legacy-fallback 확인 - [ ] config observation이 runtime route/stage와 정확히 일치 +- [ ] direct qualification `ready=5`, fresh attempt 5개, `unresolved=0`, `running=0`, `interrupted=0`, terminal evidence 5개와 infrastructure block 없음 - [ ] public preflight `ready=9` - [ ] 현재 plan이 exactly one scored run을 소유하고 사용자 권한이 명확함 - [ ] old run resume/retry/state edit 계획 없음 diff --git a/scripts/agent_benchmark/browser_cdp.py b/scripts/agent_benchmark/browser_cdp.py index adf216ed..469e0c1d 100644 --- a/scripts/agent_benchmark/browser_cdp.py +++ b/scripts/agent_benchmark/browser_cdp.py @@ -606,7 +606,53 @@ class BrowserRenderer: binary = shutil.which(binary) or "" if not binary: raise BrowserError("browser_unavailable") + transient = { + "browser_start_failed", + "browser_cdp_unavailable", + "CDP handshake failed", + "CDP handshake closed", + "CDP socket read failed", + "CDP socket closed", + "CDP socket write failed", + } + for attempt in range(3): + try: + return self._render_once( + root=root, + out=out, + viewport_list=viewport_list, + targets=targets, + binary=binary, + timeout_seconds=timeout_seconds, + ) + except BrowserError as exc: + if str(exc) not in transient or attempt == 2: + raise + self._remove_partial_screenshots(out, viewport_list) + raise BrowserError("browser_restart_exhausted") + @staticmethod + def _remove_partial_screenshots(output_root: Path, viewports) -> None: + for viewport in viewports: + target = output_root / f"screenshot-{getattr(viewport, 'id', '')}.png" + try: + if target.is_file() and not target.is_symlink(): + target.unlink() + elif target.exists() or target.is_symlink(): + raise BrowserError("screenshot_cleanup_failed") + except OSError as exc: + raise BrowserError("screenshot_cleanup_failed") from exc + + def _render_once( + self, + *, + root: Path, + out: Path, + viewport_list, + targets: list[Path], + binary: str, + timeout_seconds: int, + ) -> RenderObservation: deadline = time.monotonic() + max(3, timeout_seconds) server = _StaticServer(root) thread = threading.Thread(target=server.serve_forever, daemon=True) diff --git a/scripts/agent_benchmark/browser_cdp_test.py b/scripts/agent_benchmark/browser_cdp_test.py index 9cdf4698..2b0068d4 100644 --- a/scripts/agent_benchmark/browser_cdp_test.py +++ b/scripts/agent_benchmark/browser_cdp_test.py @@ -327,7 +327,7 @@ class BrowserProtocolTest(unittest.TestCase): "signal.signal(signal.SIGTERM, signal.SIG_IGN)\n" "child = subprocess.Popen([sys.executable, '-c', " "'import signal,time;signal.signal(signal.SIGTERM,signal.SIG_IGN);time.sleep(60)'])\n" - f"with open({str(pid_path)!r}, 'w', encoding='ascii') as handle:\n" + f"with open({str(pid_path)!r}, 'a', encoding='ascii') as handle:\n" " handle.write(f'{os.getpid()} {child.pid}\\n')\n" " handle.flush()\n" " os.fsync(handle.fileno())\n" @@ -349,7 +349,7 @@ class BrowserProtocolTest(unittest.TestCase): timeout_seconds=1, ) pids = [int(value) for value in pid_path.read_text().split()] - self.assertEqual(len(pids), 2) + self.assertEqual(len(pids), 6) for pid in pids: with self.subTest(pid=pid), self.assertRaises(ProcessLookupError): os.kill(pid, 0) @@ -360,6 +360,56 @@ class BrowserProtocolTest(unittest.TestCase): except ProcessLookupError: pass + def test_renderer_restarts_only_closed_transient_failures(self): + with tempfile.TemporaryDirectory() as raw: + root = Path(raw) + (root / "index.html").write_text("
unused
", encoding="utf-8") + viewport = SimpleNamespace(id="desktop", width=800, height=600) + renderer = BrowserRenderer(sys.executable) + calls = 0 + expected = object() + + def transient_then_success(**_kwargs): + nonlocal calls + calls += 1 + screenshot = root / "screenshot-desktop.png" + if calls < 3: + screenshot.write_bytes(b"partial") + raise BrowserError("CDP socket closed") + self.assertFalse(screenshot.exists()) + return expected + + renderer._render_once = transient_then_success + self.assertIs( + renderer.render( + workspace_root=root, + output_root=root, + viewports=(viewport,), + timeout_seconds=1, + ), + expected, + ) + self.assertEqual(calls, 3) + + for error in ("CDP socket closed", "screenshot_collision"): + with self.subTest(error=error): + calls = 0 + + def fail(**_kwargs): + nonlocal calls + calls += 1 + raise BrowserError(error) + + renderer._render_once = fail + with self.assertRaisesRegex(BrowserError, error): + renderer.render( + workspace_root=root, + output_root=root, + viewports=(viewport,), + timeout_seconds=1, + ) + self.assertEqual(calls, 3 if error == "CDP socket closed" else 1) + class _Counter(BaseHTTPRequestHandler): count = 0 diff --git a/scripts/agent_benchmark/skill_contract_test.py b/scripts/agent_benchmark/skill_contract_test.py index c743534d..1bb23b4e 100644 --- a/scripts/agent_benchmark/skill_contract_test.py +++ b/scripts/agent_benchmark/skill_contract_test.py @@ -374,6 +374,11 @@ class BenchmarkSkillContractTest(unittest.TestCase): stop_conditions, "Stop conditions must reference explicit --retry-failed resume gating", ) + self.assertIn( + "complete terminal evidence may continue to the fresh nine-cell preflight", + stop_conditions, + "Release qualification must continue after evidence-complete product failure without retry", + ) self.assertIn( "it does not require every gate to pass", safety, @@ -421,6 +426,17 @@ class BenchmarkSkillContractTest(unittest.TestCase): self.assertIn("process_exited=", skill_text) self.assertIn("artifact_passed=", skill_text) self.assertIn("five-cell direct manifest", skill_text) + for qualification_rule in ( + "fresh `ready=5`", + "exactly five fresh attempts", + "terminal controller/product/harness/process/web-validation evidence for every slot", + "no exhausted browser/CDP infrastructure block", + "Product failure, upstream HTTP rejection, generated-missing after caller failure, and timeout remain measured outcomes", + "do not trigger an implicit retry", + "Edge pre-ingress incompatibility or an exhausted browser/CDP infrastructure block stops qualification", + ): + self.assertIn(qualification_rule, skill_text) + self.assertNotIn("requires all four gates for all five cells", skill_text) # ------------------------------------------------------------------ # Template / frontmatter invariants