fix(node,edge): provider stall watchdog 구현을 제거한다
This commit is contained in:
parent
a1816a7fac
commit
4e8075d71a
88 changed files with 1004 additions and 11276 deletions
|
|
@ -21,7 +21,6 @@
|
|||
- `apps/edge/internal/service/model_queue_release.go`
|
||||
- `apps/edge/internal/service/model_queue_snapshot.go`
|
||||
- `apps/edge/internal/openai/stream_gate_runtime.go`
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
|
||||
- `apps/node/internal/adapters/config_set.go`
|
||||
- human docs: `apps/edge/README.md`
|
||||
|
||||
|
|
@ -52,9 +51,8 @@ tracked config에는 public 예시와 기본 구조만 두고, 실제 endpoint/c
|
|||
- `ConcreteProtocolProfile.ResolveOperationURL(op)`는 완성된 resolved upstream URL을 반환한다. absolute operation URL은 그대로 보존하며 relative operation path는 normalized base URL에 1회 join된다. 표기된 `/v1/...` 값은 return value가 아니라 operation-path input이다 (`models` → `GET /v1/models` 또는 `GET /anthropic/v1/models`, `chat_completions` → `POST /v1/chat/completions`, `messages` → `POST /v1/messages`, `count_tokens` → `POST /v1/messages/count_tokens`, `responses` → `POST /v1/responses`).
|
||||
- `validOperationsByDriver`는 driver별 허용 operation의 closed set이다. `openai_chat`은 `models`, `chat_completions`, `responses`, `count_tokens`를 허용한다. `anthropic_messages`는 `models`, `messages`, `count_tokens`를 허용한다. `openai_responses`는 `models`, `responses`, `count_tokens`를 허용한다.
|
||||
- `openai.provider_auth` is a legacy-mode-only request-time raw provider token forwarding rule. `enabled=false` is the default; when enabled in legacy mode, omitted fields resolve to `from_header=X-IOP-Provider-Authorization`, `target_header=Authorization`, `scheme=Bearer`, and `required=true`. Managed mode rejects this configuration and rejects a caller-supplied legacy provider credential header.
|
||||
- `openai.stream_evidence_gate` configures request-local Recovery Coordinator limits, the ingress snapshot bound, and optional semantic policy. Every supported Chat Completions, normalized Responses, provider tunnel, provider-pool, and tool-validation response already uses the `packages/go/streamgate` request runtime as its sole liveness owner. `enabled` defaults to false and controls only configured semantic filter registration/capability admission; false preserves endpoint-native compatibility inside the same runtime and does not restore a legacy response or retry owner. `max_request_fault_recovery`는 요청당 전체 fault recovery 상한(`0..3`, omitted 기본값 3, explicit 0은 모든 fault recovery 비활성화)이다. `max_strategy_fault_recovery`는 fault strategy(exact_replay/continuation_repair/schema_repair)별 상한(`0..max_request_fault_recovery`, omitted 기본값은 effective request total 상속, explicit 0은 해당 strategy 비활성화)이며 request-start 시점에 immutable runtime option snapshot으로 각 fault strategy에 동일하게 적용된다. base 또는 selector에서 활성화될 수 있는 blocking `repeat_guard`는 continuation recovery를 생성하므로 두 recovery 상한이 모두 최소 1이어야 하며, 모순된 설정은 load 시 거부한다. `max_ingress_snapshot_bytes`는 ingress snapshot 바이트 상한(`1..16777216` [16 MiB], omitted/0 기본값 16 MiB)이다. `environment`는 request-start selector snapshot이며 `dev|dev-corp`만 허용하고 omitted 기본값은 `dev`다. `filters[]`는 unique `filter` (`repeat_guard|schema_gate|provider_error`) policy이다. `enabled` omitted=true, `enforcement` omitted=`blocking`, `capability` omitted=`output.<filter>`, `hold_evidence_runes` omitted=500, `timeout_ms` omitted=5000으로 정규화하며 selector는 `environment|model_group|model|provider`로만 filter enablement/enforcement를 보정한다. base-disabled filter도 registry snapshot에 남아 더 구체적인 selector가 활성화할 수 있고, 실제 target에서 활성화된 `blocking` filter만 provider capability admission에 참여한다. `observe_only`는 evidence를 만들지만 admission을 막지 않는다. `repeat_guard` uses the configured rune bound for active request-local history/current-stream inspection and stores only bounded fingerprints, counts, and offsets in its semantic snapshot and observations. `schema_gate` and `provider_error` remain lifecycle foundations until their matcher Tasks; an unmatched provider error never creates exact replay. Config accepts no caller/agent selector.
|
||||
- `openai.stream_evidence_gate` configures request-local Recovery Coordinator limits, the ingress snapshot bound, and optional semantic policy. Every supported Chat Completions, normalized Responses, provider tunnel, provider-pool, and tool-validation response uses the `packages/go/streamgate` request runtime for response staging and configured semantic arbitration. `enabled` defaults to false and controls only configured semantic filter registration/capability admission; false preserves endpoint-native compatibility inside the same runtime and does not restore a legacy response or retry owner. `max_request_fault_recovery`는 요청당 전체 fault recovery 상한(`0..3`, omitted 기본값 3, explicit 0은 모든 fault recovery 비활성화)이다. `max_strategy_fault_recovery`는 fault strategy(exact_replay/continuation_repair/schema_repair)별 상한(`0..max_request_fault_recovery`, omitted 기본값은 effective request total 상속, explicit 0은 해당 strategy 비활성화)이며 request-start 시점에 immutable runtime option snapshot으로 각 fault strategy에 동일하게 적용된다. base 또는 selector에서 활성화될 수 있는 blocking `repeat_guard`는 continuation recovery를 생성하므로 두 recovery 상한이 모두 최소 1이어야 하며, 모순된 설정은 load 시 거부한다. `max_ingress_snapshot_bytes`는 ingress snapshot 바이트 상한(`1..16777216` [16 MiB], omitted/0 기본값 16 MiB)이다. `environment`는 request-start selector snapshot이며 `dev|dev-corp`만 허용하고 omitted 기본값은 `dev`다. `filters[]`는 unique `filter` (`repeat_guard|schema_gate|provider_error`) policy이다. `enabled` omitted=true, `enforcement` omitted=`blocking`, `capability` omitted=`output.<filter>`, `hold_evidence_runes` omitted=500, `timeout_ms` omitted=5000으로 정규화하며 selector는 `environment|model_group|model|provider`로만 filter enablement/enforcement를 보정한다. base-disabled filter도 registry snapshot에 남아 더 구체적인 selector가 활성화할 수 있고, 실제 target에서 활성화된 `blocking` filter만 provider capability admission에 참여한다. `observe_only`는 evidence를 만들지만 admission을 막지 않는다. `repeat_guard` uses the configured rune bound for active request-local history/current-stream inspection and stores only bounded fingerprints, counts, and offsets in its semantic snapshot and observations. `schema_gate` and `provider_error` remain lifecycle foundations until their matcher Tasks; an unmatched provider error never creates exact replay. Config accepts no caller/agent selector.
|
||||
- `openai.stream_evidence_gate` 설정은 request-start 시점에 snapshot으로 고정되며 in-flight request의 실행 중 refresh 영향에서 격리된다 (generation isolation). 새 generation의 설정은 이후 시작되는 새 request에만 적용된다.
|
||||
- The internal `response_stalled` recovery registration is always present for a supported OpenAI runtime request. It is not a member of `filters[]`, has no configurable capability, and does not participate in provider capability admission. It consumes only an Edge-confirmed typed handoff; configurable `provider_error` keeps its generic foundation behavior.
|
||||
- The request-start `models[].context_window_tokens` snapshot is the resume builder's target context bound. Each Chat/Responses runtime shares one request-local content/reasoning recorder across its initial and recovery event sources. A continuation rebuild uses only that recorder and the fixed directive; unknown or exceeded context rejects the rebuild before re-admission. An omitted caller temperature selects `0.2`, `0.4`, then `0.6` by continuation strategy attempt, while an explicit value is preserved. Recorder state and its raw values remain request-local, are consumed once per attempt, and are never added to config refresh state or observations. Repeat history and counters are pinned to the same request-start config generation and are not refreshable TTL/session state.
|
||||
- `openai` deep diff는 restart-required로 분류한다. `openai.principal_tokens[]`, `openai.stream_evidence_gate`, top-level 및 `openai.model_routes[].provider_id` 변경은 restart-required classifier에 포함된다.
|
||||
- Any `credential_plane` mode/TTL/cache change, TLS identity change, Control Plane attachment change, or key path change is restart-required. A refresh cannot switch between managed and legacy credential ownership or rotate process-held signing/recipient material in place.
|
||||
|
|
@ -71,26 +69,25 @@ tracked config에는 public 예시와 기본 구조만 두고, 실제 endpoint/c
|
|||
- `execution_presets[].single_request`는 operator-owned fixed single-request policy다. 설정 시 preset은 `allowed_modes=["light"]`, `stages=[plan, work, review]`의 승인된 plan→work→review 경로를 고수한다. 절대 상한은 `wall_clock_ms ≤ 1800000`, `timeout_ms ≤ 600000`, `max_tool_iterations ≤ 64`, `max_output_bytes ≤ 16777216`이며 `timeout_ms`는 `wall_clock_ms`를 초과할 수 없다. selector와 plan/review stage는 `reasoning_effort=high`를 강제하고 work stage는 `reasoning_effort`를 선언할 수 없다. `workspace_ref`는 비어있을 수 없으며 raw path, credential, Node id, endpoint를 포함하지 않는다. `templates`의 optional `plan_file`/`review_file`은 edge.yaml 상대 경로의 8192-byte UTF-8 closed grammar이며, PLAN은 deterministic `P1..Pn`, REVIEW는 `Worker Item Status`/`Worker Changes`/`Worker Verification`/`Deviations`만 허용한다. 구형 reviewer-final template은 admission에서 fail closed한다. config refresh diff reporting 시 템플릿 파일 경로나 본문은 노출되지 않고 SHA-256 digest만 보고된다. single_request preset은 `workspace_tools`를 선언할 수 없다. catalog 변경과 mapping 변경은 live-apply로 분류되며 refresh 이후 새로 시작되는 logical request에만 적용된다. admitted single-request binding은 refresh 이후에도 frozen public model, stage binding, workspace reference, limits, effective templates를 유지한다.
|
||||
- `nodes[].providers[]`는 Node 아래 resource/provider catalog다. `category`는 `api`, `cli`, `local_inference` resource kind를 나타낸다.
|
||||
- `nodes[].providers[].type`의 `seulgivibe_claude`와 `seulgivibe_openai`는 runtime type을 `openai_compat`로 정규화한다. Edge가 Node adapter payload를 만들 때 명시 provider label이 없으면 원래 Seulgivibe type alias를 `OpenAICompatAdapterConfig.provider`로 보존한다.
|
||||
- `nodes[].providers[].response_stall_timeout_ms`는 provider-originated response-stall timeout을 밀리초 단위로 선언한다. 양수 값은 그대로 사용되고, 0 또는 생략은 문서화된 기본값 `60000`을 적용한다. 음수 값과 safe duration bound를 초과하는 양수 값은 `NodeProviderConf.Validate()`에서 거부한다. effective 값은 `NodeProviderConf.EffectiveResponseStallTimeoutMS()`에서 계산한다. 이 필드는 config refresh에서 `restart_required`로 분류되며, effective-zero 등가성(생략 vs 명시적 0)은 변경으로 보고되지 않는다. request hard timeout, queue timeout, heartbeat/disconnect, CLI `response_idle_timeout_ms`는 기존 소유권을 유지한다.
|
||||
- `nodes[].providers[].response_stall_timeout_ms`는 제거된 key다. Strict load는 provider entry의 exact key를 decode 전에 fail-closed로 거부한다. 다른 unknown-key policy를 확장하지 않으며 request hard timeout, queue timeout, provider HTTP timeout, heartbeat/disconnect, CLI `response_idle_timeout_ms`는 기존 소유권을 유지한다.
|
||||
- `nodes[].providers[].id`는 전체 Edge config 안에서 중복되면 안 된다.
|
||||
- `nodes[].providers[].adapter`는 같은 Node 안의 enabled adapter instance key를 참조해야 한다. Exact instance key를 우선하고, legacy type-name route는 같은 type의 enabled instance가 정확히 하나일 때만 허용한다.
|
||||
- `nodes[].providers[].enabled`: 생략 또는 `true` → provider pool dispatch 후보에 포함. `false` → dispatch pool에서 제외. 비활성화된 provider는 status snapshot에 `status=disabled`, `health=disabled`, `capacity=0`으로 표시된다. adapter process lifecycle 변경 없음. config refresh 시 `enabled` 토글은 live-apply(restart 불필요)로 분류된다. disabled provider의 adapter reference check는 skip되지만 structural validation(type, category, models, numeric bounds)은 수행된다.
|
||||
- `nodes[].providers[].capacity`와 `long_context_capacity`는 `node_id + provider_id` resource가 소유한다. 같은 key의 legacy adapter가 함께 있으면 adapter capacity는 transport 상한이며 provider capacity는 그 이하의 admission 한도로 설정할 수 있고, 상한 초과만 load에서 거부한다. 같은 provider를 참조하는 여러 `models[].id`는 일반·long slot을 합산 공유한다. `total_context_tokens`는 runtime counter가 아니라 `context_window_tokens * long_context_capacity` 이상이어야 하는 정적 load/refresh validation 값이다.
|
||||
- `nodes[].providers[].priority`: provider-pool dispatch tie-breaker다. 기본값은 `0`이고 음수는 validation error다. dispatch는 `in_flight < capacity` 후보 중 가장 낮은 `in_flight`를 먼저 선택하며, `in_flight`가 같은 후보에서만 낮은 숫자의 `priority`를 우선한다. `in_flight`와 `priority`가 모두 같으면 기존 순환을 유지한다. priority 변경은 live-apply(restart 불필요)로 분류된다.
|
||||
- Configured provider health remains an immutable input snapshot during request execution. Confirmed current bound runtime-unavailable evidence is stored separately under `(node_id, connection_generation, provider_id)`, gates effective admission, and projects the runtime ProviderSnapshot unavailable without changing `NodeProviderConf.Health`, refresh diffs, or Node config payloads. A later exact higher-sequence available CAPABILITIES probe or a newer connection generation clears effective exclusion under the runtime contract, not through config refresh.
|
||||
- After the queue makes that authoritative overlay decision, Edge emits bounded operational evidence only: `iop_edge_provider_health_evidence_total{source,evidence_health,decision}` and `iop_edge_provider_health_transitions_total{from_health,to_health}`, plus `edge_provider_health_observation`. Sources, health values, and decisions use closed vocabularies; provider/node/run/session/adapter/target identity, payloads, and credentials are excluded. The observer is post-lock and cannot validate or mutate config/overlay state.
|
||||
- Configured provider health remains an immutable input snapshot during request execution. A normal provider or transport terminal releases the current lease without mutating `NodeProviderConf.Health`, refresh diffs, or Node config payloads.
|
||||
- `nodes[].workspaces[]` is the operator-owned bounded capability catalog for each node. Each entry is keyed by a globally unique, trimmed `ref` and declares `platform` in the closed `darwin|linux` implementation set, `root` (absolute clean path other than `/`), closed-set `operations` (`read`, `list`, `write`, `delete`, `command`), approved `commands` (id + fixed executable + fixed args, present iff `command` is in operations), `environment_allowlist` (unique portable env var names), and bounded `max_read_bytes`, `max_write_bytes`, `max_output_bytes`, `max_command_timeout_ms` (each enabled `read`, `write`, `list`, or `command` operation requires its effective positive bound; absolute maxima are 1 GiB / 1 hour). Refs are globally unique across all nodes. An empty workspaces slice is backward-compatible on any host. A non-empty Node catalog requires a supported host and every entry platform must equal that host before any root is opened; Windows and unknown hosts fail closed. The catalog is compiled into `NodeRecord.Workspaces` at load time, delivered in the Node-private config payload, and retained immutably by the workspace runtime; runtime mutation is restart-required. Raw root paths and command details never enter execution presets, caller-visible responses, provider requests, or public metadata. `workspace_ref` in `execution_presets[].single_request` references one entry by ref; operating system is runtime ownership evidence, not a caller selector.
|
||||
- Config refresh classifies any `nodes[].workspaces` change (root, capability, command template, environment allowlist, or limits) as `restart_required`. Active requests must never observe a root/capability mutation.
|
||||
- legacy single-instance adapter 설정은 load 시 named instance slice로 normalize된다.
|
||||
- `NodeConfigPayload`는 Edge가 Node에 내려주는 실행 adapter/runtime payload다.
|
||||
- `provider_id`와 effective `usage_attribution`은 OpenAI route에서 Edge service dispatch result까지 보존되는 Edge-local attribution binding이다. `response_stall_timeout_ms`는 이 attribution과 별개로 선택된 provider의 effective timeout을 `RunRequest`와 `ProviderTunnelRequest` wire field에 보존한다.
|
||||
- `provider_id`와 effective `usage_attribution`은 OpenAI route에서 Edge service dispatch result까지 보존되는 Edge-local attribution binding이다.
|
||||
- refresh 결과는 `applied`, `restart_required`, `rejected`를 구분하고, changed node/provider/model/report slice는 안정적으로 non-nil이어야 한다.
|
||||
|
||||
## refresh 분류 기준
|
||||
|
||||
- live apply 가능: Edge root `long_context_threshold_tokens`, `provider_pool.max_queue`, `provider_pool.queue_timeout_ms`, provider capacity, provider long-context capacity, provider total-context validation budget, provider priority, provider `enabled` toggle, `models[]` display/context window/provider/generation/`usage_attribution` policy mapping, `models[].execution_preset` mapping, `execution_presets[]` preset catalog, legacy node runtime concurrency metadata. 기존 lease는 유지하며 새 admission과 모든 pending item은 새 policy/candidate 상태로 재평가한다. preset catalog/mapping 변경은 refresh 이후 새로 시작되는 logical request에만 반영된다.
|
||||
- restart required: credential-plane/TLS/key references, Edge identity/listen/bootstrap/logging/metrics/console/control-plane/openai/a2a listener config, node 추가/삭제, node token/alias/agent kind, adapter 설정, provider type/category/adapter/models/health/lifecycle capability, provider-first execution fields(`provider`, `endpoint`, `base_url`, `headers`, `command`, `args`, `env`, `mode`, `resume_args`, `output_format`, `context_size`, `request_timeout_ms`) 변경, `nodes[].workspaces` 변경 (root, capability, command template, environment allowlist, limits).
|
||||
- rejected: candidate config load/validate 실패, invalid refresh mode, apply failure.
|
||||
- rejected: candidate config load/validate 실패(제거된 provider key 포함), invalid refresh mode, apply failure.
|
||||
|
||||
## 금지 사항
|
||||
|
||||
|
|
|
|||
|
|
@ -58,11 +58,9 @@ Edge는 Node 연결을 수락하고, Node는 연결 직후 등록 요청을 보
|
|||
- disconnect/reconnect: current dispatch-ready owner의 close/heartbeat timeout만 해당 connection generation을 fence한다. Edge는 같은 authoritative lifecycle에서 provider lease를 정확히 한 번 반환하고 resource를 offline/excluded로 만든 뒤 queue를 live candidate 기준으로 재평가한다. accepted Node의 ready transition은 새 generation resource를 활성화하고 기존 waiter를 즉시 pump한다. stale/rejected connection callback은 live state나 lifecycle event를 바꾸지 않는다.
|
||||
- execution: Edge가 `RunRequest`를 보내고 Node가 `RunEvent` stream으로 실행 상태를 보낸다.
|
||||
- provider raw tunnel: Edge가 기존 Edge-Node socket으로 `ProviderTunnelRequest`를 보내고 Node가 provider HTTP/SSE 요청을 연 뒤 `ProviderTunnelFrame` stream으로 provider status/header/body/end/error/usage 후보를 sequence와 함께 돌려준다. 이 경로는 OpenAI-compatible provider passthrough용이며 `RunEvent` 실행 stream과 분리된다.
|
||||
- response_stall_timeout_ms: `RunRequest.response_stall_timeout_ms`와 `ProviderTunnelRequest.response_stall_timeout_ms`는 int64 필드로, 선택된 provider의 response-stall timeout을 밀리초 단위로 운반한다. Zero는 Node가 문서화된 기본값(60000ms)을 적용함을 의미한다. Negative 또는 overflow 값은 Node 경계에서 router/provider 호출 전에 reject된다. Edge provider-pool dispatch는 winning candidate의 effective timeout을 각 요청에 복사한다. Direct/non-pool 호출은 wire에서 zero를 사용하고 Node 기본값을 적용한다.
|
||||
- response stall terminal: Node observes only the execution activity contract. On expiry it cancels and fences the local provider attempt, joins the bounded close-grace fence and an independent exact-target health probe without extending either serially, then emits exactly one normalized `RunEvent{type=error}` or tunnel `ProviderTunnelFrame{kind=ERROR}` with `failure_code=response_stalled` and populates the optional wire `ExecutionFailure` field (field 13 on `RunEvent`, field 15 on `ProviderTunnelFrame`). Terminal metadata is allowlisted (three-way health evidence as the `provider_health` status paired with the `liveness_classification` normalization — `available`/`request_stalled`, `unavailable`/`provider_unhealthy`, or `unknown`/`health_unknown`; idle duration; Node-owned run/attempt identity; fence; adapter; target; and an optional connection-scoped `health_observation_seq`); it contains no caller-controlled identity, raw payload, credential, or `recovery_eligible`. Nil and non-stalled failures leave wire `ExecutionFailure` absent while preserving legacy error string fields (`RunEvent.Error` / `ProviderTunnelFrame.Error`). `health_observation_seq` starts at one per connection and increases uniquely across the connection's normalized and tunnel observations; an unbound session omits it. Probe availability is evidence only and never resets progress, changes the fence, or authorizes retry. A confirmed fence is a capability hint only, not Node retry authorization.
|
||||
- Edge terminal handoff: transport reception identity, not payload identity, supplies `(node_id, connection_generation)`. Before a normalized or tunnel terminal can affect provider health, Edge compares that identity and the typed adapter/target evidence with the tracked immutable provider lease. A current terminal releases that lease exactly once even when optional health evidence is rejected. Edge adds `provider_id`, validated `provider_health`, and `recovery_handoff=confirmed` to every validated current bound stall before downstream routing, including sequence-stale request-local handoff; only a fresh `unavailable` observation lowers the separate runtime overlay. The handoff token is not replay approval, and Edge never adds `recovery_eligible` here.
|
||||
- CAPABILITIES recovery probe: Node resolves the requested adapter instance, runs the bounded fail-closed exact-target `ProbeHealth`, and returns stable `adapter_key`, `target`, normalized `provider_status`, and the next Session-owned `health_observation_seq`. Edge retains the command's dispatch node/generation and may clear one unavailable overlay only when a higher-sequence `available` response identifies exactly one same-generation provider binding. Empty, malformed, ambiguous, mismatched, stale, `unknown`, and `unavailable` results do not change the overlay.
|
||||
- precedence and ownership: request hard deadline, caller cancellation, and session disconnect retain their existing boundary when they win before the watchdog. A session lifetime context cancels active run and tunnel handlers on disconnect. If provider return is not confirmed during the bounded close grace, Node emits and fences the terminal but retains admission, run-manager, credential, and adapter ownership until the provider actually returns.
|
||||
- removed field reservation: `RunRequest` field 12와 `ProviderTunnelRequest` field 16은 제거된 `response_stall_timeout_ms`의 number/name reservation이다. 양쪽 field number와 name은 새 의미로 재사용할 수 없다.
|
||||
- terminal ownership: provider output silence는 wire event나 상태 전이를 만들지 않는다. Provider `END`/`ERROR`, normalized `complete`/`error`/`cancelled`, caller cancellation, request hard deadline, and session disconnect retain their existing terminal ownership.
|
||||
- terminal cleanup: transport reception identity supplies `(node_id, connection_generation)`. A current normalized or tunnel terminal releases its tracked provider lease exactly once; it does not carry a response-silence recovery handoff, mutate provider health, start a probe, or authorize replay/provider avoidance.
|
||||
- managed credential delivery: after provider selection, Edge attaches an exact `CredentialLeaseBinding` and a short-lived signed lease sealed to the selected Node. The Node opens it only after adapter-capacity admission and immediately before provider execution, verifies signature, recipient, scope, expiry, and replay state, injects the declared auth header in memory, then zeroes plaintext material.
|
||||
- provider-pool mixed dispatch: Edge service는 model group provider candidate를 선택한 뒤, 같은 selected provider/queue lease로 OpenAI-compatible provider에는 `ProviderTunnelRequest`, Ollama/native provider에는 normalized `RunRequest`를 보낸다. Edge-Node wire는 client-provided response path selector를 받지 않고, provider type만으로 후보를 제외하지 않는다.
|
||||
- cancel: Edge가 provider run id를 가진 `CancelRequest`를 보내 현재 provider 실행을 취소한다.
|
||||
|
|
@ -91,7 +89,7 @@ Edge는 Node 연결을 수락하고, Node는 연결 직후 등록 요청을 보
|
|||
- `RunEvent.metadata["openai_tool_calls"]`: OpenAI-compatible provider adapter가 native `tool_calls`를 반환했을 때 완료 이벤트에 싣는 JSON 배열이다. Edge OpenAI-compatible 표면은 이 값을 `message.tool_calls` 또는 stream `delta.tool_calls`로 복원한다. provider assistant content 텍스트를 이 값으로 파싱/합성하지 않는다.
|
||||
- `RunEvent.metadata["openai_text_tool_fallback"]`: OpenAI-compatible provider adapter가 backend native tool API 거부 후 `tools`/`tool_choice`를 제거하고 text tool-call instruction으로 재시도했을 때 `"true"`를 싣는다. 이 instruction은 backend가 system role 위치를 거부하지 않도록 leading system message에 병합한다. Edge는 이 표시가 있는 실행에서만 assistant content의 text tool-call을 OpenAI-compatible `tool_calls`로 복원할 수 있다.
|
||||
- `NodeCommandRequest.type`: 실행이 아닌 조회/제어성 명령이다. adapter execution 요청과 섞지 않는다.
|
||||
- `NodeCommandResponse.result` for CAPABILITIES uses `adapter_key`, `target`, `provider_status`, and `health_observation_seq` as the stable recovery-evidence keys. `adapter` and `instance_key` remain diagnostic capability identity; arbitrary provider metadata is not accepted as recovery evidence.
|
||||
- `NodeCommandResponse.result` for CAPABILITIES reports bounded adapter/provider capabilities for operator inspection. It is not response-silence recovery evidence and does not mutate provider admission state.
|
||||
- `NodeConfigPayload.adapters`: Edge가 Node에 내려주는 adapter instance 설정이다.
|
||||
- `NodeConfigPayload.workspaces`: the complete operator-approved workspace catalog for that Node. It includes the fixed root, closed operation list, fixed command templates, environment allowlist, and hard byte/time limits; it is not a public API or coordinator-facing projection.
|
||||
- Every workspace request `request_id`, including open, tool, artifact, cancel, and cleanup: immutable coordinator identity. The value is retained unchanged through the request-owned lifecycle and names `.iop/job/<request_id>`; Node-local execution ids must not replace or alias it.
|
||||
|
|
@ -143,16 +141,6 @@ Edge는 Node 연결을 수락하고, Node는 연결 직후 등록 요청을 보
|
|||
- Runtime composition installs the workspace handler before ready. Teardown stops the registry, runs the same bounded request cleanup for active requests, closes workspace resources before session and store resources, and applies the same order during reconnect replacement.
|
||||
- Do not open a lease before adapter capacity admission, cache plaintext across requests, accept a lease for another Node/target/revision/generation, or fall back to a different same-model credential slot after a bound route fails.
|
||||
|
||||
## 운영 증거 사영 경계
|
||||
|
||||
Node stall, Edge provider-health overlay, and Edge OpenAI recovery operational projections are local observations derived from the established terminal, health-overlay, and recovery decisions. They introduce no new Node↔Edge frame, field, ordering rule, or retry semantic. The wire protocol remains unchanged by these projections.
|
||||
|
||||
- Node observes finalized stall evidence before constructing and delivering the terminal; recovered observer failure cannot suppress terminal delivery.
|
||||
- Edge emits `iop_edge_provider_health_evidence_total`, `iop_edge_provider_health_transitions_total`, and `edge_provider_health_observation` locally after the overlay decision is finalized.
|
||||
- Edge emits `iop_edge_liveness_recovery_eligibility_total`, `iop_edge_liveness_recovery_results_total`, and `edge_liveness_recovery_observation` locally per request lifecycle.
|
||||
|
||||
Operational projections exclude raw payloads, credentials, caller-controlled identities, and unbounded identifiers from metric labels and general logs. Valid typed terminal metadata (e.g. `run_id`, `adapter`, `target` on the allowlisted stall metadata map) remains on the wire as already required by the typed terminal contract.
|
||||
|
||||
## 변경 시 확인할 코드/테스트
|
||||
|
||||
- `proto/iop/runtime.proto`
|
||||
|
|
|
|||
|
|
@ -7,19 +7,15 @@
|
|||
- status: active
|
||||
- source evidence:
|
||||
- `packages/go/execution/types.go`
|
||||
- `packages/go/execution/liveness.go`
|
||||
- `packages/go/execution/registry.go`
|
||||
- `packages/go/execution/emitter.go`
|
||||
- `packages/go/execution/failure.go`
|
||||
- `apps/node/internal/node/runtime_bridge.go`
|
||||
- `apps/node/internal/node/health_probe.go`
|
||||
- `apps/node/internal/node/command_handler.go`
|
||||
- `apps/node/internal/node/liveness_watchdog.go`
|
||||
- `apps/node/internal/transport/session.go`
|
||||
- `apps/edge/internal/service/model_queue_release.go`
|
||||
- `apps/edge/internal/service/node_command.go`
|
||||
- `apps/edge/internal/openai/stream_gate_runtime.go`
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
|
||||
|
||||
## Scope
|
||||
|
||||
|
|
@ -34,86 +30,17 @@ The execution package defines host-neutral provider primitives. It owns provider
|
|||
- Registry lookup uses provider identity and returns typed failures for missing or unavailable providers.
|
||||
- Callers must reject commands outside the closed provider-command allowlist before provider lookup.
|
||||
- Token usage remains observation data attached to execution or tunnel results.
|
||||
- `DefaultResponseStallTimeoutMS = 60000` is the documented default. `ResolveStallTimeoutMS(ms)` validates then maps zero to the default; safe positive values pass through, while negative or overflow values return an error.
|
||||
- `ClassifyRuntimeEvent` returns `start` for `EventTypeStart`, `progress` for non-empty `delta`/`message` or non-terminal usage, `terminal` for `complete`/`error`/`cancelled` (before usage check), and `none` for empty/unknown events.
|
||||
- `ClassifyProviderTunnelFrame` returns `progress` for `response_start` (with or without headers) and non-empty `body`, `terminal` for `end`/`error` (before payload check), `progress` for `usage`, and `none` for empty/unknown frames.
|
||||
- `ValidateStallTimeoutMS(ms)` rejects negative values and values exceeding `maxSafeStallTimeoutMS`; zero is allowed (use default).
|
||||
- `NodeProviderConf.EffectiveResponseStallTimeoutMS()` returns the effective timeout for a provider candidate.
|
||||
- `RunRequest.ResponseStallTimeoutMS` and `ProviderTunnelRequest.ResponseStallTimeoutMS` carry the selected provider's effective timeout; zero on the wire means the Node applies the documented default.
|
||||
- The Node wire boundary normalizes zero to `60000` and rejects negative or overflow values before router/provider invocation.
|
||||
- `response_stalled` is a stable typed failure. Node transport mappers (`runEventToProto` and `tunnelFrameToProto`) populate the optional wire `ExecutionFailure` message only for `FailureCodeResponseStalled`, attaching a defensive clone of allowlisted metadata keys (`failure_code`, `provider_health`, `liveness_classification`, `idle_duration_ms`, `run_id`, `attempt_id`, `attempt_fence`, `adapter`, `target`, and `health_observation_seq`); nil and non-stalled failures leave wire `ExecutionFailure` absent while preserving legacy error string fields (`RunEvent.Error` / `ProviderTunnelFrame.Error`). Caller metadata cannot override these values, and no raw payload, credential, or `recovery_eligible` signal is admitted.
|
||||
- The Node watchdog starts from attempt admission, resets only on the documented progress dispositions, stops on provider terminal, and emits one typed stall terminal. It does not retry providers or infer recovery eligibility. `Retryable=true` means only that the local provider ownership fence was confirmed within the bounded close grace.
|
||||
- After the watchdog claims a stall it joins two independent bounded outcomes without extending either serially — the fixed close-grace fence and the exact-target health probe — then assembles exactly one allowlisted terminal. The joined `liveness_classification`/`provider_health` pair is exactly `request_stalled`/`available`, `provider_unhealthy`/`unavailable`, or `health_unknown`/`unknown` (fail-closed default). Provider availability observed here is evidence only: it never resets progress, changes the fence, revives output, or authorizes retry, and late provider output stays fenced.
|
||||
- `health_observation_seq` is a connection-scoped monotonic sequence sourced from the transport Session. A new connection starts at zero, so the first finalized observation is one; normalized and tunnel observations on the same connection share the source and receive unique, increasing values under concurrency. Internal or unbound execution paths omit the key entirely and never encode a process-global generation.
|
||||
- `ProviderPoolDispatchRequest` carries two request-local recovery-hint fields: `AvoidProviderID` (non-empty to require a runtime-eligible alternate over the avoided provider) and `AllowAvoidedProviderFallback` (a compatibility field that liveness recovery leaves false). The queue applies identical avoidance filtering to both initial and queued re-resolution. Zero values preserve current selection behavior. This is selection policy only: it does not create a retry loop, reserve a slot, change provider priority, persist the hints, or count retries. A provider whose request produced `response_stalled` is never selected again for that request, even when its exact-target health probe reports `available`.
|
||||
- A Node `capabilities` command performs the same bounded exact-target `ProbeHealth` operation. Its stable result evidence is the requested adapter instance key (`adapter_key`), exact `target`, fail-closed normalized `provider_status`, and the next `health_observation_seq` from that same transport Session. Probe errors, unsupported probing, and adapter/instance/target mismatches report `unknown`; raw capability status is not recovery evidence.
|
||||
- Edge accepts a typed stall observation for provider-wide projection only after authoritative reception `(node_id, connection_generation)` matches the tracked immutable dispatch lease `(node_id, connection_generation, provider_id, adapter, target)`, the local attempt fence is confirmed, and the observation sequence is strictly newer. A current terminal still releases its lease exactly once when health evidence is absent, malformed, mismatched, or stale; a reception-owner mismatch changes neither overlay nor lease state.
|
||||
- Every validated current bound stall is annotated with Edge-owned `provider_id`, the validated `provider_health`, and `recovery_handoff=confirmed`, including an out-of-order terminal whose health projection is sequence-stale. Only a fresh `unavailable` observation lowers the generation-scoped runtime overlay. The token proves reception, lease binding, and local-fence handoff only; it is never `recovery_eligible` and never authorizes retry.
|
||||
- Every supported OpenAI Chat/Responses normalized or tunnel request enters one request-local StreamGate runtime, which is the sole liveness owner even when configured semantic filtering is disabled. That runtime may consume the confirmed handoff as a raw-free `response_stalled` provider error while its endpoint adapters preserve the disabled-semantic native status, headers, JSON/SSE/tunnel order, validation, usage, cancellation, and terminal behavior. It retains only the stable failure code, confirmed-handoff token, and `available|unavailable|unknown` health classification; Node/provider messages and arbitrary metadata are not copied. Exact replay additionally requires the existing uncommitted, uncancelled, side-effect-safe, snapshot-backed, shared-budget gate and is bounded to one liveness replay per request. A confirmed old terminal closes its Edge transport without another `CancelRun`; pool re-admission consumes the provider once as `AvoidProviderID` and never falls back to that stalled provider. A stall on the replacement attempt terminates without another replay.
|
||||
- The runtime overlay is keyed by `(node_id, connection_generation, provider_id)` and remains separate from configuration health. It excludes the provider from effective admission and projects it unavailable in status snapshots. Recovery requires a later CAPABILITIES result for the same current adapter/target mapping with strictly higher sequence and exact normalized `available`; malformed, ambiguous, stale-generation, unknown, and unavailable results are no-ops.
|
||||
|
||||
## Health probe contract
|
||||
|
||||
The execution package owns the stable, fail-closed probe outcome vocabulary consumed by Node terminal assembly. It is the typed three-way boundary between an inconclusive probe and a definitive provider-health classification; nothing else maps provider probe results to health.
|
||||
|
||||
- `ProviderHealth` is the stable normalized value: `request_stalled`, `provider_unhealthy`, or `health_unknown` (fail-closed default).
|
||||
- `LivenessClassification` is the stable observable category a probe outcome reduces through: `available`, `unavailable`, `timeout`, `error`, `unsupported`, `unknown`, and `identity_mismatch`.
|
||||
- `ProbeOutcome` is the typed, target-aware input; `ClassifyProbeOutcome` reduces it to a classification and `NormalizeProbeOutcome` maps it to health. The mapping is exactly: available → `request_stalled`; a validated matching unavailable result → `provider_unhealthy`; every error, timeout, unsupported adapter, unknown status, empty/mismatched adapter or target, and instance mismatch → `health_unknown`.
|
||||
- A returned error takes precedence over any reported status, so endpoint construction, request/network, non-success HTTP, and decode failures can never be confused with a positive exact-target-absent result.
|
||||
- The Node probe coordinator (`ProbeHealth`) roots its own five-second bounded context from the background, re-checks that deadline/cancellation after the probe returns, validates exact adapter and target identity (including a pinned instance key when set), and feeds only the typed normalizer. It never copies arbitrary provider metadata.
|
||||
- `ResolveProbeFunc` returns `nil` for an adapter that does not implement `ProviderProber`; a `nil` hook makes `ProbeHealth` fail closed to `health_unknown` without invoking any endpoint.
|
||||
|
||||
Probe completion is evidence only. The probe itself must never reset original request progress, change the attempt fence, authorize retry, sequence the watchdog terminal, directly mutate the Edge overlay, or infer recovery. Node owns the stall-terminal join and the connection-scoped `health_observation_seq`. Edge owns reception-generation and immutable-lease validation, the separate runtime overlay, candidate exclusion, snapshot projection, and exact later CAPABILITIES recovery. The ingress recovery host remains the sole owner of commit, cancellation, side-effect, budget, candidate, and replay eligibility decisions.
|
||||
- `FailureCode` carries provider-neutral typed failures. The removed response-silence failure and its activity, probe, health-overlay, and provider-avoidance types are not runtime concepts.
|
||||
- `RunRequest` and `ProviderTunnelRequest` do not carry a provider response-silence timeout. Silence produces no execution event, health mutation, or retry decision.
|
||||
- Provider `complete`, `error`, and `cancelled` events, caller cancellation, the request hard deadline, and transport disconnect remain the explicit terminal inputs.
|
||||
- A current provider or transport terminal releases its admission lease and request-owned resources exactly once. It does not start a health probe or select a replacement provider.
|
||||
- `ProviderPoolDispatchRequest` contains no request-local provider-avoidance hint. Candidate selection continues to use the configured provider catalog, current connection ownership, capacity, priority, and queue policy.
|
||||
- StreamGate consumes provider/transport terminals through the endpoint adapters. Output silence is not a filter input or recovery strategy.
|
||||
|
||||
## Prohibited ownership
|
||||
|
||||
The package must not own interactive shells, persistent processes, terminal emulation, working-directory mutation, resumable conversations, local quota probing, or arbitrary host command execution. It must not import application-internal packages or generated transport types.
|
||||
|
||||
## Operational evidence projections
|
||||
|
||||
The Node and Edge owners expose bounded operational projections derived exclusively from the established stall terminal, health-overlay, and recovery decisions documented above. These projections never widen the Node↔Edge wire protocol: they carry no new frame, field, ordering rule, or retry semantic, and they are emitted only after the authoritative decision is finalized.
|
||||
|
||||
### Node stall observations (owner: Node process-global)
|
||||
|
||||
- `iop_node_response_stalls_total` (counter): labels `execution_path`, `provider_health`, `liveness_classification`, `attempt_fence`. Every claimed stall increments exactly one series.
|
||||
- `iop_node_response_stall_duration_seconds` (histogram): same four labels. Samples the idle duration in seconds.
|
||||
- Dedicated structured log `node_response_stall_observation`: fields `execution_path`, `provider_health`, `liveness_classification`, `attempt_fence`, `idle_duration_ms`.
|
||||
- Label values are closed and low-cardinality: `execution_path` ∈ {`normalized`, `provider_tunnel`, `unknown`}; `provider_health` ∈ {`available`, `unavailable`, `unknown`}; `liveness_classification` ∈ {`request_stalled`, `provider_unhealthy`, `health_unknown`}; `attempt_fence` ∈ {`confirmed`, `unconfirmed`, `unknown`}.
|
||||
- Prohibited from metric labels and general logs: raw prompt/response, credential, caller metadata, `recovery_eligible`. High-cardinality inputs normalize to `unknown`.
|
||||
- Observer failure is fire-and-forget and never suppresses the terminal.
|
||||
- Source: `apps/node/internal/node/liveness_observability.go`; test: `apps/node/internal/node/liveness_observability_test.go::TestNodeLivenessObservability`.
|
||||
|
||||
### Edge provider-health overlay observations (owner: Edge service queue process-global)
|
||||
|
||||
- `iop_edge_provider_health_evidence_total` (counter): labels `source`, `evidence_health`, `decision`. Records authoritative overlay decisions.
|
||||
- `iop_edge_provider_health_transitions_total` (counter): labels `from_health`, `to_health`. Records overlay state transitions.
|
||||
- Dedicated structured log `edge_provider_health_observation`: fields `source`, `evidence_health`, `decision`, `from_health`, `to_health`, `state_changed`.
|
||||
- Label values are closed: `source` ∈ {`stall`, `probe`, `unknown`}; `evidence_health` ∈ {`available`, `unavailable`, `unknown`}; `decision` ∈ {`applied`, `rejected_stale`, `rejected_binding`, `rejected_ambiguous`, `inconclusive`}; `from_health`/`to_health` ∈ {`available`, `unavailable`, `unknown`}.
|
||||
- Prohibited from metric labels and general logs: provider, node, run, session, adapter, target, payload, or credential values.
|
||||
- Edge delivery is synchronous after decision/release/pump and after the queue lock is released; observer latency can delay handler return but cannot retain the lock or change the finalized transition.
|
||||
- Source: `apps/edge/internal/service/provider_health_observability.go`; test: `apps/edge/internal/service/provider_health_observability_test.go::TestProviderHealthObservability` and `TestProviderHealthObservabilityDoesNotExposeSentinels`.
|
||||
|
||||
### Edge OpenAI recovery observations (owner: Edge OpenAI server request-local wrapper with process-global collectors)
|
||||
|
||||
- `iop_edge_liveness_recovery_eligibility_total` (counter): labels `execution_path`, `provider_health`, `commit_state`, `eligibility`. Records eligibility decisions per liveness cycle.
|
||||
- `iop_edge_liveness_recovery_results_total` (counter): labels `execution_path`, `provider_health`, `recovery_result`. Records at most one final result per liveness cycle.
|
||||
- Dedicated structured log `edge_liveness_recovery_observation`: fields `phase`, `execution_path`, `provider_health`, `commit_state`, `eligibility`, `recovery_result`.
|
||||
- Label values are closed: `execution_path` ∈ {`normalized`, `provider_tunnel`, `unknown`}; `provider_health` ∈ {`available`, `unavailable`, `unknown`}; `commit_state` ∈ {`transport_uncommitted`, `stream_open`, `terminal_committed`, `unknown`}; `eligibility` ∈ {`eligible`, `no_owner`, `post_commit`, `unconfirmed_fence`, `caller_cancelled`, `tool_side_effect`, `budget_exhausted`, `no_candidate`, `same_provider_forbidden`, `other`}; `recovery_result` ∈ {`redispatched`, `plan_rejected`, `abort_failed`, `rebuild_failed`, `dispatch_failed`, `not_selected`, `terminal`, `other`}.
|
||||
- Prohibited from metric labels and general logs: correlation, attempt, run, session, model, provider, node, plan, shared_attempt_id, credential, or slot identifiers.
|
||||
- Each request owns one fresh wrapper; the collectors are process-global and registered once at package init.
|
||||
- `phase` is the bounded request-local cycle phase: `idle` before any eligible observation, `eligible_pending` after an `eligible` eligibility decision until the cycle resolves (redispatched, plan_rejected, abort_failed, rebuild_failed, dispatch_failed, not_selected, or terminal). Only these two values appear in the lifecycle; every other row carries one of them.
|
||||
- Empty `eligibility` and `recovery_result` rows belong to the lifecycle transitions that do not record a metric row: private filter rows that are not `filter_evaluated`, a second eligibility while `eligible_pending`, provider errors the liveness filter did not treat as a stall, and non-ExactReplay recovery observations that fall outside the private cycle. They are documented here so the safe-log field vocabulary is complete and not read as implying a missing classification.
|
||||
- Current immutable observations yield `provider_health=unknown` because the predecessor's private `filter_evaluated` observation does not carry provider health — health lives only in the request-local recovery state bridge, never in the immutable timeline. The closed classifier reserves `available` and `unavailable` for future health-bearing observations without claiming either is currently emitted.
|
||||
- Source: `apps/edge/internal/openai/liveness_recovery_observability.go`; test: `apps/edge/internal/openai/liveness_recovery_observability_test.go::TestOpenAILivenessObservationSink` and `TestOpenAILivenessRecoveryObservability`.
|
||||
|
||||
### Fresh health recovery in provider snapshots
|
||||
|
||||
A recovered provider appears in the existing Edge provider snapshot overlay as `status=available`, `health=available`, with effective capacity restored to configured values. The snapshot reflects the same `(node_id, connection_generation, provider_id)` key used by the runtime overlay. A newer connection generation does not inherit the old overlay.
|
||||
|
||||
### Leakage boundary
|
||||
|
||||
Operational projections exclude raw payloads, credentials, caller-controlled identities, and any unbounded identifier from metric labels and general structured logs. The exclusion applies to metric labels and general logs only; valid typed terminal metadata (e.g. `run_id`, `adapter`, `target` on the allowlisted stall metadata map) remains on the wire as already required by the typed terminal contract.
|
||||
|
||||
## Verification
|
||||
|
||||
- `go test -count=1 ./packages/go/execution`
|
||||
|
|
|
|||
|
|
@ -14,7 +14,6 @@
|
|||
- `apps/edge/internal/openai/usage_metrics.go`
|
||||
- `apps/edge/internal/openai/stream_gate_dispatcher.go`
|
||||
- `apps/edge/internal/openai/stream_gate_runtime.go`
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
|
||||
- `apps/edge/internal/openai/common_types.go`
|
||||
- `apps/edge/internal/openai/sse_writer.go`
|
||||
- `apps/edge/internal/openai/chat_types.go`
|
||||
|
|
@ -118,15 +117,13 @@ After provider-pool admission, Edge validates the exact route/slot/profile/model
|
|||
|
||||
Chat Completions와 Responses ingress에는 configured request snapshot 상한이 body 첫 read 전에 적용된다. body 또는 typed semantic view가 상한을 넘거나 rebuild peak 회계가 실패하면 provider admission 없이 HTTP `413`, `error.type="invalid_request_error"` 한 번으로 종료한다. 이 오류의 `message`는 내부 byte 수, snapshot reference, Core 오류 이름을 노출하지 않는다. 기존 public error body는 계속 `error.type`과 `error.message`만 가지며 size/trace/causes 같은 필드를 추가하지 않는다.
|
||||
|
||||
The bounded ingress/size error behavior is an active contract. Every supported Chat Completions, normalized Responses, and provider-tunnel request uses one request-local StreamGate runtime as the sole response and liveness owner. The runtime stages response status/headers and opening events until the first safe release, gathers all applicable filter results, and executes exactly one release, terminal, or bounded recovery outcome. `openai.stream_evidence_gate.enabled=false` preserves the existing endpoint-native compatibility behavior inside the runtime; it does not route the request to a legacy owner.
|
||||
The bounded ingress/size error behavior is an active contract. Every supported Chat Completions, normalized Responses, and provider-tunnel request uses one request-local StreamGate runtime for response staging, applicable filter arbitration, and its configured bounded recovery strategies. The runtime stages response status/headers and opening events until the first safe release, then executes exactly one release or terminal outcome. `openai.stream_evidence_gate.enabled=false` preserves endpoint-native compatibility inside the runtime; it does not route the request to a legacy owner.
|
||||
|
||||
복구 요청 조립 또는 dispatch가 실패하면 endpoint별 오류 하나만 보낸다. 내부 원인 사슬은 raw stack trace, provider endpoint/body, user prompt, output/reasoning 원문, tool args/result, 인증 정보를 포함하지 않으며 외부 JSON/SSE에 `causes`, `stack`, `trace` 같은 확장 필드로 노출하지 않는다.
|
||||
|
||||
The always-on runtime does not automatically enable a semantic detector. Only `repeat_guard`, `schema_gate`, and `provider_error` explicitly present in `openai.stream_evidence_gate.filters[]` enter the configured semantic portion of the request-start registry; `schema_gate` participates only when `metadata.scheme` is present. The private typed-stall registration remains present independently. Semantic filter selection depends on endpoint, environment, model group/model, actual provider, and execution path, never on a caller, SDK, or agent product name.
|
||||
The always-on runtime does not automatically enable a semantic detector. Only `repeat_guard`, `schema_gate`, and `provider_error` explicitly present in `openai.stream_evidence_gate.filters[]` enter the configured semantic portion of the request-start registry; `schema_gate` participates only when `metadata.scheme` is present. Semantic filter selection depends on endpoint, environment, model group/model, actual provider, and execution path, never on a caller, SDK, or agent product name.
|
||||
|
||||
For every supported Chat or Responses normalized or tunnel attempt, an Edge-confirmed typed `response_stalled` terminal is safe to recover only before any caller-visible commit and only when the request has no cancellation or tool/side-effect boundary, retains its request snapshot and recovery owner, and has remaining shared recovery budget. The replacement has a new attempt identity and prefers another provider; an exact `available` probe may permit the avoided provider only when no alternate remains. Every other typed or generic provider failure remains one sanitized terminal response and exposes no provider failure body or metadata.
|
||||
|
||||
The private liveness cycle emits operational evidence only: one `iop_edge_liveness_recovery_eligibility_total{execution_path,provider_health,commit_state,eligibility}` decision and at most one `iop_edge_liveness_recovery_results_total{execution_path,provider_health,recovery_result}` outcome. Each label is closed; the projection never labels or logs correlation, attempt, run, session, model, provider, node, plan, credential, raw payload, or terminal text. When the constructor-owned generic observation sink is active, its private liveness and selected ExactReplay lifecycle rows are replaced by `edge_liveness_recovery_observation` safe logs; explicitly installed sinks retain their original immutable observations.
|
||||
Silence is not an OpenAI-compatible terminal or recovery input. A caller cancellation, configured hard deadline, provider error/terminal, or transport disconnect yields the one applicable sanitized terminal outcome. Provider failures do not trigger watchdog-specific replay, provider avoidance, or health probing.
|
||||
|
||||
When a selected continuation plan addresses the request-local recovery source, the Rebuilder constructs a new request from retained assistant content/reasoning and the fixed English resume directive only. It never copies caller turns, Responses `input`, or caller `instructions`: Chat uses an assistant message followed by the fixed directive, while Responses uses assistant output/reasoning items plus that directive as `instructions`. The retained values are preserved byte-for-byte except for the selected content or reasoning byte cursor that excludes the repeated tail. If the caller omitted `temperature`, continuation attempts use `0.2`, `0.4`, and `0.6` in strategy-attempt order; an explicit caller temperature is preserved. A missing model context window, or a rebuilt prompt plus the fixed completion reserve above that window, fails closed before any replacement dispatch or recovery-budget consumption. This builder does not invoke a translator, local model, or `RecoveryPlanPreparer`.
|
||||
|
||||
|
|
|
|||
|
|
@ -156,12 +156,6 @@ source_evidence:
|
|||
- type: test
|
||||
path: apps/edge/internal/openai/usage_metrics_test.go
|
||||
notes: Canonical provider series, request-terminal deduplication, and provider-switch attribution
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_stall_recovery_test.go
|
||||
notes: Always-owned Chat/Responses normalized/tunnel S05 recovery and disabled-semantic compatibility matrix
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/liveness_recovery_observability_test.go
|
||||
notes: Chat/Responses normalized/tunnel liveness metric labels and default log-safety matrix
|
||||
- type: docs
|
||||
path: docs/openai-usage-grafana.md
|
||||
notes: Grafana query, daily/monthly rollup, usage origin, cloud-equivalent cost, avoided-cost ROI 조회 가이드
|
||||
|
|
@ -232,9 +226,8 @@ Edge가 OpenAI-compatible HTTP 요청을 받아 내부 `adapter + target` 실행
|
|||
| Anthropic provider-pool dispatch | Messages and count-tokens require a provider-pool model route. Native Messages requires `messages` capability and operation, while the Chat bridge requires `chat` capability and `chat_completions` operation; streaming and tools add their own capability checks. |
|
||||
| provider-normalized Messages bridge | Supported Messages compatibility headers are consumed at the bridge. Edge derives caller-neutral tool/effort/token-budget/stream requirements, selects a profile operation that preserves them, and maps effort to exact or nearest lower provider grade. Chat-compatible providers may therefore use Chat or Responses without caller-name branches. Mid-conversation `system` messages retain their ordered privileged role as Chat `role=system` or Responses system input messages; they are not downgraded to user text. Placement, text-only content, and pending-tool fences fail closed. JSON schema and tool shapes are converted for the selected wire; Gemini tool thought signatures still round-trip through opaque tool-use ids. |
|
||||
| Gemini-native agy tool continuation | Official agy 1.1.12의 Gemini-native 요청을 Chat 실행 경로로 변환한다. tool 실행 뒤 독립 `role:model` content로 전달되는 `functionResponse`는 앞선 function call과 매칭해 Chat `tool` message로 변환하며, assistant content와 response가 한 model content에 섞인 모호한 요청은 거부한다. |
|
||||
| bounded ingress and StreamGate ownership | Chat/Responses bodies are limited to 16 MiB before the first read. Every supported path delegates response-start staging, applicable filter arbitration, bounded liveness recovery, and the single terminal to `runtime/stream-evidence-gate`; `enabled` controls configured semantic policy only. |
|
||||
| typed stall terminal | Supported Chat/Responses normalized and tunnel attempts always translate only Edge-confirmed `response_stalled` terminals into a raw-free liveness recovery candidate; post-commit, cancelled, tool-bearing, missing-snapshot, exhausted, unsupported, unconfirmed, generic, and no-owner paths stay terminal. |
|
||||
| liveness operational evidence | Each private liveness cycle emits one closed eligibility counter and at most one closed final-result counter. Constructor-owned generic logs use a safe projection without identifiers or payloads, while application-installed observation sinks retain the original immutable events. |
|
||||
| bounded ingress and StreamGate | Chat/Responses bodies are limited to 16 MiB before the first read. Every supported path delegates response-start staging, applicable filter arbitration, configured bounded semantic recovery, and the single terminal to `runtime/stream-evidence-gate`; `enabled` controls configured semantic policy only. |
|
||||
| terminal semantics | Silence does not create a protocol terminal or recovery candidate. Caller cancel, configured hard deadline, provider terminal/error, and transport disconnect remain the terminal inputs. |
|
||||
| repeat-resume request shape | A selected continuation uses only request-local assistant content/reasoning plus a fixed English directive. Chat emits assistant provenance followed by the directive; Responses emits assistant output/reasoning items and places the directive in `instructions`. Caller messages, `input`, and original `instructions` are excluded. |
|
||||
| repeat history boundary | Chat and Responses use separate endpoint decoders to create a bounded raw-free role/channel/action snapshot from the current request only. User occurrences exclude assistant anchors; missing reasoning does not infer lineage or TTL state. |
|
||||
| model-driven response path | request `model`이 가리키는 provider capability가 provider raw tunnel 또는 normalized RunEvent path를 결정한다. caller metadata는 route나 response shape를 선택하지 않는다. OpenAI와 Anthropic ingress는 같은 model catalog와 provider-pool dispatch를 공유한다. |
|
||||
|
|
@ -294,9 +287,7 @@ sequenceDiagram
|
|||
- `configs/edge.yaml`의 `openai` 섹션이 listener, bearer token, legacy adapter/target, model routes, strict output을 제공한다.
|
||||
- `credential_plane.enabled` is the startup-only managed/legacy switch. Managed mode requires TLS on OpenAI ingress, CP-Edge, and Edge-Node hops; config validation rejects legacy principal/provider-auth and static provider credential sources.
|
||||
- Managed authentication and model resolution use one immutable projection view per request. Trusted principal/route/slot/revision metadata overwrites caller spoofing and remains bound across recovery admission.
|
||||
- `openai.stream_evidence_gate.enabled` defaults to false and activates configured semantic policy only. Supported OpenAI response/liveness ownership remains in the request runtime in both states; the same config also supplies the 0..3 recovery cap and up-to-16-MiB ingress snapshot bound. Changes remain restart-required.
|
||||
- A typed stall recovery re-enters provider-pool admission with the failed provider avoided. No health classification allows the stalled provider to be selected again for the same request; without an alternate, recovery terminates once.
|
||||
- `iop_edge_liveness_recovery_eligibility_total` labels are `execution_path`, `provider_health`, `commit_state`, and `eligibility`; `iop_edge_liveness_recovery_results_total` labels are `execution_path`, `provider_health`, and `recovery_result`. All are closed vocabularies and exclude request/attempt/provider/model identifiers and content.
|
||||
- `openai.stream_evidence_gate.enabled` defaults to false and activates configured semantic policy only. The same config supplies the 0..3 recovery cap and up-to-16-MiB ingress snapshot bound. Changes remain restart-required.
|
||||
- When `repeat_guard` is configured, Chat accepts plain `content`, `reasoning_content`, `reasoning`, and `reasoning_text` provenance for fingerprinting; Responses accepts its own text/reasoning/function-call item provenance. Signed, encrypted, and unknown values are canonical-only and never sanitation or observation payloads.
|
||||
- Completed action/result fingerprints provide the only request-history progress boundary. An identical consecutive action/result is no-progress; a changed completed result is progress, while a different action alone is insufficient. No caller product, session metadata, inferred TTL, or cross-request cache participates.
|
||||
- top-level `models[]`가 있으면 OpenAI model list와 provider-pool dispatch에서 legacy route보다 우선한다.
|
||||
|
|
@ -414,7 +405,7 @@ sequenceDiagram
|
|||
- 2026-08-14: Canonicalized private Chat Completions stage responses before strict Plan/Work/Review decoding. Standard OpenAI bookkeeping fields are stripped centrally, a non-null refusal remains fail-closed, and Responses/Chat now converge on the same private envelope instead of making stage codecs provider-specific.
|
||||
- 2026-08-14: Added provider `timings` to the bounded Chat response bookkeeping stripped before private stage decoding, aligning the common normalizer with the existing Ornith Work codec without admitting unknown response fields.
|
||||
- 2026-08-13: Gemini-native path parsing now keeps route IDs strict while accepting the bounded URL-encoded official caller model label `Gemini 3.6 Flash`.
|
||||
- 2026-08-06: Synchronized always-owned Chat/Responses typed-stall recovery, provider avoidance/fallback admission, and closed-label liveness operational evidence with the current runtime, contracts, and deterministic recovery tests.
|
||||
- 2026-08-15: Removed watchdog-specific OpenAI recovery and provider avoidance; existing terminal owners now determine request completion.
|
||||
- 2026-08-06: Added marked single-request Messages admission through the separate service coordinator capability, one unlabeled runtime ingress counter, buffered sanitized terminal acknowledgement, and deterministic real-POST compatibility evidence.
|
||||
- 2026-08-06: Added the marked streaming subset with fixed plan/work/review/repair progress, liveness ping, serialized monotonic text blocks, private-wire exclusion, one success/error terminal, joined ticker shutdown, and post-`message_stop` completion acknowledgement.
|
||||
- 2026-08-07: Added the private marked-request workspace tool continuation, strict closed schemas, ordered exact-generation Node round trips, immutable correlation/budgets/cancellation, and real one-POST multi-tool privacy evidence.
|
||||
|
|
|
|||
|
|
@ -12,12 +12,9 @@ source_evidence:
|
|||
- type: code
|
||||
path: packages/go/execution/types.go
|
||||
notes: Provider execution and event types
|
||||
- type: code
|
||||
path: packages/go/execution/liveness.go
|
||||
notes: Response-stall timeout default, validation, and RuntimeEvent/ProviderTunnelFrame activity classifiers
|
||||
- type: code
|
||||
path: apps/node/internal/node/runtime_bridge.go
|
||||
notes: Protobuf-to-execution translation with raw stall timeout validation before router/provider invocation
|
||||
notes: Protobuf-to-execution translation and ordered runtime event projection
|
||||
- type: code
|
||||
path: apps/edge/internal/transport/server.go
|
||||
notes: Edge-side tunnel-tolerant heartbeat and disconnect supervision
|
||||
|
|
@ -26,46 +23,31 @@ source_evidence:
|
|||
notes: Node-side tunnel-tolerant heartbeat and reconnect transport
|
||||
- type: code
|
||||
path: apps/edge/internal/service/provider_tunnel.go
|
||||
notes: Provider selection, credential binding validation, reception-aware terminal handoff, lease acquisition, and pre-send fencing
|
||||
notes: Provider selection, credential binding validation, lease acquisition, terminal handling, and pre-send fencing
|
||||
- type: code
|
||||
path: apps/edge/internal/service/model_queue_release.go
|
||||
notes: Immutable lease validation, generation/sequence-fenced runtime health overlay, recovery handoff annotation, and exactly-once release
|
||||
notes: Immutable lease validation and exactly-once release for provider and transport terminals
|
||||
- type: code
|
||||
path: apps/edge/internal/service/node_command.go
|
||||
notes: CAPABILITIES dispatch identity retention and exact available recovery evidence application
|
||||
notes: Closed Node command dispatch and bounded result handling
|
||||
- type: code
|
||||
path: apps/node/internal/node/tunnel_handler.go
|
||||
notes: Provider tunnel handling and recipient-sealed credential lease consumption
|
||||
- type: code
|
||||
path: apps/node/internal/node/liveness_watchdog.go
|
||||
notes: Shared normalized/tunnel stall coordination, close-grace ownership, serialized emission fencing, bounded probe/fence join, and connection-scoped observation sequencing
|
||||
- type: code
|
||||
path: apps/node/internal/node/health_probe.go
|
||||
notes: Bounded independent exact-target health probe coordinator consumed by the stall terminal join
|
||||
- type: code
|
||||
path: apps/node/internal/transport/session.go
|
||||
notes: Connection-scoped monotonic health-observation sequence source
|
||||
notes: Connection-scoped request handlers and cancellation ownership
|
||||
- type: code
|
||||
path: packages/go/credentiallease/envelope.go
|
||||
notes: Signed scope validation, recipient sealing, expiry, replay, and exact binding verification
|
||||
- type: test
|
||||
path: apps/node/internal/node/command_test.go
|
||||
notes: Closed provider commands plus fail-closed exact CAPABILITIES health and Session sequence regressions
|
||||
- type: test
|
||||
path: apps/edge/internal/service/provider_health_overlay_test.go
|
||||
notes: S04 binding, stale evidence, normalized/tunnel release races, overlay projection, and CAPABILITIES recovery evidence
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_stall_recovery_test.go
|
||||
notes: S05 always-owned OpenAI recovery, new attempt/provider selection, shared budget, old-transport close, and guard terminals
|
||||
notes: Closed provider command and bounded result regressions
|
||||
- type: test
|
||||
path: apps/edge/internal/transport/heartbeat_test.go
|
||||
notes: Edge heartbeat liveness profile regression
|
||||
- type: test
|
||||
path: apps/node/internal/transport/heartbeat_test.go
|
||||
notes: Node heartbeat liveness and idle-connection regressions
|
||||
- type: test
|
||||
path: apps/node/internal/node/liveness_watchdog_test.go
|
||||
notes: Manual-clock S01/S02 threshold, progress, terminal, close-grace, ownership, metadata, and late-output evidence
|
||||
- type: test
|
||||
path: apps/node/internal/node/provider_tunnel_test.go
|
||||
notes: Credential preflight admission release regression
|
||||
|
|
@ -183,24 +165,6 @@ source_evidence:
|
|||
- type: test
|
||||
path: apps/node/internal/bootstrap/workspace_runtime_test.go
|
||||
notes: Path-free startup failure, handler-before-ready composition, and registry/workspace/session/store close-order regressions
|
||||
- type: code
|
||||
path: apps/node/internal/node/liveness_observability.go
|
||||
notes: Node stall counter/histogram and dedicated structured log with closed label values and raw-payload exclusion
|
||||
- type: test
|
||||
path: apps/node/internal/node/liveness_observability_test.go
|
||||
notes: Deterministic S06 Node stall observation regression with closed label values
|
||||
- type: code
|
||||
path: apps/edge/internal/service/provider_health_observability.go
|
||||
notes: Edge overlay evidence/transition counters and dedicated structured log with closed label values and identity exclusion
|
||||
- type: test
|
||||
path: apps/edge/internal/service/provider_health_observability_test.go
|
||||
notes: Deterministic S06 Edge overlay observation regression including sentinel exclusion via TestProviderHealthObservabilityDoesNotExposeSentinels
|
||||
- type: code
|
||||
path: apps/edge/internal/openai/liveness_recovery_observability.go
|
||||
notes: Edge OpenAI eligibility/results counters and dedicated structured log with closed label values and identifier exclusion
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/liveness_recovery_observability_test.go
|
||||
notes: Deterministic S06 OpenAI recovery observation regression with closed label values
|
||||
- type: complete-log
|
||||
path: agent-task/archive/2026/08/m-iop-owned-single-request-agent-execution/25+23,24_claude_smoke_qualification/complete.log
|
||||
notes: S12 external Claude qualification with one ingress, Gemini to ornith-fast to Gemini stage order, exact workspace result, one end_turn, timing, cleanup, and redacted evidence
|
||||
|
|
@ -234,14 +198,9 @@ The shared `packages/go/execution` package contains provider lifecycle, registry
|
|||
| Work and Review handoff | Work parses the stored PLAN with deterministic `P1..Pn` IDs, projects only admitted workspace tools, and writes exactly one validated REVIEW handoff containing item status, changes, verification, and deviations. Review rereads both artifacts before provider dispatch, has no memory work payload, performs any repair/re-verification in the request-local ledger, and writes no final REVIEW page. |
|
||||
| request-owned cleanup | Node creates and inventories only `.iop/job/<request_id>` internal state, cancels and waits for all active command groups, validates the exact tree without following entries, and removes matching artifacts deepest-first with non-recursive descriptor operations. Symlinks, special files, foreign devices, identity replacements, and unowned entries fail closed. User results and sibling request state are preserved. Concurrent cleanup callers receive one bounded cached typed result. |
|
||||
| provider raw tunnel | 선택된 provider의 HTTP/SSE를 `ProviderTunnelRequest`/`ProviderTunnelFrame`으로 relay하며 순서와 단일 terminal outcome을 보장한다. |
|
||||
| response-stall activity contract | 선택된 provider의 response-stall timeout을 normalized/tunnel request에 보존한다. Node는 wire zero를 `60000ms`로 해석하고 invalid raw value를 adapter 호출 전에 거부한다. Runtime event의 terminal type은 payload/usage보다 우선하며 non-terminal usage는 progress다. |
|
||||
| Node stall watchdog | Node가 normalized run과 raw tunnel에 하나의 activity watchdog을 적용한다. progress만 timer를 reset하며, stall은 `response_stalled` terminal 하나와 Node-owned safe metadata를 만들어 normalized `RunEvent`와 raw `ProviderTunnelFrame` wire의 optional typed `ExecutionFailure` 필드에 싣는다. stall claim 뒤에는 bounded close grace fence와 독립 exact-target health probe를 직렬 확장 없이 join한다. close grace 안에 provider return이 확인된 경우만 `Retryable` capability hint를 준다. |
|
||||
| Node health evidence join | stall terminal에 three-way health evidence를 싣는다: `provider_health` status와 `liveness_classification` normalization이 `available`/`request_stalled`, `unavailable`/`provider_unhealthy`, `unknown`/`health_unknown` 쌍으로 fail-closed된다. probe 성공은 progress reset·fence 변경·retry authority가 아니며 late output은 fenced 상태를 유지한다. |
|
||||
| health observation sequence | transport Session이 connection-scoped monotonic `health_observation_seq`를 소유한다. 새 connection은 0에서 시작해 첫 finalized observation이 1이며, 같은 connection의 normalized/tunnel observation이 source를 공유해 동시에도 유일 증가값을 받는다. internal/unbound 경로는 key를 생략한다. |
|
||||
| Edge terminal health handoff | Edge validates authoritative reception node/generation plus the immutable provider/adapter/target lease before applying typed stall evidence. Every validated current bound stall receives `provider_id`, validated health, and `recovery_handoff=confirmed`, while only fresh unavailable evidence lowers a separate runtime overlay; the token never grants replay eligibility. Every valid current terminal still releases its lease exactly once. |
|
||||
| CAPABILITIES recovery | Node runs the same bounded exact-target `ProbeHealth` and returns stable adapter/target/status plus the next Session sequence. Edge recovers exactly one matching current-generation unavailable provider only from a strictly newer `available` result; malformed, ambiguous, stale, unknown, and unavailable responses are no-ops. |
|
||||
| recovery candidate preference | `ProviderPoolDispatchRequest` carries `AvoidProviderID` and `AllowAvoidedProviderFallback`. Every admission (initial and queued re-resolution) requires a runtime-eligible alternate over the avoided provider for liveness recovery. `available` probe evidence describes endpoint health but never permits re-selecting the stalled provider for the same request. Liveness replay is request-locally capped at one, so a replacement-attempt stall terminates. Zero values preserve current selection. This is selection policy only: no retry loop, slot reservation, priority change, persistence, or retry counter. |
|
||||
| OpenAI typed-stall consumption | Every supported Chat/Responses normalized or tunnel request has one unconditional runtime liveness owner, independent of configured semantic activation. It converts only the Edge-confirmed typed stall handoff into a raw-free StreamGate event, owns pre-commit eligibility, and closes the already fenced old transport before re-admission; Node does not grant replay authority. |
|
||||
| terminal ownership | Node normalized run과 raw tunnel은 caller cancel, configured hard deadline, provider terminal/error, transport disconnect만으로 끝난다. 무출력은 상태 전이나 retry input이 아니다. |
|
||||
| provider admission | Provider-pool admission selects candidates by configured capacity, priority, health, and route capability. A provider terminal releases its lease exactly once; it does not create an automatic replay or alternate-provider selection. |
|
||||
| OpenAI terminal consumption | Supported Chat/Responses normalized and tunnel paths project provider and transport terminal outcomes through their existing endpoint adapters. No watchdog-specific handoff reaches StreamGate. |
|
||||
| tunnel-tolerant liveness | Edge와 Node는 30초 heartbeat interval과 45초 response wait를 공통으로 사용해 긴 prompt prefill이나 streaming backpressure 중의 정상 connection을 조기에 끊지 않는다. |
|
||||
| reconnect/generation fencing | 현재 connection이 종료되면 해당 generation만 fence하고 Node supervisor가 reconnect한다. Heartbeat wait를 넘긴 경우의 close reason은 `heartbeat_timeout`이다. |
|
||||
| cancellation/command | `run_id`로 현재 run만 취소하며 command는 capabilities, transport status, Ollama API tunnel로 제한한다. |
|
||||
|
|
@ -265,8 +224,6 @@ The shared `packages/go/execution` package contains provider lifecycle, registry
|
|||
|
||||
IOP no longer provides persistent shell sessions, terminal emulation, process resume, local working-directory execution context, arbitrary host commands, or local quota/status probing.
|
||||
|
||||
The current spec maps reviewed Node and Edge observability producers to S06 behavior and deterministic tests. Node exposes bounded stall counters/histograms and dedicated structured logs with closed label values and raw-payload exclusion. Edge service queue exposes bounded overlay evidence/transition counters and dedicated structured logs with closed label values and identity exclusion. Edge OpenAI server exposes bounded eligibility/results counters and dedicated structured logs with closed label values and identifier exclusion. All projections are local observations and do not widen the wire protocol.
|
||||
|
||||
Single-request lifecycle observation evidence links ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, and raw-free correlation for one real POST. `iop_anthropic_single_request_ingress_total` is strictly unlabeled (no request_id, stage_id, provider identity, content, or workspace reference). Internal tool names, raw arguments, and private results are absent from public output and log projections. Stage-pure timing, cardinality-bounded labels, and privacy semantics are documented here. SDD S12 external Claude qualification on an approved IOP Node confirms the same boundary with ingress `0→1`, Gemini → ornith-fast → Gemini, one `end_turn`, the exact expected workspace result, timing, cleanup, and redacted evidence.
|
||||
|
||||
## 주요 흐름
|
||||
|
|
@ -315,14 +272,14 @@ sequenceDiagram
|
|||
- Edge-Node wire: `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- provider execution primitives: `agent-contract/inner/execution-runtime.md`
|
||||
|
||||
Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 liveness profile이다. `response_stall_timeout_ms`만 provider execution request wire에 추가되며 provider response shape은 바뀌지 않는다.
|
||||
Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 connection profile이다. Provider execution request wire에는 watchdog timeout field가 없다.
|
||||
|
||||
## 설정/데이터/이벤트
|
||||
|
||||
- Edge와 Node의 현재 heartbeat interval은 30초, response wait는 45초다.
|
||||
- 이 값은 runtime YAML model config나 `max_tokens`/context 설정이 아니라 transport 구현 상수다.
|
||||
- 45초 동안 heartbeat response가 없으면 current connection을 `heartbeat_timeout`으로 닫고 provider resource를 offline 처리한 뒤 reconnect/queue 재평가를 수행한다.
|
||||
- response-stall timeout은 provider config가 source이며 winning candidate가 re-resolution된 뒤의 request까지 같은 effective value를 보존한다. request hard timeout, queue timeout, transport heartbeat, client response-idle timeout과 timer lifecycle은 별도 소유권이다.
|
||||
- Provider catalog와 Edge-Node request wire에는 output-silence timeout이 없다. request hard timeout, queue timeout, provider HTTP timeout, transport heartbeat, client response-idle timeout은 각각 기존 owner가 처리한다.
|
||||
|
||||
## 검증
|
||||
|
||||
|
|
@ -341,9 +298,6 @@ Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 l
|
|||
- `go test -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport`
|
||||
- `go test -race -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport`
|
||||
- 실제 provider tunnel 검증은 5초를 넘는 긴 prefill과 streaming 응답 동안 Node가 connected/healthy를 유지하고, 응답이 정상 terminal을 반환하며, `heartbeat_timeout`이 발생하지 않는지 확인한다.
|
||||
- `go test -count=1 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — deterministic Node stall observation with closed label values and raw-payload exclusion.
|
||||
- `go test -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — deterministic Edge overlay evidence/transition with closed label values and identity exclusion; `TestProviderHealthObservabilityDoesNotExposeSentinels` covers the sentinel/prohibited-value guard.
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAILivenessObservationSink|TestOpenAILivenessRecoveryObservability)$'` — deterministic OpenAI recovery eligibility/results with closed label values and identifier exclusion.
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run 'TestAnthropicSingleRequestObservation'` — deterministic single-request observation evidence: ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation, and unlabeled metric assertion.
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run 'TestSingleRequest(ProviderStage|PlanStage)'` — deterministic frozen provider codec and Plan stage evidence, including required read/list-first inspection, mutation-tool exclusion, successful-result completion gate, high reasoning, the closed structured response schema, exact JSON field admission, ordered tunnel frames, deterministic frozen-template rendering, planning/internal-tool resume envelopes, and `plan.md` artifact selection.
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run 'TestSingleRequest(ProviderStage|ProviderNormalization|PlanStage|WorkStage|ReviewStage|Executor)'` — private stage operation selection, Chat/Responses conversion, nearest-lower effort (`max` → `xhigh`), default-selector pool-provider acceptance, explicit dispatch fences, and composite regression evidence.
|
||||
|
|
@ -354,13 +308,10 @@ Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 l
|
|||
|
||||
## 한계와 주의사항
|
||||
|
||||
- 30/45초 liveness profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다.
|
||||
- 30/45초 transport heartbeat profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다.
|
||||
- 45초를 넘겨 실제 heartbeat response가 없는 connection은 기존과 같이 오프라인 처리하고 reconnect한다.
|
||||
- Node owns local detection, cancellation, emission fencing, confirmed/unconfirmed ownership close, exact-target probe joining, connection-scoped observation sequencing, and the bounded `iop_node_response_stalls_total` / `iop_node_response_stall_duration_seconds` / `node_response_stall_observation` projections with closed label values.
|
||||
- Edge owns reception-generation and immutable-lease validation, the generation-scoped runtime health overlay, `iop_edge_provider_health_evidence_total` / `iop_edge_provider_health_transitions_total` / `edge_provider_health_observation` projections with closed label values, effective admission/snapshot projection, and exact later CAPABILITIES recovery.
|
||||
- The always-owned supported OpenAI ingress runtime owns commit, cancellation, side-effect, snapshot, shared-budget, candidate, and replay decisions, and exposes `iop_edge_liveness_recovery_eligibility_total` / `iop_edge_liveness_recovery_results_total` / `edge_liveness_recovery_observation` projections with closed label values.
|
||||
- Node retry and `recovery_eligible` remain prohibited. Hard deadline and connection disconnect continue to take precedence over a simultaneous stall timer.
|
||||
- Operational projections never widen the wire protocol; they carry no new frame, field, ordering rule, or retry semantic.
|
||||
- Edge owns reception-generation and immutable-lease validation, provider-pool admission, and exactly-once release on terminal events.
|
||||
- Output silence alone does not cancel a Node attempt, alter provider health, or select another provider. Explicit provider/transport terminal, caller cancellation, and the request hard deadline remain the terminal inputs.
|
||||
- Workspace admission and the private wire both fence the exact ready connection generation. The wire never exposes workspace fields through provider `RunRequest`, `NodeCommand`, or public API output. The executor exposes no caller access to `.iop`; only request-owned internal runtime code can derive and inventory `.iop/job/<request_id>`. Structured write input is required for WRITE, while legacy content-only input remains rejected. COMMAND is non-interactive and has no shell, PTY, arbitrary argv, ambient environment, path-based cwd lookup, or persistent process ownership. Cleanup never rolls back or deletes user-requested workspace results.
|
||||
- The composite single-request executor is installed at Edge input startup (`apps/edge/internal/input/manager.go`), wiring the active Plan -> Work -> Review stage pipeline for single-request execution. Private stage outcomes use the implemented closed S11 terminal policy and stop without retry/fallback or a second request. Deterministic local activation and terminal evidence are proven, and S12 (`claude-smoke`) qualifies the external Claude path on an approved IOP Node.
|
||||
- Single-request observation evidence (ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation) is documented and tested. `iop_anthropic_single_request_ingress_total` is strictly unlabeled. Internal tool names, raw arguments, and private results are absent from public output and log projections. SDD S12 confirms the external path with one ingress, the expected stage sequence, one terminal, exact output, timing, cleanup, and redacted evidence.
|
||||
|
|
@ -379,13 +330,7 @@ Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 l
|
|||
- 2026-08-07: Implemented the S11 `error-cancel` boundary: one frozen service terminal disposition, request-local typed stage classification, fixed-hash repetition/no-progress detection, shared buffered/SSE Anthropic mapping, silent disconnect cancellation, private-partial suppression for `max_tokens`, and deterministic one-ingress/one-terminal/no-second-request evidence. The Edge-Node protobuf wire is unchanged and S12 remains pending.
|
||||
- 2026-08-07: Installed the composite single-request executor at Edge input startup (`apps/edge/internal/input/manager.go`), activating the Plan -> Work -> Review stage pipeline. Production construction evidence is test-covered (`apps/edge/internal/input/manager_test.go`), while actual Claude/Mac external qualification remains explicitly deferred to S12 (`claude-smoke`).
|
||||
- 2026-08-02: provider tunnel의 긴 prompt prefill과 streaming backpressure를 정상 traffic으로 허용하도록 Edge/Node heartbeat profile을 30초 interval/45초 wait로 복원한 현재 구현과 회귀 검증을 반영했다 (`apps/edge/internal/transport/server.go`, `apps/node/internal/transport/client.go`).
|
||||
- 2026-08-04: provider response-stall timeout의 config validation, selected-candidate propagation, Node adapter-visible retention, and activity classification contract를 반영했다.
|
||||
- 2026-08-04: Added the shared Node run/tunnel watchdog coordinator, serialized tunnel emission fence, pre-provider admission cleanup, disconnect-bound handler lifetime, and deterministic S01/S02 manual-clock evidence.
|
||||
- 2026-08-04: Joined the bounded close-grace fence and the independent exact-target health probe into one stall terminal carrying three-way health evidence, and added the connection-scoped `health_observation_seq` sourced from the transport Session.
|
||||
- 2026-08-05: Added authoritative Edge terminal handoff, immutable lease binding, generation/sequence-fenced runtime provider health, exactly-once normalized/tunnel release, and fail-closed Session-sequenced CAPABILITIES recovery without config-health mutation or replay authorization.
|
||||
- 2026-08-05: Added runtime-local OpenAI consumption of confirmed typed stalls, including cancel-free old-transport close and provider-pool avoidance hints for ExactReplay.
|
||||
- 2026-08-05: Made supported OpenAI Chat/Responses normalized and tunnel liveness ownership unconditional and added S05 recovery/guard evidence independent of semantic policy activation.
|
||||
- 2026-08-06: Mapped reviewed Node, Edge overlay, and OpenAI recovery observability producers to S06 behavior with deterministic test evidence. Node exposes `iop_node_response_stalls_total`, `iop_node_response_stall_duration_seconds`, and `node_response_stall_observation` (source: `apps/node/internal/node/liveness_observability.go`; test: `TestNodeLivenessObservability`). Edge service queue exposes `iop_edge_provider_health_evidence_total`, `iop_edge_provider_health_transitions_total`, and `edge_provider_health_observation` (source: `apps/edge/internal/service/provider_health_observability.go`; test: `TestProviderHealthObservability`, `TestProviderHealthObservabilityDoesNotExposeSentinels`). Edge OpenAI server exposes `iop_edge_liveness_recovery_eligibility_total`, `iop_edge_liveness_recovery_results_total`, and `edge_liveness_recovery_observation` (source: `apps/edge/internal/openai/liveness_recovery_observability.go`; test: `TestOpenAILivenessObservationSink`, `TestOpenAILivenessRecoveryObservability`). All projections carry only closed, low-cardinality label values and exclude raw payloads, credentials, and unbounded identifiers from metric labels and general logs. The wire protocol is unchanged.
|
||||
- 2026-08-15: Removed the provider output-silence timeout, Node cancellation/probe path, Edge health/replay handoff, and their observations. Existing explicit terminal owners and transport heartbeat remain separate.
|
||||
- 2026-08-06: Added implemented Edge workspace admission for single-request execution: an opaque `workspace_ref` binds to one configured ready Node generation and a closed capability projection before executor startup. Admission rejects unavailable, foreign, pending, malformed, and stale candidates without fallback or reselection; Node-private wire, executor, filesystem path, and symlink enforcement remain deferred.
|
||||
- 2026-08-06: Added the dedicated Edge-Node workspace wire. `NodeConfigPayload` now delivers the approved catalog; `WorkspaceOpen`/`Tool`/`Cancel`/`Cleanup` messages have closed typed outcomes, immutable coordinator identities, parser registration, and an optional Node handler. Edge dispatch is generation-fenced and context cancellation sends one typed cancel. Node filesystem and process execution are intentionally deferred.
|
||||
- 2026-08-06: Completed the reviewed workspace file boundary repair. Edge now sends only frozen request authority, Node admits immutable catalog subsets/lower limits, and structured write reaches the file executor while legacy incomplete input remains rejected. The Go 1.24-compatible descriptor-relative no-follow write path validates before effects, bounded list processing retains fixed state, startup errors are path-free, and composition proves handler-before-ready plus workspace-before-session/store teardown. Command execution/cancellation and cleanup remain deferred.
|
||||
|
|
|
|||
|
|
@ -8,10 +8,7 @@ source_evidence:
|
|||
notes: Edge config, provider pool, config refresh, Node payload 연결 계약
|
||||
- type: code
|
||||
path: packages/go/config/provider_types.go
|
||||
notes: provider/model catalog 설정 타입, response_stall_timeout_ms validation과 effective helper
|
||||
- type: code
|
||||
path: packages/go/execution/liveness.go
|
||||
notes: Stall timeout default, validation, and effective helper used by config
|
||||
notes: provider/model catalog 설정 타입과 validation
|
||||
- type: code
|
||||
path: packages/go/config/edge_types.go
|
||||
notes: Edge root provider_pool canonical queue policy 타입과 기본값
|
||||
|
|
@ -38,13 +35,10 @@ source_evidence:
|
|||
notes: provider 전역 lease, 공통 pending 상한, global enqueue 순서와 long-context admission
|
||||
- type: code
|
||||
path: apps/edge/internal/service/model_queue_release.go
|
||||
notes: Lease release, disconnect/reconnect candidate rebuild, global queue pump, and generation/sequence-fenced runtime health transitions
|
||||
notes: Lease release, disconnect/reconnect candidate rebuild, and global queue pump
|
||||
- type: code
|
||||
path: apps/edge/internal/service/model_queue_snapshot.go
|
||||
notes: Config-preserving effective runtime health and capacity projection
|
||||
- type: code
|
||||
path: apps/edge/internal/service/provider_health_observability.go
|
||||
notes: Post-decision bounded metrics and safe structured-log projection
|
||||
notes: Configured health, connectivity, capacity, and queue-pressure projection
|
||||
- type: code
|
||||
path: apps/edge/internal/service/status_provider.go
|
||||
notes: lease state와 candidate pressure 기반 online/offline provider snapshot
|
||||
|
|
@ -90,12 +84,6 @@ source_evidence:
|
|||
- type: test
|
||||
path: apps/edge/internal/service/status_provider_test.go
|
||||
notes: cross-model candidate pressure와 offline/reconnect snapshot 검증
|
||||
- type: test
|
||||
path: apps/edge/internal/service/provider_health_overlay_test.go
|
||||
notes: Runtime-unavailable admission/snapshot gating, config immutability, and exact CAPABILITIES recovery
|
||||
- type: test
|
||||
path: apps/edge/internal/service/provider_health_observability_test.go
|
||||
notes: Normalized/tunnel decision projection, stale/recovery counters, private registry isolation, and lock-safe observation
|
||||
- type: test
|
||||
path: apps/edge/internal/bootstrap/reconnect_readiness_integration_test.go
|
||||
notes: dispatch-ready reconnect가 기존 queued waiter를 실제 Node terminal까지 수렴시키는 검증
|
||||
|
|
@ -118,14 +106,14 @@ Edge 설정에서 provider-pool이 어떻게 모델 실행 후보를 고르고,
|
|||
| usage attribution policy | `models[].usage_attribution`은 `provider|model_group`만 허용하고 생략 시 provider 귀속으로 해석한다. model-group 귀속은 운영자의 명시적 opt-in이다. |
|
||||
| provider mapping | `models[].providers`는 provider id를 실제 served model name으로 매핑한다. |
|
||||
| node provider catalog | `nodes[].providers[]`는 Node 아래 resource/provider catalog이며 provider id는 Edge config에서 전역 유일해야 한다. |
|
||||
| response-stall timeout | `response_stall_timeout_ms`는 provider별 response-stall timeout이다. zero/omitted는 `60000ms`, invalid negative/overflow 값은 validation error이며 selected candidate의 effective 값은 normalized/tunnel request에 보존된다. |
|
||||
| terminal ownership | provider catalog에는 watchdog timeout이 없다. request hard timeout, queue timeout, provider HTTP timeout, cancel, transport terminal은 각 기존 owner가 처리한다. |
|
||||
| config validation | config load가 provider id 참조, served model membership, numeric bounds, long-context budget을 검증한다. |
|
||||
| provider 후보 필터링 | dispatch는 dispatch-ready connection을 가진 Node의 provider 후보 중 catalog match, enabled, healthy/available, capacity 조건을 만족하는 후보만 사용한다. protocol profile capability(`messages`, `chat`, `responses`, `streaming`, `tool_calling`, `count_tokens`, `models`)는 operation별 admission에 사용된다. |
|
||||
| provider 전역 capacity/priority dispatch | `node_id + provider_id` lease가 여러 model group의 일반·long in-flight를 합산한다. available 후보 중 낮은 in-flight를 고르고 동률이면 낮은 `priority`와 round-robin을 적용한다. |
|
||||
| provider-pool 공통 queue policy | Edge root `provider_pool.max_queue`가 모든 model group의 전체 pending 상한을, `queue_timeout_ms`가 각 pending request timeout을 소유한다. |
|
||||
| global queue 재평가 | lease 반환, capacity/priority/enabled refresh, disconnect/reconnect 뒤 global enqueue 순서에서 현재 dispatch 가능한 가장 이른 waiter부터 candidate를 다시 구성한다. |
|
||||
| provider snapshot | 일반·long in-flight는 provider lease state, queued 값은 Edge queue에서 해당 provider를 후보로 포함하는 고유 pending request pressure에서 계산한다. offline provider는 catalog identity를 유지하고 effective 수치를 0으로 보고한다. |
|
||||
| runtime provider health overlay | A confirmed current bound unavailable stall lowers a separate `(node_id, connection_generation, provider_id)` overlay. The provider is excluded from effective admission and its snapshot projects unavailable with zero effective capacity/counters, while configured health remains unchanged. Only a later exact higher-sequence available CAPABILITIES probe recovers it; inconclusive evidence is a no-op. Post-decision metrics/logs expose only closed source, health, decision, and state-change values; they contain no resource identity or raw request/response data. |
|
||||
| provider availability | Configured provider health and current connection ownership are admission inputs. A normal provider or transport terminal releases the current lease once without mutating config health or initiating a probe/replay path. |
|
||||
| mixed provider execution path | 같은 model group의 OpenAI-compatible provider와 Ollama/native provider를 같은 후보군으로 두며, 선택된 provider capability로 passthrough 또는 normalized 실행 경로를 결정한다. OpenAI-compatible provider는 `openai_chat`, `anthropic_messages`, 또는 `openai_responses` driver로 해석된다. |
|
||||
| provider operation normalization | `protocol_profiles[].normalization.effort` records operation-scoped provider wire, supported normalized grades, tools compatibility, and explicit token-budget compatibility. Request admission uses these facts rather than caller identity; exact grade misses use only the nearest lower declared grade and never upgrade. |
|
||||
| long-context admission | estimated input token이 threshold 이상이면 `context_class=long`으로 분류하고, provider long slot이 있으면 일반 capacity slot과 함께 점유한다. |
|
||||
|
|
@ -165,9 +153,9 @@ sequenceDiagram
|
|||
Service->>Queue: dispatch-ready provider 후보 선택(capacity + priority)
|
||||
Queue-->>Service: selected provider + served target
|
||||
alt selected provider supports OpenAI-compatible call
|
||||
Service->>Node: ProviderTunnelRequest(adapter, served target, response-stall timeout)
|
||||
Service->>Node: ProviderTunnelRequest(adapter, served target)
|
||||
else selected provider is Ollama/native
|
||||
Service->>Node: RunRequest(adapter, served target, response-stall timeout)
|
||||
Service->>Node: RunRequest(adapter, served target)
|
||||
end
|
||||
|
||||
participant Operator
|
||||
|
|
@ -193,7 +181,6 @@ sequenceDiagram
|
|||
- `protocol_profiles` is the top-level catalog of custom overlays. A `ProtocolProfileConf` supplies `base`, `driver`, `base_url`, operation paths, `auth`, `capabilities`, `model_mapping`, `normalization`, and `extensions`; `base` inheritance is separate from legacy provider-type normalization.
|
||||
- `normalization.effort[operation]` must reference a declared operation and a recognized wire. Grade keys are the closed `none|low|medium|high|xhigh|max` order. Runtime mapping takes the exact key or nearest lower key; canonical mapped values cannot exceed the source grade. `with_tools` and `token_budget` describe whether that operation preserves the corresponding semantic combination.
|
||||
- `nodes[].providers[].profile` selects a catalog entry. Config normalization resolves that selection (or a legacy type alias) into the runtime-only `RuntimeProfile` snapshot; the source YAML remains a selector plus catalog, not a per-model overlay.
|
||||
- `nodes[].providers[].response_stall_timeout_ms` is validated at config load: zero/omitted resolves to `60000ms`; safe positive values are retained; negative and duration-overflow values are rejected. Its effective value is immutable for the selected provider attempt and survives queue re-resolution for both execution paths.
|
||||
- Profile catalog and provider-selector changes are restart-required. Snapshot immutability describes loaded runtime state and does not make those changes live-applicable.
|
||||
- `ConcreteProtocolProfile.MapModel(model)`은 provider의 model alias 정규화를 수행한다. provider가 model mapping을 정의하면 IOP external `model` key를 provider served target으로 변환한다.
|
||||
- `ConcreteProtocolProfile.ResolveOperationURL(op)` returns the complete resolved upstream URL. Absolute operation URLs are returned unchanged, while relative operation paths are joined once to the normalized base URL; the listed `/v1/...` values are operation-path inputs, not return values.
|
||||
|
|
@ -206,10 +193,10 @@ sequenceDiagram
|
|||
- `nodes[].providers[].capacity`와 `long_context_capacity`는 provider resource 속성이고 같은 provider를 공유하는 model alias가 합산 점유한다. 같은 key의 legacy adapter capacity는 transport 상한이므로 provider admission capacity는 그 이하일 수 있고 상한 초과만 거부한다. `total_context_tokens`는 runtime ledger가 아니라 `context_window_tokens * long_context_capacity` 정적 validation 값이다.
|
||||
- `models[].usage_attribution`은 생략 시 `provider`, 명시값은 `provider|model_group`만 허용한다. 변경은 model catalog policy 변경으로 live apply되며 `models["<id>"].usage_attribution` 경로로 보고한다.
|
||||
- provider `enabled=false`는 dispatch pool에서 제외하지만 adapter process lifecycle 변경을 의미하지 않는다.
|
||||
- Runtime health is not a config-refresh field. The overlay never rewrites `nodes[].providers[].health`, is discarded across connection generations, and participates only in effective candidate eligibility and snapshot projection.
|
||||
- Runtime terminal handling does not rewrite `nodes[].providers[].health`; provider health remains an operator-owned config input.
|
||||
- accepted registration은 provider candidate를 바로 복구하지 않는다. Node가 config 적용과 handler 설치 뒤 ready ack를 받아야 해당 generation이 candidate, connected snapshot, refresh push 대상이 되며 이 transition이 stranded provider-pool waiter를 재평가한다.
|
||||
- provider capacity, long-context capacity, priority, enabled toggle, root queue policy와 model generation policy는 live apply 대상으로 분류된다. apply는 기존 lease를 보존하고 이후 admission 및 모든 관련 waiter의 live candidate/deadline을 새 값으로 재평가한다.
|
||||
- `response_stall_timeout_ms` 변경은 restart-required다. request hard timeout, queue timeout, heartbeat/disconnect, client response-idle timeout과 watchdog timer lifecycle은 별도 소유권이다.
|
||||
- request hard timeout, queue timeout, heartbeat/disconnect, and client response-idle timeout retain separate existing ownership.
|
||||
- Edge listener, control plane, openai/a2a listener, bootstrap artifact path, node 추가/삭제, node token/alias, adapter 설정 변경은 restart-required 대상이다.
|
||||
- `openai.principal_tokens[]`는 `token_ref`와 `token_hash_sha256` 중복을 거부하고, raw token 원문은 tracked config에 저장하지 않는다.
|
||||
- 여러 `openai.principal_tokens[]` entry가 같은 `principal_ref`를 공유할 수 있으며, 이때 `token_ref`가 앱/통합/용도별 사용량 분해 기준이다.
|
||||
|
|
@ -231,7 +218,7 @@ sequenceDiagram
|
|||
|
||||
## 한계와 주의사항
|
||||
|
||||
- Active health coverage is intentionally limited to confirmed response-stall evidence and explicit exact-target CAPABILITIES recovery. It is not a general background provider health polling system.
|
||||
- Provider health is a configured admission input; this flow does not add background health polling or output-silence recovery.
|
||||
- refresh admin API는 operator-local 표면이다. 접근 제어 없이 public interface에 노출하지 않는다.
|
||||
- Stream Evidence Gate의 request-local lifecycle과 지원 OpenAI 경로는 `agent-spec/runtime/stream-evidence-gate.md`에서 관리한다.
|
||||
- adapter structural 변경은 contract상 restart-required로 분류된다. Node handler가 registry swap을 지원하더라도 Edge refresh classifier가 허용한 변경만 apply해야 한다.
|
||||
|
|
@ -260,9 +247,7 @@ sequenceDiagram
|
|||
- 2026-08-01: protocol profile catalog/selector ownership, runtime-only profile resolution, and restart-required refresh semantics were synchronized with config source.
|
||||
- 2026-08-02: Synchronized the managed credential mode switch, TLS/key prerequisites, legacy-auth exclusion, projected route binding, and restart-required credential-plane classification with current validation/runtime source.
|
||||
- 2026-08-02: Added the `glm_coding` built-in profile alongside `glm` (General API), both exposing only `models` + `chat_completions` with Bearer auth and no Responses. Endpoint selection is driven by external model IDs mapped to distinct provider IDs. No automatic fallback between General API and Coding Plan. Both are comment-only in the example config and disabled by default. Coding Plan usage is subject to current Z.AI subscription terms.
|
||||
- 2026-08-04: Added provider response-stall timeout validation/default, restart-required refresh classification, selected-candidate propagation, and Node retention. Timer/watchdog lifecycle remains out of scope.
|
||||
- 2026-08-05: Added the separate generation-scoped runtime provider health overlay, effective admission/snapshot exclusion, config-health immutability, and exact higher-sequence CAPABILITIES recovery.
|
||||
- 2026-08-05: Added post-decision provider-health operational evidence with bounded counters and structured logs, isolated from overlay state and provider identity.
|
||||
- 2026-08-15: Removed the provider output-silence timeout setting, wire propagation, refresh classification, and runtime health overlay. Strict load rejects the removed key and normal terminal handling leaves configured health unchanged.
|
||||
- 2026-08-09: Synchronized the single-request effective-template boundary: relative-only `plan_file`/`review_file` resolution against the `edge.yaml` directory with pre-access absolute rejection, independent per-file default fallback, load-time regular-file/size/UTF-8/grammar rejection, digest-only refresh diff evidence, and admission-time freezing so a refresh reaches only newly admitted requests (`packages/go/config/load.go`, `apps/edge/internal/configrefresh/classify.go`, `apps/edge/internal/openai/single_request_preset_binding.go`).
|
||||
- 2026-08-13: Added operation-scoped provider effort normalization facts and nearest-lower, never-upgrade mapping semantics to the protocol profile catalog.
|
||||
- 2026-08-06: Synchronized the fixed single-request policy (`execution_presets[].single_request`) absolute caps, plan→work→review stage shape, opaque `workspace_ref`, live-apply classification, and snapshot-isolation semantics with current code, contract, and classifier implementation.
|
||||
|
|
|
|||
|
|
@ -30,15 +30,9 @@ source_evidence:
|
|||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_pipeline_test.go
|
||||
notes: Chat/Responses tunnel의 exact-wire terminal, split tool identity, non-2xx lifecycle 검증
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_stall_recovery_test.go
|
||||
notes: S05 endpoint/path/semantic recovery matrix, shared budget, candidate identity, transport close, guard terminals, and disabled-semantic compatibility
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/filter_observation_sink_test.go
|
||||
notes: raw-free observation allowlist와 correlation 검증
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/liveness_recovery_observability_test.go
|
||||
notes: request-local closed-label liveness metrics, safe default-log projection, and explicit-sink forwarding
|
||||
- type: code
|
||||
path: apps/edge/internal/openai/single_request_anthropic_stream.go
|
||||
notes: Separate marked service-to-Anthropic progress/ping/terminal projector that does not enter the generic gate
|
||||
|
|
@ -66,8 +60,8 @@ codec이 정규화한 provider event를 downstream에 쓰기 전에 evidence와
|
|||
| repeat-resume builder | A selected continuation plan can consume one request-local content/reasoning snapshot and build endpoint-native Chat or Responses resume input with the fixed English directive, without caller history or another model call. |
|
||||
| active repeat guard | Request-local Chat/Responses history fingerprints, a Unicode rolling pending window, and committed look-behind produce sanitized pass, continuation, repeated-action safe-stop, or side-effect fatal decisions. |
|
||||
| host re-admission | 현재 provider ownership을 닫은 뒤 optional one-shot prepare, rebuild, budget consume, 단일 dispatch 순서로 새 actual model/provider/path binding을 설치한다. |
|
||||
| raw-free observation | request correlation, attempt/epoch, filter/rule, decision, recovery와 bounded sanitized cause/evidence만 timeline sink로 보낸다. The OpenAI liveness projection additionally emits one closed eligibility metric and at most one closed final-result metric per private cycle. |
|
||||
| typed stall handoff | Every supported OpenAI Chat/Responses normalized or tunnel request has one always-on runtime liveness owner. It maps only an Edge-confirmed `response_stalled` terminal to a raw-free provider error and evaluates ExactReplay through the existing commit/cancel/side-effect/snapshot/shared-budget contract. |
|
||||
| raw-free observation | request correlation, attempt/epoch, filter/rule, decision, recovery와 bounded sanitized cause/evidence만 timeline sink로 보낸다. |
|
||||
| terminal boundary | Silence is not a StreamGate event. Provider/transport terminals and caller cancellation are projected by their existing endpoint adapters; watchdog-specific recovery is absent. |
|
||||
| separate marked Anthropic projection | The single-request coordinator's fixed plan/work/review/repair summaries, `event: ping`, content indices, and endpoint terminal are owned by a separate serialized service-to-endpoint projector. They do not become normalized gate events, filters, release decisions, or recovery inputs. |
|
||||
|
||||
## 범위
|
||||
|
|
@ -109,13 +103,11 @@ sequenceDiagram
|
|||
|
||||
## 설정/데이터/이벤트
|
||||
|
||||
- `openai.stream_evidence_gate.enabled` defaults to `false` and controls only configured semantic filters and their capability admission. The Core owns the supported response/liveness lifecycle in both states, while disabled mode preserves endpoint-native compatibility through runtime adapters.
|
||||
- `openai.stream_evidence_gate.enabled` defaults to `false` and controls only configured semantic filters and their capability admission. Disabled mode preserves endpoint-native compatibility through runtime adapters.
|
||||
- `max_request_fault_recovery`는 0..3, `max_strategy_fault_recovery`는 0..request-total이고 생략 시 request-total을 상속한다. base 또는 selector에서 활성화될 수 있는 blocking `repeat_guard`는 두 상한이 모두 최소 1이어야 한다.
|
||||
- `max_ingress_snapshot_bytes`는 1..16777216이며 생략 시 16 MiB다. raw body limit은 첫 read 전에 적용되고 canonical body, typed view와 rebuild peak가 같은 request-local ledger에 포함된다.
|
||||
- Stream Evidence Gate 설정 변경은 현재 restart-required다. request가 시작된 뒤 config/registry snapshot은 바뀌지 않는다.
|
||||
- The production Core registry includes the common Noop filter, configured active `repeat_guard`, schema/provider-error lifecycle foundations, and applicable request-local tool validation. Repeat detection uses the configured 500-rune default, never time-based release, and returns a continuation only before a tool/side-effect boundary. Provider-error still records unmatched errors as pass until its matcher Task.
|
||||
- The private typed-stall evaluator is always registered for supported requests and is independent from configured semantic `filters[]` and provider capability admission. It closes a confirmed old transport without a duplicate cancel and passes the failed provider once to pool re-admission; only `available` permits avoided-provider fallback.
|
||||
- Liveness metrics use only `execution_path`, `provider_health`, `commit_state`, `eligibility`, and `recovery_result` closed vocabularies. Constructor-owned generic zap logging is replaced for the private liveness/ExactReplay rows with a safe projection; a sink supplied through `SetObservationSink` still receives the original immutable observations.
|
||||
- Resume recording is bounded by the ingress snapshot limit and is reset for every attempt. The Rebuilder consumes it once after the owning attempt is aborted. It uses the request-start model catalog context window and fails before dispatch when the window is unknown or the rebuilt prompt plus its completion reserve does not fit.
|
||||
- A repeat continuation cursor is a UTF-8 byte boundary for content or reasoning. Already committed look-behind fixes the cursor at the released channel boundary; the pending duplicate is discarded, and a byte-identical replacement prefix is suppressed once. Omitted temperature uses `0.2`, `0.4`, and `0.6` by strategy attempt; explicit temperature is preserved.
|
||||
- Marked single-request Anthropic progress consumes only the coordinator's closed public enum in its endpoint projector. Its pings and terminal lock do not pass through the Core registry, mutate request-start gate snapshots, or enable generic filters/recovery.
|
||||
|
|
@ -141,7 +133,5 @@ sequenceDiagram
|
|||
- 2026-07-28: Chat/Responses tunnel의 terminal wire queue, split tool identity와 non-2xx provider-error lifecycle 근거로 normalized Responses runtime 범위와 foundation 한계를 현재 구현에 맞췄다.
|
||||
- 2026-07-28: Added the request-local Chat/Responses repeat-resume builder, its bounded recorder lifecycle, fixed directive, caller-history exclusion, and context-window fail-closed boundary.
|
||||
- 2026-07-29: Activated request-local history/current-stream repeat detection, Unicode safe cursors, no-progress action safe-stop, one-shot prefix suppression, and continuation temperature candidates.
|
||||
- 2026-08-05: Added raw-free `response_stalled` mapping and runtime-local confirmed-handoff recovery ownership for OpenAI StreamGate attempts.
|
||||
- 2026-08-05: Made supported Chat/Responses normalized and tunnel liveness ownership unconditional, isolated semantic activation to configured filters/capability admission, and added deterministic S05 recovery/guard/compatibility evidence.
|
||||
- 2026-08-06: Added request-local liveness eligibility/result metrics and constructor-default-only safe observation-log projection.
|
||||
- 2026-08-15: Removed watchdog terminal mapping and its recovery/observation path. Silence now leaves the request active until an existing terminal input arrives.
|
||||
- 2026-08-06: Recorded the marked single-request Anthropic projector as a separate service-to-endpoint boundary without expanding generic gate events, filters, release, recovery, or observation semantics.
|
||||
|
|
|
|||
|
|
@ -34,7 +34,6 @@ class RunRequest extends $pb.GeneratedMessage {
|
|||
$core.Iterable<$core.MapEntry<$core.String, $core.String>>? metadata,
|
||||
$core.String? sessionId,
|
||||
$core.bool? background,
|
||||
$fixnum.Int64? responseStallTimeoutMs,
|
||||
}) {
|
||||
final result = create();
|
||||
if (runId != null) result.runId = runId;
|
||||
|
|
@ -46,8 +45,6 @@ class RunRequest extends $pb.GeneratedMessage {
|
|||
if (metadata != null) result.metadata.addEntries(metadata);
|
||||
if (sessionId != null) result.sessionId = sessionId;
|
||||
if (background != null) result.background = background;
|
||||
if (responseStallTimeoutMs != null)
|
||||
result.responseStallTimeoutMs = responseStallTimeoutMs;
|
||||
return result;
|
||||
}
|
||||
|
||||
|
|
@ -79,7 +76,6 @@ class RunRequest extends $pb.GeneratedMessage {
|
|||
packageName: const $pb.PackageName('iop'))
|
||||
..aOS(9, _omitFieldNames ? '' : 'sessionId')
|
||||
..aOB(11, _omitFieldNames ? '' : 'background')
|
||||
..aInt64(12, _omitFieldNames ? '' : 'responseStallTimeoutMs')
|
||||
..hasRequiredFields = false;
|
||||
|
||||
@$core.Deprecated('See https://github.com/google/protobuf.dart/issues/998.')
|
||||
|
|
@ -178,19 +174,6 @@ class RunRequest extends $pb.GeneratedMessage {
|
|||
$core.bool hasBackground() => $_has(8);
|
||||
@$pb.TagNumber(11)
|
||||
void clearBackground() => $_clearField(11);
|
||||
|
||||
/// response_stall_timeout_ms is the selected provider's response-stall
|
||||
/// timeout in milliseconds. Zero means the Node applies the documented
|
||||
/// default (300000). Negative or overflow values are rejected at the Node
|
||||
/// boundary before router/provider invocation.
|
||||
@$pb.TagNumber(12)
|
||||
$fixnum.Int64 get responseStallTimeoutMs => $_getI64(9);
|
||||
@$pb.TagNumber(12)
|
||||
set responseStallTimeoutMs($fixnum.Int64 value) => $_setInt64(9, value);
|
||||
@$pb.TagNumber(12)
|
||||
$core.bool hasResponseStallTimeoutMs() => $_has(9);
|
||||
@$pb.TagNumber(12)
|
||||
void clearResponseStallTimeoutMs() => $_clearField(12);
|
||||
}
|
||||
|
||||
/// RunEvent is a streaming execution event.
|
||||
|
|
@ -415,7 +398,6 @@ class ProviderTunnelRequest extends $pb.GeneratedMessage {
|
|||
$core.String? operation,
|
||||
SignedCredentialLease? credentialLease,
|
||||
CredentialLeaseBinding? credentialBinding,
|
||||
$fixnum.Int64? responseStallTimeoutMs,
|
||||
}) {
|
||||
final result = create();
|
||||
if (runId != null) result.runId = runId;
|
||||
|
|
@ -433,8 +415,6 @@ class ProviderTunnelRequest extends $pb.GeneratedMessage {
|
|||
if (operation != null) result.operation = operation;
|
||||
if (credentialLease != null) result.credentialLease = credentialLease;
|
||||
if (credentialBinding != null) result.credentialBinding = credentialBinding;
|
||||
if (responseStallTimeoutMs != null)
|
||||
result.responseStallTimeoutMs = responseStallTimeoutMs;
|
||||
return result;
|
||||
}
|
||||
|
||||
|
|
@ -478,7 +458,6 @@ class ProviderTunnelRequest extends $pb.GeneratedMessage {
|
|||
..aOM<CredentialLeaseBinding>(
|
||||
15, _omitFieldNames ? '' : 'credentialBinding',
|
||||
subBuilder: CredentialLeaseBinding.create)
|
||||
..aInt64(16, _omitFieldNames ? '' : 'responseStallTimeoutMs')
|
||||
..hasRequiredFields = false;
|
||||
|
||||
@$core.Deprecated('See https://github.com/google/protobuf.dart/issues/998.')
|
||||
|
|
@ -636,19 +615,6 @@ class ProviderTunnelRequest extends $pb.GeneratedMessage {
|
|||
void clearCredentialBinding() => $_clearField(15);
|
||||
@$pb.TagNumber(15)
|
||||
CredentialLeaseBinding ensureCredentialBinding() => $_ensure(14);
|
||||
|
||||
/// response_stall_timeout_ms is the selected provider's response-stall
|
||||
/// timeout in milliseconds. Zero means the Node applies the documented
|
||||
/// default (300000). Negative or overflow values are rejected at the Node
|
||||
/// boundary before router/provider invocation.
|
||||
@$pb.TagNumber(16)
|
||||
$fixnum.Int64 get responseStallTimeoutMs => $_getI64(15);
|
||||
@$pb.TagNumber(16)
|
||||
set responseStallTimeoutMs($fixnum.Int64 value) => $_setInt64(15, value);
|
||||
@$pb.TagNumber(16)
|
||||
$core.bool hasResponseStallTimeoutMs() => $_has(15);
|
||||
@$pb.TagNumber(16)
|
||||
void clearResponseStallTimeoutMs() => $_clearField(16);
|
||||
}
|
||||
|
||||
class CredentialLeaseScope extends $pb.GeneratedMessage {
|
||||
|
|
|
|||
|
|
@ -211,20 +211,14 @@ const RunRequest$json = {
|
|||
},
|
||||
{'1': 'session_id', '3': 9, '4': 1, '5': 9, '10': 'sessionId'},
|
||||
{'1': 'background', '3': 11, '4': 1, '5': 8, '10': 'background'},
|
||||
{
|
||||
'1': 'response_stall_timeout_ms',
|
||||
'3': 12,
|
||||
'4': 1,
|
||||
'5': 3,
|
||||
'10': 'responseStallTimeoutMs'
|
||||
},
|
||||
],
|
||||
'3': [RunRequest_MetadataEntry$json],
|
||||
'9': [
|
||||
{'1': 4, '2': 5},
|
||||
{'1': 10, '2': 11},
|
||||
{'1': 12, '2': 13},
|
||||
],
|
||||
'10': ['workspace', 'session_mode'],
|
||||
'10': ['workspace', 'session_mode', 'response_stall_timeout_ms'],
|
||||
};
|
||||
|
||||
@$core.Deprecated('Use runRequestDescriptor instead')
|
||||
|
|
@ -245,9 +239,9 @@ final $typed_data.Uint8List runRequestDescriptor = $convert.base64Decode(
|
|||
'YuU3RydWN0UgVpbnB1dBIfCgt0aW1lb3V0X3NlYxgHIAEoBVIKdGltZW91dFNlYxI5CghtZXRh'
|
||||
'ZGF0YRgIIAMoCzIdLmlvcC5SdW5SZXF1ZXN0Lk1ldGFkYXRhRW50cnlSCG1ldGFkYXRhEh0KCn'
|
||||
'Nlc3Npb25faWQYCSABKAlSCXNlc3Npb25JZBIeCgpiYWNrZ3JvdW5kGAsgASgIUgpiYWNrZ3Jv'
|
||||
'dW5kEjkKGXJlc3BvbnNlX3N0YWxsX3RpbWVvdXRfbXMYDCABKANSFnJlc3BvbnNlU3RhbGxUaW'
|
||||
'1lb3V0TXMaOwoNTWV0YWRhdGFFbnRyeRIQCgNrZXkYASABKAlSA2tleRIUCgV2YWx1ZRgCIAEo'
|
||||
'CVIFdmFsdWU6AjgBSgQIBBAFSgQIChALUgl3b3Jrc3BhY2VSDHNlc3Npb25fbW9kZQ==');
|
||||
'dW5kGjsKDU1ldGFkYXRhRW50cnkSEAoDa2V5GAEgASgJUgNrZXkSFAoFdmFsdWUYAiABKAlSBX'
|
||||
'ZhbHVlOgI4AUoECAQQBUoECAoQC0oECAwQDVIJd29ya3NwYWNlUgxzZXNzaW9uX21vZGVSGXJl'
|
||||
'c3BvbnNlX3N0YWxsX3RpbWVvdXRfbXM=');
|
||||
|
||||
@$core.Deprecated('Use runEventDescriptor instead')
|
||||
const RunEvent$json = {
|
||||
|
|
@ -353,18 +347,15 @@ const ProviderTunnelRequest$json = {
|
|||
'6': '.iop.CredentialLeaseBinding',
|
||||
'10': 'credentialBinding'
|
||||
},
|
||||
{
|
||||
'1': 'response_stall_timeout_ms',
|
||||
'3': 16,
|
||||
'4': 1,
|
||||
'5': 3,
|
||||
'10': 'responseStallTimeoutMs'
|
||||
},
|
||||
],
|
||||
'3': [
|
||||
ProviderTunnelRequest_HeadersEntry$json,
|
||||
ProviderTunnelRequest_MetadataEntry$json
|
||||
],
|
||||
'9': [
|
||||
{'1': 16, '2': 17},
|
||||
],
|
||||
'10': ['response_stall_timeout_ms'],
|
||||
};
|
||||
|
||||
@$core.Deprecated('Use providerTunnelRequestDescriptor instead')
|
||||
|
|
@ -399,10 +390,10 @@ final $typed_data.Uint8List providerTunnelRequestDescriptor = $convert.base64Dec
|
|||
'Vzc2lvbl9pZBgMIAEoCVIJc2Vzc2lvbklkEhwKCW9wZXJhdGlvbhgNIAEoCVIJb3BlcmF0aW9u'
|
||||
'EkUKEGNyZWRlbnRpYWxfbGVhc2UYDiABKAsyGi5pb3AuU2lnbmVkQ3JlZGVudGlhbExlYXNlUg'
|
||||
'9jcmVkZW50aWFsTGVhc2USSgoSY3JlZGVudGlhbF9iaW5kaW5nGA8gASgLMhsuaW9wLkNyZWRl'
|
||||
'bnRpYWxMZWFzZUJpbmRpbmdSEWNyZWRlbnRpYWxCaW5kaW5nEjkKGXJlc3BvbnNlX3N0YWxsX3'
|
||||
'RpbWVvdXRfbXMYECABKANSFnJlc3BvbnNlU3RhbGxUaW1lb3V0TXMaOgoMSGVhZGVyc0VudHJ5'
|
||||
'EhAKA2tleRgBIAEoCVIDa2V5EhQKBXZhbHVlGAIgASgJUgV2YWx1ZToCOAEaOwoNTWV0YWRhdG'
|
||||
'FFbnRyeRIQCgNrZXkYASABKAlSA2tleRIUCgV2YWx1ZRgCIAEoCVIFdmFsdWU6AjgB');
|
||||
'bnRpYWxMZWFzZUJpbmRpbmdSEWNyZWRlbnRpYWxCaW5kaW5nGjoKDEhlYWRlcnNFbnRyeRIQCg'
|
||||
'NrZXkYASABKAlSA2tleRIUCgV2YWx1ZRgCIAEoCVIFdmFsdWU6AjgBGjsKDU1ldGFkYXRhRW50'
|
||||
'cnkSEAoDa2V5GAEgASgJUgNrZXkSFAoFdmFsdWUYAiABKAlSBXZhbHVlOgI4AUoECBAQEVIZcm'
|
||||
'VzcG9uc2Vfc3RhbGxfdGltZW91dF9tcw==');
|
||||
|
||||
@$core.Deprecated('Use credentialLeaseScopeDescriptor instead')
|
||||
const CredentialLeaseScope$json = {
|
||||
|
|
|
|||
|
|
@ -73,7 +73,6 @@ func NewRuntime(cfg *config.EdgeConfig) (*Runtime, error) {
|
|||
|
||||
bus := edgeevents.NewBus()
|
||||
svc := edgeservice.New(registry, bus)
|
||||
svc.SetProviderHealthLogger(logger.Named("provider-health"))
|
||||
// Install the service-owned lifecycle projection before creating input
|
||||
// servers, so every accepted request observes the same bounded sink.
|
||||
svc.SetSingleRequestObservationLogger(logger.Named("single-request"))
|
||||
|
|
|
|||
|
|
@ -95,13 +95,12 @@ type providerKey struct {
|
|||
LifecycleCapabilities []string
|
||||
Enabled bool
|
||||
|
||||
Provider string
|
||||
Endpoint string
|
||||
BaseURL string
|
||||
Headers map[string]string
|
||||
ContextSize int
|
||||
RequestTimeoutMS int
|
||||
ResponseStallTimeoutMS int64
|
||||
Provider string
|
||||
Endpoint string
|
||||
BaseURL string
|
||||
Headers map[string]string
|
||||
ContextSize int
|
||||
RequestTimeoutMS int
|
||||
}
|
||||
|
||||
func buildProviderIndex(cfg *config.EdgeConfig) map[string]providerKey {
|
||||
|
|
@ -110,26 +109,25 @@ func buildProviderIndex(cfg *config.EdgeConfig) map[string]providerKey {
|
|||
nodeKey := nodeIdentity(node, i)
|
||||
for _, p := range node.Providers {
|
||||
idx[p.ID] = providerKey{
|
||||
NodeKey: nodeKey,
|
||||
Type: p.Type,
|
||||
Category: p.Category,
|
||||
Adapter: p.Adapter,
|
||||
Profile: p.Profile,
|
||||
Models: append([]string(nil), p.Models...),
|
||||
Health: p.Health,
|
||||
Capacity: p.Capacity,
|
||||
TotalContextTokens: p.TotalContextTokens,
|
||||
LongContextCapacity: p.LongContextCapacity,
|
||||
Priority: p.Priority,
|
||||
LifecycleCapabilities: append([]string(nil), p.LifecycleCapabilities...),
|
||||
Enabled: config.ProviderEnabled(p),
|
||||
Provider: p.Provider,
|
||||
Endpoint: p.Endpoint,
|
||||
BaseURL: p.BaseURL,
|
||||
Headers: cloneStringMap(p.Headers),
|
||||
ContextSize: p.ContextSize,
|
||||
RequestTimeoutMS: p.RequestTimeoutMS,
|
||||
ResponseStallTimeoutMS: p.EffectiveResponseStallTimeoutMS(),
|
||||
NodeKey: nodeKey,
|
||||
Type: p.Type,
|
||||
Category: p.Category,
|
||||
Adapter: p.Adapter,
|
||||
Profile: p.Profile,
|
||||
Models: append([]string(nil), p.Models...),
|
||||
Health: p.Health,
|
||||
Capacity: p.Capacity,
|
||||
TotalContextTokens: p.TotalContextTokens,
|
||||
LongContextCapacity: p.LongContextCapacity,
|
||||
Priority: p.Priority,
|
||||
LifecycleCapabilities: append([]string(nil), p.LifecycleCapabilities...),
|
||||
Enabled: config.ProviderEnabled(p),
|
||||
Provider: p.Provider,
|
||||
Endpoint: p.Endpoint,
|
||||
BaseURL: p.BaseURL,
|
||||
Headers: cloneStringMap(p.Headers),
|
||||
ContextSize: p.ContextSize,
|
||||
RequestTimeoutMS: p.RequestTimeoutMS,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -289,7 +287,6 @@ func appendProviderStructuralChanges(changes *[]Change, current, candidate map[s
|
|||
appendDeepIfChanged(changes, fmt.Sprintf("nodes[].providers[%q].headers", provID), StatusRestartRequired, cur.Headers, next.Headers)
|
||||
appendIfChanged(changes, fmt.Sprintf("nodes[].providers[%q].context_size", provID), StatusRestartRequired, cur.ContextSize, next.ContextSize)
|
||||
appendIfChanged(changes, fmt.Sprintf("nodes[].providers[%q].request_timeout_ms", provID), StatusRestartRequired, cur.RequestTimeoutMS, next.RequestTimeoutMS)
|
||||
appendIfChanged(changes, fmt.Sprintf("nodes[].providers[%q].response_stall_timeout_ms", provID), StatusRestartRequired, cur.ResponseStallTimeoutMS, next.ResponseStallTimeoutMS)
|
||||
}
|
||||
for provID := range candidate {
|
||||
if _, exists := current[provID]; !exists {
|
||||
|
|
|
|||
|
|
@ -1,71 +0,0 @@
|
|||
package configrefresh_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
configrefresh "iop/apps/edge/internal/configrefresh"
|
||||
)
|
||||
|
||||
func TestProviderResponseStallTimeoutRefreshClassification(t *testing.T) {
|
||||
base := `server:
|
||||
listen: "0.0.0.0:9090"
|
||||
nodes:
|
||||
- id: "node-1"
|
||||
token: "tok-1"
|
||||
adapters:
|
||||
vllm:
|
||||
enabled: true
|
||||
endpoint: "http://127.0.0.1:8000/v1"
|
||||
providers:
|
||||
- id: "prov-a"
|
||||
type: "vllm"
|
||||
category: "api"
|
||||
adapter: "vllm"
|
||||
endpoint: "http://127.0.0.1:8000/v1"
|
||||
models: ["m"]
|
||||
capacity: 2
|
||||
`
|
||||
dir := t.TempDir()
|
||||
currentPath, candidatePath := filepath.Join(dir, "current.yaml"), filepath.Join(dir, "candidate.yaml")
|
||||
if err := os.WriteFile(currentPath, []byte(base), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
current, err := configrefresh.LoadCandidate(currentPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
raw string
|
||||
want bool
|
||||
}{
|
||||
{name: "positive is restart required", raw: "45000", want: true},
|
||||
{name: "explicit zero matches omitted", raw: "0"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
candidateYAML := strings.Replace(base, "capacity: 2\n", "capacity: 2\n response_stall_timeout_ms: "+tc.raw+"\n", 1)
|
||||
if err := os.WriteFile(candidatePath, []byte(candidateYAML), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
candidate, err := configrefresh.LoadCandidate(candidatePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := false
|
||||
for _, change := range configrefresh.Classify(current, candidate).Changes {
|
||||
if change.Path == `nodes[].providers["prov-a"].response_stall_timeout_ms` {
|
||||
found = true
|
||||
if change.Class != configrefresh.StatusRestartRequired {
|
||||
t.Errorf("change class = %v", change.Class)
|
||||
}
|
||||
}
|
||||
}
|
||||
if found != tc.want {
|
||||
t.Errorf("change found = %t, want %t", found, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -10,7 +10,7 @@ import (
|
|||
|
||||
// streamBufferedChatCompletion serves a buffered (strict or tool-bearing) SSE
|
||||
// chat completion through the request runtime. Semantic filters remain
|
||||
// configurable, while request-local liveness recovery is always registered.
|
||||
// configurable.
|
||||
func (s *Server) streamBufferedChatCompletion(w http.ResponseWriter, dc *chatDispatchContext, handle edgeservice.RunResult, flusher http.Flusher) {
|
||||
s.runOpenAIBufferedChatStreamGate(w, flusher, dc, handle, true)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,8 +33,7 @@ func chatSubmitRunRequest(dispatch routeDispatch, req chatCompletionRequest, pro
|
|||
}
|
||||
|
||||
// completeChatCompletion serves a non-streaming chat completion through the
|
||||
// request runtime, which owns liveness recovery independently of semantic
|
||||
// filter activation. This surface supplies the buffered event source and JSON
|
||||
// request runtime. This surface supplies the buffered event source and JSON
|
||||
// renderer that preserve the endpoint-native response contract.
|
||||
func (s *Server) completeChatCompletion(w http.ResponseWriter, dc *chatDispatchContext, handle edgeservice.RunResult) {
|
||||
s.runOpenAIBufferedChatStreamGate(w, nil, dc, handle, false)
|
||||
|
|
|
|||
|
|
@ -1,440 +0,0 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/promauto"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
// This file implements the request-local liveness recovery observation
|
||||
// projection (SDD S06). It sits between StreamGate's immutable
|
||||
// FilterObservation timeline and the configured downstream sink. For each
|
||||
// request runtime, Server.observationSink() returns one fresh
|
||||
// openAILivenessObservationSink. That wrapper watches the predecessor-owned
|
||||
// private liveness filter (openAIStallRecoveryFilterID / openai.liveness) plus
|
||||
// the ExactReplay recovery lifecycle it arms, and emits exactly one bounded
|
||||
// eligibility observation and at most one final result per liveness cycle. It
|
||||
// never changes filter arbitration, recovery budgets, dispatch, or any Core
|
||||
// observation; sink/metric/log failures stay observation-only.
|
||||
|
||||
// liveness_recovery_observability metric label vocabularies. All values are
|
||||
// closed and low-cardinality: correlation/request/attempt/run/session/model/
|
||||
// provider/node/lease/slot/credential identifiers and raw prompt/response/tool
|
||||
// content are never used as labels or logged.
|
||||
|
||||
const (
|
||||
livenessMetricEligibilityName = "iop_edge_liveness_recovery_eligibility_total"
|
||||
livenessMetricResultsName = "iop_edge_liveness_recovery_results_total"
|
||||
|
||||
livenessLogMessage = "edge_liveness_recovery_observation"
|
||||
)
|
||||
|
||||
// execution_path label values.
|
||||
const (
|
||||
livenessPathNormalized = "normalized"
|
||||
livenessPathProviderTunnel = "provider_tunnel"
|
||||
livenessPathUnknown = "unknown"
|
||||
)
|
||||
|
||||
// provider_health label values.
|
||||
const (
|
||||
livenessHealthAvailable = "available"
|
||||
livenessHealthUnavailable = "unavailable"
|
||||
livenessHealthUnknown = "unknown"
|
||||
)
|
||||
|
||||
// commit_state label values (Core's closed CommitState plus unknown fallback).
|
||||
const (
|
||||
livenessCommitUncommitted = "transport_uncommitted"
|
||||
livenessCommitStreamOpen = "stream_open"
|
||||
livenessCommitTerminal = "terminal_committed"
|
||||
livenessCommitUnknown = "unknown"
|
||||
)
|
||||
|
||||
// eligibility label values. Only the values the predecessor filter can produce
|
||||
// are reachable today; the rest are reserved so the closed vocabulary does not
|
||||
// have to change if the predecessor's descriptor set grows. See
|
||||
// classifyLivenessEligibility for the exact descriptor mapping.
|
||||
const (
|
||||
livenessEligibilityEligible = "eligible"
|
||||
livenessEligibilityNoOwner = "no_owner"
|
||||
livenessEligibilityPostCommit = "post_commit"
|
||||
livenessEligibilityUnconfirmedFence = "unconfirmed_fence"
|
||||
livenessEligibilityCallerCancelled = "caller_cancelled"
|
||||
livenessEligibilityToolSideEffect = "tool_side_effect"
|
||||
livenessEligibilityBudgetExhausted = "budget_exhausted"
|
||||
livenessEligibilityNoCandidate = "no_candidate"
|
||||
livenessEligibilitySameProviderForbidden = "same_provider_forbidden"
|
||||
livenessEligibilityOther = "other"
|
||||
)
|
||||
|
||||
// recovery_result label values.
|
||||
const (
|
||||
livenessResultRedispatched = "redispatched"
|
||||
livenessResultPlanRejected = "plan_rejected"
|
||||
livenessResultAbortFailed = "abort_failed"
|
||||
livenessResultRebuildFailed = "rebuild_failed"
|
||||
livenessResultDispatchFailed = "dispatch_failed"
|
||||
livenessResultNotSelected = "not_selected"
|
||||
livenessResultTerminal = "terminal"
|
||||
livenessResultOther = "other"
|
||||
)
|
||||
|
||||
// Predecessor descriptor codes. These mirror the sanitized evidence descriptor
|
||||
// strings emitted by openAIStallRecoveryFilter.Evaluate in
|
||||
// stream_gate_filters.go. They are not exported constants there, so they are
|
||||
// re-declared here and covered by a test that drives the real filter, so a
|
||||
// predecessor change is caught rather than silently mismapped.
|
||||
const (
|
||||
livenessDescriptorConfirmed = "response_stalled_confirmed"
|
||||
livenessDescriptorUnconfirmed = "response_stalled_unconfirmed"
|
||||
livenessDescriptorIneligible = "response_stalled_ineligible"
|
||||
livenessDescriptorProviderIgnored = "provider_error_ignored"
|
||||
)
|
||||
|
||||
// livenessRecoveryCollectors is one Prometheus collector set for the liveness
|
||||
// recovery projection. The production set is registered exactly once at package
|
||||
// initialization against the default registerer; tests construct isolated sets
|
||||
// against an explicit registry. NewServer, observationSink(), and the request
|
||||
// wrapper never register collectors.
|
||||
type livenessRecoveryCollectors struct {
|
||||
eligibility *prometheus.CounterVec
|
||||
results *prometheus.CounterVec
|
||||
}
|
||||
|
||||
// newLivenessRecoveryCollectors registers the eligibility and result counters
|
||||
// against reg. A nil reg falls back to the default registerer.
|
||||
func newLivenessRecoveryCollectors(reg prometheus.Registerer) *livenessRecoveryCollectors {
|
||||
if reg == nil {
|
||||
reg = prometheus.DefaultRegisterer
|
||||
}
|
||||
factory := promauto.With(reg)
|
||||
return &livenessRecoveryCollectors{
|
||||
eligibility: factory.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: livenessMetricEligibilityName,
|
||||
Help: "Private OpenAI liveness recovery eligibility decisions by execution path, provider health, commit state, and sanitized eligibility.",
|
||||
}, []string{"execution_path", "provider_health", "commit_state", "eligibility"}),
|
||||
results: factory.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: livenessMetricResultsName,
|
||||
Help: "Final OpenAI liveness recovery results by execution path, provider health, and recovery result.",
|
||||
}, []string{"execution_path", "provider_health", "recovery_result"}),
|
||||
}
|
||||
}
|
||||
|
||||
// defaultLivenessRecoveryCollectors is the process-global production collector
|
||||
// set. It is created exactly once here and shared by every default Server.
|
||||
var defaultLivenessRecoveryCollectors = newLivenessRecoveryCollectors(prometheus.DefaultRegisterer)
|
||||
|
||||
// livenessPhase is the bounded request-local cycle phase.
|
||||
type livenessPhase int
|
||||
|
||||
const (
|
||||
livenessPhaseIdle livenessPhase = iota
|
||||
livenessPhaseEligiblePending
|
||||
)
|
||||
|
||||
func (p livenessPhase) String() string {
|
||||
if p == livenessPhaseEligiblePending {
|
||||
return "eligible_pending"
|
||||
}
|
||||
return "idle"
|
||||
}
|
||||
|
||||
// openAILivenessObservationSink is the request-local wrapper around the
|
||||
// configured downstream observation sink. It retains only a mutex-protected
|
||||
// bounded phase plus the current cycle's closed classification values; no raw
|
||||
// identifiers are held. Every method is safe for concurrent use because
|
||||
// parallel filter evaluation can emit observations from multiple goroutines.
|
||||
type openAILivenessObservationSink struct {
|
||||
downstream streamgate.ObservationSink
|
||||
logger *zap.Logger
|
||||
suppressDefault bool
|
||||
collectors *livenessRecoveryCollectors
|
||||
|
||||
mu sync.Mutex
|
||||
phase livenessPhase
|
||||
cyclePath string
|
||||
cycleHealth string
|
||||
cycleCommit string
|
||||
cycleElig string
|
||||
recoverySeen bool
|
||||
resultDone bool
|
||||
}
|
||||
|
||||
var _ streamgate.ObservationSink = (*openAILivenessObservationSink)(nil)
|
||||
|
||||
// newOpenAILivenessObservationSink builds a fresh request-local wrapper. A nil
|
||||
// downstream defaults to NoopObservationSink; a nil logger defaults to a no-op
|
||||
// logger; a nil collector set defaults to the process-global production set.
|
||||
// suppressDefault is true only when downstream is the Server's constructor-owned
|
||||
// default generic zap sink.
|
||||
func newOpenAILivenessObservationSink(downstream streamgate.ObservationSink, logger *zap.Logger, suppressDefault bool, collectors *livenessRecoveryCollectors) *openAILivenessObservationSink {
|
||||
if downstream == nil {
|
||||
downstream = streamgate.NoopObservationSink{}
|
||||
}
|
||||
if logger == nil {
|
||||
logger = zap.NewNop()
|
||||
}
|
||||
if collectors == nil {
|
||||
collectors = defaultLivenessRecoveryCollectors
|
||||
}
|
||||
return &openAILivenessObservationSink{
|
||||
downstream: downstream,
|
||||
logger: logger,
|
||||
suppressDefault: suppressDefault,
|
||||
collectors: collectors,
|
||||
}
|
||||
}
|
||||
|
||||
// Emit projects the observation into the liveness metrics and, when this
|
||||
// wrapper owns the constructor-default generic sink, replaces the suppressed
|
||||
// high-cardinality generic log with the safe edge_liveness_recovery_observation
|
||||
// entry for consumed private-liveness/ExactReplay rows. Every other observation
|
||||
// is forwarded unchanged to the downstream sink. Metric projection always runs;
|
||||
// suppression and the safe log only apply on the default-sink path.
|
||||
func (s *openAILivenessObservationSink) Emit(ctx context.Context, obs streamgate.FilterObservation) error {
|
||||
s.mu.Lock()
|
||||
consumed, elig, result := s.project(obs)
|
||||
phase := s.phase.String()
|
||||
s.mu.Unlock()
|
||||
|
||||
if s.suppressDefault && consumed {
|
||||
s.writeSafeLog(phase, obs, elig, result)
|
||||
return nil
|
||||
}
|
||||
// Observation delivery is deliberately best-effort. A custom sink failure
|
||||
// must not feed back into filter arbitration or recovery ownership.
|
||||
_ = s.downstream.Emit(ctx, obs)
|
||||
return nil
|
||||
}
|
||||
|
||||
// project updates the request-local phase and records metrics for one
|
||||
// observation. It returns whether the observation belongs to the private
|
||||
// liveness cycle (and must be kept off the generic writer) plus the eligibility
|
||||
// and result recorded on this call (empty when none). It must be called with
|
||||
// s.mu held.
|
||||
func (s *openAILivenessObservationSink) project(obs streamgate.FilterObservation) (consumed bool, elig string, result string) {
|
||||
kind := obs.Kind()
|
||||
|
||||
if isLivenessFilterObservation(obs) {
|
||||
if kind != streamgate.ObservationKindFilterEvaluated {
|
||||
// filter_evaluation_started or any other private filter row: keep it
|
||||
// off the generic writer but record no metric.
|
||||
return true, "", ""
|
||||
}
|
||||
descriptor := livenessDescriptor(obs)
|
||||
eligibility, cycle := classifyLivenessEligibility(descriptor)
|
||||
if !cycle {
|
||||
// A provider error the liveness filter did not treat as a stall.
|
||||
return true, "", ""
|
||||
}
|
||||
if s.phase == livenessPhaseEligiblePending {
|
||||
// Deduplicate a second eligibility while a cycle is still open.
|
||||
return true, "", ""
|
||||
}
|
||||
s.cyclePath = classifyLivenessPath(obs.AttemptTarget().ExecutionPath())
|
||||
s.cycleHealth = classifyLivenessHealth(livenessProviderHealth(obs))
|
||||
s.cycleCommit = classifyLivenessCommit(obs.CommitState())
|
||||
s.cycleElig = eligibility
|
||||
s.recoverySeen = false
|
||||
s.resultDone = false
|
||||
s.collectors.eligibility.WithLabelValues(s.cyclePath, s.cycleHealth, s.cycleCommit, eligibility).Inc()
|
||||
if eligibility == livenessEligibilityEligible {
|
||||
s.phase = livenessPhaseEligiblePending
|
||||
return true, eligibility, ""
|
||||
}
|
||||
// Ineligible decisions finish immediately with a terminal result.
|
||||
return true, eligibility, s.recordResult(livenessResultTerminal)
|
||||
}
|
||||
|
||||
if s.phase != livenessPhaseEligiblePending {
|
||||
return false, "", ""
|
||||
}
|
||||
|
||||
switch kind {
|
||||
case streamgate.ObservationKindRecoveryPlanSelected:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
// A different recovery strategy won arbitration. This private
|
||||
// liveness cycle was not selected, but the unrelated lifecycle
|
||||
// observation must retain its normal downstream visibility.
|
||||
return false, "", s.recordResult(livenessResultNotSelected)
|
||||
}
|
||||
s.recoverySeen = true
|
||||
return true, "", ""
|
||||
case streamgate.ObservationKindRecoveryAttemptAborted,
|
||||
streamgate.ObservationKindRecoveryPrepared,
|
||||
streamgate.ObservationKindRecoveryRebuilt:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
return false, "", ""
|
||||
}
|
||||
s.recoverySeen = true
|
||||
return true, "", ""
|
||||
case streamgate.ObservationKindRecoveryDispatched:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
return false, "", ""
|
||||
}
|
||||
return true, "", s.recordResult(livenessResultRedispatched)
|
||||
case streamgate.ObservationKindRecoveryPlanRejected:
|
||||
// Core intentionally omits Recovery from plan_rejected observations;
|
||||
// while this private cycle is pending, the row is its final rejection.
|
||||
return true, "", s.recordResult(livenessResultPlanRejected)
|
||||
case streamgate.ObservationKindRecoveryAttemptAbortFailed:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
return false, "", ""
|
||||
}
|
||||
return true, "", s.recordResult(livenessResultAbortFailed)
|
||||
case streamgate.ObservationKindRecoveryRebuildFailed:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
return false, "", ""
|
||||
}
|
||||
return true, "", s.recordResult(livenessResultRebuildFailed)
|
||||
case streamgate.ObservationKindRecoveryDispatchFailed:
|
||||
if !isExactReplayRecovery(obs) {
|
||||
return false, "", ""
|
||||
}
|
||||
return true, "", s.recordResult(livenessResultDispatchFailed)
|
||||
case streamgate.ObservationKindTerminalCommitted:
|
||||
// The terminal itself stays on the generic writer; it only finalizes the
|
||||
// liveness result when recovery ended without an explicit lifecycle row.
|
||||
final := livenessResultTerminal
|
||||
if !s.recoverySeen {
|
||||
final = livenessResultNotSelected
|
||||
}
|
||||
return false, "", s.recordResult(final)
|
||||
}
|
||||
return false, "", ""
|
||||
}
|
||||
|
||||
// isExactReplayRecovery reports whether a lifecycle row belongs to the
|
||||
// private liveness strategy. Plan rejection is the one Core lifecycle row
|
||||
// without Recovery metadata and is handled explicitly in project.
|
||||
func isExactReplayRecovery(obs streamgate.FilterObservation) bool {
|
||||
recovery := obs.Recovery()
|
||||
return recovery != nil && recovery.Strategy() == streamgate.RecoveryStrategyExactReplay
|
||||
}
|
||||
|
||||
// recordResult increments the result counter once per cycle and resets the
|
||||
// phase so a later provider stall can open a new bounded cycle. It must be
|
||||
// called with s.mu held. The returned value is the recorded result, or "" when
|
||||
// a result was already recorded for this cycle.
|
||||
func (s *openAILivenessObservationSink) recordResult(result string) string {
|
||||
if s.resultDone {
|
||||
return ""
|
||||
}
|
||||
s.collectors.results.WithLabelValues(s.cyclePath, s.cycleHealth, result).Inc()
|
||||
s.resultDone = true
|
||||
s.phase = livenessPhaseIdle
|
||||
return result
|
||||
}
|
||||
|
||||
// writeSafeLog writes the bounded replacement for the suppressed generic log.
|
||||
// Only phase and the closed classification labels are recorded; no identifiers
|
||||
// or raw content are ever present. Values are recomputed from the observation
|
||||
// through closed maps so nothing high-cardinality can leak.
|
||||
func (s *openAILivenessObservationSink) writeSafeLog(phase string, obs streamgate.FilterObservation, elig, result string) {
|
||||
s.logger.Info(livenessLogMessage,
|
||||
zap.String("phase", phase),
|
||||
zap.String("execution_path", classifyLivenessPath(obs.AttemptTarget().ExecutionPath())),
|
||||
zap.String("provider_health", classifyLivenessHealth(livenessProviderHealth(obs))),
|
||||
zap.String("commit_state", classifyLivenessCommit(obs.CommitState())),
|
||||
zap.String("eligibility", elig),
|
||||
zap.String("recovery_result", result),
|
||||
)
|
||||
}
|
||||
|
||||
// isLivenessFilterObservation reports whether obs was attributed to the
|
||||
// predecessor-owned private liveness filter.
|
||||
func isLivenessFilterObservation(obs streamgate.FilterObservation) bool {
|
||||
attr := obs.Attribution()
|
||||
if attr == nil {
|
||||
return false
|
||||
}
|
||||
return attr.FilterID() == openAIStallRecoveryFilterID
|
||||
}
|
||||
|
||||
// livenessDescriptor returns the sanitized evidence descriptor code carried by a
|
||||
// private liveness filter_evaluated observation, or "" when absent.
|
||||
func livenessDescriptor(obs streamgate.FilterObservation) string {
|
||||
ev := obs.Evidence()
|
||||
if ev == nil {
|
||||
return ""
|
||||
}
|
||||
return ev.DescriptorCode()
|
||||
}
|
||||
|
||||
// livenessProviderHealth returns the raw provider-health signal carried by the
|
||||
// observation. The predecessor's private filter_evaluated observation does not
|
||||
// carry provider health (health lives only in the request-local recovery state
|
||||
// bridge, never in the immutable timeline), so this is currently always empty
|
||||
// and classifyLivenessHealth resolves it to unknown. The seam is kept so a
|
||||
// future health-bearing observation maps without a projection change.
|
||||
func livenessProviderHealth(_ streamgate.FilterObservation) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// classifyLivenessEligibility maps a sanitized descriptor to a closed
|
||||
// eligibility value and reports whether the descriptor opens a liveness cycle.
|
||||
// provider_error_ignored (and any unrecognized descriptor that is not a stall)
|
||||
// does not open a cycle.
|
||||
func classifyLivenessEligibility(descriptor string) (eligibility string, cycle bool) {
|
||||
switch descriptor {
|
||||
case livenessDescriptorConfirmed:
|
||||
return livenessEligibilityEligible, true
|
||||
case livenessDescriptorUnconfirmed:
|
||||
return livenessEligibilityUnconfirmedFence, true
|
||||
case livenessDescriptorIneligible:
|
||||
// The predecessor collapses post-commit, tool-side-effect, caller-cancel,
|
||||
// and missing-request-ref into one ineligible descriptor, so the exact
|
||||
// reason is not recoverable from the immutable timeline.
|
||||
return livenessEligibilityOther, true
|
||||
case livenessDescriptorProviderIgnored, "":
|
||||
return "", false
|
||||
default:
|
||||
return livenessEligibilityOther, true
|
||||
}
|
||||
}
|
||||
|
||||
// classifyLivenessPath maps an execution path to the closed path vocabulary.
|
||||
func classifyLivenessPath(path string) string {
|
||||
switch path {
|
||||
case livenessPathNormalized:
|
||||
return livenessPathNormalized
|
||||
case livenessPathProviderTunnel:
|
||||
return livenessPathProviderTunnel
|
||||
default:
|
||||
return livenessPathUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// classifyLivenessHealth maps a raw provider-health signal to the closed health
|
||||
// vocabulary with an unknown fallback.
|
||||
func classifyLivenessHealth(health string) string {
|
||||
switch health {
|
||||
case livenessHealthAvailable:
|
||||
return livenessHealthAvailable
|
||||
case livenessHealthUnavailable:
|
||||
return livenessHealthUnavailable
|
||||
default:
|
||||
return livenessHealthUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// classifyLivenessCommit maps Core's commit state to the closed commit
|
||||
// vocabulary with an unknown fallback.
|
||||
func classifyLivenessCommit(cs streamgate.CommitState) string {
|
||||
switch cs {
|
||||
case streamgate.CommitStateTransportUncommitted:
|
||||
return livenessCommitUncommitted
|
||||
case streamgate.CommitStateStreamOpen:
|
||||
return livenessCommitStreamOpen
|
||||
case streamgate.CommitStateTerminalCommitted:
|
||||
return livenessCommitTerminal
|
||||
default:
|
||||
return livenessCommitUnknown
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -659,7 +659,7 @@ func TestStreamGateEnabledToolValidationHasSingleRecoveryOwner(t *testing.T) {
|
|||
|
||||
t.Run("semantic filters disabled still use the core budget", func(t *testing.T) {
|
||||
// The semantic filter switch does not select the request runtime. The
|
||||
// always-on Core liveness owner therefore applies the same zero budget.
|
||||
// request-local Core therefore applies the same zero recovery budget.
|
||||
srv, fake := streamGateToolValidationServer(t, false, 0, invalidToolCallRun(), validToolCallRun())
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(streamGateToolValidationBody)))
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ func (s *openAIResponsesEventSource) NextEvent(ctx context.Context) (streamgate.
|
|||
text, reasoning, _, toolCalls, usage, _, err := collectRunResult(ctx, s.handle.Stream(), s.handle.WaitTimeout())
|
||||
if err != nil {
|
||||
s.holder.store(openAIResponsesAttemptResult{dispatch: s.handle.Dispatch(), collectErr: err})
|
||||
return newOpenAIProviderErrorEventFromFailure(openAIExecutionFailureFromError(err), streamGateErrorRunFailed)
|
||||
return newOpenAIProviderErrorEvent(streamGateErrorRunFailed)
|
||||
}
|
||||
text, reasoning, _ = normalizeCompletionOutput(s.dc.outputPolicy, text, reasoning, false)
|
||||
result := openAIResponsesAttemptResult{text: text, reasoning: reasoning, toolCalls: toolCalls, usage: usage, dispatch: s.handle.Dispatch()}
|
||||
|
|
@ -1180,11 +1180,7 @@ func (s *Server) buildOpenAIResponsesStreamGateRuntime(dc *responsesDispatchCont
|
|||
// which both normalized and tunnel replacement requests are derived, rather
|
||||
// than retaining caller-derived Run/PrepareRun state from a generic tunnel
|
||||
// runtime.
|
||||
func (s *Server) buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc *responsesDispatchContext, initial openAIAttemptTransport, dispatch edgeservice.RunDispatch, closeInitial func(), sink openAIStreamGateSink, registry streamgate.FilterRegistrySnapshot, stallStates ...*openAIStallRecoveryState) (*streamgate.RequestRuntime, *openAIStreamGateUsageHolder, error) {
|
||||
var stallState *openAIStallRecoveryState
|
||||
if len(stallStates) > 0 {
|
||||
stallState = stallStates[0]
|
||||
}
|
||||
func (s *Server) buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc *responsesDispatchContext, initial openAIAttemptTransport, dispatch edgeservice.RunDispatch, closeInitial func(), sink openAIStreamGateSink, registry streamgate.FilterRegistrySnapshot) (*streamgate.RequestRuntime, *openAIStreamGateUsageHolder, error) {
|
||||
holderSink, ok := sink.(*openAIResponsesReleaseSink)
|
||||
if !ok {
|
||||
if composite, compositeOK := sink.(*openAICompositeReleaseSink); compositeOK {
|
||||
|
|
@ -1267,7 +1263,7 @@ func (s *Server) buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc *responsesD
|
|||
}
|
||||
return newOpenAIRecoverySourceEventSource(src, recoverySource), nil
|
||||
}
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), newOpenAIResponsesRecoveryAdmissionBuilder(s, dc, state), factory, dc.usage, stallState, sink)
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), newOpenAIResponsesRecoveryAdmissionBuilder(s, dc, state), factory, dc.usage, sink)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
@ -1280,7 +1276,6 @@ func (s *Server) buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc *responsesD
|
|||
controller := &openAIAttemptController{
|
||||
service: s.service, dispatch: dispatch, closeTransport: closeInitial,
|
||||
usageRecorder: dc.usage, usageBinding: initial.usageBinding, usage: initial.usage,
|
||||
stall: stallState,
|
||||
compatibilitySink: sink,
|
||||
}
|
||||
binding, err := streamgate.NewAttemptBinding(
|
||||
|
|
@ -1380,21 +1375,14 @@ func (s *Server) runOpenAIResponsesStreamGateAttempt(w http.ResponseWriter, dc *
|
|||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
stallState, stallRegistration, err := openAIStallRecoveryRegistration(fctx)
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
closeInitial()
|
||||
dc.finishUsageRequest(usageStatusError, openAIAttemptResponseMode(initial.path))
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx, stallRegistration)
|
||||
if err != nil {
|
||||
closeInitial()
|
||||
dc.finishUsageRequest(usageStatusError, openAIAttemptResponseMode(initial.path))
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
runtime, _, err := s.buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc, initial, dispatch, closeInitial, sink, registry, stallState)
|
||||
runtime, _, err := s.buildOpenAIResponsesStreamGateRuntimeFromAttempt(dc, initial, dispatch, closeInitial, sink, registry)
|
||||
if err != nil {
|
||||
closeInitial()
|
||||
dc.finishUsageRequest(usageStatusError, openAIAttemptResponseMode(initial.path))
|
||||
|
|
|
|||
|
|
@ -74,8 +74,6 @@ type Server struct {
|
|||
logger *zap.Logger
|
||||
server *http.Server
|
||||
obsSink streamgate.ObservationSink
|
||||
obsSinkIsDefault bool
|
||||
livenessCollectors *livenessRecoveryCollectors
|
||||
principalProjection authprojection.Reader
|
||||
credentialMode credentialMode
|
||||
executionPresets []config.ExecutionPreset
|
||||
|
|
@ -121,8 +119,6 @@ func NewServer(cfg config.EdgeOpenAIConf, svc runService, logger *zap.Logger) *S
|
|||
service: svc,
|
||||
logger: logger,
|
||||
obsSink: newZapFilterObservationSink(logger),
|
||||
obsSinkIsDefault: true,
|
||||
livenessCollectors: defaultLivenessRecoveryCollectors,
|
||||
requestCoordinator: newLogicalRequestCoordinator(logicalRequestCoordinatorOptions{}),
|
||||
artifactFrontiers: newArtifactFrontierStore(defaultArtifactFrontierCapacity),
|
||||
lightFlows: newHotPathLightStore(defaultHotPathLightCapacity),
|
||||
|
|
@ -307,10 +303,7 @@ func (s *Server) emitHotPathObservation(ctx context.Context, projection hotPathL
|
|||
// SetObservationSink replaces the default observation sink used to emit
|
||||
// streamgate_filter_observation entries for this server's request runtimes.
|
||||
// A nil sink installs a NoopObservationSink so observation failures can never
|
||||
// alter response behavior. Every call transfers ownership to the application:
|
||||
// the constructor-owned-default flag is cleared so the request-local liveness
|
||||
// projection never suppresses forwarding to an explicitly installed sink, even
|
||||
// when that sink is another *zapFilterObservationSink of the built-in type.
|
||||
// alter response behavior.
|
||||
func (s *Server) SetObservationSink(sink streamgate.ObservationSink) {
|
||||
s.mu.Lock()
|
||||
if sink == nil {
|
||||
|
|
@ -318,26 +311,18 @@ func (s *Server) SetObservationSink(sink streamgate.ObservationSink) {
|
|||
} else {
|
||||
s.obsSink = sink
|
||||
}
|
||||
s.obsSinkIsDefault = false
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// observationSink returns a fresh request-local liveness observation projection
|
||||
// wrapping the configured downstream sink. The wrapper only suppresses the
|
||||
// private-liveness/ExactReplay rows from the generic writer when the downstream
|
||||
// is this server's constructor-owned default sink; every explicitly installed
|
||||
// sink receives the original immutable observations.
|
||||
// observationSink returns the configured observation sink.
|
||||
func (s *Server) observationSink() streamgate.ObservationSink {
|
||||
s.mu.RLock()
|
||||
downstream := s.obsSink
|
||||
logger := s.logger
|
||||
suppressDefault := s.obsSinkIsDefault
|
||||
collectors := s.livenessCollectors
|
||||
s.mu.RUnlock()
|
||||
if downstream == nil {
|
||||
downstream = streamgate.NoopObservationSink{}
|
||||
}
|
||||
return newOpenAILivenessObservationSink(downstream, logger, suppressDefault, collectors)
|
||||
return downstream
|
||||
}
|
||||
|
||||
// SetLongContextThreshold sets the input-token threshold at or above which a
|
||||
|
|
|
|||
|
|
@ -176,11 +176,11 @@ func TestSingleRequestProviderStageUsesFrozenOptionsAndDispatch(t *testing.T) {
|
|||
}
|
||||
|
||||
// Verify exact dispatch parameters captured in ProviderPoolDispatchRequest
|
||||
if captured.Run.NodeRef != "node" || captured.Run.ModelGroupKey != "plan-model" || captured.Run.ProviderID != "gemini" || captured.Run.UsageAttribution != "principal" || captured.Run.SessionID != "session" || captured.Run.TimeoutSec != 60 || captured.Run.MaxQueue != 10 || captured.Run.QueueTimeoutMS != 5000 || !captured.Run.ProviderPool || captured.Run.RunID != "" || captured.Run.Adapter != "" || captured.Run.Target != "" || captured.Run.Prompt != "" || captured.Run.Input != nil || captured.Run.Background != false || captured.Run.Metadata != nil || captured.Run.EstimatedInputTokens != 0 || captured.Run.ContextClass != "" || captured.Run.ResponseStallTimeoutMS != 0 {
|
||||
if captured.Run.NodeRef != "node" || captured.Run.ModelGroupKey != "plan-model" || captured.Run.ProviderID != "gemini" || captured.Run.UsageAttribution != "principal" || captured.Run.SessionID != "session" || captured.Run.TimeoutSec != 60 || captured.Run.MaxQueue != 10 || captured.Run.QueueTimeoutMS != 5000 || !captured.Run.ProviderPool || captured.Run.RunID != "" || captured.Run.Adapter != "" || captured.Run.Target != "" || captured.Run.Prompt != "" || captured.Run.Input != nil || captured.Run.Background != false || captured.Run.Metadata != nil || captured.Run.EstimatedInputTokens != 0 || captured.Run.ContextClass != "" {
|
||||
t.Fatalf("unexpected Run dispatch: %+v", captured.Run)
|
||||
}
|
||||
|
||||
if captured.Tunnel.NodeRef != "node" || captured.Tunnel.ModelGroupKey != "plan-model" || captured.Tunnel.ProviderID != "gemini" || captured.Tunnel.UsageAttribution != "principal" || captured.Tunnel.Adapter != "openai_compat" || captured.Tunnel.Target != "gemini-3.6-flash" || captured.Tunnel.SessionID != "session" || captured.Tunnel.Method != http.MethodPost || captured.Tunnel.Path != "/v1/chat/completions" || captured.Tunnel.Operation != "" || captured.Tunnel.Stream || captured.Tunnel.TimeoutSec != 60 || captured.Tunnel.MaxQueue != 10 || captured.Tunnel.QueueTimeoutMS != 5000 || !captured.Tunnel.ProviderPool || captured.Tunnel.RunID != "" || captured.Tunnel.Headers != nil || captured.Tunnel.Body != nil || captured.Tunnel.Metadata != nil || captured.Tunnel.EstimatedInputTokens != 0 || captured.Tunnel.ContextClass != "" || captured.Tunnel.ResponseStallTimeoutMS != 0 {
|
||||
if captured.Tunnel.NodeRef != "node" || captured.Tunnel.ModelGroupKey != "plan-model" || captured.Tunnel.ProviderID != "gemini" || captured.Tunnel.UsageAttribution != "principal" || captured.Tunnel.Adapter != "openai_compat" || captured.Tunnel.Target != "gemini-3.6-flash" || captured.Tunnel.SessionID != "session" || captured.Tunnel.Method != http.MethodPost || captured.Tunnel.Path != "/v1/chat/completions" || captured.Tunnel.Operation != "" || captured.Tunnel.Stream || captured.Tunnel.TimeoutSec != 60 || captured.Tunnel.MaxQueue != 10 || captured.Tunnel.QueueTimeoutMS != 5000 || !captured.Tunnel.ProviderPool || captured.Tunnel.RunID != "" || captured.Tunnel.Headers != nil || captured.Tunnel.Body != nil || captured.Tunnel.Metadata != nil || captured.Tunnel.EstimatedInputTokens != 0 || captured.Tunnel.ContextClass != "" {
|
||||
t.Fatalf("unexpected Tunnel request: %+v", captured.Tunnel)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -138,7 +138,6 @@ type openAIAttemptDispatcher struct {
|
|||
eventSource openAIAttemptEventSourceFactory
|
||||
state *openAIRecoveryAdmissionState
|
||||
usage *openAIUsageRecorder
|
||||
stall *openAIStallRecoveryState
|
||||
sink openAIStreamGateSink
|
||||
holder *openAIBufferedResultHolder
|
||||
}
|
||||
|
|
@ -161,8 +160,6 @@ func newOpenAIAttemptDispatcher(
|
|||
switch value := arg.(type) {
|
||||
case *openAIUsageRecorder:
|
||||
dispatcher.usage = value
|
||||
case *openAIStallRecoveryState:
|
||||
dispatcher.stall = value
|
||||
case openAIStreamGateSink:
|
||||
dispatcher.sink = value
|
||||
case *openAIBufferedResultHolder:
|
||||
|
|
@ -205,13 +202,6 @@ func (d *openAIAttemptDispatcher) DispatchAttempt(ctx context.Context, request s
|
|||
if err := admission.validate(); err != nil {
|
||||
return streamgate.AttemptBinding{}, err
|
||||
}
|
||||
if admission.kind == openAIAdmissionPool {
|
||||
if providerID, allowFallback, ok := d.stall.consumeAdmission(); ok {
|
||||
admission.pool.AvoidProviderID = providerID
|
||||
admission.pool.AllowAvoidedProviderFallback = allowFallback
|
||||
}
|
||||
}
|
||||
|
||||
transport, dispatch, closeTransport, err := d.dispatch(ctx, admission)
|
||||
if err != nil {
|
||||
d.state.record(err)
|
||||
|
|
@ -236,7 +226,6 @@ func (d *openAIAttemptDispatcher) DispatchAttempt(ctx context.Context, request s
|
|||
usageRecorder: d.usage,
|
||||
usageBinding: transport.usageBinding,
|
||||
usage: transport.usage,
|
||||
stall: d.stall,
|
||||
compatibilitySink: d.sink,
|
||||
}
|
||||
abortDispatched := func() {
|
||||
|
|
@ -354,8 +343,7 @@ func openAIAttemptBindingProvider(dispatch edgeservice.RunDispatch) string {
|
|||
return provider
|
||||
}
|
||||
// Core requires a non-empty attempt binding even for legacy direct routes
|
||||
// that predate stable provider ids. The liveness handoff never admits this
|
||||
// sentinel as a recovery candidate.
|
||||
// that predate stable provider ids.
|
||||
return openAIUnspecifiedProviderID
|
||||
}
|
||||
|
||||
|
|
@ -379,7 +367,6 @@ type openAIAttemptController struct {
|
|||
usageRecorder *openAIUsageRecorder
|
||||
usageBinding usageDispatchBinding
|
||||
usage *openAIAttemptUsage
|
||||
stall *openAIStallRecoveryState
|
||||
compatibilitySink openAIStreamGateSink
|
||||
}
|
||||
|
||||
|
|
@ -420,13 +407,9 @@ func (c *openAIAttemptController) AbortAttempt(ctx context.Context) error {
|
|||
}
|
||||
c.recordUsage()
|
||||
|
||||
// A typed response_stalled terminal with an Edge-confirmed local fence has
|
||||
// already closed Node ownership. Preserve that authority by closing only the
|
||||
// request-local transport; all other recoveries retain CancelRun behavior.
|
||||
confirmedTerminal := c.stall.claimConfirmedClose(openAIStreamGateSafeToken("attempt", c.dispatch.RunID))
|
||||
compatibilityTerminal := openAICompatibilityProviderTerminal(c.compatibilitySink)
|
||||
var cancelErr error
|
||||
if !confirmedTerminal && !compatibilityTerminal && c.dispatch.RunID != "" {
|
||||
if !compatibilityTerminal && c.dispatch.RunID != "" {
|
||||
_, cancelErr = c.service.CancelRun(ctx, edgeservice.CancelRunRequest{
|
||||
NodeRef: c.dispatch.NodeID, RunID: c.dispatch.RunID,
|
||||
})
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import (
|
|||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
"iop/packages/go/streamgate"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
type dispatcherEventSource struct{}
|
||||
|
|
@ -59,7 +60,6 @@ type dispatcherServiceSpy struct {
|
|||
cancelCalls int
|
||||
closeCalls int
|
||||
lastHeaders map[string]string
|
||||
lastPool edgeservice.ProviderPoolDispatchRequest
|
||||
}
|
||||
|
||||
func (s *dispatcherServiceSpy) dispatch(path string) edgeservice.RunDispatch {
|
||||
|
|
@ -83,7 +83,6 @@ func (s *dispatcherServiceSpy) SubmitProviderTunnel(_ context.Context, request e
|
|||
|
||||
func (s *dispatcherServiceSpy) SubmitProviderPool(_ context.Context, request edgeservice.ProviderPoolDispatchRequest) (*edgeservice.ProviderPoolDispatchResult, error) {
|
||||
s.poolCalls++
|
||||
s.lastPool = request
|
||||
if s.poolPath == "provider_tunnel" {
|
||||
tunnel := request.Tunnel
|
||||
var err error
|
||||
|
|
@ -185,21 +184,29 @@ func TestOpenAIAttemptDispatcherExistingAdmissionSurfaces(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestOpenAIAttemptControllerConfirmedStall(t *testing.T) {
|
||||
func TestOpenAIAttemptControllerTerminalOwnershipExactlyOnce(t *testing.T) {
|
||||
service := &dispatcherServiceSpy{}
|
||||
state := &openAIStallRecoveryState{}
|
||||
state.arm("attempt.attempt-normalized", "provider.actual", "available")
|
||||
controller := &openAIAttemptController{
|
||||
terminal := &openAIChatLiveTerminalState{}
|
||||
terminal.setProviderTerminal()
|
||||
providerTerminal := &openAIAttemptController{
|
||||
service: service,
|
||||
dispatch: service.dispatch("normalized"),
|
||||
closeTransport: func() { service.closeCalls++ },
|
||||
stall: state,
|
||||
compatibilitySink: &openAIChatSSEReleaseSink{
|
||||
liveTerminal: terminal,
|
||||
},
|
||||
}
|
||||
if err := controller.AbortAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("confirmed AbortAttempt: %v", err)
|
||||
if err := providerTerminal.AbortAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("provider terminal AbortAttempt: %v", err)
|
||||
}
|
||||
if err := providerTerminal.AbortAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("duplicate provider terminal AbortAttempt: %v", err)
|
||||
}
|
||||
if err := providerTerminal.CloseAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("provider terminal CloseAttempt after abort: %v", err)
|
||||
}
|
||||
if service.cancelCalls != 0 || service.closeCalls != 1 {
|
||||
t.Fatalf("confirmed terminal cancel/close = %d/%d, want 0/1", service.cancelCalls, service.closeCalls)
|
||||
t.Fatalf("provider terminal cancel/close = %d/%d, want 0/1", service.cancelCalls, service.closeCalls)
|
||||
}
|
||||
|
||||
ordinary := &openAIAttemptController{
|
||||
|
|
@ -210,33 +217,38 @@ func TestOpenAIAttemptControllerConfirmedStall(t *testing.T) {
|
|||
if err := ordinary.AbortAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("ordinary AbortAttempt: %v", err)
|
||||
}
|
||||
if err := ordinary.AbortAttempt(context.Background()); err != nil {
|
||||
t.Fatalf("duplicate ordinary AbortAttempt: %v", err)
|
||||
}
|
||||
if service.cancelCalls != 1 || service.closeCalls != 2 {
|
||||
t.Fatalf("ordinary recovery cancel/close = %d/%d, want 1/2", service.cancelCalls, service.closeCalls)
|
||||
t.Fatalf("ordinary abort cancel/close = %d/%d, want 1/2", service.cancelCalls, service.closeCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIAttemptDispatcherStalledProvider(t *testing.T) {
|
||||
service := &dispatcherServiceSpy{poolPath: "normalized"}
|
||||
rebuilder, ref, dispatcher := newDispatcherFixture(t, service, func(_ context.Context, _ streamgate.RebuiltRequest, body []byte) (openAIAttemptAdmission, error) {
|
||||
return openAIAttemptAdmission{kind: openAIAdmissionPool, pool: edgeservice.ProviderPoolDispatchRequest{
|
||||
Run: edgeservice.SubmitRunRequest{ModelGroupKey: "alias", ProviderPool: true},
|
||||
Tunnel: edgeservice.SubmitProviderTunnelRequest{Path: openAIRebuildEndpointChat, Body: body},
|
||||
}}, nil
|
||||
})
|
||||
state := &openAIStallRecoveryState{}
|
||||
state.arm("attempt.old", "provider.stalled", "available")
|
||||
if !state.claimConfirmedClose("attempt.old") {
|
||||
t.Fatal("failed to arm confirmed close")
|
||||
func TestOpenAIRunEventSourceTreatsTypedFailureAsGeneralTerminal(t *testing.T) {
|
||||
failure := &iop.ExecutionFailure{
|
||||
Code: "deprecated_typed_failure",
|
||||
Retryable: true,
|
||||
Metadata: map[string]string{"provider_health": "unavailable"},
|
||||
}
|
||||
dispatcher.stall = state
|
||||
request := rebuiltRequestForDispatcher(t, rebuilder, ref, "plan.stalled-provider")
|
||||
binding, err := dispatcher.DispatchAttempt(context.Background(), request)
|
||||
if err != nil {
|
||||
t.Fatalf("dispatch recovery: %v", err)
|
||||
source := newOpenAIRunEventSource(
|
||||
edgeservice.RunStream{Events: bufferedRunEvents(&iop.RunEvent{Type: "error", Failure: failure})},
|
||||
time.Second,
|
||||
nil,
|
||||
)
|
||||
if event, err := source.NextEvent(t.Context()); err != nil || event.Kind() != streamgate.EventKindResponseStart {
|
||||
t.Fatalf("response start = (%s, %v)", event.Kind(), err)
|
||||
}
|
||||
defer binding.Controller().AbortAttempt(context.Background())
|
||||
if service.lastPool.AvoidProviderID != "provider.stalled" || service.lastPool.AllowAvoidedProviderFallback {
|
||||
t.Fatalf("recovery pool hints = %#v", service.lastPool)
|
||||
event, err := source.NextEvent(t.Context())
|
||||
if err != nil || event.Kind() != streamgate.EventKindProviderError {
|
||||
t.Fatalf("terminal = (%s, %v)", event.Kind(), err)
|
||||
}
|
||||
terminal, err := event.AsProviderError()
|
||||
if err != nil || terminal.ExternalDesc() == nil || terminal.ExternalDesc().Code() != streamGateErrorRunFailed {
|
||||
t.Fatalf("provider terminal = (%v, %v)", terminal.ExternalDesc(), err)
|
||||
}
|
||||
if failure.GetMetadata()["provider_health"] != "unavailable" || len(failure.GetMetadata()) != 1 {
|
||||
t.Fatalf("typed failure metadata mutated: %#v", failure.GetMetadata())
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -694,111 +694,6 @@ func batchHasProviderError(batch streamgate.EvidenceBatch) bool {
|
|||
return false
|
||||
}
|
||||
|
||||
const (
|
||||
openAIStallRecoveryFilterID = "openai.response_stalled"
|
||||
openAIStallRecoveryFilterRuleID = "response_stalled_exact_replay"
|
||||
openAIStallRecoveryConsumerID = "openai.liveness"
|
||||
openAIStallRecoveryPriority = 100
|
||||
)
|
||||
|
||||
// openAIStallRecoveryFilter is an internal, always-present liveness owner for
|
||||
// supported OpenAI ingress. It is deliberately outside configurable semantic
|
||||
// filter policy and provider capability admission.
|
||||
type openAIStallRecoveryFilter struct {
|
||||
streamgate.FilterBase
|
||||
requestRef string
|
||||
state *openAIStallRecoveryState
|
||||
}
|
||||
|
||||
func newOpenAIStallRecoveryFilter(requestRef string, state *openAIStallRecoveryState) (*openAIStallRecoveryFilter, error) {
|
||||
base, err := streamgate.NewFilterBase(openAIStallRecoveryFilterID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &openAIStallRecoveryFilter{FilterBase: base, requestRef: requestRef, state: state}, nil
|
||||
}
|
||||
|
||||
func (f *openAIStallRecoveryFilter) Applies(streamgate.FilterContext) bool { return true }
|
||||
|
||||
func (f *openAIStallRecoveryFilter) HoldRequirement(streamgate.FilterContext) streamgate.FilterHoldRequirement {
|
||||
req, _ := streamgate.NewFilterHoldRequirementNone(
|
||||
streamGateChannelDefault, []streamgate.EventKind{streamgate.EventKindProviderError},
|
||||
)
|
||||
return req
|
||||
}
|
||||
|
||||
func (f *openAIStallRecoveryFilter) Evaluate(_ context.Context, fctx streamgate.FilterContext, batch streamgate.EvidenceBatch) (streamgate.FilterDecision, error) {
|
||||
descriptor := "provider_error_ignored"
|
||||
var health string
|
||||
for _, event := range batch.Events() {
|
||||
if event.Kind() != streamgate.EventKindProviderError {
|
||||
continue
|
||||
}
|
||||
terminal, err := event.AsProviderError()
|
||||
external := terminal.ExternalDesc()
|
||||
if err != nil || external == nil || external.Code() != openAIStallFailureCode {
|
||||
continue
|
||||
}
|
||||
confirmed := false
|
||||
for _, cause := range terminal.FailureCauses().All() {
|
||||
if cause.Stage() == openAIStallHandoffStage && cause.Code() == openAIStallHandoffCause {
|
||||
confirmed = true
|
||||
}
|
||||
if cause.Stage() == openAIStallHealthStage {
|
||||
health = cause.Code()
|
||||
}
|
||||
}
|
||||
if confirmed && (health == "available" || health == "unavailable" || health == "unknown") {
|
||||
descriptor = "response_stalled_confirmed"
|
||||
break
|
||||
}
|
||||
descriptor = "response_stalled_unconfirmed"
|
||||
}
|
||||
|
||||
decisionKind := streamgate.FilterDecisionKindPass
|
||||
var intent *streamgate.RecoveryIntent
|
||||
if descriptor == "response_stalled_confirmed" {
|
||||
unsafe := fctx.CommitState() != streamgate.CommitStateTransportUncommitted || fctx.HasToolSideEffect() || f.requestRef == "" || batchHasToolEvidence(batch)
|
||||
if unsafe || !f.state.claimRecovery() {
|
||||
descriptor = "response_stalled_ineligible"
|
||||
} else {
|
||||
directive, err := streamgate.NewRecoveryDirectiveExact(f.requestRef)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
createdIntent, err := streamgate.NewRecoveryIntent(streamgate.RecoveryStrategyExactReplay, directive, openAIStallFailureCode, openAIStallRecoveryPriority)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
intent = &createdIntent
|
||||
f.state.arm(fctx.AttemptID(), fctx.ActualProvider(), health)
|
||||
decisionKind = streamgate.FilterDecisionKindViolation
|
||||
}
|
||||
}
|
||||
ts := batch.CapturedAt()
|
||||
if ts.IsZero() {
|
||||
ts = time.Now()
|
||||
}
|
||||
evidence, err := streamgate.NewSanitizedEvidence(streamgate.EventKindProviderError, streamGateChannelDefault, openAIStallRecoveryFilterRuleID, descriptor, openAIOutputFilterFingerprint(openAIStallRecoveryFilterRuleID, descriptor), 1, 0, streamgate.FilterOutcomeKindEvaluated, ts)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
return streamgate.NewFilterDecision(decisionKind, openAIStallRecoveryConsumerID, f.ID(), openAIStallRecoveryFilterRuleID, evidence, intent)
|
||||
}
|
||||
|
||||
func batchHasToolEvidence(batch streamgate.EvidenceBatch) bool {
|
||||
for _, events := range [][]streamgate.NormalizedEvent{batch.Events(), batch.ChannelPending()[streamGateChannelDefault], batch.CommittedLookBehind()[streamGateChannelDefault]} {
|
||||
for _, event := range events {
|
||||
if event.Kind() == streamgate.EventKindToolCallFragment {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var _ streamgate.Filter = (*openAIStallRecoveryFilter)(nil)
|
||||
|
||||
// openAIOutputFilterFingerprint derives a stable, raw-free fingerprint from the
|
||||
// rule id and a sanitized descriptor so evidence carries no provider text.
|
||||
func openAIOutputFilterFingerprint(ruleID, descriptor string) streamgate.FixedFingerprint {
|
||||
|
|
|
|||
|
|
@ -35,75 +35,6 @@ type openAIIngressSnapshot struct {
|
|||
closed bool
|
||||
}
|
||||
|
||||
// openAIStallRecoveryState is the narrow request-local bridge between the
|
||||
// private liveness filter and recovery dispatch. It retains only Edge-owned
|
||||
// provider identity plus the allowlisted probe classification; no provider
|
||||
// error text, request body, or arbitrary failure metadata enters this state.
|
||||
type openAIStallRecoveryState struct {
|
||||
mu sync.Mutex
|
||||
attemptID string
|
||||
providerID string
|
||||
health string
|
||||
confirmedForClose bool
|
||||
recoveryClaimed bool
|
||||
}
|
||||
|
||||
func (s *openAIStallRecoveryState) claimRecovery() bool {
|
||||
if s == nil {
|
||||
return false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.recoveryClaimed {
|
||||
return false
|
||||
}
|
||||
s.recoveryClaimed = true
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *openAIStallRecoveryState) arm(attemptID, providerID, health string) {
|
||||
if s == nil || attemptID == "" || providerID == "" || providerID == openAIUnspecifiedProviderID {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.attemptID = attemptID
|
||||
s.providerID = providerID
|
||||
s.health = health
|
||||
s.confirmedForClose = false
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAIStallRecoveryState) claimConfirmedClose(attemptID string) bool {
|
||||
if s == nil {
|
||||
return false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.attemptID != attemptID || s.providerID == "" || s.confirmedForClose {
|
||||
return false
|
||||
}
|
||||
s.confirmedForClose = true
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *openAIStallRecoveryState) consumeAdmission() (providerID string, allowFallback bool, ok bool) {
|
||||
if s == nil {
|
||||
return "", false, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if !s.confirmedForClose || s.providerID == "" {
|
||||
return "", false, false
|
||||
}
|
||||
// Health-probe availability proves only that the endpoint is reachable. It
|
||||
// never makes the exact request that just stalled safe to replay on the same
|
||||
// provider.
|
||||
providerID, allowFallback = s.providerID, false
|
||||
s.attemptID, s.providerID, s.health = "", "", ""
|
||||
s.confirmedForClose = false
|
||||
return providerID, allowFallback, true
|
||||
}
|
||||
|
||||
// readOpenAIIngressBody installs the HTTP body limit before reading. The
|
||||
// standard library reader performs a limit+1 probe internally, so an exact
|
||||
// limit body succeeds and the first excess byte is reported as overflow.
|
||||
|
|
|
|||
|
|
@ -939,74 +939,6 @@ func TestOpenAITunnelHTTPErrorLifecycle(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestOpenAIDirectResponsesSemanticDisabledStallTerminal drives the generic
|
||||
// direct Responses tunnel runtime (not the provider-pool Responses wrapper).
|
||||
// Once raw Responses SSE is visible, a typed stall must discard a staged
|
||||
// response.completed and append exactly one native error plus [DONE].
|
||||
func TestOpenAIDirectResponsesSemanticDisabledStallTerminal(t *testing.T) {
|
||||
srv := NewServer(config.EdgeOpenAIConf{TimeoutSec: 5}, &providerFakeRunService{}, nil)
|
||||
rawBody := []byte(`{"model":"client-model","stream":true,"input":"hi"}`)
|
||||
requestCtx := newTestRequestContext(t, routeDispatch{Adapter: "openai-compat", Target: "served-model", TimeoutSec: 5}, rawBody)
|
||||
req := openAITunnelStreamGateRequest{
|
||||
route: routeDispatch{Adapter: "openai-compat", Target: "served-model", TimeoutSec: 5},
|
||||
ingress: requestCtx.ingress, endpoint: openAIRebuildEndpointResponses,
|
||||
method: http.MethodPost, path: "/v1/responses", operation: string(config.OperationResponses),
|
||||
stream: true, modelGroupKey: "client-model", semanticSet: true, semanticEnabled: false,
|
||||
authorize: func(context.Context) (map[string]string, error) { return nil, nil },
|
||||
rewriteBody: func(body []byte, _ string) ([]byte, error) { return body, nil },
|
||||
}
|
||||
frames := bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte(
|
||||
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp-direct\",\"object\":\"response\",\"status\":\"in_progress\"},\"sequence_number\":1}\n\n" +
|
||||
"data: {\"type\":\"response.output_text.delta\",\"item_id\":\"msg-direct\",\"output_index\":0,\"content_index\":0,\"delta\":\"direct-prefix\",\"sequence_number\":2}\n\n" +
|
||||
"data: {\"type\":\"response.completed\",\"response\":{\"id\":\"raw-provider-terminal-must-stay-hidden\",\"object\":\"response\",\"status\":\"completed\"},\"sequence_number\":3}\n\n",
|
||||
)},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure("available")},
|
||||
)
|
||||
closeCount := 0
|
||||
handle := &countingDirectTunnelHandle{
|
||||
fakeTunnelHandle: fakeTunnelHandle{dispatch: edgeservice.RunDispatch{
|
||||
RunID: "direct-responses-stall", ModelGroupKey: "client-model", Adapter: "openai-compat",
|
||||
Target: "served-model", ProviderID: "provider-a", ExecutionPath: string(edgeservice.ProviderPoolPathTunnel),
|
||||
}, frames: frames},
|
||||
closed: &closeCount,
|
||||
}
|
||||
fctx, err := srv.openAITunnelOutputFilterContext(req)
|
||||
if err != nil {
|
||||
t.Fatalf("openAITunnelOutputFilterContext: %v", err)
|
||||
}
|
||||
stallState, registration, err := openAIStallRecoveryRegistration(fctx)
|
||||
if err != nil {
|
||||
t.Fatalf("openAIStallRecoveryRegistration: %v", err)
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(srv.streamGateConfig(), fctx, registration)
|
||||
if err != nil {
|
||||
t.Fatalf("openAIStreamGateRegistrySnapshotFor: %v", err)
|
||||
}
|
||||
w := newRecordingResponseWriter()
|
||||
sink := newOpenAIResponsesPoolReleaseSink(w, &openAIResponsesResultHolder{}, newOpenAIStreamGateCodecSelector(openAIStreamGateCodecTunnel))
|
||||
runtime, _, err := srv.buildOpenAITunnelStreamGateRuntime(req, handle, sink, registry, stallState)
|
||||
if err != nil {
|
||||
t.Fatalf("buildOpenAITunnelStreamGateRuntime: %v", err)
|
||||
}
|
||||
runErr := runtime.Run(t.Context())
|
||||
committed, success := sink.terminalStatus()
|
||||
if closeErr := runtime.CloseRequestResources(t.Context(), false); closeErr != nil {
|
||||
t.Fatalf("CloseRequestResources: %v", closeErr)
|
||||
}
|
||||
body := w.body.String()
|
||||
if runErr != nil || !committed || success || w.code != http.StatusOK || closeCount != 1 {
|
||||
t.Fatalf("runtime=(err=%v committed=%v success=%v status=%d closes=%d body=%q)", runErr, committed, success, w.code, closeCount, body)
|
||||
}
|
||||
if !strings.Contains(body, "direct-prefix") || strings.Contains(body, "raw-provider-terminal-must-stay-hidden") || strings.Contains(body, "provider body") || strings.Contains(body, "raw provider metadata") {
|
||||
t.Fatalf("direct Responses terminal leaked or lost wire: %q", body)
|
||||
}
|
||||
if strings.Count(body, `"type":"error"`) != 1 || strings.Count(body, "data: [DONE]") != 1 || strings.Contains(body, `"type":"response.completed"`) {
|
||||
t.Fatalf("direct Responses terminal count mismatch: %q", body)
|
||||
}
|
||||
}
|
||||
|
||||
type countingDirectTunnelHandle struct {
|
||||
fakeTunnelHandle
|
||||
closed *int
|
||||
|
|
|
|||
|
|
@ -281,9 +281,8 @@ func streamgateSelectorType(s string) (streamgate.PolicySelectorType, bool) {
|
|||
// no scheme neither registers nor requires it. The returned slices are the
|
||||
// request-stable inputs to a generation-bound FilterRegistrySnapshot.
|
||||
func openAIOutputFilterRegistrations(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext) ([]streamgate.FilterRegistration, []streamgate.FilterPolicyLayer, error) {
|
||||
// The response runtime and its private liveness registration are always
|
||||
// present on supported OpenAI paths. This gate controls configured semantic
|
||||
// filters and their capability admission only.
|
||||
// The response runtime remains present on supported OpenAI paths. This gate
|
||||
// controls configured semantic filters and their capability admission only.
|
||||
if !gateCfg.Enabled {
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -207,7 +207,7 @@ func (s *openAIChatSSEReleaseSink) CommitTerminal(ctx context.Context, tr stream
|
|||
}
|
||||
|
||||
message := openAIStreamGateErrorMessage(tr)
|
||||
if !s.semanticEnabled && s.liveTerminal != nil && message != openAIStallFailureCode {
|
||||
if !s.semanticEnabled && s.liveTerminal != nil {
|
||||
if compatibilityMessage := s.liveTerminal.getErrorMessage(); compatibilityMessage != "" {
|
||||
message = compatibilityMessage
|
||||
}
|
||||
|
|
@ -218,7 +218,7 @@ func (s *openAIChatSSEReleaseSink) CommitTerminal(ctx context.Context, tr stream
|
|||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
}
|
||||
if !s.wroteHeader {
|
||||
if !s.semanticEnabled && message != openAIStallFailureCode {
|
||||
if !s.semanticEnabled {
|
||||
s.commitHeaderLocked(http.StatusOK)
|
||||
writeSSEErrorWithType(s.w, s.flusher, "run_error", message)
|
||||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@ import (
|
|||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
|
@ -81,60 +80,6 @@ func newOpenAIProviderErrorEvent(code string) (streamgate.NormalizedEvent, error
|
|||
return streamgate.NewProviderErrorEvent(streamGateChannelDefault, desc, causes, time.Now())
|
||||
}
|
||||
|
||||
const (
|
||||
openAIStallFailureCode = "response_stalled"
|
||||
openAIStallHandoffCause = "confirmed"
|
||||
openAIStallHandoffStage = "recovery_handoff"
|
||||
openAIStallHealthStage = "provider_health"
|
||||
openAIStallAttemptFenceKey = "attempt_fence"
|
||||
openAIStallHandoffKey = "recovery_handoff"
|
||||
openAIStallProviderIDKey = "provider_id"
|
||||
openAIStallProviderHealthKey = "provider_health"
|
||||
)
|
||||
|
||||
// newOpenAIProviderErrorEventFromFailure admits only the typed, Edge-confirmed
|
||||
// stall handoff into the Core contract. The proto failure itself is never
|
||||
// copied: its arbitrary message and metadata remain outside StreamGate.
|
||||
func newOpenAIProviderErrorEventFromFailure(failure *iop.ExecutionFailure, fallback string) (streamgate.NormalizedEvent, error) {
|
||||
if failure == nil || failure.GetCode() != openAIStallFailureCode || !failure.GetRetryable() {
|
||||
return newOpenAIProviderErrorEvent(fallback)
|
||||
}
|
||||
metadata := failure.GetMetadata()
|
||||
health := metadata[openAIStallProviderHealthKey]
|
||||
if metadata["failure_code"] != openAIStallFailureCode ||
|
||||
metadata[openAIStallAttemptFenceKey] != openAIStallHandoffCause ||
|
||||
metadata[openAIStallHandoffKey] != openAIStallHandoffCause ||
|
||||
metadata[openAIStallProviderIDKey] == "" ||
|
||||
(health != "available" && health != "unavailable" && health != "unknown") {
|
||||
return newOpenAIProviderErrorEvent(fallback)
|
||||
}
|
||||
desc, err := streamgate.NewExternalDescriptor("provider_error", openAIStallFailureCode, openAIStallFailureCode, "")
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
handoff, err := streamgate.NewFailureCause(openAIStallHandoffStage, metadata[openAIStallHandoffKey], "", "", "")
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
healthCause, err := streamgate.NewFailureCause(openAIStallHealthStage, health, "", "", "")
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
causes, err := streamgate.NewFailureCauseChain([]streamgate.FailureCause{handoff, healthCause})
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
return streamgate.NewProviderErrorEvent(streamGateChannelDefault, desc, causes, time.Now())
|
||||
}
|
||||
|
||||
func openAIExecutionFailureFromError(err error) *iop.ExecutionFailure {
|
||||
var terminal *openAIRunTerminalError
|
||||
if errors.As(err, &terminal) {
|
||||
return terminal.executionFailure()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// openAIStreamGateUsageHolder carries the final attempt observation used by
|
||||
// response renderers. Provider metrics use the separate per-attempt owner and
|
||||
// never discard an aborted attempt when recovery replaces it.
|
||||
|
|
@ -292,7 +237,7 @@ func (s *openAIRunEventSource) NextEvent(ctx context.Context) (streamgate.Normal
|
|||
}
|
||||
return streamgate.NewTerminalEvent(streamGateChannelDefault, time.Now())
|
||||
case "error", "cancelled":
|
||||
if s.chat != nil && (event.GetFailure() == nil || event.GetFailure().GetCode() != openAIStallFailureCode) {
|
||||
if s.chat != nil {
|
||||
message := event.GetError()
|
||||
if message == "" {
|
||||
message = event.GetMessage()
|
||||
|
|
@ -303,7 +248,7 @@ func (s *openAIRunEventSource) NextEvent(ctx context.Context) (streamgate.Normal
|
|||
s.chat.terminal.setErrorMessage(message)
|
||||
s.chat.terminal.setProviderTerminal()
|
||||
}
|
||||
return newOpenAIProviderErrorEventFromFailure(event.GetFailure(), streamGateErrorRunFailed)
|
||||
return newOpenAIProviderErrorEvent(streamGateErrorRunFailed)
|
||||
default:
|
||||
continue
|
||||
}
|
||||
|
|
@ -539,7 +484,7 @@ func (s *openAIBufferedChatEventSource) NextEvent(ctx context.Context) (streamga
|
|||
return streamgate.NormalizedEvent{}, ctx.Err()
|
||||
}
|
||||
s.holder.set(openAIBufferedAttemptResult{dispatch: s.handle.Dispatch(), collectErr: err})
|
||||
return newOpenAIProviderErrorEventFromFailure(openAIExecutionFailureFromError(err), streamGateErrorRunFailed)
|
||||
return newOpenAIProviderErrorEvent(streamGateErrorRunFailed)
|
||||
}
|
||||
verr := result.toolValidationErr
|
||||
if verr == nil {
|
||||
|
|
@ -817,20 +762,18 @@ func (s *openAITunnelEventSource) translateFrame(frame *iop.ProviderTunnelFrame)
|
|||
return nil, nil
|
||||
|
||||
case iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR:
|
||||
if frame.GetFailure() == nil || frame.GetFailure().GetCode() != openAIStallFailureCode {
|
||||
message := frame.GetError()
|
||||
if message == "" {
|
||||
message = "provider tunnel failed"
|
||||
}
|
||||
s.compatState.setCompatibilityProviderTerminal(message)
|
||||
// The generic provider-error compatibility path owns its legacy
|
||||
// pre/post-start behavior and must not acquire an endpoint-authored
|
||||
// SSE terminal merely because endpoint parsing is always enabled.
|
||||
s.compatState.mu.Lock()
|
||||
s.compatState.endpoint = ""
|
||||
s.compatState.mu.Unlock()
|
||||
message := frame.GetError()
|
||||
if message == "" {
|
||||
message = "provider tunnel failed"
|
||||
}
|
||||
ev, err := newOpenAIProviderErrorEventFromFailure(frame.GetFailure(), streamGateErrorTunnelFailed)
|
||||
s.compatState.setCompatibilityProviderTerminal(message)
|
||||
// The generic provider-error compatibility path owns its legacy
|
||||
// pre/post-start behavior and must not acquire an endpoint-authored
|
||||
// SSE terminal merely because endpoint parsing is always enabled.
|
||||
s.compatState.mu.Lock()
|
||||
s.compatState.endpoint = ""
|
||||
s.compatState.mu.Unlock()
|
||||
ev, err := newOpenAIProviderErrorEvent(streamGateErrorTunnelFailed)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -909,7 +852,7 @@ func openAIStreamGateRegistrySnapshot() (streamgate.FilterRegistrySnapshot, erro
|
|||
// one request: the always-applicable Noop mechanics filter, the configured
|
||||
// semantic output filters (repeat/schema/provider-error) translated from the
|
||||
// supplied stream_evidence_gate policy, plus any request-local extra
|
||||
// registrations (e.g. the typed-stall recovery and tool validation gates).
|
||||
// registrations such as tool validation gates.
|
||||
func openAIStreamGateRegistrySnapshotFor(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext, extra ...streamgate.FilterRegistration) (streamgate.FilterRegistrySnapshot, error) {
|
||||
regs, err := openAIStreamGateNoopRegistrations()
|
||||
if err != nil {
|
||||
|
|
@ -924,19 +867,6 @@ func openAIStreamGateRegistrySnapshotFor(gateCfg config.StreamEvidenceGateConf,
|
|||
return streamgate.NewFilterRegistrySnapshot(streamGateConfigGeneration, regs, policies)
|
||||
}
|
||||
|
||||
func openAIStallRecoveryRegistration(fctx openAIOutputFilterContext) (*openAIStallRecoveryState, streamgate.FilterRegistration, error) {
|
||||
state := &openAIStallRecoveryState{}
|
||||
filter, err := newOpenAIStallRecoveryFilter(fctx.requestRef, state)
|
||||
if err != nil {
|
||||
return nil, streamgate.FilterRegistration{}, err
|
||||
}
|
||||
registration, err := streamgate.NewFilterRegistration(filter, streamGateNoopCapability, true, streamgate.FilterEnforcementBlocking, streamGateFilterTimeout, openAIStallRecoveryPriority)
|
||||
if err != nil {
|
||||
return nil, streamgate.FilterRegistration{}, err
|
||||
}
|
||||
return state, registration, nil
|
||||
}
|
||||
|
||||
// streamGateConfig returns a copy of the request-stable stream-gate config the
|
||||
// request runtime pins at request start (generation isolation).
|
||||
func (s *Server) streamGateConfig() config.StreamEvidenceGateConf {
|
||||
|
|
@ -1244,7 +1174,6 @@ type openAIChatStreamGateConfig struct {
|
|||
preparer streamgate.RecoveryPlanPreparer
|
||||
prepFactory streamgate.RecoveryPreparationSnapshotFactory
|
||||
obsSink streamgate.ObservationSink
|
||||
stallState *openAIStallRecoveryState
|
||||
liveTerminal *openAIChatLiveTerminalState
|
||||
semanticEnabled bool
|
||||
}
|
||||
|
|
@ -1339,7 +1268,7 @@ func (s *Server) buildOpenAIChatStreamGateRuntimeFor(dc *chatDispatchContext, cf
|
|||
return nil, nil, err
|
||||
}
|
||||
build := newOpenAIChatRecoveryAdmissionBuilder(s, dc, cfg.holder)
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), build, s.newOpenAIChatAttemptEventSourceFactory(dc, cfg, usage), dc.usage, cfg.stallState, cfg.sink, cfg.holder)
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), build, s.newOpenAIChatAttemptEventSourceFactory(dc, cfg, usage), dc.usage, cfg.sink, cfg.holder)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
@ -1363,7 +1292,6 @@ func (s *Server) buildOpenAIChatStreamGateRuntimeFor(dc *chatDispatchContext, cf
|
|||
initialController := &openAIAttemptController{
|
||||
service: s.service, dispatch: dispatch, closeTransport: cfg.closeAll,
|
||||
usageRecorder: dc.usage, usageBinding: cfg.initial.usageBinding, usage: cfg.initial.usage,
|
||||
stall: cfg.stallState,
|
||||
compatibilitySink: func() openAIStreamGateSink { sink, _ := cfg.sink.(openAIStreamGateSink); return sink }(),
|
||||
}
|
||||
initialBinding, err := streamgate.NewAttemptBinding(
|
||||
|
|
@ -1502,15 +1430,7 @@ func (s *Server) runOpenAIChatStreamGate(w http.ResponseWriter, flusher http.Flu
|
|||
dc.finishUsageRequest(usageStatusError, responseModeNormalized)
|
||||
return
|
||||
}
|
||||
stallState, stallRegistration, err := openAIStallRecoveryRegistration(fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate chat liveness registration failed", zap.Error(err))
|
||||
writeSSEErrorWithType(w, flusher, "run_error", "stream gate runtime unavailable")
|
||||
dc.finishUsageRequest(usageStatusError, responseModeNormalized)
|
||||
return
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx, stallRegistration)
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate chat registry build failed", zap.Error(err))
|
||||
|
|
@ -1528,7 +1448,6 @@ func (s *Server) runOpenAIChatStreamGate(w http.ResponseWriter, flusher http.Flu
|
|||
selector: selector,
|
||||
registry: registry,
|
||||
obsSink: s.observationSink(),
|
||||
stallState: stallState,
|
||||
liveTerminal: liveTerminal,
|
||||
semanticEnabled: semanticEnabled,
|
||||
}, sink, func() {
|
||||
|
|
@ -1589,31 +1508,21 @@ func (s *Server) newOpenAIBufferedChatStreamGateConfig(
|
|||
normalized := newOpenAIBufferedChatReleaseSink(s, w, flusher, dc, stream, holder)
|
||||
sink := s.openAIChatCompositeSink(w, flusher, dc, selector, normalized)
|
||||
|
||||
fctx, err := s.openAIChatOutputFilterContext(dc)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{}, nil, err
|
||||
}
|
||||
stallState, stallRegistration, err := openAIStallRecoveryRegistration(fctx)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{}, nil, err
|
||||
}
|
||||
extraFilters = append(extraFilters, stallRegistration)
|
||||
registry, err := s.openAIChatStreamGateRegistry(dc, holder, extraFilters)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{}, nil, err
|
||||
}
|
||||
return openAIChatStreamGateConfig{
|
||||
writer: w,
|
||||
mode: openAIChatGateModeBuffered,
|
||||
initial: openAIAttemptTransport{path: openAIAdmissionRun, run: handle},
|
||||
dispatch: handle.Dispatch(),
|
||||
closeAll: handle.Close,
|
||||
sink: sink,
|
||||
selector: selector,
|
||||
registry: registry,
|
||||
holder: holder,
|
||||
obsSink: s.observationSink(),
|
||||
stallState: stallState,
|
||||
writer: w,
|
||||
mode: openAIChatGateModeBuffered,
|
||||
initial: openAIAttemptTransport{path: openAIAdmissionRun, run: handle},
|
||||
dispatch: handle.Dispatch(),
|
||||
closeAll: handle.Close,
|
||||
sink: sink,
|
||||
selector: selector,
|
||||
registry: registry,
|
||||
holder: holder,
|
||||
obsSink: s.observationSink(),
|
||||
}, sink, nil
|
||||
}
|
||||
|
||||
|
|
@ -1708,15 +1617,6 @@ func (s *Server) newOpenAIChatPoolStreamGateConfig(
|
|||
}
|
||||
sink := s.openAIChatCompositeSink(w, flusher, dc, selector, normalized)
|
||||
|
||||
fctx, err := s.openAIChatOutputFilterContext(dc)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{closeAll: closeAll}, nil, err
|
||||
}
|
||||
stallState, stallRegistration, err := openAIStallRecoveryRegistration(fctx)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{closeAll: closeAll}, nil, err
|
||||
}
|
||||
extraFilters = append(extraFilters, stallRegistration)
|
||||
registry, err := s.openAIChatStreamGateRegistry(dc, holder, extraFilters)
|
||||
if err != nil {
|
||||
return openAIChatStreamGateConfig{closeAll: closeAll}, nil, err
|
||||
|
|
@ -1732,7 +1632,6 @@ func (s *Server) newOpenAIChatPoolStreamGateConfig(
|
|||
registry: registry,
|
||||
holder: holder,
|
||||
obsSink: s.observationSink(),
|
||||
stallState: stallState,
|
||||
semanticEnabled: s.streamGateSemanticEnabled(),
|
||||
}
|
||||
if liveSink, ok := normalized.(*openAIChatSSEReleaseSink); ok {
|
||||
|
|
@ -1847,12 +1746,7 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
handle edgeservice.ProviderTunnelResult,
|
||||
sink streamgate.ReleaseSink,
|
||||
registry streamgate.FilterRegistrySnapshot,
|
||||
stallStates ...*openAIStallRecoveryState,
|
||||
) (*streamgate.RequestRuntime, *openAIStreamGateUsageHolder, error) {
|
||||
var stallState *openAIStallRecoveryState
|
||||
if len(stallStates) > 0 {
|
||||
stallState = stallStates[0]
|
||||
}
|
||||
usage := &openAIStreamGateUsageHolder{}
|
||||
var responseSink *openAIResponsesPoolReleaseSink
|
||||
if req.endpoint == openAIRebuildEndpointResponses {
|
||||
|
|
@ -1894,7 +1788,7 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
tracking := &openAIStreamGateUsageTrackingTunnelSource{openAITunnelEventSource: src, usage: usage, attempt: transport.usage}
|
||||
return newOpenAIRecoverySourceEventSource(tracking, recoverySource), nil
|
||||
}
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), build, eventSourceFactory, req.usage, stallState, sink)
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), build, eventSourceFactory, req.usage, sink)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
@ -1933,7 +1827,6 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
initialController := &openAIAttemptController{
|
||||
service: s.service, dispatch: dispatch, closeTransport: handle.Close,
|
||||
usageRecorder: req.usage, usageBinding: initialTransport.usageBinding, usage: initialTransport.usage,
|
||||
stall: stallState,
|
||||
compatibilitySink: func() openAIStreamGateSink { typed, _ := sink.(openAIStreamGateSink); return typed }(),
|
||||
}
|
||||
initialBinding, err := streamgate.NewAttemptBinding(
|
||||
|
|
@ -2024,15 +1917,7 @@ func (s *Server) runOpenAITunnelStreamGate(w http.ResponseWriter, r *http.Reques
|
|||
usageRecorder.FinishRequest(usageStatusError, responseModePassthrough)
|
||||
return
|
||||
}
|
||||
stallState, stallRegistration, err := openAIStallRecoveryRegistration(fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate tunnel liveness registration failed", zap.Error(err))
|
||||
writeError(w, http.StatusInternalServerError, "provider_tunnel_error", "stream gate runtime unavailable")
|
||||
usageRecorder.FinishRequest(usageStatusError, responseModePassthrough)
|
||||
return
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx, stallRegistration)
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate tunnel registry build failed", zap.Error(err))
|
||||
|
|
@ -2040,7 +1925,7 @@ func (s *Server) runOpenAITunnelStreamGate(w http.ResponseWriter, r *http.Reques
|
|||
usageRecorder.FinishRequest(usageStatusError, responseModePassthrough)
|
||||
return
|
||||
}
|
||||
rt, _, err := s.buildOpenAITunnelStreamGateRuntime(req, handle, sink, registry, stallState)
|
||||
rt, _, err := s.buildOpenAITunnelStreamGateRuntime(req, handle, sink, registry)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate tunnel runtime build failed", zap.Error(err))
|
||||
|
|
|
|||
|
|
@ -1,796 +0,0 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
func confirmedStallFailure(health string) *iop.ExecutionFailure {
|
||||
return &iop.ExecutionFailure{
|
||||
Code: openAIStallFailureCode,
|
||||
Retryable: true,
|
||||
Message: "provider body, prompt, and credentials must not escape",
|
||||
Metadata: map[string]string{
|
||||
"failure_code": openAIStallFailureCode,
|
||||
"attempt_fence": "confirmed",
|
||||
"recovery_handoff": "confirmed",
|
||||
"provider_id": "provider-a",
|
||||
"provider_health": health,
|
||||
"untrusted": "raw provider metadata",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func stallFilterContext(t *testing.T, commit streamgate.CommitState, sideEffect bool) streamgate.FilterContext {
|
||||
t.Helper()
|
||||
ctx, err := streamgate.NewFilterContextBuilder(streamGateConfigGeneration, "attempt.run-a").
|
||||
SetEndpoint(openAIRebuildEndpointChat).
|
||||
SetActualProvider("provider-a").
|
||||
SetCommitState(commit).
|
||||
SetHasToolSideEffect(sideEffect).
|
||||
Build()
|
||||
if err != nil {
|
||||
t.Fatalf("build filter context: %v", err)
|
||||
}
|
||||
return ctx
|
||||
}
|
||||
|
||||
func stallBatch(t *testing.T, event streamgate.NormalizedEvent, commit streamgate.CommitState, pending ...streamgate.NormalizedEvent) streamgate.EvidenceBatch {
|
||||
t.Helper()
|
||||
batch, err := streamgate.NewEvidenceBatch([]streamgate.NormalizedEvent{event}, map[string][]streamgate.NormalizedEvent{streamGateChannelDefault: pending}, nil, nil, true, commit, time.Now())
|
||||
if err != nil {
|
||||
t.Fatalf("build stall batch: %v", err)
|
||||
}
|
||||
return batch
|
||||
}
|
||||
|
||||
func TestOpenAIStallEventMapping(t *testing.T) {
|
||||
event, err := newOpenAIProviderErrorEventFromFailure(confirmedStallFailure("unknown"), streamGateErrorRunFailed)
|
||||
if err != nil {
|
||||
t.Fatalf("map confirmed stall: %v", err)
|
||||
}
|
||||
terminal, err := event.AsProviderError()
|
||||
if err != nil {
|
||||
t.Fatalf("AsProviderError: %v", err)
|
||||
}
|
||||
if desc := terminal.ExternalDesc(); desc == nil || desc.Code() != openAIStallFailureCode || desc.Message() != openAIStallFailureCode {
|
||||
t.Fatalf("descriptor = %#v", desc)
|
||||
}
|
||||
for _, cause := range terminal.FailureCauses().All() {
|
||||
if cause.Code() == "provider body, prompt, and credentials must not escape" || cause.Code() == "raw provider metadata" {
|
||||
t.Fatalf("raw failure data leaked into causes: %#v", cause)
|
||||
}
|
||||
}
|
||||
|
||||
generic, err := newOpenAIProviderErrorEventFromFailure(&iop.ExecutionFailure{Code: "other", Message: "raw"}, streamGateErrorRunFailed)
|
||||
if err != nil {
|
||||
t.Fatalf("map generic failure: %v", err)
|
||||
}
|
||||
genericTerminal, _ := generic.AsProviderError()
|
||||
if got := genericTerminal.ExternalDesc().Code(); got != streamGateErrorRunFailed {
|
||||
t.Fatalf("generic descriptor code = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIStallRecoveryFilter(t *testing.T) {
|
||||
for _, health := range []string{"available", "unavailable", "unknown"} {
|
||||
t.Run(health, func(t *testing.T) {
|
||||
state := &openAIStallRecoveryState{}
|
||||
filter, err := newOpenAIStallRecoveryFilter("openai.ingress.1", state)
|
||||
if err != nil {
|
||||
t.Fatalf("new filter: %v", err)
|
||||
}
|
||||
event, err := newOpenAIProviderErrorEventFromFailure(confirmedStallFailure(health), streamGateErrorRunFailed)
|
||||
if err != nil {
|
||||
t.Fatalf("map failure: %v", err)
|
||||
}
|
||||
decision, err := filter.Evaluate(context.Background(), stallFilterContext(t, streamgate.CommitStateTransportUncommitted, false), stallBatch(t, event, streamgate.CommitStateTransportUncommitted))
|
||||
if err != nil {
|
||||
t.Fatalf("evaluate: %v", err)
|
||||
}
|
||||
if decision.Kind() != streamgate.FilterDecisionKindViolation || decision.RecoveryIntent() == nil {
|
||||
t.Fatalf("decision = %#v", decision)
|
||||
}
|
||||
if !state.claimConfirmedClose("attempt.run-a") {
|
||||
t.Fatal("confirmed state was not armed")
|
||||
}
|
||||
provider, fallback, ok := state.consumeAdmission()
|
||||
if !ok || provider != "provider-a" || fallback {
|
||||
t.Fatalf("admission hint = %q/%t/%t", provider, fallback, ok)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIStallRecoveryIneligibleAfterCommitOrTool(t *testing.T) {
|
||||
event, err := newOpenAIProviderErrorEventFromFailure(confirmedStallFailure("available"), streamGateErrorRunFailed)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
commit streamgate.CommitState
|
||||
sideEffect bool
|
||||
}{
|
||||
{"post_commit", streamgate.CommitStateStreamOpen, false},
|
||||
{"tool_side_effect", streamgate.CommitStateTransportUncommitted, true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
state := &openAIStallRecoveryState{}
|
||||
filter, _ := newOpenAIStallRecoveryFilter("openai.ingress.1", state)
|
||||
decision, err := filter.Evaluate(context.Background(), stallFilterContext(t, tc.commit, tc.sideEffect), stallBatch(t, event, tc.commit))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if decision.Kind() != streamgate.FilterDecisionKindPass || decision.RecoveryIntent() != nil {
|
||||
t.Fatalf("unsafe decision = %#v", decision)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIStallRecoveryIsBoundedToOneReplay(t *testing.T) {
|
||||
state := &openAIStallRecoveryState{}
|
||||
filter, err := newOpenAIStallRecoveryFilter("openai.ingress.1", state)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
event, err := newOpenAIProviderErrorEventFromFailure(confirmedStallFailure("available"), streamGateErrorRunFailed)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := stallFilterContext(t, streamgate.CommitStateTransportUncommitted, false)
|
||||
batch := stallBatch(t, event, streamgate.CommitStateTransportUncommitted)
|
||||
first, err := filter.Evaluate(context.Background(), ctx, batch)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := filter.Evaluate(context.Background(), ctx, batch)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first.Kind() != streamgate.FilterDecisionKindViolation || first.RecoveryIntent() == nil {
|
||||
t.Fatalf("first stall decision = %#v", first)
|
||||
}
|
||||
if second.Kind() != streamgate.FilterDecisionKindPass || second.RecoveryIntent() != nil {
|
||||
t.Fatalf("second stall decision = %#v", second)
|
||||
}
|
||||
}
|
||||
|
||||
func stallMatrixSuccessAttempt(endpoint, path string, stream bool, runID, provider, marker string) scriptedPoolAttempt {
|
||||
attempt := scriptedPoolAttempt{path: path, runID: runID, provider: provider, target: "served-" + provider}
|
||||
if path == string(edgeservice.ProviderPoolPathNormalized) {
|
||||
attempt.runEvents = bufferedRunEvents(
|
||||
&iop.RunEvent{RunId: runID, Type: "delta", Delta: marker},
|
||||
&iop.RunEvent{RunId: runID, Type: "complete", Metadata: map[string]string{"finish_reason": "stop"}},
|
||||
)
|
||||
return attempt
|
||||
}
|
||||
body := []byte(fmt.Sprintf(`{"id":"chat-recovered","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":%q},"finish_reason":"stop"}]}`, marker))
|
||||
if endpoint == openAIRebuildEndpointResponses {
|
||||
if stream {
|
||||
body = []byte(fmt.Sprintf("data: {\"type\":\"response.output_text.delta\",\"delta\":%q}\n\ndata: {\"type\":\"response.completed\"}\n\ndata: [DONE]\n\n", marker))
|
||||
} else {
|
||||
body = []byte(fmt.Sprintf(`{"id":"resp-recovered","object":"response","status":"completed","output_text":%q,"output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":%q}]}]}`, marker, marker))
|
||||
}
|
||||
}
|
||||
contentType := "application/json"
|
||||
if stream {
|
||||
contentType = "text/event-stream"
|
||||
}
|
||||
attempt.frames = bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": contentType}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: body},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true},
|
||||
)
|
||||
return attempt
|
||||
}
|
||||
|
||||
func stallMatrixFailureAttempt(path, runID, provider, health string) scriptedPoolAttempt {
|
||||
attempt := scriptedPoolAttempt{path: path, runID: runID, provider: provider, target: "served-" + provider}
|
||||
if path == string(edgeservice.ProviderPoolPathNormalized) {
|
||||
attempt.runEvents = bufferedRunEvents(&iop.RunEvent{RunId: runID, Type: "error", Failure: confirmedStallFailure(health)})
|
||||
} else {
|
||||
attempt.frames = bufferedTunnelFrames(&iop.ProviderTunnelFrame{RunId: runID, Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure(health)})
|
||||
}
|
||||
return attempt
|
||||
}
|
||||
|
||||
func stallMatrixServer(service runService, semantic bool, budget int) *Server {
|
||||
srv := NewServer(config.EdgeOpenAIConf{
|
||||
TimeoutSec: 5,
|
||||
StreamEvidenceGate: config.StreamEvidenceGateConf{
|
||||
Enabled: semantic, MaxRequestFaultRecovery: &budget,
|
||||
},
|
||||
}, service, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: "matrix-model", Providers: map[string]string{"provider-a": "served-a", "provider-b": "served-b"},
|
||||
}})
|
||||
return srv
|
||||
}
|
||||
|
||||
func runStallMatrixHandler(t *testing.T, srv *Server, endpoint string, stream bool, ctx context.Context, bodyOverride ...string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
path := "/v1/chat/completions"
|
||||
body := fmt.Sprintf(`{"model":"matrix-model","stream":%t,"messages":[{"role":"user","content":"hi"}]}`, stream)
|
||||
if endpoint == openAIRebuildEndpointResponses {
|
||||
path = "/v1/responses"
|
||||
body = fmt.Sprintf(`{"model":"matrix-model","stream":%t,"input":"hi"}`, stream)
|
||||
}
|
||||
if len(bodyOverride) > 0 && bodyOverride[0] != "" {
|
||||
body = bodyOverride[0]
|
||||
}
|
||||
r := httptest.NewRequest(http.MethodPost, path, strings.NewReader(body))
|
||||
if ctx != nil {
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
if endpoint == openAIRebuildEndpointChat {
|
||||
srv.handleChatCompletions(w, r)
|
||||
} else {
|
||||
srv.handleResponses(w, r)
|
||||
}
|
||||
return w
|
||||
}
|
||||
|
||||
func stallPoolRequests(service *scriptedPoolRunService) []edgeservice.ProviderPoolDispatchRequest {
|
||||
service.mu.Lock()
|
||||
defer service.mu.Unlock()
|
||||
return append([]edgeservice.ProviderPoolDispatchRequest(nil), service.poolRequests...)
|
||||
}
|
||||
|
||||
func countStallMatrixString(values []string, want string) int {
|
||||
count := 0
|
||||
for _, value := range values {
|
||||
if value == want {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
func assertStallAttemptClosedOnce(t *testing.T, service *scriptedPoolRunService, path, runID string) {
|
||||
t.Helper()
|
||||
_, _, runCloses, tunnelCloses, _, _ := service.snapshot()
|
||||
closes := runCloses
|
||||
if path == string(edgeservice.ProviderPoolPathTunnel) {
|
||||
closes = tunnelCloses
|
||||
}
|
||||
if got := countStallMatrixString(closes, runID); got != 1 {
|
||||
t.Fatalf("transport close count for %s = %d, want 1 (run=%v tunnel=%v)", runID, got, runCloses, tunnelCloses)
|
||||
}
|
||||
}
|
||||
|
||||
func logicalFinishStallAttempt(runID, content string, committed bool) scriptedPoolAttempt {
|
||||
wire := []byte(fmt.Sprintf("data: {\"id\":\"logical-finish\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":%q},\"finish_reason\":\"stop\"}]}\n\n", content))
|
||||
if committed {
|
||||
// The first content frame opens the stream. The following finish-only
|
||||
// frame has no semantic event, so its identifiable wire remains pending
|
||||
// until [DONE]/END and must be discarded by the later stall terminal.
|
||||
wire = append(
|
||||
[]byte("data: {\"id\":\"logical-finish\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"committed-prefix\"},\"finish_reason\":null}]}\n\n"),
|
||||
[]byte(fmt.Sprintf("data: {\"id\":%q,\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n", content))...,
|
||||
)
|
||||
}
|
||||
return scriptedPoolAttempt{
|
||||
path: string(edgeservice.ProviderPoolPathTunnel), runID: runID, provider: "provider-a", target: "served-a",
|
||||
frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure("available")},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
func logicalFinishMatrixServer(service runService, holdRunes int) *Server {
|
||||
budget := 1
|
||||
srv := NewServer(config.EdgeOpenAIConf{
|
||||
TimeoutSec: 5,
|
||||
StreamEvidenceGate: config.StreamEvidenceGateConf{
|
||||
Enabled: true, MaxRequestFaultRecovery: &budget,
|
||||
Filters: []config.StreamGateFilterPolicyConf{{
|
||||
Filter: config.StreamGateFilterRepeatGuard, Enforcement: config.StreamGateFilterEnforcementBlocking,
|
||||
Priority: 10, HoldEvidenceRunes: holdRunes,
|
||||
}},
|
||||
},
|
||||
}, service, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: "matrix-model", ContextWindowTokens: 8192,
|
||||
Providers: map[string]string{"provider-a": "served-a", "provider-b": "served-b"},
|
||||
}})
|
||||
return srv
|
||||
}
|
||||
|
||||
// TestOpenAIStallAfterLogicalFinishMatrix drives the observed shape through
|
||||
// the production tunnel codec/Core/sink: finish_reason is pending protocol
|
||||
// wire, while only the typed stall terminal decides replay or sanitized close.
|
||||
func TestOpenAIStallAfterLogicalFinishMatrix(t *testing.T) {
|
||||
t.Run("pre-commit pending finish replays once", func(t *testing.T) {
|
||||
const pending = "pending-finish-must-stay-hidden"
|
||||
const recovered = "recovered-output-only"
|
||||
service := newScriptedPoolRunService(
|
||||
logicalFinishStallAttempt("logical-pre-a", pending, false),
|
||||
stallMatrixSuccessAttempt(openAIRebuildEndpointChat, string(edgeservice.ProviderPoolPathTunnel), true, "logical-pre-b", "provider-b", recovered),
|
||||
)
|
||||
w := runStallMatrixHandler(t, logicalFinishMatrixServer(service, 500), openAIRebuildEndpointChat, true, nil)
|
||||
body := w.Body.String()
|
||||
if w.Code != http.StatusOK || service.poolSubmits() != 2 || !strings.Contains(body, recovered) || strings.Contains(body, pending) {
|
||||
t.Fatalf("pre-commit recovery=(status=%d dispatches=%d body=%q)", w.Code, service.poolSubmits(), body)
|
||||
}
|
||||
if strings.Contains(body, "provider body") || strings.Contains(body, "raw provider metadata") {
|
||||
t.Fatalf("raw failure data leaked: %q", body)
|
||||
}
|
||||
requests := stallPoolRequests(service)
|
||||
if len(requests) != 2 || requests[1].AvoidProviderID != "provider-a" || requests[1].AllowAvoidedProviderFallback {
|
||||
t.Fatalf("pre-commit replay requests=%+v", requests)
|
||||
}
|
||||
if strings.Count(body, recovered) != 1 {
|
||||
t.Fatalf("recovered output count is not one: %q", body)
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "logical-pre-a")
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "logical-pre-b")
|
||||
})
|
||||
|
||||
t.Run("post-commit finish emits one sanitized terminal", func(t *testing.T) {
|
||||
const pending = "pending-finish-after-commit"
|
||||
service := newScriptedPoolRunService(
|
||||
logicalFinishStallAttempt("logical-post-a", pending, true),
|
||||
stallMatrixSuccessAttempt(openAIRebuildEndpointChat, string(edgeservice.ProviderPoolPathTunnel), true, "forbidden", "provider-b", "must-not-replay"),
|
||||
)
|
||||
w := runStallMatrixHandler(t, logicalFinishMatrixServer(service, 1), openAIRebuildEndpointChat, true, nil)
|
||||
body := w.Body.String()
|
||||
if w.Code != http.StatusOK || service.poolSubmits() != 1 || !strings.Contains(body, "committed-prefix") || strings.Contains(body, "must-not-replay") {
|
||||
t.Fatalf("post-commit recovery=(status=%d dispatches=%d body=%q)", w.Code, service.poolSubmits(), body)
|
||||
}
|
||||
if strings.Count(body, `"type":"run_error"`) != 1 || strings.Count(body, "data: [DONE]") != 1 {
|
||||
t.Fatalf("post-commit terminal count mismatch: %q", body)
|
||||
}
|
||||
if strings.Contains(body, "provider body") || strings.Contains(body, "raw provider metadata") || strings.Contains(body, pending) {
|
||||
t.Fatalf("pending/raw bytes leaked after commit: %q", body)
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "logical-post-a")
|
||||
})
|
||||
}
|
||||
|
||||
func semanticDisabledCommittedStallAttempt(endpoint, runID, pending string) scriptedPoolAttempt {
|
||||
body := []byte(
|
||||
"data: {\"id\":\"disabled-chat\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"disabled-committed-prefix\"},\"finish_reason\":null}]}\n\n" +
|
||||
fmt.Sprintf("data: {\"id\":%q,\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n", pending),
|
||||
)
|
||||
if endpoint == openAIRebuildEndpointResponses {
|
||||
body = []byte(
|
||||
"data: {\"type\":\"response.created\",\"response\":{\"id\":\"resp-disabled\",\"object\":\"response\",\"status\":\"in_progress\"},\"sequence_number\":1}\n\n" +
|
||||
"data: {\"type\":\"response.output_text.delta\",\"item_id\":\"msg-disabled\",\"output_index\":0,\"content_index\":0,\"delta\":\"disabled-committed-prefix\",\"sequence_number\":2}\n\n" +
|
||||
fmt.Sprintf("data: {\"type\":\"response.completed\",\"response\":{\"id\":%q,\"object\":\"response\",\"status\":\"completed\"},\"sequence_number\":3}\n\n", pending),
|
||||
)
|
||||
}
|
||||
return scriptedPoolAttempt{
|
||||
path: string(edgeservice.ProviderPoolPathTunnel), runID: runID, provider: "provider-a", target: "served-a",
|
||||
frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: body},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure("available")},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAISemanticGateDisabledStallTerminalMatrix proves that endpoint
|
||||
// framing and the private typed-stall owner remain active when semantic filters
|
||||
// are disabled. Pre-commit stalls replay once; an opened Chat/Responses stream
|
||||
// never replays and receives one sanitized endpoint-native terminal.
|
||||
func TestOpenAISemanticGateDisabledStallTerminalMatrix(t *testing.T) {
|
||||
for _, endpoint := range []string{openAIRebuildEndpointChat, openAIRebuildEndpointResponses} {
|
||||
t.Run(endpoint+"/pre-commit", func(t *testing.T) {
|
||||
const recovered = "semantic-disabled-recovered"
|
||||
service := newScriptedPoolRunService(
|
||||
stallMatrixFailureAttempt(string(edgeservice.ProviderPoolPathTunnel), "disabled-pre-a", "provider-a", "available"),
|
||||
stallMatrixSuccessAttempt(endpoint, string(edgeservice.ProviderPoolPathTunnel), true, "disabled-pre-b", "provider-b", recovered),
|
||||
)
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), endpoint, true, nil)
|
||||
body := w.Body.String()
|
||||
if w.Code != http.StatusOK || service.poolSubmits() != 2 || strings.Count(body, recovered) != 1 {
|
||||
t.Fatalf("pre-commit response=(status=%d dispatches=%d body=%q)", w.Code, service.poolSubmits(), body)
|
||||
}
|
||||
if strings.Contains(body, "provider body") || strings.Contains(body, "raw provider metadata") {
|
||||
t.Fatalf("raw failure data leaked: %q", body)
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "disabled-pre-a")
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "disabled-pre-b")
|
||||
})
|
||||
|
||||
t.Run(endpoint+"/post-commit", func(t *testing.T) {
|
||||
const pending = "semantic-disabled-pending-terminal"
|
||||
service := newScriptedPoolRunService(
|
||||
semanticDisabledCommittedStallAttempt(endpoint, "disabled-post-a", pending),
|
||||
stallMatrixSuccessAttempt(endpoint, string(edgeservice.ProviderPoolPathTunnel), true, "forbidden", "provider-b", "must-not-replay"),
|
||||
)
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), endpoint, true, nil)
|
||||
body := w.Body.String()
|
||||
if w.Code != http.StatusOK || service.poolSubmits() != 1 || !strings.Contains(body, "disabled-committed-prefix") || strings.Contains(body, "must-not-replay") {
|
||||
t.Fatalf("post-commit response=(status=%d dispatches=%d body=%q)", w.Code, service.poolSubmits(), body)
|
||||
}
|
||||
if strings.Contains(body, pending) || strings.Contains(body, "provider body") || strings.Contains(body, "raw provider metadata") {
|
||||
t.Fatalf("pending/raw failure data leaked: %q", body)
|
||||
}
|
||||
if strings.Count(body, "data: [DONE]") != 1 {
|
||||
t.Fatalf("DONE terminal count mismatch: %q", body)
|
||||
}
|
||||
if endpoint == openAIRebuildEndpointChat {
|
||||
if strings.Count(body, `"type":"run_error"`) != 1 {
|
||||
t.Fatalf("Chat error terminal count mismatch: %q", body)
|
||||
}
|
||||
} else if strings.Count(body, `"type":"error"`) != 1 || strings.Contains(body, `"type":"response.completed"`) {
|
||||
t.Fatalf("Responses error terminal mismatch: %q", body)
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, string(edgeservice.ProviderPoolPathTunnel), "disabled-post-a")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAIStallRecoveryMatrix proves S05 through the supported production
|
||||
// handlers and the production runtime adapter. It covers every endpoint/path/
|
||||
// semantic-policy recovery product, then exercises the shared budget and every
|
||||
// zero-recovery safety guard.
|
||||
func TestOpenAIStallRecoveryMatrix(t *testing.T) {
|
||||
type recoveryCase struct {
|
||||
name string
|
||||
endpoint string
|
||||
initialPath string
|
||||
replacementPath string
|
||||
stream bool
|
||||
semantic bool
|
||||
}
|
||||
tunnelPath := string(edgeservice.ProviderPoolPathTunnel)
|
||||
normPath := string(edgeservice.ProviderPoolPathNormalized)
|
||||
recoveryCases := []recoveryCase{
|
||||
{name: "chat", endpoint: openAIRebuildEndpointChat, initialPath: normPath, replacementPath: normPath, stream: false, semantic: false},
|
||||
{name: "chat", endpoint: openAIRebuildEndpointChat, initialPath: normPath, replacementPath: normPath, stream: false, semantic: true},
|
||||
{name: "chat", endpoint: openAIRebuildEndpointChat, initialPath: tunnelPath, replacementPath: tunnelPath, stream: false, semantic: false},
|
||||
{name: "chat", endpoint: openAIRebuildEndpointChat, initialPath: tunnelPath, replacementPath: tunnelPath, stream: false, semantic: true},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: normPath, replacementPath: normPath, stream: false, semantic: false},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: normPath, replacementPath: normPath, stream: false, semantic: true},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: tunnelPath, stream: false, semantic: false},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: tunnelPath, stream: false, semantic: true},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: normPath, stream: false, semantic: false},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: normPath, stream: false, semantic: true},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: normPath, replacementPath: tunnelPath, stream: false, semantic: false},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: normPath, replacementPath: tunnelPath, stream: false, semantic: true},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: tunnelPath, stream: true, semantic: false},
|
||||
{name: "responses", endpoint: openAIRebuildEndpointResponses, initialPath: tunnelPath, replacementPath: tunnelPath, stream: true, semantic: true},
|
||||
}
|
||||
for _, tc := range recoveryCases {
|
||||
pathLabel := tc.initialPath
|
||||
if tc.initialPath != tc.replacementPath {
|
||||
pathLabel = fmt.Sprintf("%s_to_%s", tc.initialPath, tc.replacementPath)
|
||||
}
|
||||
name := fmt.Sprintf("recover/%s/%s/stream=%t/semantic=%t", tc.name, pathLabel, tc.stream, tc.semantic)
|
||||
t.Run(name, func(t *testing.T) {
|
||||
marker := fmt.Sprintf("recovered-%s-%s-stream-%t", tc.endpoint, pathLabel, tc.stream)
|
||||
service := newScriptedPoolRunService(
|
||||
stallMatrixFailureAttempt(tc.initialPath, "attempt-a", "provider-a", "unavailable"),
|
||||
stallMatrixSuccessAttempt(tc.endpoint, tc.replacementPath, tc.stream, "attempt-b", "provider-b", marker),
|
||||
)
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, tc.semantic, 1), tc.endpoint, tc.stream, nil)
|
||||
if w.Code != http.StatusOK || !strings.Contains(w.Body.String(), marker) {
|
||||
t.Fatalf("recovered response=(status=%d body=%q)", w.Code, w.Body.String())
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "provider body") || strings.Contains(w.Body.String(), "raw provider metadata") {
|
||||
t.Fatalf("raw stall data leaked: %q", w.Body.String())
|
||||
}
|
||||
if tc.endpoint == openAIRebuildEndpointChat && strings.Count(w.Body.String(), `"object":"chat.completion"`) != 1 {
|
||||
t.Fatalf("chat terminal count is not one: %q", w.Body.String())
|
||||
}
|
||||
if tc.endpoint == openAIRebuildEndpointResponses && tc.stream {
|
||||
if strings.Count(w.Body.String(), `"type":"response.completed"`) != 1 || strings.Count(w.Body.String(), "data: [DONE]") != 1 {
|
||||
t.Fatalf("streaming Responses terminal count is not one: %q", w.Body.String())
|
||||
}
|
||||
} else if tc.endpoint == openAIRebuildEndpointResponses && strings.Count(w.Body.String(), `"object":"response"`) != 1 {
|
||||
t.Fatalf("responses terminal count is not one: %q", w.Body.String())
|
||||
}
|
||||
requests := stallPoolRequests(service)
|
||||
if len(requests) != 2 || requests[1].AvoidProviderID != "provider-a" || requests[1].AllowAvoidedProviderFallback {
|
||||
t.Fatalf("re-admission requests=%+v, want one provider-a avoidance without fallback", requests)
|
||||
}
|
||||
pools, cancels, _, _, runRequests, tunnelRequests := service.snapshot()
|
||||
if pools != 2 || len(cancels) != 0 {
|
||||
t.Fatalf("dispatch/cancel lifecycle=(%d,%v), want (2,none)", pools, cancels)
|
||||
}
|
||||
if tc.replacementPath == normPath {
|
||||
if len(runRequests) == 0 {
|
||||
t.Fatalf("expected at least one normalized run request, got 0")
|
||||
}
|
||||
replacementRun := runRequests[len(runRequests)-1]
|
||||
if replacementRun.TimeoutSec != 5 {
|
||||
t.Fatalf("normalized replacement TimeoutSec = %d, want ingress timeout 5", replacementRun.TimeoutSec)
|
||||
}
|
||||
wantPrompt := "user: hi"
|
||||
wantEstimate := 2
|
||||
wantEstimateStr := "2"
|
||||
if tc.endpoint == openAIRebuildEndpointResponses {
|
||||
wantPrompt = "hi"
|
||||
wantEstimate = 7
|
||||
wantEstimateStr = "7"
|
||||
}
|
||||
if replacementRun.Prompt != wantPrompt {
|
||||
t.Fatalf("normalized replacement Prompt = %q, want %q", replacementRun.Prompt, wantPrompt)
|
||||
}
|
||||
if tc.endpoint == openAIRebuildEndpointResponses {
|
||||
if prompt, ok := replacementRun.Input["prompt"].(string); !ok || prompt != "hi" {
|
||||
t.Fatalf("normalized replacement Input[\"prompt\"] = %v, want hi", replacementRun.Input["prompt"])
|
||||
}
|
||||
}
|
||||
if replacementRun.Metadata["openai_model"] != "matrix-model" || replacementRun.Metadata["openai_stream"] != fmt.Sprintf("%t", tc.stream) {
|
||||
t.Fatalf("normalized replacement metadata = %v, want model matrix-model and stream %t", replacementRun.Metadata, tc.stream)
|
||||
}
|
||||
if replacementRun.Metadata["strict_output"] != "false" ||
|
||||
replacementRun.Metadata["estimated_input_tokens"] != wantEstimateStr ||
|
||||
replacementRun.Metadata["context_class"] != "normal" {
|
||||
t.Fatalf("normalized replacement derived metadata = %v", replacementRun.Metadata)
|
||||
}
|
||||
if replacementRun.MaxQueue != 0 || replacementRun.QueueTimeoutMS != 0 {
|
||||
t.Fatalf("normalized replacement queue fields=(%d,%d), want (0,0)", replacementRun.MaxQueue, replacementRun.QueueTimeoutMS)
|
||||
}
|
||||
if replacementRun.EstimatedInputTokens != wantEstimate || replacementRun.ContextClass != "normal" {
|
||||
t.Fatalf("normalized replacement estimate/class=(%d,%q), want (%d,normal)", replacementRun.EstimatedInputTokens, replacementRun.ContextClass, wantEstimate)
|
||||
}
|
||||
} else {
|
||||
if len(tunnelRequests) == 0 {
|
||||
t.Fatalf("expected at least one provider tunnel request, got 0")
|
||||
}
|
||||
replacementTunnel := tunnelRequests[len(tunnelRequests)-1]
|
||||
if replacementTunnel.TimeoutSec != 5 {
|
||||
t.Fatalf("tunnel replacement TimeoutSec = %d, want 5", replacementTunnel.TimeoutSec)
|
||||
}
|
||||
if replacementTunnel.Stream != tc.stream {
|
||||
t.Fatalf("tunnel replacement Stream = %t, want %t", replacementTunnel.Stream, tc.stream)
|
||||
}
|
||||
if replacementTunnel.Metadata["openai_model"] != "matrix-model" || replacementTunnel.Metadata["openai_stream"] != fmt.Sprintf("%t", tc.stream) {
|
||||
t.Fatalf("tunnel replacement metadata = %v, want model matrix-model and stream %t", replacementTunnel.Metadata, tc.stream)
|
||||
}
|
||||
if replacementTunnel.EstimatedInputTokens <= 0 || replacementTunnel.ContextClass == "" {
|
||||
t.Fatalf("tunnel replacement estimate/class invalid: estimate=%d class=%q", replacementTunnel.EstimatedInputTokens, replacementTunnel.ContextClass)
|
||||
}
|
||||
if replacementTunnel.BuildBody == nil {
|
||||
t.Fatalf("tunnel replacement BuildBody is nil")
|
||||
}
|
||||
rebuilt, err := replacementTunnel.BuildBody("served-b")
|
||||
if err != nil {
|
||||
t.Fatalf("tunnel replacement BuildBody failed: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(rebuilt), `"model":"served-b"`) {
|
||||
t.Fatalf("tunnel replacement body missing target model served-b: %q", string(rebuilt))
|
||||
}
|
||||
if tc.endpoint == openAIRebuildEndpointResponses {
|
||||
if !strings.Contains(string(rebuilt), `"input":"hi"`) || !strings.Contains(string(rebuilt), fmt.Sprintf(`"stream":%t`, tc.stream)) {
|
||||
t.Fatalf("tunnel replacement body lost input or stream: %q", string(rebuilt))
|
||||
}
|
||||
}
|
||||
}
|
||||
if tc.endpoint == openAIRebuildEndpointResponses && tc.stream && strings.Contains(w.Body.String(), `"type":"error"`) {
|
||||
t.Fatalf("streaming Responses rendered an error terminal: %q", w.Body.String())
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, tc.initialPath, "attempt-a")
|
||||
assertStallAttemptClosedOnce(t, service, tc.replacementPath, "attempt-b")
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("stall replay never grants same-provider fallback", func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathNormalized)
|
||||
service := newScriptedPoolRunService(
|
||||
stallMatrixFailureAttempt(path, "available-a", "provider-a", "available"),
|
||||
stallMatrixSuccessAttempt(openAIRebuildEndpointChat, path, false, "available-b", "provider-a", "same-provider-recovered"),
|
||||
)
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), openAIRebuildEndpointChat, false, nil)
|
||||
requests := stallPoolRequests(service)
|
||||
if w.Code != http.StatusOK || len(requests) != 2 || requests[1].AvoidProviderID != "provider-a" || requests[1].AllowAvoidedProviderFallback {
|
||||
t.Fatalf("available fallback response=%d/%q requests=%+v", w.Code, w.Body.String(), requests)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("shared budget emits one sanitized terminal", func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathNormalized)
|
||||
service := newScriptedPoolRunService(
|
||||
stallMatrixFailureAttempt(path, "budget-a", "provider-a", "unavailable"),
|
||||
stallMatrixFailureAttempt(path, "budget-b", "provider-b", "unavailable"),
|
||||
stallMatrixSuccessAttempt(openAIRebuildEndpointChat, path, false, "budget-c", "provider-a", "must-not-dispatch"),
|
||||
)
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, true, 1), openAIRebuildEndpointChat, false, nil)
|
||||
if service.poolSubmits() != 2 || w.Code != http.StatusBadGateway || strings.Count(w.Body.String(), `"type":"run_error"`) != 1 || strings.Contains(w.Body.String(), "provider body") || strings.Contains(w.Body.String(), "must-not-dispatch") {
|
||||
t.Fatalf("budget terminal=(dispatches=%d status=%d body=%q)", service.poolSubmits(), w.Code, w.Body.String())
|
||||
}
|
||||
assertStallAttemptClosedOnce(t, service, path, "budget-a")
|
||||
assertStallAttemptClosedOnce(t, service, path, "budget-b")
|
||||
})
|
||||
|
||||
for _, guard := range []struct {
|
||||
name string
|
||||
budget int
|
||||
failure *iop.ExecutionFailure
|
||||
}{
|
||||
{name: "generic-unconfirmed", budget: 1, failure: &iop.ExecutionFailure{Code: openAIStallFailureCode, Retryable: true, Message: "secret generic"}},
|
||||
{name: "exhausted", budget: 0, failure: confirmedStallFailure("unknown")},
|
||||
{name: "unsupported-health", budget: 1, failure: func() *iop.ExecutionFailure {
|
||||
f := confirmedStallFailure("unknown")
|
||||
f.Metadata[openAIStallProviderHealthKey] = "unsupported"
|
||||
return f
|
||||
}()},
|
||||
} {
|
||||
t.Run("guard/"+guard.name, func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathNormalized)
|
||||
attempt := scriptedPoolAttempt{path: path, runID: "guard-" + guard.name, provider: "provider-a", target: "served-a", runEvents: bufferedRunEvents(&iop.RunEvent{Type: "error", Failure: guard.failure})}
|
||||
service := newScriptedPoolRunService(attempt, stallMatrixSuccessAttempt(openAIRebuildEndpointChat, path, false, "forbidden", "provider-b", "must-not-render"))
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, true, guard.budget), openAIRebuildEndpointChat, false, nil)
|
||||
if service.poolSubmits() != 1 || w.Code != http.StatusBadGateway || strings.Contains(w.Body.String(), "secret") || strings.Contains(w.Body.String(), "must-not-render") {
|
||||
t.Fatalf("guard result=(dispatches=%d status=%d body=%q)", service.poolSubmits(), w.Code, w.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("guard/committed", func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathTunnel)
|
||||
wire := []byte("data: {\"id\":\"partial\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"committed\"},\"finish_reason\":null}]}\n\n")
|
||||
attempt := scriptedPoolAttempt{path: path, runID: "committed-a", provider: "provider-a", target: "served-a", frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure("available")},
|
||||
)}
|
||||
service := newScriptedPoolRunService(attempt, stallMatrixSuccessAttempt(openAIRebuildEndpointChat, path, false, "forbidden", "provider-b", "must-not-render"))
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, true, 1), openAIRebuildEndpointChat, true, nil)
|
||||
if service.poolSubmits() != 1 || !strings.Contains(w.Body.String(), "committed") || strings.Contains(w.Body.String(), "must-not-render") {
|
||||
t.Fatalf("committed guard=(dispatches=%d status=%d body=%q)", service.poolSubmits(), w.Code, w.Body.String())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("guard/caller-cancelled", func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathNormalized)
|
||||
openEvents := make(chan *iop.RunEvent)
|
||||
service := newScriptedPoolRunService(scriptedPoolAttempt{path: path, runID: "cancelled-a", provider: "provider-a", target: "served-a", runEvents: openEvents})
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, true, 1), openAIRebuildEndpointChat, false, ctx)
|
||||
if service.poolSubmits() != 1 || w.Code != http.StatusRequestTimeout {
|
||||
t.Fatalf("cancel guard=(dispatches=%d status=%d body=%q)", service.poolSubmits(), w.Code, w.Body.String())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("guard/tool-side-effect", func(t *testing.T) {
|
||||
path := string(edgeservice.ProviderPoolPathTunnel)
|
||||
toolWire := []byte("data: {\"id\":\"tool\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call-1\",\"type\":\"function\",\"function\":{\"name\":\"act\",\"arguments\":\"{}\"}}]},\"finish_reason\":null}]}\n\n")
|
||||
attempt := scriptedPoolAttempt{path: path, runID: "tool-a", provider: "provider-a", target: "served-a", frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: toolWire},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: confirmedStallFailure("available")},
|
||||
)}
|
||||
service := newScriptedPoolRunService(attempt, stallMatrixSuccessAttempt(openAIRebuildEndpointChat, path, false, "forbidden", "provider-b", "must-not-render"))
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, true, 1), openAIRebuildEndpointChat, true, nil)
|
||||
if service.poolSubmits() != 1 || !strings.Contains(w.Body.String(), "call-1") || strings.Contains(w.Body.String(), "must-not-render") {
|
||||
t.Fatalf("tool guard=(dispatches=%d status=%d body=%q)", service.poolSubmits(), w.Code, w.Body.String())
|
||||
}
|
||||
})
|
||||
|
||||
for _, owner := range []struct {
|
||||
name string
|
||||
requestRef string
|
||||
register bool
|
||||
}{
|
||||
{name: "missing-snapshot", requestRef: "", register: true},
|
||||
{name: "no-owner", register: false},
|
||||
} {
|
||||
t.Run("guard/"+owner.name, func(t *testing.T) {
|
||||
raw := []byte(`{"model":"matrix-model","stream":true,"messages":[{"role":"user","content":"hi"}]}`)
|
||||
service := &fakeRunService{}
|
||||
srv := NewServer(config.EdgeOpenAIConf{Adapter: "ollama"}, service, nil)
|
||||
base := newTestRequestContext(t, routeDispatch{Adapter: "ollama", Target: "served-a", TimeoutSec: 5}, raw)
|
||||
var req chatCompletionRequest
|
||||
if err := json.Unmarshal(raw, &req); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dc := srv.newChatDispatchContext(base, req, "hi", strictOutputPolicy{})
|
||||
handle := &fakeRunResult{dispatch: edgeservice.RunDispatch{RunID: owner.name + "-a", NodeID: "node-a", ProviderID: "provider-a", ModelGroupKey: "matrix-model", Target: "served-a"}, events: bufferedRunEvents(&iop.RunEvent{Type: "error", Failure: confirmedStallFailure("unknown")})}
|
||||
var registry streamgate.FilterRegistrySnapshot
|
||||
var err error
|
||||
if owner.register {
|
||||
_, registration, regErr := openAIStallRecoveryRegistration(openAIOutputFilterContext{requestRef: owner.requestRef})
|
||||
if regErr != nil {
|
||||
t.Fatal(regErr)
|
||||
}
|
||||
registry, err = openAIStreamGateRegistrySnapshotWith(registration)
|
||||
} else {
|
||||
registry, err = openAIStreamGateRegistrySnapshot()
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
sink := newOpenAIChatSSEReleaseSink(w, nil, "chatcmpl-guard", time.Now().Unix(), "matrix-model")
|
||||
runtime, _, err := srv.buildOpenAIChatStreamGateRuntime(dc, handle, sink, registry)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runErr := runtime.Run(t.Context())
|
||||
_ = runtime.CloseRequestResources(t.Context(), runErr == nil)
|
||||
if len(service.reqsSnapshot()) != 0 || !strings.Contains(w.Body.String(), openAIStallFailureCode) || strings.Contains(w.Body.String(), "provider body") {
|
||||
t.Fatalf("owner guard=(dispatches=%d runErr=%v body=%q)", len(service.reqsSnapshot()), runErr, w.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAISemanticGateDisabledCompatibility proves that the always-owned
|
||||
// runtime preserves endpoint-native behavior while semantic filters are off.
|
||||
func TestOpenAISemanticGateDisabledCompatibility(t *testing.T) {
|
||||
t.Run("chat/normalized/sse", func(t *testing.T) {
|
||||
service := newScriptedPoolRunService(scriptedPoolAttempt{
|
||||
path: string(edgeservice.ProviderPoolPathNormalized), runID: "compat-chat-run", provider: "provider-a", target: "served-a",
|
||||
runEvents: bufferedRunEvents(
|
||||
&iop.RunEvent{Type: "reasoning_delta", Delta: "private compatibility reasoning"},
|
||||
&iop.RunEvent{Type: "delta", Delta: "compatibility chat"},
|
||||
&iop.RunEvent{Type: "complete", Metadata: map[string]string{"finish_reason": "length"}, Usage: &iop.Usage{InputTokens: 2, OutputTokens: 3}},
|
||||
),
|
||||
})
|
||||
srv := stallMatrixServer(service, false, 1)
|
||||
w := runStallMatrixHandler(t, srv, openAIRebuildEndpointChat, true, nil)
|
||||
body := w.Body.String()
|
||||
if srv.streamGateSemanticEnabled() || w.Code != http.StatusOK || w.Header().Get("Content-Type") != "text/event-stream" || !strings.Contains(body, "compatibility chat") || !strings.Contains(body, `"reasoning_content":"private compatibility reasoning"`) || !strings.Contains(body, `"finish_reason":"length"`) || strings.Count(body, "data: [DONE]") != 1 || service.poolSubmits() != 1 {
|
||||
t.Fatalf("normalized Chat compatibility=(status=%d headers=%v dispatches=%d body=%q)", w.Code, w.Header(), service.poolSubmits(), body)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("chat/tunnel/sse-byte-order", func(t *testing.T) {
|
||||
wire := []byte("data: {\"id\":\"compat-chat\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"compat tunnel\"},\"finish_reason\":null}]}\n\ndata: {\"id\":\"compat-chat\",\"object\":\"chat.completion.chunk\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"length\"}]}\n\ndata: [DONE]\n\n")
|
||||
service := newScriptedPoolRunService(scriptedPoolAttempt{path: string(edgeservice.ProviderPoolPathTunnel), runID: "compat-chat-tunnel", provider: "provider-a", target: "served-a", frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "text/event-stream", "X-Compat": "chat"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire[:len(wire)/2]},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire[len(wire)/2:]},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true},
|
||||
)})
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), openAIRebuildEndpointChat, true, nil)
|
||||
if w.Code != http.StatusOK || w.Header().Get("X-Compat") != "chat" || w.Body.String() != string(wire) || strings.Count(w.Body.String(), "data: [DONE]") != 1 || service.poolSubmits() != 1 {
|
||||
t.Fatalf("tunnel Chat compatibility=(status=%d headers=%v dispatches=%d body=%q want=%q)", w.Code, w.Header(), service.poolSubmits(), w.Body.String(), wire)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("responses/normalized/json", func(t *testing.T) {
|
||||
service := newScriptedPoolRunService(scriptedPoolAttempt{
|
||||
path: string(edgeservice.ProviderPoolPathNormalized), runID: "compat-responses-run", provider: "provider-a", target: "served-a",
|
||||
runEvents: bufferedRunEvents(
|
||||
&iop.RunEvent{Type: "reasoning_delta", Delta: "compatibility reasoning"},
|
||||
&iop.RunEvent{Type: "delta", Delta: "compatibility responses"},
|
||||
&iop.RunEvent{Type: "complete", Usage: &iop.Usage{InputTokens: 4, OutputTokens: 5, ReasoningTokens: 2}},
|
||||
),
|
||||
})
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), openAIRebuildEndpointResponses, false, nil)
|
||||
var response responsesResponse
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &response); err != nil {
|
||||
t.Fatalf("decode Responses compatibility: %v body=%q", err, w.Body.String())
|
||||
}
|
||||
if w.Code != http.StatusOK || response.OutputText != "compatibility responses" || response.Usage.TotalTokens != 9 || strings.Count(w.Body.String(), `"object":"response"`) != 1 || service.poolSubmits() != 1 {
|
||||
t.Fatalf("normalized Responses compatibility=(status=%d response=%+v dispatches=%d body=%q)", w.Code, response, service.poolSubmits(), w.Body.String())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("responses/tunnel/json-byte-order", func(t *testing.T) {
|
||||
wire := []byte(`{"id":"compat-responses","object":"response","status":"completed","output_text":"compat tunnel responses","output":[]}`)
|
||||
service := newScriptedPoolRunService(scriptedPoolAttempt{path: string(edgeservice.ProviderPoolPathTunnel), runID: "compat-responses-tunnel", provider: "provider-a", target: "served-a", frames: bufferedTunnelFrames(
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START, StatusCode: http.StatusOK, Headers: map[string]string{"Content-Type": "application/json", "X-Compat": "responses"}},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire[:31]},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: wire[31:]},
|
||||
&iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true},
|
||||
)})
|
||||
w := runStallMatrixHandler(t, stallMatrixServer(service, false, 1), openAIRebuildEndpointResponses, false, nil)
|
||||
if w.Code != http.StatusOK || w.Header().Get("X-Compat") != "responses" || w.Body.String() != string(wire) || strings.Count(w.Body.String(), `"object":"response"`) != 1 || service.poolSubmits() != 1 {
|
||||
t.Fatalf("tunnel Responses compatibility=(status=%d headers=%v dispatches=%d body=%q want=%q)", w.Code, w.Header(), service.poolSubmits(), w.Body.String(), wire)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
@ -81,9 +81,6 @@ func (m *modelQueueManager) findAvailableNodeLocked(group *modelQueueGroup, cand
|
|||
if !live {
|
||||
continue
|
||||
}
|
||||
if !m.candidateRuntimeHealthyLocked(&c) {
|
||||
continue
|
||||
}
|
||||
if c.capacity <= 0 {
|
||||
continue
|
||||
}
|
||||
|
|
@ -148,99 +145,6 @@ func (m *modelQueueManager) findAvailableNodeLocked(group *modelQueueGroup, cand
|
|||
|
||||
// candidateLess provides a deterministic ordering for equal-inflight/priority rotation:
|
||||
// providerID first, then nodeID.
|
||||
|
||||
// candidateRecoveryEligibleLocked reports whether c is a valid identity target
|
||||
// for the recovery preference: it must live-resolve to an enabled provider with
|
||||
// positive configured capacity, be runtime-healthy, and (for provider-pool
|
||||
// candidates) neither orphaned nor generation-fenced. Momentary in-flight
|
||||
// saturation is deliberately NOT considered — a busy but eligible alternate is
|
||||
// still preferred over the avoided provider, and the request queues for it. This
|
||||
// mirrors the eligibility findAvailableNodeLocked applies at selection time,
|
||||
// minus the transient capacity check, so the "does an eligible alternate remain"
|
||||
// decision matches what the scheduler can actually dispatch. Must be called with
|
||||
// m.mu held.
|
||||
func (m *modelQueueManager) candidateRecoveryEligibleLocked(c *candidateNode) bool {
|
||||
live, ok := m.liveCandidateLocked(c)
|
||||
if !ok || live.capacity <= 0 {
|
||||
return false
|
||||
}
|
||||
if !m.candidateRuntimeHealthyLocked(&live) {
|
||||
return false
|
||||
}
|
||||
if c.providerID != "" {
|
||||
key := providerResourceKey{nodeID: c.entry.NodeID, providerID: c.providerID}
|
||||
if res, exists := m.resources[key]; exists {
|
||||
if res.orphan || !res.enabled {
|
||||
return false
|
||||
}
|
||||
if !generationEligible(c.generation, res) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// applyRecoveryPreferenceLocked applies the request-local avoided-provider
|
||||
// preference AFTER current runtime eligibility, under m.mu. It partitions the
|
||||
// candidates using candidateRecoveryEligibleLocked so that only a genuinely
|
||||
// runtime-eligible alternate suppresses the avoided provider — an unhealthy,
|
||||
// orphaned, or disabled alternate identity can no longer starve an explicit
|
||||
// same-provider fallback.
|
||||
//
|
||||
// Returns (preferred, rejected):
|
||||
// - avoidProviderID empty: the input is returned unchanged, rejected=false.
|
||||
// - an eligible alternate exists: only the non-avoided candidates are
|
||||
// returned, rejected=false (the avoided provider is dropped).
|
||||
// - no eligible alternate and fallback allowed and the avoided provider is
|
||||
// eligible: the avoided candidates are returned, rejected=false.
|
||||
// - no eligible alternate, the avoided provider is eligible, and fallback is
|
||||
// not permitted: (nil, true) — a request-policy terminal rejection.
|
||||
// - nothing eligible at all: (nil, false) — the caller maps the empty result
|
||||
// to provider-unavailable, not a policy rejection.
|
||||
//
|
||||
// Must be called with m.mu held.
|
||||
func (m *modelQueueManager) applyRecoveryPreferenceLocked(candidates []candidateNode, recovery recoveryCandidatePolicy) ([]candidateNode, bool) {
|
||||
if !recovery.active() || len(candidates) == 0 {
|
||||
return candidates, false
|
||||
}
|
||||
|
||||
var alternates []candidateNode
|
||||
var avoided []candidateNode
|
||||
eligibleAlternate := false
|
||||
avoidedEligible := false
|
||||
for i := range candidates {
|
||||
if candidates[i].providerID == recovery.avoidProviderID {
|
||||
avoided = append(avoided, candidates[i])
|
||||
if m.candidateRecoveryEligibleLocked(&candidates[i]) {
|
||||
avoidedEligible = true
|
||||
}
|
||||
continue
|
||||
}
|
||||
alternates = append(alternates, candidates[i])
|
||||
if m.candidateRecoveryEligibleLocked(&candidates[i]) {
|
||||
eligibleAlternate = true
|
||||
}
|
||||
}
|
||||
|
||||
if eligibleAlternate {
|
||||
return alternates, false
|
||||
}
|
||||
// No runtime-eligible alternate remains: the avoided provider may only be
|
||||
// re-selected with explicit fallback permission and only while it is itself
|
||||
// eligible.
|
||||
if recovery.allowAvoidedProviderFallback && avoidedEligible {
|
||||
return avoided, false
|
||||
}
|
||||
// Fallback not permitted. If the avoided provider is the sole eligible
|
||||
// candidate the request policy rejected it (terminal); otherwise nothing is
|
||||
// eligible and the caller reports provider-unavailable.
|
||||
if avoidedEligible {
|
||||
return nil, true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func candidateLess(a, b *candidateNode) bool {
|
||||
if b == nil {
|
||||
return true
|
||||
|
|
@ -314,10 +218,6 @@ func (m *modelQueueManager) reserveCandidateLocked(group *modelQueueGroup, candi
|
|||
if !eligible || live.capacity <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
if !m.candidateRuntimeHealthyLocked(&live) {
|
||||
return 0, false
|
||||
}
|
||||
|
||||
slot := candidate.slotKey()
|
||||
if candidate.providerID != "" {
|
||||
res, ok := m.resourceForCandidateLocked(candidate)
|
||||
|
|
@ -460,11 +360,7 @@ func (m *modelQueueManager) pumpAllLocked() {
|
|||
// Must be called with m.mu held.
|
||||
func (m *modelQueueManager) resolveQueuedCandidatesLocked(item *queueItem) ([]candidateNode, resolveOutcome, error) {
|
||||
if item.resolveCandidates == nil {
|
||||
filtered := m.filterRuntimeHealthyCandidatesLocked(item.candidates)
|
||||
if len(item.candidates) > 0 && len(filtered) == 0 {
|
||||
return nil, resolveNoCandidates, nil
|
||||
}
|
||||
return m.applyQueuedRecoveryPreferenceLocked(filtered, item.recovery)
|
||||
return item.candidates, resolveOk, nil
|
||||
}
|
||||
candidates, err := item.resolveCandidates()
|
||||
if err != nil {
|
||||
|
|
@ -501,29 +397,7 @@ func (m *modelQueueManager) resolveQueuedCandidatesLocked(item *queueItem) ([]ca
|
|||
// to dispatch to. Treat as no-live-candidate terminal.
|
||||
return nil, resolveNoCandidates, nil
|
||||
}
|
||||
filtered = m.filterRuntimeHealthyCandidatesLocked(filtered)
|
||||
if len(filtered) == 0 {
|
||||
return nil, resolveNoCandidates, nil
|
||||
}
|
||||
return m.applyQueuedRecoveryPreferenceLocked(filtered, item.recovery)
|
||||
}
|
||||
|
||||
// applyQueuedRecoveryPreferenceLocked applies the request-local recovery
|
||||
// preference to an already runtime-eligible queued candidate set and maps the
|
||||
// result to a pump resolveOutcome: a request-policy rejection becomes the typed
|
||||
// terminal error (no reservation), an empty preferred set becomes
|
||||
// resolveNoCandidates (provider-unavailable), and a non-empty set continues to
|
||||
// selection. A zero-value policy returns the candidates unchanged. Must be
|
||||
// called with m.mu held.
|
||||
func (m *modelQueueManager) applyQueuedRecoveryPreferenceLocked(candidates []candidateNode, recovery recoveryCandidatePolicy) ([]candidateNode, resolveOutcome, error) {
|
||||
preferred, rejected := m.applyRecoveryPreferenceLocked(candidates, recovery)
|
||||
if rejected {
|
||||
return nil, resolveTerminalError, ErrProviderPoolCandidateRejected
|
||||
}
|
||||
if len(preferred) == 0 {
|
||||
return nil, resolveNoCandidates, nil
|
||||
}
|
||||
return preferred, resolveOk, nil
|
||||
return filtered, resolveOk, nil
|
||||
}
|
||||
|
||||
// pumpOnceLocked expires timed-out items and dispatches the earliest globally
|
||||
|
|
@ -624,19 +498,10 @@ func (m *modelQueueManager) admit(ctx context.Context, groupKey, adapter, target
|
|||
return candidate, err
|
||||
}
|
||||
|
||||
// admitWithReason preserves the recovery-free admission signature every existing
|
||||
// caller uses (legacy runs, provider tunnels, direct fixtures). It delegates to
|
||||
// admitWithRecovery with a zero-value recovery policy, so those paths keep their
|
||||
// current candidate-selection behavior untouched.
|
||||
// admitWithReason is the shared admission core for provider-pool and legacy
|
||||
// dispatch. Candidate selection is driven only by current configuration,
|
||||
// connectivity, capacity, priority, and queue state.
|
||||
func (m *modelQueueManager) admitWithReason(ctx context.Context, groupKey, adapter, target string, candidates []candidateNode, policy groupPolicy, resolveCandidates func() ([]candidateNode, error), long bool, providerPool bool) (*candidateNode, string, error) {
|
||||
return m.admitWithRecovery(ctx, groupKey, adapter, target, candidates, policy, resolveCandidates, long, providerPool, recoveryCandidatePolicy{})
|
||||
}
|
||||
|
||||
// admitWithRecovery is the admission core. The recovery policy is applied after
|
||||
// current runtime-health filtering under the same lock as selection, and is
|
||||
// stamped onto the queued item so every pump re-resolution reapplies the
|
||||
// identical request-local avoided-provider preference.
|
||||
func (m *modelQueueManager) admitWithRecovery(ctx context.Context, groupKey, adapter, target string, candidates []candidateNode, policy groupPolicy, resolveCandidates func() ([]candidateNode, error), long bool, providerPool bool, recovery recoveryCandidatePolicy) (*candidateNode, string, error) {
|
||||
m.mu.Lock()
|
||||
|
||||
group := m.getOrCreateGroupLocked(groupKey, policy)
|
||||
|
|
@ -650,26 +515,9 @@ func (m *modelQueueManager) admitWithRecovery(ctx context.Context, groupKey, ada
|
|||
if group.target == "" {
|
||||
group.target = target
|
||||
}
|
||||
if providerPool {
|
||||
candidates = m.filterRuntimeHealthyCandidatesLocked(candidates)
|
||||
if len(candidates) == 0 {
|
||||
m.mu.Unlock()
|
||||
return nil, "", fmt.Errorf("model group %q: %w", groupKey, errProviderUnavailable)
|
||||
}
|
||||
// Recovery preference is linearized behind runtime-health filtering under
|
||||
// the queue lock: only a runtime-eligible alternate suppresses the avoided
|
||||
// provider, and a fully rejected policy is a typed terminal without a
|
||||
// reservation. A zero-value policy leaves candidates unchanged.
|
||||
preferred, rejected := m.applyRecoveryPreferenceLocked(candidates, recovery)
|
||||
if rejected {
|
||||
m.mu.Unlock()
|
||||
return nil, "", ErrProviderPoolCandidateRejected
|
||||
}
|
||||
if len(preferred) == 0 {
|
||||
m.mu.Unlock()
|
||||
return nil, "", fmt.Errorf("model group %q: %w", groupKey, errProviderUnavailable)
|
||||
}
|
||||
candidates = preferred
|
||||
if providerPool && len(candidates) == 0 {
|
||||
m.mu.Unlock()
|
||||
return nil, "", fmt.Errorf("model group %q: %w", groupKey, errProviderUnavailable)
|
||||
}
|
||||
|
||||
candidate := m.findAvailableNodeLocked(group, candidates, long)
|
||||
|
|
@ -764,7 +612,6 @@ func (m *modelQueueManager) admitWithRecovery(ctx context.Context, groupKey, ada
|
|||
long: long,
|
||||
providerPool: providerPool,
|
||||
reason: reason,
|
||||
recovery: recovery,
|
||||
}
|
||||
m.enqueueItemLocked(group, item, resolveCandidates)
|
||||
m.mu.Unlock()
|
||||
|
|
|
|||
|
|
@ -2,14 +2,9 @@ package service
|
|||
|
||||
import (
|
||||
"fmt"
|
||||
"strconv"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
const recoveryHandoffConfirmed = "confirmed"
|
||||
|
||||
type receivedTerminalDisposition uint8
|
||||
|
||||
const (
|
||||
|
|
@ -65,103 +60,12 @@ func (m *modelQueueManager) releaseLeaseLocked(leaseID uint64) bool {
|
|||
return true
|
||||
}
|
||||
|
||||
// receivedHealthEvidence is the fully validated Node health evidence carried by
|
||||
// one typed response-stalled terminal. It contains no caller-controlled fields.
|
||||
type receivedHealthEvidence struct {
|
||||
providerHealth string
|
||||
sequence uint64
|
||||
}
|
||||
|
||||
func parseReceivedHealthEvidence(runID string, failure *iop.ExecutionFailure) (receivedHealthEvidence, bool) {
|
||||
if failure == nil || failure.GetCode() != string(runtime.FailureCodeResponseStalled) || !failure.GetRetryable() {
|
||||
return receivedHealthEvidence{}, false
|
||||
}
|
||||
metadata := failure.GetMetadata()
|
||||
if metadata["failure_code"] != string(runtime.FailureCodeResponseStalled) ||
|
||||
metadata["attempt_fence"] != "confirmed" ||
|
||||
metadata["run_id"] != runID || metadata["attempt_id"] != runID ||
|
||||
metadata["adapter"] == "" || metadata["target"] == "" {
|
||||
return receivedHealthEvidence{}, false
|
||||
}
|
||||
sequence, err := strconv.ParseUint(metadata["health_observation_seq"], 10, 64)
|
||||
if err != nil || sequence == 0 {
|
||||
return receivedHealthEvidence{}, false
|
||||
}
|
||||
health := metadata["provider_health"]
|
||||
classification := metadata["liveness_classification"]
|
||||
switch {
|
||||
case health == string(runtime.ProviderStatusUnavailable) && classification == string(runtime.ProviderUnhealthy):
|
||||
case health == string(runtime.ProviderStatusAvailable) && classification == string(runtime.RequestStalled):
|
||||
case health == string(runtime.ProviderStatusUnknown) && classification == string(runtime.HealthUnknown):
|
||||
default:
|
||||
return receivedHealthEvidence{}, false
|
||||
}
|
||||
return receivedHealthEvidence{providerHealth: health, sequence: sequence}, true
|
||||
}
|
||||
|
||||
func annotateRecoveryHandoff(failure *iop.ExecutionFailure, envelopeMetadata *map[string]string, providerID, providerHealth string) {
|
||||
if failure.Metadata == nil {
|
||||
failure.Metadata = make(map[string]string)
|
||||
}
|
||||
failure.Metadata["provider_id"] = providerID
|
||||
failure.Metadata["provider_health"] = providerHealth
|
||||
failure.Metadata["recovery_handoff"] = recoveryHandoffConfirmed
|
||||
if envelopeMetadata == nil {
|
||||
return
|
||||
}
|
||||
if *envelopeMetadata == nil {
|
||||
*envelopeMetadata = make(map[string]string)
|
||||
}
|
||||
(*envelopeMetadata)["provider_id"] = providerID
|
||||
(*envelopeMetadata)["provider_health"] = providerHealth
|
||||
(*envelopeMetadata)["recovery_handoff"] = recoveryHandoffConfirmed
|
||||
}
|
||||
|
||||
// applyReceivedHealthEvidenceLocked sequence-fences one fully bound terminal.
|
||||
// Every accepted observation advances the high-water mark. Only unavailable
|
||||
// lowers effective provider health; available/unknown stall observations never
|
||||
// recover an already unavailable provider.
|
||||
func (m *modelQueueManager) applyReceivedHealthEvidenceLocked(lease *providerLease, evidence receivedHealthEvidence) providerHealthObservation {
|
||||
observation := providerHealthObservation{
|
||||
source: "stall", evidenceHealth: evidence.providerHealth, decision: "inconclusive",
|
||||
}
|
||||
if lease == nil || lease.providerID == "" || lease.adapter == "" || lease.target == "" {
|
||||
observation.decision = "rejected_binding"
|
||||
return observation
|
||||
}
|
||||
key := providerRuntimeHealthKey{
|
||||
nodeID: lease.nodeID, generation: lease.generation, providerID: lease.providerID,
|
||||
}
|
||||
overlay := m.runtimeHealth[key]
|
||||
observation.fromHealth = runtimeOverlayHealth(overlay)
|
||||
observation.toHealth = observation.fromHealth
|
||||
if overlay != nil && evidence.sequence <= overlay.observationSeq {
|
||||
observation.decision = "rejected_stale"
|
||||
return observation
|
||||
}
|
||||
if overlay == nil {
|
||||
overlay = &providerRuntimeHealthOverlay{}
|
||||
m.runtimeHealth[key] = overlay
|
||||
}
|
||||
overlay.observationSeq = evidence.sequence
|
||||
if evidence.providerHealth == string(runtime.ProviderStatusUnavailable) {
|
||||
overlay.adapter = lease.adapter
|
||||
overlay.target = lease.target
|
||||
overlay.unavailable = true
|
||||
}
|
||||
observation.decision = "applied"
|
||||
observation.toHealth = runtimeOverlayHealth(overlay)
|
||||
observation.stateChanged = observation.fromHealth != observation.toHealth
|
||||
return observation
|
||||
}
|
||||
|
||||
// settleReceivedTerminal validates authoritative reception identity against the
|
||||
// immutable lease before any correctness state changes. A current terminal
|
||||
// releases once even when its optional health evidence is missing or rejected.
|
||||
// A mismatched node/generation is rejected and cannot release another owner's
|
||||
// lease. For accepted bound stall evidence, handoff annotation, any fresh overlay
|
||||
// transition, release, and queue pumping all occur under m.mu.
|
||||
func (m *modelQueueManager) settleReceivedTerminal(nodeID string, generation uint64, runID string, failure *iop.ExecutionFailure, envelopeMetadata *map[string]string) receivedTerminalDisposition {
|
||||
// releases once; failure details never alter provider health or candidate
|
||||
// selection. A mismatched node/generation is rejected and cannot release
|
||||
// another owner's lease.
|
||||
func (m *modelQueueManager) settleReceivedTerminal(nodeID string, generation uint64, runID string) receivedTerminalDisposition {
|
||||
if runID == "" {
|
||||
return receivedTerminalUntracked
|
||||
}
|
||||
|
|
@ -183,130 +87,13 @@ func (m *modelQueueManager) settleReceivedTerminal(nodeID string, generation uin
|
|||
return receivedTerminalRejected
|
||||
}
|
||||
|
||||
var observation *providerHealthObservation
|
||||
if evidence, ok := parseReceivedHealthEvidence(runID, failure); ok {
|
||||
metadata := failure.GetMetadata()
|
||||
if lease.providerID != "" && metadata["adapter"] == lease.adapter && metadata["target"] == lease.target {
|
||||
// Handoff confirms authoritative reception, immutable lease binding,
|
||||
// and the local attempt fence. Sequence freshness governs only the
|
||||
// provider-wide overlay; an out-of-order terminal still carries its
|
||||
// request-local handoff and still releases its own lease.
|
||||
annotateRecoveryHandoff(failure, envelopeMetadata, lease.providerID, evidence.providerHealth)
|
||||
result := m.applyReceivedHealthEvidenceLocked(lease, evidence)
|
||||
observation = &result
|
||||
} else {
|
||||
result := providerHealthObservation{source: "stall", evidenceHealth: evidence.providerHealth, decision: "rejected_binding"}
|
||||
observation = &result
|
||||
}
|
||||
}
|
||||
|
||||
if m.releaseLeaseLocked(leaseID) {
|
||||
m.pumpAllLocked()
|
||||
}
|
||||
m.mu.Unlock()
|
||||
m.observeProviderHealth(observation)
|
||||
return receivedTerminalAccepted
|
||||
}
|
||||
|
||||
// resolveCurrentProbeProviderLocked resolves CAPABILITIES evidence against the
|
||||
// authoritative current Node provider catalog. Runtime overlays are a health
|
||||
// projection, not an identity source: a healthy sibling provider with the same
|
||||
// adapter/target must make the evidence ambiguous as well. Must be called with
|
||||
// m.mu held.
|
||||
func (m *modelQueueManager) resolveCurrentProbeProviderLocked(nodeID, adapter, target string) (string, bool) {
|
||||
if m.store == nil {
|
||||
return "", false
|
||||
}
|
||||
record, ok := m.store.FindByID(nodeID)
|
||||
if !ok || record == nil {
|
||||
return "", false
|
||||
}
|
||||
|
||||
matchedProviderID := ""
|
||||
for _, provider := range record.Providers {
|
||||
if provider.ID == "" || providerAdapterKey(provider) != adapter || !providerCanServe(provider, target) {
|
||||
continue
|
||||
}
|
||||
if matchedProviderID != "" {
|
||||
return "", false
|
||||
}
|
||||
matchedProviderID = provider.ID
|
||||
}
|
||||
return matchedProviderID, matchedProviderID != ""
|
||||
}
|
||||
|
||||
// applyProviderProbeEvidence offers one CAPABILITIES health observation to the
|
||||
// current provider catalog. The exact adapter/target must identify one and only
|
||||
// one current-generation provider. A strictly newer available observation is
|
||||
// recorded even if the provider is already effectively available, so a delayed
|
||||
// lower-sequence terminal cannot later mark it unavailable. Only an actual
|
||||
// unavailable-to-available transition pumps the queue and reports recovery.
|
||||
func (m *modelQueueManager) applyProviderProbeEvidence(nodeID string, generation uint64, adapter, target string, status runtime.ProviderStatus, sequence uint64, isCurrentOwner func() bool) bool {
|
||||
observation := providerHealthObservation{source: "probe", evidenceHealth: string(status), decision: "inconclusive"}
|
||||
if nodeID == "" || generation == 0 || adapter == "" || target == "" ||
|
||||
status != runtime.ProviderStatusAvailable || sequence == 0 {
|
||||
m.observeProviderHealth(&observation)
|
||||
return false
|
||||
}
|
||||
m.mu.Lock()
|
||||
if isCurrentOwner != nil && !isCurrentOwner() {
|
||||
m.mu.Unlock()
|
||||
observation.decision = "rejected_binding"
|
||||
m.observeProviderHealth(&observation)
|
||||
return false
|
||||
}
|
||||
|
||||
providerID, ok := m.resolveCurrentProbeProviderLocked(nodeID, adapter, target)
|
||||
if !ok {
|
||||
m.mu.Unlock()
|
||||
observation.decision = "rejected_ambiguous"
|
||||
m.observeProviderHealth(&observation)
|
||||
return false
|
||||
}
|
||||
key := providerRuntimeHealthKey{nodeID: nodeID, generation: generation, providerID: providerID}
|
||||
overlay := m.runtimeHealth[key]
|
||||
observation.fromHealth = runtimeOverlayHealth(overlay)
|
||||
observation.toHealth = observation.fromHealth
|
||||
if overlay != nil && sequence <= overlay.observationSeq {
|
||||
m.mu.Unlock()
|
||||
observation.decision = "rejected_stale"
|
||||
m.observeProviderHealth(&observation)
|
||||
return false
|
||||
}
|
||||
if overlay == nil {
|
||||
overlay = &providerRuntimeHealthOverlay{}
|
||||
m.runtimeHealth[key] = overlay
|
||||
}
|
||||
recovered := overlay.unavailable && overlay.adapter == adapter && overlay.target == target
|
||||
overlay.observationSeq = sequence
|
||||
if overlay.unavailable && !recovered {
|
||||
m.mu.Unlock()
|
||||
observation.decision = "rejected_binding"
|
||||
observation.toHealth = runtimeOverlayHealth(overlay)
|
||||
m.observeProviderHealth(&observation)
|
||||
return false
|
||||
}
|
||||
overlay.adapter = adapter
|
||||
overlay.target = target
|
||||
overlay.unavailable = false
|
||||
if recovered {
|
||||
m.pumpAllLocked()
|
||||
}
|
||||
observation.decision = "applied"
|
||||
observation.toHealth = runtimeOverlayHealth(overlay)
|
||||
observation.stateChanged = observation.fromHealth != observation.toHealth
|
||||
m.mu.Unlock()
|
||||
m.observeProviderHealth(&observation)
|
||||
return recovered
|
||||
}
|
||||
|
||||
func runtimeOverlayHealth(overlay *providerRuntimeHealthOverlay) string {
|
||||
if overlay != nil && overlay.unavailable {
|
||||
return string(runtime.ProviderStatusUnavailable)
|
||||
}
|
||||
return string(runtime.ProviderStatusAvailable)
|
||||
}
|
||||
|
||||
// fenceNodeGenerationLocked fences the disconnected connection identified by
|
||||
// (nodeID, generation): it settles leases through the exactly-once release path
|
||||
// so each provider resource counter is returned per lease, and marks matching
|
||||
|
|
@ -333,11 +120,6 @@ func (m *modelQueueManager) fenceNodeGenerationLocked(nodeID string, generation
|
|||
}
|
||||
res.orphan = true
|
||||
}
|
||||
for key := range m.runtimeHealth {
|
||||
if key.nodeID == nodeID && (fenceAll || key.generation <= generation) {
|
||||
delete(m.runtimeHealth, key)
|
||||
}
|
||||
}
|
||||
return settledLease
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -46,22 +46,8 @@ func (m *modelQueueManager) getSnapshotForNodeLocked(nodeID string, rec *edgenod
|
|||
|
||||
capVal := prov.Capacity
|
||||
inflight, queued, longInflight, longQueued := m.providerSnapshotStatsLocked(nodeID, prov.ID, pressure)
|
||||
generation := uint64(0)
|
||||
if resource := m.resources[providerResourceKey{nodeID: nodeID, providerID: prov.ID}]; resource != nil {
|
||||
generation = resource.generation
|
||||
}
|
||||
runtimeUnavailable := connected && m.providerRuntimeUnavailableLocked(nodeID, generation, prov.ID)
|
||||
status := effectiveStatus(connected)
|
||||
health := effectiveHealth(connected, prov.Health)
|
||||
if runtimeUnavailable {
|
||||
status = "unavailable"
|
||||
health = "unavailable"
|
||||
capVal = 0
|
||||
inflight = 0
|
||||
queued = 0
|
||||
longInflight = 0
|
||||
longQueued = 0
|
||||
}
|
||||
|
||||
snaps = append(snaps, &iop.ProviderSnapshot{
|
||||
Adapter: prov.Adapter,
|
||||
|
|
@ -75,13 +61,13 @@ func (m *modelQueueManager) getSnapshotForNodeLocked(nodeID string, rec *edgenod
|
|||
Category: string(prov.Category),
|
||||
ServedModels: servedModels,
|
||||
LoadRatio: func() float32 {
|
||||
if !connected || runtimeUnavailable || capVal <= 0 {
|
||||
if !connected || capVal <= 0 {
|
||||
return 0
|
||||
}
|
||||
return float32(inflight) / float32(capVal)
|
||||
}(),
|
||||
LifecycleCapabilities: lifecycleCaps,
|
||||
LongContextCapacity: int32(effectiveCount(connected && !runtimeUnavailable, prov.LongContextCapacity)),
|
||||
LongContextCapacity: int32(effectiveCount(connected, prov.LongContextCapacity)),
|
||||
LongInFlight: int32(effectiveCount(connected, longInflight)),
|
||||
LongQueued: int32(effectiveCount(connected, longQueued)),
|
||||
})
|
||||
|
|
|
|||
|
|
@ -99,8 +99,7 @@ type candidateNode struct {
|
|||
// resolved before the owning connection disconnected or was superseded by a
|
||||
// reconnect — out of reserve and dispatch handoff. Zero means untracked
|
||||
// (legacy/direct candidates and hand-built fixtures) and is never fenced.
|
||||
generation uint64
|
||||
responseStallTimeoutMS int64
|
||||
generation uint64
|
||||
}
|
||||
|
||||
// slotKey returns a unique slot key for inflight accounting.
|
||||
|
|
@ -187,56 +186,11 @@ type providerLease struct {
|
|||
runID string
|
||||
}
|
||||
|
||||
// providerRuntimeHealthKey scopes runtime health to one provider on one Node
|
||||
// connection. A reconnect receives a new generation and therefore never
|
||||
// inherits health evidence observed on the superseded connection.
|
||||
type providerRuntimeHealthKey struct {
|
||||
nodeID string
|
||||
generation uint64
|
||||
providerID string
|
||||
}
|
||||
|
||||
// providerRuntimeHealthOverlay is deliberately separate from the config-owned
|
||||
// provider resource. observationSeq is the high-water mark for every validated
|
||||
// bound observation, while unavailable changes effective admission/snapshot
|
||||
// health only. adapter and target retain the exact binding that lowered the
|
||||
// provider so only the same exact-target status probe may recover it.
|
||||
type providerRuntimeHealthOverlay struct {
|
||||
adapter string
|
||||
target string
|
||||
observationSeq uint64
|
||||
unavailable bool
|
||||
}
|
||||
|
||||
type admitResult struct {
|
||||
candidate *candidateNode
|
||||
err error
|
||||
}
|
||||
|
||||
// recoveryCandidatePolicy carries the request-local avoided-provider recovery
|
||||
// hint through immediate admission and every queued re-resolution. It is a pure
|
||||
// value with no persistence beyond the request/queue item lifetime: the queue
|
||||
// stores it on the pending item only so the pump reapplies the identical
|
||||
// preference the caller submitted.
|
||||
//
|
||||
// The zero value (empty avoidProviderID, false allowAvoidedProviderFallback)
|
||||
// disables recovery entirely, so every non-recovery admission path preserves the
|
||||
// current candidate-selection behavior.
|
||||
type recoveryCandidatePolicy struct {
|
||||
// avoidProviderID, when non-empty, marks the provider the caller wants to
|
||||
// avoid. A runtime-eligible alternate is always preferred over it.
|
||||
avoidProviderID string
|
||||
// allowAvoidedProviderFallback permits re-selecting the avoided provider,
|
||||
// but only when no runtime-eligible alternate remains. It is the caller's
|
||||
// explicit, probe-backed permission and never derived from overlay state.
|
||||
allowAvoidedProviderFallback bool
|
||||
}
|
||||
|
||||
// active reports whether the policy expresses an avoided-provider preference.
|
||||
func (p recoveryCandidatePolicy) active() bool {
|
||||
return p.avoidProviderID != ""
|
||||
}
|
||||
|
||||
// queueItem is one pending admission. candidates carry the request's resource
|
||||
// identity (node, provider, served target) only: capacity, long-context capacity,
|
||||
// priority, and the enabled switch are re-read from live state at dispatch time,
|
||||
|
|
@ -252,9 +206,6 @@ type queueItem struct {
|
|||
providerPool bool // true when this item is enqueued under the provider-pool policy scope
|
||||
reason string
|
||||
enqueueSeq uint64
|
||||
// recovery is the request-local avoided-provider preference reapplied on
|
||||
// every pump re-resolution. Zero value for non-recovery admissions.
|
||||
recovery recoveryCandidatePolicy
|
||||
}
|
||||
|
||||
type modelQueueGroup struct {
|
||||
|
|
@ -269,11 +220,8 @@ type modelQueueGroup struct {
|
|||
}
|
||||
|
||||
type modelQueueManager struct {
|
||||
mu sync.Mutex
|
||||
// healthObserver receives immutable post-decision projections only. It is
|
||||
// never called while mu is held.
|
||||
healthObserver providerHealthObserver
|
||||
groups map[string]*modelQueueGroup
|
||||
mu sync.Mutex
|
||||
groups map[string]*modelQueueGroup
|
||||
// leases holds every live lease by id. Admission inserts under the same
|
||||
// critical section that reserves the resource, and release deletes under the
|
||||
// same critical section that frees it, so the lease map is the single source
|
||||
|
|
@ -289,10 +237,6 @@ type modelQueueManager struct {
|
|||
enqueueSeq uint64
|
||||
store *edgenode.NodeStore
|
||||
resources map[providerResourceKey]*providerResourceState
|
||||
// runtimeHealth is a generation-scoped overlay. It never mutates NodeStore
|
||||
// provider config and is guarded by the same lock as leases/resources so
|
||||
// health transitions and admission observe one linearized state.
|
||||
runtimeHealth map[providerRuntimeHealthKey]*providerRuntimeHealthOverlay
|
||||
// providerPoolPolicy is the canonical root policy shared by every
|
||||
// provider-pool admission. It replaces the legacy per-provider first-encounter
|
||||
// heuristic and makes max_queue a hard cap across all model groups for the
|
||||
|
|
@ -303,13 +247,11 @@ type modelQueueManager struct {
|
|||
|
||||
func newModelQueueManager(store *edgenode.NodeStore) *modelQueueManager {
|
||||
return &modelQueueManager{
|
||||
healthObserver: defaultHealthObserver(),
|
||||
groups: make(map[string]*modelQueueGroup),
|
||||
leases: make(map[uint64]*providerLease),
|
||||
leaseByRun: make(map[string]uint64),
|
||||
store: store,
|
||||
resources: make(map[providerResourceKey]*providerResourceState),
|
||||
runtimeHealth: make(map[providerRuntimeHealthKey]*providerRuntimeHealthOverlay),
|
||||
groups: make(map[string]*modelQueueGroup),
|
||||
leases: make(map[uint64]*providerLease),
|
||||
leaseByRun: make(map[string]uint64),
|
||||
store: store,
|
||||
resources: make(map[providerResourceKey]*providerResourceState),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -493,13 +435,6 @@ func (m *modelQueueManager) activateNodeGenerationLocked(nodeID string, generati
|
|||
if nodeID == "" {
|
||||
return
|
||||
}
|
||||
// Preserve a same-generation duplicate activation, but discard every older
|
||||
// generation's runtime evidence. Configuration remains untouched.
|
||||
for key := range m.runtimeHealth {
|
||||
if key.nodeID == nodeID && key.generation != generation {
|
||||
delete(m.runtimeHealth, key)
|
||||
}
|
||||
}
|
||||
for _, res := range m.resources {
|
||||
if res.nodeID != nodeID {
|
||||
continue
|
||||
|
|
@ -511,37 +446,6 @@ func (m *modelQueueManager) activateNodeGenerationLocked(nodeID string, generati
|
|||
}
|
||||
}
|
||||
|
||||
func (m *modelQueueManager) providerRuntimeUnavailableLocked(nodeID string, generation uint64, providerID string) bool {
|
||||
if providerID == "" {
|
||||
return false
|
||||
}
|
||||
overlay := m.runtimeHealth[providerRuntimeHealthKey{
|
||||
nodeID: nodeID, generation: generation, providerID: providerID,
|
||||
}]
|
||||
return overlay != nil && overlay.unavailable
|
||||
}
|
||||
|
||||
func (m *modelQueueManager) candidateRuntimeHealthyLocked(candidate *candidateNode) bool {
|
||||
if candidate == nil || candidate.entry == nil || candidate.providerID == "" {
|
||||
return true
|
||||
}
|
||||
return !m.providerRuntimeUnavailableLocked(candidate.entry.NodeID, candidate.generation, candidate.providerID)
|
||||
}
|
||||
|
||||
// filterRuntimeHealthyCandidatesLocked removes runtime-unavailable provider
|
||||
// candidates while preserving candidate order and legacy candidates. It is
|
||||
// used by both immediate admission and queued re-resolution so effective
|
||||
// provider eligibility has one source of truth.
|
||||
func (m *modelQueueManager) filterRuntimeHealthyCandidatesLocked(candidates []candidateNode) []candidateNode {
|
||||
filtered := make([]candidateNode, 0, len(candidates))
|
||||
for i := range candidates {
|
||||
if m.candidateRuntimeHealthyLocked(&candidates[i]) {
|
||||
filtered = append(filtered, candidates[i])
|
||||
}
|
||||
}
|
||||
return filtered
|
||||
}
|
||||
|
||||
// findLastColon returns the index of the last ':' in s, or -1 if not found.
|
||||
func findLastColon(s string) int {
|
||||
for i := len(s) - 1; i >= 0; i-- {
|
||||
|
|
|
|||
|
|
@ -4,12 +4,10 @@ import (
|
|||
"context"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
|
|
@ -137,21 +135,6 @@ func (s *Service) sendNodeCommand(req NodeCommandRequestSpec, cmdType iop.NodeCo
|
|||
if resp.GetError() != "" {
|
||||
return NodeCommandView{}, fmt.Errorf("node reported error: %s", resp.GetError())
|
||||
}
|
||||
if cmdType == iop.NodeCommandType_NODE_COMMAND_TYPE_CAPABILITIES && s.queue != nil {
|
||||
if evidence, ok := capabilitiesProbeEvidenceFromResponse(commandReq, resp); ok {
|
||||
s.queue.applyProviderProbeEvidence(
|
||||
entry.NodeID,
|
||||
entry.ConnectionGeneration,
|
||||
evidence.adapter,
|
||||
evidence.target,
|
||||
evidence.status,
|
||||
evidence.sequence,
|
||||
func() bool {
|
||||
return s.registry != nil && s.registry.IsCurrentOwnerGeneration(entry.NodeID, entry.ConnectionGeneration)
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
return NodeCommandView{
|
||||
NodeID: entry.NodeID,
|
||||
NodeLabel: nodeLabel(entry),
|
||||
|
|
@ -163,39 +146,3 @@ func (s *Service) sendNodeCommand(req NodeCommandRequestSpec, cmdType iop.NodeCo
|
|||
ProviderSnapshots: resp.GetProviderSnapshots(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
type capabilitiesProbeEvidence struct {
|
||||
adapter string
|
||||
target string
|
||||
status runtime.ProviderStatus
|
||||
sequence uint64
|
||||
}
|
||||
|
||||
// capabilitiesProbeEvidenceFromResponse accepts only the stable, exact binding
|
||||
// emitted by the Node CAPABILITIES probe. Older Nodes omit the sequence and are
|
||||
// harmless no-ops. Empty/malformed identity, response-envelope mismatch, and
|
||||
// non-baseline status values also fail closed.
|
||||
func capabilitiesProbeEvidenceFromResponse(req *iop.NodeCommandRequest, resp *iop.NodeCommandResponse) (capabilitiesProbeEvidence, bool) {
|
||||
if req == nil || resp == nil || req.GetType() != iop.NodeCommandType_NODE_COMMAND_TYPE_CAPABILITIES ||
|
||||
resp.GetType() != iop.NodeCommandType_NODE_COMMAND_TYPE_CAPABILITIES {
|
||||
return capabilitiesProbeEvidence{}, false
|
||||
}
|
||||
adapter := strings.TrimSpace(req.GetAdapter())
|
||||
target := strings.TrimSpace(req.GetTarget())
|
||||
if adapter == "" || target == "" || resp.GetAdapter() != adapter || resp.GetTarget() != target {
|
||||
return capabilitiesProbeEvidence{}, false
|
||||
}
|
||||
result := resp.GetResult()
|
||||
if strings.TrimSpace(result["adapter_key"]) != adapter || strings.TrimSpace(result["target"]) != target {
|
||||
return capabilitiesProbeEvidence{}, false
|
||||
}
|
||||
sequence, err := strconv.ParseUint(result["health_observation_seq"], 10, 64)
|
||||
if err != nil || sequence == 0 {
|
||||
return capabilitiesProbeEvidence{}, false
|
||||
}
|
||||
status := runtime.ProviderStatus(strings.TrimSpace(result["provider_status"]))
|
||||
if normalized := runtime.NormalizeProviderStatus(status); normalized != status {
|
||||
return capabilitiesProbeEvidence{}, false
|
||||
}
|
||||
return capabilitiesProbeEvidence{adapter: adapter, target: target, status: status, sequence: sequence}, true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,176 +0,0 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/zap"
|
||||
)
|
||||
|
||||
const (
|
||||
providerHealthEvidenceMetric = "iop_edge_provider_health_evidence_total"
|
||||
providerHealthTransitionMetric = "iop_edge_provider_health_transitions_total"
|
||||
providerHealthObservationLogKey = "edge_provider_health_observation"
|
||||
)
|
||||
|
||||
// providerHealthObservation is an immutable, identity-free projection of a
|
||||
// health-overlay decision. The queue constructs it while holding its lock and
|
||||
// sends it to the observer only after the decision, release, and pump finish.
|
||||
type providerHealthObservation struct {
|
||||
source string
|
||||
evidenceHealth string
|
||||
decision string
|
||||
fromHealth string
|
||||
toHealth string
|
||||
stateChanged bool
|
||||
}
|
||||
|
||||
type providerHealthObserver interface {
|
||||
Observe(providerHealthObservation)
|
||||
}
|
||||
|
||||
type providerHealthMetrics struct {
|
||||
evidence *prometheus.CounterVec
|
||||
transitions *prometheus.CounterVec
|
||||
}
|
||||
|
||||
type providerHealthObservability struct {
|
||||
metrics *providerHealthMetrics
|
||||
mu sync.RWMutex
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
var defaultProviderHealthMetrics struct {
|
||||
once sync.Once
|
||||
metrics *providerHealthMetrics
|
||||
}
|
||||
|
||||
func defaultHealthObserver() providerHealthObserver {
|
||||
return &providerHealthObservability{
|
||||
metrics: defaultProviderHealthCollectorSet(),
|
||||
logger: zap.NewNop(),
|
||||
}
|
||||
}
|
||||
|
||||
func defaultProviderHealthCollectorSet() *providerHealthMetrics {
|
||||
defaultProviderHealthMetrics.once.Do(func() {
|
||||
defaultProviderHealthMetrics.metrics = newProviderHealthMetrics(prometheus.DefaultRegisterer)
|
||||
})
|
||||
return defaultProviderHealthMetrics.metrics
|
||||
}
|
||||
|
||||
// newProviderHealthObservability creates an isolated observer for tests when
|
||||
// reg is a private registry. Production callers use defaultHealthObserver.
|
||||
func newProviderHealthObservability(reg prometheus.Registerer, logger *zap.Logger) *providerHealthObservability {
|
||||
if logger == nil {
|
||||
logger = zap.NewNop()
|
||||
}
|
||||
return &providerHealthObservability{metrics: newProviderHealthMetrics(reg), logger: logger}
|
||||
}
|
||||
|
||||
func newProviderHealthMetrics(reg prometheus.Registerer) *providerHealthMetrics {
|
||||
metrics := &providerHealthMetrics{
|
||||
evidence: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: providerHealthEvidenceMetric,
|
||||
Help: "Authoritative Edge provider health-overlay evidence decisions.",
|
||||
}, []string{"source", "evidence_health", "decision"}),
|
||||
transitions: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Name: providerHealthTransitionMetric,
|
||||
Help: "Authoritative Edge provider health-overlay state transitions.",
|
||||
}, []string{"from_health", "to_health"}),
|
||||
}
|
||||
if reg == nil {
|
||||
return metrics
|
||||
}
|
||||
metrics.evidence = registerProviderHealthCounter(reg, metrics.evidence)
|
||||
metrics.transitions = registerProviderHealthCounter(reg, metrics.transitions)
|
||||
return metrics
|
||||
}
|
||||
|
||||
func registerProviderHealthCounter(reg prometheus.Registerer, counter *prometheus.CounterVec) *prometheus.CounterVec {
|
||||
if err := reg.Register(counter); err != nil {
|
||||
if alreadyRegistered, ok := err.(prometheus.AlreadyRegisteredError); ok {
|
||||
if existing, ok := alreadyRegistered.ExistingCollector.(*prometheus.CounterVec); ok {
|
||||
return existing
|
||||
}
|
||||
}
|
||||
}
|
||||
return counter
|
||||
}
|
||||
|
||||
func (o *providerHealthObservability) SetLogger(logger *zap.Logger) {
|
||||
if o == nil || logger == nil {
|
||||
return
|
||||
}
|
||||
o.mu.Lock()
|
||||
o.logger = logger
|
||||
o.mu.Unlock()
|
||||
}
|
||||
|
||||
func (o *providerHealthObservability) Observe(observation providerHealthObservation) {
|
||||
if o == nil || o.metrics == nil {
|
||||
return
|
||||
}
|
||||
// Prometheus counters do not return errors. The projection is deliberately
|
||||
// bounded before it reaches either metrics or logs.
|
||||
source := normalizeProviderHealthSource(observation.source)
|
||||
evidenceHealth := normalizeProviderHealth(observation.evidenceHealth)
|
||||
decision := normalizeProviderHealthDecision(observation.decision)
|
||||
fromHealth := normalizeProviderHealth(observation.fromHealth)
|
||||
toHealth := normalizeProviderHealth(observation.toHealth)
|
||||
o.metrics.evidence.WithLabelValues(source, evidenceHealth, decision).Inc()
|
||||
if observation.stateChanged {
|
||||
o.metrics.transitions.WithLabelValues(fromHealth, toHealth).Inc()
|
||||
}
|
||||
o.mu.RLock()
|
||||
logger := o.logger
|
||||
o.mu.RUnlock()
|
||||
if logger == nil {
|
||||
return
|
||||
}
|
||||
logger.Info(providerHealthObservationLogKey,
|
||||
zap.String("source", source),
|
||||
zap.String("evidence_health", evidenceHealth),
|
||||
zap.String("decision", decision),
|
||||
zap.String("from_health", fromHealth),
|
||||
zap.String("to_health", toHealth),
|
||||
zap.Bool("state_changed", observation.stateChanged),
|
||||
)
|
||||
}
|
||||
|
||||
func normalizeProviderHealthSource(source string) string {
|
||||
switch source {
|
||||
case "stall", "probe":
|
||||
return source
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeProviderHealth(value string) string {
|
||||
switch value {
|
||||
case "available", "unavailable":
|
||||
return value
|
||||
default:
|
||||
return "unknown"
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeProviderHealthDecision(decision string) string {
|
||||
switch decision {
|
||||
case "applied", "rejected_stale", "rejected_binding", "rejected_ambiguous", "inconclusive":
|
||||
return decision
|
||||
default:
|
||||
return "inconclusive"
|
||||
}
|
||||
}
|
||||
|
||||
func (m *modelQueueManager) observeProviderHealth(observation *providerHealthObservation) {
|
||||
if observation == nil || m == nil || m.healthObserver == nil {
|
||||
return
|
||||
}
|
||||
// Observation is non-authoritative. A custom observer must not be able to
|
||||
// turn a released lease or pumped queue back into a failed terminal path.
|
||||
defer func() { _ = recover() }()
|
||||
m.healthObserver.Observe(*observation)
|
||||
}
|
||||
|
|
@ -1,397 +0,0 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest/observer"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
edgeevents "iop/apps/edge/internal/events"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
func installProviderHealthTestObserver(t *testing.T, svc *Service) (*prometheus.Registry, *observer.ObservedLogs) {
|
||||
t.Helper()
|
||||
registry := prometheus.NewRegistry()
|
||||
core, logs := observer.New(zap.InfoLevel)
|
||||
svc.queue.mu.Lock()
|
||||
svc.queue.healthObserver = newProviderHealthObservability(registry, zap.New(core))
|
||||
svc.queue.mu.Unlock()
|
||||
return registry, logs
|
||||
}
|
||||
|
||||
func metricValue(t *testing.T, registry *prometheus.Registry, name string, want map[string]string) float64 {
|
||||
t.Helper()
|
||||
families, err := registry.Gather()
|
||||
if err != nil {
|
||||
t.Fatalf("gather metrics: %v", err)
|
||||
}
|
||||
for _, family := range families {
|
||||
if family.GetName() != name {
|
||||
continue
|
||||
}
|
||||
for _, metric := range family.Metric {
|
||||
if metricHasLabels(metric, want) {
|
||||
return metric.GetCounter().GetValue()
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func metricHasLabels(metric *dto.Metric, want map[string]string) bool {
|
||||
if len(metric.Label) != len(want) {
|
||||
return false
|
||||
}
|
||||
for _, label := range metric.Label {
|
||||
if want[label.GetName()] != label.GetValue() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func assertPublicProviderSnapshot(t *testing.T, snapshots []NodeSnapshot, nodeID, providerID string, wantStatus, wantHealth string, wantCapacity int32) {
|
||||
t.Helper()
|
||||
for _, snap := range snapshots {
|
||||
if snap.NodeID != nodeID {
|
||||
continue
|
||||
}
|
||||
for _, ps := range snap.ProviderSnapshots {
|
||||
if ps.GetId() == providerID {
|
||||
if ps.GetStatus() != wantStatus || ps.GetHealth() != wantHealth || ps.GetCapacity() != wantCapacity {
|
||||
t.Fatalf("snapshot for %s/%s = (status=%q, health=%q, capacity=%d), want (%q, %q, %d)",
|
||||
nodeID, providerID, ps.GetStatus(), ps.GetHealth(), ps.GetCapacity(),
|
||||
wantStatus, wantHealth, wantCapacity)
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Fatalf("snapshot for %s/%s not found", nodeID, providerID)
|
||||
}
|
||||
|
||||
func TestProviderHealthObservability(t *testing.T) {
|
||||
for _, executionPath := range []string{"normalized", "tunnel"} {
|
||||
t.Run(executionPath, func(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() {
|
||||
_ = edgeConn.Close()
|
||||
_ = nodeConn.Close()
|
||||
})
|
||||
parserMap := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.NodeCommandRequest{}): func(data []byte) (proto.Message, error) {
|
||||
message := &iop.NodeCommandRequest{}
|
||||
return message, proto.Unmarshal(data, message)
|
||||
},
|
||||
toki.TypeNameOf(&iop.NodeCommandResponse{}): func(data []byte) (proto.Message, error) {
|
||||
message := &iop.NodeCommandResponse{}
|
||||
return message, proto.Unmarshal(data, message)
|
||||
},
|
||||
}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parserMap)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parserMap)
|
||||
toki.AddRequestListenerTyped(&nodeClient.Communicator, func(request *iop.NodeCommandRequest) (*iop.NodeCommandResponse, error) {
|
||||
return &iop.NodeCommandResponse{
|
||||
RequestId: request.GetRequestId(), Type: request.GetType(),
|
||||
Adapter: request.GetAdapter(), Target: request.GetTarget(), SessionId: request.GetSessionId(),
|
||||
Result: map[string]string{
|
||||
"adapter_key": request.GetAdapter(), "target": request.GetTarget(),
|
||||
"provider_status": "available", "health_observation_seq": "4",
|
||||
},
|
||||
}, nil
|
||||
})
|
||||
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, edgeClient)
|
||||
registry, logs := installProviderHealthTestObserver(t, svc)
|
||||
|
||||
// 1. Unavailable terminal (sequence 3)
|
||||
addBoundOverlayLease(t, svc.queue, "run-unhealthy", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
if executionPath == "normalized" {
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-unhealthy", overlayAdapter, overlayTarget, 3))
|
||||
} else {
|
||||
svc.HandleReceivedProviderTunnelFrame(entry.NodeID, entry.ConnectionGeneration, &iop.ProviderTunnelFrame{
|
||||
RunId: "run-unhealthy", TunnelId: "tunnel-observability",
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR,
|
||||
Failure: stallFailure("run-unhealthy", overlayAdapter, overlayTarget, "unavailable", "provider_unhealthy", 3),
|
||||
})
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 3)
|
||||
assertPublicProviderSnapshot(t, svc.ListNodeSnapshots(), entry.NodeID, overlayProviderID, "unavailable", "unavailable", 0)
|
||||
|
||||
// 2. Stale terminal (sequence 3) delivered through selected executionPath handler
|
||||
addBoundOverlayLease(t, svc.queue, "run-stale", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
stale := stallFailure("run-stale", overlayAdapter, overlayTarget, "available", "request_stalled", 3)
|
||||
if executionPath == "normalized" {
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, &iop.RunEvent{RunId: "run-stale", Type: "error", Failure: stale})
|
||||
} else {
|
||||
svc.HandleReceivedProviderTunnelFrame(entry.NodeID, entry.ConnectionGeneration, &iop.ProviderTunnelFrame{
|
||||
RunId: "run-stale", TunnelId: "tunnel-stale",
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR,
|
||||
Failure: stale,
|
||||
})
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 3)
|
||||
assertPublicProviderSnapshot(t, svc.ListNodeSnapshots(), entry.NodeID, overlayProviderID, "unavailable", "unavailable", 0)
|
||||
|
||||
// 3. Recovery via production Capabilities path (higher-sequence probe 4)
|
||||
if _, err := svc.Capabilities(context.Background(), NodeCommandRequestSpec{
|
||||
NodeRef: entry.NodeID, Adapter: overlayAdapter, Target: overlayTarget,
|
||||
}); err != nil {
|
||||
t.Fatalf("Capabilities recovery: %v", err)
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, false, 4)
|
||||
assertPublicProviderSnapshot(t, svc.ListNodeSnapshots(), entry.NodeID, overlayProviderID, "available", "available", 1)
|
||||
|
||||
if got := metricValue(t, registry, providerHealthEvidenceMetric, map[string]string{"source": "stall", "evidence_health": "unavailable", "decision": "applied"}); got != 1 {
|
||||
t.Fatalf("applied unhealthy metric = %v, want 1", got)
|
||||
}
|
||||
if got := metricValue(t, registry, providerHealthEvidenceMetric, map[string]string{"source": "stall", "evidence_health": "available", "decision": "rejected_stale"}); got != 1 {
|
||||
t.Fatalf("stale rejection metric = %v, want 1", got)
|
||||
}
|
||||
if got := metricValue(t, registry, providerHealthEvidenceMetric, map[string]string{"source": "probe", "evidence_health": "available", "decision": "applied"}); got != 1 {
|
||||
t.Fatalf("recovery metric = %v, want 1", got)
|
||||
}
|
||||
if got := metricValue(t, registry, providerHealthTransitionMetric, map[string]string{"from_health": "available", "to_health": "unavailable"}); got != 1 {
|
||||
t.Fatalf("unhealthy transition metric = %v, want 1", got)
|
||||
}
|
||||
if got := metricValue(t, registry, providerHealthTransitionMetric, map[string]string{"from_health": "unavailable", "to_health": "available"}); got != 1 {
|
||||
t.Fatalf("recovery transition metric = %v, want 1", got)
|
||||
}
|
||||
|
||||
entries := logs.All()
|
||||
if len(entries) != 3 {
|
||||
t.Fatalf("health observation logs = %d, want 3", len(entries))
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if entry.Message != providerHealthObservationLogKey {
|
||||
t.Fatalf("unexpected log message %q", entry.Message)
|
||||
}
|
||||
for _, field := range entry.Context {
|
||||
if strings.Contains(field.Key, "provider") || strings.Contains(field.Key, "node") || strings.Contains(field.Key, "run") || strings.Contains(field.Key, "session") || strings.Contains(field.Key, "adapter") || strings.Contains(field.Key, "target") {
|
||||
t.Fatalf("identity-bearing log field %q", field.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("duplicate evidence is observed exactly once", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
registry, _ := installProviderHealthTestObserver(t, svc)
|
||||
addBoundOverlayLease(t, svc.queue, "run-once", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent("run-once", overlayAdapter, overlayTarget, 1)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
if got := metricValue(t, registry, providerHealthEvidenceMetric, map[string]string{"source": "stall", "evidence_health": "unavailable", "decision": "applied"}); got != 1 {
|
||||
t.Fatalf("duplicate terminal observations = %v, want 1", got)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("default collectors are reused", func(t *testing.T) {
|
||||
for range 4 {
|
||||
_ = New(nil, edgeevents.NewBus())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
type blockingProviderHealthObserver struct {
|
||||
started chan struct{}
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
func (o *blockingProviderHealthObserver) Observe(providerHealthObservation) {
|
||||
close(o.started)
|
||||
<-o.release
|
||||
}
|
||||
|
||||
func TestProviderHealthObservabilityRunsAfterQueueUnlock(t *testing.T) {
|
||||
svc, entry, record := newProviderHealthOverlayService(t, nil)
|
||||
blocking := &blockingProviderHealthObserver{started: make(chan struct{}), release: make(chan struct{})}
|
||||
svc.queue.mu.Lock()
|
||||
svc.queue.healthObserver = blocking
|
||||
svc.queue.mu.Unlock()
|
||||
addBoundOverlayLease(t, svc.queue, "run-lock", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-lock", overlayAdapter, overlayTarget, 1))
|
||||
close(done)
|
||||
}()
|
||||
select {
|
||||
case <-blocking.started:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("observer was not called")
|
||||
}
|
||||
// This snapshot needs modelQueueManager.mu. It must complete while the
|
||||
// observer remains blocked, proving the post-decision placement.
|
||||
snapshotDone := make(chan struct{})
|
||||
go func() {
|
||||
_ = svc.queue.getSnapshotForNode(entry.NodeID, record, true)
|
||||
close(snapshotDone)
|
||||
}()
|
||||
select {
|
||||
case <-snapshotDone:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("observer retained modelQueueManager.mu")
|
||||
}
|
||||
close(blocking.release)
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("terminal did not return after observer release")
|
||||
}
|
||||
if leaseCount(svc.queue) != 0 {
|
||||
t.Fatal("blocking observer prevented lease release")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderHealthObservabilityDoesNotExposeSentinels(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
registry, logs := installProviderHealthTestObserver(t, svc)
|
||||
|
||||
forbiddenValues := []string{
|
||||
"SECRET_NODE_ID_12345",
|
||||
"SECRET_PROVIDER_ID_67890",
|
||||
"SECRET_RUN_ID_ABCDE",
|
||||
"SECRET_SESSION_ID_FGHIJ",
|
||||
"SECRET_ADAPTER_KEY_KLMNO",
|
||||
"SECRET_TARGET_MODEL_PQRST",
|
||||
"SECRET_ERROR_MESSAGE_UVWXY",
|
||||
"SECRET_PROMPT_BODY_Z0123",
|
||||
"SECRET_BEARER_TOKEN_45678",
|
||||
"SECRET_EVENT_NODE_ID_11111",
|
||||
"SECRET_EVENT_SESSION_ID_22222",
|
||||
"SECRET_EVENT_MESSAGE_33333",
|
||||
"SECRET_EVENT_ERROR_44444",
|
||||
"SECRET_EVENT_DELTA_99999",
|
||||
"SECRET_EVENT_NODE_ALIAS_AAAAA",
|
||||
"SECRET_FRAME_RUN_ID_55555",
|
||||
"SECRET_FRAME_NODE_ID_66666",
|
||||
"SECRET_FRAME_TUNNEL_ID_BBBBB",
|
||||
"SECRET_FRAME_NODE_ALIAS_CCCCC",
|
||||
"SECRET_HEADER_KEY_77777",
|
||||
"SECRET_FRAME_ERROR_88888",
|
||||
}
|
||||
|
||||
addBoundOverlayLease(t, svc.queue, forbiddenValues[2], overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
|
||||
failure := &iop.ExecutionFailure{
|
||||
Code: string(runtime.FailureCodeResponseStalled),
|
||||
Message: forbiddenValues[6],
|
||||
Retryable: true,
|
||||
Metadata: map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled),
|
||||
"provider_health": "unavailable",
|
||||
"liveness_classification": "provider_unhealthy",
|
||||
"idle_duration_ms": "300000",
|
||||
"run_id": forbiddenValues[2],
|
||||
"session_id": forbiddenValues[3],
|
||||
"adapter": overlayAdapter,
|
||||
"target": overlayTarget,
|
||||
"health_observation_seq": "1",
|
||||
"node_id": forbiddenValues[0],
|
||||
"provider_id": forbiddenValues[1],
|
||||
"raw_adapter": forbiddenValues[4],
|
||||
"raw_target": forbiddenValues[5],
|
||||
"body": forbiddenValues[7],
|
||||
"authorization": forbiddenValues[8],
|
||||
},
|
||||
}
|
||||
|
||||
event := &iop.RunEvent{
|
||||
RunId: forbiddenValues[2],
|
||||
Type: "error",
|
||||
Delta: "SECRET_EVENT_DELTA_99999",
|
||||
NodeId: "SECRET_EVENT_NODE_ID_11111",
|
||||
NodeAlias: "SECRET_EVENT_NODE_ALIAS_AAAAA",
|
||||
SessionId: "SECRET_EVENT_SESSION_ID_22222",
|
||||
Message: "SECRET_EVENT_MESSAGE_33333",
|
||||
Error: "SECRET_EVENT_ERROR_44444",
|
||||
Failure: failure,
|
||||
Metadata: failure.Metadata,
|
||||
}
|
||||
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
|
||||
frameRunID := "SECRET_FRAME_RUN_ID_55555"
|
||||
addBoundOverlayLease(t, svc.queue, frameRunID, overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
|
||||
frameFailure := &iop.ExecutionFailure{
|
||||
Code: string(runtime.FailureCodeResponseStalled),
|
||||
Message: forbiddenValues[6],
|
||||
Retryable: true,
|
||||
Metadata: map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled),
|
||||
"provider_health": "unavailable",
|
||||
"liveness_classification": "provider_unhealthy",
|
||||
"idle_duration_ms": "300000",
|
||||
"run_id": frameRunID,
|
||||
"session_id": forbiddenValues[3],
|
||||
"adapter": overlayAdapter,
|
||||
"target": overlayTarget,
|
||||
"health_observation_seq": "2",
|
||||
"node_id": forbiddenValues[0],
|
||||
"provider_id": forbiddenValues[1],
|
||||
"raw_adapter": forbiddenValues[4],
|
||||
"raw_target": forbiddenValues[5],
|
||||
"body": forbiddenValues[7],
|
||||
"authorization": forbiddenValues[8],
|
||||
},
|
||||
}
|
||||
|
||||
frame := &iop.ProviderTunnelFrame{
|
||||
RunId: frameRunID,
|
||||
TunnelId: "SECRET_FRAME_TUNNEL_ID_BBBBB",
|
||||
NodeId: "SECRET_FRAME_NODE_ID_66666",
|
||||
NodeAlias: "SECRET_FRAME_NODE_ALIAS_CCCCC",
|
||||
Headers: map[string]string{"SECRET_HEADER_KEY_77777": forbiddenValues[8]},
|
||||
Body: []byte(forbiddenValues[7]),
|
||||
Error: "SECRET_FRAME_ERROR_88888",
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR,
|
||||
Failure: frameFailure,
|
||||
Metadata: frameFailure.Metadata,
|
||||
}
|
||||
|
||||
svc.HandleReceivedProviderTunnelFrame(entry.NodeID, entry.ConnectionGeneration, frame)
|
||||
|
||||
forbiddenValues = append(forbiddenValues, entry.NodeID, overlayProviderID, overlayAdapter, overlayTarget)
|
||||
|
||||
families, err := registry.Gather()
|
||||
if err != nil {
|
||||
t.Fatalf("gather metrics: %v", err)
|
||||
}
|
||||
for _, family := range families {
|
||||
for _, metric := range family.Metric {
|
||||
for _, label := range metric.Label {
|
||||
for _, secret := range forbiddenValues {
|
||||
if strings.Contains(label.GetName(), secret) || strings.Contains(label.GetValue(), secret) {
|
||||
t.Fatalf("forbidden value %q leaked in metric label %s=%s", secret, label.GetName(), label.GetValue())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for _, entry := range logs.All() {
|
||||
for _, secret := range forbiddenValues {
|
||||
if strings.Contains(entry.Message, secret) {
|
||||
t.Fatalf("forbidden value %q leaked in log message: %s", secret, entry.Message)
|
||||
}
|
||||
for _, field := range entry.Context {
|
||||
if strings.Contains(field.Key, secret) || strings.Contains(fmt.Sprint(field.Interface), secret) || strings.Contains(field.String, secret) {
|
||||
t.Fatalf("forbidden value %q leaked in log field %s", secret, field.Key)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,474 +0,0 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
edgeevents "iop/apps/edge/internal/events"
|
||||
edgenode "iop/apps/edge/internal/node"
|
||||
"iop/packages/go/config"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
const (
|
||||
overlayNodeID = "node-overlay"
|
||||
overlayProviderID = "provider-overlay"
|
||||
overlayAdapter = "vllm-overlay"
|
||||
overlayTarget = "model-overlay"
|
||||
overlayGroup = "group-overlay"
|
||||
)
|
||||
|
||||
func newProviderHealthOverlayService(t *testing.T, client *toki.TcpClient) (*Service, *edgenode.NodeEntry, *edgenode.NodeRecord) {
|
||||
t.Helper()
|
||||
registry := edgenode.NewRegistry()
|
||||
entry := &edgenode.NodeEntry{NodeID: overlayNodeID, Alias: "overlay", Client: client}
|
||||
registry.Register(entry)
|
||||
|
||||
record := &edgenode.NodeRecord{
|
||||
ID: overlayNodeID,
|
||||
Adapters: config.AdaptersConf{VllmInstances: []config.VllmInstanceConf{{
|
||||
Name: overlayAdapter, Enabled: true, Capacity: 1,
|
||||
}}},
|
||||
Providers: []config.NodeProviderConf{{
|
||||
ID: overlayProviderID, Type: "vllm", Category: config.CategoryAPI,
|
||||
Adapter: overlayAdapter, Models: []string{overlayTarget}, Health: "available", Capacity: 1,
|
||||
}},
|
||||
}
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(record)
|
||||
svc := New(registry, edgeevents.NewBus())
|
||||
svc.SetRuntimeConfig(store, []config.ModelCatalogEntry{{
|
||||
ID: overlayGroup, Providers: map[string]string{overlayProviderID: overlayTarget},
|
||||
}}, NewGroupPolicy(16, 30*time.Second))
|
||||
svc.HandleNodeConnect(entry.NodeID, entry.ConnectionGeneration)
|
||||
return svc, entry, record
|
||||
}
|
||||
|
||||
func addBoundOverlayLease(t *testing.T, queue *modelQueueManager, runID, providerID, adapter, target string, generation uint64) {
|
||||
t.Helper()
|
||||
queue.mu.Lock()
|
||||
defer queue.mu.Unlock()
|
||||
group := queue.getOrCreateGroupLocked(overlayGroup, NewGroupPolicy(16, 30*time.Second))
|
||||
group.adapter = adapter
|
||||
group.target = target
|
||||
queue.leaseSeq++
|
||||
leaseID := queue.leaseSeq
|
||||
lease := &providerLease{
|
||||
id: leaseID, groupKey: overlayGroup, nodeID: overlayNodeID, providerID: providerID,
|
||||
generation: generation, adapter: adapter, target: target,
|
||||
state: leaseStateTracked, runID: runID,
|
||||
}
|
||||
queue.leases[leaseID] = lease
|
||||
queue.leaseByRun[runID] = leaseID
|
||||
if providerID == "" {
|
||||
group.inflight[overlayNodeID]++
|
||||
return
|
||||
}
|
||||
key := providerResourceKey{nodeID: overlayNodeID, providerID: providerID}
|
||||
resource := queue.resources[key]
|
||||
if resource == nil {
|
||||
resource = &providerResourceState{
|
||||
nodeID: overlayNodeID, providerID: providerID, capacity: 1, enabled: true, generation: generation,
|
||||
}
|
||||
queue.resources[key] = resource
|
||||
}
|
||||
resource.reserve(false)
|
||||
}
|
||||
|
||||
func stallFailure(runID, adapter, target, providerHealth, classification string, sequence uint64) *iop.ExecutionFailure {
|
||||
return &iop.ExecutionFailure{
|
||||
Code: string(runtime.FailureCodeResponseStalled), Message: "provider response stalled", Retryable: true,
|
||||
Metadata: map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled), "provider_health": providerHealth,
|
||||
"liveness_classification": classification, "idle_duration_ms": "300000",
|
||||
"run_id": runID, "attempt_id": runID, "attempt_fence": "confirmed",
|
||||
"adapter": adapter, "target": target, "health_observation_seq": fmt.Sprint(sequence),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func unavailableRunEvent(runID, adapter, target string, sequence uint64) *iop.RunEvent {
|
||||
failure := stallFailure(runID, adapter, target, "unavailable", "provider_unhealthy", sequence)
|
||||
metadata := make(map[string]string, len(failure.GetMetadata()))
|
||||
for key, value := range failure.GetMetadata() {
|
||||
metadata[key] = value
|
||||
}
|
||||
return &iop.RunEvent{RunId: runID, Type: "error", Failure: failure, Metadata: metadata}
|
||||
}
|
||||
|
||||
func assertOverlayUnavailable(t *testing.T, queue *modelQueueManager, generation uint64, want bool, wantSequence uint64) {
|
||||
t.Helper()
|
||||
queue.mu.Lock()
|
||||
defer queue.mu.Unlock()
|
||||
overlay := queue.runtimeHealth[providerRuntimeHealthKey{
|
||||
nodeID: overlayNodeID, generation: generation, providerID: overlayProviderID,
|
||||
}]
|
||||
if overlay == nil {
|
||||
if want || wantSequence != 0 {
|
||||
t.Fatalf("runtime overlay missing, want unavailable=%v sequence=%d", want, wantSequence)
|
||||
}
|
||||
return
|
||||
}
|
||||
if overlay.unavailable != want || overlay.observationSeq != wantSequence {
|
||||
t.Fatalf("runtime overlay=(unavailable=%v sequence=%d), want (%v,%d)", overlay.unavailable, overlay.observationSeq, want, wantSequence)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReceivedRunFailureHealthOverlayTable(t *testing.T) {
|
||||
t.Run("missing provider identity releases but cannot project", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-missing-provider", "", overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent("run-missing-provider", overlayAdapter, overlayTarget, 1)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
if leaseCount(svc.queue) != 0 {
|
||||
t.Fatal("valid terminal did not release its provider-less lease")
|
||||
}
|
||||
if event.GetMetadata()["recovery_handoff"] != "" || len(svc.queue.runtimeHealth) != 0 {
|
||||
t.Fatalf("provider-less evidence affected handoff/overlay: event=%#v overlay=%#v", event.GetMetadata(), svc.queue.runtimeHealth)
|
||||
}
|
||||
})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
nodeID string
|
||||
generation func(uint64) uint64
|
||||
}{
|
||||
{name: "wrong reception node", nodeID: "other-node", generation: func(generation uint64) uint64 { return generation }},
|
||||
{name: "stale reception generation", nodeID: overlayNodeID, generation: func(generation uint64) uint64 { return generation + 1 }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
runID := "run-" + tc.name
|
||||
addBoundOverlayLease(t, svc.queue, runID, overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent(runID, overlayAdapter, overlayTarget, 1)
|
||||
svc.HandleReceivedRunLifecycleEvent(tc.nodeID, tc.generation(entry.ConnectionGeneration), event)
|
||||
if leaseCount(svc.queue) != 1 || len(svc.queue.runtimeHealth) != 0 {
|
||||
t.Fatalf("wrong reception changed correctness state: leases=%d overlay=%#v", leaseCount(svc.queue), svc.queue.runtimeHealth)
|
||||
}
|
||||
svc.HandleRunLifecycleEvent(event)
|
||||
})
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
adapter string
|
||||
target string
|
||||
}{
|
||||
{name: "adapter binding mismatch", adapter: "other-adapter", target: overlayTarget},
|
||||
{name: "target binding mismatch", adapter: overlayAdapter, target: "other-target"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
runID := "run-" + tc.name
|
||||
addBoundOverlayLease(t, svc.queue, runID, overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent(runID, tc.adapter, tc.target, 1)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
if leaseCount(svc.queue) != 0 || len(svc.queue.runtimeHealth) != 0 || event.GetMetadata()["recovery_handoff"] != "" {
|
||||
t.Fatalf("mismatched binding changed overlay/handoff: event=%#v overlay=%#v", event.GetMetadata(), svc.queue.runtimeHealth)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
t.Run("fresh unavailable lowers admission and snapshot without config mutation", func(t *testing.T) {
|
||||
svc, entry, record := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-unavailable", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent("run-unavailable", overlayAdapter, overlayTarget, 3)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
|
||||
if event.GetMetadata()["recovery_handoff"] != "confirmed" || event.GetMetadata()["provider_id"] != overlayProviderID ||
|
||||
event.GetFailure().GetMetadata()["recovery_handoff"] != "confirmed" {
|
||||
t.Fatalf("confirmed handoff annotation missing: event=%#v failure=%#v", event.GetMetadata(), event.GetFailure().GetMetadata())
|
||||
}
|
||||
if event.GetMetadata()["recovery_eligible"] != "" || event.GetFailure().GetMetadata()["recovery_eligible"] != "" {
|
||||
t.Fatal("Edge handoff invented recovery eligibility")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 3)
|
||||
if record.Providers[0].Health != "available" {
|
||||
t.Fatalf("config health mutated to %q", record.Providers[0].Health)
|
||||
}
|
||||
snapshot := svc.queue.getSnapshotForNode(entry.NodeID, record, true)[0]
|
||||
if snapshot.GetStatus() != "unavailable" || snapshot.GetHealth() != "unavailable" || snapshot.GetCapacity() != 0 {
|
||||
t.Fatalf("effective snapshot did not project overlay: %#v", snapshot)
|
||||
}
|
||||
candidate := candidateNode{
|
||||
entry: entry, capacity: 1, providerID: overlayProviderID, adapter: overlayAdapter,
|
||||
servedTarget: overlayTarget, generation: entry.ConnectionGeneration,
|
||||
}
|
||||
if _, err := svc.queue.admit(context.Background(), overlayGroup, overlayAdapter, overlayTarget, []candidateNode{candidate}, NewGroupPolicy(16, time.Second), nil, false, true); err == nil {
|
||||
t.Fatal("runtime-unavailable provider remained admissible")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("available stall advances fence but does not recover", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-lower", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-lower", overlayAdapter, overlayTarget, 5))
|
||||
|
||||
addBoundOverlayLease(t, svc.queue, "run-available-stall", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
failure := stallFailure("run-available-stall", overlayAdapter, overlayTarget, "available", "request_stalled", 6)
|
||||
event := &iop.RunEvent{RunId: "run-available-stall", Type: "error", Failure: failure, Metadata: map[string]string{}}
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 6)
|
||||
})
|
||||
|
||||
t.Run("unknown stall advances fence but does not recover", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-lower-unknown", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-lower-unknown", overlayAdapter, overlayTarget, 9))
|
||||
|
||||
addBoundOverlayLease(t, svc.queue, "run-unknown-stall", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
failure := stallFailure("run-unknown-stall", overlayAdapter, overlayTarget, "unknown", "health_unknown", 10)
|
||||
event := &iop.RunEvent{RunId: "run-unknown-stall", Type: "error", Failure: failure, Metadata: map[string]string{}}
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 10)
|
||||
})
|
||||
|
||||
t.Run("equal or lower sequence is a projection no-op", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-first", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-first", overlayAdapter, overlayTarget, 8))
|
||||
for _, sequence := range []uint64{8, 7} {
|
||||
runID := fmt.Sprintf("run-stale-%d", sequence)
|
||||
addBoundOverlayLease(t, svc.queue, runID, overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
event := unavailableRunEvent(runID, overlayAdapter, overlayTarget, sequence)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, event)
|
||||
if event.GetMetadata()["recovery_handoff"] != "confirmed" {
|
||||
t.Fatalf("stale sequence %d lost its request-local handoff", sequence)
|
||||
}
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 8)
|
||||
if leaseCount(svc.queue) != 0 {
|
||||
t.Fatal("stale-but-valid terminals did not release exactly once")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("new generation does not inherit unavailable overlay", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-old-generation", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-old-generation", overlayAdapter, overlayTarget, 4))
|
||||
svc.queue.mu.Lock()
|
||||
svc.queue.activateNodeGenerationLocked(entry.NodeID, entry.ConnectionGeneration+1)
|
||||
candidate := &candidateNode{entry: &edgenode.NodeEntry{NodeID: entry.NodeID}, providerID: overlayProviderID, generation: entry.ConnectionGeneration + 1}
|
||||
healthy := svc.queue.candidateRuntimeHealthyLocked(candidate)
|
||||
svc.queue.mu.Unlock()
|
||||
if !healthy {
|
||||
t.Fatal("new connection generation inherited old runtime health")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestReceivedNormalizedAndTunnelFailureReleaseOnce(t *testing.T) {
|
||||
for _, executionPath := range []string{"normalized", "tunnel"} {
|
||||
t.Run(executionPath, func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-release-once", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
const racers = 16
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(racers)
|
||||
for i := 0; i < racers; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if executionPath == "normalized" {
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-release-once", overlayAdapter, overlayTarget, 1))
|
||||
return
|
||||
}
|
||||
failure := stallFailure("run-release-once", overlayAdapter, overlayTarget, "unavailable", "provider_unhealthy", 1)
|
||||
svc.HandleReceivedProviderTunnelFrame(entry.NodeID, entry.ConnectionGeneration, &iop.ProviderTunnelFrame{
|
||||
RunId: "run-release-once", TunnelId: "tunnel-release-once",
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Failure: failure,
|
||||
})
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if leaseCount(svc.queue) != 0 {
|
||||
t.Fatalf("%s lease remained after terminal race", executionPath)
|
||||
}
|
||||
inFlight, longInFlight := providerResourceCounts(svc.queue, entry.NodeID, overlayProviderID)
|
||||
if inFlight != 0 || longInFlight != 0 {
|
||||
t.Fatalf("%s counters=(%d,%d), want zero", executionPath, inFlight, longInFlight)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReceivedTunnelFailureHandoffBeforeRoute(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-tunnel-handoff", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
frames, unsubscribe := svc.tunnels.subscribe("tunnel-handoff", 1)
|
||||
defer unsubscribe()
|
||||
failure := stallFailure("run-tunnel-handoff", overlayAdapter, overlayTarget, "unavailable", "provider_unhealthy", 1)
|
||||
svc.HandleReceivedProviderTunnelFrame(entry.NodeID, entry.ConnectionGeneration, &iop.ProviderTunnelFrame{
|
||||
RunId: "run-tunnel-handoff", TunnelId: "tunnel-handoff",
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR,
|
||||
Failure: failure, Metadata: map[string]string{},
|
||||
})
|
||||
select {
|
||||
case frame := <-frames:
|
||||
if frame.GetMetadata()["recovery_handoff"] != "confirmed" || frame.GetFailure().GetMetadata()["provider_id"] != overlayProviderID {
|
||||
t.Fatalf("routed terminal missed validated annotation: %#v", frame)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("validated tunnel terminal was not routed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderHealthOverlayCapabilitiesRecovery(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() {
|
||||
_ = edgeConn.Close()
|
||||
_ = nodeConn.Close()
|
||||
})
|
||||
parserMap := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.NodeCommandRequest{}): func(data []byte) (proto.Message, error) {
|
||||
message := &iop.NodeCommandRequest{}
|
||||
return message, proto.Unmarshal(data, message)
|
||||
},
|
||||
toki.TypeNameOf(&iop.NodeCommandResponse{}): func(data []byte) (proto.Message, error) {
|
||||
message := &iop.NodeCommandResponse{}
|
||||
return message, proto.Unmarshal(data, message)
|
||||
},
|
||||
}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parserMap)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parserMap)
|
||||
toki.AddRequestListenerTyped(&nodeClient.Communicator, func(request *iop.NodeCommandRequest) (*iop.NodeCommandResponse, error) {
|
||||
return &iop.NodeCommandResponse{
|
||||
RequestId: request.GetRequestId(), Type: request.GetType(),
|
||||
Adapter: request.GetAdapter(), Target: request.GetTarget(), SessionId: request.GetSessionId(),
|
||||
Result: map[string]string{
|
||||
"adapter_key": request.GetAdapter(), "target": request.GetTarget(),
|
||||
"provider_status": "available", "health_observation_seq": "2",
|
||||
},
|
||||
}, nil
|
||||
})
|
||||
|
||||
svc, entry, record := newProviderHealthOverlayService(t, edgeClient)
|
||||
addBoundOverlayLease(t, svc.queue, "run-needs-recovery", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-needs-recovery", overlayAdapter, overlayTarget, 1))
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 1)
|
||||
|
||||
if _, err := svc.Capabilities(context.Background(), NodeCommandRequestSpec{
|
||||
NodeRef: entry.NodeID, Adapter: overlayAdapter, Target: overlayTarget,
|
||||
}); err != nil {
|
||||
t.Fatalf("CAPABILITIES recovery probe: %v", err)
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, false, 2)
|
||||
if record.Providers[0].Health != "available" {
|
||||
t.Fatalf("recovery mutated config health to %q", record.Providers[0].Health)
|
||||
}
|
||||
snapshot := svc.queue.getSnapshotForNode(entry.NodeID, record, true)[0]
|
||||
if snapshot.GetStatus() != "available" || snapshot.GetHealth() != "available" || snapshot.GetCapacity() != 1 {
|
||||
t.Fatalf("recovered snapshot=%#v", snapshot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderHealthOverlayCapabilitiesRecoveryRejectsCatalogAmbiguity(t *testing.T) {
|
||||
svc, entry, record := newProviderHealthOverlayService(t, nil)
|
||||
record.Providers = append(record.Providers, config.NodeProviderConf{
|
||||
ID: "provider-healthy", Type: "vllm", Category: config.CategoryAPI,
|
||||
Adapter: overlayAdapter, Models: []string{overlayTarget}, Health: "available", Capacity: 1,
|
||||
})
|
||||
addBoundOverlayLease(t, svc.queue, "run-catalog-ambiguity", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-catalog-ambiguity", overlayAdapter, overlayTarget, 1))
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 1)
|
||||
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusAvailable, 2, func() bool { return true }); recovered {
|
||||
t.Fatal("ambiguous current catalog recovered provider")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 1)
|
||||
}
|
||||
|
||||
func TestProviderHealthOverlayCapabilitiesRecoveryPreservesAvailableHighWater(t *testing.T) {
|
||||
svc, entry, record := newProviderHealthOverlayService(t, nil)
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusAvailable, 2, func() bool { return true }); recovered {
|
||||
t.Fatal("already available provider reported recovery")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, false, 2)
|
||||
|
||||
addBoundOverlayLease(t, svc.queue, "run-delayed-unavailable", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-delayed-unavailable", overlayAdapter, overlayTarget, 1))
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, false, 2)
|
||||
if record.Providers[0].Health != "available" {
|
||||
t.Fatalf("delayed terminal mutated config health to %q", record.Providers[0].Health)
|
||||
}
|
||||
snapshot := svc.queue.getSnapshotForNode(entry.NodeID, record, true)[0]
|
||||
if snapshot.GetStatus() != "available" || snapshot.GetHealth() != "available" || snapshot.GetCapacity() != 1 {
|
||||
t.Fatalf("available high-water was reversed: %#v", snapshot)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderHealthOverlayCapabilitiesRecoveryRejectsInconclusiveEvidence(t *testing.T) {
|
||||
t.Run("unknown and unavailable do not advance the recovery fence", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-probe-fence", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-probe-fence", overlayAdapter, overlayTarget, 4))
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusUnknown, 6, func() bool { return true }); recovered {
|
||||
t.Fatal("unknown probe recovered provider")
|
||||
}
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusUnavailable, 7, func() bool { return true }); recovered {
|
||||
t.Fatal("unavailable probe recovered provider")
|
||||
}
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusAvailable, 4, func() bool { return true }); recovered {
|
||||
t.Fatal("equal-sequence available probe recovered provider")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 4)
|
||||
})
|
||||
|
||||
t.Run("stale generation is rejected", func(t *testing.T) {
|
||||
svc, entry, _ := newProviderHealthOverlayService(t, nil)
|
||||
addBoundOverlayLease(t, svc.queue, "run-stale-generation", overlayProviderID, overlayAdapter, overlayTarget, entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-stale-generation", overlayAdapter, overlayTarget, 1))
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, overlayTarget, runtime.ProviderStatusAvailable, 2, func() bool { return false }); recovered {
|
||||
t.Fatal("stale generation recovered provider")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 1)
|
||||
})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
result map[string]string
|
||||
adapter string
|
||||
target string
|
||||
}{
|
||||
{name: "missing sequence", result: map[string]string{"adapter_key": overlayAdapter, "target": overlayTarget, "provider_status": "available"}, adapter: overlayAdapter, target: overlayTarget},
|
||||
{name: "malformed sequence", result: map[string]string{"adapter_key": overlayAdapter, "target": overlayTarget, "provider_status": "available", "health_observation_seq": "bad"}, adapter: overlayAdapter, target: overlayTarget},
|
||||
{name: "binding mismatch", result: map[string]string{"adapter_key": "other", "target": overlayTarget, "provider_status": "available", "health_observation_seq": "2"}, adapter: overlayAdapter, target: overlayTarget},
|
||||
{name: "unknown status", result: map[string]string{"adapter_key": overlayAdapter, "target": overlayTarget, "provider_status": "corrupt", "health_observation_seq": "2"}, adapter: overlayAdapter, target: overlayTarget},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
request := &iop.NodeCommandRequest{Type: iop.NodeCommandType_NODE_COMMAND_TYPE_CAPABILITIES, Adapter: tc.adapter, Target: tc.target}
|
||||
response := &iop.NodeCommandResponse{Type: request.GetType(), Adapter: tc.adapter, Target: tc.target, Result: tc.result}
|
||||
if _, ok := capabilitiesProbeEvidenceFromResponse(request, response); ok {
|
||||
t.Fatalf("malformed evidence accepted: %#v", response)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderHealthOverlayCapabilitiesRecoveryRequiresLoweredBinding(t *testing.T) {
|
||||
svc, entry, record := newProviderHealthOverlayService(t, nil)
|
||||
record.Providers[0].Models = []string{"target-a", "target-b"}
|
||||
|
||||
addBoundOverlayLease(t, svc.queue, "run-lower-b", overlayProviderID, overlayAdapter, "target-b", entry.ConnectionGeneration)
|
||||
svc.HandleReceivedRunLifecycleEvent(entry.NodeID, entry.ConnectionGeneration, unavailableRunEvent("run-lower-b", overlayAdapter, "target-b", 1))
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 1)
|
||||
|
||||
// Newer available evidence for target-a advances sequence high-water mark to 2 but does not recover target-b lowered overlay.
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, "target-a", runtime.ProviderStatusAvailable, 2, func() bool { return true }); recovered {
|
||||
t.Fatal("cross-target available probe recovered provider lowered for another target")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, true, 2)
|
||||
|
||||
// Matching target-b available evidence at sequence 3 recovers the provider overlay.
|
||||
if recovered := svc.queue.applyProviderProbeEvidence(entry.NodeID, entry.ConnectionGeneration, overlayAdapter, "target-b", runtime.ProviderStatusAvailable, 3, func() bool { return true }); !recovered {
|
||||
t.Fatal("matching target-b available probe failed to recover provider overlay")
|
||||
}
|
||||
assertOverlayUnavailable(t, svc.queue, entry.ConnectionGeneration, false, 3)
|
||||
}
|
||||
|
|
@ -89,30 +89,13 @@ func (e *ProviderPoolOperationUnsupportedError) Unwrap() error {
|
|||
// a single one-shot provider-pool dispatch. SubmitProviderPool uses exactly
|
||||
// one queue admission to select a candidate, then dispatches only the
|
||||
// execution path indicated by the candidate's executionPath.
|
||||
//
|
||||
// AvoidProviderID is a request-local recovery hint. When non-empty, every
|
||||
// admission (initial and queued re-resolution) prefers a runtime-eligible
|
||||
// alternate provider over the avoided one. The avoided provider is only
|
||||
// retained when no alternate exists AND AllowAvoidedProviderFallback is
|
||||
// true AND the provider is still runtime eligible — the explicit fallback
|
||||
// permission is the only way to re-select the avoided provider. Liveness
|
||||
// recovery never grants that permission: a health probe proves endpoint
|
||||
// availability, not safety of replaying the request that just stalled.
|
||||
//
|
||||
// Zero values (empty AvoidProviderID, false AllowAvoidedProviderFallback)
|
||||
// preserve the current candidate selection behavior.
|
||||
//
|
||||
// This is selection policy only: it does not create a retry loop, reserve
|
||||
// a slot, change provider priority, persist the hints, or count retries.
|
||||
type ProviderPoolDispatchRequest struct {
|
||||
Run SubmitRunRequest
|
||||
Tunnel SubmitProviderTunnelRequest
|
||||
PrepareProtocolTunnel prepareProtocolTunnelFunc
|
||||
PrepareTunnel prepareTunnelFunc
|
||||
PrepareRun prepareRunFunc
|
||||
AcceptCandidate ProviderPoolCandidatePredicate
|
||||
AvoidProviderID string
|
||||
AllowAvoidedProviderFallback bool
|
||||
Run SubmitRunRequest
|
||||
Tunnel SubmitProviderTunnelRequest
|
||||
PrepareProtocolTunnel prepareProtocolTunnelFunc
|
||||
PrepareTunnel prepareTunnelFunc
|
||||
PrepareRun prepareRunFunc
|
||||
AcceptCandidate ProviderPoolCandidatePredicate
|
||||
}
|
||||
|
||||
// ProviderPoolDispatchResult describes which execution path was selected and
|
||||
|
|
@ -156,17 +139,6 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
}
|
||||
}
|
||||
|
||||
// Request-local avoided-provider recovery preference. This is selection policy
|
||||
// only: it does not create a retry loop, reserve a slot, change provider
|
||||
// priority, persist the hints, or count retries. The queue owns application —
|
||||
// it applies the preference after runtime-health filtering under its lock for
|
||||
// both this immediate admission and every queued re-resolution — so a
|
||||
// zero-value policy preserves the current candidate set.
|
||||
recovery := recoveryCandidatePolicy{
|
||||
avoidProviderID: req.AvoidProviderID,
|
||||
allowAvoidedProviderFallback: req.AllowAvoidedProviderFallback,
|
||||
}
|
||||
|
||||
// Provider-pool dispatch uses the canonical policy from the runtime snapshot.
|
||||
var policy groupPolicy
|
||||
if req.Run.ProviderPool {
|
||||
|
|
@ -197,14 +169,10 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
return nil, ErrProviderPoolCandidateRejected
|
||||
}
|
||||
}
|
||||
// The avoided-provider recovery preference is NOT applied here: the
|
||||
// queue reapplies it under its lock in resolveQueuedCandidatesLocked,
|
||||
// after runtime-health and orphan filtering, so re-resolution honors
|
||||
// the same request-local hint against genuinely eligible candidates.
|
||||
return resolved, nil
|
||||
}
|
||||
}
|
||||
selected, queueReason, err := s.queue.admitWithRecovery(ctx, req.Run.ModelGroupKey, req.Run.Adapter, req.Run.Target, candidates, policy, resolveCandidates, long, req.Run.ProviderPool, recovery)
|
||||
selected, queueReason, err := s.queue.admitWithReason(ctx, req.Run.ModelGroupKey, req.Run.Adapter, req.Run.Target, candidates, policy, resolveCandidates, long, req.Run.ProviderPool)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -236,7 +204,6 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
return nil, err
|
||||
}
|
||||
}
|
||||
runReq.ResponseStallTimeoutMS = selected.responseStallTimeoutMS
|
||||
return s.dispatchProviderPoolRun(ctx, runReq, adapter, target, selected, queueReason, reservation)
|
||||
|
||||
default:
|
||||
|
|
@ -357,8 +324,6 @@ func (s *Service) dispatchProviderPoolTunnel(
|
|||
}
|
||||
tunnelReq = tunnelReqPrepared
|
||||
}
|
||||
tunnelReq.ResponseStallTimeoutMS = selected.responseStallTimeoutMS
|
||||
|
||||
tunnelReqResolved, runID, err := buildProviderTunnelRequest(tunnelReq, adapter, target)
|
||||
if err != nil {
|
||||
reservation.release("build-error")
|
||||
|
|
@ -430,8 +395,6 @@ func (s *Service) dispatchProviderPoolRun(
|
|||
) (*ProviderPoolDispatchResult, error) {
|
||||
req.Adapter = adapter
|
||||
req.Target = target
|
||||
req.ResponseStallTimeoutMS = selected.responseStallTimeoutMS
|
||||
|
||||
runReq, runID, err := BuildRunRequest(req)
|
||||
if err != nil {
|
||||
reservation.release("build-error")
|
||||
|
|
@ -470,23 +433,22 @@ func (s *Service) dispatchProviderPoolRun(
|
|||
}
|
||||
|
||||
disp := RunDispatch{
|
||||
RunID: runID,
|
||||
NodeID: selected.entry.NodeID,
|
||||
NodeLabel: nodeLabel(selected.entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
ResponseStallTimeoutMS: dispatchResponseStallTimeout(runReq.GetResponseStallTimeoutMs()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: selected.providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: selected.providerType,
|
||||
ExecutionPath: string(selected.executionPath),
|
||||
QueueReason: queueReason,
|
||||
RunID: runID,
|
||||
NodeID: selected.entry.NodeID,
|
||||
NodeLabel: nodeLabel(selected.entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: selected.providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: selected.providerType,
|
||||
ExecutionPath: string(selected.executionPath),
|
||||
QueueReason: queueReason,
|
||||
}
|
||||
disp.ProfileID, disp.ProfileDriver = profileFacts(selected.profile)
|
||||
if selected.profile != nil {
|
||||
|
|
|
|||
|
|
@ -1,649 +0,0 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
edgeevents "iop/apps/edge/internal/events"
|
||||
edgenode "iop/apps/edge/internal/node"
|
||||
"iop/packages/go/config"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// The recovery-preference tests exercise the production admission path
|
||||
// (admitWithRecovery → applyRecoveryPreferenceLocked → findAvailableNodeLocked →
|
||||
// reserveCandidateLocked) rather than the pure helper. The avoided provider
|
||||
// "prov-a-primary" sorts before the alternate "prov-b-backup", so the plain
|
||||
// rotation would pick the avoided provider; a case that instead selects the
|
||||
// alternate proves the preference actually changed the dispatched candidate.
|
||||
const (
|
||||
recoveryNodeID = "node-recovery"
|
||||
recoveryAvoidID = "prov-a-primary"
|
||||
recoveryAltID = "prov-b-backup"
|
||||
recoveryServed = "served-x"
|
||||
recoveryGroupKey = "recovery-model"
|
||||
recoveryAvoidAdap = "vllm-a"
|
||||
recoveryAltAdap = "vllm-b"
|
||||
)
|
||||
|
||||
// newRecoveryQueueFixture builds a store-backed queue manager with two
|
||||
// capacity-1 providers on one node and the provider-pool policy seeded from the
|
||||
// store, matching how production reconciles resources before admission.
|
||||
func newRecoveryQueueFixture(t *testing.T) (*modelQueueManager, *edgenode.NodeEntry) {
|
||||
t.Helper()
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: recoveryNodeID,
|
||||
Runtime: config.RuntimeConf{Concurrency: 1},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{ID: recoveryAvoidID, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
{ID: recoveryAltID, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
},
|
||||
})
|
||||
m := newModelQueueManager(store)
|
||||
m.setProviderPoolPolicyLocked(store, NewGroupPolicy(4, 5*time.Second))
|
||||
entry := &edgenode.NodeEntry{NodeID: recoveryNodeID}
|
||||
return m, entry
|
||||
}
|
||||
|
||||
func recoveryCandidate(entry *edgenode.NodeEntry, providerID string) candidateNode {
|
||||
return candidateNode{
|
||||
entry: entry,
|
||||
providerID: providerID,
|
||||
servedTarget: recoveryServed,
|
||||
capacity: 1,
|
||||
generation: entry.ConnectionGeneration,
|
||||
}
|
||||
}
|
||||
|
||||
// markRecoveryUnavailable installs a runtime-health overlay that lowers one
|
||||
// provider on the candidate's generation, without mutating any config.
|
||||
func markRecoveryUnavailable(m *modelQueueManager, entry *edgenode.NodeEntry, providerID string) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
m.runtimeHealth[providerRuntimeHealthKey{
|
||||
nodeID: entry.NodeID,
|
||||
generation: entry.ConnectionGeneration,
|
||||
providerID: providerID,
|
||||
}] = &providerRuntimeHealthOverlay{
|
||||
adapter: recoveryAvoidAdap,
|
||||
target: recoveryServed,
|
||||
observationSeq: 1,
|
||||
unavailable: true,
|
||||
}
|
||||
}
|
||||
|
||||
// TestProviderRecoverySelectionImmediateAdmission drives the immediate
|
||||
// provider-pool admission path for every recovery branch and asserts the
|
||||
// selected provider (or typed rejection), exactly one lease per dispatch, and
|
||||
// that every counter settles back to zero after release with no forbidden
|
||||
// reservation on a rejected policy.
|
||||
func TestProviderRecoverySelectionImmediateAdmission(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
candidates []string // provider ids present in the request
|
||||
unavailable string // provider id lowered by runtime overlay, or ""
|
||||
recovery recoveryCandidatePolicy
|
||||
wantProviderID string // expected dispatched provider, or "" when rejected
|
||||
wantErr error // expected terminal error, or nil on dispatch
|
||||
}{
|
||||
{
|
||||
name: "eligible_alternate_preferred_over_avoided",
|
||||
candidates: []string{recoveryAvoidID, recoveryAltID},
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID},
|
||||
wantProviderID: recoveryAltID,
|
||||
},
|
||||
{
|
||||
name: "same_only_fallback_true_selects_avoided",
|
||||
candidates: []string{recoveryAvoidID},
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID, allowAvoidedProviderFallback: true},
|
||||
wantProviderID: recoveryAvoidID,
|
||||
},
|
||||
{
|
||||
name: "same_only_fallback_false_rejects",
|
||||
candidates: []string{recoveryAvoidID},
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID},
|
||||
wantErr: ErrProviderPoolCandidateRejected,
|
||||
},
|
||||
{
|
||||
name: "unavailable_alternate_fallback_true_selects_avoided",
|
||||
candidates: []string{recoveryAvoidID, recoveryAltID},
|
||||
unavailable: recoveryAltID,
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID, allowAvoidedProviderFallback: true},
|
||||
wantProviderID: recoveryAvoidID,
|
||||
},
|
||||
{
|
||||
name: "unavailable_alternate_fallback_false_rejects",
|
||||
candidates: []string{recoveryAvoidID, recoveryAltID},
|
||||
unavailable: recoveryAltID,
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID},
|
||||
wantErr: ErrProviderPoolCandidateRejected,
|
||||
},
|
||||
{
|
||||
name: "unavailable_avoided_selects_alternate",
|
||||
candidates: []string{recoveryAvoidID, recoveryAltID},
|
||||
unavailable: recoveryAvoidID,
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID},
|
||||
wantProviderID: recoveryAltID,
|
||||
},
|
||||
{
|
||||
name: "same_only_runtime_unavailable_is_terminal",
|
||||
candidates: []string{recoveryAvoidID},
|
||||
unavailable: recoveryAvoidID,
|
||||
recovery: recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID, allowAvoidedProviderFallback: true},
|
||||
wantErr: errProviderUnavailable,
|
||||
},
|
||||
{
|
||||
name: "empty_hints_dispatches_by_rotation",
|
||||
candidates: []string{recoveryAvoidID, recoveryAltID},
|
||||
recovery: recoveryCandidatePolicy{},
|
||||
wantProviderID: recoveryAvoidID, // lowest providerID wins the rotation
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m, entry := newRecoveryQueueFixture(t)
|
||||
if tc.unavailable != "" {
|
||||
markRecoveryUnavailable(m, entry, tc.unavailable)
|
||||
}
|
||||
candidates := make([]candidateNode, 0, len(tc.candidates))
|
||||
for _, id := range tc.candidates {
|
||||
candidates = append(candidates, recoveryCandidate(entry, id))
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
|
||||
defer cancel()
|
||||
selected, _, err := m.admitWithRecovery(ctx, recoveryGroupKey, "", recoveryServed, candidates, groupPolicy{}, nil, false, true, tc.recovery)
|
||||
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(err, tc.wantErr) {
|
||||
t.Fatalf("err=%v, want %v", err, tc.wantErr)
|
||||
}
|
||||
if selected != nil {
|
||||
t.Fatalf("rejected policy reserved candidate %+v", selected)
|
||||
}
|
||||
if lc := leaseCount(m); lc != 0 {
|
||||
t.Fatalf("leaseCount=%d after rejection, want 0", lc)
|
||||
}
|
||||
for _, id := range []string{recoveryAvoidID, recoveryAltID} {
|
||||
if inflight, _ := providerResourceCounts(m, recoveryNodeID, id); inflight != 0 {
|
||||
t.Fatalf("provider %s in-flight=%d after rejection, want 0", id, inflight)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("admit err=%v, want dispatch of %s", err, tc.wantProviderID)
|
||||
}
|
||||
if selected == nil || selected.providerID != tc.wantProviderID {
|
||||
t.Fatalf("selected=%+v, want providerID=%s", selected, tc.wantProviderID)
|
||||
}
|
||||
if lc := leaseCount(m); lc != 1 {
|
||||
t.Fatalf("leaseCount=%d after dispatch, want exactly 1", lc)
|
||||
}
|
||||
if inflight, _ := providerResourceCounts(m, recoveryNodeID, tc.wantProviderID); inflight != 1 {
|
||||
t.Fatalf("provider %s in-flight=%d after dispatch, want 1", tc.wantProviderID, inflight)
|
||||
}
|
||||
|
||||
// Release the lease and confirm every counter settles.
|
||||
m.releaseLease(selected.leaseID, "test-settle")
|
||||
if lc := leaseCount(m); lc != 0 {
|
||||
t.Fatalf("leaseCount=%d after release, want 0", lc)
|
||||
}
|
||||
if inflight, _ := providerResourceCounts(m, recoveryNodeID, tc.wantProviderID); inflight != 0 {
|
||||
t.Fatalf("provider %s in-flight=%d after release, want 0", tc.wantProviderID, inflight)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// recoveryAdmitResult carries a queued admission outcome back to the test body.
|
||||
type recoveryAdmitResult struct {
|
||||
candidate *candidateNode
|
||||
err error
|
||||
}
|
||||
|
||||
// TestProviderRecoverySelectionQueuedReresolution proves the queued path
|
||||
// reapplies the identical request-local recovery policy after a runtime-health
|
||||
// overlay change lands between enqueue and pump: an eligible alternate that
|
||||
// disappears either promotes the avoided provider under explicit fallback or
|
||||
// yields a typed terminal rejection when fallback is not permitted.
|
||||
func TestProviderRecoverySelectionQueuedReresolution(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
fallback bool
|
||||
wantProviderID string
|
||||
wantErr error
|
||||
}{
|
||||
{name: "fallback_true_promotes_avoided", fallback: true, wantProviderID: recoveryAvoidID},
|
||||
{name: "fallback_false_rejects", fallback: false, wantErr: ErrProviderPoolCandidateRejected},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
m, entry := newRecoveryQueueFixture(t)
|
||||
|
||||
// Occupy the alternate's only slot so a recovery request that prefers
|
||||
// it must queue instead of dispatching immediately.
|
||||
filler, _, err := m.admitWithRecovery(t.Context(), "filler-group", "", recoveryServed,
|
||||
[]candidateNode{recoveryCandidate(entry, recoveryAltID)}, groupPolicy{}, nil, false, true, recoveryCandidatePolicy{})
|
||||
if err != nil || filler == nil || filler.providerID != recoveryAltID {
|
||||
t.Fatalf("filler admit: candidate=%+v err=%v", filler, err)
|
||||
}
|
||||
|
||||
resolver := func() ([]candidateNode, error) {
|
||||
return []candidateNode{
|
||||
recoveryCandidate(entry, recoveryAvoidID),
|
||||
recoveryCandidate(entry, recoveryAltID),
|
||||
}, nil
|
||||
}
|
||||
|
||||
resultCh := make(chan recoveryAdmitResult, 1)
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 5*time.Second)
|
||||
defer cancel()
|
||||
go func() {
|
||||
candidate, _, admitErr := m.admitWithRecovery(ctx, recoveryGroupKey, "", recoveryServed,
|
||||
[]candidateNode{recoveryCandidate(entry, recoveryAvoidID), recoveryCandidate(entry, recoveryAltID)},
|
||||
groupPolicy{}, resolver, false, true,
|
||||
recoveryCandidatePolicy{avoidProviderID: recoveryAvoidID, allowAvoidedProviderFallback: tc.fallback})
|
||||
resultCh <- recoveryAdmitResult{candidate: candidate, err: admitErr}
|
||||
}()
|
||||
|
||||
requireProviderPoolPending(t, m, 1)
|
||||
|
||||
// Overlay change before pump: the alternate becomes runtime-unavailable.
|
||||
markRecoveryUnavailable(m, entry, recoveryAltID)
|
||||
m.mu.Lock()
|
||||
m.pumpAllLocked()
|
||||
m.mu.Unlock()
|
||||
|
||||
result := <-resultCh
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(result.err, tc.wantErr) {
|
||||
t.Fatalf("queued err=%v, want %v", result.err, tc.wantErr)
|
||||
}
|
||||
if result.candidate != nil {
|
||||
t.Fatalf("queued rejection reserved candidate %+v", result.candidate)
|
||||
}
|
||||
// Only the filler lease remains.
|
||||
if lc := leaseCount(m); lc != 1 {
|
||||
t.Fatalf("leaseCount=%d after queued rejection, want 1 (filler only)", lc)
|
||||
}
|
||||
} else {
|
||||
if result.err != nil {
|
||||
t.Fatalf("queued admit err=%v, want dispatch of %s", result.err, tc.wantProviderID)
|
||||
}
|
||||
if result.candidate == nil || result.candidate.providerID != tc.wantProviderID {
|
||||
t.Fatalf("queued selected=%+v, want providerID=%s", result.candidate, tc.wantProviderID)
|
||||
}
|
||||
if inflight, _ := providerResourceCounts(m, recoveryNodeID, tc.wantProviderID); inflight != 1 {
|
||||
t.Fatalf("provider %s in-flight=%d after queued dispatch, want 1", tc.wantProviderID, inflight)
|
||||
}
|
||||
if lc := leaseCount(m); lc != 2 {
|
||||
t.Fatalf("leaseCount=%d after queued dispatch, want 2 (filler + recovery)", lc)
|
||||
}
|
||||
m.releaseLease(result.candidate.leaseID, "test-settle")
|
||||
}
|
||||
|
||||
// The pending queue must have drained in both branches.
|
||||
m.mu.Lock()
|
||||
pending := m.pendingProviderPoolCountLocked()
|
||||
m.mu.Unlock()
|
||||
if pending != 0 {
|
||||
t.Fatalf("pending=%d after pump, want 0", pending)
|
||||
}
|
||||
|
||||
m.releaseLease(filler.leaseID, "test-cleanup")
|
||||
if lc := leaseCount(m); lc != 0 {
|
||||
t.Fatalf("leaseCount=%d after full cleanup, want 0", lc)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestProviderRecoverySelectionServiceDispatchPrefersAlternate exercises the
|
||||
// full SubmitProviderPool surface over net.Pipe: with the avoided provider
|
||||
// suppressed, the returned DispatchInfo names the alternate, and exactly one
|
||||
// provider tunnel request reaches the node (the avoided provider is never
|
||||
// dispatched).
|
||||
func TestProviderRecoverySelectionServiceDispatchPrefersAlternate(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() {
|
||||
_ = edgeConn.Close()
|
||||
_ = nodeConn.Close()
|
||||
})
|
||||
|
||||
parserMap := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.ProviderTunnelRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parserMap)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parserMap)
|
||||
|
||||
var capturedMu sync.Mutex
|
||||
var capturedCount int
|
||||
var capturedAdapter string
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelRequest](&nodeClient.Communicator, func(req *iop.ProviderTunnelRequest) {
|
||||
capturedMu.Lock()
|
||||
capturedCount++
|
||||
// The alternate and avoided providers use distinct adapter instances
|
||||
// (vllm-b vs vllm-a) but the same served target, so the wire adapter is
|
||||
// the identity that proves which provider was dispatched.
|
||||
capturedAdapter = req.GetAdapter()
|
||||
capturedMu.Unlock()
|
||||
})
|
||||
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: recoveryNodeID,
|
||||
Runtime: config.RuntimeConf{Concurrency: 4},
|
||||
Adapters: config.AdaptersConf{
|
||||
VllmInstances: []config.VllmInstanceConf{
|
||||
{Name: recoveryAvoidAdap, Enabled: true, Endpoint: "http://127.0.0.1:8000/v1"},
|
||||
{Name: recoveryAltAdap, Enabled: true, Endpoint: "http://127.0.0.1:8001/v1"},
|
||||
},
|
||||
},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{ID: recoveryAvoidID, Adapter: recoveryAvoidAdap, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
{ID: recoveryAltID, Adapter: recoveryAltAdap, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
},
|
||||
})
|
||||
|
||||
reg := edgenode.NewRegistry()
|
||||
reg.Register(&edgenode.NodeEntry{
|
||||
NodeID: recoveryNodeID,
|
||||
LifecycleState: edgenode.LifecycleConnected,
|
||||
Client: edgeClient,
|
||||
CredentialRecipientKeyID: "recipient-recovery",
|
||||
CredentialRecipientPublicKey: make([]byte, 32),
|
||||
})
|
||||
|
||||
svc := New(reg, edgeevents.NewBus())
|
||||
svc.SetNodeStore(store)
|
||||
svc.SetModelCatalog([]config.ModelCatalogEntry{
|
||||
{ID: recoveryGroupKey, Providers: map[string]string{recoveryAvoidID: recoveryServed, recoveryAltID: recoveryServed}},
|
||||
})
|
||||
|
||||
result, err := svc.SubmitProviderPool(context.Background(), ProviderPoolDispatchRequest{
|
||||
Run: SubmitRunRequest{
|
||||
ModelGroupKey: recoveryGroupKey,
|
||||
ProviderPool: true,
|
||||
Background: true,
|
||||
},
|
||||
AvoidProviderID: recoveryAvoidID,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SubmitProviderPool: %v", err)
|
||||
}
|
||||
if result == nil || result.Path != ProviderPoolPathTunnel {
|
||||
t.Fatalf("result=%+v, want tunnel path", result)
|
||||
}
|
||||
if result.Tunnel != nil {
|
||||
defer result.Tunnel.Close()
|
||||
}
|
||||
if result.DispatchInfo.ProviderID != recoveryAltID {
|
||||
t.Fatalf("DispatchInfo.ProviderID=%q, want %q (avoided provider must be suppressed)", result.DispatchInfo.ProviderID, recoveryAltID)
|
||||
}
|
||||
|
||||
waitForCondition(t, func() bool {
|
||||
capturedMu.Lock()
|
||||
defer capturedMu.Unlock()
|
||||
return capturedCount == 1
|
||||
}, "expected exactly one provider tunnel request to reach the node")
|
||||
|
||||
capturedMu.Lock()
|
||||
defer capturedMu.Unlock()
|
||||
if capturedCount != 1 {
|
||||
t.Fatalf("captured %d provider tunnel requests, want exactly 1", capturedCount)
|
||||
}
|
||||
if capturedAdapter != recoveryAltAdap {
|
||||
t.Fatalf("wire adapter=%q, want %q (avoided provider was dispatched)", capturedAdapter, recoveryAltAdap)
|
||||
}
|
||||
if got := inflightRunCount(svc.queue); got != 1 {
|
||||
t.Fatalf("inflight run count=%d after single dispatch, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProviderRecoverySelectionServiceQueuedReresolution crosses the public
|
||||
// SubmitProviderPool surface with its default resolver. The alternate first
|
||||
// fills its capacity; a queued recovery request has no operation or custom
|
||||
// predicate, then re-resolves against a changed live catalog when the filler
|
||||
// lease releases. This proves the request-local recovery policy survives the
|
||||
// public queued path rather than only a queue-core fixture.
|
||||
func TestProviderRecoverySelectionServiceQueuedReresolution(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
fallback bool
|
||||
wantProviderID string
|
||||
wantRecoveryWire int
|
||||
wantErr error
|
||||
}{
|
||||
{name: "fallback_true_dispatches_the_now_only_avoided_provider", fallback: true, wantProviderID: recoveryAvoidID, wantRecoveryWire: 2},
|
||||
{name: "fallback_false_terminates_without_avoided_dispatch", fallback: false, wantRecoveryWire: 1, wantErr: ErrProviderPoolCandidateRejected},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() {
|
||||
_ = edgeConn.Close()
|
||||
_ = nodeConn.Close()
|
||||
})
|
||||
|
||||
parserMap := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.ProviderTunnelRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parserMap)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parserMap)
|
||||
|
||||
var capturedMu sync.Mutex
|
||||
var capturedAdapters []string
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelRequest](&nodeClient.Communicator, func(req *iop.ProviderTunnelRequest) {
|
||||
capturedMu.Lock()
|
||||
capturedAdapters = append(capturedAdapters, req.GetAdapter())
|
||||
capturedMu.Unlock()
|
||||
})
|
||||
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: recoveryNodeID,
|
||||
Runtime: config.RuntimeConf{Concurrency: 4},
|
||||
Adapters: config.AdaptersConf{VllmInstances: []config.VllmInstanceConf{
|
||||
{Name: recoveryAvoidAdap, Enabled: true, Endpoint: "http://127.0.0.1:8000/v1"},
|
||||
{Name: recoveryAltAdap, Enabled: true, Endpoint: "http://127.0.0.1:8001/v1"},
|
||||
}},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{ID: recoveryAvoidID, Adapter: recoveryAvoidAdap, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
{ID: recoveryAltID, Adapter: recoveryAltAdap, Type: "vllm", Models: []string{recoveryServed}, Health: "available", Capacity: 1},
|
||||
},
|
||||
})
|
||||
|
||||
reg := edgenode.NewRegistry()
|
||||
reg.Register(&edgenode.NodeEntry{
|
||||
NodeID: recoveryNodeID,
|
||||
LifecycleState: edgenode.LifecycleConnected,
|
||||
Client: edgeClient,
|
||||
CredentialRecipientKeyID: "recipient-recovery",
|
||||
CredentialRecipientPublicKey: make([]byte, 32),
|
||||
})
|
||||
svc := New(reg, edgeevents.NewBus())
|
||||
svc.SetNodeStore(store)
|
||||
|
||||
// Fill the alternate before the recovery request sees both candidates.
|
||||
svc.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: recoveryGroupKey, Providers: map[string]string{recoveryAltID: recoveryServed},
|
||||
}})
|
||||
filler, err := svc.SubmitProviderPool(t.Context(), ProviderPoolDispatchRequest{
|
||||
Run: SubmitRunRequest{ModelGroupKey: recoveryGroupKey, ProviderPool: true, Background: true},
|
||||
})
|
||||
if err != nil || filler == nil || filler.DispatchInfo.ProviderID != recoveryAltID {
|
||||
t.Fatalf("alternate filler: result=%+v err=%v", filler, err)
|
||||
}
|
||||
waitForCondition(t, func() bool {
|
||||
capturedMu.Lock()
|
||||
defer capturedMu.Unlock()
|
||||
return len(capturedAdapters) == 1
|
||||
}, "expected one alternate filler tunnel dispatch")
|
||||
|
||||
// The recovery request has the default empty operation and no custom
|
||||
// predicate. While the alternate is capacity-full it must remain queued.
|
||||
svc.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: recoveryGroupKey, Providers: map[string]string{recoveryAvoidID: recoveryServed, recoveryAltID: recoveryServed},
|
||||
}})
|
||||
resultCh := make(chan *ProviderPoolDispatchResult, 1)
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
result, submitErr := svc.SubmitProviderPool(t.Context(), ProviderPoolDispatchRequest{
|
||||
Run: SubmitRunRequest{ModelGroupKey: recoveryGroupKey, ProviderPool: true, Background: true},
|
||||
AvoidProviderID: recoveryAvoidID,
|
||||
AllowAvoidedProviderFallback: tc.fallback,
|
||||
})
|
||||
resultCh <- result
|
||||
errCh <- submitErr
|
||||
}()
|
||||
requireProviderPoolPending(t, svc.queue, 1)
|
||||
|
||||
// Re-resolution must observe the changed catalog, not the enqueue-time
|
||||
// slice. Releasing the filler is the production queue pump trigger.
|
||||
svc.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: recoveryGroupKey, Providers: map[string]string{recoveryAvoidID: recoveryServed},
|
||||
}})
|
||||
svc.queue.releaseRun(filler.DispatchInfo.RunID, "test-release-filler")
|
||||
|
||||
result := <-resultCh
|
||||
err = <-errCh
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(err, tc.wantErr) {
|
||||
t.Fatalf("queued SubmitProviderPool err=%v, want %v", err, tc.wantErr)
|
||||
}
|
||||
if result != nil {
|
||||
t.Fatalf("terminal recovery returned result=%+v", result)
|
||||
}
|
||||
} else {
|
||||
if err != nil || result == nil || result.DispatchInfo.ProviderID != tc.wantProviderID {
|
||||
t.Fatalf("queued recovery result=%+v err=%v, want provider %q", result, err, tc.wantProviderID)
|
||||
}
|
||||
if lc := leaseCount(svc.queue); lc != 1 {
|
||||
t.Fatalf("leaseCount=%d after recovery dispatch, want exactly 1", lc)
|
||||
}
|
||||
svc.queue.releaseRun(result.DispatchInfo.RunID, "test-release-recovery")
|
||||
}
|
||||
|
||||
waitForCondition(t, func() bool {
|
||||
capturedMu.Lock()
|
||||
defer capturedMu.Unlock()
|
||||
return len(capturedAdapters) == tc.wantRecoveryWire
|
||||
}, "unexpected provider tunnel dispatch count")
|
||||
capturedMu.Lock()
|
||||
gotAdapters := append([]string(nil), capturedAdapters...)
|
||||
capturedMu.Unlock()
|
||||
if gotAdapters[0] != recoveryAltAdap {
|
||||
t.Fatalf("filler adapter=%q, want %q", gotAdapters[0], recoveryAltAdap)
|
||||
}
|
||||
if tc.fallback && gotAdapters[1] != recoveryAvoidAdap {
|
||||
t.Fatalf("fallback adapter=%q, want permitted avoided adapter %q", gotAdapters[1], recoveryAvoidAdap)
|
||||
}
|
||||
if lc := leaseCount(svc.queue); lc != 0 {
|
||||
t.Fatalf("leaseCount=%d after cleanup, want 0", lc)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestProviderRecoverySelectionServiceRejectsUnavailableOrUnknownAvoidedProvider
|
||||
// covers the same-only terminal branches through SubmitProviderPool. Neither a
|
||||
// runtime-unavailable avoided provider nor a configured-unknown one may reserve
|
||||
// a lease or emit a provider tunnel request, even when same-provider fallback is
|
||||
// explicitly permitted.
|
||||
func TestProviderRecoverySelectionServiceRejectsUnavailableOrUnknownAvoidedProvider(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
configuredHealth string
|
||||
markRuntimeOffline bool
|
||||
wantErr error
|
||||
}{
|
||||
{name: "runtime_unavailable", configuredHealth: "available", markRuntimeOffline: true, wantErr: errProviderUnavailable},
|
||||
{name: "configured_unknown", configuredHealth: "unknown"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() {
|
||||
_ = edgeConn.Close()
|
||||
_ = nodeConn.Close()
|
||||
})
|
||||
parserMap := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.ProviderTunnelRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parserMap)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parserMap)
|
||||
var capturedMu sync.Mutex
|
||||
captured := 0
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelRequest](&nodeClient.Communicator, func(*iop.ProviderTunnelRequest) {
|
||||
capturedMu.Lock()
|
||||
captured++
|
||||
capturedMu.Unlock()
|
||||
})
|
||||
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: recoveryNodeID,
|
||||
Runtime: config.RuntimeConf{Concurrency: 1},
|
||||
Adapters: config.AdaptersConf{VllmInstances: []config.VllmInstanceConf{{
|
||||
Name: recoveryAvoidAdap, Enabled: true, Endpoint: "http://127.0.0.1:8000/v1",
|
||||
}}},
|
||||
Providers: []config.NodeProviderConf{{
|
||||
ID: recoveryAvoidID, Adapter: recoveryAvoidAdap, Type: "vllm", Models: []string{recoveryServed}, Health: tc.configuredHealth, Capacity: 1,
|
||||
}},
|
||||
})
|
||||
reg := edgenode.NewRegistry()
|
||||
entry := &edgenode.NodeEntry{NodeID: recoveryNodeID, LifecycleState: edgenode.LifecycleConnected, Client: edgeClient}
|
||||
reg.Register(entry)
|
||||
svc := New(reg, edgeevents.NewBus())
|
||||
svc.SetNodeStore(store)
|
||||
svc.SetModelCatalog([]config.ModelCatalogEntry{{
|
||||
ID: recoveryGroupKey, Providers: map[string]string{recoveryAvoidID: recoveryServed},
|
||||
}})
|
||||
if tc.markRuntimeOffline {
|
||||
markRecoveryUnavailable(svc.queue, entry, recoveryAvoidID)
|
||||
}
|
||||
|
||||
result, err := svc.SubmitProviderPool(t.Context(), ProviderPoolDispatchRequest{
|
||||
Run: SubmitRunRequest{ModelGroupKey: recoveryGroupKey, ProviderPool: true, Background: true},
|
||||
AvoidProviderID: recoveryAvoidID,
|
||||
AllowAvoidedProviderFallback: true,
|
||||
})
|
||||
if tc.wantErr != nil {
|
||||
if !errors.Is(err, tc.wantErr) {
|
||||
t.Fatalf("SubmitProviderPool err=%v, want %v", err, tc.wantErr)
|
||||
}
|
||||
} else if err == nil {
|
||||
t.Fatal("configured-unknown provider unexpectedly dispatched")
|
||||
}
|
||||
if result != nil {
|
||||
t.Fatalf("terminal branch returned result=%+v", result)
|
||||
}
|
||||
if lc := leaseCount(svc.queue); lc != 0 {
|
||||
t.Fatalf("leaseCount=%d after terminal branch, want 0", lc)
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
capturedMu.Lock()
|
||||
defer capturedMu.Unlock()
|
||||
if captured != 0 {
|
||||
t.Fatalf("captured %d provider tunnel requests after terminal branch, want 0", captured)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -289,7 +289,6 @@ func applyProviderDispatchFields(c *candidateNode, prov config.NodeProviderConf)
|
|||
profile := prov.RuntimeProfile.Clone()
|
||||
c.profile = &profile
|
||||
}
|
||||
c.responseStallTimeoutMS = prov.EffectiveResponseStallTimeoutMS()
|
||||
}
|
||||
|
||||
// isProviderAvailable checks provider health status. Only "available" (and
|
||||
|
|
|
|||
|
|
@ -55,12 +55,11 @@ func TestSubmitRunProviderPoolRewritesAdapterAndTarget(t *testing.T) {
|
|||
},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{
|
||||
ID: "prov-vllm-01",
|
||||
Adapter: "vllm-gpu",
|
||||
Models: []string{"served-qwen"},
|
||||
Health: "available",
|
||||
Capacity: 2,
|
||||
ResponseStallTimeoutMS: 45000,
|
||||
ID: "prov-vllm-01",
|
||||
Adapter: "vllm-gpu",
|
||||
Models: []string{"served-qwen"},
|
||||
Health: "available",
|
||||
Capacity: 2,
|
||||
},
|
||||
},
|
||||
})
|
||||
|
|
@ -92,7 +91,7 @@ func TestSubmitRunProviderPoolRewritesAdapterAndTarget(t *testing.T) {
|
|||
|
||||
select {
|
||||
case got := <-capturedReq:
|
||||
if got.GetAdapter() != "vllm-gpu" || got.GetTarget() != "served-qwen" || got.GetRunId() != "run-pool-test-001" || got.GetResponseStallTimeoutMs() != 45000 {
|
||||
if got.GetAdapter() != "vllm-gpu" || got.GetTarget() != "served-qwen" || got.GetRunId() != "run-pool-test-001" {
|
||||
t.Fatalf("unexpected RunRequest: %+v", got)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
|
|
|
|||
|
|
@ -1,369 +0,0 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
edgeevents "iop/apps/edge/internal/events"
|
||||
edgenode "iop/apps/edge/internal/node"
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
func TestProviderCandidateResponseStallTimeout(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
raw int64
|
||||
want int64
|
||||
}{
|
||||
{name: "omitted defaults", want: execution.DefaultResponseStallTimeoutMS},
|
||||
{name: "configured value", raw: 45000, want: 45000},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
candidate := candidateNode{}
|
||||
applyProviderDispatchFields(&candidate, config.NodeProviderConf{ResponseStallTimeoutMS: tc.raw})
|
||||
if got := candidate.responseStallTimeoutMS; got != tc.want {
|
||||
t.Errorf("response stall timeout = %d, want %d", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDirectDispatchUsesZeroWireStallTimeout(t *testing.T) {
|
||||
t.Run("normalized", func(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() { _ = edgeConn.Close(); _ = nodeConn.Close() })
|
||||
parser := toki.ParserMap{toki.TypeNameOf(&iop.RunRequest{}): func(b []byte) (proto.Message, error) { m := &iop.RunRequest{}; return m, proto.Unmarshal(b, m) }}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parser)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parser)
|
||||
wires := make(chan *iop.RunRequest, 1)
|
||||
toki.AddListenerTyped[*iop.RunRequest](&nodeClient.Communicator, func(req *iop.RunRequest) { wires <- proto.Clone(req).(*iop.RunRequest) })
|
||||
svc := directStallTimeoutService(edgeClient)
|
||||
result, err := svc.SubmitRun(context.Background(), SubmitRunRequest{NodeRef: "direct-node", RunID: "direct-run", Adapter: "adapter", Target: "target", Background: true, ResponseStallTimeoutMS: 45000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := result.Dispatch().ResponseStallTimeoutMS; got != execution.DefaultResponseStallTimeoutMS {
|
||||
t.Fatalf("dispatch timeout = %d", got)
|
||||
}
|
||||
select {
|
||||
case wire := <-wires:
|
||||
if got := wire.GetResponseStallTimeoutMs(); got != 0 {
|
||||
t.Fatalf("wire timeout = %d, want 0", got)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("did not receive RunRequest")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("tunnel", func(t *testing.T) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() { _ = edgeConn.Close(); _ = nodeConn.Close() })
|
||||
parser := toki.ParserMap{toki.TypeNameOf(&iop.ProviderTunnelRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
}}
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parser)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parser)
|
||||
wires := make(chan *iop.ProviderTunnelRequest, 1)
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelRequest](&nodeClient.Communicator, func(req *iop.ProviderTunnelRequest) { wires <- proto.Clone(req).(*iop.ProviderTunnelRequest) })
|
||||
svc := directStallTimeoutService(edgeClient)
|
||||
result, err := svc.SubmitProviderTunnel(context.Background(), SubmitProviderTunnelRequest{NodeRef: "direct-node", RunID: "direct-tunnel", Adapter: "adapter", Target: "target", ResponseStallTimeoutMS: 45000})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer result.Close()
|
||||
if got := result.Dispatch().ResponseStallTimeoutMS; got != execution.DefaultResponseStallTimeoutMS {
|
||||
t.Fatalf("dispatch timeout = %d", got)
|
||||
}
|
||||
select {
|
||||
case wire := <-wires:
|
||||
if got := wire.GetResponseStallTimeoutMs(); got != 0 {
|
||||
t.Fatalf("wire timeout = %d, want 0", got)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("did not receive ProviderTunnelRequest")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func directStallTimeoutService(client *toki.TcpClient) *Service {
|
||||
registry := edgenode.NewRegistry()
|
||||
registry.Register(&edgenode.NodeEntry{NodeID: "direct-node", Client: client, DispatchReady: true})
|
||||
return New(registry, edgeevents.NewBus())
|
||||
}
|
||||
|
||||
type timeoutMatrixTestCase struct {
|
||||
name string
|
||||
isTunnel bool
|
||||
isQueued bool
|
||||
wantProvID string
|
||||
wantTarget string
|
||||
wantTimeout int64
|
||||
wantExecPath string
|
||||
wantQueueReason string
|
||||
}
|
||||
|
||||
func TestProviderPoolResponseStallTimeoutIdentityMatrix(t *testing.T) {
|
||||
tests := []timeoutMatrixTestCase{
|
||||
{name: "normalized_immediate", isTunnel: false, isQueued: false, wantProvID: "prov-1", wantTarget: "target-1", wantTimeout: 30000, wantExecPath: "normalized", wantQueueReason: "dispatched"},
|
||||
{name: "normalized_queued", isTunnel: false, isQueued: true, wantProvID: "prov-2", wantTarget: "target-2", wantTimeout: 60000, wantExecPath: "normalized", wantQueueReason: "capacity_full"},
|
||||
{name: "tunnel_immediate", isTunnel: true, isQueued: false, wantProvID: "prov-1", wantTarget: "target-1", wantTimeout: 30000, wantExecPath: "provider_tunnel", wantQueueReason: "dispatched"},
|
||||
{name: "tunnel_queued", isTunnel: true, isQueued: true, wantProvID: "prov-2", wantTarget: "target-2", wantTimeout: 60000, wantExecPath: "provider_tunnel", wantQueueReason: "capacity_full"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
runTimeoutMatrixSubtest(t, tc)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func runTimeoutMatrixSubtest(t *testing.T, tc timeoutMatrixTestCase) {
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
t.Cleanup(func() { _ = edgeConn.Close(); _ = nodeConn.Close() })
|
||||
|
||||
provType := "ollama"
|
||||
if tc.isTunnel {
|
||||
provType = "vllm"
|
||||
}
|
||||
|
||||
runWires := make(chan *iop.RunRequest, 2)
|
||||
tunnelWires := make(chan *iop.ProviderTunnelRequest, 2)
|
||||
edgeClient, _ := setupTimeoutMatrixClients(edgeConn, nodeConn, tc.isTunnel, runWires, tunnelWires)
|
||||
|
||||
groupKey := "group-timeout-identity"
|
||||
svc, store, catalog, policy := setupTimeoutMatrixService(edgeClient, provType, groupKey)
|
||||
|
||||
resDispatch, closeResult := executeTimeoutMatrixSubmit(t, svc, store, catalog, policy, groupKey, provType, tc)
|
||||
if closeResult != nil {
|
||||
defer closeResult()
|
||||
}
|
||||
|
||||
assertTimeoutMatrixDispatch(t, resDispatch, tc)
|
||||
assertTimeoutMatrixWire(t, tc, runWires, tunnelWires)
|
||||
|
||||
if closeResult != nil {
|
||||
closeResult()
|
||||
closeResult = nil
|
||||
}
|
||||
svc.HandleNodeDisconnect("node-timeout-matrix", 0, "test-cleanup")
|
||||
assertQueueSettled(t, svc.queue)
|
||||
}
|
||||
|
||||
func setupTimeoutMatrixClients(edgeConn, nodeConn net.Conn, isTunnel bool, runWires chan *iop.RunRequest, tunnelWires chan *iop.ProviderTunnelRequest) (*toki.TcpClient, *toki.TcpClient) {
|
||||
var parser toki.ParserMap
|
||||
if isTunnel {
|
||||
parser = toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.ProviderTunnelRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
} else {
|
||||
parser = toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.RunRequest{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.RunRequest{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
edgeClient := toki.NewTcpClient(edgeConn, 0, 0, parser)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, parser)
|
||||
|
||||
if isTunnel {
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelRequest](&nodeClient.Communicator, func(req *iop.ProviderTunnelRequest) {
|
||||
tunnelWires <- proto.Clone(req).(*iop.ProviderTunnelRequest)
|
||||
})
|
||||
} else {
|
||||
toki.AddListenerTyped[*iop.RunRequest](&nodeClient.Communicator, func(req *iop.RunRequest) {
|
||||
runWires <- proto.Clone(req).(*iop.RunRequest)
|
||||
})
|
||||
}
|
||||
return edgeClient, nodeClient
|
||||
}
|
||||
|
||||
func buildTimeoutMatrixStore(provType, health1 string) *edgenode.NodeStore {
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: "node-timeout-matrix",
|
||||
Runtime: config.RuntimeConf{Concurrency: 2},
|
||||
Adapters: config.AdaptersConf{
|
||||
OllamaInstances: []config.OllamaInstanceConf{{Name: "shared-adapter", Enabled: true}},
|
||||
VllmInstances: []config.VllmInstanceConf{{Name: "shared-adapter", Enabled: true}},
|
||||
},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{ID: "prov-1", Type: provType, Adapter: "shared-adapter", Models: []string{"target-1"}, Health: health1, Capacity: 1, ResponseStallTimeoutMS: 30000},
|
||||
{ID: "prov-2", Type: provType, Adapter: "shared-adapter", Models: []string{"target-2"}, Health: "available", Capacity: 1, ResponseStallTimeoutMS: 60000},
|
||||
},
|
||||
})
|
||||
return store
|
||||
}
|
||||
|
||||
func setupTimeoutMatrixService(edgeClient *toki.TcpClient, provType, groupKey string) (*Service, *edgenode.NodeStore, []config.ModelCatalogEntry, groupPolicy) {
|
||||
catalog := []config.ModelCatalogEntry{
|
||||
{ID: groupKey, Providers: map[string]string{"prov-1": "target-1", "prov-2": "target-2"}},
|
||||
}
|
||||
store := buildTimeoutMatrixStore(provType, "available")
|
||||
reg := edgenode.NewRegistry()
|
||||
reg.Register(&edgenode.NodeEntry{
|
||||
NodeID: "node-timeout-matrix",
|
||||
LifecycleState: edgenode.LifecycleConnected,
|
||||
Client: edgeClient,
|
||||
DispatchReady: true,
|
||||
})
|
||||
svc := New(reg, edgeevents.NewBus())
|
||||
svc.SetNodeStore(store)
|
||||
svc.SetModelCatalog(catalog)
|
||||
policy := groupPolicyFromStore(store, reg.AllReady(), "shared-adapter", "target-1")
|
||||
return svc, store, catalog, policy
|
||||
}
|
||||
|
||||
func executeTimeoutMatrixSubmit(t *testing.T, svc *Service, store *edgenode.NodeStore, catalog []config.ModelCatalogEntry, policy groupPolicy, groupKey, provType string, tc timeoutMatrixTestCase) (RunDispatch, func()) {
|
||||
runID := "run-" + tc.name
|
||||
if !tc.isQueued {
|
||||
if tc.isTunnel {
|
||||
res, err := svc.SubmitProviderTunnel(context.Background(), SubmitProviderTunnelRequest{RunID: runID, ModelGroupKey: groupKey, ProviderPool: true})
|
||||
if err != nil {
|
||||
t.Fatalf("immediate tunnel submit error: %v", err)
|
||||
}
|
||||
return res.Dispatch(), res.Close
|
||||
}
|
||||
res, err := svc.SubmitRun(context.Background(), SubmitRunRequest{RunID: runID, ModelGroupKey: groupKey, ProviderPool: true, Background: true})
|
||||
if err != nil {
|
||||
t.Fatalf("immediate normalized submit error: %v", err)
|
||||
}
|
||||
return res.Dispatch(), res.Close
|
||||
}
|
||||
|
||||
cands, pol, err := svc.resolveProviderPoolCandidates(SubmitRunRequest{ModelGroupKey: groupKey, ProviderPool: true}, store, catalog)
|
||||
if err != nil || len(cands) < 2 {
|
||||
t.Fatalf("resolve candidates: err=%v len=%d", err, len(cands))
|
||||
}
|
||||
sel1, _, err1 := svc.queue.admitWithReason(t.Context(), groupKey, "shared-adapter", "target-1", cands, pol, nil, false, true)
|
||||
if err1 != nil {
|
||||
t.Fatalf("admit prov-1: %v", err1)
|
||||
}
|
||||
r1 := newQueueReservation(svc.queue, sel1)
|
||||
|
||||
sel2, _, err2 := svc.queue.admitWithReason(t.Context(), groupKey, "shared-adapter", "target-2", cands, pol, nil, false, true)
|
||||
if err2 != nil {
|
||||
r1.release("cleanup-prov1")
|
||||
t.Fatalf("admit prov-2: %v", err2)
|
||||
}
|
||||
r2 := newQueueReservation(svc.queue, sel2)
|
||||
|
||||
type submitOut struct {
|
||||
dispatch RunDispatch
|
||||
close func()
|
||||
err error
|
||||
}
|
||||
outCh := make(chan submitOut, 1)
|
||||
|
||||
go func() {
|
||||
if tc.isTunnel {
|
||||
res, err := svc.SubmitProviderTunnel(context.Background(), SubmitProviderTunnelRequest{RunID: runID, ModelGroupKey: groupKey, ProviderPool: true})
|
||||
if err != nil {
|
||||
outCh <- submitOut{err: err}
|
||||
return
|
||||
}
|
||||
outCh <- submitOut{dispatch: res.Dispatch(), close: res.Close}
|
||||
} else {
|
||||
res, err := svc.SubmitRun(context.Background(), SubmitRunRequest{RunID: runID, ModelGroupKey: groupKey, ProviderPool: true, Background: true})
|
||||
if err != nil {
|
||||
outCh <- submitOut{err: err}
|
||||
return
|
||||
}
|
||||
outCh <- submitOut{dispatch: res.Dispatch(), close: res.Close}
|
||||
}
|
||||
}()
|
||||
|
||||
requireProviderPoolPending(t, svc.queue, 1)
|
||||
store2 := buildTimeoutMatrixStore(provType, "disabled")
|
||||
svc.SetRuntimeConfig(store2, catalog, policy)
|
||||
requireProviderPoolPending(t, svc.queue, 1)
|
||||
r2.release("make-prov2-available")
|
||||
|
||||
select {
|
||||
case out := <-outCh:
|
||||
r1.release("cleanup-prov1")
|
||||
if out.err != nil {
|
||||
t.Fatalf("queued submit error: %v", out.err)
|
||||
}
|
||||
return out.dispatch, out.close
|
||||
case <-time.After(3 * time.Second):
|
||||
r1.release("cleanup-prov1")
|
||||
t.Fatal("timed out waiting for queued submit result")
|
||||
return RunDispatch{}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func assertTimeoutMatrixDispatch(t *testing.T, disp RunDispatch, tc timeoutMatrixTestCase) {
|
||||
runID := "run-" + tc.name
|
||||
if got := disp.RunID; got != runID {
|
||||
t.Errorf("RunID = %q, want %q", got, runID)
|
||||
}
|
||||
if got := disp.ProviderID; got != tc.wantProvID {
|
||||
t.Errorf("ProviderID = %q, want %q", got, tc.wantProvID)
|
||||
}
|
||||
if got := disp.Adapter; got != "shared-adapter" {
|
||||
t.Errorf("Adapter = %q, want %q", got, "shared-adapter")
|
||||
}
|
||||
if got := disp.Target; got != tc.wantTarget {
|
||||
t.Errorf("Target = %q, want %q", got, tc.wantTarget)
|
||||
}
|
||||
if got := disp.ResponseStallTimeoutMS; got != tc.wantTimeout {
|
||||
t.Errorf("ResponseStallTimeoutMS = %d, want %d", got, tc.wantTimeout)
|
||||
}
|
||||
if got := disp.ExecutionPath; got != tc.wantExecPath {
|
||||
t.Errorf("ExecutionPath = %q, want %q", got, tc.wantExecPath)
|
||||
}
|
||||
if got := disp.QueueReason; got != tc.wantQueueReason {
|
||||
t.Errorf("QueueReason = %q, want %q", got, tc.wantQueueReason)
|
||||
}
|
||||
}
|
||||
|
||||
func assertTimeoutMatrixWire(t *testing.T, tc timeoutMatrixTestCase, runWires chan *iop.RunRequest, tunnelWires chan *iop.ProviderTunnelRequest) {
|
||||
runID := "run-" + tc.name
|
||||
if tc.isTunnel {
|
||||
wire := recvWire(t, tunnelWires, "ProviderTunnelRequest")
|
||||
if got := wire.GetRunId(); got != runID {
|
||||
t.Errorf("wire RunId = %q, want %q", got, runID)
|
||||
}
|
||||
if got := wire.GetTunnelId(); got != runID+"-tunnel" {
|
||||
t.Errorf("wire TunnelId = %q, want %q", got, runID+"-tunnel")
|
||||
}
|
||||
if got := wire.GetAdapter(); got != "shared-adapter" {
|
||||
t.Errorf("wire Adapter = %q, want %q", got, "shared-adapter")
|
||||
}
|
||||
if got := wire.GetTarget(); got != tc.wantTarget {
|
||||
t.Errorf("wire Target = %q, want %q", got, tc.wantTarget)
|
||||
}
|
||||
if got := wire.GetResponseStallTimeoutMs(); got != tc.wantTimeout {
|
||||
t.Errorf("wire ResponseStallTimeoutMs = %d, want %d", got, tc.wantTimeout)
|
||||
}
|
||||
assertNoExtra(t, tunnelWires, "ProviderTunnelRequest")
|
||||
} else {
|
||||
wire := recvWire(t, runWires, "RunRequest")
|
||||
if got := wire.GetRunId(); got != runID {
|
||||
t.Errorf("wire RunId = %q, want %q", got, runID)
|
||||
}
|
||||
if got := wire.GetAdapter(); got != "shared-adapter" {
|
||||
t.Errorf("wire Adapter = %q, want %q", got, "shared-adapter")
|
||||
}
|
||||
if got := wire.GetTarget(); got != tc.wantTarget {
|
||||
t.Errorf("wire Target = %q, want %q", got, tc.wantTarget)
|
||||
}
|
||||
if got := wire.GetResponseStallTimeoutMs(); got != tc.wantTimeout {
|
||||
t.Errorf("wire ResponseStallTimeoutMs = %d, want %d", got, tc.wantTimeout)
|
||||
}
|
||||
assertNoExtra(t, runWires, "RunRequest")
|
||||
}
|
||||
}
|
||||
|
|
@ -94,7 +94,7 @@ func (s *Service) HandleReceivedProviderTunnelFrame(nodeID string, generation ui
|
|||
return
|
||||
}
|
||||
if isTerminalProviderTunnelFrame(frame) && s.queue != nil {
|
||||
disposition := s.queue.settleReceivedTerminal(nodeID, generation, frame.GetRunId(), frame.GetFailure(), &frame.Metadata)
|
||||
disposition := s.queue.settleReceivedTerminal(nodeID, generation, frame.GetRunId())
|
||||
if disposition == receivedTerminalRejected {
|
||||
return
|
||||
}
|
||||
|
|
@ -127,17 +127,16 @@ type SubmitProviderTunnelRequest struct {
|
|||
// BuildBody, when set, produces the provider request body from the final
|
||||
// resolved target (provider-pool admission rewrites the target to the
|
||||
// winning candidate's served model). It takes precedence over Body.
|
||||
BuildBody func(target string) ([]byte, error)
|
||||
Stream bool
|
||||
TimeoutSec int
|
||||
MaxQueue int
|
||||
QueueTimeoutMS int
|
||||
Metadata map[string]string
|
||||
EstimatedInputTokens int
|
||||
ContextClass string
|
||||
ProviderPool bool
|
||||
CredentialBinding *CredentialBinding
|
||||
ResponseStallTimeoutMS int64
|
||||
BuildBody func(target string) ([]byte, error)
|
||||
Stream bool
|
||||
TimeoutSec int
|
||||
MaxQueue int
|
||||
QueueTimeoutMS int
|
||||
Metadata map[string]string
|
||||
EstimatedInputTokens int
|
||||
ContextClass string
|
||||
ProviderPool bool
|
||||
CredentialBinding *CredentialBinding
|
||||
}
|
||||
|
||||
// CredentialBinding contains only authenticated, secret-free route facts.
|
||||
|
|
@ -260,7 +259,6 @@ func (s *Service) submitProviderTunnelQueued(ctx context.Context, req SubmitProv
|
|||
target = selected.servedTarget
|
||||
}
|
||||
|
||||
req.ResponseStallTimeoutMS = selected.responseStallTimeoutMS
|
||||
tunnelReq, runID, err := buildProviderTunnelRequest(req, adapter, target)
|
||||
if err != nil {
|
||||
reservation.release("build-error")
|
||||
|
|
@ -311,9 +309,6 @@ func (s *Service) submitProviderTunnelDirectContext(ctx context.Context, req Sub
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// A direct tunnel has no selected provider candidate. Preserve the
|
||||
// zero-on-wire Node-default contract instead of accepting caller ownership.
|
||||
req.ResponseStallTimeoutMS = 0
|
||||
tunnelReq, _, err := buildProviderTunnelRequest(req, req.Adapter, req.Target)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -469,24 +464,23 @@ func (s *Service) openProviderTunnel(entry *edgenode.NodeEntry, tunnelReq *iop.P
|
|||
|
||||
return &ProviderTunnelHandle{
|
||||
RunDispatch: RunDispatch{
|
||||
RunID: runID,
|
||||
NodeID: entry.NodeID,
|
||||
NodeLabel: nodeLabel(entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: tunnelReq.GetAdapter(),
|
||||
Target: tunnelReq.GetTarget(),
|
||||
SessionID: tunnelReq.GetSessionId(),
|
||||
TimeoutSec: int(tunnelReq.GetTimeoutSec()),
|
||||
ResponseStallTimeoutMS: dispatchResponseStallTimeout(tunnelReq.GetResponseStallTimeoutMs()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: providerType,
|
||||
ExecutionPath: executionPath,
|
||||
CredentialSlotRef: credentialSlotRef,
|
||||
CredentialRevision: credentialRevision,
|
||||
QueueReason: queueReason,
|
||||
RunID: runID,
|
||||
NodeID: entry.NodeID,
|
||||
NodeLabel: nodeLabel(entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: tunnelReq.GetAdapter(),
|
||||
Target: tunnelReq.GetTarget(),
|
||||
SessionID: tunnelReq.GetSessionId(),
|
||||
TimeoutSec: int(tunnelReq.GetTimeoutSec()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: providerType,
|
||||
ExecutionPath: executionPath,
|
||||
CredentialSlotRef: credentialSlotRef,
|
||||
CredentialRevision: credentialRevision,
|
||||
QueueReason: queueReason,
|
||||
},
|
||||
TunnelID: tunnelReq.GetTunnelId(),
|
||||
frames: out,
|
||||
|
|
@ -530,19 +524,18 @@ func buildProviderTunnelRequest(req SubmitProviderTunnelRequest, adapter, target
|
|||
metadata[k] = v
|
||||
}
|
||||
return &iop.ProviderTunnelRequest{
|
||||
RunId: runID,
|
||||
TunnelId: runID + "-tunnel",
|
||||
Adapter: adapter,
|
||||
Target: target,
|
||||
Method: req.Method,
|
||||
Path: req.Path,
|
||||
Operation: req.Operation,
|
||||
Headers: headers,
|
||||
Body: body,
|
||||
Stream: req.Stream,
|
||||
TimeoutSec: int32(normalizeTimeoutSec(req.TimeoutSec)),
|
||||
Metadata: metadata,
|
||||
SessionId: NormalizeSessionID(req.SessionID),
|
||||
ResponseStallTimeoutMs: req.ResponseStallTimeoutMS,
|
||||
RunId: runID,
|
||||
TunnelId: runID + "-tunnel",
|
||||
Adapter: adapter,
|
||||
Target: target,
|
||||
Method: req.Method,
|
||||
Path: req.Path,
|
||||
Operation: req.Operation,
|
||||
Headers: headers,
|
||||
Body: body,
|
||||
Stream: req.Stream,
|
||||
TimeoutSec: int32(normalizeTimeoutSec(req.TimeoutSec)),
|
||||
Metadata: metadata,
|
||||
SessionId: NormalizeSessionID(req.SessionID),
|
||||
}, runID, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -145,13 +145,12 @@ func newProviderTunnelTestEnv(t *testing.T) *providerTunnelTestEnv {
|
|||
},
|
||||
Providers: []config.NodeProviderConf{
|
||||
{
|
||||
ID: "prov-vllm-01",
|
||||
Adapter: "vllm-gpu",
|
||||
Type: "vllm",
|
||||
Models: []string{"served-qwen"},
|
||||
Health: "available",
|
||||
Capacity: 1,
|
||||
ResponseStallTimeoutMS: 45000,
|
||||
ID: "prov-vllm-01",
|
||||
Adapter: "vllm-gpu",
|
||||
Type: "vllm",
|
||||
Models: []string{"served-qwen"},
|
||||
Health: "available",
|
||||
Capacity: 1,
|
||||
},
|
||||
},
|
||||
})
|
||||
|
|
@ -211,7 +210,7 @@ func TestSubmitProviderTunnelProviderPoolSendsRequestAndReleasesSlotOnEnd(t *tes
|
|||
waitForCondition(t, func() bool { return env.capturedRequest() != nil },
|
||||
"fake node did not receive ProviderTunnelRequest")
|
||||
captured := env.capturedRequest()
|
||||
if captured.GetAdapter() != "vllm-gpu" || captured.GetTarget() != "served-qwen" || captured.GetResponseStallTimeoutMs() != 45000 {
|
||||
if captured.GetAdapter() != "vllm-gpu" || captured.GetTarget() != "served-qwen" {
|
||||
t.Errorf("wire adapter/target: got %q/%q", captured.GetAdapter(), captured.GetTarget())
|
||||
}
|
||||
if !strings.Contains(string(captured.GetBody()), `"model":"served-qwen"`) {
|
||||
|
|
|
|||
|
|
@ -57,9 +57,6 @@ func (s *Service) submitRunDirect(req SubmitRunRequest) (RunResult, error) {
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Only provider-pool selection owns a non-zero wire value. Direct callers
|
||||
// retain the Node's zero-on-wire default regardless of DTO input.
|
||||
req.ResponseStallTimeoutMS = 0
|
||||
return s.dispatchToEntry(entry, req)
|
||||
}
|
||||
|
||||
|
|
@ -97,8 +94,6 @@ func (s *Service) submitRunQueued(ctx context.Context, req SubmitRunRequest) (Ru
|
|||
if selected.servedTarget != "" {
|
||||
req.Target = selected.servedTarget
|
||||
}
|
||||
req.ResponseStallTimeoutMS = selected.responseStallTimeoutMS
|
||||
|
||||
runReq, runID, err := BuildRunRequest(req)
|
||||
if err != nil {
|
||||
reservation.release("build-error")
|
||||
|
|
@ -133,23 +128,22 @@ func (s *Service) submitRunQueued(ctx context.Context, req SubmitRunRequest) (Ru
|
|||
reservation.handOff()
|
||||
|
||||
return newRunHandle(RunDispatch{
|
||||
RunID: runID,
|
||||
NodeID: selected.entry.NodeID,
|
||||
NodeLabel: nodeLabel(selected.entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
ResponseStallTimeoutMS: dispatchResponseStallTimeout(runReq.GetResponseStallTimeoutMs()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: selected.providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: selected.providerType,
|
||||
ExecutionPath: string(selected.executionPath),
|
||||
QueueReason: queueReason,
|
||||
RunID: runID,
|
||||
NodeID: selected.entry.NodeID,
|
||||
NodeLabel: nodeLabel(selected.entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: selected.providerID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
ProviderType: selected.providerType,
|
||||
ExecutionPath: string(selected.executionPath),
|
||||
QueueReason: queueReason,
|
||||
}, sub), nil
|
||||
}
|
||||
|
||||
|
|
@ -183,20 +177,19 @@ func (s *Service) dispatchToEntry(entry *edgenode.NodeEntry, req SubmitRunReques
|
|||
}
|
||||
|
||||
return newRunHandle(RunDispatch{
|
||||
RunID: runID,
|
||||
NodeID: entry.NodeID,
|
||||
NodeLabel: nodeLabel(entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
ResponseStallTimeoutMS: dispatchResponseStallTimeout(runReq.GetResponseStallTimeoutMs()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: req.ProviderID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
QueueReason: "dispatched",
|
||||
RunID: runID,
|
||||
NodeID: entry.NodeID,
|
||||
NodeLabel: nodeLabel(entry),
|
||||
ModelGroupKey: req.ModelGroupKey,
|
||||
Adapter: runReq.GetAdapter(),
|
||||
Target: runReq.GetTarget(),
|
||||
SessionID: runReq.GetSessionId(),
|
||||
Background: runReq.GetBackground(),
|
||||
TimeoutSec: int(runReq.GetTimeoutSec()),
|
||||
EstimatedInputTokens: req.EstimatedInputTokens,
|
||||
ContextClass: req.ContextClass,
|
||||
ProviderID: req.ProviderID,
|
||||
UsageAttribution: req.UsageAttribution,
|
||||
QueueReason: "dispatched",
|
||||
}, sub), nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -38,37 +38,35 @@ type SubmitRunRequest struct {
|
|||
// provider-pool catalog keyed by ModelGroupKey. Adapter and Target are
|
||||
// resolved per-candidate by resolveProviderPoolCandidates; the winning
|
||||
// candidate's ServedTarget is written into Target before BuildRunRequest.
|
||||
ProviderPool bool
|
||||
ResponseStallTimeoutMS int64
|
||||
ProviderPool bool
|
||||
}
|
||||
|
||||
// RunDispatch describes a dispatched run in surface-neutral terms. It is the
|
||||
// metadata any caller (console, HTTP, future RPC) needs after submission.
|
||||
type RunDispatch struct {
|
||||
RunID string
|
||||
NodeID string
|
||||
NodeLabel string
|
||||
ModelGroupKey string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
TimeoutSec int
|
||||
ResponseStallTimeoutMS int64
|
||||
EstimatedInputTokens int
|
||||
ContextClass string
|
||||
ProviderID string
|
||||
UsageAttribution string
|
||||
ProviderType string // non-empty for provider-pool dispatches
|
||||
ExecutionPath string // non-empty for provider-pool dispatches
|
||||
ProfileID string
|
||||
ProfileDriver string
|
||||
ProfileOperation string
|
||||
ProfileToolCallWire string
|
||||
ProfileCapabilities []string
|
||||
CredentialSlotRef string
|
||||
CredentialRevision uint64
|
||||
QueueReason string
|
||||
RunID string
|
||||
NodeID string
|
||||
NodeLabel string
|
||||
ModelGroupKey string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
TimeoutSec int
|
||||
EstimatedInputTokens int
|
||||
ContextClass string
|
||||
ProviderID string
|
||||
UsageAttribution string
|
||||
ProviderType string // non-empty for provider-pool dispatches
|
||||
ExecutionPath string // non-empty for provider-pool dispatches
|
||||
ProfileID string
|
||||
ProfileDriver string
|
||||
ProfileOperation string
|
||||
ProfileToolCallWire string
|
||||
ProfileCapabilities []string
|
||||
CredentialSlotRef string
|
||||
CredentialRevision uint64
|
||||
QueueReason string
|
||||
}
|
||||
|
||||
// RunStream carries asynchronous events for a dispatched foreground run.
|
||||
|
|
|
|||
|
|
@ -8,19 +8,11 @@ import (
|
|||
"google.golang.org/protobuf/types/known/structpb"
|
||||
|
||||
eventpkg "iop/packages/go/events"
|
||||
"iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
var lastRunIDNanos atomic.Int64
|
||||
|
||||
func dispatchResponseStallTimeout(ms int64) int64 {
|
||||
if ms == 0 {
|
||||
return execution.DefaultResponseStallTimeoutMS
|
||||
}
|
||||
return ms
|
||||
}
|
||||
|
||||
func NewRunID() string {
|
||||
return newRunIDAt(time.Now().UnixNano())
|
||||
}
|
||||
|
|
@ -65,14 +57,13 @@ func BuildRunRequest(req SubmitRunRequest) (*iop.RunRequest, string, error) {
|
|||
metadata[k] = v
|
||||
}
|
||||
return &iop.RunRequest{
|
||||
RunId: runID,
|
||||
Adapter: req.Adapter,
|
||||
Target: req.Target,
|
||||
SessionId: NormalizeSessionID(req.SessionID),
|
||||
Background: req.Background,
|
||||
Input: input,
|
||||
TimeoutSec: int32(normalizeTimeoutSec(req.TimeoutSec)),
|
||||
Metadata: metadata,
|
||||
ResponseStallTimeoutMs: req.ResponseStallTimeoutMS,
|
||||
RunId: runID,
|
||||
Adapter: req.Adapter,
|
||||
Target: req.Target,
|
||||
SessionId: NormalizeSessionID(req.SessionID),
|
||||
Background: req.Background,
|
||||
Input: input,
|
||||
TimeoutSec: int32(normalizeTimeoutSec(req.TimeoutSec)),
|
||||
Metadata: metadata,
|
||||
}, runID, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,8 +5,6 @@ import (
|
|||
"fmt"
|
||||
"sync"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
edgeevents "iop/apps/edge/internal/events"
|
||||
edgenode "iop/apps/edge/internal/node"
|
||||
"iop/packages/go/config"
|
||||
|
|
@ -182,18 +180,6 @@ func New(registry *edgenode.Registry, events *edgeevents.Bus) *Service {
|
|||
return s
|
||||
}
|
||||
|
||||
// SetProviderHealthLogger binds the Edge runtime logger to the bounded
|
||||
// provider-health observer. Bootstrap calls it before transport handlers start;
|
||||
// tests may replace the observer directly with a private registry fixture.
|
||||
func (s *Service) SetProviderHealthLogger(logger *zap.Logger) {
|
||||
if s == nil || s.queue == nil {
|
||||
return
|
||||
}
|
||||
if observer, ok := s.queue.healthObserver.(*providerHealthObservability); ok {
|
||||
observer.SetLogger(logger)
|
||||
}
|
||||
}
|
||||
|
||||
// HandleRunLifecycleEvent releases the lease owning a terminated run. The
|
||||
// transport calls it directly, ahead of the observability fanout, because the
|
||||
// event bus drops into full subscriber channels: lease accounting must not
|
||||
|
|
@ -214,7 +200,7 @@ func (s *Service) HandleReceivedRunLifecycleEvent(nodeID string, generation uint
|
|||
if event == nil || s.queue == nil || !isTerminalRunEvent(event) {
|
||||
return
|
||||
}
|
||||
s.queue.settleReceivedTerminal(nodeID, generation, event.GetRunId(), event.GetFailure(), &event.Metadata)
|
||||
s.queue.settleReceivedTerminal(nodeID, generation, event.GetRunId())
|
||||
}
|
||||
|
||||
// HandleNodeDisconnect fences the leases held by the disconnecting connection
|
||||
|
|
|
|||
|
|
@ -8,10 +8,42 @@ import (
|
|||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest/observer"
|
||||
)
|
||||
|
||||
func metricValue(t *testing.T, registry *prometheus.Registry, name string, want map[string]string) float64 {
|
||||
t.Helper()
|
||||
families, err := registry.Gather()
|
||||
if err != nil {
|
||||
t.Fatalf("gather metrics: %v", err)
|
||||
}
|
||||
for _, family := range families {
|
||||
if family.GetName() != name {
|
||||
continue
|
||||
}
|
||||
for _, metric := range family.Metric {
|
||||
if metricHasLabels(metric, want) {
|
||||
return metric.GetCounter().GetValue()
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func metricHasLabels(metric *dto.Metric, want map[string]string) bool {
|
||||
if len(metric.Label) != len(want) {
|
||||
return false
|
||||
}
|
||||
for _, label := range metric.Label {
|
||||
if want[label.GetName()] != label.GetValue() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func TestSingleRequestMetrics(t *testing.T) {
|
||||
if first, second := defaultSingleRequestCollectorSet(), defaultSingleRequestCollectorSet(); first != second {
|
||||
t.Fatal("default collector set was registered more than once")
|
||||
|
|
|
|||
|
|
@ -622,20 +622,11 @@ func TestBuildConfigPayload_AllAdaptersSettingsNil(t *testing.T) {
|
|||
func TestEdgeParserMap_ExecutionFailureRoundTrip(t *testing.T) {
|
||||
parsers := edgeParserMap()
|
||||
failure := &iop.ExecutionFailure{
|
||||
Code: "response_stalled",
|
||||
Message: "provider response stalled",
|
||||
Code: "provider_error",
|
||||
Message: "provider error",
|
||||
Retryable: true,
|
||||
Metadata: map[string]string{
|
||||
"failure_code": "response_stalled",
|
||||
"provider_health": "available",
|
||||
"liveness_classification": "request_stalled",
|
||||
"idle_duration_ms": "5000",
|
||||
"run_id": "run-1",
|
||||
"attempt_id": "run-1",
|
||||
"attempt_fence": "confirmed",
|
||||
"adapter": "ollama",
|
||||
"target": "llama3",
|
||||
"health_observation_seq": "1",
|
||||
"provider": "upstream",
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -643,7 +634,7 @@ func TestEdgeParserMap_ExecutionFailureRoundTrip(t *testing.T) {
|
|||
event := &iop.RunEvent{
|
||||
RunId: "run-1",
|
||||
Type: "error",
|
||||
Error: "provider response stalled",
|
||||
Error: "provider error",
|
||||
Failure: failure,
|
||||
NodeId: "node-1",
|
||||
Metadata: failure.Metadata,
|
||||
|
|
@ -664,10 +655,10 @@ func TestEdgeParserMap_ExecutionFailureRoundTrip(t *testing.T) {
|
|||
if got.GetFailure() == nil {
|
||||
t.Fatal("expected non-nil Failure on parsed RunEvent")
|
||||
}
|
||||
if got.GetFailure().GetCode() != "response_stalled" || !got.GetFailure().GetRetryable() {
|
||||
if got.GetFailure().GetCode() != "provider_error" || !got.GetFailure().GetRetryable() {
|
||||
t.Fatalf("unexpected Failure: %+v", got.GetFailure())
|
||||
}
|
||||
if got.GetFailure().GetMetadata()["provider_health"] != "available" {
|
||||
if got.GetFailure().GetMetadata()["provider"] != "upstream" {
|
||||
t.Fatalf("unexpected metadata: %+v", got.GetFailure().GetMetadata())
|
||||
}
|
||||
})
|
||||
|
|
@ -678,7 +669,7 @@ func TestEdgeParserMap_ExecutionFailureRoundTrip(t *testing.T) {
|
|||
TunnelId: "tunnel-1",
|
||||
Sequence: 5,
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR,
|
||||
Error: "provider response stalled",
|
||||
Error: "provider error",
|
||||
Failure: failure,
|
||||
NodeId: "node-1",
|
||||
Metadata: failure.Metadata,
|
||||
|
|
@ -699,10 +690,10 @@ func TestEdgeParserMap_ExecutionFailureRoundTrip(t *testing.T) {
|
|||
if got.GetFailure() == nil {
|
||||
t.Fatal("expected non-nil Failure on parsed ProviderTunnelFrame")
|
||||
}
|
||||
if got.GetFailure().GetCode() != "response_stalled" || !got.GetFailure().GetRetryable() {
|
||||
if got.GetFailure().GetCode() != "provider_error" || !got.GetFailure().GetRetryable() {
|
||||
t.Fatalf("unexpected Failure: %+v", got.GetFailure())
|
||||
}
|
||||
if got.GetFailure().GetMetadata()["liveness_classification"] != "request_stalled" {
|
||||
if got.GetFailure().GetMetadata()["provider"] != "upstream" {
|
||||
t.Fatalf("unexpected metadata: %+v", got.GetFailure().GetMetadata())
|
||||
}
|
||||
})
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@ func (n *Node) OnCommandRequest(ctx context.Context, sess *transport.Session, re
|
|||
|
||||
switch cmdType {
|
||||
case runtime.CommandTypeCapabilities:
|
||||
return n.handleCapabilitiesCommand(execCtx, sess, req), nil
|
||||
return n.handleCapabilitiesCommand(execCtx, req), nil
|
||||
case runtime.CommandTypeTransportStatus:
|
||||
return n.handleTransportStatusCommand(sess, req), nil
|
||||
default:
|
||||
|
|
@ -46,7 +46,7 @@ func (n *Node) OnCommandRequest(ctx context.Context, sess *transport.Session, re
|
|||
}
|
||||
}
|
||||
|
||||
func (n *Node) handleCapabilitiesCommand(ctx context.Context, sess *transport.Session, req *iop.NodeCommandRequest) *iop.NodeCommandResponse {
|
||||
func (n *Node) handleCapabilitiesCommand(ctx context.Context, req *iop.NodeCommandRequest) *iop.NodeCommandResponse {
|
||||
adapter, err := n.router.LookupAdapter(req.GetAdapter())
|
||||
if err != nil {
|
||||
return n.commandErrorResponse(req, fmt.Sprintf("node: %s", err.Error()))
|
||||
|
|
@ -57,20 +57,22 @@ func (n *Node) handleCapabilitiesCommand(ctx context.Context, sess *transport.Se
|
|||
}
|
||||
|
||||
targets := append([]string(nil), caps.Targets...)
|
||||
// CAPABILITIES health is a real bounded exact-target probe, not the raw
|
||||
// Capabilities status and not an adapter-specific error mapping. ProbeHealth
|
||||
// validates the adapter type, instance key, and target and collapses every
|
||||
// inconclusive path to unknown. The request adapter key and target remain the
|
||||
// immutable Edge binding carried in the response envelope/result.
|
||||
healthEvidence := ProbeHealth(caps.AdapterName, caps.InstanceKey, req.GetTarget(), ResolveProbeFunc(adapter))
|
||||
providerStatus := runtime.ProviderStatusUnknown
|
||||
switch healthEvidence.Health {
|
||||
case runtime.RequestStalled:
|
||||
providerStatus = runtime.ProviderStatusAvailable
|
||||
case runtime.ProviderUnhealthy:
|
||||
providerStatus = runtime.ProviderStatusUnavailable
|
||||
providerStatus := caps.ProviderStatus
|
||||
providerDetail := ""
|
||||
|
||||
if prober, ok := adapter.(runtime.ProviderProber); ok {
|
||||
probeRes, probeErr := prober.ProbeProvider(ctx, req.GetTarget())
|
||||
if probeErr != nil {
|
||||
providerStatus = runtime.ProviderStatusUnavailable
|
||||
providerDetail = probeErr.Error()
|
||||
} else {
|
||||
providerStatus = probeRes.Status
|
||||
providerDetail = probeRes.Detail
|
||||
if len(probeRes.Targets) > 0 {
|
||||
targets = append([]string(nil), probeRes.Targets...)
|
||||
}
|
||||
}
|
||||
}
|
||||
providerDetail := healthEvidence.Detail
|
||||
|
||||
sort.Strings(targets)
|
||||
|
||||
|
|
@ -85,9 +87,7 @@ func (n *Node) handleCapabilitiesCommand(ctx context.Context, sess *transport.Se
|
|||
|
||||
result := map[string]string{
|
||||
"adapter": caps.AdapterName,
|
||||
"adapter_key": req.GetAdapter(),
|
||||
"instance_key": caps.InstanceKey,
|
||||
"target": req.GetTarget(),
|
||||
"targets": strings.Join(targets, ","),
|
||||
"max_concurrency": strconv.Itoa(caps.MaxConcurrency),
|
||||
"provider_status": string(runtime.NormalizeProviderStatus(providerStatus)),
|
||||
|
|
@ -95,9 +95,6 @@ func (n *Node) handleCapabilitiesCommand(ctx context.Context, sess *transport.Se
|
|||
"in_flight": strconv.Itoa(inFlight),
|
||||
"queued": strconv.Itoa(queued),
|
||||
}
|
||||
if sess != nil {
|
||||
result["health_observation_seq"] = strconv.FormatUint(sess.NextHealthObservationSeq(), 10)
|
||||
}
|
||||
if providerDetail != "" {
|
||||
result["provider_detail"] = providerDetail
|
||||
}
|
||||
|
|
@ -105,7 +102,6 @@ func (n *Node) handleCapabilitiesCommand(ctx context.Context, sess *transport.Se
|
|||
providerSnapshot := &iop.ProviderSnapshot{
|
||||
Adapter: req.GetAdapter(),
|
||||
Status: string(runtime.NormalizeProviderStatus(providerStatus)),
|
||||
Health: string(runtime.NormalizeProviderStatus(providerStatus)),
|
||||
Capacity: int32(caps.MaxConcurrency),
|
||||
InFlight: int32(inFlight),
|
||||
Queued: int32(queued),
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@ package node_test
|
|||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
|
@ -69,8 +68,8 @@ func (a *providerCommandAdapter) ProbeProvider(_ context.Context, target string)
|
|||
return result, a.probeErr
|
||||
}
|
||||
|
||||
func TestCapabilitiesHealthEvidence(t *testing.T) {
|
||||
t.Run("exact available evidence is session sequenced", func(t *testing.T) {
|
||||
func TestCapabilitiesCommandUsesAdapterStatusWithoutRecoveryEvidence(t *testing.T) {
|
||||
t.Run("available provider status has no recovery fields", func(t *testing.T) {
|
||||
adapter := &providerCommandAdapter{probe: runtime.ProviderProbeResult{
|
||||
AdapterName: "provider", Target: "model", Status: runtime.ProviderStatusAvailable,
|
||||
}}
|
||||
|
|
@ -78,7 +77,7 @@ func TestCapabilitiesHealthEvidence(t *testing.T) {
|
|||
n, _ := makeNode(t, router)
|
||||
sess := &transport.Session{}
|
||||
|
||||
for wantSeq := uint64(1); wantSeq <= 2; wantSeq++ {
|
||||
for range 2 {
|
||||
resp, err := n.OnCommandRequest(context.Background(), sess, &iop.NodeCommandRequest{
|
||||
RequestId: "caps", Type: iop.NodeCommandType_NODE_COMMAND_TYPE_CAPABILITIES,
|
||||
Adapter: "provider", Target: "model",
|
||||
|
|
@ -87,14 +86,16 @@ func TestCapabilitiesHealthEvidence(t *testing.T) {
|
|||
t.Fatalf("response=%v err=%v", resp, err)
|
||||
}
|
||||
result := resp.GetResult()
|
||||
if result["adapter_key"] != "provider" || result["target"] != "model" || result["provider_status"] != "available" {
|
||||
t.Fatalf("unstable exact evidence: %#v", result)
|
||||
if result["provider_status"] != "available" {
|
||||
t.Fatalf("provider status: %#v", result)
|
||||
}
|
||||
if result["health_observation_seq"] != strconv.FormatUint(wantSeq, 10) {
|
||||
t.Fatalf("sequence=%q, want %d", result["health_observation_seq"], wantSeq)
|
||||
for _, key := range []string{"adapter_key", "target", "health_observation_seq"} {
|
||||
if _, ok := result[key]; ok {
|
||||
t.Fatalf("recovery field %q remained in capabilities result: %#v", key, result)
|
||||
}
|
||||
}
|
||||
if len(resp.GetProviderSnapshots()) != 1 || resp.GetProviderSnapshots()[0].GetHealth() != "available" {
|
||||
t.Fatalf("provider snapshot did not carry normalized health: %#v", resp.GetProviderSnapshots())
|
||||
if len(resp.GetProviderSnapshots()) != 1 || resp.GetProviderSnapshots()[0].GetStatus() != "available" || resp.GetProviderSnapshots()[0].GetHealth() != "" {
|
||||
t.Fatalf("provider snapshot retained recovery health: %#v", resp.GetProviderSnapshots())
|
||||
}
|
||||
}
|
||||
if adapter.probes != 2 {
|
||||
|
|
@ -108,8 +109,8 @@ func TestCapabilitiesHealthEvidence(t *testing.T) {
|
|||
err error
|
||||
wantStatus string
|
||||
}{
|
||||
{name: "transport error", result: runtime.ProviderProbeResult{AdapterName: "provider", Target: "model", Status: runtime.ProviderStatusAvailable}, err: errors.New("probe failed"), wantStatus: "unknown"},
|
||||
{name: "identity mismatch", result: runtime.ProviderProbeResult{AdapterName: "other", Target: "model", Status: runtime.ProviderStatusAvailable}, wantStatus: "unknown"},
|
||||
{name: "transport error", result: runtime.ProviderProbeResult{AdapterName: "provider", Target: "model", Status: runtime.ProviderStatusAvailable}, err: errors.New("probe failed"), wantStatus: "unavailable"},
|
||||
{name: "adapter result", result: runtime.ProviderProbeResult{AdapterName: "other", Target: "model", Status: runtime.ProviderStatusAvailable}, wantStatus: "available"},
|
||||
{name: "unknown", result: runtime.ProviderProbeResult{AdapterName: "provider", Target: "model", Status: runtime.ProviderStatusUnknown}, wantStatus: "unknown"},
|
||||
{name: "exact unavailable", result: runtime.ProviderProbeResult{AdapterName: "provider", Target: "model", Status: runtime.ProviderStatusUnavailable}, wantStatus: "unavailable"},
|
||||
} {
|
||||
|
|
@ -127,8 +128,8 @@ func TestCapabilitiesHealthEvidence(t *testing.T) {
|
|||
if got := resp.GetResult()["provider_status"]; got != tc.wantStatus {
|
||||
t.Fatalf("provider_status=%q, want %q", got, tc.wantStatus)
|
||||
}
|
||||
if resp.GetResult()["health_observation_seq"] != "1" {
|
||||
t.Fatalf("missing Session sequence: %#v", resp.GetResult())
|
||||
if _, ok := resp.GetResult()["health_observation_seq"]; ok {
|
||||
t.Fatalf("recovery sequence remained: %#v", resp.GetResult())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,157 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
)
|
||||
|
||||
// healthProbeCeiling is the independent upper bound on a single exact-target
|
||||
// health probe. The probe never inherits the stalled execution request's
|
||||
// context, deadline, or cancellation; it always roots a fresh deadline from
|
||||
// the background so a canceling parent cannot cut the evidence short. It is a
|
||||
// var rather than a const so deterministic tests can lower it without scheduler
|
||||
// sleeps; production always observes the five-second ceiling.
|
||||
var healthProbeCeiling = 5 * time.Second
|
||||
|
||||
// probeFunc is the injectable hook over ProviderProber.ProbeProvider. Tests
|
||||
// inject deterministic providers and observe the bounded context; production
|
||||
// resolves the adapter's ProviderProber implementation through ResolveProbeFunc.
|
||||
type probeFunc func(ctx context.Context, target string) (runtime.ProviderProbeResult, error)
|
||||
|
||||
// ResolveProbeFunc returns a probe hook bound to the adapter's ProviderProber
|
||||
// implementation, or nil when the adapter does not support active probing. A
|
||||
// nil hook makes ProbeHealth fail closed to HealthUnknown without invoking any
|
||||
// provider endpoint.
|
||||
func ResolveProbeFunc(adapter runtime.Provider) probeFunc {
|
||||
prober, ok := adapter.(runtime.ProviderProber)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
return prober.ProbeProvider
|
||||
}
|
||||
|
||||
// HealthProbeEvidence is the fail-closed evidence returned by ProbeHealth. It
|
||||
// carries only stable, coordinator-owned values: it never copies arbitrary
|
||||
// provider metadata, resets progress, changes the attempt fence, or authorizes
|
||||
// retry. Terminal assembly may consume Health as evidence only.
|
||||
type HealthProbeEvidence struct {
|
||||
Health runtime.ProviderHealth
|
||||
Status runtime.ProviderStatus
|
||||
Detail string
|
||||
}
|
||||
|
||||
// ProbeHealth performs a single bounded exact-target health probe of the named
|
||||
// adapter and target, independent of any stalled execution request. It roots
|
||||
// its own deadline from the background, runs the hook concurrently so a probe
|
||||
// that ignores context cancellation cannot hold the coordinator past the
|
||||
// independent ceiling, re-checks that deadline/cancellation after the probe
|
||||
// returns, validates that the probe confirmed the exact adapter and target
|
||||
// identity, and returns only the fail-closed normalized evidence.
|
||||
//
|
||||
// adapterName and instanceKey identify the stalled execution's required
|
||||
// provider (instanceKey may be empty for single-instance adapters); target is
|
||||
// the exact target that stalled. probe is the injectable ProviderProber hook,
|
||||
// or nil when the adapter does not support probing. ProbeHealth never calls
|
||||
// observer progress/reset, never changes the attempt fence, and never
|
||||
// authorizes retry.
|
||||
func ProbeHealth(adapterName, instanceKey, target string, probe probeFunc) HealthProbeEvidence {
|
||||
probeCtx, cancel := context.WithTimeout(context.Background(), healthProbeCeiling)
|
||||
defer cancel()
|
||||
|
||||
outcome := runtime.ProbeOutcome{
|
||||
ExpectedAdapter: adapterName,
|
||||
ExpectedInstance: instanceKey,
|
||||
ExpectedTarget: target,
|
||||
}
|
||||
if probe == nil {
|
||||
outcome.Err = runtime.ErrProbeUnsupported
|
||||
return finalizeHealthProbe(outcome)
|
||||
}
|
||||
|
||||
return finalizeHealthProbe(runProbe(probeCtx, target, probe, outcome))
|
||||
}
|
||||
|
||||
// probeCallResult is the typed result the probe goroutine reports to the
|
||||
// coordinator. It lets the coordinator select a completed probe against its
|
||||
// independent deadline without holding return time hostage to a hook that
|
||||
// ignores context cancellation.
|
||||
type probeCallResult struct {
|
||||
result runtime.ProviderProbeResult
|
||||
err error
|
||||
}
|
||||
|
||||
// runProbe invokes the probe hook on a background goroutine and selects its
|
||||
// result against the independent probe context. The result channel is buffered
|
||||
// to size one so a late-finishing hook can complete and send after the
|
||||
// coordinator has already returned, without blocking. On the deadline branch
|
||||
// the context error is surfaced as health_unknown through the normalizer; on
|
||||
// the result branch the post-result context recheck is preserved so a
|
||||
// simultaneously expired deadline still wins fail-closed. runProbe never calls
|
||||
// observer progress/reset, never changes the attempt fence, and never
|
||||
// authorizes retry.
|
||||
func runProbe(probeCtx context.Context, target string, probe probeFunc, outcome runtime.ProbeOutcome) runtime.ProbeOutcome {
|
||||
resultCh := make(chan probeCallResult, 1)
|
||||
go func() {
|
||||
res, err := probe(probeCtx, target)
|
||||
resultCh <- probeCallResult{result: res, err: err}
|
||||
}()
|
||||
|
||||
select {
|
||||
case call := <-resultCh:
|
||||
err := call.err
|
||||
// Re-check the independent deadline/cancellation even when the probe
|
||||
// returns nil error: a probe that ignored its bound context must still
|
||||
// be treated as inconclusive rather than allowed to manufacture a
|
||||
// definitive result. A result racing a simultaneous deadline expiry
|
||||
// therefore stays fail-closed.
|
||||
if err == nil && probeCtx.Err() != nil {
|
||||
err = probeCtx.Err()
|
||||
}
|
||||
outcome.AdapterName = call.result.AdapterName
|
||||
outcome.InstanceKey = call.result.InstanceKey
|
||||
outcome.Target = call.result.Target
|
||||
outcome.Status = call.result.Status
|
||||
outcome.Err = err
|
||||
case <-probeCtx.Done():
|
||||
outcome.Err = probeCtx.Err()
|
||||
}
|
||||
return outcome
|
||||
}
|
||||
|
||||
// finalizeHealthProbe normalizes the probe outcome and packages the stable
|
||||
// evidence. It is the single path that feeds the typed outcome normalizer.
|
||||
func finalizeHealthProbe(outcome runtime.ProbeOutcome) HealthProbeEvidence {
|
||||
classification := runtime.ClassifyProbeOutcome(outcome)
|
||||
return HealthProbeEvidence{
|
||||
Health: runtime.HealthFromClassification(classification),
|
||||
Status: runtime.NormalizeProviderStatus(outcome.Status),
|
||||
Detail: healthProbeDetail(outcome, classification),
|
||||
}
|
||||
}
|
||||
|
||||
// healthProbeDetail returns a short, coordinator-owned reason string for the
|
||||
// evidence. It never copies arbitrary provider metadata; only the probe's own
|
||||
// error message (when present) is surfaced for diagnostics.
|
||||
func healthProbeDetail(outcome runtime.ProbeOutcome, classification runtime.LivenessClassification) string {
|
||||
switch classification {
|
||||
case runtime.LivenessAvailable:
|
||||
return "exact target available"
|
||||
case runtime.LivenessUnavailable:
|
||||
return "exact target unavailable"
|
||||
case runtime.LivenessTimeout:
|
||||
return "probe timed out"
|
||||
case runtime.LivenessUnsupported:
|
||||
return "adapter does not support probing"
|
||||
case runtime.LivenessIdentityMismatch:
|
||||
return "probe identity did not match request"
|
||||
case runtime.LivenessUnknown:
|
||||
return "probe returned unknown status"
|
||||
default:
|
||||
if outcome.Err != nil {
|
||||
return outcome.Err.Error()
|
||||
}
|
||||
return "probe inconclusive"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,339 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
)
|
||||
|
||||
// recordingProbe captures the context the coordinator passed to the probe hook
|
||||
// so tests can assert it is live, independent, and exactly bounded.
|
||||
type recordingProbe struct {
|
||||
ctx context.Context
|
||||
result runtime.ProviderProbeResult
|
||||
err error
|
||||
calls int
|
||||
probeFn func(ctx context.Context, target string) (runtime.ProviderProbeResult, error)
|
||||
}
|
||||
|
||||
func (r *recordingProbe) probe(ctx context.Context, target string) (runtime.ProviderProbeResult, error) {
|
||||
r.calls++
|
||||
r.ctx = ctx
|
||||
if r.probeFn != nil {
|
||||
return r.probeFn(ctx, target)
|
||||
}
|
||||
return r.result, r.err
|
||||
}
|
||||
|
||||
func TestProbeHealthAvailableYieldsRequestStalled(t *testing.T) {
|
||||
rec := &recordingProbe{result: runtime.ProviderProbeResult{
|
||||
AdapterName: "vllm", InstanceKey: "vllm-gpu", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}}
|
||||
ev := ProbeHealth("vllm", "vllm-gpu", "m-a", rec.probe)
|
||||
if ev.Health != runtime.RequestStalled {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.RequestStalled)
|
||||
}
|
||||
if ev.Status != runtime.ProviderStatusAvailable {
|
||||
t.Errorf("Status: got %q, want available", ev.Status)
|
||||
}
|
||||
if rec.calls != 1 {
|
||||
t.Errorf("probe called %d times, want 1", rec.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthUnavailableYieldsProviderUnhealthy(t *testing.T) {
|
||||
rec := &recordingProbe{result: runtime.ProviderProbeResult{
|
||||
AdapterName: "ollama", Target: "m-b",
|
||||
Status: runtime.ProviderStatusUnavailable,
|
||||
}}
|
||||
ev := ProbeHealth("ollama", "", "m-b", rec.probe)
|
||||
if ev.Health != runtime.ProviderUnhealthy {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.ProviderUnhealthy)
|
||||
}
|
||||
if ev.Status != runtime.ProviderStatusUnavailable {
|
||||
t.Errorf("Status: got %q, want unavailable", ev.Status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthTransportErrorYieldsHealthUnknown(t *testing.T) {
|
||||
boom := errors.New("connection refused")
|
||||
rec := &recordingProbe{
|
||||
result: runtime.ProviderProbeResult{AdapterName: "vllm", Target: "m-a"},
|
||||
err: boom,
|
||||
}
|
||||
ev := ProbeHealth("vllm", "", "m-a", rec.probe)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Status != runtime.ProviderStatusUnknown {
|
||||
t.Errorf("Status: got %q, want unknown", ev.Status)
|
||||
}
|
||||
if ev.Detail != boom.Error() {
|
||||
t.Errorf("Detail: got %q, want %q", ev.Detail, boom.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthDeadlineExceededYieldsHealthUnknown(t *testing.T) {
|
||||
rec := &recordingProbe{
|
||||
result: runtime.ProviderProbeResult{AdapterName: "vllm", Target: "m-a"},
|
||||
err: context.DeadlineExceeded,
|
||||
}
|
||||
ev := ProbeHealth("vllm", "", "m-a", rec.probe)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Detail != "probe timed out" {
|
||||
t.Errorf("Detail: got %q, want probe timed out", ev.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthUnsupportedAdapterYieldsHealthUnknown(t *testing.T) {
|
||||
ev := ProbeHealth("worker", "", "m-a", nil)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Detail != "adapter does not support probing" {
|
||||
t.Errorf("Detail: got %q", ev.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthIdentityMismatchYieldsHealthUnknown(t *testing.T) {
|
||||
// The probe confirms a different adapter/target than the request required.
|
||||
rec := &recordingProbe{result: runtime.ProviderProbeResult{
|
||||
AdapterName: "ollama", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}}
|
||||
ev := ProbeHealth("vllm", "", "m-a", rec.probe)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Detail != "probe identity did not match request" {
|
||||
t.Errorf("Detail: got %q", ev.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeHealthPinnedInstanceMismatchYieldsHealthUnknown(t *testing.T) {
|
||||
rec := &recordingProbe{result: runtime.ProviderProbeResult{
|
||||
AdapterName: "vllm", InstanceKey: "vllm-gpu", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}}
|
||||
ev := ProbeHealth("vllm", "vllm-other", "m-a", rec.probe)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProbeHealthRechecksDeadlineWhenProbeIgnoresContext proves the coordinator
|
||||
// re-checks its independent deadline after the probe returns nil error. The
|
||||
// ceiling is lowered to the past so the rooted context is already expired; a
|
||||
// probe that ignores that context and reports available must still be
|
||||
// classified inconclusive. No scheduler sleep is used.
|
||||
func TestProbeHealthRechecksDeadlineWhenProbeIgnoresContext(t *testing.T) {
|
||||
saved := healthProbeCeiling
|
||||
healthProbeCeiling = -1 * time.Millisecond
|
||||
defer func() { healthProbeCeiling = saved }()
|
||||
|
||||
rec := &recordingProbe{result: runtime.ProviderProbeResult{
|
||||
AdapterName: "vllm", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}}
|
||||
ev := ProbeHealth("vllm", "", "m-a", rec.probe)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q after ignored deadline", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Detail != "probe timed out" {
|
||||
t.Errorf("Detail: got %q, want probe timed out", ev.Detail)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProbeHealthReturnsWhenBlockedHookOutlivesContext proves the coordinator
|
||||
// returns at its independent ceiling even when the prober ignores context
|
||||
// cancellation and never returns. It exercises the unexported context-taking
|
||||
// runProbe helper with a manually canceled context: the hook signals started,
|
||||
// the test cancels the context, the coordinator must return fail-closed
|
||||
// (health_unknown / probe timed out) while the hook is still blocked, and only
|
||||
// then does the test release the hook so no goroutine leaks. No time.Sleep,
|
||||
// wall-clock polling, live provider, or arbitrary provider metadata is used.
|
||||
func TestProbeHealthReturnsWhenBlockedHookOutlivesContext(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
|
||||
started := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
done := make(chan runtime.ProbeOutcome, 1)
|
||||
|
||||
probe := func(_ context.Context, _ string) (runtime.ProviderProbeResult, error) {
|
||||
started <- struct{}{}
|
||||
<-release
|
||||
return runtime.ProviderProbeResult{}, nil
|
||||
}
|
||||
|
||||
go func() {
|
||||
done <- runProbe(ctx, "m-a", probe, runtime.ProbeOutcome{
|
||||
ExpectedAdapter: "vllm",
|
||||
ExpectedTarget: "m-a",
|
||||
})
|
||||
}()
|
||||
|
||||
<-started
|
||||
|
||||
// Cancel the manual context. The coordinator must return fail-closed while
|
||||
// the hook is still blocked on release.
|
||||
cancel()
|
||||
|
||||
select {
|
||||
case got := <-done:
|
||||
if class := runtime.ClassifyProbeOutcome(got); class != runtime.LivenessTimeout {
|
||||
t.Fatalf("classification: got %q, want %q", class, runtime.LivenessTimeout)
|
||||
}
|
||||
ev := finalizeHealthProbe(got)
|
||||
if ev.Health != runtime.HealthUnknown {
|
||||
t.Fatalf("Health: got %q, want %q while hook still blocked", ev.Health, runtime.HealthUnknown)
|
||||
}
|
||||
if ev.Detail != "probe timed out" {
|
||||
t.Errorf("Detail: got %q, want probe timed out", ev.Detail)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("coordinator did not return within 2s after context cancel; hook held it past the ceiling")
|
||||
}
|
||||
|
||||
// Release the blocked hook so the probe goroutine finishes and no goroutine
|
||||
// leaks past the test.
|
||||
close(release)
|
||||
}
|
||||
|
||||
// TestProbeHealthReceivesIndependentBoundedContext proves the probe hook
|
||||
// receives a live, independently rooted, exactly bounded context: it has its
|
||||
// own deadline near the ceiling and is not derived from any canceled execution
|
||||
// request (the coordinator takes no execution context by design). The context
|
||||
// state is snapshotted inside the probe hook because ProbeHealth cancels its
|
||||
// rooted context after returning.
|
||||
func TestProbeHealthReceivesIndependentBoundedContext(t *testing.T) {
|
||||
var (
|
||||
observedAt time.Time
|
||||
observedDeadline time.Time
|
||||
hasDeadline bool
|
||||
observedErr error
|
||||
observedPtr interface{ Done() <-chan struct{} }
|
||||
)
|
||||
probe := func(ctx context.Context, target string) (runtime.ProviderProbeResult, error) {
|
||||
observedAt = time.Now()
|
||||
observedDeadline, hasDeadline = ctx.Deadline()
|
||||
observedErr = ctx.Err()
|
||||
observedPtr = ctx
|
||||
return runtime.ProviderProbeResult{
|
||||
AdapterName: "vllm", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}, nil
|
||||
}
|
||||
_ = ProbeHealth("vllm", "", "m-a", probe)
|
||||
|
||||
if observedErr != nil {
|
||||
t.Fatalf("probe context not live: %v", observedErr)
|
||||
}
|
||||
if !hasDeadline {
|
||||
t.Fatal("probe context has no deadline")
|
||||
}
|
||||
if !observedDeadline.After(observedAt) {
|
||||
t.Fatalf("probe deadline %v is not in the future (now %v)", observedDeadline, observedAt)
|
||||
}
|
||||
if got := observedDeadline.Sub(observedAt); got > healthProbeCeiling {
|
||||
t.Fatalf("probe bound %v exceeds ceiling %v", got, healthProbeCeiling)
|
||||
}
|
||||
// The rooted context must not be tied to a caller-supplied context.
|
||||
cancelCtx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if observedPtr == cancelCtx {
|
||||
t.Fatal("probe context must not be a caller-supplied context")
|
||||
}
|
||||
}
|
||||
|
||||
// TestProbeHealthRootsFromBackground proves a cancelled caller-side context
|
||||
// cannot cut the probe short: the coordinator takes no execution context by
|
||||
// design, so the probe still observes a live, bounded context and a definitive
|
||||
// result despite an unrelated canceled context existing in the caller.
|
||||
func TestProbeHealthRootsFromBackground(t *testing.T) {
|
||||
saved := healthProbeCeiling
|
||||
healthProbeCeiling = 50 * time.Millisecond
|
||||
defer func() { healthProbeCeiling = saved }()
|
||||
|
||||
// A separate canceled context exists in the caller; the coordinator must
|
||||
// not be derived from it.
|
||||
_, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
var observedErr error
|
||||
probe := func(ctx context.Context, target string) (runtime.ProviderProbeResult, error) {
|
||||
observedErr = ctx.Err()
|
||||
return runtime.ProviderProbeResult{
|
||||
AdapterName: "vllm", Target: "m-a",
|
||||
Status: runtime.ProviderStatusAvailable,
|
||||
}, nil
|
||||
}
|
||||
ev := ProbeHealth("vllm", "", "m-a", probe)
|
||||
if ev.Health != runtime.RequestStalled {
|
||||
t.Fatalf("Health: got %q, want %q (caller cancellation must not affect probe)", ev.Health, runtime.RequestStalled)
|
||||
}
|
||||
if observedErr != nil {
|
||||
t.Fatalf("probe context was not live despite a canceled caller-side context: %v", observedErr)
|
||||
}
|
||||
}
|
||||
|
||||
type stubProberProvider struct {
|
||||
probed bool
|
||||
}
|
||||
|
||||
func (s *stubProberProvider) Name() string { return "stub" }
|
||||
func (s *stubProberProvider) Capabilities(_ context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: "stub"}, nil
|
||||
}
|
||||
func (s *stubProberProvider) Execute(_ context.Context, _ runtime.ExecutionSpec, _ runtime.EventSink) error {
|
||||
return nil
|
||||
}
|
||||
func (s *stubProberProvider) ProbeProvider(_ context.Context, _ string) (runtime.ProviderProbeResult, error) {
|
||||
s.probed = true
|
||||
return runtime.ProviderProbeResult{AdapterName: "stub", Target: "m-a", Status: runtime.ProviderStatusAvailable}, nil
|
||||
}
|
||||
|
||||
type stubPlainProvider struct{}
|
||||
|
||||
func (s *stubPlainProvider) Name() string { return "plain" }
|
||||
func (s *stubPlainProvider) Capabilities(_ context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: "plain"}, nil
|
||||
}
|
||||
func (s *stubPlainProvider) Execute(_ context.Context, _ runtime.ExecutionSpec, _ runtime.EventSink) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestResolveProbeFunc(t *testing.T) {
|
||||
t.Run("prober_adapter_returns_hook", func(t *testing.T) {
|
||||
stub := &stubProberProvider{}
|
||||
probe := ResolveProbeFunc(stub)
|
||||
if probe == nil {
|
||||
t.Fatal("expected non-nil probe hook for prober adapter")
|
||||
}
|
||||
res, err := probe(context.Background(), "m-a")
|
||||
if err != nil || res.Status != runtime.ProviderStatusAvailable {
|
||||
t.Fatalf("unexpected probe result: %+v err=%v", res, err)
|
||||
}
|
||||
if !stub.probed {
|
||||
t.Fatal("probe hook did not invoke ProviderProber.ProbeProvider")
|
||||
}
|
||||
})
|
||||
t.Run("plain_adapter_returns_nil", func(t *testing.T) {
|
||||
if ResolveProbeFunc(&stubPlainProvider{}) != nil {
|
||||
t.Fatal("expected nil probe hook for non-prober adapter")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestProbeHealthViaResolveProbeFuncEndToEnd(t *testing.T) {
|
||||
ev := ProbeHealth("stub", "", "m-a", ResolveProbeFunc(&stubProberProvider{}))
|
||||
if ev.Health != runtime.RequestStalled {
|
||||
t.Fatalf("Health: got %q, want %q", ev.Health, runtime.RequestStalled)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,140 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
)
|
||||
|
||||
// healthObservationSequencer allocates connection-scoped, monotonically
|
||||
// increasing health-observation sequence values. Only a live bound transport
|
||||
// Session provides one; internal or unbound execution paths pass nil so the
|
||||
// terminal omits health_observation_seq and never invents a process-global
|
||||
// generation.
|
||||
type healthObservationSequencer interface {
|
||||
NextHealthObservationSeq() uint64
|
||||
}
|
||||
|
||||
func contextStillActive(ctx context.Context) bool { return ctx == nil || ctx.Err() == nil }
|
||||
|
||||
// stallObservation is the bounded evidence joined after the watchdog claims a
|
||||
// stall. It carries only Node-owned values; probe evidence is observation only
|
||||
// and never changes the fence, resets progress, or authorizes retry.
|
||||
type stallObservation struct {
|
||||
fence string
|
||||
idle time.Duration
|
||||
health HealthProbeEvidence
|
||||
seq uint64
|
||||
hasSeq bool
|
||||
}
|
||||
|
||||
func stallObservationFrom(result attemptResult, idle time.Duration, seq healthObservationSequencer) stallObservation {
|
||||
obs := stallObservation{fence: result.fence, idle: idle, health: result.health}
|
||||
if seq != nil {
|
||||
obs.seq = seq.NextHealthObservationSeq()
|
||||
obs.hasSeq = true
|
||||
}
|
||||
return obs
|
||||
}
|
||||
|
||||
// stallMetadata builds the single allowlisted stall-terminal metadata map.
|
||||
func stallMetadata(runID, adapter, target string, obs stallObservation) map[string]string {
|
||||
classification := obs.health.Health
|
||||
if classification == "" {
|
||||
classification = runtime.HealthUnknown
|
||||
}
|
||||
providerStatus := runtime.ProviderStatusUnknown
|
||||
switch classification {
|
||||
case runtime.RequestStalled:
|
||||
providerStatus = runtime.ProviderStatusAvailable
|
||||
case runtime.ProviderUnhealthy:
|
||||
providerStatus = runtime.ProviderStatusUnavailable
|
||||
}
|
||||
metadata := map[string]string{
|
||||
"failure_code": string(runtime.FailureCodeResponseStalled),
|
||||
"provider_health": string(providerStatus),
|
||||
"liveness_classification": string(classification),
|
||||
"idle_duration_ms": strconv.FormatInt(obs.idle.Milliseconds(), 10),
|
||||
"run_id": runID,
|
||||
"attempt_id": runID,
|
||||
"attempt_fence": obs.fence,
|
||||
"adapter": adapter,
|
||||
"target": target,
|
||||
}
|
||||
if obs.hasSeq {
|
||||
metadata["health_observation_seq"] = strconv.FormatUint(obs.seq, 10)
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
type healthProbe func() HealthProbeEvidence
|
||||
|
||||
func healthProbeFor(adapter runtime.Provider, adapterName, instanceKey, target string) healthProbe {
|
||||
resolved := ResolveProbeFunc(adapter)
|
||||
return func() HealthProbeEvidence {
|
||||
return ProbeHealth(adapterName, instanceKey, target, resolved)
|
||||
}
|
||||
}
|
||||
|
||||
func runHealthProbe(probe healthProbe) HealthProbeEvidence {
|
||||
if probe == nil {
|
||||
return HealthProbeEvidence{Health: runtime.HealthUnknown, Status: runtime.ProviderStatusUnknown}
|
||||
}
|
||||
return probe()
|
||||
}
|
||||
|
||||
type attemptResult struct {
|
||||
providerErr error
|
||||
stalled bool
|
||||
providerReturned bool
|
||||
fence string
|
||||
health HealthProbeEvidence
|
||||
}
|
||||
|
||||
type attemptCleanup struct {
|
||||
once sync.Once
|
||||
fn func()
|
||||
}
|
||||
|
||||
func newAttemptCleanup(fn func()) *attemptCleanup { return &attemptCleanup{fn: fn} }
|
||||
|
||||
func (c *attemptCleanup) run() {
|
||||
if c != nil {
|
||||
c.once.Do(c.fn)
|
||||
}
|
||||
}
|
||||
|
||||
func (c *attemptCleanup) afterProviderReturn(providerDone <-chan error) {
|
||||
go func() {
|
||||
<-providerDone
|
||||
c.run()
|
||||
}()
|
||||
}
|
||||
|
||||
func startProviderAttempt(execute func() error) <-chan error {
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- execute() }()
|
||||
return done
|
||||
}
|
||||
|
||||
// joinStallEvidence starts the fixed close-grace fence and exact-target probe
|
||||
// together, then waits for both bounded outcomes without serial extension.
|
||||
func joinStallEvidence(clock attemptClock, providerDone <-chan error, probe healthProbe) attemptResult {
|
||||
probeDone := make(chan HealthProbeEvidence, 1)
|
||||
go func() { probeDone <- runHealthProbe(probe) }()
|
||||
|
||||
grace := clock.NewTimer(defaultAttemptCloseGrace)
|
||||
result := attemptResult{stalled: true, fence: "unconfirmed"}
|
||||
select {
|
||||
case result.providerErr = <-providerDone:
|
||||
result.providerReturned = true
|
||||
result.fence = "confirmed"
|
||||
case <-grace.C():
|
||||
}
|
||||
grace.Stop()
|
||||
result.health = <-probeDone
|
||||
return result
|
||||
}
|
||||
|
|
@ -1,433 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// TestStalledTerminalsCloneSafeMetadata proves the normalized stall terminal
|
||||
// clones Node-owned metadata into the Failure map, the event map, and the
|
||||
// protobuf map without sharing a mutable alias, and that caller-provided
|
||||
// spoof values in the execution spec never leak into the terminal.
|
||||
func TestStalledTerminalsCloneSafeMetadata(t *testing.T) {
|
||||
spec := runtime.ExecutionSpec{RunID: "node-run", Adapter: "adapter", Target: "target", Metadata: map[string]string{"run_id": "spoof", "attempt_id": "spoof", "provider_health": "spoof", "liveness_classification": "spoof", "health_observation_seq": "spoof", "recovery_eligible": "true", "secret": "leak"}}
|
||||
obs := stallObservation{fence: "confirmed", idle: 2 * time.Second, health: HealthProbeEvidence{Health: runtime.RequestStalled, Status: runtime.ProviderStatusAvailable}, seq: 7, hasSeq: true}
|
||||
event := stalledRuntimeEvent(spec, obs)
|
||||
if event.Failure.Code != runtime.FailureCodeResponseStalled || !event.Failure.Retryable {
|
||||
t.Fatalf("failure = %#v", event.Failure)
|
||||
}
|
||||
if event.Metadata["run_id"] != "node-run" || event.Metadata["attempt_id"] != "node-run" || event.Metadata["recovery_eligible"] != "" || event.Metadata["secret"] != "" {
|
||||
t.Fatalf("unsafe normalized metadata = %#v", event.Metadata)
|
||||
}
|
||||
// Node-owned health evidence and the connection-scoped observation sequence
|
||||
// overwrite any caller-provided spoof values.
|
||||
if event.Metadata["provider_health"] != "available" || event.Metadata["liveness_classification"] != "request_stalled" || event.Metadata["health_observation_seq"] != "7" {
|
||||
t.Fatalf("health evidence not applied to normalized metadata = %#v", event.Metadata)
|
||||
}
|
||||
sender := &recordingProtoSender{}
|
||||
sink := &sessionSink{sess: sender}
|
||||
if err := sink.Emit(context.Background(), event); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wire := sender.snapshot()[0].(*iop.RunEvent)
|
||||
// The Failure map, event map, and protobuf map must carry identical safe
|
||||
// values without sharing a mutable alias.
|
||||
for _, key := range []string{"attempt_fence", "provider_health", "liveness_classification", "health_observation_seq"} {
|
||||
if event.Failure.Metadata[key] != event.Metadata[key] || wire.GetMetadata()[key] != event.Metadata[key] {
|
||||
t.Fatalf("normalized failure/event/protobuf disagree on %q: %q / %q / %q", key, event.Failure.Metadata[key], event.Metadata[key], wire.GetMetadata()[key])
|
||||
}
|
||||
}
|
||||
if wire.GetFailure() == nil || wire.GetFailure().GetCode() != "response_stalled" || !wire.GetFailure().GetRetryable() {
|
||||
t.Fatalf("wire.Failure mismatch: %#v", wire.GetFailure())
|
||||
}
|
||||
if wire.GetFailure().GetMetadata()["recovery_eligible"] != "" || wire.GetFailure().GetMetadata()["secret"] != "" {
|
||||
t.Fatalf("wire.Failure contains unsafe metadata: %#v", wire.GetFailure().GetMetadata())
|
||||
}
|
||||
event.Metadata["attempt_fence"] = "mutated"
|
||||
if event.Failure.Metadata["attempt_fence"] != "confirmed" || wire.GetMetadata()["attempt_fence"] != "confirmed" || wire.GetFailure().GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatal("normalized failure, event, and protobuf metadata alias")
|
||||
}
|
||||
|
||||
tunnelObs := stallObservation{fence: "unconfirmed", idle: 2 * time.Second, health: HealthProbeEvidence{Health: runtime.ProviderUnhealthy, Status: runtime.ProviderStatusUnavailable}, seq: 8, hasSeq: true}
|
||||
frame := stalledTunnelFrame(runtime.ProviderTunnelRequest{RunID: "node-run", Adapter: "adapter", Target: "target", Metadata: spec.Metadata}, tunnelObs)
|
||||
protoFrame := tunnelFrameToProto(frame, "node", "alias")
|
||||
if protoFrame.GetMetadata()["provider_health"] != "unavailable" || protoFrame.GetMetadata()["liveness_classification"] != "provider_unhealthy" || protoFrame.GetMetadata()["health_observation_seq"] != "8" {
|
||||
t.Fatalf("tunnel health evidence not applied = %#v", protoFrame.GetMetadata())
|
||||
}
|
||||
if protoFrame.GetFailure() == nil || protoFrame.GetFailure().GetCode() != "response_stalled" || protoFrame.GetFailure().GetRetryable() {
|
||||
t.Fatalf("protoFrame.Failure mismatch: %#v", protoFrame.GetFailure())
|
||||
}
|
||||
if protoFrame.GetFailure().GetMetadata()["provider_health"] != "unavailable" || protoFrame.GetFailure().GetMetadata()["liveness_classification"] != "provider_unhealthy" || protoFrame.GetFailure().GetMetadata()["health_observation_seq"] != "8" {
|
||||
t.Fatalf("protoFrame.Failure metadata mismatch = %#v", protoFrame.GetFailure().GetMetadata())
|
||||
}
|
||||
if protoFrame.GetFailure().GetMetadata()["recovery_eligible"] != "" || protoFrame.GetFailure().GetMetadata()["secret"] != "" {
|
||||
t.Fatalf("protoFrame.Failure contains unsafe metadata = %#v", protoFrame.GetFailure().GetMetadata())
|
||||
}
|
||||
frame.Metadata["attempt_fence"] = "mutated"
|
||||
if protoFrame.GetMetadata()["attempt_fence"] != "unconfirmed" || protoFrame.GetMetadata()["recovery_eligible"] != "" || protoFrame.GetMetadata()["secret"] != "" {
|
||||
t.Fatalf("unsafe or aliased tunnel metadata = %#v", protoFrame.GetMetadata())
|
||||
}
|
||||
}
|
||||
|
||||
// TestStallMetadataMapsThreeWayHealthEvidence proves the joined metadata carries
|
||||
// each of the three stable health outcomes, fails closed to unknown on zero
|
||||
// evidence, and includes the connection-scoped sequence only when one was
|
||||
// allocated.
|
||||
func TestStallMetadataMapsThreeWayHealthEvidence(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
obs stallObservation
|
||||
wantHealth string
|
||||
wantClass string
|
||||
wantSeqPresent bool
|
||||
wantSeq string
|
||||
}{
|
||||
{"available maps to request_stalled", stallObservation{fence: "confirmed", health: HealthProbeEvidence{Health: runtime.RequestStalled, Status: runtime.ProviderStatusAvailable}, seq: 1, hasSeq: true}, "available", "request_stalled", true, "1"},
|
||||
{"unavailable maps to provider_unhealthy", stallObservation{fence: "unconfirmed", health: HealthProbeEvidence{Health: runtime.ProviderUnhealthy, Status: runtime.ProviderStatusUnavailable}, seq: 2, hasSeq: true}, "unavailable", "provider_unhealthy", true, "2"},
|
||||
{"unknown status maps to health_unknown", stallObservation{fence: "confirmed", health: HealthProbeEvidence{Health: runtime.HealthUnknown, Status: runtime.ProviderStatusUnknown}, seq: 3, hasSeq: true}, "unknown", "health_unknown", true, "3"},
|
||||
{"zero evidence fails closed and omits seq", stallObservation{fence: "unconfirmed"}, "unknown", "health_unknown", false, ""},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
metadata := stallMetadata("run", "adapter", "target", tc.obs)
|
||||
if metadata["failure_code"] != string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("failure_code = %q", metadata["failure_code"])
|
||||
}
|
||||
if metadata["provider_health"] != tc.wantHealth || metadata["liveness_classification"] != tc.wantClass {
|
||||
t.Fatalf("health = %q, classification = %q", metadata["provider_health"], metadata["liveness_classification"])
|
||||
}
|
||||
if metadata["attempt_fence"] != tc.obs.fence || metadata["run_id"] != "run" || metadata["attempt_id"] != "run" || metadata["adapter"] != "adapter" || metadata["target"] != "target" {
|
||||
t.Fatalf("ownership metadata = %#v", metadata)
|
||||
}
|
||||
seq, present := metadata["health_observation_seq"]
|
||||
if present != tc.wantSeqPresent || seq != tc.wantSeq {
|
||||
t.Fatalf("health_observation_seq present=%v value=%q, want present=%v value=%q", present, seq, tc.wantSeqPresent, tc.wantSeq)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestStallMetadataFailsClosedOnContradictoryProbeStatus proves the terminal
|
||||
// pair never emits a definitive provider status paired with an inconclusive
|
||||
// classification. When the raw probe reports available or unavailable but the
|
||||
// normalized classification is HealthUnknown (identity mismatch, timeout, or
|
||||
// probe error), both provider_health and liveness_classification must resolve
|
||||
// to unknown/health_unknown on both the normalized and tunnel terminal paths.
|
||||
func TestStallMetadataFailsClosedOnContradictoryProbeStatus(t *testing.T) {
|
||||
contradictory := []struct {
|
||||
name string
|
||||
obs stallObservation
|
||||
}{
|
||||
{"raw available with unknown classification", stallObservation{fence: "confirmed", health: HealthProbeEvidence{Health: runtime.HealthUnknown, Status: runtime.ProviderStatusAvailable}, seq: 10, hasSeq: true}},
|
||||
{"raw unavailable with unknown classification", stallObservation{fence: "unconfirmed", health: HealthProbeEvidence{Health: runtime.HealthUnknown, Status: runtime.ProviderStatusUnavailable}, seq: 11, hasSeq: true}},
|
||||
}
|
||||
for _, tc := range contradictory {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
// Normalized terminal path.
|
||||
metadata := stallMetadata("run", "adapter", "target", tc.obs)
|
||||
if metadata["provider_health"] != string(runtime.ProviderStatusUnknown) {
|
||||
t.Fatalf("normalized provider_health = %q, want %q", metadata["provider_health"], runtime.ProviderStatusUnknown)
|
||||
}
|
||||
if metadata["liveness_classification"] != string(runtime.HealthUnknown) {
|
||||
t.Fatalf("normalized liveness_classification = %q, want %q", metadata["liveness_classification"], runtime.HealthUnknown)
|
||||
}
|
||||
if metadata["failure_code"] != string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("failure_code = %q", metadata["failure_code"])
|
||||
}
|
||||
|
||||
// Tunnel terminal path via stalledTunnelFrame.
|
||||
tunnelObs := tc.obs
|
||||
req := runtime.ProviderTunnelRequest{RunID: "run", Adapter: "adapter", Target: "target"}
|
||||
frame := stalledTunnelFrame(req, tunnelObs)
|
||||
protoFrame := tunnelFrameToProto(frame, "node", "alias")
|
||||
if protoFrame.GetMetadata()["provider_health"] != string(runtime.ProviderStatusUnknown) {
|
||||
t.Fatalf("tunnel provider_health = %q, want %q", protoFrame.GetMetadata()["provider_health"], runtime.ProviderStatusUnknown)
|
||||
}
|
||||
if protoFrame.GetMetadata()["liveness_classification"] != string(runtime.HealthUnknown) {
|
||||
t.Fatalf("tunnel liveness_classification = %q, want %q", protoFrame.GetMetadata()["liveness_classification"], runtime.HealthUnknown)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunWatchdogJoinsHealthEvidence proves the normalized stall terminal joins
|
||||
// the bounded exact-target probe result. The probe runs on an independent,
|
||||
// still-live context after the request was canceled, and its three-way outcome
|
||||
// reaches the terminal without changing the confirmed fence or reviving the run.
|
||||
func TestRunWatchdogJoinsHealthEvidence(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
reply probeReply
|
||||
wantHealth string
|
||||
wantClass string
|
||||
}{
|
||||
{"available maps to request_stalled", probeReply{result: runtime.ProviderProbeResult{AdapterName: "run-health", Target: "target", Status: runtime.ProviderStatusAvailable}}, "available", "request_stalled"},
|
||||
{"unavailable maps to provider_unhealthy", probeReply{result: runtime.ProviderProbeResult{AdapterName: "run-health", Target: "target", Status: runtime.ProviderStatusUnavailable}}, "unavailable", "provider_unhealthy"},
|
||||
{"probe error fails closed to unknown", probeReply{err: errors.New("probe transport failure")}, "unknown", "health_unknown"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newProbingWatchdogAdapter("run-health")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-health", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
probe := <-adapter.probeCalls
|
||||
if probe.target != "target" {
|
||||
t.Fatalf("probe target = %q", probe.target)
|
||||
}
|
||||
if probe.ctx.Err() != nil {
|
||||
t.Fatal("health probe inherited the canceled request context")
|
||||
}
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
adapter.probeReturn <- tc.reply
|
||||
adapter.runReturn <- nil // provider returns within grace -> confirmed
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
terminal := waitRunEvent(t, pipe.events)
|
||||
meta := terminal.GetMetadata()
|
||||
if terminal.GetType() != string(runtime.EventTypeError) || meta["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
if meta["provider_health"] != tc.wantHealth || meta["liveness_classification"] != tc.wantClass {
|
||||
t.Fatalf("health evidence = %q/%q, want %q/%q", meta["provider_health"], meta["liveness_classification"], tc.wantHealth, tc.wantClass)
|
||||
}
|
||||
if meta["health_observation_seq"] != "1" {
|
||||
t.Fatalf("health_observation_seq = %q, want 1", meta["health_observation_seq"])
|
||||
}
|
||||
// Exactly one terminal; late provider output remains fenced.
|
||||
_ = call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-health", Type: runtime.EventTypeDelta, Delta: "late"})
|
||||
select {
|
||||
case extra := <-pipe.events:
|
||||
t.Fatalf("late or duplicate event = %+v", extra)
|
||||
default:
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestTunnelWatchdogJoinsHealthEvidence proves the tunnel ERROR terminal joins
|
||||
// the bounded probe result under an unconfirmed close fence while retaining
|
||||
// provider-owned cleanup until the provider actually returns.
|
||||
func TestTunnelWatchdogJoinsHealthEvidence(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newProbingWatchdogAdapter("tunnel-health")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-health", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
probe := <-adapter.probeCalls
|
||||
if probe.ctx.Err() != nil {
|
||||
t.Fatal("tunnel health probe inherited the canceled request context")
|
||||
}
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: "tunnel-health", Target: "target", Status: runtime.ProviderStatusAvailable}}
|
||||
grace.fire() // provider does not return within grace -> unconfirmed
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
meta := terminal.GetMetadata()
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || meta["attempt_fence"] != "unconfirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
if meta["provider_health"] != "available" || meta["liveness_classification"] != "request_stalled" || meta["health_observation_seq"] != "1" {
|
||||
t.Fatalf("tunnel health evidence = %#v", meta)
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 1 || !n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("unconfirmed tunnel released ownership before provider return")
|
||||
}
|
||||
adapter.tunnelReturn <- nil
|
||||
waitForOwnershipRelease(t, n, adapter.Name(), "tunnel provider return did not release ownership")
|
||||
select {
|
||||
case extra := <-pipe.frames:
|
||||
t.Fatalf("late or duplicate frame = %+v", extra)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunWatchdogProbeEvidenceDoesNotResetProgress proves a positive
|
||||
// availability probe is evidence only: it never suppresses the stall terminal,
|
||||
// arms another activity timer, or revives local ownership.
|
||||
func TestRunWatchdogProbeEvidenceDoesNotResetProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newProbingWatchdogAdapter("run-noreset")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-noreset", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
<-adapter.probeCalls
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: "run-noreset", Target: "target", Status: runtime.ProviderStatusAvailable}}
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
terminal := waitRunEvent(t, pipe.events)
|
||||
if terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["failure_code"] != string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("available probe suppressed the stall terminal: %+v", terminal)
|
||||
}
|
||||
// Only the stall and close-grace timers were armed; probe evidence never reset
|
||||
// the activity watchdog.
|
||||
if clock.count() != 2 {
|
||||
t.Fatalf("probe evidence armed an extra timer: %d timers", clock.count())
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 0 || n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("available probe revived local ownership")
|
||||
}
|
||||
}
|
||||
|
||||
// TestWatchdogHealthObservationSeqIsConnectionScoped proves the sequence source
|
||||
// is shared by normalized and tunnel attempts on one Session, increases per
|
||||
// finalized observation, and resets on a new connection.
|
||||
func TestWatchdogHealthObservationSeqIsConnectionScoped(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("seq-adapter")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
runSeq := driveNormalizedConfirmedStall(t, n, pipe, adapter, clock, "seq-run", 0)
|
||||
if runSeq != "1" {
|
||||
t.Fatalf("first normalized observation seq = %q, want 1", runSeq)
|
||||
}
|
||||
tunnelSeq := driveTunnelConfirmedStall(t, n, pipe, adapter, clock, "seq-tunnel", "tunnel", 2)
|
||||
if tunnelSeq != "2" {
|
||||
t.Fatalf("tunnel observation seq on same connection = %q, want 2", tunnelSeq)
|
||||
}
|
||||
|
||||
pipe2 := newWatchdogPipe(t)
|
||||
resetSeq := driveNormalizedConfirmedStall(t, n, pipe2, adapter, clock, "seq-run-2", 4)
|
||||
if resetSeq != "1" {
|
||||
t.Fatalf("new-connection observation seq = %q, want 1", resetSeq)
|
||||
}
|
||||
}
|
||||
|
||||
func driveNormalizedConfirmedStall(t *testing.T, n *Node, pipe *watchdogPipe, adapter *controlledWatchdogAdapter, clock *manualAttemptClock, runID string, firstTimer int) string {
|
||||
t.Helper()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: runID, Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock.waitTimer(t, firstTimer).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, firstTimer+1)
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
return waitRunEvent(t, pipe.events).GetMetadata()["health_observation_seq"]
|
||||
}
|
||||
|
||||
func driveTunnelConfirmedStall(t *testing.T, n *Node, pipe *watchdogPipe, adapter *controlledWatchdogAdapter, clock *manualAttemptClock, runID, tunnelID string, firstTimer int) string {
|
||||
t.Helper()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: runID, TunnelId: tunnelID, Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, firstTimer).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, firstTimer+1)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
return waitTunnelFrame(t, pipe.frames).GetMetadata()["health_observation_seq"]
|
||||
}
|
||||
|
||||
// TestWatchdogOmitsHealthObservationSeqWithoutBoundSession proves an internal or
|
||||
// unbound execution path omits the sequence key entirely while health evidence
|
||||
// still fails closed to unknown.
|
||||
func TestWatchdogOmitsHealthObservationSeqWithoutBoundSession(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-nilseq")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
ticket, err := n.admissionFor(adapter.Name(), runtime.Capabilities{MaxConcurrency: 1}).acquire()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := runtime.ProviderTunnelRequest{RunID: "tunnel-nilseq", TunnelID: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMS: 1000}
|
||||
execCtx, cancel := context.WithCancel(context.Background())
|
||||
h := &runHandle{runID: tr.RunID, adapter: tr.Adapter, target: tr.Target, cancel: cancel, done: make(chan struct{})}
|
||||
n.runs.register(h)
|
||||
sender := &recordingProtoSender{}
|
||||
sink := &tunnelSink{sess: sender, observer: newAttemptObserver(clock, time.Second)}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- n.executeTunnelAttempt(execCtx, cancel, adapter, tr, sink, ticket, h, nil, nil, nil) }()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.tunnelReturn <- nil // confirmed
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
frames := sender.snapshot()
|
||||
if len(frames) != 1 {
|
||||
t.Fatalf("emitted frames = %d, want 1", len(frames))
|
||||
}
|
||||
meta := frames[0].(*iop.ProviderTunnelFrame).GetMetadata()
|
||||
if _, present := meta["health_observation_seq"]; present {
|
||||
t.Fatalf("unbound-session terminal carried a sequence: %#v", meta)
|
||||
}
|
||||
if meta["provider_health"] != "unknown" || meta["liveness_classification"] != "health_unknown" {
|
||||
t.Fatalf("nil-probe health = %#v", meta)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStallMetadataNormalizedAndTunnelParity(t *testing.T) {
|
||||
spec := runtime.ExecutionSpec{RunID: "parity-run", Adapter: "ollama", Target: "llama3"}
|
||||
obs := stallObservation{fence: "confirmed", idle: 3 * time.Second, health: HealthProbeEvidence{Health: runtime.RequestStalled, Status: runtime.ProviderStatusAvailable}, seq: 10, hasSeq: true}
|
||||
|
||||
normEvent := stalledRuntimeEvent(spec, obs)
|
||||
normProto := runEventToProto(normEvent, "node-1", "session-1", false)
|
||||
|
||||
tunnelReq := runtime.ProviderTunnelRequest{RunID: "parity-run", TunnelID: "tunnel-1", Adapter: "ollama", Target: "llama3"}
|
||||
tunnelFrame := stalledTunnelFrame(tunnelReq, obs)
|
||||
tunnelProto := tunnelFrameToProto(tunnelFrame, "node-1", "alias-1")
|
||||
|
||||
if normProto.GetFailure() == nil || tunnelProto.GetFailure() == nil {
|
||||
t.Fatalf("expected non-nil failure on both paths: norm=%#v tunnel=%#v", normProto.GetFailure(), tunnelProto.GetFailure())
|
||||
}
|
||||
if normProto.GetFailure().GetCode() != tunnelProto.GetFailure().GetCode() {
|
||||
t.Fatalf("code mismatch: norm=%q tunnel=%q", normProto.GetFailure().GetCode(), tunnelProto.GetFailure().GetCode())
|
||||
}
|
||||
if normProto.GetFailure().GetRetryable() != tunnelProto.GetFailure().GetRetryable() {
|
||||
t.Fatalf("retryable mismatch: norm=%v tunnel=%v", normProto.GetFailure().GetRetryable(), tunnelProto.GetFailure().GetRetryable())
|
||||
}
|
||||
for _, key := range []string{"failure_code", "provider_health", "liveness_classification", "idle_duration_ms", "run_id", "attempt_id", "attempt_fence", "adapter", "target", "health_observation_seq"} {
|
||||
if normProto.GetFailure().GetMetadata()[key] != tunnelProto.GetFailure().GetMetadata()[key] {
|
||||
t.Fatalf("metadata key %q mismatch: norm=%q tunnel=%q", key, normProto.GetFailure().GetMetadata()[key], tunnelProto.GetFailure().GetMetadata()[key])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure proto import is used by the test file (kept for compatibility).
|
||||
var _ = proto.Clone
|
||||
|
|
@ -1,191 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"go.uber.org/zap"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
)
|
||||
|
||||
// nodeLivenessObserver emits bounded, operator-queryable evidence for every
|
||||
// exactly-once claimed stall on either execution path. It is process-global in
|
||||
// production so repeated Node construction never re-registers metric names,
|
||||
// and it is test-injectable so package tests can verify the closed label set
|
||||
// and the safe log contract without touching the default prometheus registerer.
|
||||
//
|
||||
// The observer never changes stall detection, fence/probe ordering, terminal
|
||||
// delivery, or request/session/raw prompt/response handling. Observer failure
|
||||
// or disabled logging cannot suppress the terminal.
|
||||
type nodeLivenessObserver struct {
|
||||
stalls *prometheus.CounterVec
|
||||
duration *prometheus.HistogramVec
|
||||
logger *zap.Logger
|
||||
}
|
||||
|
||||
// productionStalls is the process-global counter registered once against the
|
||||
// default Prometheus registerer. Every Node reuses this single instance.
|
||||
var productionStalls *prometheus.CounterVec
|
||||
|
||||
// productionDuration is the process-global histogram registered once against
|
||||
// the default Prometheus registerer. Every Node reuses this single instance.
|
||||
var productionDuration *prometheus.HistogramVec
|
||||
|
||||
// init registers the production collector set exactly once with the default
|
||||
// Prometheus registerer. Per-Node construction never calls promauto or
|
||||
// MustRegister; test constructors supply an isolated registerer instead.
|
||||
func init() {
|
||||
productionStalls = prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Namespace: "iop",
|
||||
Subsystem: "node",
|
||||
Name: "response_stalls_total",
|
||||
Help: "Total claimed response stalls grouped by execution path, provider health, liveness classification, and attempt fence.",
|
||||
}, []string{"execution_path", "provider_health", "liveness_classification", "attempt_fence"})
|
||||
prometheus.MustRegister(productionStalls)
|
||||
|
||||
productionDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Namespace: "iop",
|
||||
Subsystem: "node",
|
||||
Name: "response_stall_duration_seconds",
|
||||
Help: "Idle duration in seconds for every claimed response stall.",
|
||||
Buckets: prometheus.ExponentialBuckets(0.05, 2, 10),
|
||||
NativeHistogramBucketFactor: 1.1,
|
||||
NativeHistogramMaxBucketNumber: 100,
|
||||
NativeHistogramMinResetDuration: 1 << 60,
|
||||
}, []string{"execution_path", "provider_health", "liveness_classification", "attempt_fence"})
|
||||
prometheus.MustRegister(productionDuration)
|
||||
}
|
||||
|
||||
// newProductionNodeLivenessObserver returns the shared production observer.
|
||||
// Tests must not call this; they call newNodeLivenessObserverForTest
|
||||
// with a private prometheus.Registry to avoid polluting the default registerer.
|
||||
func newProductionNodeLivenessObserver(logger *zap.Logger) *nodeLivenessObserver {
|
||||
return &nodeLivenessObserver{
|
||||
stalls: productionStalls,
|
||||
duration: productionDuration,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// newNodeLivenessObserverForTest returns an observer backed by a private
|
||||
// prometheus.Registry. The returned observer's Stalls and Duration fields
|
||||
// expose the underlying collectors so tests can inspect gathered metrics
|
||||
// without touching the process-wide default registerer.
|
||||
func newNodeLivenessObserverForTest(logger *zap.Logger, reg prometheus.Registerer) *nodeLivenessObserver {
|
||||
stalls := prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Namespace: "iop",
|
||||
Subsystem: "node",
|
||||
Name: "response_stalls_total",
|
||||
Help: "Total claimed response stalls grouped by execution path, provider health, liveness classification, and attempt fence.",
|
||||
}, []string{"execution_path", "provider_health", "liveness_classification", "attempt_fence"})
|
||||
reg.MustRegister(stalls)
|
||||
|
||||
duration := prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Namespace: "iop",
|
||||
Subsystem: "node",
|
||||
Name: "response_stall_duration_seconds",
|
||||
Help: "Idle duration in seconds for every claimed response stall.",
|
||||
Buckets: prometheus.ExponentialBuckets(0.05, 2, 10),
|
||||
NativeHistogramBucketFactor: 1.1,
|
||||
NativeHistogramMaxBucketNumber: 100,
|
||||
NativeHistogramMinResetDuration: 1 << 60,
|
||||
}, []string{"execution_path", "provider_health", "liveness_classification", "attempt_fence"})
|
||||
reg.MustRegister(duration)
|
||||
|
||||
return &nodeLivenessObserver{
|
||||
stalls: stalls,
|
||||
duration: duration,
|
||||
logger: logger,
|
||||
}
|
||||
}
|
||||
|
||||
// executionPathAllowlist enumerates the only values the observer accepts for
|
||||
// the execution_path label. Anything else is normalized to "unknown".
|
||||
var executionPathAllowlist = map[string]struct{}{
|
||||
"normalized": {},
|
||||
"provider_tunnel": {},
|
||||
}
|
||||
|
||||
// healthAllowlist enumerates the only values the observer accepts for the
|
||||
// provider_health label. Anything else is normalized to "unknown".
|
||||
var healthAllowlist = map[runtime.ProviderStatus]runtime.ProviderStatus{
|
||||
runtime.ProviderStatusAvailable: runtime.ProviderStatusAvailable,
|
||||
runtime.ProviderStatusUnavailable: runtime.ProviderStatusUnavailable,
|
||||
}
|
||||
|
||||
// classificationAllowlist enumerates the only values the observer accepts for
|
||||
// the liveness_classification label. Anything else is normalized to "health_unknown".
|
||||
var classificationAllowlist = map[runtime.ProviderHealth]runtime.ProviderHealth{
|
||||
runtime.RequestStalled: runtime.RequestStalled,
|
||||
runtime.ProviderUnhealthy: runtime.ProviderUnhealthy,
|
||||
}
|
||||
|
||||
// fenceAllowlist enumerates the only values the observer accepts for the
|
||||
// attempt_fence label. Anything else is normalized to "unknown".
|
||||
var fenceAllowlist = map[string]struct{}{
|
||||
"confirmed": {},
|
||||
"unconfirmed": {},
|
||||
}
|
||||
|
||||
// normalizeNodeLivenessLabels returns the closed four-tuple of label values
|
||||
// for the counter, histogram, and dedicated structured log. Every value is
|
||||
// validated against its allowlist; anything outside is normalized to "unknown"
|
||||
// so a future classification or status never leaks an unbounded cardinality
|
||||
// into the metric series.
|
||||
func normalizeNodeLivenessLabels(executionPath string, obs stallObservation) [4]string {
|
||||
var path string
|
||||
if _, ok := executionPathAllowlist[executionPath]; ok {
|
||||
path = executionPath
|
||||
} else {
|
||||
path = "unknown"
|
||||
}
|
||||
|
||||
health := runtime.ProviderStatusUnknown
|
||||
if v, ok := healthAllowlist[obs.health.Status]; ok {
|
||||
health = v
|
||||
}
|
||||
|
||||
classification := runtime.HealthUnknown
|
||||
if v, ok := classificationAllowlist[obs.health.Health]; ok {
|
||||
classification = v
|
||||
}
|
||||
|
||||
var fence string
|
||||
if _, ok := fenceAllowlist[obs.fence]; ok {
|
||||
fence = obs.fence
|
||||
} else {
|
||||
fence = "unknown"
|
||||
}
|
||||
|
||||
return [4]string{path, string(health), string(classification), fence}
|
||||
}
|
||||
|
||||
// Observe emits one counter observation, one duration sample, and one
|
||||
// structured log entry for the given claimed stall. It is invoked exactly
|
||||
// once per claimed stall from the production watchdog seams.
|
||||
//
|
||||
// Observer failure never suppresses the terminal: metrics and logs are
|
||||
// fire-and-forget evidence; the terminal is the delivery contract.
|
||||
func (o *nodeLivenessObserver) Observe(executionPath string, obs stallObservation) {
|
||||
if o == nil {
|
||||
return
|
||||
}
|
||||
defer func() { _ = recover() }()
|
||||
|
||||
labels := normalizeNodeLivenessLabels(executionPath, obs)
|
||||
|
||||
o.stalls.WithLabelValues(labels[0], labels[1], labels[2], labels[3]).Inc()
|
||||
o.duration.WithLabelValues(labels[0], labels[1], labels[2], labels[3]).Observe(obs.idle.Seconds())
|
||||
|
||||
if o.logger == nil {
|
||||
return
|
||||
}
|
||||
|
||||
o.logger.Info(
|
||||
"node_response_stall_observation",
|
||||
zap.String("execution_path", labels[0]),
|
||||
zap.String("provider_health", labels[1]),
|
||||
zap.String("liveness_classification", labels[2]),
|
||||
zap.String("attempt_fence", labels[3]),
|
||||
zap.Int64("idle_duration_ms", obs.idle.Milliseconds()),
|
||||
)
|
||||
}
|
||||
|
|
@ -1,680 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
dto "github.com/prometheus/client_model/go"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zapcore"
|
||||
"google.golang.org/protobuf/types/known/structpb"
|
||||
|
||||
"iop/apps/node/internal/store"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// TestNodeLivenessObservability proves the bounded Node stall-observability
|
||||
// contract on deterministic normalized and tunnel fixtures. It covers the four
|
||||
// path/health outcomes (available/request-stalled and unavailable/provider-
|
||||
// unhealthy on both paths), verifies exact metric families/labels and allow-
|
||||
// listed values, asserts one dedicated log per claimed stall, and rejects
|
||||
// high-cardinality raw values from both the metric labels and the structured
|
||||
// log. It also proves unknown label normalization, logger panic containment, and
|
||||
// repeated default Node construction.
|
||||
func TestNodeLivenessObservability(t *testing.T) {
|
||||
t.Run("normalized/request-stalled", testNormalizedRequestStalled)
|
||||
t.Run("normalized/provider-unhealthy", testNormalizedProviderUnhealthy)
|
||||
t.Run("provider_tunnel/request-stalled", testTunnelRequestStalled)
|
||||
t.Run("provider_tunnel/provider-unhealthy", testTunnelProviderUnhealthy)
|
||||
t.Run("unknown-normalization", testUnknownNormalization)
|
||||
t.Run("failure-isolation", testFailureIsolation)
|
||||
t.Run("repeated-default-construction", testRepeatedDefaultConstruction)
|
||||
}
|
||||
|
||||
type evidenceExpectation struct {
|
||||
path string
|
||||
health string
|
||||
classification string
|
||||
fence string
|
||||
counter float64
|
||||
histogramCount uint64
|
||||
idleMS int64
|
||||
hostileSentinels []string
|
||||
}
|
||||
|
||||
func assertNodeLivenessEvidence(t *testing.T, reg *prometheus.Registry, logs *testLogCore, exp evidenceExpectation) {
|
||||
t.Helper()
|
||||
|
||||
gathered, err := reg.Gather()
|
||||
if err != nil {
|
||||
t.Fatalf("gather error: %v", err)
|
||||
}
|
||||
|
||||
// 1. Counter assertion
|
||||
wantCounter := findMetric(gathered, "iop_node_response_stalls_total")
|
||||
if wantCounter == nil {
|
||||
t.Fatal("counter iop_node_response_stalls_total not found")
|
||||
}
|
||||
if len(wantCounter.GetMetric()) != 1 {
|
||||
t.Fatalf("counter metric series count = %d, want 1", len(wantCounter.GetMetric()))
|
||||
}
|
||||
gotCounter := wantCounter.GetMetric()[0]
|
||||
if gotCounter.GetCounter().GetValue() != exp.counter {
|
||||
t.Fatalf("counter value = %v, want %v", gotCounter.GetCounter().GetValue(), exp.counter)
|
||||
}
|
||||
counterLabelMap := dtoLabelMap(gotCounter.GetLabel())
|
||||
if len(counterLabelMap) != 4 {
|
||||
t.Fatalf("counter label count = %d, want 4 (labels=%v)", len(counterLabelMap), counterLabelMap)
|
||||
}
|
||||
assertLabel(t, counterLabelMap, "execution_path", exp.path)
|
||||
assertLabel(t, counterLabelMap, "provider_health", exp.health)
|
||||
assertLabel(t, counterLabelMap, "liveness_classification", exp.classification)
|
||||
assertLabel(t, counterLabelMap, "attempt_fence", exp.fence)
|
||||
|
||||
// 2. Histogram assertion
|
||||
wantHist := findMetric(gathered, "iop_node_response_stall_duration_seconds")
|
||||
if wantHist == nil {
|
||||
t.Fatal("histogram iop_node_response_stall_duration_seconds not found")
|
||||
}
|
||||
if len(wantHist.GetMetric()) != 1 {
|
||||
t.Fatalf("histogram metric series count = %d, want 1", len(wantHist.GetMetric()))
|
||||
}
|
||||
gotHist := wantHist.GetMetric()[0]
|
||||
if gotHist.GetHistogram().GetSampleCount() != exp.histogramCount {
|
||||
t.Fatalf("histogram sample count = %d, want %d", gotHist.GetHistogram().GetSampleCount(), exp.histogramCount)
|
||||
}
|
||||
expectedSec := float64(exp.idleMS) / 1000.0
|
||||
if gotHist.GetHistogram().GetSampleSum() < expectedSec*0.99 || gotHist.GetHistogram().GetSampleSum() > expectedSec*1.01 {
|
||||
t.Fatalf("histogram sample sum = %v, want ~%v", gotHist.GetHistogram().GetSampleSum(), expectedSec)
|
||||
}
|
||||
histLabelMap := dtoLabelMap(gotHist.GetLabel())
|
||||
if len(histLabelMap) != 4 {
|
||||
t.Fatalf("histogram label count = %d, want 4 (labels=%v)", len(histLabelMap), histLabelMap)
|
||||
}
|
||||
assertLabel(t, histLabelMap, "execution_path", exp.path)
|
||||
assertLabel(t, histLabelMap, "provider_health", exp.health)
|
||||
assertLabel(t, histLabelMap, "liveness_classification", exp.classification)
|
||||
assertLabel(t, histLabelMap, "attempt_fence", exp.fence)
|
||||
|
||||
// 3. Log entry assertion
|
||||
logs.mu.Lock()
|
||||
entries := make([]testLogEntry, len(logs.entries))
|
||||
copy(entries, logs.entries)
|
||||
logs.mu.Unlock()
|
||||
|
||||
var matching []testLogEntry
|
||||
for _, entry := range entries {
|
||||
if entry.Message == "node_response_stall_observation" {
|
||||
matching = append(matching, entry)
|
||||
}
|
||||
}
|
||||
if len(matching) != 1 {
|
||||
t.Fatalf("dedicated stall observation log count = %d, want 1 (total log entries = %d)", len(matching), len(entries))
|
||||
}
|
||||
entry := matching[0]
|
||||
if entry.Level != zapcore.InfoLevel {
|
||||
t.Fatalf("log level = %v, want Info", entry.Level)
|
||||
}
|
||||
if len(entry.Fields) != 5 {
|
||||
t.Fatalf("log field count = %d, want 5 (fields=%+v)", len(entry.Fields), entry.Fields)
|
||||
}
|
||||
|
||||
var foundPath, foundHealth, foundClass, foundFence bool
|
||||
var foundDuration int64
|
||||
var durationType zapcore.FieldType
|
||||
for _, f := range entry.Fields {
|
||||
switch f.Key {
|
||||
case "execution_path":
|
||||
foundPath = true
|
||||
if f.String != exp.path {
|
||||
t.Fatalf("field execution_path = %q, want %q", f.String, exp.path)
|
||||
}
|
||||
case "provider_health":
|
||||
foundHealth = true
|
||||
if f.String != exp.health {
|
||||
t.Fatalf("field provider_health = %q, want %q", f.String, exp.health)
|
||||
}
|
||||
case "liveness_classification":
|
||||
foundClass = true
|
||||
if f.String != exp.classification {
|
||||
t.Fatalf("field liveness_classification = %q, want %q", f.String, exp.classification)
|
||||
}
|
||||
case "attempt_fence":
|
||||
foundFence = true
|
||||
if f.String != exp.fence {
|
||||
t.Fatalf("field attempt_fence = %q, want %q", f.String, exp.fence)
|
||||
}
|
||||
case "idle_duration_ms":
|
||||
foundDuration = f.Integer
|
||||
durationType = f.Type
|
||||
default:
|
||||
t.Fatalf("unexpected log field key %q", f.Key)
|
||||
}
|
||||
}
|
||||
if !foundPath || !foundHealth || !foundClass || !foundFence {
|
||||
t.Fatalf("missing expected string fields in log entry: %+v", entry.Fields)
|
||||
}
|
||||
if durationType != zapcore.Int64Type {
|
||||
t.Fatalf("idle_duration_ms type = %v, want Int64Type (%v)", durationType, zapcore.Int64Type)
|
||||
}
|
||||
if foundDuration != exp.idleMS {
|
||||
t.Fatalf("idle_duration_ms value = %d, want %d", foundDuration, exp.idleMS)
|
||||
}
|
||||
|
||||
// 4. Encoded JSON field assertions
|
||||
encoder := zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig())
|
||||
item, err := encoder.EncodeEntry(zapcore.Entry{
|
||||
Level: entry.Level,
|
||||
Message: entry.Message,
|
||||
}, entry.Fields)
|
||||
if err != nil {
|
||||
t.Fatalf("encode log entry: %v", err)
|
||||
}
|
||||
encodedJSON := item.String()
|
||||
expectedNumJSON := fmt.Sprintf(`"idle_duration_ms":%d`, exp.idleMS)
|
||||
if !strings.Contains(encodedJSON, expectedNumJSON) {
|
||||
t.Fatalf("encoded JSON log %q does not contain expected numeric field %q", encodedJSON, expectedNumJSON)
|
||||
}
|
||||
|
||||
// 5. Hostile sentinel rejection
|
||||
for _, sentinel := range exp.hostileSentinels {
|
||||
if sentinel == "" {
|
||||
continue
|
||||
}
|
||||
for _, mf := range gathered {
|
||||
for _, m := range mf.GetMetric() {
|
||||
for _, l := range m.GetLabel() {
|
||||
if l.GetName() == sentinel || strings.Contains(l.GetName(), sentinel) {
|
||||
t.Fatalf("sentinel %q leaked into metric label name %q", sentinel, l.GetName())
|
||||
}
|
||||
if l.GetValue() == sentinel || strings.Contains(l.GetValue(), sentinel) {
|
||||
t.Fatalf("sentinel %q leaked into metric label value %q", sentinel, l.GetValue())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if strings.Contains(encodedJSON, sentinel) {
|
||||
t.Fatalf("sentinel %q leaked into encoded JSON log %q", sentinel, encodedJSON)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func assertNoAdditionalTerminal[T any](t *testing.T, ch <-chan T) {
|
||||
t.Helper()
|
||||
select {
|
||||
case msg := <-ch:
|
||||
t.Fatalf("unexpected additional terminal message: %+v", msg)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func testNormalizedRequestStalled(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
logger, logs := newTestLogger()
|
||||
adapterName := "hostile-adapter-norm-avail"
|
||||
target := "hostile-target-norm-avail"
|
||||
runID := "obs-norm-avail-spoof-run-id"
|
||||
sessionID := "spoof-session-norm-avail"
|
||||
requestID := "spoof-request-id-norm-avail"
|
||||
prompt := "raw-prompt-norm-avail"
|
||||
response := "raw-response-norm-avail"
|
||||
credential := "raw-credential-norm-avail"
|
||||
|
||||
sentinels := []string{runID, sessionID, adapterName, target, requestID, prompt, response, credential}
|
||||
|
||||
adapter := newProbingWatchdogAdapter(adapterName)
|
||||
n := newNodeWithObserver(t, adapter, reg, logger)
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{
|
||||
RunId: runID,
|
||||
Adapter: adapter.Name(),
|
||||
Target: target,
|
||||
SessionId: sessionID,
|
||||
ResponseStallTimeoutMs: 500,
|
||||
Input: &structpb.Struct{Fields: map[string]*structpb.Value{"prompt": structpb.NewStringValue(prompt)}},
|
||||
Metadata: map[string]string{"request_id": requestID, "response": response, "credential": credential},
|
||||
})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: target, Status: runtime.ProviderStatusAvailable}}
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
terminal := waitRunEvent(t, pipe.events)
|
||||
if terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
|
||||
assertNodeLivenessEvidence(t, reg, logs, evidenceExpectation{
|
||||
path: "normalized",
|
||||
health: "available",
|
||||
classification: "request_stalled",
|
||||
fence: "confirmed",
|
||||
counter: 1,
|
||||
histogramCount: 1,
|
||||
idleMS: 500,
|
||||
hostileSentinels: sentinels,
|
||||
})
|
||||
}
|
||||
|
||||
func testNormalizedProviderUnhealthy(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
logger, logs := newTestLogger()
|
||||
adapterName := "hostile-adapter-norm-unavail"
|
||||
target := "hostile-target-norm-unavail"
|
||||
runID := "obs-norm-unavail-spoof-run-id"
|
||||
sessionID := "spoof-session-norm-unavail"
|
||||
requestID := "spoof-request-id-norm-unavail"
|
||||
prompt := "raw-prompt-norm-unavail"
|
||||
response := "raw-response-norm-unavail"
|
||||
credential := "raw-credential-norm-unavail"
|
||||
|
||||
sentinels := []string{runID, sessionID, adapterName, target, requestID, prompt, response, credential}
|
||||
|
||||
adapter := newProbingWatchdogAdapter(adapterName)
|
||||
n := newNodeWithObserver(t, adapter, reg, logger)
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{
|
||||
RunId: runID,
|
||||
Adapter: adapter.Name(),
|
||||
Target: target,
|
||||
SessionId: sessionID,
|
||||
ResponseStallTimeoutMs: 500,
|
||||
Input: &structpb.Struct{Fields: map[string]*structpb.Value{"prompt": structpb.NewStringValue(prompt)}},
|
||||
Metadata: map[string]string{"request_id": requestID, "response": response, "credential": credential},
|
||||
})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: target, Status: runtime.ProviderStatusUnavailable}}
|
||||
|
||||
grace := clock.waitTimer(t, 1)
|
||||
grace.fire()
|
||||
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
terminal := waitRunEvent(t, pipe.events)
|
||||
if terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "unconfirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
|
||||
assertNodeLivenessEvidence(t, reg, logs, evidenceExpectation{
|
||||
path: "normalized",
|
||||
health: "unavailable",
|
||||
classification: "provider_unhealthy",
|
||||
fence: "unconfirmed",
|
||||
counter: 1,
|
||||
histogramCount: 1,
|
||||
idleMS: 500,
|
||||
hostileSentinels: sentinels,
|
||||
})
|
||||
}
|
||||
|
||||
func testTunnelRequestStalled(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
logger, logs := newTestLogger()
|
||||
adapterName := "hostile-adapter-tun-avail"
|
||||
target := "hostile-target-tun-avail"
|
||||
runID := "obs-tun-avail-spoof-run-id"
|
||||
tunnelID := "tunnel-obs-spoof-id"
|
||||
sessionID := "spoof-session-tun-avail"
|
||||
requestID := "spoof-request-id-tun-avail"
|
||||
headerVal := "raw-header-tun-avail"
|
||||
bodyVal := "raw-body-tun-avail"
|
||||
responseVal := "raw-response-tun-avail"
|
||||
credentialVal := "raw-credential-tun-avail"
|
||||
|
||||
sentinels := []string{runID, tunnelID, adapterName, target, sessionID, requestID, headerVal, bodyVal, responseVal, credentialVal}
|
||||
|
||||
adapter := newProbingWatchdogAdapter(adapterName)
|
||||
n := newNodeWithObserver(t, adapter, reg, logger)
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{
|
||||
RunId: runID,
|
||||
TunnelId: tunnelID,
|
||||
Adapter: adapter.Name(),
|
||||
Target: target,
|
||||
SessionId: sessionID,
|
||||
Headers: map[string]string{"authorization": credentialVal, "request_id": requestID, "x-header": headerVal},
|
||||
Body: []byte(bodyVal),
|
||||
Metadata: map[string]string{"response": responseVal},
|
||||
ResponseStallTimeoutMs: 500,
|
||||
})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
if call.req.RunID != runID || call.req.TunnelID != tunnelID || call.req.Adapter != adapter.Name() || call.req.Target != target || call.req.SessionID != sessionID || call.req.Headers["authorization"] != credentialVal || call.req.Headers["request_id"] != requestID || call.req.Headers["x-header"] != headerVal || string(call.req.Body) != bodyVal || call.req.Metadata["response"] != responseVal {
|
||||
t.Fatalf("captured tunnel request mismatch: %#v", call.req)
|
||||
}
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: target, Status: runtime.ProviderStatusAvailable}}
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
|
||||
assertNodeLivenessEvidence(t, reg, logs, evidenceExpectation{
|
||||
path: "provider_tunnel",
|
||||
health: "available",
|
||||
classification: "request_stalled",
|
||||
fence: "confirmed",
|
||||
counter: 1,
|
||||
histogramCount: 1,
|
||||
idleMS: 500,
|
||||
hostileSentinels: sentinels,
|
||||
})
|
||||
}
|
||||
|
||||
func testTunnelProviderUnhealthy(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
logger, logs := newTestLogger()
|
||||
adapterName := "hostile-adapter-tun-unavail"
|
||||
target := "hostile-target-tun-unavail"
|
||||
runID := "obs-tun-unavail-spoof-run-id"
|
||||
tunnelID := "tunnel-unavail-spoof-id"
|
||||
sessionID := "spoof-session-tun-unavail"
|
||||
requestID := "spoof-request-id-tun-unavail"
|
||||
headerVal := "raw-header-tun-unavail"
|
||||
bodyVal := "raw-body-tun-unavail"
|
||||
responseVal := "raw-response-tun-unavail"
|
||||
credentialVal := "raw-credential-tun-unavail"
|
||||
|
||||
sentinels := []string{runID, tunnelID, adapterName, target, sessionID, requestID, headerVal, bodyVal, responseVal, credentialVal}
|
||||
|
||||
adapter := newProbingWatchdogAdapter(adapterName)
|
||||
n := newNodeWithObserver(t, adapter, reg, logger)
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{
|
||||
RunId: runID,
|
||||
TunnelId: tunnelID,
|
||||
Adapter: adapter.Name(),
|
||||
Target: target,
|
||||
SessionId: sessionID,
|
||||
Headers: map[string]string{"authorization": credentialVal, "request_id": requestID, "x-header": headerVal},
|
||||
Body: []byte(bodyVal),
|
||||
Metadata: map[string]string{"response": responseVal},
|
||||
ResponseStallTimeoutMs: 500,
|
||||
})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
if call.req.RunID != runID || call.req.TunnelID != tunnelID || call.req.Adapter != adapter.Name() || call.req.Target != target || call.req.SessionID != sessionID || call.req.Headers["authorization"] != credentialVal || call.req.Headers["request_id"] != requestID || call.req.Headers["x-header"] != headerVal || string(call.req.Body) != bodyVal || call.req.Metadata["response"] != responseVal {
|
||||
t.Fatalf("captured tunnel request mismatch: %#v", call.req)
|
||||
}
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: target, Status: runtime.ProviderStatusUnavailable}}
|
||||
|
||||
grace := clock.waitTimer(t, 1)
|
||||
grace.fire()
|
||||
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "unconfirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
|
||||
assertNodeLivenessEvidence(t, reg, logs, evidenceExpectation{
|
||||
path: "provider_tunnel",
|
||||
health: "unavailable",
|
||||
classification: "provider_unhealthy",
|
||||
fence: "unconfirmed",
|
||||
counter: 1,
|
||||
histogramCount: 1,
|
||||
idleMS: 500,
|
||||
hostileSentinels: sentinels,
|
||||
})
|
||||
}
|
||||
|
||||
func testUnknownNormalization(t *testing.T) {
|
||||
labels := normalizeNodeLivenessLabels("invalid_path", stallObservation{
|
||||
health: HealthProbeEvidence{
|
||||
Status: runtime.ProviderStatus("invalid_status"),
|
||||
Health: runtime.ProviderHealth("invalid_health"),
|
||||
},
|
||||
fence: "invalid_fence",
|
||||
})
|
||||
want := [4]string{"unknown", "unknown", "health_unknown", "unknown"}
|
||||
if labels != want {
|
||||
t.Fatalf("normalizeNodeLivenessLabels = %v, want %v", labels, want)
|
||||
}
|
||||
}
|
||||
|
||||
type panickingLogCore struct{}
|
||||
|
||||
func (p *panickingLogCore) Enabled(zapcore.Level) bool { return true }
|
||||
func (p *panickingLogCore) With([]zap.Field) zapcore.Core { return p }
|
||||
func (p *panickingLogCore) Check(e zapcore.Entry, ce *zapcore.CheckedEntry) *zapcore.CheckedEntry {
|
||||
return ce.AddCore(e, p)
|
||||
}
|
||||
func (p *panickingLogCore) Write(zapcore.Entry, []zap.Field) error {
|
||||
panic("simulated logger panic")
|
||||
}
|
||||
func (p *panickingLogCore) Sync() error { return nil }
|
||||
|
||||
func testFailureIsolation(t *testing.T) {
|
||||
t.Run("normalized", func(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
panickingLogger := zap.New(&panickingLogCore{})
|
||||
adapter := newProbingWatchdogAdapter("obs-panic-norm")
|
||||
n := newNodeWithObserver(t, adapter, reg, zap.NewNop())
|
||||
n.liveness.logger = panickingLogger
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{
|
||||
RunId: "obs-panic-norm",
|
||||
Adapter: adapter.Name(),
|
||||
Target: "target",
|
||||
ResponseStallTimeoutMs: 500,
|
||||
})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: "target", Status: runtime.ProviderStatusAvailable}}
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v, want errProviderResponseStalled", err)
|
||||
}
|
||||
terminal := waitRunEvent(t, pipe.events)
|
||||
if terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
assertNoAdditionalTerminal(t, pipe.events)
|
||||
})
|
||||
|
||||
t.Run("tunnel", func(t *testing.T) {
|
||||
reg := prometheus.NewRegistry()
|
||||
panickingLogger := zap.New(&panickingLogCore{})
|
||||
adapter := newProbingWatchdogAdapter("obs-panic-tun")
|
||||
n := newNodeWithObserver(t, adapter, reg, zap.NewNop())
|
||||
n.liveness.logger = panickingLogger
|
||||
pipe := newWatchdogPipe(t)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{
|
||||
RunId: "obs-panic-tun",
|
||||
TunnelId: "tunnel-panic",
|
||||
Adapter: adapter.Name(),
|
||||
Target: "target",
|
||||
ResponseStallTimeoutMs: 500,
|
||||
})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock := n.watchdogClock.(*manualAttemptClock)
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1)
|
||||
adapter.probeReturn <- probeReply{result: runtime.ProviderProbeResult{AdapterName: adapter.Name(), Target: "target", Status: runtime.ProviderStatusAvailable}}
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v, want errProviderResponseStalled", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
assertNoAdditionalTerminal(t, pipe.frames)
|
||||
})
|
||||
}
|
||||
|
||||
func testRepeatedDefaultConstruction(t *testing.T) {
|
||||
st, err := store.New(":memory:", zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = st.Close() }()
|
||||
|
||||
for i := 0; i < 50; i++ {
|
||||
_ = New("node-dup-"+string(rune('a'+i%26)), &noopRouter{}, st, 0, io.Discard, zap.NewNop(), nil)
|
||||
}
|
||||
}
|
||||
|
||||
// --- Test helpers ---
|
||||
|
||||
type testLogEntry struct {
|
||||
Level zapcore.Level
|
||||
Message string
|
||||
Fields []zap.Field
|
||||
}
|
||||
|
||||
type testLogCore struct {
|
||||
mu sync.Mutex
|
||||
entries []testLogEntry
|
||||
}
|
||||
|
||||
func newTestLogCore() *testLogCore {
|
||||
return &testLogCore{}
|
||||
}
|
||||
|
||||
func (c *testLogCore) Enabled(lvl zapcore.Level) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
func (c *testLogCore) With(fields []zap.Field) zapcore.Core {
|
||||
return c
|
||||
}
|
||||
|
||||
func (c *testLogCore) Check(entry zapcore.Entry, ce *zapcore.CheckedEntry) *zapcore.CheckedEntry {
|
||||
if c.Enabled(entry.Level) {
|
||||
return ce.AddCore(entry, c)
|
||||
}
|
||||
return ce
|
||||
}
|
||||
|
||||
func (c *testLogCore) Write(entry zapcore.Entry, fields []zap.Field) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.entries = append(c.entries, testLogEntry{
|
||||
Level: entry.Level,
|
||||
Message: entry.Message,
|
||||
Fields: fields,
|
||||
})
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *testLogCore) Sync() error { return nil }
|
||||
|
||||
func newTestLogger() (*zap.Logger, *testLogCore) {
|
||||
core := newTestLogCore()
|
||||
logger := zap.New(core)
|
||||
return logger, core
|
||||
}
|
||||
|
||||
func findMetric(gathered []*dto.MetricFamily, name string) *dto.MetricFamily {
|
||||
for _, mf := range gathered {
|
||||
if mf.GetName() == name {
|
||||
return mf
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func dtoLabelMap(labels []*dto.LabelPair) map[string]string {
|
||||
m := make(map[string]string, len(labels))
|
||||
for _, l := range labels {
|
||||
m[l.GetName()] = l.GetValue()
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
func assertLabel(t *testing.T, labels map[string]string, name, want string) {
|
||||
t.Helper()
|
||||
got, ok := labels[name]
|
||||
if !ok {
|
||||
t.Fatalf("label %q missing, labels=%v", name, labels)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("label %s = %q, want %q", name, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
type noopRouter struct{}
|
||||
|
||||
func (r *noopRouter) Resolve(_ context.Context, _ runtime.RunRequest) (runtime.ExecutionSpec, error) {
|
||||
return runtime.ExecutionSpec{}, errors.New("noop")
|
||||
}
|
||||
func (r *noopRouter) ResolveAdapter(_ context.Context, _ runtime.RunRequest) (runtime.ExecutionSpec, runtime.Provider, error) {
|
||||
return runtime.ExecutionSpec{}, nil, errors.New("noop")
|
||||
}
|
||||
func (r *noopRouter) LookupAdapter(_ string) (runtime.Provider, error) {
|
||||
return nil, errors.New("noop")
|
||||
}
|
||||
func (r *noopRouter) GetAdapter(_ string) (runtime.Provider, bool) {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func newNodeWithObserver(t *testing.T, adapter runtime.ProviderTunnelAdapter, reg prometheus.Registerer, logger *zap.Logger) *Node {
|
||||
t.Helper()
|
||||
st, err := store.New(":memory:", zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
n := New("node-obs", &watchdogRouter{adapter: adapter}, st, 0, io.Discard, logger, nil)
|
||||
n.watchdogClock = newManualAttemptClock()
|
||||
n.liveness = newNodeLivenessObserverForTest(logger, reg)
|
||||
return n
|
||||
}
|
||||
|
|
@ -1,516 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"iop/packages/go/credentiallease"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// defaultAttemptCloseGrace bounds the wait after cancellation used to decide
|
||||
// whether an adapter has actually relinquished local ownership.
|
||||
const defaultAttemptCloseGrace = 5 * time.Second
|
||||
|
||||
var errProviderResponseStalled = errors.New("provider response stalled")
|
||||
|
||||
// attemptClock is intentionally small so package tests can provide a manual
|
||||
// timer without relying on scheduler sleeps.
|
||||
type attemptClock interface {
|
||||
NewTimer(time.Duration) attemptTimer
|
||||
Now() time.Time
|
||||
}
|
||||
type attemptTimer interface {
|
||||
C() <-chan time.Time
|
||||
Stop() bool
|
||||
Reset(time.Duration) bool
|
||||
}
|
||||
type realAttemptClock struct{}
|
||||
type realAttemptTimer struct{ timer *time.Timer }
|
||||
|
||||
func (realAttemptClock) NewTimer(d time.Duration) attemptTimer {
|
||||
return realAttemptTimer{timer: time.NewTimer(d)}
|
||||
}
|
||||
func (realAttemptClock) Now() time.Time { return time.Now() }
|
||||
func (t realAttemptTimer) C() <-chan time.Time { return t.timer.C }
|
||||
func (t realAttemptTimer) Stop() bool { return t.timer.Stop() }
|
||||
func (t realAttemptTimer) Reset(d time.Duration) bool { return t.timer.Reset(d) }
|
||||
|
||||
// attemptObserver owns one response activity timer. Terminal and fenced
|
||||
// states are monotonic so late provider output can never revive an attempt.
|
||||
type attemptObserver struct {
|
||||
mu sync.Mutex
|
||||
clock attemptClock
|
||||
timer attemptTimer
|
||||
deadline time.Duration
|
||||
expiresAt time.Time
|
||||
terminal bool
|
||||
fenced bool
|
||||
epoch uint64
|
||||
|
||||
// beforeExpiryCapture and afterExpiryCapture are deterministic ordering
|
||||
// seams for package tests that exercise the receive-before-capture race.
|
||||
beforeExpiryCapture func()
|
||||
afterExpiryCapture func(bool)
|
||||
}
|
||||
|
||||
func newAttemptObserver(clock attemptClock, timeout time.Duration) *attemptObserver {
|
||||
if clock == nil {
|
||||
clock = realAttemptClock{}
|
||||
}
|
||||
// Record the arm's scheduled deadline before creating its timer. A very
|
||||
// short timer can signal while NewTimer is still returning; that signal is
|
||||
// nevertheless the current arm and must not be rejected as stale.
|
||||
armedAt := clock.Now()
|
||||
o := &attemptObserver{clock: clock, deadline: timeout, expiresAt: armedAt.Add(timeout)}
|
||||
o.timer = clock.NewTimer(timeout)
|
||||
return o
|
||||
}
|
||||
func (o *attemptObserver) expired() <-chan time.Time { return o.timer.C() }
|
||||
func (o *attemptObserver) observe(disposition runtime.ProviderActivityDisposition) {
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
if o.terminal || o.fenced {
|
||||
return
|
||||
}
|
||||
switch disposition {
|
||||
case runtime.DispositionProgress:
|
||||
// A timer can have an unread expiry while progress arrives. Drain that
|
||||
// expiry before rearming, advance the epoch, and record the next arm's
|
||||
// scheduled deadline before Reset can make it observable. A stale expiry
|
||||
// consumed by the watchdog cannot then fence the reset attempt whether its
|
||||
// validity is captured before or after this reset.
|
||||
if !o.timer.Stop() {
|
||||
select {
|
||||
case <-o.timer.C():
|
||||
default:
|
||||
}
|
||||
}
|
||||
o.epoch++
|
||||
o.expiresAt = o.clock.Now().Add(o.deadline)
|
||||
o.timer.Reset(o.deadline)
|
||||
case runtime.DispositionTerminal:
|
||||
o.terminal = true
|
||||
o.timer.Stop()
|
||||
}
|
||||
}
|
||||
|
||||
type attemptExpiry struct{ epoch uint64 }
|
||||
|
||||
// expiryForSignal binds a consumed timer signal to the arm that produced it.
|
||||
// A progress reset advances expiresAt past a stale signal's fire time and bumps
|
||||
// the epoch, so a signal received before that reset is rejected here even
|
||||
// though its epoch was never captured against the old arm. The returned epoch
|
||||
// continues to guard the post-capture race up to claimFence, where a progress
|
||||
// reset that begins after this capture is also rejected.
|
||||
func (o *attemptObserver) expiryForSignal(firedAt time.Time) (attemptExpiry, bool) {
|
||||
if o.beforeExpiryCapture != nil {
|
||||
o.beforeExpiryCapture()
|
||||
}
|
||||
o.mu.Lock()
|
||||
valid := !o.terminal && !o.fenced && !firedAt.Before(o.expiresAt)
|
||||
expiry := attemptExpiry{epoch: o.epoch}
|
||||
o.mu.Unlock()
|
||||
if o.afterExpiryCapture != nil {
|
||||
o.afterExpiryCapture(valid)
|
||||
}
|
||||
if !valid {
|
||||
return attemptExpiry{}, false
|
||||
}
|
||||
return expiry, true
|
||||
}
|
||||
|
||||
func (o *attemptObserver) claimFence(expiry attemptExpiry) bool {
|
||||
o.mu.Lock()
|
||||
defer o.mu.Unlock()
|
||||
if o.terminal || o.fenced || o.epoch != expiry.epoch {
|
||||
return false
|
||||
}
|
||||
o.fenced = true
|
||||
o.timer.Stop()
|
||||
return true
|
||||
}
|
||||
|
||||
func cloneLivenessMetadata(metadata map[string]string) map[string]string {
|
||||
cloned := make(map[string]string, len(metadata))
|
||||
for key, value := range metadata {
|
||||
cloned[key] = value
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
func stalledRuntimeEvent(spec runtime.ExecutionSpec, obs stallObservation) runtime.RuntimeEvent {
|
||||
metadata := stallMetadata(spec.RunID, spec.Adapter, spec.Target, obs)
|
||||
return runtime.RuntimeEvent{RunID: spec.RunID, Type: runtime.EventTypeError, Timestamp: time.Now(), Error: "provider response stalled",
|
||||
Failure: &runtime.Failure{Code: runtime.FailureCodeResponseStalled, Message: "provider response stalled", Retryable: obs.fence == "confirmed", Metadata: cloneLivenessMetadata(metadata)}, Metadata: cloneLivenessMetadata(metadata)}
|
||||
}
|
||||
func stalledTunnelFrame(req runtime.ProviderTunnelRequest, obs stallObservation) runtime.ProviderTunnelFrame {
|
||||
metadata := stallMetadata(req.RunID, req.Adapter, req.Target, obs)
|
||||
return runtime.ProviderTunnelFrame{
|
||||
RunID: req.RunID,
|
||||
TunnelID: req.TunnelID,
|
||||
Kind: runtime.ProviderTunnelFrameKindError,
|
||||
Error: "provider response stalled",
|
||||
Timestamp: time.Now(),
|
||||
Failure: &runtime.Failure{
|
||||
Code: runtime.FailureCodeResponseStalled,
|
||||
Message: "provider response stalled",
|
||||
Retryable: obs.fence == "confirmed",
|
||||
Metadata: cloneLivenessMetadata(metadata),
|
||||
},
|
||||
Metadata: cloneLivenessMetadata(metadata),
|
||||
}
|
||||
}
|
||||
|
||||
// awaitAttempt owns the race between provider return, the request boundary,
|
||||
// and the response-stall timer. A confirmed result means provider return was
|
||||
// observed within close grace; otherwise resource cleanup remains provider-owned.
|
||||
func awaitAttempt(
|
||||
execCtx context.Context,
|
||||
cancel context.CancelFunc,
|
||||
clock attemptClock,
|
||||
observer *attemptObserver,
|
||||
claimStall func(attemptExpiry) bool,
|
||||
providerDone <-chan error,
|
||||
probe healthProbe,
|
||||
) attemptResult {
|
||||
for {
|
||||
select {
|
||||
case providerErr := <-providerDone:
|
||||
return attemptResult{providerErr: providerErr, providerReturned: true}
|
||||
case firedAt := <-observer.expired():
|
||||
expiry, valid := observer.expiryForSignal(firedAt)
|
||||
if !valid || !contextStillActive(execCtx) || !claimStall(expiry) {
|
||||
continue
|
||||
}
|
||||
cancel()
|
||||
return joinStallEvidence(clock, providerDone, probe)
|
||||
case <-execCtx.Done():
|
||||
cancel()
|
||||
return attemptResult{providerErr: <-providerDone, providerReturned: true}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Node) executeNormalizedAttempt(
|
||||
ctx, execCtx context.Context,
|
||||
cancel context.CancelFunc,
|
||||
adapter runtime.Provider,
|
||||
spec runtime.ExecutionSpec,
|
||||
ticket *admissionTicket,
|
||||
h *runHandle,
|
||||
sender protoSender,
|
||||
probe healthProbe,
|
||||
seq healthObservationSequencer,
|
||||
) error {
|
||||
observer := newAttemptObserver(n.watchdogClock, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond)
|
||||
sink := &terminalDeferringSink{
|
||||
inner: &sessionSink{sess: sender, out: n.out, nodeID: n.nodeID, sessionID: normalizeSessionID(spec.SessionID), background: spec.Background},
|
||||
observer: observer,
|
||||
}
|
||||
providerDone := startProviderAttempt(func() error { return adapter.Execute(execCtx, spec, sink) })
|
||||
cleanup := newAttemptCleanup(func() {
|
||||
ticket.release()
|
||||
cancel()
|
||||
n.runs.deregister(spec.RunID)
|
||||
close(h.done)
|
||||
})
|
||||
result := awaitAttempt(execCtx, cancel, n.watchdogClock, observer, sink.claimStall, providerDone, probe)
|
||||
if !result.stalled {
|
||||
return n.finishNormalizedAttempt(ctx, spec, sink, cleanup, result.providerErr)
|
||||
}
|
||||
|
||||
obs := stallObservationFrom(result, time.Duration(spec.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
n.liveness.Observe("normalized", obs)
|
||||
sink.queueClaimedTerminal(stalledRuntimeEvent(spec, obs))
|
||||
n.completeRun(spec, errProviderResponseStalled)
|
||||
if result.providerReturned {
|
||||
cleanup.run()
|
||||
} else {
|
||||
cleanup.afterProviderReturn(providerDone)
|
||||
}
|
||||
if err := sink.Flush(context.Background()); err != nil {
|
||||
n.logger.Warn("session: flush stalled terminal", zap.Error(err))
|
||||
}
|
||||
return errProviderResponseStalled
|
||||
}
|
||||
|
||||
func (n *Node) finishNormalizedAttempt(
|
||||
ctx context.Context,
|
||||
spec runtime.ExecutionSpec,
|
||||
sink *terminalDeferringSink,
|
||||
cleanup *attemptCleanup,
|
||||
execErr error,
|
||||
) error {
|
||||
cleanup.run()
|
||||
if !sink.hasTerminalObserved() {
|
||||
if synthErr := n.synthAndEmitTerminal(ctx, sink, spec, execErr); synthErr != nil && execErr == nil {
|
||||
execErr = synthErr
|
||||
}
|
||||
}
|
||||
n.completeRun(spec, execErr)
|
||||
if flushErr := sink.Flush(context.Background()); flushErr != nil && execErr == nil {
|
||||
return flushErr
|
||||
}
|
||||
return execErr
|
||||
}
|
||||
|
||||
func (n *Node) consumeTunnelCredential(
|
||||
ctx context.Context,
|
||||
req *iop.ProviderTunnelRequest,
|
||||
tr *runtime.ProviderTunnelRequest,
|
||||
) (*credentiallease.Material, error) {
|
||||
if n.credentialConsumer == nil && req.GetCredentialLease() == nil && req.GetCredentialBinding() == nil {
|
||||
return nil, nil
|
||||
}
|
||||
if n.credentialConsumer == nil || req.GetCredentialLease() == nil || req.GetCredentialBinding() == nil {
|
||||
return nil, errors.New("node: credential lease is required")
|
||||
}
|
||||
envelope, err := credentiallease.FromProto(req.GetCredentialLease())
|
||||
if err != nil {
|
||||
return nil, errors.New("node: credential lease rejected")
|
||||
}
|
||||
material, err := n.credentialConsumer.Consume(ctx, envelope, credentiallease.ExpectedFromProto(req.GetCredentialBinding()))
|
||||
if err != nil {
|
||||
return nil, errors.New("node: credential lease rejected")
|
||||
}
|
||||
tr.Credential = &runtime.ProviderCredential{HeaderName: material.HeaderName, Scheme: material.Scheme, Secret: material.Secret}
|
||||
return material, nil
|
||||
}
|
||||
|
||||
func (n *Node) executeTunnelAttempt(
|
||||
execCtx context.Context,
|
||||
cancel context.CancelFunc,
|
||||
adapter runtime.ProviderTunnelAdapter,
|
||||
tr runtime.ProviderTunnelRequest,
|
||||
sink *tunnelSink,
|
||||
ticket *admissionTicket,
|
||||
h *runHandle,
|
||||
material *credentiallease.Material,
|
||||
probe healthProbe,
|
||||
seq healthObservationSequencer,
|
||||
) error {
|
||||
providerDone := startProviderAttempt(func() error { return adapter.TunnelProvider(execCtx, tr, sink) })
|
||||
cleanup := newAttemptCleanup(func() {
|
||||
cancel()
|
||||
if tr.Credential != nil {
|
||||
tr.Credential.Zero()
|
||||
}
|
||||
if material != nil {
|
||||
material.Zero()
|
||||
}
|
||||
ticket.release()
|
||||
n.runs.deregister(tr.RunID)
|
||||
close(h.done)
|
||||
})
|
||||
result := awaitAttempt(execCtx, cancel, n.watchdogClock, sink.observer, sink.claimStall, providerDone, probe)
|
||||
if !result.stalled {
|
||||
cleanup.run()
|
||||
if result.providerErr != nil {
|
||||
n.logger.Warn("provider tunnel error", zap.String("run_id", tr.RunID), zap.String("tunnel_id", tr.TunnelID), zap.Error(result.providerErr))
|
||||
}
|
||||
return result.providerErr
|
||||
}
|
||||
|
||||
if result.providerReturned {
|
||||
cleanup.run()
|
||||
} else {
|
||||
cleanup.afterProviderReturn(providerDone)
|
||||
}
|
||||
obs := stallObservationFrom(result, time.Duration(tr.ResponseStallTimeoutMS)*time.Millisecond, seq)
|
||||
n.liveness.Observe("provider_tunnel", obs)
|
||||
_ = sink.emitClaimedTerminal(context.Background(), stalledTunnelFrame(tr, obs))
|
||||
return errProviderResponseStalled
|
||||
}
|
||||
|
||||
// terminalDeferringSink holds normalized terminal output until Node-local
|
||||
// admission has released its slot. emitMu is the single emission authority for
|
||||
// accepted provider events and a watchdog fence claim.
|
||||
type terminalDeferringSink struct {
|
||||
inner runtime.EventSink
|
||||
observer *attemptObserver
|
||||
|
||||
emitMu sync.Mutex
|
||||
mu sync.Mutex
|
||||
deferring bool
|
||||
terminalObserved bool
|
||||
fenced bool
|
||||
deferred []runtime.RuntimeEvent
|
||||
|
||||
// beforeStallClaim is a deterministic ordering seam for package tests.
|
||||
beforeStallClaim func()
|
||||
afterStallClaim func(bool)
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) Emit(ctx context.Context, event runtime.RuntimeEvent) error {
|
||||
s.emitMu.Lock()
|
||||
defer s.emitMu.Unlock()
|
||||
s.mu.Lock()
|
||||
if s.terminalObserved || s.fenced {
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
if s.observer != nil {
|
||||
s.observer.observe(runtime.ClassifyRuntimeEvent(event))
|
||||
}
|
||||
if runtime.IsTerminalEvent(event.Type) {
|
||||
s.terminalObserved = true
|
||||
}
|
||||
if s.deferring || runtime.IsTerminalEvent(event.Type) {
|
||||
s.deferring = true
|
||||
s.deferred = append(s.deferred, event)
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
s.mu.Unlock()
|
||||
return s.inner.Emit(ctx, event)
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) claimStall(expiry attemptExpiry) bool {
|
||||
if s.beforeStallClaim != nil {
|
||||
s.beforeStallClaim()
|
||||
}
|
||||
s.emitMu.Lock()
|
||||
s.mu.Lock()
|
||||
if s.terminalObserved || s.fenced || (s.observer != nil && !s.observer.claimFence(expiry)) {
|
||||
s.mu.Unlock()
|
||||
s.emitMu.Unlock()
|
||||
if s.afterStallClaim != nil {
|
||||
s.afterStallClaim(false)
|
||||
}
|
||||
return false
|
||||
}
|
||||
s.fenced, s.terminalObserved, s.deferring = true, true, true
|
||||
s.mu.Unlock()
|
||||
s.emitMu.Unlock()
|
||||
if s.afterStallClaim != nil {
|
||||
s.afterStallClaim(true)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) queueClaimedTerminal(event runtime.RuntimeEvent) {
|
||||
s.emitMu.Lock()
|
||||
defer s.emitMu.Unlock()
|
||||
s.mu.Lock()
|
||||
s.deferred = append(s.deferred, event)
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) Flush(ctx context.Context) error {
|
||||
s.emitMu.Lock()
|
||||
defer s.emitMu.Unlock()
|
||||
s.mu.Lock()
|
||||
events := append([]runtime.RuntimeEvent(nil), s.deferred...)
|
||||
s.deferred = nil
|
||||
s.deferring = false
|
||||
s.mu.Unlock()
|
||||
for _, event := range events {
|
||||
if err := s.inner.Emit(ctx, event); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) hasTerminalObserved() bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.terminalObserved
|
||||
}
|
||||
|
||||
// tunnelSink holds its emission lock through Send. A watchdog fence therefore
|
||||
// cannot overtake a frame that was accepted before the fence claim.
|
||||
type tunnelSink struct {
|
||||
sess protoSender
|
||||
nodeID string
|
||||
nodeAlias string
|
||||
observer *attemptObserver
|
||||
mu sync.Mutex
|
||||
terminal bool
|
||||
fenced bool
|
||||
|
||||
// beforeStallClaim is a deterministic ordering seam for package tests.
|
||||
beforeStallClaim func()
|
||||
afterStallClaim func(bool)
|
||||
}
|
||||
|
||||
func (s *tunnelSink) EmitTunnelFrame(ctx context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal || s.fenced {
|
||||
return nil
|
||||
}
|
||||
disposition := runtime.ClassifyProviderTunnelFrame(frame)
|
||||
if s.observer != nil {
|
||||
s.observer.observe(disposition)
|
||||
}
|
||||
if disposition == runtime.DispositionTerminal {
|
||||
s.terminal = true
|
||||
}
|
||||
return s.emit(ctx, frame)
|
||||
}
|
||||
|
||||
func (s *tunnelSink) claimStall(expiry attemptExpiry) bool {
|
||||
if s.beforeStallClaim != nil {
|
||||
s.beforeStallClaim()
|
||||
}
|
||||
s.mu.Lock()
|
||||
if s.terminal || s.fenced || (s.observer != nil && !s.observer.claimFence(expiry)) {
|
||||
s.mu.Unlock()
|
||||
if s.afterStallClaim != nil {
|
||||
s.afterStallClaim(false)
|
||||
}
|
||||
return false
|
||||
}
|
||||
s.fenced, s.terminal = true, true
|
||||
s.mu.Unlock()
|
||||
if s.afterStallClaim != nil {
|
||||
s.afterStallClaim(true)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *tunnelSink) emitClaimedTerminal(ctx context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.emit(ctx, frame)
|
||||
}
|
||||
|
||||
func (s *tunnelSink) emit(ctx context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
tf := tunnelFrameToProto(frame, s.nodeID, s.nodeAlias)
|
||||
if s.sess != nil {
|
||||
return s.sess.Send(tf)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func tunnelFrameToProto(frame runtime.ProviderTunnelFrame, nodeID, nodeAlias string) *iop.ProviderTunnelFrame {
|
||||
var usage *iop.Usage
|
||||
if frame.Usage != nil {
|
||||
usage = &iop.Usage{InputTokens: int32(frame.Usage.InputTokens), OutputTokens: int32(frame.Usage.OutputTokens), ReasoningTokens: int32(frame.Usage.ReasoningTokens), CachedInputTokens: int32(frame.Usage.CachedInputTokens)}
|
||||
}
|
||||
protoKind := iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_UNSPECIFIED
|
||||
switch frame.Kind {
|
||||
case runtime.ProviderTunnelFrameKindResponseStart:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START
|
||||
case runtime.ProviderTunnelFrameKindBody:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY
|
||||
case runtime.ProviderTunnelFrameKindEnd:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END
|
||||
case runtime.ProviderTunnelFrameKindError:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR
|
||||
case runtime.ProviderTunnelFrameKindUsage:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_USAGE
|
||||
}
|
||||
return &iop.ProviderTunnelFrame{
|
||||
RunId: frame.RunID, TunnelId: frame.TunnelID, Sequence: frame.Sequence, Kind: protoKind,
|
||||
StatusCode: int32(frame.StatusCode), Headers: frame.Headers, Body: frame.Body, End: frame.End,
|
||||
Error: frame.Error, Failure: executionFailureToProto(frame.Failure), Usage: usage, Metadata: cloneLivenessMetadata(frame.Metadata), Timestamp: frame.Timestamp.UnixNano(),
|
||||
NodeId: nodeID, NodeAlias: nodeAlias,
|
||||
}
|
||||
}
|
||||
|
|
@ -1,382 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"iop/packages/go/credentiallease"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// TestRunWatchdogLifecycle covers confirmed fence with exact grace, unconfirmed
|
||||
// ownership retention until provider return, caller cancel winning the timer
|
||||
// race, and an already-expired deadline bypassing the watchdog.
|
||||
func TestRunWatchdogLifecycle(t *testing.T) {
|
||||
t.Run("confirmed fence and exact grace", testRunWatchdogConfirmed)
|
||||
t.Run("unconfirmed retains ownership until provider return", testRunWatchdogUnconfirmed)
|
||||
t.Run("caller cancel wins timer race", testRunWatchdogCancelPrecedence)
|
||||
t.Run("hard deadline retains boundary", testRunWatchdogDeadlinePrecedence)
|
||||
}
|
||||
|
||||
func testRunWatchdogConfirmed(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-confirmed")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-confirmed", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
requireTimerDurations(t, stallTimer, time.Second)
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
event := waitRunEvent(t, pipe.events)
|
||||
if event.GetType() != string(runtime.EventTypeError) || event.GetMetadata()["attempt_fence"] != "confirmed" || event.GetMetadata()["idle_duration_ms"] != "1000" {
|
||||
t.Fatalf("stall event = %+v", event)
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 0 || n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("confirmed provider return retained local ownership")
|
||||
}
|
||||
_ = call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-confirmed", Type: runtime.EventTypeDelta, Delta: "late"})
|
||||
select {
|
||||
case extra := <-pipe.events:
|
||||
t.Fatalf("late or duplicate event = %+v", extra)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func testRunWatchdogUnconfirmed(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-unconfirmed")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-unconfirmed", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 2000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
_ = call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-unconfirmed", Type: runtime.EventTypeDelta, Delta: "progress"})
|
||||
if progress := waitRunEvent(t, pipe.events); progress.GetType() != string(runtime.EventTypeDelta) {
|
||||
t.Fatalf("progress event = %+v", progress)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, 2*time.Second, 2*time.Second)
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
grace.fire()
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
if event := waitRunEvent(t, pipe.events); event.GetMetadata()["attempt_fence"] != "unconfirmed" {
|
||||
t.Fatalf("stall event = %+v", event)
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 1 || !n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("unconfirmed attempt released ownership before provider return")
|
||||
}
|
||||
_ = call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-unconfirmed", Type: runtime.EventTypeDelta, Delta: "late"})
|
||||
adapter.runReturn <- nil
|
||||
waitForOwnershipRelease(t, n, adapter.Name(), "provider return did not release retained ownership")
|
||||
select {
|
||||
case extra := <-pipe.events:
|
||||
t.Fatalf("late or duplicate event = %+v", extra)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func testRunWatchdogCancelPrecedence(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-cancel")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(ctx, pipe.sess, &iop.RunRequest{RunId: "run-cancel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
timer := clock.waitTimer(t, 0)
|
||||
cancel()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
timer.fire()
|
||||
adapter.runReturn <- runtime.ErrRunCancelled
|
||||
if err := <-done; err != runtime.ErrRunCancelled {
|
||||
t.Fatalf("cancel result = %v", err)
|
||||
}
|
||||
event := waitRunEvent(t, pipe.events)
|
||||
if event.GetType() != string(runtime.EventTypeCancelled) || event.GetMetadata()["failure_code"] == string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("cancel event relabeled as stall: %+v", event)
|
||||
}
|
||||
if clock.count() != 1 {
|
||||
t.Fatalf("cancel created close-grace timer: %d timers", clock.count())
|
||||
}
|
||||
}
|
||||
|
||||
func testRunWatchdogDeadlinePrecedence(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-deadline")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second))
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(ctx, pipe.sess, &iop.RunRequest{RunId: "run-deadline", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
waitContextCanceled(t, call.ctx)
|
||||
adapter.runReturn <- context.DeadlineExceeded
|
||||
if err := <-done; err != context.DeadlineExceeded {
|
||||
t.Fatalf("deadline result = %v", err)
|
||||
}
|
||||
stallTimer.fire()
|
||||
event := waitRunEvent(t, pipe.events)
|
||||
if event.GetType() != string(runtime.EventTypeError) || event.GetError() != context.DeadlineExceeded.Error() || event.GetMetadata()["failure_code"] == string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("deadline event relabeled as stall: %+v", event)
|
||||
}
|
||||
if clock.count() != 1 {
|
||||
t.Fatalf("deadline created close-grace timer: %d timers", clock.count())
|
||||
}
|
||||
}
|
||||
|
||||
// TestTunnelWatchdogLifecycle covers unconfirmed fence dropping late frames,
|
||||
// confirmed fence, provider terminal stopping the clock, and credential
|
||||
// ownership following provider return.
|
||||
func TestTunnelWatchdogLifecycle(t *testing.T) {
|
||||
t.Run("unconfirmed fence drops late frames", testTunnelWatchdogUnconfirmed)
|
||||
t.Run("confirmed fence", testTunnelWatchdogConfirmed)
|
||||
t.Run("provider terminal stops clock", testTunnelProviderTerminalStopsClock)
|
||||
t.Run("credential ownership follows provider return", testTunnelCredentialOwnership)
|
||||
}
|
||||
|
||||
func testTunnelWatchdogUnconfirmed(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-unconfirmed")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-run", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1500})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "tunnel-run", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindBody, Body: []byte("progress")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY {
|
||||
t.Fatalf("progress frame = %+v", frame)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, 1500*time.Millisecond, 1500*time.Millisecond)
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
grace.fire()
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "unconfirmed" {
|
||||
t.Fatalf("stall terminal = %+v", terminal)
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 1 || !n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("unconfirmed tunnel released ownership before provider return")
|
||||
}
|
||||
_ = call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "tunnel-run", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindUsage})
|
||||
adapter.tunnelReturn <- nil
|
||||
waitForOwnershipRelease(t, n, adapter.Name(), "tunnel provider return did not release ownership")
|
||||
select {
|
||||
case extra := <-pipe.frames:
|
||||
t.Fatalf("late or duplicate frame = %+v", extra)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func testTunnelWatchdogConfirmed(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-confirmed")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-confirmed-run", TunnelId: "tunnel-confirmed", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
if terminal := waitTunnelFrame(t, pipe.frames); terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("stall terminal = %+v", terminal)
|
||||
}
|
||||
if activeAdapterAttempts(n, adapter.Name()) != 0 || n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("confirmed tunnel retained ownership")
|
||||
}
|
||||
}
|
||||
|
||||
func testTunnelProviderTerminalStopsClock(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-terminal")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "terminal-run", TunnelId: "terminal-tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
timer := clock.waitTimer(t, 0)
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "terminal-run", TunnelID: "terminal-tunnel", Kind: runtime.ProviderTunnelFrameKindEnd, End: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = waitTunnelFrame(t, pipe.frames)
|
||||
_, stopped := timer.snapshot()
|
||||
if !stopped {
|
||||
t.Fatal("provider terminal did not stop tunnel watchdog")
|
||||
}
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func testTunnelCredentialOwnership(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-credential")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
ticket, err := n.admissionFor(adapter.Name(), runtime.Capabilities{MaxConcurrency: 1}).acquire()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := []byte("provider-secret")
|
||||
credential := &runtime.ProviderCredential{HeaderName: "Authorization", Scheme: "Bearer", Secret: secret}
|
||||
material := &credentiallease.Material{HeaderName: credential.HeaderName, Scheme: credential.Scheme, Secret: secret}
|
||||
tr := runtime.ProviderTunnelRequest{RunID: "credential-run", TunnelID: "credential-tunnel", Adapter: adapter.Name(), Target: "target", Credential: credential, ResponseStallTimeoutMS: 1000}
|
||||
execCtx, cancel := context.WithCancel(context.Background())
|
||||
h := &runHandle{runID: tr.RunID, adapter: tr.Adapter, target: tr.Target, cancel: cancel, done: make(chan struct{})}
|
||||
n.runs.register(h)
|
||||
sink := &tunnelSink{sess: noopSender{}, observer: newAttemptObserver(clock, time.Second)}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.executeTunnelAttempt(execCtx, cancel, adapter, tr, sink, ticket, h, material, nil, nil)
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
clock.waitTimer(t, 1).fire()
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
if string(credential.Secret) != "provider-secret" || string(material.Secret) != "provider-secret" || activeAdapterAttempts(n, adapter.Name()) != 1 || !n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("unconfirmed tunnel did not retain credential and local ownership")
|
||||
}
|
||||
adapter.tunnelReturn <- nil
|
||||
select {
|
||||
case <-h.done:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("credential cleanup did not follow provider return")
|
||||
}
|
||||
if credential.Secret != nil || material.Secret != nil || activeAdapterAttempts(n, adapter.Name()) != 0 || n.runs.hasAnyActiveRuns() {
|
||||
t.Fatal("provider return did not zero credentials and release local ownership")
|
||||
}
|
||||
}
|
||||
|
||||
type tunnelTerminalOwnership struct {
|
||||
admissionReleased bool
|
||||
runDeregistered bool
|
||||
credentialsZeroed bool
|
||||
handleClosed bool
|
||||
}
|
||||
|
||||
type tunnelTerminalInspector struct {
|
||||
ownership func() tunnelTerminalOwnership
|
||||
seen chan tunnelTerminalOwnership
|
||||
}
|
||||
|
||||
func (s *tunnelTerminalInspector) Send(message proto.Message) error {
|
||||
frame, ok := message.(*iop.ProviderTunnelFrame)
|
||||
if ok && frame.GetKind() == iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR {
|
||||
s.seen <- s.ownership()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// TestTunnelConfirmedFenceClosesOwnershipBeforeTerminal proves the confirmed
|
||||
// terminal is visible to the edge only after admission, run deregistration,
|
||||
// credential zeroing, and handle closure have all completed.
|
||||
func TestTunnelConfirmedFenceClosesOwnershipBeforeTerminal(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-confirmed-ownership")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
ticket, err := n.admissionFor(adapter.Name(), runtime.Capabilities{MaxConcurrency: 1}).acquire()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
credential := &runtime.ProviderCredential{HeaderName: "Authorization", Scheme: "Bearer", Secret: []byte("provider-secret")}
|
||||
material := &credentiallease.Material{HeaderName: credential.HeaderName, Scheme: credential.Scheme, Secret: []byte("provider-secret")}
|
||||
tr := runtime.ProviderTunnelRequest{RunID: "tunnel-confirmed-ownership", TunnelID: "tunnel", Adapter: adapter.Name(), Target: "target", Credential: credential, ResponseStallTimeoutMS: 1000}
|
||||
execCtx, cancel := context.WithCancel(context.Background())
|
||||
h := &runHandle{runID: tr.RunID, adapter: tr.Adapter, target: tr.Target, cancel: cancel, done: make(chan struct{})}
|
||||
n.runs.register(h)
|
||||
inspector := &tunnelTerminalInspector{seen: make(chan tunnelTerminalOwnership, 1)}
|
||||
inspector.ownership = func() tunnelTerminalOwnership {
|
||||
ownership := tunnelTerminalOwnership{
|
||||
admissionReleased: activeAdapterAttempts(n, adapter.Name()) == 0,
|
||||
runDeregistered: !n.runs.hasAnyActiveRuns(),
|
||||
credentialsZeroed: credential.Secret == nil && material.Secret == nil,
|
||||
}
|
||||
select {
|
||||
case <-h.done:
|
||||
ownership.handleClosed = true
|
||||
default:
|
||||
}
|
||||
return ownership
|
||||
}
|
||||
sink := &tunnelSink{sess: inspector, observer: newAttemptObserver(clock, time.Second)}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.executeTunnelAttempt(execCtx, cancel, adapter, tr, sink, ticket, h, material, nil, nil)
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
clock.waitTimer(t, 0).fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
ownership := <-inspector.seen
|
||||
if !ownership.admissionReleased || !ownership.runDeregistered || !ownership.credentialsZeroed || !ownership.handleClosed {
|
||||
t.Fatalf("confirmed terminal was visible before local ownership closed: %+v", ownership)
|
||||
}
|
||||
}
|
||||
|
||||
func waitForOwnershipRelease(t *testing.T, n *Node, adapter, failure string) {
|
||||
t.Helper()
|
||||
deadline := time.After(2 * time.Second)
|
||||
for activeAdapterAttempts(n, adapter) != 0 || n.runs.hasAnyActiveRuns() {
|
||||
select {
|
||||
case <-deadline:
|
||||
t.Fatal(failure)
|
||||
default:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,863 +0,0 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"iop/apps/node/internal/store"
|
||||
"iop/apps/node/internal/transport"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
type manualAttemptTimer struct {
|
||||
mu sync.Mutex
|
||||
ch chan time.Time
|
||||
now func() time.Time
|
||||
advanceTo func(time.Time)
|
||||
durations []time.Duration
|
||||
scheduled time.Time
|
||||
stopped bool
|
||||
fired bool
|
||||
beforeReset func()
|
||||
}
|
||||
|
||||
func newManualAttemptTimer(d time.Duration, now func() time.Time, advanceTo func(time.Time)) *manualAttemptTimer {
|
||||
scheduled := now().Add(d)
|
||||
return &manualAttemptTimer{ch: make(chan time.Time, 1), now: now, advanceTo: advanceTo, durations: []time.Duration{d}, scheduled: scheduled}
|
||||
}
|
||||
func (t *manualAttemptTimer) C() <-chan time.Time { return t.ch }
|
||||
func (t *manualAttemptTimer) Stop() bool {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
wasActive := !t.stopped && !t.fired
|
||||
t.stopped = true
|
||||
return wasActive
|
||||
}
|
||||
func (t *manualAttemptTimer) Reset(d time.Duration) bool {
|
||||
t.mu.Lock()
|
||||
beforeReset := t.beforeReset
|
||||
t.mu.Unlock()
|
||||
if beforeReset != nil {
|
||||
beforeReset()
|
||||
}
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
wasStopped := t.stopped
|
||||
t.stopped = false
|
||||
t.fired = false
|
||||
t.durations = append(t.durations, d)
|
||||
t.scheduled = t.now().Add(d)
|
||||
return wasStopped
|
||||
}
|
||||
func (t *manualAttemptTimer) fire() {
|
||||
t.mu.Lock()
|
||||
stopped, fired, scheduled := t.stopped, t.fired, t.scheduled
|
||||
if !stopped && !fired {
|
||||
t.fired = true
|
||||
}
|
||||
t.mu.Unlock()
|
||||
if !stopped && !fired {
|
||||
t.advanceTo(scheduled)
|
||||
t.ch <- scheduled
|
||||
}
|
||||
}
|
||||
func (t *manualAttemptTimer) fireStaleArmDuringReset() {
|
||||
t.mu.Lock()
|
||||
scheduled := t.scheduled
|
||||
t.mu.Unlock()
|
||||
t.advanceTo(scheduled)
|
||||
t.ch <- scheduled
|
||||
}
|
||||
func (t *manualAttemptTimer) snapshot() ([]time.Duration, bool) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return append([]time.Duration(nil), t.durations...), t.stopped
|
||||
}
|
||||
|
||||
type manualAttemptClock struct {
|
||||
mu sync.Mutex
|
||||
timers []*manualAttemptTimer
|
||||
created chan struct{}
|
||||
now time.Time
|
||||
beforeTimerReturn func(*manualAttemptTimer)
|
||||
}
|
||||
|
||||
func newManualAttemptClock() *manualAttemptClock {
|
||||
return &manualAttemptClock{created: make(chan struct{}, 16), now: time.Unix(0, 0)}
|
||||
}
|
||||
|
||||
// Now returns a strictly increasing timestamp. Timers retain their scheduled
|
||||
// deadline separately, so a delayed manual fire cannot be mistaken for the
|
||||
// clock's later read time.
|
||||
func (c *manualAttemptClock) Now() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.now = c.now.Add(time.Millisecond)
|
||||
return c.now
|
||||
}
|
||||
func (c *manualAttemptClock) current() time.Time {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return c.now
|
||||
}
|
||||
func (c *manualAttemptClock) advanceTo(at time.Time) {
|
||||
c.mu.Lock()
|
||||
if c.now.Before(at) {
|
||||
c.now = at
|
||||
}
|
||||
c.mu.Unlock()
|
||||
}
|
||||
func (c *manualAttemptClock) NewTimer(d time.Duration) attemptTimer {
|
||||
timer := newManualAttemptTimer(d, c.current, c.advanceTo)
|
||||
c.mu.Lock()
|
||||
c.timers = append(c.timers, timer)
|
||||
beforeTimerReturn := c.beforeTimerReturn
|
||||
c.mu.Unlock()
|
||||
c.created <- struct{}{}
|
||||
if beforeTimerReturn != nil {
|
||||
beforeTimerReturn(timer)
|
||||
}
|
||||
return timer
|
||||
}
|
||||
func (c *manualAttemptClock) waitTimer(t *testing.T, index int) *manualAttemptTimer {
|
||||
t.Helper()
|
||||
for {
|
||||
c.mu.Lock()
|
||||
if len(c.timers) > index {
|
||||
timer := c.timers[index]
|
||||
c.mu.Unlock()
|
||||
return timer
|
||||
}
|
||||
c.mu.Unlock()
|
||||
select {
|
||||
case <-c.created:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatalf("timer %d was not created", index)
|
||||
}
|
||||
}
|
||||
}
|
||||
func (c *manualAttemptClock) count() int {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
return len(c.timers)
|
||||
}
|
||||
|
||||
type controlledRunCall struct {
|
||||
ctx context.Context
|
||||
spec runtime.ExecutionSpec
|
||||
sink runtime.EventSink
|
||||
}
|
||||
|
||||
type controlledTunnelCall struct {
|
||||
ctx context.Context
|
||||
req runtime.ProviderTunnelRequest
|
||||
sink runtime.ProviderTunnelSink
|
||||
}
|
||||
|
||||
type controlledWatchdogAdapter struct {
|
||||
name string
|
||||
runCalls chan controlledRunCall
|
||||
tunnelCalls chan controlledTunnelCall
|
||||
runReturn chan error
|
||||
tunnelReturn chan error
|
||||
maxConcurrent int
|
||||
}
|
||||
|
||||
func newControlledWatchdogAdapter(name string) *controlledWatchdogAdapter {
|
||||
return &controlledWatchdogAdapter{
|
||||
name: name, runCalls: make(chan controlledRunCall, 1), tunnelCalls: make(chan controlledTunnelCall, 1),
|
||||
runReturn: make(chan error, 1), tunnelReturn: make(chan error, 1), maxConcurrent: 1,
|
||||
}
|
||||
}
|
||||
func (a *controlledWatchdogAdapter) Name() string { return a.name }
|
||||
func (a *controlledWatchdogAdapter) Capabilities(context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: a.name, Targets: []string{"target"}, MaxConcurrency: a.maxConcurrent}, nil
|
||||
}
|
||||
func (a *controlledWatchdogAdapter) Execute(ctx context.Context, spec runtime.ExecutionSpec, sink runtime.EventSink) error {
|
||||
a.runCalls <- controlledRunCall{ctx: ctx, spec: spec, sink: sink}
|
||||
return <-a.runReturn
|
||||
}
|
||||
func (a *controlledWatchdogAdapter) TunnelProvider(ctx context.Context, req runtime.ProviderTunnelRequest, sink runtime.ProviderTunnelSink) error {
|
||||
a.tunnelCalls <- controlledTunnelCall{ctx: ctx, req: req, sink: sink}
|
||||
return <-a.tunnelReturn
|
||||
}
|
||||
|
||||
// probeCall records one invocation of the injected health probe so tests can
|
||||
// assert the probe received an independent, still-live context after the
|
||||
// stalled request was canceled.
|
||||
type probeCall struct {
|
||||
ctx context.Context
|
||||
target string
|
||||
}
|
||||
|
||||
type probeReply struct {
|
||||
result runtime.ProviderProbeResult
|
||||
err error
|
||||
}
|
||||
|
||||
// probingWatchdogAdapter is a controlledWatchdogAdapter that also implements
|
||||
// runtime.ProviderProber. ProbeProvider blocks on a channel so tests drive the
|
||||
// independent bounded health probe deterministically and observe the context it
|
||||
// received.
|
||||
type probingWatchdogAdapter struct {
|
||||
*controlledWatchdogAdapter
|
||||
probeCalls chan probeCall
|
||||
probeReturn chan probeReply
|
||||
}
|
||||
|
||||
func newProbingWatchdogAdapter(name string) *probingWatchdogAdapter {
|
||||
return &probingWatchdogAdapter{
|
||||
controlledWatchdogAdapter: newControlledWatchdogAdapter(name),
|
||||
probeCalls: make(chan probeCall, 1),
|
||||
probeReturn: make(chan probeReply, 1),
|
||||
}
|
||||
}
|
||||
|
||||
func (a *probingWatchdogAdapter) ProbeProvider(ctx context.Context, target string) (runtime.ProviderProbeResult, error) {
|
||||
a.probeCalls <- probeCall{ctx: ctx, target: target}
|
||||
reply := <-a.probeReturn
|
||||
return reply.result, reply.err
|
||||
}
|
||||
|
||||
type watchdogRouter struct{ adapter runtime.ProviderTunnelAdapter }
|
||||
|
||||
func (r *watchdogRouter) Resolve(_ context.Context, req runtime.RunRequest) (runtime.ExecutionSpec, error) {
|
||||
return runtime.ExecutionSpec{
|
||||
RunID: req.RunID, Adapter: r.adapter.Name(), Target: req.Target, SessionID: req.SessionID,
|
||||
Background: req.Background, Input: req.Input, TimeoutSec: req.TimeoutSec, Metadata: req.Metadata,
|
||||
ResponseStallTimeoutMS: req.ResponseStallTimeoutMS,
|
||||
}, nil
|
||||
}
|
||||
func (r *watchdogRouter) ResolveAdapter(ctx context.Context, req runtime.RunRequest) (runtime.ExecutionSpec, runtime.Provider, error) {
|
||||
spec, err := r.Resolve(ctx, req)
|
||||
return spec, r.adapter, err
|
||||
}
|
||||
func (r *watchdogRouter) LookupAdapter(name string) (runtime.Provider, error) {
|
||||
if name != r.adapter.Name() {
|
||||
return nil, fmt.Errorf("adapter %q not found", name)
|
||||
}
|
||||
return r.adapter, nil
|
||||
}
|
||||
func (r *watchdogRouter) GetAdapter(name string) (runtime.Provider, bool) {
|
||||
if name == r.adapter.Name() {
|
||||
return r.adapter, true
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
func newWatchdogNode(t *testing.T, adapter runtime.ProviderTunnelAdapter, clock *manualAttemptClock) *Node {
|
||||
t.Helper()
|
||||
st, err := store.New(":memory:", zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
n := New("node-watchdog", &watchdogRouter{adapter: adapter}, st, 0, io.Discard, zap.NewNop(), nil)
|
||||
n.watchdogClock = clock
|
||||
return n
|
||||
}
|
||||
|
||||
type watchdogPipe struct {
|
||||
edge *toki.TcpClient
|
||||
sess *transport.Session
|
||||
events chan *iop.RunEvent
|
||||
frames chan *iop.ProviderTunnelFrame
|
||||
}
|
||||
|
||||
func newWatchdogPipe(t *testing.T) *watchdogPipe {
|
||||
t.Helper()
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
edgeParsers := toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.RunEvent{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.RunEvent{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
toki.TypeNameOf(&iop.ProviderTunnelFrame{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.ProviderTunnelFrame{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
}
|
||||
edge := toki.NewTcpClient(edgeConn, 0, 0, edgeParsers)
|
||||
nodeClient := toki.NewTcpClient(nodeConn, 0, 0, toki.ParserMap{})
|
||||
pipe := &watchdogPipe{
|
||||
edge: edge, sess: transport.ExportNewSession(nodeClient, zap.NewNop(), "node-watchdog", "watchdog"),
|
||||
events: make(chan *iop.RunEvent, 8), frames: make(chan *iop.ProviderTunnelFrame, 8),
|
||||
}
|
||||
toki.AddListenerTyped[*iop.RunEvent](&edge.Communicator, func(event *iop.RunEvent) {
|
||||
pipe.events <- proto.Clone(event).(*iop.RunEvent)
|
||||
})
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelFrame](&edge.Communicator, func(frame *iop.ProviderTunnelFrame) {
|
||||
pipe.frames <- proto.Clone(frame).(*iop.ProviderTunnelFrame)
|
||||
})
|
||||
t.Cleanup(func() { _ = edge.Close(); _ = nodeClient.Close() })
|
||||
return pipe
|
||||
}
|
||||
|
||||
func waitContextCanceled(t *testing.T, ctx context.Context) {
|
||||
t.Helper()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("provider context was not canceled")
|
||||
}
|
||||
}
|
||||
|
||||
func waitRunEvent(t *testing.T, events <-chan *iop.RunEvent) *iop.RunEvent {
|
||||
t.Helper()
|
||||
select {
|
||||
case event := <-events:
|
||||
return event
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("run event was not emitted")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func waitTunnelFrame(t *testing.T, frames <-chan *iop.ProviderTunnelFrame) *iop.ProviderTunnelFrame {
|
||||
t.Helper()
|
||||
select {
|
||||
case frame := <-frames:
|
||||
return frame
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("tunnel frame was not emitted")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func requireTimerDurations(t *testing.T, timer *manualAttemptTimer, want ...time.Duration) {
|
||||
t.Helper()
|
||||
got, _ := timer.snapshot()
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("timer durations = %v, want %v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("timer durations = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func activeAdapterAttempts(n *Node, adapter string) int {
|
||||
n.adapterGatesMu.Lock()
|
||||
gate := n.adapterGates[adapter]
|
||||
n.adapterGatesMu.Unlock()
|
||||
if gate == nil {
|
||||
return 0
|
||||
}
|
||||
return gate.activeCount()
|
||||
}
|
||||
|
||||
func TestAttemptObserverProgressResetsAndFenceIsMonotonic(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
observer := newAttemptObserver(clock, time.Second)
|
||||
timer := clock.waitTimer(t, 0)
|
||||
observer.observe(runtime.DispositionNone)
|
||||
requireTimerDurations(t, timer, time.Second)
|
||||
observer.observe(runtime.DispositionProgress)
|
||||
requireTimerDurations(t, timer, time.Second, time.Second)
|
||||
timer.fire()
|
||||
expiry, valid := observer.expiryForSignal(<-observer.expired())
|
||||
if !valid || !observer.claimFence(expiry) || observer.claimFence(expiry) {
|
||||
t.Fatal("fence claim was not monotonic")
|
||||
}
|
||||
observer.observe(runtime.DispositionProgress)
|
||||
requireTimerDurations(t, timer, time.Second, time.Second)
|
||||
}
|
||||
|
||||
func TestAttemptObserverCurrentArmSignalSurvivesImmediateFire(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
clock.beforeTimerReturn = func(timer *manualAttemptTimer) { timer.fire() }
|
||||
observer := newAttemptObserver(clock, time.Nanosecond)
|
||||
|
||||
expiry, valid := observer.expiryForSignal(<-observer.expired())
|
||||
if !valid {
|
||||
t.Fatal("current timer signal was rejected because expiry bookkeeping followed the fire")
|
||||
}
|
||||
if !observer.claimFence(expiry) {
|
||||
t.Fatal("current timer signal did not claim the fence")
|
||||
}
|
||||
}
|
||||
|
||||
type recordingProtoSender struct {
|
||||
mu sync.Mutex
|
||||
messages []proto.Message
|
||||
entered chan struct{}
|
||||
release chan struct{}
|
||||
once sync.Once
|
||||
}
|
||||
|
||||
func (s *recordingProtoSender) Send(message proto.Message) error {
|
||||
if s.entered != nil {
|
||||
s.once.Do(func() {
|
||||
close(s.entered)
|
||||
<-s.release
|
||||
})
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.messages = append(s.messages, proto.Clone(message))
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
func (s *recordingProtoSender) snapshot() []proto.Message {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]proto.Message(nil), s.messages...)
|
||||
}
|
||||
|
||||
func TestTunnelSinkStallClaimSerializesAcceptedFrame(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
sender := &recordingProtoSender{entered: make(chan struct{}), release: make(chan struct{})}
|
||||
sink := &tunnelSink{sess: sender, observer: newAttemptObserver(clock, time.Second)}
|
||||
bodyDone := make(chan error, 1)
|
||||
go func() {
|
||||
bodyDone <- sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{Kind: runtime.ProviderTunnelFrameKindBody, Body: []byte("accepted")})
|
||||
}()
|
||||
<-sender.entered
|
||||
if sink.mu.TryLock() {
|
||||
sink.mu.Unlock()
|
||||
t.Fatal("tunnel emission lock was released before accepted frame Send completed")
|
||||
}
|
||||
close(sender.release)
|
||||
if err := <-bodyDone; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
timer := clock.waitTimer(t, 0)
|
||||
timer.fire()
|
||||
expiry, valid := sink.observer.expiryForSignal(<-sink.observer.expired())
|
||||
if !valid || !sink.claimStall(expiry) {
|
||||
t.Fatal("stall claim failed after accepted frame completed")
|
||||
}
|
||||
if err := sink.emitClaimedTerminal(context.Background(), stalledTunnelFrame(runtime.ProviderTunnelRequest{RunID: "run", TunnelID: "tunnel"}, stallObservation{fence: "confirmed", idle: time.Second})); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{Kind: runtime.ProviderTunnelFrameKindUsage}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
messages := sender.snapshot()
|
||||
if len(messages) != 2 {
|
||||
t.Fatalf("sent frames = %d, want body then terminal", len(messages))
|
||||
}
|
||||
if messages[0].(*iop.ProviderTunnelFrame).GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY || messages[1].(*iop.ProviderTunnelFrame).GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR {
|
||||
t.Fatalf("frame order = %v, %v", messages[0], messages[1])
|
||||
}
|
||||
}
|
||||
|
||||
// TestTunnelWatchdogFinishFrameWithoutEndStallsOnce reproduces a provider
|
||||
// stream that has emitted a logical OpenAI finish frame but has not closed its
|
||||
// HTTP response. The finish frame is body progress, not a transport terminal:
|
||||
// the watchdog must expire once and fence every later usage/END frame.
|
||||
func TestTunnelWatchdogFinishFrameWithoutEndStallsOnce(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-finish-without-end")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{
|
||||
RunId: "tunnel-finish-without-end", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000,
|
||||
})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{
|
||||
RunID: "tunnel-finish-without-end", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindResponseStart,
|
||||
StatusCode: 200, Headers: map[string]string{"Content-Type": "text/event-stream"},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
logicalFinish := []byte("data: {\"choices\":[{\"delta\":{\"content\":\"finished body\"},\"finish_reason\":\"stop\"}]}\n\n")
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{
|
||||
RunID: "tunnel-finish-without-end", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindBody, Body: logicalFinish,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START {
|
||||
t.Fatalf("response start = %+v", frame)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY || string(frame.GetBody()) != string(logicalFinish) {
|
||||
t.Fatalf("logical finish body = %+v", frame)
|
||||
}
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second, time.Second)
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
terminal := waitTunnelFrame(t, pipe.frames)
|
||||
if terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR ||
|
||||
terminal.GetFailure().GetCode() != string(runtime.FailureCodeResponseStalled) ||
|
||||
terminal.GetMetadata()["failure_code"] != string(runtime.FailureCodeResponseStalled) {
|
||||
t.Fatalf("stall terminal = %+v", terminal)
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
|
||||
// These are the exact late frames an adapter can race after cancellation.
|
||||
// Both must be accepted as no-ops by the fenced sink.
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{
|
||||
RunID: "tunnel-finish-without-end", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindUsage,
|
||||
Usage: &runtime.UsageStats{InputTokens: 1, OutputTokens: 1},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{
|
||||
RunID: "tunnel-finish-without-end", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindEnd, End: true,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case frame := <-pipe.frames:
|
||||
t.Fatalf("late frame escaped the terminal fence: %+v", frame)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunWatchdogStaleExpiryYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-stale-expiry")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-stale-expiry", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
sink := call.sink.(*terminalDeferringSink)
|
||||
claimStarted := make(chan struct{})
|
||||
releaseClaim := make(chan struct{})
|
||||
claimResult := make(chan bool, 1)
|
||||
var firstClaim sync.Once
|
||||
sink.beforeStallClaim = func() {
|
||||
firstClaim.Do(func() {
|
||||
close(claimStarted)
|
||||
<-releaseClaim
|
||||
})
|
||||
}
|
||||
sink.afterStallClaim = func(claimed bool) { claimResult <- claimed }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
stallTimer.fire()
|
||||
<-claimStarted // The old timer was consumed before provider progress arrives.
|
||||
if err := call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-stale-expiry", Type: runtime.EventTypeDelta, Delta: "progress"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if event := waitRunEvent(t, pipe.events); event.GetType() != string(runtime.EventTypeDelta) {
|
||||
t.Fatalf("progress event = %+v", event)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
close(releaseClaim)
|
||||
if claimed := <-claimResult; claimed {
|
||||
t.Fatal("stale normalized expiry fenced after progress reset the watchdog")
|
||||
}
|
||||
select {
|
||||
case event := <-pipe.events:
|
||||
t.Fatalf("stale normalized expiry emitted terminal: %+v", event)
|
||||
default:
|
||||
}
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
if terminal := waitRunEvent(t, pipe.events); terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
if claimed := <-claimResult; !claimed {
|
||||
t.Fatal("reset normalized expiry did not claim the watchdog fence")
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
||||
func TestTunnelWatchdogStaleExpiryYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-stale-expiry")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-stale-expiry", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
sink := call.sink.(*tunnelSink)
|
||||
claimStarted := make(chan struct{})
|
||||
releaseClaim := make(chan struct{})
|
||||
claimResult := make(chan bool, 1)
|
||||
var firstClaim sync.Once
|
||||
sink.beforeStallClaim = func() {
|
||||
firstClaim.Do(func() {
|
||||
close(claimStarted)
|
||||
<-releaseClaim
|
||||
})
|
||||
}
|
||||
sink.afterStallClaim = func(claimed bool) { claimResult <- claimed }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
stallTimer.fire()
|
||||
<-claimStarted // The old timer was consumed before the accepted frame progresses the attempt.
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "tunnel-stale-expiry", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindBody, Body: []byte("progress")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY {
|
||||
t.Fatalf("progress frame = %+v", frame)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
close(releaseClaim)
|
||||
if claimed := <-claimResult; claimed {
|
||||
t.Fatal("stale tunnel expiry fenced after progress reset the watchdog")
|
||||
}
|
||||
select {
|
||||
case frame := <-pipe.frames:
|
||||
t.Fatalf("stale tunnel expiry emitted terminal: %+v", frame)
|
||||
default:
|
||||
}
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
if terminal := waitTunnelFrame(t, pipe.frames); terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
if claimed := <-claimResult; !claimed {
|
||||
t.Fatal("reset tunnel expiry did not claim the watchdog fence")
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
||||
func TestRunWatchdogOldArmFireDuringResetYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-old-arm-during-reset")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-old-arm-during-reset", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
sink := call.sink.(*terminalDeferringSink)
|
||||
captureResults := make(chan bool, 2)
|
||||
sink.observer.afterExpiryCapture = func(valid bool) { captureResults <- valid }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
clock.advanceTo(clock.current().Add(time.Second))
|
||||
stallTimer.beforeReset = stallTimer.fireStaleArmDuringReset
|
||||
if err := call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-old-arm-during-reset", Type: runtime.EventTypeDelta, Delta: "progress"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if event := waitRunEvent(t, pipe.events); event.GetType() != string(runtime.EventTypeDelta) {
|
||||
t.Fatalf("progress event = %+v", event)
|
||||
}
|
||||
if valid := <-captureResults; valid {
|
||||
t.Fatal("old normalized arm was accepted while progress reset the watchdog")
|
||||
}
|
||||
if err := call.ctx.Err(); err != nil {
|
||||
t.Fatal("old normalized arm canceled the provider before the reset threshold")
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
if valid := <-captureResults; !valid {
|
||||
t.Fatal("reset normalized arm was not accepted after its full threshold")
|
||||
}
|
||||
if terminal := waitRunEvent(t, pipe.events); terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
||||
func TestTunnelWatchdogOldArmFireDuringResetYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-old-arm-during-reset")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-old-arm-during-reset", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
sink := call.sink.(*tunnelSink)
|
||||
captureResults := make(chan bool, 2)
|
||||
sink.observer.afterExpiryCapture = func(valid bool) { captureResults <- valid }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
clock.advanceTo(clock.current().Add(time.Second))
|
||||
stallTimer.beforeReset = stallTimer.fireStaleArmDuringReset
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "tunnel-old-arm-during-reset", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindBody, Body: []byte("progress")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY {
|
||||
t.Fatalf("progress frame = %+v", frame)
|
||||
}
|
||||
if valid := <-captureResults; valid {
|
||||
t.Fatal("old tunnel arm was accepted while progress reset the watchdog")
|
||||
}
|
||||
if err := call.ctx.Err(); err != nil {
|
||||
t.Fatal("old tunnel arm canceled the provider before the reset threshold")
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
if valid := <-captureResults; !valid {
|
||||
t.Fatal("reset tunnel arm was not accepted after its full threshold")
|
||||
}
|
||||
if terminal := waitTunnelFrame(t, pipe.frames); terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
||||
func TestRunWatchdogStaleExpiryBeforeCaptureYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("run-stale-before-capture")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnRunRequest(context.Background(), pipe.sess, &iop.RunRequest{RunId: "run-stale-before-capture", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.runCalls
|
||||
sink := call.sink.(*terminalDeferringSink)
|
||||
captureStarted := make(chan struct{})
|
||||
releaseCapture := make(chan struct{})
|
||||
captureResults := make(chan bool, 4)
|
||||
var firstCapture sync.Once
|
||||
sink.observer.beforeExpiryCapture = func() {
|
||||
firstCapture.Do(func() {
|
||||
close(captureStarted)
|
||||
<-releaseCapture
|
||||
})
|
||||
}
|
||||
sink.observer.afterExpiryCapture = func(valid bool) { captureResults <- valid }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
stallTimer.fire()
|
||||
<-captureStarted // The old timer signal was received before its validity is captured.
|
||||
if err := call.sink.Emit(context.Background(), runtime.RuntimeEvent{RunID: "run-stale-before-capture", Type: runtime.EventTypeDelta, Delta: "progress"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if event := waitRunEvent(t, pipe.events); event.GetType() != string(runtime.EventTypeDelta) {
|
||||
t.Fatalf("progress event = %+v", event)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
close(releaseCapture)
|
||||
if valid := <-captureResults; valid {
|
||||
t.Fatal("stale normalized expiry captured as valid after progress reset the watchdog")
|
||||
}
|
||||
if err := call.ctx.Err(); err != nil {
|
||||
t.Fatal("stale normalized expiry canceled the provider before its reset threshold")
|
||||
}
|
||||
select {
|
||||
case event := <-pipe.events:
|
||||
t.Fatalf("stale normalized expiry emitted terminal: %+v", event)
|
||||
default:
|
||||
}
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.runReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("run result = %v", err)
|
||||
}
|
||||
if valid := <-captureResults; !valid {
|
||||
t.Fatal("reset normalized expiry was not captured as valid after its full threshold")
|
||||
}
|
||||
if terminal := waitRunEvent(t, pipe.events); terminal.GetType() != string(runtime.EventTypeError) || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
||||
func TestTunnelWatchdogStaleExpiryBeforeCaptureYieldsToProgress(t *testing.T) {
|
||||
clock := newManualAttemptClock()
|
||||
adapter := newControlledWatchdogAdapter("tunnel-stale-before-capture")
|
||||
n := newWatchdogNode(t, adapter, clock)
|
||||
pipe := newWatchdogPipe(t)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- n.OnProviderTunnelRequest(context.Background(), pipe.sess, &iop.ProviderTunnelRequest{RunId: "tunnel-stale-before-capture", TunnelId: "tunnel", Adapter: adapter.Name(), Target: "target", ResponseStallTimeoutMs: 1000})
|
||||
}()
|
||||
call := <-adapter.tunnelCalls
|
||||
sink := call.sink.(*tunnelSink)
|
||||
captureStarted := make(chan struct{})
|
||||
releaseCapture := make(chan struct{})
|
||||
captureResults := make(chan bool, 4)
|
||||
var firstCapture sync.Once
|
||||
sink.observer.beforeExpiryCapture = func() {
|
||||
firstCapture.Do(func() {
|
||||
close(captureStarted)
|
||||
<-releaseCapture
|
||||
})
|
||||
}
|
||||
sink.observer.afterExpiryCapture = func(valid bool) { captureResults <- valid }
|
||||
|
||||
stallTimer := clock.waitTimer(t, 0)
|
||||
stallTimer.fire()
|
||||
<-captureStarted // The old timer signal was received before its validity is captured.
|
||||
if err := call.sink.EmitTunnelFrame(context.Background(), runtime.ProviderTunnelFrame{RunID: "tunnel-stale-before-capture", TunnelID: "tunnel", Kind: runtime.ProviderTunnelFrameKindBody, Body: []byte("progress")}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if frame := waitTunnelFrame(t, pipe.frames); frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY {
|
||||
t.Fatalf("progress frame = %+v", frame)
|
||||
}
|
||||
requireTimerDurations(t, stallTimer, time.Second, time.Second)
|
||||
close(releaseCapture)
|
||||
if valid := <-captureResults; valid {
|
||||
t.Fatal("stale tunnel expiry captured as valid after progress reset the watchdog")
|
||||
}
|
||||
if err := call.ctx.Err(); err != nil {
|
||||
t.Fatal("stale tunnel expiry canceled the provider before its reset threshold")
|
||||
}
|
||||
select {
|
||||
case frame := <-pipe.frames:
|
||||
t.Fatalf("stale tunnel expiry emitted terminal: %+v", frame)
|
||||
default:
|
||||
}
|
||||
|
||||
stallTimer.fire()
|
||||
waitContextCanceled(t, call.ctx)
|
||||
grace := clock.waitTimer(t, 1)
|
||||
adapter.tunnelReturn <- nil
|
||||
if err := <-done; err != errProviderResponseStalled {
|
||||
t.Fatalf("tunnel result = %v", err)
|
||||
}
|
||||
if valid := <-captureResults; !valid {
|
||||
t.Fatal("reset tunnel expiry was not captured as valid after its full threshold")
|
||||
}
|
||||
if terminal := waitTunnelFrame(t, pipe.frames); terminal.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || terminal.GetMetadata()["attempt_fence"] != "confirmed" {
|
||||
t.Fatalf("terminal = %+v", terminal)
|
||||
}
|
||||
requireTimerDurations(t, grace, defaultAttemptCloseGrace)
|
||||
}
|
||||
|
|
@ -32,12 +32,6 @@ type Node struct {
|
|||
credentialConsumer *credentiallease.Consumer
|
||||
workspaceMu sync.RWMutex
|
||||
workspaceRuntime *workspace.Runtime
|
||||
watchdogClock attemptClock
|
||||
|
||||
// liveness is the bounded stall-observability observer. Production Nodes
|
||||
// share one process-global collector set; tests inject an isolated registry
|
||||
// via the test-only constructor path in liveness_observability.go.
|
||||
liveness *nodeLivenessObserver
|
||||
}
|
||||
|
||||
func (n *Node) SetCredentialConsumer(consumer *credentiallease.Consumer) {
|
||||
|
|
@ -86,7 +80,5 @@ func New(
|
|||
out: out,
|
||||
logger: logger,
|
||||
currentConfigSet: initialConfigSet,
|
||||
watchdogClock: realAttemptClock{},
|
||||
liveness: newProductionNodeLivenessObserver(logger),
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,16 +26,15 @@ type fixedRouter struct {
|
|||
|
||||
func (r *fixedRouter) Resolve(_ context.Context, req runtime.RunRequest) (runtime.ExecutionSpec, error) {
|
||||
return runtime.ExecutionSpec{
|
||||
RunID: req.RunID,
|
||||
Adapter: r.adapterName,
|
||||
Target: req.Target,
|
||||
SessionID: req.SessionID,
|
||||
Background: req.Background,
|
||||
Policy: req.Policy,
|
||||
Input: req.Input,
|
||||
TimeoutSec: req.TimeoutSec,
|
||||
Metadata: req.Metadata,
|
||||
ResponseStallTimeoutMS: req.ResponseStallTimeoutMS,
|
||||
RunID: req.RunID,
|
||||
Adapter: r.adapterName,
|
||||
Target: req.Target,
|
||||
SessionID: req.SessionID,
|
||||
Background: req.Background,
|
||||
Policy: req.Policy,
|
||||
Input: req.Input,
|
||||
TimeoutSec: req.TimeoutSec,
|
||||
Metadata: req.Metadata,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,54 +0,0 @@
|
|||
package node_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// TestNodeSuccessfulTunnelFramesCarryNoHealthEvidence proves health evidence and
|
||||
// the connection-scoped observation sequence are confined to the stall terminal:
|
||||
// a successful tunnel over a bound session emits no frame carrying stall/health
|
||||
// metadata.
|
||||
func TestNodeSuccessfulTunnelFramesCarryNoHealthEvidence(t *testing.T) {
|
||||
mta := &mockTunnelAdapter{t: t, expectedReq: runtime.ProviderTunnelRequest{RunID: "run-health-scope", TunnelID: "tunnel-health-scope"}}
|
||||
router := &fixedRouter{adapterName: "openai_compat", adapters: map[string]runtime.Provider{"openai_compat": mta}}
|
||||
n, _ := makeNode(t, router)
|
||||
|
||||
edgeSide, sess := buildSessionTestPipeForNode(t)
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 8)
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelFrame](&edgeSide.Communicator, func(tf *iop.ProviderTunnelFrame) {
|
||||
frames <- proto.Clone(tf).(*iop.ProviderTunnelFrame)
|
||||
})
|
||||
|
||||
if err := n.OnProviderTunnelRequest(context.Background(), sess, &iop.ProviderTunnelRequest{
|
||||
RunId: "run-health-scope", TunnelId: "tunnel-health-scope", Adapter: "openai_compat", Target: "qwen", Method: "POST", Path: "/v1/chat/completions",
|
||||
}); err != nil {
|
||||
t.Fatalf("tunnel: %v", err)
|
||||
}
|
||||
|
||||
stallKeys := []string{"provider_health", "liveness_classification", "health_observation_seq", "attempt_fence", "failure_code"}
|
||||
deadline := time.After(2 * time.Second)
|
||||
for {
|
||||
select {
|
||||
case tf := <-frames:
|
||||
meta := tf.GetMetadata()
|
||||
for _, key := range stallKeys {
|
||||
if _, present := meta[key]; present {
|
||||
t.Fatalf("successful tunnel frame leaked stall/health key %q: %#v", key, meta)
|
||||
}
|
||||
}
|
||||
if tf.GetKind() == iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END {
|
||||
return
|
||||
}
|
||||
case <-deadline:
|
||||
t.Fatal("terminal END frame was not observed")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -6,8 +6,6 @@ import (
|
|||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"io"
|
||||
"math"
|
||||
"net"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
|
|
@ -18,10 +16,7 @@ import (
|
|||
"go.uber.org/zap"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"iop/apps/node/internal/adapters"
|
||||
"iop/apps/node/internal/node"
|
||||
"iop/apps/node/internal/router"
|
||||
"iop/apps/node/internal/store"
|
||||
"iop/apps/node/internal/transport"
|
||||
"iop/packages/go/credentiallease"
|
||||
runtime "iop/packages/go/execution"
|
||||
|
|
@ -110,6 +105,30 @@ type cancelAwareTunnelAdapter struct {
|
|||
observedCancel chan struct{}
|
||||
}
|
||||
|
||||
type heldTerminalTunnelAdapter struct {
|
||||
countingAdapter
|
||||
started chan struct{}
|
||||
release chan struct{}
|
||||
last runtime.ProviderTunnelRequest
|
||||
}
|
||||
|
||||
func (a *heldTerminalTunnelAdapter) Name() string { return "held-tunnel" }
|
||||
func (a *heldTerminalTunnelAdapter) Capabilities(context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: a.Name(), MaxConcurrency: 1}, nil
|
||||
}
|
||||
func (a *heldTerminalTunnelAdapter) TunnelProvider(ctx context.Context, req runtime.ProviderTunnelRequest, sink runtime.ProviderTunnelSink) error {
|
||||
a.last = req
|
||||
close(a.started)
|
||||
select {
|
||||
case <-a.release:
|
||||
return sink.EmitTunnelFrame(ctx, runtime.ProviderTunnelFrame{
|
||||
RunID: req.RunID, TunnelID: req.TunnelID, Kind: runtime.ProviderTunnelFrameKindEnd, End: true, Timestamp: time.Now(),
|
||||
})
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (a *cancelAwareTunnelAdapter) Name() string { return "openai_compat" }
|
||||
func (a *cancelAwareTunnelAdapter) Capabilities(_ context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: "openai_compat", Targets: []string{"qwen"}}, nil
|
||||
|
|
@ -631,159 +650,68 @@ func TestNodeOnProviderTunnelRequest_AdapterErrorNoDuplicate(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestOnProviderTunnelRequestRejectsNegativeStallTimeout verifies that a
|
||||
// tunnel request with a negative response_stall_timeout_ms is rejected
|
||||
// before reaching the adapter.
|
||||
func TestOnProviderTunnelRequestRejectsNegativeStallTimeout(t *testing.T) {
|
||||
set, err := adapters.BuildConfigSet(&iop.NodeConfigPayload{}, zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatalf("BuildConfigSet: %v", err)
|
||||
}
|
||||
rtr := router.New(set.Registry, zap.NewNop())
|
||||
st, err := store.New(":memory:", zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatalf("store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
|
||||
n := node.New("test-node", rtr, st, 1, io.Discard, zap.NewNop(), set)
|
||||
|
||||
// Build a tunnel request with negative stall timeout.
|
||||
req := &iop.ProviderTunnelRequest{
|
||||
RunId: "tunnel-neg",
|
||||
TunnelId: "tunnel-neg-tunnel",
|
||||
Adapter: "mock",
|
||||
Target: "echo",
|
||||
Method: "POST",
|
||||
Path: "/v1/chat/completions",
|
||||
ResponseStallTimeoutMs: -1,
|
||||
}
|
||||
|
||||
// The negative timeout should be rejected before the adapter is looked up.
|
||||
err = n.OnProviderTunnelRequest(context.Background(), nil, req)
|
||||
if err == nil {
|
||||
t.Fatal("expected error for negative stall timeout")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "response_stall_timeout_ms") {
|
||||
t.Fatalf("expected error mentioning response_stall_timeout_ms, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnProviderTunnelRequestAcceptsZeroStallTimeout verifies that a tunnel
|
||||
// request with zero response_stall_timeout_ms passes validation (Node applies
|
||||
// the documented default).
|
||||
func TestOnProviderTunnelRequestAcceptsZeroStallTimeout(t *testing.T) {
|
||||
set, err := adapters.BuildConfigSet(&iop.NodeConfigPayload{}, zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatalf("BuildConfigSet: %v", err)
|
||||
}
|
||||
rtr := router.New(set.Registry, zap.NewNop())
|
||||
st, err := store.New(":memory:", zap.NewNop())
|
||||
if err != nil {
|
||||
t.Fatalf("store: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = st.Close() })
|
||||
|
||||
n := node.New("test-node", rtr, st, 1, io.Discard, zap.NewNop(), set)
|
||||
|
||||
req := &iop.ProviderTunnelRequest{
|
||||
RunId: "tunnel-zero",
|
||||
TunnelId: "tunnel-zero-tunnel",
|
||||
Adapter: "mock",
|
||||
Target: "echo",
|
||||
Method: "POST",
|
||||
Path: "/v1/chat/completions",
|
||||
ResponseStallTimeoutMs: 0,
|
||||
}
|
||||
|
||||
// Zero should pass validation and attempt adapter lookup.
|
||||
// The mock adapter is not registered, so we expect a lookup error,
|
||||
// not a stall timeout error.
|
||||
err = n.OnProviderTunnelRequest(context.Background(), nil, req)
|
||||
if err != nil && strings.Contains(err.Error(), "response_stall_timeout_ms") {
|
||||
t.Fatalf("zero stall timeout should not be rejected: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnProviderTunnelRequestRetainsValidatedStallTimeout(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
raw int64
|
||||
want int64
|
||||
bad bool
|
||||
}{
|
||||
{name: "zero defaults", raw: 0, want: runtime.DefaultResponseStallTimeoutMS},
|
||||
{name: "positive preserved", raw: 45000, want: 45000},
|
||||
{name: "exact safe boundary", raw: math.MaxInt64 / int64(time.Millisecond), want: math.MaxInt64 / int64(time.Millisecond)},
|
||||
{name: "overflow rejected", raw: math.MaxInt64/int64(time.Millisecond) + 1, bad: true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
adapter := &stallCaptureTunnelAdapter{}
|
||||
rtr := &fixedRouter{adapterName: adapter.Name(), adapters: map[string]runtime.Provider{adapter.Name(): adapter}}
|
||||
n, _ := makeNode(t, rtr)
|
||||
err := n.OnProviderTunnelRequest(context.Background(), nil, &iop.ProviderTunnelRequest{
|
||||
RunId: "tunnel-stall-" + tc.name,
|
||||
TunnelId: "tunnel-stall-id",
|
||||
Adapter: adapter.Name(),
|
||||
Target: "qwen",
|
||||
TimeoutSec: 17,
|
||||
ResponseStallTimeoutMs: tc.raw,
|
||||
})
|
||||
if (err != nil) != tc.bad {
|
||||
t.Fatalf("OnProviderTunnelRequest error = %v, want bad=%t", err, tc.bad)
|
||||
}
|
||||
if tc.bad {
|
||||
if got := atomic.LoadInt32(&adapter.calls); got != 0 {
|
||||
t.Fatalf("tunnel adapter calls = %d, want 0", got)
|
||||
}
|
||||
return
|
||||
}
|
||||
if adapter.last.ResponseStallTimeoutMS != tc.want {
|
||||
t.Errorf("response stall timeout = %d, want %d", adapter.last.ResponseStallTimeoutMS, tc.want)
|
||||
}
|
||||
if adapter.last.TimeoutSec != 17 {
|
||||
t.Errorf("hard timeout = %d, want 17", adapter.last.TimeoutSec)
|
||||
}
|
||||
func TestOnProviderTunnelRequestHonorsLifecycleTerminal(t *testing.T) {
|
||||
adapter := &heldTerminalTunnelAdapter{started: make(chan struct{}), release: make(chan struct{})}
|
||||
rtr := &fixedRouter{adapterName: adapter.Name(), adapters: map[string]runtime.Provider{adapter.Name(): adapter}}
|
||||
n, _ := makeNode(t, rtr)
|
||||
edgeSide, sess := buildSessionTestPipeForNode(t)
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 2)
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelFrame](&edgeSide.Communicator, func(frame *iop.ProviderTunnelFrame) {
|
||||
frames <- proto.Clone(frame).(*iop.ProviderTunnelFrame)
|
||||
})
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
errCh <- n.OnProviderTunnelRequest(context.Background(), sess, &iop.ProviderTunnelRequest{
|
||||
RunId: "run-held-tunnel", TunnelId: "held-tunnel", Adapter: adapter.Name(), Target: "qwen",
|
||||
})
|
||||
}()
|
||||
select {
|
||||
case <-adapter.started:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("tunnel adapter did not start")
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
select {
|
||||
case err := <-errCh:
|
||||
t.Fatalf("silent tunnel ended before lifecycle terminal: %v", err)
|
||||
default:
|
||||
}
|
||||
close(adapter.release)
|
||||
if err := <-errCh; err != nil {
|
||||
t.Fatalf("provider terminal result: %v", err)
|
||||
}
|
||||
select {
|
||||
case frame := <-frames:
|
||||
if frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END {
|
||||
t.Fatalf("terminal frame = %+v", frame)
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("provider terminal frame not delivered")
|
||||
}
|
||||
select {
|
||||
case extra := <-frames:
|
||||
t.Fatalf("duplicate terminal frame: %+v", extra)
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnProviderTunnelRequestInvalidStallTimeoutKeepsCorrelation(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
raw int64
|
||||
}{
|
||||
{name: "negative", raw: -1},
|
||||
{name: "overflow", raw: math.MaxInt64/int64(time.Millisecond) + 1},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
adapter := &stallCaptureTunnelAdapter{}
|
||||
rtr := &fixedRouter{adapterName: adapter.Name(), adapters: map[string]runtime.Provider{adapter.Name(): adapter}}
|
||||
n, _ := makeNode(t, rtr)
|
||||
edgeSide, sess := buildSessionTestPipeForNode(t)
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 2)
|
||||
toki.AddListenerTyped[*iop.ProviderTunnelFrame](&edgeSide.Communicator, func(frame *iop.ProviderTunnelFrame) {
|
||||
frames <- proto.Clone(frame).(*iop.ProviderTunnelFrame)
|
||||
})
|
||||
|
||||
req := &iop.ProviderTunnelRequest{RunId: "run-" + tc.name, TunnelId: "tunnel-" + tc.name, Adapter: adapter.Name(), Target: "qwen", ResponseStallTimeoutMs: tc.raw}
|
||||
err := n.OnProviderTunnelRequest(context.Background(), sess, req)
|
||||
if err == nil || !strings.Contains(err.Error(), "response_stall_timeout_ms") {
|
||||
t.Fatalf("validation error = %v", err)
|
||||
}
|
||||
frame := <-frames
|
||||
if frame.GetRunId() != req.GetRunId() || frame.GetTunnelId() != req.GetTunnelId() || frame.GetKind() != iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR || !strings.Contains(frame.GetError(), "response_stall_timeout_ms") {
|
||||
t.Fatalf("error frame = %+v", frame)
|
||||
}
|
||||
if got := atomic.LoadInt32(&adapter.calls); got != 0 {
|
||||
t.Fatalf("adapter calls = %d, want 0", got)
|
||||
}
|
||||
select {
|
||||
case extra := <-frames:
|
||||
t.Fatalf("unexpected second error frame: %+v", extra)
|
||||
default:
|
||||
}
|
||||
t.Run("hard deadline remains terminal owner", func(t *testing.T) {
|
||||
deadlineAdapter := &cancelAwareTunnelAdapter{
|
||||
started: make(chan struct{}),
|
||||
observedCancel: make(chan struct{}),
|
||||
}
|
||||
deadlineRouter := &fixedRouter{adapterName: deadlineAdapter.Name(), adapters: map[string]runtime.Provider{deadlineAdapter.Name(): deadlineAdapter}}
|
||||
deadlineNode, _ := makeNode(t, deadlineRouter)
|
||||
err := deadlineNode.OnProviderTunnelRequest(context.Background(), nil, &iop.ProviderTunnelRequest{
|
||||
RunId: "run-tunnel-deadline", TunnelId: "tunnel-deadline", Adapter: deadlineAdapter.Name(), Target: "qwen",
|
||||
TimeoutSec: 1,
|
||||
})
|
||||
}
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("hard deadline result = %v, want context.DeadlineExceeded", err)
|
||||
}
|
||||
select {
|
||||
case <-deadlineAdapter.observedCancel:
|
||||
default:
|
||||
t.Fatal("tunnel adapter did not observe hard deadline")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,16 +3,11 @@ package node_test
|
|||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
toki "git.toki-labs.com/toki/proto-socket/go"
|
||||
"go.uber.org/zap"
|
||||
"google.golang.org/protobuf/proto"
|
||||
|
||||
"iop/apps/node/internal/transport"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
|
|
@ -53,6 +48,32 @@ type blockingAdapter struct {
|
|||
done chan struct{}
|
||||
}
|
||||
|
||||
// heldTerminalAdapter stays silent until the test releases it, then emits one
|
||||
// provider-owned terminal event. Context cancellation remains the only other
|
||||
// completion path.
|
||||
type heldTerminalAdapter struct {
|
||||
started chan struct{}
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
func newHeldTerminalAdapter() *heldTerminalAdapter {
|
||||
return &heldTerminalAdapter{started: make(chan struct{}), release: make(chan struct{})}
|
||||
}
|
||||
|
||||
func (a *heldTerminalAdapter) Name() string { return "held-terminal" }
|
||||
func (a *heldTerminalAdapter) Capabilities(context.Context) (runtime.Capabilities, error) {
|
||||
return runtime.Capabilities{AdapterName: a.Name(), MaxConcurrency: 1}, nil
|
||||
}
|
||||
func (a *heldTerminalAdapter) Execute(ctx context.Context, spec runtime.ExecutionSpec, sink runtime.EventSink) error {
|
||||
close(a.started)
|
||||
select {
|
||||
case <-a.release:
|
||||
return sink.Emit(ctx, runtime.RuntimeEvent{RunID: spec.RunID, Type: runtime.EventTypeComplete, Timestamp: time.Now()})
|
||||
case <-ctx.Done():
|
||||
return runtime.ErrRunCancelled
|
||||
}
|
||||
}
|
||||
|
||||
func newBlockingAdapter() *blockingAdapter {
|
||||
return &blockingAdapter{started: make(chan struct{}), done: make(chan struct{})}
|
||||
}
|
||||
|
|
@ -147,51 +168,48 @@ func TestOnRunRequest_Success(t *testing.T) {
|
|||
if run.Status != "completed" {
|
||||
t.Fatalf("expected completed status, got %q", run.Status)
|
||||
}
|
||||
if adapter.lastSpec.ResponseStallTimeoutMS != runtime.DefaultResponseStallTimeoutMS {
|
||||
t.Fatalf("default response stall timeout = %d, want %d", adapter.lastSpec.ResponseStallTimeoutMS, runtime.DefaultResponseStallTimeoutMS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnRunRequestRetainsValidatedStallTimeout(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
raw int64
|
||||
want int64
|
||||
wantError bool
|
||||
}{
|
||||
{name: "positive override", raw: 45000, want: 45000},
|
||||
{name: "negative rejected before adapter", raw: -1, wantError: true},
|
||||
{name: "overflow rejected before adapter", raw: 99999999999999, wantError: true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
adapter := &countingAdapter{}
|
||||
rtr := &fixedRouter{adapterName: "test", adapters: map[string]runtime.Provider{"test": adapter}}
|
||||
n, _ := makeNode(t, rtr)
|
||||
err := n.OnRunRequest(context.Background(), &transport.Session{}, &iop.RunRequest{
|
||||
RunId: "run-stall-" + tc.name,
|
||||
Adapter: "test",
|
||||
Target: "v1",
|
||||
TimeoutSec: 17,
|
||||
ResponseStallTimeoutMs: tc.raw,
|
||||
})
|
||||
if (err != nil) != tc.wantError {
|
||||
t.Fatalf("OnRunRequest error = %v, want error=%t", err, tc.wantError)
|
||||
}
|
||||
if tc.wantError {
|
||||
if got := atomic.LoadInt32(&adapter.executeCalls); got != 0 {
|
||||
t.Fatalf("adapter execute calls = %d, want 0", got)
|
||||
}
|
||||
return
|
||||
}
|
||||
if adapter.lastSpec.ResponseStallTimeoutMS != tc.want {
|
||||
t.Errorf("response stall timeout = %d, want %d", adapter.lastSpec.ResponseStallTimeoutMS, tc.want)
|
||||
}
|
||||
if adapter.lastSpec.TimeoutSec != 17 {
|
||||
t.Errorf("hard timeout = %d, want 17", adapter.lastSpec.TimeoutSec)
|
||||
}
|
||||
func TestOnRunRequestHonorsLifecycleTerminal(t *testing.T) {
|
||||
t.Run("silence ends only on provider terminal", func(t *testing.T) {
|
||||
adapter := newHeldTerminalAdapter()
|
||||
rtr := &fixedRouter{adapterName: adapter.Name(), adapters: map[string]runtime.Provider{adapter.Name(): adapter}}
|
||||
n, st := makeNode(t, rtr)
|
||||
if err := n.OnRunRequest(context.Background(), &transport.Session{}, &iop.RunRequest{
|
||||
RunId: "run-held-terminal", Adapter: adapter.Name(), Target: "v1", Background: true,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case <-adapter.started:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("adapter did not start")
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
select {
|
||||
case <-adapter.release:
|
||||
t.Fatal("provider release channel changed unexpectedly")
|
||||
default:
|
||||
}
|
||||
if run, err := st.GetRun(context.Background(), "run-held-terminal"); err != nil || run == nil || run.Status != "running" {
|
||||
t.Fatalf("run before provider terminal = %+v, err=%v", run, err)
|
||||
}
|
||||
close(adapter.release)
|
||||
requireStatus(t, st, "run-held-terminal", "completed")
|
||||
})
|
||||
|
||||
t.Run("hard deadline remains terminal owner", func(t *testing.T) {
|
||||
adapter := newBlockingAdapter()
|
||||
rtr := &fixedRouter{adapterName: "blocking", adapters: map[string]runtime.Provider{"blocking": adapter}}
|
||||
n, st := makeNode(t, rtr)
|
||||
err := n.OnRunRequest(context.Background(), &transport.Session{}, &iop.RunRequest{
|
||||
RunId: "run-deadline", Adapter: "blocking", Target: "v1", TimeoutSec: 1,
|
||||
})
|
||||
}
|
||||
if !errors.Is(err, runtime.ErrRunCancelled) {
|
||||
t.Fatalf("deadline result = %v, want ErrRunCancelled", err)
|
||||
}
|
||||
requireStatus(t, st, "run-deadline", "cancelled")
|
||||
})
|
||||
}
|
||||
|
||||
func TestOnRunRequest_ForegroundAdapterErrorReturned(t *testing.T) {
|
||||
|
|
@ -411,48 +429,3 @@ func TestResolveAdapterErrorObservedByEdge(t *testing.T) {
|
|||
t.Fatalf("expected resolve prefix, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOnRunRequestSuccessTerminalCarriesNoHealthEvidence proves the normalized
|
||||
// health evidence and connection-scoped observation sequence are confined to the
|
||||
// stall terminal: a successful run over a bound session emits a completion event
|
||||
// with no stall/health metadata.
|
||||
func TestOnRunRequestSuccessTerminalCarriesNoHealthEvidence(t *testing.T) {
|
||||
adapter := &countingAdapter{}
|
||||
router := &fixedRouter{adapterName: "test", adapters: map[string]runtime.Provider{"test": adapter}}
|
||||
n, _ := makeNode(t, router)
|
||||
|
||||
edgeConn, nodeConn := net.Pipe()
|
||||
edge := toki.NewTcpClient(edgeConn, 0, 0, toki.ParserMap{
|
||||
toki.TypeNameOf(&iop.RunEvent{}): func(b []byte) (proto.Message, error) {
|
||||
m := &iop.RunEvent{}
|
||||
return m, proto.Unmarshal(b, m)
|
||||
},
|
||||
})
|
||||
nodeSide := toki.NewTcpClient(nodeConn, 0, 0, toki.ParserMap{})
|
||||
t.Cleanup(func() { _ = edge.Close(); _ = nodeSide.Close() })
|
||||
events := make(chan *iop.RunEvent, 8)
|
||||
toki.AddListenerTyped[*iop.RunEvent](&edge.Communicator, func(e *iop.RunEvent) {
|
||||
events <- proto.Clone(e).(*iop.RunEvent)
|
||||
})
|
||||
sess := transport.ExportNewSession(nodeSide, zap.NewNop(), "node-id-1", "alias-1")
|
||||
|
||||
if err := n.OnRunRequest(context.Background(), sess, &iop.RunRequest{RunId: "run-health-scope", Adapter: "test", Target: "v1"}); err != nil {
|
||||
t.Fatalf("run request: %v", err)
|
||||
}
|
||||
|
||||
stallKeys := []string{"provider_health", "liveness_classification", "health_observation_seq", "attempt_fence", "failure_code"}
|
||||
select {
|
||||
case ev := <-events:
|
||||
if ev.GetType() != string(runtime.EventTypeComplete) {
|
||||
t.Fatalf("terminal type = %q, want complete", ev.GetType())
|
||||
}
|
||||
meta := ev.GetMetadata()
|
||||
for _, key := range stallKeys {
|
||||
if _, present := meta[key]; present {
|
||||
t.Fatalf("successful run terminal leaked stall/health key %q: %#v", key, meta)
|
||||
}
|
||||
}
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("no run terminal emitted")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -21,10 +21,6 @@ func (n *Node) OnRunRequest(ctx context.Context, sess *transport.Session, req *i
|
|||
rr := runRequestFromProto(req)
|
||||
printEdgeMessage(n.out, rr.Input)
|
||||
|
||||
if err := n.validateRunStallTimeout(sess, req, &rr); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
n.configSetMu.RLock()
|
||||
configLocked := true
|
||||
defer func() {
|
||||
|
|
@ -89,15 +85,45 @@ func (n *Node) OnRunRequest(ctx context.Context, sess *transport.Session, req *i
|
|||
n.configSetMu.RUnlock()
|
||||
|
||||
var sender protoSender = noopSender{}
|
||||
var seq healthObservationSequencer
|
||||
if sess != nil && sess.IsAlive() {
|
||||
sender = sess
|
||||
seq = sess
|
||||
}
|
||||
probe := healthProbeFor(adapter, caps.AdapterName, caps.InstanceKey, spec.Target)
|
||||
runSink := &terminalDeferringSink{inner: &sessionSink{
|
||||
sess: sender,
|
||||
out: n.out,
|
||||
nodeID: n.nodeID,
|
||||
sessionID: normalizeSessionID(spec.SessionID),
|
||||
background: spec.Background,
|
||||
}}
|
||||
|
||||
run := func() error {
|
||||
return n.executeNormalizedAttempt(ctx, execCtx, cancel, adapter, spec, ticket, h, sender, probe, seq)
|
||||
released := false
|
||||
releaseTicket := func() {
|
||||
if !released {
|
||||
ticket.release()
|
||||
released = true
|
||||
}
|
||||
}
|
||||
defer releaseTicket()
|
||||
defer cancel()
|
||||
defer n.runs.deregister(spec.RunID)
|
||||
defer close(h.done)
|
||||
|
||||
execErr := adapter.Execute(execCtx, spec, runSink)
|
||||
releaseTicket()
|
||||
if !runSink.hasTerminalObserved() {
|
||||
if synthErr := n.synthAndEmitTerminal(ctx, runSink, spec, execErr); synthErr != nil && execErr == nil {
|
||||
execErr = synthErr
|
||||
}
|
||||
}
|
||||
n.completeRun(spec, execErr)
|
||||
if flushErr := runSink.Flush(context.Background()); flushErr != nil {
|
||||
n.logger.Warn("session: flush terminal events", zap.String("run_id", spec.RunID), zap.Error(flushErr))
|
||||
if execErr == nil {
|
||||
return flushErr
|
||||
}
|
||||
}
|
||||
return execErr
|
||||
}
|
||||
|
||||
if spec.Background {
|
||||
|
|
|
|||
|
|
@ -1,17 +1,14 @@
|
|||
package node
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"iop/apps/node/internal/transport"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
// runRequestFromProto is the Edge-Node wire boundary. Common runtime packages
|
||||
// remain independent of protobuf and Node transport details. ResponseStallTimeoutMS
|
||||
// is deliberately left unset here: the handler validates the raw wire value and
|
||||
// assigns the effective timeout via ValidateStallTimeoutOnWire before routing.
|
||||
// remain independent of protobuf and Node transport details. The deprecated
|
||||
// response-stall timeout stays accepted on the wire for mixed-version peers but
|
||||
// is intentionally not copied into the execution request.
|
||||
func runRequestFromProto(req *iop.RunRequest) runtime.RunRequest {
|
||||
return runtime.RunRequest{
|
||||
RunID: req.GetRunId(),
|
||||
|
|
@ -26,51 +23,6 @@ func runRequestFromProto(req *iop.RunRequest) runtime.RunRequest {
|
|||
}
|
||||
}
|
||||
|
||||
var allowlistedLivenessMetadataKeys = map[string]bool{
|
||||
"failure_code": true,
|
||||
"provider_health": true,
|
||||
"liveness_classification": true,
|
||||
"idle_duration_ms": true,
|
||||
"run_id": true,
|
||||
"attempt_id": true,
|
||||
"attempt_fence": true,
|
||||
"adapter": true,
|
||||
"target": true,
|
||||
"health_observation_seq": true,
|
||||
}
|
||||
|
||||
func allowlistedLivenessMetadata(metadata map[string]string) map[string]string {
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
var filtered map[string]string
|
||||
for k, v := range metadata {
|
||||
if allowlistedLivenessMetadataKeys[k] {
|
||||
if filtered == nil {
|
||||
filtered = make(map[string]string)
|
||||
}
|
||||
filtered[k] = v
|
||||
}
|
||||
}
|
||||
return filtered
|
||||
}
|
||||
|
||||
func executionFailureToProto(failure *runtime.Failure) *iop.ExecutionFailure {
|
||||
if failure == nil || failure.Code != runtime.FailureCodeResponseStalled {
|
||||
return nil
|
||||
}
|
||||
msg := failure.Message
|
||||
if msg == "" {
|
||||
msg = failure.Error()
|
||||
}
|
||||
return &iop.ExecutionFailure{
|
||||
Code: string(failure.Code),
|
||||
Message: msg,
|
||||
Retryable: failure.Retryable,
|
||||
Metadata: allowlistedLivenessMetadata(failure.Metadata),
|
||||
}
|
||||
}
|
||||
|
||||
// runEventToProto preserves the existing Edge-Node event values while
|
||||
// translating the host-neutral common event into the Node wire response.
|
||||
func runEventToProto(event runtime.RuntimeEvent, nodeID, sessionID string, background bool) *iop.RunEvent {
|
||||
|
|
@ -84,7 +36,6 @@ func runEventToProto(event runtime.RuntimeEvent, nodeID, sessionID string, backg
|
|||
Delta: event.Delta,
|
||||
Message: event.Message,
|
||||
Error: errorMessage,
|
||||
Failure: executionFailureToProto(event.Failure),
|
||||
Metadata: event.Metadata,
|
||||
Timestamp: event.Timestamp.UnixNano(),
|
||||
SessionId: sessionID,
|
||||
|
|
@ -101,47 +52,11 @@ func runEventToProto(event runtime.RuntimeEvent, nodeID, sessionID string, backg
|
|||
}
|
||||
return wireEvent
|
||||
}
|
||||
|
||||
// ValidateStallTimeoutOnWire validates a raw wire value and returns the
|
||||
// effective timeout before the request reaches the router or provider. Zero
|
||||
// resolves to the documented default; safe positive values pass through;
|
||||
// negative and overflow values are rejected instead of being silently defaulted.
|
||||
func ValidateStallTimeoutOnWire(ms int64) (int64, error) {
|
||||
effective, err := runtime.ResolveStallTimeoutMS(ms)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("response_stall_timeout_ms: %w", err)
|
||||
}
|
||||
return effective, nil
|
||||
}
|
||||
|
||||
func applyValidatedRunStallTimeout(req *iop.RunRequest, runReq *runtime.RunRequest) error {
|
||||
effective, err := ValidateStallTimeoutOnWire(req.GetResponseStallTimeoutMs())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
runReq.ResponseStallTimeoutMS = effective
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *Node) validateRunStallTimeout(sess *transport.Session, req *iop.RunRequest, runReq *runtime.RunRequest) error {
|
||||
if err := applyValidatedRunStallTimeout(req, runReq); err != nil {
|
||||
n.sendPreExecuteError(sess, req.GetRunId(), req.GetSessionId(), req.GetBackground(), n.nodeID, err.Error())
|
||||
return fmt.Errorf("node: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func providerTunnelRequestFromProto(req *iop.ProviderTunnelRequest) (runtime.ProviderTunnelRequest, error) {
|
||||
tr := runtime.ProviderTunnelRequest{
|
||||
func providerTunnelRequestFromProto(req *iop.ProviderTunnelRequest) runtime.ProviderTunnelRequest {
|
||||
return runtime.ProviderTunnelRequest{
|
||||
RunID: req.GetRunId(), TunnelID: req.GetTunnelId(), Adapter: req.GetAdapter(), Target: req.GetTarget(),
|
||||
Method: req.GetMethod(), Path: req.GetPath(), Operation: req.GetOperation(), Headers: req.GetHeaders(),
|
||||
Body: req.GetBody(), Stream: req.GetStream(), TimeoutSec: int(req.GetTimeoutSec()), Metadata: req.GetMetadata(),
|
||||
SessionID: req.GetSessionId(),
|
||||
}
|
||||
effective, err := ValidateStallTimeoutOnWire(req.GetResponseStallTimeoutMs())
|
||||
if err != nil {
|
||||
return tr, err
|
||||
}
|
||||
tr.ResponseStallTimeoutMS = effective
|
||||
return tr, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,7 +5,6 @@ import (
|
|||
"testing"
|
||||
"time"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/types/known/structpb"
|
||||
|
||||
runtime "iop/packages/go/execution"
|
||||
|
|
@ -89,172 +88,18 @@ func TestRunEventToProtoUsesTypedFailureMessageAsFallback(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestRunRequestFromProtoLeavesRawStallTimeoutForHandlerValidation(t *testing.T) {
|
||||
// The handler, not the protobuf mapper, validates and resolves the raw value.
|
||||
req := &iop.RunRequest{
|
||||
RunId: "r1",
|
||||
Adapter: "ollama",
|
||||
func TestRuntimeEventToProtoLeavesDeprecatedFailureFieldEmpty(t *testing.T) {
|
||||
event := runtime.RuntimeEvent{
|
||||
RunID: "run-2",
|
||||
Type: runtime.EventTypeError,
|
||||
Error: "provider error",
|
||||
Failure: &runtime.Failure{Code: runtime.FailureCodeProvider, Message: "provider error"},
|
||||
}
|
||||
runtimeReq := runRequestFromProto(req)
|
||||
if runtimeReq.ResponseStallTimeoutMS != 0 {
|
||||
t.Errorf("zero wire must remain raw before handler validation, got %d", runtimeReq.ResponseStallTimeoutMS)
|
||||
wire := runEventToProto(event, "node-1", "session-1", false)
|
||||
if wire.GetFailure() != nil {
|
||||
t.Fatalf("deprecated wire failure populated: %#v", wire.GetFailure())
|
||||
}
|
||||
|
||||
// Positive override → passes through.
|
||||
req2 := &iop.RunRequest{
|
||||
RunId: "r2",
|
||||
Adapter: "ollama",
|
||||
ResponseStallTimeoutMs: 60000,
|
||||
}
|
||||
runtimeReq2 := runRequestFromProto(req2)
|
||||
if runtimeReq2.ResponseStallTimeoutMS != 0 {
|
||||
t.Errorf("positive wire value must remain raw before handler validation, got %d", runtimeReq2.ResponseStallTimeoutMS)
|
||||
if wire.GetError() != "provider error" {
|
||||
t.Fatalf("error string = %q, want provider error", wire.GetError())
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStallTimeoutOnWire(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
raw int64
|
||||
want int64
|
||||
bad bool
|
||||
}{
|
||||
{name: "zero defaults", raw: 0, want: runtime.DefaultResponseStallTimeoutMS},
|
||||
{name: "positive preserved", raw: 60000, want: 60000},
|
||||
{name: "negative rejected", raw: -1, bad: true},
|
||||
{name: "overflow rejected", raw: 99999999999999, bad: true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got, err := ValidateStallTimeoutOnWire(tc.raw)
|
||||
if (err != nil) != tc.bad {
|
||||
t.Fatalf("ValidateStallTimeoutOnWire(%d) error = %v, want bad=%t", tc.raw, err, tc.bad)
|
||||
}
|
||||
if !tc.bad && got != tc.want {
|
||||
t.Errorf("ValidateStallTimeoutOnWire(%d) = %d, want %d", tc.raw, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseStallTimeoutWireRoundTrip(t *testing.T) {
|
||||
values := []int64{0, 60000, -1, (int64(1) << 62) / int64(time.Millisecond)}
|
||||
for _, value := range values {
|
||||
t.Run("run", func(t *testing.T) {
|
||||
original := &iop.RunRequest{ResponseStallTimeoutMs: value}
|
||||
encoded, err := proto.Marshal(original)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
decoded := &iop.RunRequest{}
|
||||
if err := proto.Unmarshal(encoded, decoded); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if decoded.GetResponseStallTimeoutMs() != value {
|
||||
t.Fatalf("RunRequest round trip = %d, want %d", decoded.GetResponseStallTimeoutMs(), value)
|
||||
}
|
||||
})
|
||||
t.Run("tunnel", func(t *testing.T) {
|
||||
original := &iop.ProviderTunnelRequest{ResponseStallTimeoutMs: value}
|
||||
encoded, err := proto.Marshal(original)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
decoded := &iop.ProviderTunnelRequest{}
|
||||
if err := proto.Unmarshal(encoded, decoded); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if decoded.GetResponseStallTimeoutMs() != value {
|
||||
t.Fatalf("ProviderTunnelRequest round trip = %d, want %d", decoded.GetResponseStallTimeoutMs(), value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRuntimeEventToProtoPreservesTypedFailure(t *testing.T) {
|
||||
t.Run("stalled failure populated with allowlisted metadata", func(t *testing.T) {
|
||||
inputMeta := map[string]string{
|
||||
"failure_code": "response_stalled",
|
||||
"provider_health": "available",
|
||||
"liveness_classification": "request_stalled",
|
||||
"idle_duration_ms": "5000",
|
||||
"run_id": "run-1",
|
||||
"attempt_id": "run-1",
|
||||
"attempt_fence": "confirmed",
|
||||
"adapter": "ollama",
|
||||
"target": "llama3",
|
||||
"health_observation_seq": "1",
|
||||
"recovery_eligible": "true",
|
||||
"secret_key": "sensitive",
|
||||
}
|
||||
event := runtime.RuntimeEvent{
|
||||
RunID: eventTypeStalledRunID(),
|
||||
Type: runtime.EventTypeError,
|
||||
Error: "provider response stalled",
|
||||
Failure: &runtime.Failure{
|
||||
Code: runtime.FailureCodeResponseStalled,
|
||||
Message: "provider response stalled",
|
||||
Retryable: true,
|
||||
Metadata: inputMeta,
|
||||
},
|
||||
}
|
||||
|
||||
wire := runEventToProto(event, "node-1", "session-1", false)
|
||||
|
||||
if wire.GetFailure() == nil {
|
||||
t.Fatal("expected non-nil wire.Failure")
|
||||
}
|
||||
if wire.GetFailure().GetCode() != "response_stalled" {
|
||||
t.Fatalf("code = %q, want response_stalled", wire.GetFailure().GetCode())
|
||||
}
|
||||
if wire.GetFailure().GetMessage() != "provider response stalled" {
|
||||
t.Fatalf("message = %q", wire.GetFailure().GetMessage())
|
||||
}
|
||||
if !wire.GetFailure().GetRetryable() {
|
||||
t.Fatal("expected retryable = true")
|
||||
}
|
||||
|
||||
meta := wire.GetFailure().GetMetadata()
|
||||
if meta["provider_health"] != "available" || meta["liveness_classification"] != "request_stalled" || meta["health_observation_seq"] != "1" {
|
||||
t.Fatalf("allowlisted metadata missing or invalid = %#v", meta)
|
||||
}
|
||||
if meta["recovery_eligible"] != "" || meta["secret_key"] != "" {
|
||||
t.Fatalf("non-allowlisted metadata present in wire failure: %#v", meta)
|
||||
}
|
||||
|
||||
// Verify defensive cloning: mutating input map must not alter wire failure metadata
|
||||
inputMeta["attempt_fence"] = "mutated"
|
||||
if meta["attempt_fence"] != "confirmed" {
|
||||
t.Fatal("wire failure metadata shared mutable alias with input metadata")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("non-stalled failure leaves wire failure nil", func(t *testing.T) {
|
||||
event := runtime.RuntimeEvent{
|
||||
RunID: "run-2",
|
||||
Type: runtime.EventTypeError,
|
||||
Error: "cancelled error",
|
||||
Failure: &runtime.Failure{Code: runtime.FailureCodeCancelled, Message: "cancelled error"},
|
||||
}
|
||||
wire := runEventToProto(event, "node-1", "session-1", false)
|
||||
if wire.GetFailure() != nil {
|
||||
t.Fatalf("expected nil wire.Failure for non-stalled code, got %#v", wire.GetFailure())
|
||||
}
|
||||
if wire.GetError() != "cancelled error" {
|
||||
t.Fatalf("error string = %q, want cancelled error", wire.GetError())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("nil failure leaves wire failure nil", func(t *testing.T) {
|
||||
event := runtime.RuntimeEvent{
|
||||
RunID: "run-3",
|
||||
Type: runtime.EventTypeComplete,
|
||||
}
|
||||
wire := runEventToProto(event, "node-1", "session-1", false)
|
||||
if wire.GetFailure() != nil {
|
||||
t.Fatalf("expected nil wire.Failure for nil failure, got %#v", wire.GetFailure())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func eventTypeStalledRunID() string { return "run-1" }
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/types/known/structpb"
|
||||
|
|
@ -21,6 +22,65 @@ type noopSender struct{}
|
|||
|
||||
func (noopSender) Send(proto.Message) error { return nil }
|
||||
|
||||
// terminalDeferringSink holds terminal events until Node-local admission has
|
||||
// released its slot. Edge uses terminal run events to advance queued work, so
|
||||
// emitting them before the local slot is free can over-dispatch back into Node.
|
||||
type terminalDeferringSink struct {
|
||||
inner runtime.EventSink
|
||||
|
||||
emitMu sync.Mutex
|
||||
mu sync.Mutex
|
||||
deferring bool
|
||||
terminalObserved bool
|
||||
deferred []runtime.RuntimeEvent
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) Emit(ctx context.Context, event runtime.RuntimeEvent) error {
|
||||
s.emitMu.Lock()
|
||||
defer s.emitMu.Unlock()
|
||||
|
||||
s.mu.Lock()
|
||||
if s.terminalObserved {
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
if runtime.IsTerminalEvent(event.Type) {
|
||||
s.terminalObserved = true
|
||||
}
|
||||
if s.deferring || runtime.IsTerminalEvent(event.Type) {
|
||||
s.deferring = true
|
||||
s.deferred = append(s.deferred, event)
|
||||
s.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
s.mu.Unlock()
|
||||
return s.inner.Emit(ctx, event)
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) Flush(ctx context.Context) error {
|
||||
s.emitMu.Lock()
|
||||
defer s.emitMu.Unlock()
|
||||
|
||||
s.mu.Lock()
|
||||
events := append([]runtime.RuntimeEvent(nil), s.deferred...)
|
||||
s.deferred = nil
|
||||
s.deferring = false
|
||||
s.mu.Unlock()
|
||||
|
||||
for _, event := range events {
|
||||
if err := s.inner.Emit(ctx, event); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *terminalDeferringSink) hasTerminalObserved() bool {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.terminalObserved
|
||||
}
|
||||
|
||||
// sessionSink wraps a transport.Session to implement runtime.EventSink.
|
||||
type sessionSink struct {
|
||||
sess protoSender
|
||||
|
|
@ -34,7 +94,6 @@ type sessionSink struct {
|
|||
|
||||
func (s *sessionSink) Emit(_ context.Context, event runtime.RuntimeEvent) error {
|
||||
s.printEvent(event)
|
||||
event.Metadata = cloneLivenessMetadata(event.Metadata)
|
||||
return s.sess.Send(runEventToProto(event, s.nodeID, s.sessionID, s.background))
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,12 +2,15 @@ package node
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"iop/apps/node/internal/transport"
|
||||
"iop/packages/go/credentiallease"
|
||||
runtime "iop/packages/go/execution"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
|
@ -21,11 +24,7 @@ func (n *Node) OnProviderTunnelRequest(ctx context.Context, sess *transport.Sess
|
|||
zap.String("target", req.GetTarget()),
|
||||
)
|
||||
|
||||
tr, err := providerTunnelRequestFromProto(req)
|
||||
if err != nil {
|
||||
n.sendTunnelError(sess, tr, fmt.Errorf("node: %w", err))
|
||||
return fmt.Errorf("node: %w", err)
|
||||
}
|
||||
tr := providerTunnelRequestFromProto(req)
|
||||
|
||||
n.configSetMu.RLock()
|
||||
configLocked := true
|
||||
|
|
@ -81,24 +80,20 @@ func (n *Node) OnProviderTunnelRequest(ctx context.Context, sess *transport.Sess
|
|||
}
|
||||
|
||||
var sender protoSender = noopSender{}
|
||||
var seq healthObservationSequencer
|
||||
nodeID := n.nodeID
|
||||
nodeAlias := ""
|
||||
if sess != nil {
|
||||
if sess.IsAlive() {
|
||||
sender = sess
|
||||
seq = sess
|
||||
}
|
||||
nodeID = sess.NodeID()
|
||||
nodeAlias = sess.Alias()
|
||||
}
|
||||
|
||||
observer := newAttemptObserver(n.watchdogClock, time.Duration(tr.ResponseStallTimeoutMS)*time.Millisecond)
|
||||
sink := &tunnelSink{
|
||||
sess: sender,
|
||||
nodeID: nodeID,
|
||||
nodeAlias: nodeAlias,
|
||||
observer: observer,
|
||||
}
|
||||
|
||||
execCtx, cancel := context.WithCancel(ctx)
|
||||
|
|
@ -116,12 +111,50 @@ func (n *Node) OnProviderTunnelRequest(ctx context.Context, sess *transport.Sess
|
|||
}
|
||||
n.runs.register(h)
|
||||
|
||||
probe := healthProbeFor(adapter, caps.AdapterName, caps.InstanceKey, tr.Target)
|
||||
|
||||
configLocked = false
|
||||
n.configSetMu.RUnlock()
|
||||
preProviderOwned = false
|
||||
return n.executeTunnelAttempt(execCtx, cancel, tunnelAdapter, tr, sink, ticket, h, material, probe, seq)
|
||||
defer func() {
|
||||
cancel()
|
||||
if tr.Credential != nil {
|
||||
tr.Credential.Zero()
|
||||
}
|
||||
if material != nil {
|
||||
material.Zero()
|
||||
}
|
||||
ticket.release()
|
||||
n.runs.deregister(tr.RunID)
|
||||
close(h.done)
|
||||
}()
|
||||
|
||||
if err := tunnelAdapter.TunnelProvider(execCtx, tr, sink); err != nil {
|
||||
n.logger.Warn("provider tunnel error", zap.String("run_id", tr.RunID), zap.String("tunnel_id", tr.TunnelID), zap.Error(err))
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *Node) consumeTunnelCredential(
|
||||
ctx context.Context,
|
||||
req *iop.ProviderTunnelRequest,
|
||||
tr *runtime.ProviderTunnelRequest,
|
||||
) (*credentiallease.Material, error) {
|
||||
if n.credentialConsumer == nil && req.GetCredentialLease() == nil && req.GetCredentialBinding() == nil {
|
||||
return nil, nil
|
||||
}
|
||||
if n.credentialConsumer == nil || req.GetCredentialLease() == nil || req.GetCredentialBinding() == nil {
|
||||
return nil, errors.New("node: credential lease is required")
|
||||
}
|
||||
envelope, err := credentiallease.FromProto(req.GetCredentialLease())
|
||||
if err != nil {
|
||||
return nil, errors.New("node: credential lease rejected")
|
||||
}
|
||||
material, err := n.credentialConsumer.Consume(ctx, envelope, credentiallease.ExpectedFromProto(req.GetCredentialBinding()))
|
||||
if err != nil {
|
||||
return nil, errors.New("node: credential lease rejected")
|
||||
}
|
||||
tr.Credential = &runtime.ProviderCredential{HeaderName: material.HeaderName, Scheme: material.Scheme, Secret: material.Secret}
|
||||
return material, nil
|
||||
}
|
||||
|
||||
func (n *Node) sendTunnelError(sess *transport.Session, tr runtime.ProviderTunnelRequest, err error) {
|
||||
|
|
@ -140,3 +173,60 @@ func (n *Node) sendTunnelError(sess *transport.Session, tr runtime.ProviderTunne
|
|||
}
|
||||
_ = sess.Send(tf)
|
||||
}
|
||||
|
||||
// tunnelSink serializes provider frames and accepts at most one terminal.
|
||||
type tunnelSink struct {
|
||||
sess protoSender
|
||||
nodeID string
|
||||
nodeAlias string
|
||||
mu sync.Mutex
|
||||
terminal bool
|
||||
}
|
||||
|
||||
func (s *tunnelSink) EmitTunnelFrame(_ context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.terminal {
|
||||
return nil
|
||||
}
|
||||
if frame.Kind == runtime.ProviderTunnelFrameKindEnd || frame.Kind == runtime.ProviderTunnelFrameKindError {
|
||||
s.terminal = true
|
||||
}
|
||||
if s.sess == nil {
|
||||
return nil
|
||||
}
|
||||
return s.sess.Send(tunnelFrameToProto(frame, s.nodeID, s.nodeAlias))
|
||||
}
|
||||
|
||||
func tunnelFrameToProto(frame runtime.ProviderTunnelFrame, nodeID, nodeAlias string) *iop.ProviderTunnelFrame {
|
||||
var usage *iop.Usage
|
||||
if frame.Usage != nil {
|
||||
usage = &iop.Usage{
|
||||
InputTokens: int32(frame.Usage.InputTokens),
|
||||
OutputTokens: int32(frame.Usage.OutputTokens),
|
||||
ReasoningTokens: int32(frame.Usage.ReasoningTokens),
|
||||
CachedInputTokens: int32(frame.Usage.CachedInputTokens),
|
||||
}
|
||||
}
|
||||
|
||||
protoKind := iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_UNSPECIFIED
|
||||
switch frame.Kind {
|
||||
case runtime.ProviderTunnelFrameKindResponseStart:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START
|
||||
case runtime.ProviderTunnelFrameKindBody:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY
|
||||
case runtime.ProviderTunnelFrameKindEnd:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END
|
||||
case runtime.ProviderTunnelFrameKindError:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR
|
||||
case runtime.ProviderTunnelFrameKindUsage:
|
||||
protoKind = iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_USAGE
|
||||
}
|
||||
|
||||
return &iop.ProviderTunnelFrame{
|
||||
RunId: frame.RunID, TunnelId: frame.TunnelID, Sequence: frame.Sequence, Kind: protoKind,
|
||||
StatusCode: int32(frame.StatusCode), Headers: frame.Headers, Body: frame.Body, End: frame.End,
|
||||
Error: frame.Error, Usage: usage, Metadata: frame.Metadata, Timestamp: frame.Timestamp.UnixNano(),
|
||||
NodeId: nodeID, NodeAlias: nodeAlias,
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -43,16 +43,15 @@ func (r *defaultRouter) resolveWithRegistry(req runtime.RunRequest, reg *runtime
|
|||
}
|
||||
|
||||
spec := runtime.ExecutionSpec{
|
||||
RunID: req.RunID,
|
||||
Adapter: adapterName,
|
||||
Target: req.Target,
|
||||
SessionID: req.SessionID,
|
||||
Background: req.Background,
|
||||
Policy: req.Policy,
|
||||
Input: req.Input,
|
||||
TimeoutSec: req.TimeoutSec,
|
||||
Metadata: req.Metadata,
|
||||
ResponseStallTimeoutMS: req.ResponseStallTimeoutMS,
|
||||
RunID: req.RunID,
|
||||
Adapter: adapterName,
|
||||
Target: req.Target,
|
||||
SessionID: req.SessionID,
|
||||
Background: req.Background,
|
||||
Policy: req.Policy,
|
||||
Input: req.Input,
|
||||
TimeoutSec: req.TimeoutSec,
|
||||
Metadata: req.Metadata,
|
||||
}
|
||||
|
||||
r.logger.Debug("resolved execution spec",
|
||||
|
|
|
|||
|
|
@ -417,7 +417,6 @@ nodes:
|
|||
health: "healthy"
|
||||
capacity: 1
|
||||
priority: 50
|
||||
# response_stall_timeout_ms: 60000 # omitted → uses documented default
|
||||
# Seulgivibe OpenAI-compatible provider examples. Keep endpoint values
|
||||
# illustrative and provide user tokens per request via openai.provider_auth.
|
||||
# - id: "seulgivibe-claude"
|
||||
|
|
|
|||
|
|
@ -72,20 +72,13 @@ nodes:
|
|||
|
||||
`models[]`를 사용할 경우 `openai.adapter`/`openai.target`은 하향 호환 fallback이다. 실제 routing은 `nodes[].providers[].id`를 기준으로 provider-pool에서 선택된다.
|
||||
|
||||
### Provider response-stall timeout ownership
|
||||
### Terminal ownership
|
||||
|
||||
dev-runtime에서 긴 응답의 정지 판정은 아래 순서를 유지한다.
|
||||
|
||||
```text
|
||||
Node provider response_stall_timeout_ms = 60000
|
||||
Node close/probe join ceiling = 5000
|
||||
Agent dispatcher silence safety = 70000
|
||||
Edge request hard timeout > 70000
|
||||
```
|
||||
긴 무출력은 실행 상태 전이나 오류가 아니다. 요청은 caller cancel, configured hard timeout, provider terminal/error, transport disconnect 중 하나가 올 때까지 유지된다. provider queue timeout과 HTTP client timeout도 각 기존 경계에서만 적용된다.
|
||||
|
||||
dev-runtime은 blocking `repeat_guard`와 함께 `max_request_fault_recovery: 1`, `max_strategy_fault_recovery: 1`을 사용하고 Pi agent retry는 비활성화한다. 따라서 반복 출력에는 Stream Evidence Gate가 요청 내부 continuation recovery를 최대 한 번 수행하고, 외부 Pi 재시도나 중첩 요청은 만들지 않는다.
|
||||
|
||||
`response_stall_timeout_ms` 변경은 restart-required다. `config check`와 `config refresh --mode dry-run`에서 이를 확인한 뒤 Edge와 Node를 같은 source ref로 rebuild/restart하고, 각 binary의 build identity와 실행 중인 process identity를 다시 대조한다. tracked 검증 근거에는 source/build/config 식별자, 단조 시간, terminal 개수와 결과 분류만 남기며 prompt, output, token, credential 원문은 기록하지 않는다.
|
||||
`config check`와 `config refresh --mode dry-run`에서 현재 설정을 확인한 뒤 Edge와 Node를 같은 source ref로 rebuild/restart하고, 각 binary의 build identity와 실행 중인 process identity를 다시 대조한다. tracked 검증 근거에는 source/build/config 식별자, 단조 시간, terminal 개수와 결과 분류만 남기며 prompt, output, token, credential 원문은 기록하지 않는다.
|
||||
|
||||
확인:
|
||||
|
||||
|
|
|
|||
|
|
@ -281,7 +281,7 @@ func rejectLegacyProviderConfig(settings map[string]any) error {
|
|||
}
|
||||
if (path == "console" && key == "agent") || key == "agent_"+"kind" || key == "workspace_"+"required" ||
|
||||
(key == "cli" && strings.Contains(path, "adapters")) ||
|
||||
(strings.Contains(path, "providers[") && isLegacyProviderProcessField(key)) {
|
||||
(strings.Contains(path, "providers[") && (isLegacyProviderProcessField(key) || key == "response_stall_timeout_ms")) {
|
||||
return fmt.Errorf("legacy provider configuration field %q is not supported", childPath)
|
||||
}
|
||||
if strings.Contains(path, "providers[") && (key == "type" || key == "category") && strings.EqualFold(fmt.Sprint(child), "cli") {
|
||||
|
|
|
|||
50
packages/go/config/provider_stall_timeout_removed_test.go
Normal file
50
packages/go/config/provider_stall_timeout_removed_test.go
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
package config_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"iop/packages/go/config"
|
||||
)
|
||||
|
||||
func TestLoadEdgeRejectsRemovedResponseStallTimeout(t *testing.T) {
|
||||
const base = `
|
||||
long_context_threshold_tokens: 1
|
||||
nodes:
|
||||
- id: n1
|
||||
alias: n1
|
||||
token: token
|
||||
providers:
|
||||
- id: provider-1
|
||||
type: ollama
|
||||
category: local_inference
|
||||
models: [model-1]
|
||||
base_url: http://127.0.0.1:11434
|
||||
capacity: 1
|
||||
`
|
||||
|
||||
dir := t.TempDir()
|
||||
removedPath := filepath.Join(dir, "removed.yaml")
|
||||
removed := base + " response_stall_timeout_ms: 60000\n"
|
||||
if err := os.WriteFile(removedPath, []byte(removed), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := config.LoadEdge(removedPath)
|
||||
if err == nil {
|
||||
t.Fatal("LoadEdge() error = nil, want removed response_stall_timeout_ms rejection")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "legacy provider configuration") ||
|
||||
!strings.Contains(err.Error(), "nodes[0].providers[0].response_stall_timeout_ms") {
|
||||
t.Fatalf("LoadEdge() error = %q, want exact removed provider field rejection", err)
|
||||
}
|
||||
|
||||
currentPath := filepath.Join(dir, "current.yaml")
|
||||
if err := os.WriteFile(currentPath, []byte(base), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := config.LoadEdge(currentPath); err != nil {
|
||||
t.Fatalf("LoadEdge() without removed field error = %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
package config_test
|
||||
|
||||
import (
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/execution"
|
||||
)
|
||||
|
||||
func TestNodeProviderResponseStallTimeoutValidation(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
raw int64
|
||||
want int64
|
||||
bad bool
|
||||
}{
|
||||
{name: "omitted defaults", want: execution.DefaultResponseStallTimeoutMS},
|
||||
{name: "positive preserved", raw: 60000, want: 60000},
|
||||
{name: "exact safe boundary preserved", raw: math.MaxInt64 / int64(time.Millisecond), want: math.MaxInt64 / int64(time.Millisecond)},
|
||||
{name: "first overflowing millisecond rejected", raw: math.MaxInt64/int64(time.Millisecond) + 1, bad: true},
|
||||
{name: "negative rejected", raw: -1, bad: true},
|
||||
{name: "overflow rejected", raw: 99999999999999, bad: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
provider := config.NodeProviderConf{ID: "p1", Type: "vllm", Category: config.CategoryAPI, Models: []string{"m"}, ResponseStallTimeoutMS: tc.raw}
|
||||
err := provider.Validate()
|
||||
if (err != nil) != tc.bad {
|
||||
t.Fatalf("Validate() error = %v, want bad=%t", err, tc.bad)
|
||||
}
|
||||
if tc.bad {
|
||||
if !strings.Contains(err.Error(), "response_stall_timeout_ms") {
|
||||
t.Fatalf("error = %q", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if got := provider.EffectiveResponseStallTimeoutMS(); got != tc.want {
|
||||
t.Errorf("effective timeout = %d, want %d", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -3,8 +3,6 @@ package config
|
|||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"iop/packages/go/execution"
|
||||
)
|
||||
|
||||
// Category represents the provider category.
|
||||
|
|
@ -97,24 +95,6 @@ type NodeProviderConf struct {
|
|||
Headers map[string]string `mapstructure:"headers" yaml:"headers,omitempty"`
|
||||
ContextSize int `mapstructure:"context_size" yaml:"context_size,omitempty"`
|
||||
RequestTimeoutMS int `mapstructure:"request_timeout_ms" yaml:"request_timeout_ms,omitempty"`
|
||||
// ResponseStallTimeoutMS is the provider-originated response-stall timeout
|
||||
// in milliseconds. It is carried on every provider-first and legacy route
|
||||
// request so the downstream watchdog has one effective value per dispatched
|
||||
// attempt. Zero is treated as the documented default (60000 ms). Negative
|
||||
// values and positive values that cannot safely become a time.Duration are
|
||||
// rejected by Validate. The effective value is returned by
|
||||
// EffectiveResponseStallTimeoutMS.
|
||||
ResponseStallTimeoutMS int64 `mapstructure:"response_stall_timeout_ms" yaml:"response_stall_timeout_ms,omitempty"`
|
||||
}
|
||||
|
||||
// EffectiveResponseStallTimeoutMS returns the effective response-stall timeout
|
||||
// in milliseconds. Validate rejects negative and overflow values at load, so
|
||||
// here only zero maps to the shared default and safe positives pass through.
|
||||
func (p NodeProviderConf) EffectiveResponseStallTimeoutMS() int64 {
|
||||
if p.ResponseStallTimeoutMS == 0 {
|
||||
return execution.DefaultResponseStallTimeoutMS
|
||||
}
|
||||
return p.ResponseStallTimeoutMS
|
||||
}
|
||||
|
||||
// Validate checks internal consistency of the provider candidate config.
|
||||
|
|
@ -161,20 +141,6 @@ func (p NodeProviderConf) Validate() error {
|
|||
if p.LongContextCapacity > 0 && p.TotalContextTokens <= 0 {
|
||||
return fmt.Errorf("nodes[].providers[%q].total_context_tokens must be positive when long_context_capacity > 0", id)
|
||||
}
|
||||
if err := validateStallTimeout(p.ResponseStallTimeoutMS, id); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateStallTimeout enforces the response_stall_timeout_ms contract:
|
||||
// zero selects the default, positive values must fit safely into a
|
||||
// time.Duration, and negative values are rejected.
|
||||
func validateStallTimeout(ms int64, id string) error {
|
||||
if err := execution.ValidateStallTimeoutMS(ms); err != nil {
|
||||
return fmt.Errorf("nodes[].providers[%q].response_stall_timeout_ms: %w", id, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,6 @@ const (
|
|||
FailureCodeUnavailable FailureCode = "unavailable"
|
||||
FailureCodeQuotaExhausted FailureCode = "quota_exhausted"
|
||||
FailureCodeProvider FailureCode = "provider_error"
|
||||
FailureCodeResponseStalled FailureCode = "response_stalled"
|
||||
FailureCodeInternal FailureCode = "internal"
|
||||
)
|
||||
|
||||
|
|
@ -122,7 +121,6 @@ func isKnownFailureCode(code FailureCode) bool {
|
|||
FailureCodeUnavailable,
|
||||
FailureCodeQuotaExhausted,
|
||||
FailureCodeProvider,
|
||||
FailureCodeResponseStalled,
|
||||
FailureCodeInternal:
|
||||
return true
|
||||
default:
|
||||
|
|
|
|||
|
|
@ -43,8 +43,8 @@ func TestFailureCodecNormalizesUnknownCode(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestFailureCodecPreservesResponseStalled(t *testing.T) {
|
||||
input := &Failure{Code: FailureCodeResponseStalled, Retryable: true, Metadata: map[string]string{"attempt_fence": "confirmed"}}
|
||||
func TestFailureCodecPreservesProviderFailure(t *testing.T) {
|
||||
input := &Failure{Code: FailureCodeProvider, Retryable: true, Metadata: map[string]string{"attempt": "one"}}
|
||||
payload, err := EncodeFailure(input)
|
||||
if err != nil {
|
||||
t.Fatalf("EncodeFailure() error = %v", err)
|
||||
|
|
@ -53,7 +53,7 @@ func TestFailureCodecPreservesResponseStalled(t *testing.T) {
|
|||
if err != nil {
|
||||
t.Fatalf("DecodeFailure() error = %v", err)
|
||||
}
|
||||
if output.Code != FailureCodeResponseStalled || !output.Retryable || output.Metadata["attempt_fence"] != "confirmed" {
|
||||
if output.Code != FailureCodeProvider || !output.Retryable || output.Metadata["attempt"] != "one" {
|
||||
t.Fatalf("round trip = %#v", output)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,296 +0,0 @@
|
|||
package execution
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultResponseStallTimeoutMS is the default response-stall timeout in
|
||||
// milliseconds. It is used when no provider-configured value is available
|
||||
// (zero wire value, omitted config, direct/legacy dispatch).
|
||||
const DefaultResponseStallTimeoutMS = 60000
|
||||
|
||||
// maxSafeStallTimeoutMS is the largest millisecond value that can safely
|
||||
// become a time.Duration without overflow. Values above this bound are
|
||||
// rejected by the config validator and treated as invalid on the wire.
|
||||
const maxSafeStallTimeoutMS = math.MaxInt64 / int64(time.Millisecond)
|
||||
|
||||
// ResolveStallTimeoutMS validates and normalizes a raw response-stall timeout
|
||||
// value in one pass. It is the single validate-then-normalize entry point used
|
||||
// by config load and both Node wire boundaries: zero maps to the documented
|
||||
// default, safe positive values pass through unchanged, and negative or
|
||||
// duration-overflow values return a StallTimeoutValidationError before any
|
||||
// router or provider invocation. It never silently converts an invalid value.
|
||||
func ResolveStallTimeoutMS(ms int64) (int64, error) {
|
||||
if err := ValidateStallTimeoutMS(ms); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if ms == 0 {
|
||||
return DefaultResponseStallTimeoutMS, nil
|
||||
}
|
||||
return ms, nil
|
||||
}
|
||||
|
||||
// ValidateStallTimeoutMS returns nil when ms is zero (use default) or a
|
||||
// positive value that can safely become a time.Duration in milliseconds.
|
||||
// Negative values and values exceeding the safe duration bound are rejected.
|
||||
// It is the single validation entry point used by config and the wire boundary.
|
||||
func ValidateStallTimeoutMS(ms int64) error {
|
||||
if ms < 0 {
|
||||
return &StallTimeoutValidationError{
|
||||
Value: ms,
|
||||
Msg: "response_stall_timeout_ms must be non-negative",
|
||||
}
|
||||
}
|
||||
if ms > maxSafeStallTimeoutMS {
|
||||
return &StallTimeoutValidationError{
|
||||
Value: ms,
|
||||
Msg: "response_stall_timeout_ms exceeds safe duration bound",
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// StallTimeoutValidationError is returned when a response_stall_timeout_ms
|
||||
// value is negative or exceeds the safe duration bound.
|
||||
type StallTimeoutValidationError struct {
|
||||
Value int64
|
||||
Msg string
|
||||
}
|
||||
|
||||
func (e *StallTimeoutValidationError) Error() string {
|
||||
if e.Msg != "" {
|
||||
return e.Msg
|
||||
}
|
||||
return "invalid response_stall_timeout_ms"
|
||||
}
|
||||
|
||||
// ProviderActivityDisposition classifies a provider output signal for the
|
||||
// watchdog. The classifier is the single source of truth for progress and
|
||||
// terminal decisions; handlers never switch on kind independently.
|
||||
type ProviderActivityDisposition string
|
||||
|
||||
const (
|
||||
// DispositionNone means the signal carries no provider progress
|
||||
// information and must not reset the watchdog timer.
|
||||
DispositionNone ProviderActivityDisposition = "none"
|
||||
// DispositionStart establishes the initial baseline for the watchdog.
|
||||
// It is emitted once per run before any progress signals and lets the
|
||||
// observer record a known starting point without conflating that
|
||||
// transition with later progress resets.
|
||||
DispositionStart ProviderActivityDisposition = "start"
|
||||
// DispositionProgress means the provider is actively making progress
|
||||
// and must reset the watchdog timer.
|
||||
DispositionProgress ProviderActivityDisposition = "progress"
|
||||
// DispositionTerminal means the provider has produced a terminal
|
||||
// signal (complete, error, cancelled, end). The watchdog must stop
|
||||
// observing this run.
|
||||
DispositionTerminal ProviderActivityDisposition = "terminal"
|
||||
)
|
||||
|
||||
// ClassifyRuntimeEvent classifies a RuntimeEvent into a ProviderActivityDisposition.
|
||||
// Terminality is decided by the event type, never by token counts.
|
||||
//
|
||||
// Rules:
|
||||
// - start → DispositionStart
|
||||
// - complete/error/cancelled → DispositionTerminal (takes precedence over any payload/usage)
|
||||
// - non-terminal delta/reasoning_delta with non-empty delta/message or a usage observation → DispositionProgress
|
||||
// - empty/unknown type → DispositionNone
|
||||
func ClassifyRuntimeEvent(event RuntimeEvent) ProviderActivityDisposition {
|
||||
switch event.Type {
|
||||
case EventTypeStart:
|
||||
return DispositionStart
|
||||
case EventTypeComplete, EventTypeError, EventTypeCancelled:
|
||||
return DispositionTerminal
|
||||
case EventTypeDelta, EventTypeReasoningDelta:
|
||||
if event.Delta != "" || event.Message != "" || event.Usage != nil {
|
||||
return DispositionProgress
|
||||
}
|
||||
return DispositionNone
|
||||
default:
|
||||
return DispositionNone
|
||||
}
|
||||
}
|
||||
|
||||
// ClassifyProviderTunnelFrame classifies a ProviderTunnelFrame into a
|
||||
// ProviderActivityDisposition.
|
||||
//
|
||||
// Rules:
|
||||
// - response_start (including headers) → DispositionProgress
|
||||
// - non-empty body → DispositionProgress
|
||||
// - usage frame → DispositionProgress
|
||||
// - end/error → DispositionTerminal (takes precedence over payload)
|
||||
// - empty/unknown kind → DispositionNone
|
||||
func ClassifyProviderTunnelFrame(frame ProviderTunnelFrame) ProviderActivityDisposition {
|
||||
switch frame.Kind {
|
||||
case ProviderTunnelFrameKindEnd, ProviderTunnelFrameKindError:
|
||||
return DispositionTerminal
|
||||
case ProviderTunnelFrameKindResponseStart:
|
||||
// response_start with or without headers is progress.
|
||||
return DispositionProgress
|
||||
case ProviderTunnelFrameKindBody:
|
||||
if len(frame.Body) > 0 {
|
||||
return DispositionProgress
|
||||
}
|
||||
return DispositionNone
|
||||
case ProviderTunnelFrameKindUsage:
|
||||
// A usage frame is always progress for the tunnel path; the watchdog
|
||||
// observes token consumption as active provider work.
|
||||
return DispositionProgress
|
||||
default:
|
||||
return DispositionNone
|
||||
}
|
||||
}
|
||||
|
||||
// ErrProbeUnsupported is carried in a ProbeOutcome when an adapter does not
|
||||
// implement active provider probing. It is one of the inconclusive outcomes
|
||||
// the probe normalizer collapses to HealthUnknown rather than treating as a
|
||||
// definitive available or exact-target-unavailable result.
|
||||
var ErrProbeUnsupported = errors.New("execution: adapter does not support provider probing")
|
||||
|
||||
// ProviderHealth is the stable, fail-closed classification of a provider's
|
||||
// health as observed by a single bounded exact-target probe. It is the only
|
||||
// value terminal assembly consumes from a probe: probe completion is evidence
|
||||
// only and must never reset original request progress, change the attempt
|
||||
// fence, or authorize retry.
|
||||
type ProviderHealth string
|
||||
|
||||
const (
|
||||
// HealthUnknown is the fail-closed default. The probe could not establish
|
||||
// a definitive available or exact-target-unavailable result. Every error,
|
||||
// timeout, unsupported adapter, unknown status, and identity mismatch maps
|
||||
// here.
|
||||
HealthUnknown ProviderHealth = "health_unknown"
|
||||
// ProviderUnhealthy means a valid probe positively reported the exact
|
||||
// target as absent.
|
||||
ProviderUnhealthy ProviderHealth = "provider_unhealthy"
|
||||
// RequestStalled means a valid probe positively reported the exact target
|
||||
// as available, corroborating that the stalled request targets a live
|
||||
// target rather than a missing endpoint.
|
||||
RequestStalled ProviderHealth = "request_stalled"
|
||||
)
|
||||
|
||||
// LivenessClassification is the stable, observable category a bounded
|
||||
// exact-target probe outcome reduces to before it becomes a ProviderHealth.
|
||||
// It exists so every fail-closed branch is independently testable; the
|
||||
// normalizer is the single mapping from classification to health.
|
||||
type LivenessClassification string
|
||||
|
||||
const (
|
||||
// LivenessAvailable means a valid probe reported the exact target present.
|
||||
LivenessAvailable LivenessClassification = "available"
|
||||
// LivenessUnavailable means a valid probe reported the exact target absent.
|
||||
LivenessUnavailable LivenessClassification = "unavailable"
|
||||
// LivenessTimeout means the bounded probe context expired before a result.
|
||||
LivenessTimeout LivenessClassification = "timeout"
|
||||
// LivenessError means the probe returned a transport, protocol, or decode
|
||||
// error that is not itself a definitive target-absent result.
|
||||
LivenessError LivenessClassification = "error"
|
||||
// LivenessUnsupported means the adapter does not implement active probing.
|
||||
LivenessUnsupported LivenessClassification = "unsupported"
|
||||
// LivenessUnknown means the probe returned an unrecognized status.
|
||||
LivenessUnknown LivenessClassification = "unknown"
|
||||
// LivenessIdentityMismatch means the probe identity did not match the
|
||||
// requested adapter or target identity.
|
||||
LivenessIdentityMismatch LivenessClassification = "identity_mismatch"
|
||||
)
|
||||
|
||||
// ProbeOutcome is the typed, target-aware input to the fail-closed probe
|
||||
// outcome normalizer. The coordinator validates and populates every field
|
||||
// from a single bounded exact-target probe attempt; the normalizer never
|
||||
// copies arbitrary provider metadata from it.
|
||||
type ProbeOutcome struct {
|
||||
// AdapterName is the adapter identity reported by the probe result.
|
||||
AdapterName string
|
||||
// InstanceKey is the stable registry instance key reported by the probe.
|
||||
InstanceKey string
|
||||
// Target is the exact target reported by the probe result.
|
||||
Target string
|
||||
// Status is the normalized provider status reported by the probe.
|
||||
Status ProviderStatus
|
||||
// Err is the inconclusive error returned by the probe, if any.
|
||||
Err error
|
||||
// ExpectedAdapter is the adapter identity the caller required.
|
||||
ExpectedAdapter string
|
||||
// ExpectedInstance is the instance key the caller required; empty means the
|
||||
// caller does not pin a specific registry instance.
|
||||
ExpectedInstance string
|
||||
// ExpectedTarget is the exact target the caller required.
|
||||
ExpectedTarget string
|
||||
}
|
||||
|
||||
// ClassifyProbeOutcome reduces a bounded exact-target probe outcome to its
|
||||
// stable liveness classification. It is pure and fail-closed: any error, probe
|
||||
// expiry, unsupported adapter, unknown status, or identity mismatch is an
|
||||
// inconclusive classification rather than a definitive one. A returned error
|
||||
// takes precedence over any reported status.
|
||||
func ClassifyProbeOutcome(outcome ProbeOutcome) LivenessClassification {
|
||||
if outcome.Err != nil {
|
||||
if errors.Is(outcome.Err, context.Canceled) || errors.Is(outcome.Err, context.DeadlineExceeded) {
|
||||
return LivenessTimeout
|
||||
}
|
||||
if errors.Is(outcome.Err, ErrProbeUnsupported) {
|
||||
return LivenessUnsupported
|
||||
}
|
||||
return LivenessError
|
||||
}
|
||||
if !probeIdentityValid(outcome) {
|
||||
return LivenessIdentityMismatch
|
||||
}
|
||||
switch outcome.Status {
|
||||
case ProviderStatusAvailable:
|
||||
return LivenessAvailable
|
||||
case ProviderStatusUnavailable:
|
||||
return LivenessUnavailable
|
||||
default:
|
||||
return LivenessUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// HealthFromClassification maps a liveness classification to its stable
|
||||
// ProviderHealth value. Available yields RequestStalled, unavailable yields
|
||||
// ProviderUnhealthy, and every inconclusive classification yields
|
||||
// HealthUnknown.
|
||||
func HealthFromClassification(classification LivenessClassification) ProviderHealth {
|
||||
switch classification {
|
||||
case LivenessAvailable:
|
||||
return RequestStalled
|
||||
case LivenessUnavailable:
|
||||
return ProviderUnhealthy
|
||||
default:
|
||||
return HealthUnknown
|
||||
}
|
||||
}
|
||||
|
||||
// NormalizeProbeOutcome maps a bounded exact-target probe outcome to its
|
||||
// stable fail-closed ProviderHealth value. It is the composition of
|
||||
// ClassifyProbeOutcome and HealthFromClassification: a validated matching
|
||||
// available result yields RequestStalled, a validated matching unavailable
|
||||
// result yields ProviderUnhealthy, and every error, timeout, unsupported
|
||||
// adapter, unknown status, and identity mismatch yields HealthUnknown. It is
|
||||
// pure and side-effect free.
|
||||
func NormalizeProbeOutcome(outcome ProbeOutcome) ProviderHealth {
|
||||
return HealthFromClassification(ClassifyProbeOutcome(outcome))
|
||||
}
|
||||
|
||||
// probeIdentityValid reports whether a probe result's adapter and target
|
||||
// identity is non-empty and exactly matches what the caller required. When the
|
||||
// caller pins an instance key, the probe must confirm it. An empty or
|
||||
// mismatched identity is inconclusive and must fail closed.
|
||||
func probeIdentityValid(outcome ProbeOutcome) bool {
|
||||
if outcome.AdapterName == "" || outcome.ExpectedAdapter == "" {
|
||||
return false
|
||||
}
|
||||
if outcome.Target == "" || outcome.ExpectedTarget == "" {
|
||||
return false
|
||||
}
|
||||
if outcome.AdapterName != outcome.ExpectedAdapter || outcome.Target != outcome.ExpectedTarget {
|
||||
return false
|
||||
}
|
||||
if outcome.ExpectedInstance != "" && outcome.InstanceKey != outcome.ExpectedInstance {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
|
@ -1,401 +0,0 @@
|
|||
package execution_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"iop/packages/go/execution"
|
||||
)
|
||||
|
||||
func TestResolveStallTimeoutMS(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
ms int64
|
||||
want int64
|
||||
wantErr bool
|
||||
}{
|
||||
{"zero maps to default", 0, execution.DefaultResponseStallTimeoutMS, false},
|
||||
{"default passes through", execution.DefaultResponseStallTimeoutMS, execution.DefaultResponseStallTimeoutMS, false},
|
||||
{"custom positive passes through", 60000, 60000, false},
|
||||
{"small positive passes through", 1, 1, false},
|
||||
{"exact safe boundary passes through", math.MaxInt64 / int64(time.Millisecond), math.MaxInt64 / int64(time.Millisecond), false},
|
||||
{"first overflowing millisecond rejected", math.MaxInt64/int64(time.Millisecond) + 1, 0, true},
|
||||
{"negative rejected", -1, 0, true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got, err := execution.ResolveStallTimeoutMS(tc.ms)
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Fatalf("ResolveStallTimeoutMS(%d) error = %v, want error=%t", tc.ms, err, tc.wantErr)
|
||||
}
|
||||
if !tc.wantErr && got != tc.want {
|
||||
t.Errorf("ResolveStallTimeoutMS(%d) = %d, want %d", tc.ms, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStallTimeoutMS(t *testing.T) {
|
||||
if err := execution.ValidateStallTimeoutMS(300000); err != nil {
|
||||
t.Errorf("expected nil for 300000, got %v", err)
|
||||
}
|
||||
if err := execution.ValidateStallTimeoutMS(1); err != nil {
|
||||
t.Errorf("expected nil for 1, got %v", err)
|
||||
}
|
||||
if err := execution.ValidateStallTimeoutMS(0); err != nil {
|
||||
t.Errorf("expected nil for 0 (use default), got %v", err)
|
||||
}
|
||||
if err := execution.ValidateStallTimeoutMS(-1); err == nil {
|
||||
t.Error("expected error for -1")
|
||||
}
|
||||
if err := execution.ValidateStallTimeoutMS(math.MaxInt64/int64(time.Millisecond) + 1); err == nil {
|
||||
t.Error("expected error for overflow value")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStallTimeoutValidationError(t *testing.T) {
|
||||
e := &execution.StallTimeoutValidationError{Value: -1, Msg: "must be positive"}
|
||||
if e.Error() != "must be positive" {
|
||||
t.Errorf("Error() = %q, want 'must be positive'", e.Error())
|
||||
}
|
||||
e2 := &execution.StallTimeoutValidationError{Value: 0}
|
||||
if e2.Error() != "invalid response_stall_timeout_ms" {
|
||||
t.Errorf("Error() = %q, want 'invalid response_stall_timeout_ms'", e2.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyRuntimeEvent(t *testing.T) {
|
||||
now := time.Now()
|
||||
cases := []struct {
|
||||
name string
|
||||
ev execution.RuntimeEvent
|
||||
want execution.ProviderActivityDisposition
|
||||
}{
|
||||
{
|
||||
name: "start event",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeStart, RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionStart,
|
||||
},
|
||||
{
|
||||
name: "delta with text",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeDelta, Delta: "hello", RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "delta with message",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeDelta, Message: "hi", RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "reasoning_delta with text",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeReasoningDelta, Delta: "thinking...", RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{name: "delta with zero usage", ev: execution.RuntimeEvent{Type: execution.EventTypeDelta, Usage: &execution.UsageStats{}, RunID: "r1", Timestamp: now}, want: execution.DispositionProgress},
|
||||
{name: "delta with token usage", ev: execution.RuntimeEvent{Type: execution.EventTypeDelta, Usage: &execution.UsageStats{OutputTokens: 1}, RunID: "r1", Timestamp: now}, want: execution.DispositionProgress},
|
||||
{name: "reasoning delta with token usage", ev: execution.RuntimeEvent{Type: execution.EventTypeReasoningDelta, Usage: &execution.UsageStats{ReasoningTokens: 1}, RunID: "r1", Timestamp: now}, want: execution.DispositionProgress},
|
||||
{name: "delta empty no usage", ev: execution.RuntimeEvent{Type: execution.EventTypeDelta, RunID: "r1", Timestamp: now}, want: execution.DispositionNone},
|
||||
{
|
||||
name: "complete with usage",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeComplete, Usage: &execution.UsageStats{OutputTokens: 10}, RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
{
|
||||
name: "complete without usage",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeComplete, RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
{name: "error with payload and usage", ev: execution.RuntimeEvent{Type: execution.EventTypeError, Delta: "last", Error: "boom", Usage: &execution.UsageStats{OutputTokens: 1}, RunID: "r1", Timestamp: now}, want: execution.DispositionTerminal},
|
||||
{name: "cancelled with payload and usage", ev: execution.RuntimeEvent{Type: execution.EventTypeCancelled, Message: "last", Usage: &execution.UsageStats{InputTokens: 1}, RunID: "r1", Timestamp: now}, want: execution.DispositionTerminal},
|
||||
{
|
||||
name: "unknown type",
|
||||
ev: execution.RuntimeEvent{Type: "unknown", RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionNone,
|
||||
},
|
||||
{
|
||||
name: "delta with terminal usage takes terminal",
|
||||
ev: execution.RuntimeEvent{Type: execution.EventTypeComplete, Delta: "last", Usage: &execution.UsageStats{OutputTokens: 5}, RunID: "r1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := execution.ClassifyRuntimeEvent(tc.ev)
|
||||
if got != tc.want {
|
||||
t.Errorf("ClassifyRuntimeEvent: got %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyProviderTunnelFrame(t *testing.T) {
|
||||
now := time.Now()
|
||||
cases := []struct {
|
||||
name string
|
||||
f execution.ProviderTunnelFrame
|
||||
want execution.ProviderActivityDisposition
|
||||
}{
|
||||
{
|
||||
name: "response_start",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindResponseStart, RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "response_start with headers",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindResponseStart, Headers: map[string]string{"content-type": "text/event-stream"}, RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "body with data",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindBody, Body: []byte("hello"), RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "body empty",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindBody, RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionNone,
|
||||
},
|
||||
{
|
||||
name: "end",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindEnd, RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
{
|
||||
name: "error",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindError, Error: "provider timeout", RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
{
|
||||
name: "usage with tokens",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindUsage, Usage: &execution.UsageStats{OutputTokens: 10}, RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionProgress,
|
||||
},
|
||||
{
|
||||
name: "unknown kind",
|
||||
f: execution.ProviderTunnelFrame{Kind: "bogus", RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionNone,
|
||||
},
|
||||
{
|
||||
name: "end with body takes terminal",
|
||||
f: execution.ProviderTunnelFrame{Kind: execution.ProviderTunnelFrameKindEnd, Body: []byte("final"), RunID: "r1", TunnelID: "t1", Timestamp: now},
|
||||
want: execution.DispositionTerminal,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
got := execution.ClassifyProviderTunnelFrame(tc.f)
|
||||
if got != tc.want {
|
||||
t.Errorf("ClassifyProviderTunnelFrame: got %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// assertProbeOutcome checks both the liveness classification and the normalized
|
||||
// health for a probe outcome, keeping the table-driven probe tests compact.
|
||||
func assertProbeOutcome(t *testing.T, outcome execution.ProbeOutcome, wantClass execution.LivenessClassification, wantHealth execution.ProviderHealth) {
|
||||
t.Helper()
|
||||
if gotClass := execution.ClassifyProbeOutcome(outcome); gotClass != wantClass {
|
||||
t.Errorf("ClassifyProbeOutcome: got %q, want %q", gotClass, wantClass)
|
||||
}
|
||||
if gotHealth := execution.NormalizeProbeOutcome(outcome); gotHealth != wantHealth {
|
||||
t.Errorf("NormalizeProbeOutcome: got %q, want %q", gotHealth, wantHealth)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyProbeOutcomeDefinitive(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
outcome execution.ProbeOutcome
|
||||
wantClass execution.LivenessClassification
|
||||
wantHealth execution.ProviderHealth
|
||||
}{
|
||||
{
|
||||
name: "matching available",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Status: execution.ProviderStatusAvailable,
|
||||
},
|
||||
wantClass: execution.LivenessAvailable, wantHealth: execution.RequestStalled,
|
||||
},
|
||||
{
|
||||
name: "matching unavailable",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "ollama", ExpectedAdapter: "ollama",
|
||||
Target: "m-b", ExpectedTarget: "m-b",
|
||||
Status: execution.ProviderStatusUnavailable,
|
||||
},
|
||||
wantClass: execution.LivenessUnavailable, wantHealth: execution.ProviderUnhealthy,
|
||||
},
|
||||
{
|
||||
name: "matching available with pinned instance",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
InstanceKey: "vllm-gpu", ExpectedInstance: "vllm-gpu",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Status: execution.ProviderStatusAvailable,
|
||||
},
|
||||
wantClass: execution.LivenessAvailable, wantHealth: execution.RequestStalled,
|
||||
},
|
||||
{
|
||||
name: "pinned instance mismatch stays inconclusive",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
InstanceKey: "vllm-gpu", ExpectedInstance: "vllm-other",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Status: execution.ProviderStatusAvailable,
|
||||
},
|
||||
wantClass: execution.LivenessIdentityMismatch, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
assertProbeOutcome(t, tc.outcome, tc.wantClass, tc.wantHealth)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyProbeOutcomeInconclusive(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
outcome execution.ProbeOutcome
|
||||
wantClass execution.LivenessClassification
|
||||
wantHealth execution.ProviderHealth
|
||||
}{
|
||||
{
|
||||
name: "transport error takes precedence over available status",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Status: execution.ProviderStatusAvailable, Err: errors.New("boom"),
|
||||
},
|
||||
wantClass: execution.LivenessError, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
{
|
||||
name: "deadline exceeded is timeout",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Err: context.DeadlineExceeded,
|
||||
},
|
||||
wantClass: execution.LivenessTimeout, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
{
|
||||
name: "cancellation is timeout",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Err: context.Canceled,
|
||||
},
|
||||
wantClass: execution.LivenessTimeout, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
{
|
||||
name: "unsupported adapter",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "worker", ExpectedAdapter: "worker",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Err: execution.ErrProbeUnsupported,
|
||||
},
|
||||
wantClass: execution.LivenessUnsupported, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
{
|
||||
name: "wrapped unsupported is still unsupported",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "worker", ExpectedAdapter: "worker",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Err: fmt.Errorf("resolve: %w", execution.ErrProbeUnsupported),
|
||||
},
|
||||
wantClass: execution.LivenessUnsupported, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
{
|
||||
name: "unknown status",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a",
|
||||
Status: execution.ProviderStatusUnknown,
|
||||
},
|
||||
wantClass: execution.LivenessUnknown, wantHealth: execution.HealthUnknown,
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
assertProbeOutcome(t, tc.outcome, tc.wantClass, tc.wantHealth)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyProbeOutcomeIdentity(t *testing.T) {
|
||||
avail := execution.ProviderStatusAvailable
|
||||
mismatch := execution.LivenessIdentityMismatch
|
||||
cases := []struct {
|
||||
name string
|
||||
outcome execution.ProbeOutcome
|
||||
}{
|
||||
{
|
||||
name: "empty adapter identity",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a", Status: avail,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "empty expected adapter",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "",
|
||||
Target: "m-a", ExpectedTarget: "m-a", Status: avail,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "empty target identity",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "", ExpectedTarget: "m-a", Status: avail,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "mismatched adapter",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "ollama", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-a", Status: avail,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "mismatched target",
|
||||
outcome: execution.ProbeOutcome{
|
||||
AdapterName: "vllm", ExpectedAdapter: "vllm",
|
||||
Target: "m-a", ExpectedTarget: "m-b", Status: avail,
|
||||
},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
assertProbeOutcome(t, tc.outcome, mismatch, execution.HealthUnknown)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHealthFromClassification(t *testing.T) {
|
||||
cases := []struct {
|
||||
class execution.LivenessClassification
|
||||
want execution.ProviderHealth
|
||||
}{
|
||||
{execution.LivenessAvailable, execution.RequestStalled},
|
||||
{execution.LivenessUnavailable, execution.ProviderUnhealthy},
|
||||
{execution.LivenessTimeout, execution.HealthUnknown},
|
||||
{execution.LivenessError, execution.HealthUnknown},
|
||||
{execution.LivenessUnsupported, execution.HealthUnknown},
|
||||
{execution.LivenessUnknown, execution.HealthUnknown},
|
||||
{execution.LivenessIdentityMismatch, execution.HealthUnknown},
|
||||
{execution.LivenessClassification("bogus"), execution.HealthUnknown},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := execution.HealthFromClassification(tc.class); got != tc.want {
|
||||
t.Errorf("HealthFromClassification(%q): got %q, want %q", tc.class, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -16,16 +16,15 @@ const DefaultSessionID = "default"
|
|||
var ErrRunCancelled = errors.New("run cancelled")
|
||||
|
||||
type ExecutionSpec struct {
|
||||
RunID string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
Policy map[string]any
|
||||
Input map[string]any
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
ResponseStallTimeoutMS int64
|
||||
RunID string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
Policy map[string]any
|
||||
Input map[string]any
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
type EventType string
|
||||
|
|
@ -95,16 +94,15 @@ type Capabilities struct {
|
|||
|
||||
// RunRequest is the host-neutral representation of an incoming run request.
|
||||
type RunRequest struct {
|
||||
RunID string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
Policy map[string]any
|
||||
Input map[string]any
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
ResponseStallTimeoutMS int64
|
||||
RunID string
|
||||
Adapter string
|
||||
Target string
|
||||
SessionID string
|
||||
Background bool
|
||||
Policy map[string]any
|
||||
Input map[string]any
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
}
|
||||
|
||||
type CommandType string
|
||||
|
|
@ -195,15 +193,14 @@ type ProviderTunnelRequest struct {
|
|||
// "messages", "models"). When set, the Node adapter resolves the request
|
||||
// URL from the concrete profile's operation path. When empty, the legacy
|
||||
// Path field is used as a mixed-version fallback.
|
||||
Operation string
|
||||
Headers map[string]string
|
||||
Body []byte
|
||||
Stream bool
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
SessionID string
|
||||
Credential *ProviderCredential
|
||||
ResponseStallTimeoutMS int64
|
||||
Operation string
|
||||
Headers map[string]string
|
||||
Body []byte
|
||||
Stream bool
|
||||
TimeoutSec int
|
||||
Metadata map[string]string
|
||||
SessionID string
|
||||
Credential *ProviderCredential
|
||||
}
|
||||
|
||||
// ProviderCredential is request-local plaintext owned by the Node adapter.
|
||||
|
|
|
|||
|
|
@ -472,23 +472,18 @@ func (NodeConfigRefreshStatus) EnumDescriptor() ([]byte, []int) {
|
|||
|
||||
// RunRequest initiates an adapter execution on a node.
|
||||
type RunRequest struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
RunId string `protobuf:"bytes,1,opt,name=run_id,json=runId,proto3" json:"run_id,omitempty"`
|
||||
Adapter string `protobuf:"bytes,2,opt,name=adapter,proto3" json:"adapter,omitempty"`
|
||||
Target string `protobuf:"bytes,3,opt,name=target,proto3" json:"target,omitempty"`
|
||||
Policy *structpb.Struct `protobuf:"bytes,5,opt,name=policy,proto3" json:"policy,omitempty"`
|
||||
Input *structpb.Struct `protobuf:"bytes,6,opt,name=input,proto3" json:"input,omitempty"`
|
||||
TimeoutSec int32 `protobuf:"varint,7,opt,name=timeout_sec,json=timeoutSec,proto3" json:"timeout_sec,omitempty"`
|
||||
Metadata map[string]string `protobuf:"bytes,8,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
SessionId string `protobuf:"bytes,9,opt,name=session_id,json=sessionId,proto3" json:"session_id,omitempty"`
|
||||
Background bool `protobuf:"varint,11,opt,name=background,proto3" json:"background,omitempty"`
|
||||
// response_stall_timeout_ms is the selected provider's response-stall
|
||||
// timeout in milliseconds. Zero means the Node applies the documented
|
||||
// default (300000). Negative or overflow values are rejected at the Node
|
||||
// boundary before router/provider invocation.
|
||||
ResponseStallTimeoutMs int64 `protobuf:"varint,12,opt,name=response_stall_timeout_ms,json=responseStallTimeoutMs,proto3" json:"response_stall_timeout_ms,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
RunId string `protobuf:"bytes,1,opt,name=run_id,json=runId,proto3" json:"run_id,omitempty"`
|
||||
Adapter string `protobuf:"bytes,2,opt,name=adapter,proto3" json:"adapter,omitempty"`
|
||||
Target string `protobuf:"bytes,3,opt,name=target,proto3" json:"target,omitempty"`
|
||||
Policy *structpb.Struct `protobuf:"bytes,5,opt,name=policy,proto3" json:"policy,omitempty"`
|
||||
Input *structpb.Struct `protobuf:"bytes,6,opt,name=input,proto3" json:"input,omitempty"`
|
||||
TimeoutSec int32 `protobuf:"varint,7,opt,name=timeout_sec,json=timeoutSec,proto3" json:"timeout_sec,omitempty"`
|
||||
Metadata map[string]string `protobuf:"bytes,8,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"`
|
||||
SessionId string `protobuf:"bytes,9,opt,name=session_id,json=sessionId,proto3" json:"session_id,omitempty"`
|
||||
Background bool `protobuf:"varint,11,opt,name=background,proto3" json:"background,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *RunRequest) Reset() {
|
||||
|
|
@ -584,13 +579,6 @@ func (x *RunRequest) GetBackground() bool {
|
|||
return false
|
||||
}
|
||||
|
||||
func (x *RunRequest) GetResponseStallTimeoutMs() int64 {
|
||||
if x != nil {
|
||||
return x.ResponseStallTimeoutMs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// RunEvent is a streaming execution event.
|
||||
type RunEvent struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
|
|
@ -762,13 +750,8 @@ type ProviderTunnelRequest struct {
|
|||
// credential_binding is the independently resolved Edge dispatch binding
|
||||
// the Node compares byte-for-byte with the signed lease before consumption.
|
||||
CredentialBinding *CredentialLeaseBinding `protobuf:"bytes,15,opt,name=credential_binding,json=credentialBinding,proto3" json:"credential_binding,omitempty"`
|
||||
// response_stall_timeout_ms is the selected provider's response-stall
|
||||
// timeout in milliseconds. Zero means the Node applies the documented
|
||||
// default (300000). Negative or overflow values are rejected at the Node
|
||||
// boundary before router/provider invocation.
|
||||
ResponseStallTimeoutMs int64 `protobuf:"varint,16,opt,name=response_stall_timeout_ms,json=responseStallTimeoutMs,proto3" json:"response_stall_timeout_ms,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *ProviderTunnelRequest) Reset() {
|
||||
|
|
@ -906,13 +889,6 @@ func (x *ProviderTunnelRequest) GetCredentialBinding() *CredentialLeaseBinding {
|
|||
return nil
|
||||
}
|
||||
|
||||
func (x *ProviderTunnelRequest) GetResponseStallTimeoutMs() int64 {
|
||||
if x != nil {
|
||||
return x.ResponseStallTimeoutMs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type CredentialLeaseScope struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
LeaseId string `protobuf:"bytes,1,opt,name=lease_id,json=leaseId,proto3" json:"lease_id,omitempty"`
|
||||
|
|
@ -4557,7 +4533,7 @@ var File_proto_iop_runtime_proto protoreflect.FileDescriptor
|
|||
|
||||
const file_proto_iop_runtime_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x17proto/iop/runtime.proto\x12\x03iop\x1a\x1cgoogle/protobuf/struct.proto\"\xed\x03\n" +
|
||||
"\x17proto/iop/runtime.proto\x12\x03iop\x1a\x1cgoogle/protobuf/struct.proto\"\xd3\x03\n" +
|
||||
"\n" +
|
||||
"RunRequest\x12\x15\n" +
|
||||
"\x06run_id\x18\x01 \x01(\tR\x05runId\x12\x18\n" +
|
||||
|
|
@ -4572,12 +4548,11 @@ const file_proto_iop_runtime_proto_rawDesc = "" +
|
|||
"session_id\x18\t \x01(\tR\tsessionId\x12\x1e\n" +
|
||||
"\n" +
|
||||
"background\x18\v \x01(\bR\n" +
|
||||
"background\x129\n" +
|
||||
"\x19response_stall_timeout_ms\x18\f \x01(\x03R\x16responseStallTimeoutMs\x1a;\n" +
|
||||
"background\x1a;\n" +
|
||||
"\rMetadataEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01J\x04\b\x04\x10\x05J\x04\b\n" +
|
||||
"\x10\vR\tworkspaceR\fsession_mode\"\xd9\x03\n" +
|
||||
"\x10\vJ\x04\b\f\x10\rR\tworkspaceR\fsession_modeR\x19response_stall_timeout_ms\"\xd9\x03\n" +
|
||||
"\bRunEvent\x12\x15\n" +
|
||||
"\x06run_id\x18\x01 \x01(\tR\x05runId\x12\x12\n" +
|
||||
"\x04type\x18\x02 \x01(\tR\x04type\x12\x14\n" +
|
||||
|
|
@ -4600,7 +4575,7 @@ const file_proto_iop_runtime_proto_rawDesc = "" +
|
|||
"\afailure\x18\r \x01(\v2\x15.iop.ExecutionFailureR\afailure\x1a;\n" +
|
||||
"\rMetadataEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x83\x06\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xe9\x05\n" +
|
||||
"\x15ProviderTunnelRequest\x12\x15\n" +
|
||||
"\x06run_id\x18\x01 \x01(\tR\x05runId\x12\x1b\n" +
|
||||
"\ttunnel_id\x18\x02 \x01(\tR\btunnelId\x12\x18\n" +
|
||||
|
|
@ -4619,14 +4594,13 @@ const file_proto_iop_runtime_proto_rawDesc = "" +
|
|||
"session_id\x18\f \x01(\tR\tsessionId\x12\x1c\n" +
|
||||
"\toperation\x18\r \x01(\tR\toperation\x12E\n" +
|
||||
"\x10credential_lease\x18\x0e \x01(\v2\x1a.iop.SignedCredentialLeaseR\x0fcredentialLease\x12J\n" +
|
||||
"\x12credential_binding\x18\x0f \x01(\v2\x1b.iop.CredentialLeaseBindingR\x11credentialBinding\x129\n" +
|
||||
"\x19response_stall_timeout_ms\x18\x10 \x01(\x03R\x16responseStallTimeoutMs\x1a:\n" +
|
||||
"\x12credential_binding\x18\x0f \x01(\v2\x1b.iop.CredentialLeaseBindingR\x11credentialBinding\x1a:\n" +
|
||||
"\fHeadersEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\x1a;\n" +
|
||||
"\rMetadataEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xd2\x04\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01J\x04\b\x10\x10\x11R\x19response_stall_timeout_ms\"\xd2\x04\n" +
|
||||
"\x14CredentialLeaseScope\x12\x19\n" +
|
||||
"\blease_id\x18\x01 \x01(\tR\aleaseId\x12#\n" +
|
||||
"\rprincipal_ref\x18\x02 \x01(\tR\fprincipalRef\x12.\n" +
|
||||
|
|
|
|||
98
proto/gen/iop/runtime_reservation_test.go
Normal file
98
proto/gen/iop/runtime_reservation_test.go
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
package iop
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"google.golang.org/protobuf/encoding/protowire"
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/reflect/protoreflect"
|
||||
)
|
||||
|
||||
func TestRuntimeRemovedStallFieldsRemainReserved(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
desc protoreflect.MessageDescriptor
|
||||
number protoreflect.FieldNumber
|
||||
}{
|
||||
{name: "RunRequest", desc: (&RunRequest{}).ProtoReflect().Descriptor(), number: 12},
|
||||
{name: "ProviderTunnelRequest", desc: (&ProviderTunnelRequest{}).ProtoReflect().Descriptor(), number: 16},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if !tt.desc.ReservedRanges().Has(tt.number) {
|
||||
t.Fatalf("field %d is not reserved", tt.number)
|
||||
}
|
||||
if tt.desc.Fields().ByNumber(tt.number) != nil {
|
||||
t.Fatalf("field %d has been reused", tt.number)
|
||||
}
|
||||
if !hasReservedName(tt.desc, "response_stall_timeout_ms") {
|
||||
t.Fatal("removed field name response_stall_timeout_ms is not reserved")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRuntimeRemovedStallFieldsRemainUnknown(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
number protowire.Number
|
||||
newMsg func() proto.Message
|
||||
}{
|
||||
{name: "RunRequest", number: 12, newMsg: func() proto.Message { return &RunRequest{} }},
|
||||
{name: "ProviderTunnelRequest", number: 16, newMsg: func() proto.Message { return &ProviderTunnelRequest{} }},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
wire := protowire.AppendTag(nil, tt.number, protowire.VarintType)
|
||||
wire = protowire.AppendVarint(wire, 60000)
|
||||
|
||||
first := tt.newMsg()
|
||||
if err := proto.Unmarshal(wire, first); err != nil {
|
||||
t.Fatalf("Unmarshal() error = %v", err)
|
||||
}
|
||||
assertUnknownVarint(t, first.ProtoReflect().GetUnknown(), tt.number, 60000)
|
||||
|
||||
roundTrip, err := proto.Marshal(first)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal() error = %v", err)
|
||||
}
|
||||
second := tt.newMsg()
|
||||
if err := proto.Unmarshal(roundTrip, second); err != nil {
|
||||
t.Fatalf("round-trip Unmarshal() error = %v", err)
|
||||
}
|
||||
assertUnknownVarint(t, second.ProtoReflect().GetUnknown(), tt.number, 60000)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func hasReservedName(desc protoreflect.MessageDescriptor, want protoreflect.Name) bool {
|
||||
names := desc.ReservedNames()
|
||||
for i := 0; i < names.Len(); i++ {
|
||||
if names.Get(i) == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func assertUnknownVarint(t *testing.T, wire []byte, wantNumber protowire.Number, wantValue uint64) {
|
||||
t.Helper()
|
||||
for len(wire) > 0 {
|
||||
number, wireType, tagLen := protowire.ConsumeTag(wire)
|
||||
if tagLen < 0 {
|
||||
t.Fatalf("ConsumeTag() error = %v", protowire.ParseError(tagLen))
|
||||
}
|
||||
wire = wire[tagLen:]
|
||||
value, valueLen := protowire.ConsumeVarint(wire)
|
||||
if wireType != protowire.VarintType || valueLen < 0 {
|
||||
t.Fatalf("unknown field %d has invalid wire type/value", number)
|
||||
}
|
||||
if number == wantNumber && value == wantValue {
|
||||
return
|
||||
}
|
||||
wire = wire[valueLen:]
|
||||
}
|
||||
t.Fatalf("unknown varint field %d=%d was not preserved", wantNumber, wantValue)
|
||||
}
|
||||
|
|
@ -8,8 +8,10 @@ option go_package = "iop/proto/gen/iop";
|
|||
|
||||
// RunRequest initiates an adapter execution on a node.
|
||||
message RunRequest {
|
||||
reserved 4, 10;
|
||||
// 12 was response_stall_timeout_ms, removed with the stall watchdog.
|
||||
reserved 4, 10, 12;
|
||||
reserved "workspace", "session_mode";
|
||||
reserved "response_stall_timeout_ms";
|
||||
string run_id = 1;
|
||||
string adapter = 2;
|
||||
string target = 3;
|
||||
|
|
@ -19,11 +21,6 @@ message RunRequest {
|
|||
map<string, string> metadata = 8;
|
||||
string session_id = 9;
|
||||
bool background = 11;
|
||||
// response_stall_timeout_ms is the selected provider's response-stall
|
||||
// timeout in milliseconds. Zero means the Node applies the documented
|
||||
// default (300000). Negative or overflow values are rejected at the Node
|
||||
// boundary before router/provider invocation.
|
||||
int64 response_stall_timeout_ms = 12;
|
||||
}
|
||||
|
||||
// RunEvent is a streaming execution event.
|
||||
|
|
@ -81,11 +78,9 @@ message ProviderTunnelRequest {
|
|||
// credential_binding is the independently resolved Edge dispatch binding
|
||||
// the Node compares byte-for-byte with the signed lease before consumption.
|
||||
CredentialLeaseBinding credential_binding = 15;
|
||||
// response_stall_timeout_ms is the selected provider's response-stall
|
||||
// timeout in milliseconds. Zero means the Node applies the documented
|
||||
// default (300000). Negative or overflow values are rejected at the Node
|
||||
// boundary before router/provider invocation.
|
||||
int64 response_stall_timeout_ms = 16;
|
||||
// 16 was response_stall_timeout_ms, removed with the stall watchdog.
|
||||
reserved 16;
|
||||
reserved "response_stall_timeout_ms";
|
||||
}
|
||||
|
||||
message CredentialLeaseScope {
|
||||
|
|
|
|||
|
|
@ -303,7 +303,6 @@ $(cat "$PROVIDER_HEADER_BLOCK")
|
|||
- "lemonade-profile-alias"
|
||||
health: available
|
||||
capacity: 4
|
||||
response_stall_timeout_ms: 200
|
||||
EOF
|
||||
|
||||
cat > "$NODE_CONFIG" <<EOF
|
||||
|
|
@ -442,7 +441,7 @@ FAILURE_MATCHES="$TMP_DIR/failure-matches.txt"
|
|||
grep -i -E "node reported error|error run_id=|\[[^]]+-evt\] error|panic:" "$EDGE_OUT" "$NODE_OUT" > "$FAILURE_MATCHES" || true
|
||||
if [ "$MODE" = "fake" ]; then
|
||||
UNEXPECTED_FAILURES="$TMP_DIR/unexpected-failures.txt"
|
||||
grep -vi -E "response_stalled|provider response stalled" "$FAILURE_MATCHES" > "$UNEXPECTED_FAILURES" || true
|
||||
cp "$FAILURE_MATCHES" "$UNEXPECTED_FAILURES"
|
||||
else
|
||||
UNEXPECTED_FAILURES="$FAILURE_MATCHES"
|
||||
fi
|
||||
|
|
@ -453,7 +452,6 @@ if [ -s "$UNEXPECTED_FAILURES" ]; then
|
|||
fi
|
||||
|
||||
if [ "$MODE" = "fake" ]; then
|
||||
grep -qi -E "response_stalled|provider response stalled" "$EDGE_OUT" "$NODE_OUT"
|
||||
kill "$LEMONADE_PID"
|
||||
wait "$LEMONADE_PID" 2>/dev/null || true
|
||||
if kill -0 "$LEMONADE_PID" 2>/dev/null; then
|
||||
|
|
|
|||
Loading…
Reference in a new issue