iop/agent-contract/inner/execution-runtime.md
toki fef1f7a9dc feat(liveness): provider 실행 stall 관측을 구현한다
Node의 provider progress 기반 stall timeout, watchdog fencing과 bounded health probe evidence를 실행 경로에 반영한다. Edge-Node 계약과 구현 스펙, 테스트 및 Milestone 완료 evidence를 현재 상태와 맞춘다.
2026-08-05 09:45:14 +09:00

7.2 KiB

Provider Execution Runtime Contract

Contract metadata

  • id: iop.execution-runtime
  • boundary: inner
  • status: active
  • source evidence:
    • packages/go/execution/types.go
    • packages/go/execution/liveness.go
    • packages/go/execution/registry.go
    • packages/go/execution/emitter.go
    • packages/go/execution/failure.go
    • apps/node/internal/node/runtime_bridge.go
    • apps/node/internal/node/health_probe.go
    • apps/node/internal/node/liveness_watchdog.go
    • apps/node/internal/transport/session.go

Scope

The execution package defines host-neutral provider primitives. It owns provider registration, lifecycle, execution events, typed failures, cancellation, token usage, and the three provider commands capabilities, transport_status, and ollama_api.

session_id is an opaque correlation value. It does not select, create, resume, or terminate a process. Repeated requests with the same value are independent executions. Cancellation targets a non-empty run_id only.

Requirements

  • Providers implement the narrow Provider interface and may expose optional lifecycle, command, or tunnel capabilities.
  • Event emitters preserve order and publish exactly one terminal event.
  • Registry lookup uses provider identity and returns typed failures for missing or unavailable providers.
  • Callers must reject commands outside the closed provider-command allowlist before provider lookup.
  • Token usage remains observation data attached to execution or tunnel results.
  • DefaultResponseStallTimeoutMS = 300000 is the documented default. ResolveStallTimeoutMS(ms) validates then maps zero to the default; safe positive values pass through, while negative or overflow values return an error.
  • ClassifyRuntimeEvent returns start for EventTypeStart, progress for non-empty delta/message or non-terminal usage, terminal for complete/error/cancelled (before usage check), and none for empty/unknown events.
  • ClassifyProviderTunnelFrame returns progress for response_start (with or without headers) and non-empty body, terminal for end/error (before payload check), progress for usage, and none for empty/unknown frames.
  • ValidateStallTimeoutMS(ms) rejects negative values and values exceeding maxSafeStallTimeoutMS; zero is allowed (use default).
  • NodeProviderConf.EffectiveResponseStallTimeoutMS() returns the effective timeout for a provider candidate.
  • RunRequest.ResponseStallTimeoutMS and ProviderTunnelRequest.ResponseStallTimeoutMS carry the selected provider's effective timeout; zero on the wire means the Node applies the documented default.
  • The Node wire boundary normalizes zero to 300000 and rejects negative or overflow values before router/provider invocation.
  • response_stalled is a stable typed failure. Its allowlisted metadata includes the failure code, the joined three-way exact-target health evidence (Edge-visible provider_health status and normalized liveness_classification), idle duration, Node-owned run/attempt identity, the local close fence, adapter, target, and an optional connection-scoped health_observation_seq; caller metadata cannot override these values, and no raw payload, credential, or recovery signal is admitted.
  • The Node watchdog starts from attempt admission, resets only on the documented progress dispositions, stops on provider terminal, and emits one typed stall terminal. It does not retry providers or infer recovery eligibility. Retryable=true means only that the local provider ownership fence was confirmed within the bounded close grace.
  • After the watchdog claims a stall it joins two independent bounded outcomes without extending either serially — the fixed close-grace fence and the exact-target health probe — then assembles exactly one allowlisted terminal. The joined liveness_classification/provider_health pair is exactly request_stalled/available, provider_unhealthy/unavailable, or health_unknown/unknown (fail-closed default). Provider availability observed here is evidence only: it never resets progress, changes the fence, revives output, or authorizes retry, and late provider output stays fenced.
  • health_observation_seq is a connection-scoped monotonic sequence sourced from the transport Session. A new connection starts at zero, so the first finalized observation is one; normalized and tunnel observations on the same connection share the source and receive unique, increasing values under concurrency. Internal or unbound execution paths omit the key entirely and never encode a process-global generation.

Health probe contract

The execution package owns the stable, fail-closed probe outcome vocabulary consumed by Node terminal assembly. It is the typed three-way boundary between an inconclusive probe and a definitive provider-health classification; nothing else maps provider probe results to health.

  • ProviderHealth is the stable normalized value: request_stalled, provider_unhealthy, or health_unknown (fail-closed default).
  • LivenessClassification is the stable observable category a probe outcome reduces through: available, unavailable, timeout, error, unsupported, unknown, and identity_mismatch.
  • ProbeOutcome is the typed, target-aware input; ClassifyProbeOutcome reduces it to a classification and NormalizeProbeOutcome maps it to health. The mapping is exactly: available → request_stalled; a validated matching unavailable result → provider_unhealthy; every error, timeout, unsupported adapter, unknown status, empty/mismatched adapter or target, and instance mismatch → health_unknown.
  • A returned error takes precedence over any reported status, so endpoint construction, request/network, non-success HTTP, and decode failures can never be confused with a positive exact-target-absent result.
  • The Node probe coordinator (ProbeHealth) roots its own five-second bounded context from the background, re-checks that deadline/cancellation after the probe returns, validates exact adapter and target identity (including a pinned instance key when set), and feeds only the typed normalizer. It never copies arbitrary provider metadata.
  • ResolveProbeFunc returns nil for an adapter that does not implement ProviderProber; a nil hook makes ProbeHealth fail closed to health_unknown without invoking any endpoint.

Probe completion is evidence only. The probe itself must never reset original request progress, change the attempt fence, authorize retry, sequence the watchdog terminal, drive the Edge overlay, or infer recovery. Node owns the stall-terminal join and the connection-scoped health_observation_seq; Edge reception-generation binding, stale-observation validation, the Edge health overlay, candidate exclusion, retry, recovery, and configuration remain owned by later slices.

Prohibited ownership

The package must not own interactive shells, persistent processes, terminal emulation, working-directory mutation, resumable conversations, local quota probing, or arbitrary host command execution. It must not import application-internal packages or generated transport types.

Verification

  • go test -count=1 ./packages/go/execution
  • go test -race -count=1 ./packages/go/execution
  • go vet ./packages/go/execution