iop/agent-contract/inner/execution-runtime.md

Provider Execution Runtime Contract

Contract metadata

• id: iop.execution-runtime
• boundary: inner
• status: active
• source evidence:
  • packages/go/execution/types.go
  • packages/go/execution/liveness.go
  • packages/go/execution/registry.go
  • packages/go/execution/emitter.go
  • packages/go/execution/failure.go
  • apps/node/internal/node/runtime_bridge.go
  • apps/node/internal/node/health_probe.go
  • apps/node/internal/node/command_handler.go
  • apps/node/internal/node/liveness_watchdog.go
  • apps/node/internal/transport/session.go
  • apps/edge/internal/service/model_queue_release.go
  • apps/edge/internal/service/node_command.go
  • apps/edge/internal/openai/stream_gate_runtime.go
  • apps/edge/internal/openai/stream_gate_stall_recovery_test.go

Scope

The execution package defines host-neutral provider primitives. It owns provider registration, lifecycle, execution events, typed failures, cancellation, token usage, and the three provider commands capabilities, transport_status, and ollama_api.

session_id is an opaque correlation value. It does not select, create, resume, or terminate a process. Repeated requests with the same value are independent executions. Cancellation targets a non-empty run_id only.

Requirements

• Providers implement the narrow Provider interface and may expose optional lifecycle, command, or tunnel capabilities.
• Event emitters preserve order and publish exactly one terminal event.
• Registry lookup uses provider identity and returns typed failures for missing or unavailable providers.
• Callers must reject commands outside the closed provider-command allowlist before provider lookup.
• Token usage remains observation data attached to execution or tunnel results.
• DefaultResponseStallTimeoutMS = 300000 is the documented default. ResolveStallTimeoutMS(ms) validates then maps zero to the default; safe positive values pass through, while negative or overflow values return an error.
• ClassifyRuntimeEvent returns start for EventTypeStart, progress for non-empty delta/message or non-terminal usage, terminal for complete/error/cancelled (before usage check), and none for empty/unknown events.
• ClassifyProviderTunnelFrame returns progress for response_start (with or without headers) and non-empty body, terminal for end/error (before payload check), progress for usage, and none for empty/unknown frames.
• ValidateStallTimeoutMS(ms) rejects negative values and values exceeding maxSafeStallTimeoutMS; zero is allowed (use default).
• NodeProviderConf.EffectiveResponseStallTimeoutMS() returns the effective timeout for a provider candidate.
• RunRequest.ResponseStallTimeoutMS and ProviderTunnelRequest.ResponseStallTimeoutMS carry the selected provider's effective timeout; zero on the wire means the Node applies the documented default.
• The Node wire boundary normalizes zero to 300000 and rejects negative or overflow values before router/provider invocation.
• response_stalled is a stable typed failure. Node transport mappers (runEventToProto and tunnelFrameToProto) populate the optional wire ExecutionFailure message only for FailureCodeResponseStalled, attaching a defensive clone of allowlisted metadata keys (failure_code, provider_health, liveness_classification, idle_duration_ms, run_id, attempt_id, attempt_fence, adapter, target, and health_observation_seq); nil and non-stalled failures leave wire ExecutionFailure absent while preserving legacy error string fields (RunEvent.Error / ProviderTunnelFrame.Error). Caller metadata cannot override these values, and no raw payload, credential, or recovery_eligible signal is admitted.
• The Node watchdog starts from attempt admission, resets only on the documented progress dispositions, stops on provider terminal, and emits one typed stall terminal. It does not retry providers or infer recovery eligibility. Retryable=true means only that the local provider ownership fence was confirmed within the bounded close grace.
• After the watchdog claims a stall it joins two independent bounded outcomes without extending either serially — the fixed close-grace fence and the exact-target health probe — then assembles exactly one allowlisted terminal. The joined liveness_classification/provider_health pair is exactly request_stalled/available, provider_unhealthy/unavailable, or health_unknown/unknown (fail-closed default). Provider availability observed here is evidence only: it never resets progress, changes the fence, revives output, or authorizes retry, and late provider output stays fenced.
• health_observation_seq is a connection-scoped monotonic sequence sourced from the transport Session. A new connection starts at zero, so the first finalized observation is one; normalized and tunnel observations on the same connection share the source and receive unique, increasing values under concurrency. Internal or unbound execution paths omit the key entirely and never encode a process-global generation.
• ProviderPoolDispatchRequest carries two request-local recovery-hint fields: AvoidProviderID (non-empty to prefer a runtime-eligible alternate over the avoided provider) and AllowAvoidedProviderFallback (explicit permission to retain the avoided provider when no alternate exists and it remains runtime eligible). The queue applies identical avoidance filtering to both initial and queued re-resolution. Zero values preserve current selection behavior. This is selection policy only: it does not create a retry loop, reserve a slot, change provider priority, persist the hints, or count retries. The fallback permission is always derived from exact probe-backed available evidence by the caller (never from current overlay state).
• A Node capabilities command performs the same bounded exact-target ProbeHealth operation. Its stable result evidence is the requested adapter instance key (adapter_key), exact target, fail-closed normalized provider_status, and the next health_observation_seq from that same transport Session. Probe errors, unsupported probing, and adapter/instance/target mismatches report unknown; raw capability status is not recovery evidence.
• Edge accepts a typed stall observation for provider-wide projection only after authoritative reception (node_id, connection_generation) matches the tracked immutable dispatch lease (node_id, connection_generation, provider_id, adapter, target), the local attempt fence is confirmed, and the observation sequence is strictly newer. A current terminal still releases its lease exactly once when health evidence is absent, malformed, mismatched, or stale; a reception-owner mismatch changes neither overlay nor lease state.
• Every validated current bound stall is annotated with Edge-owned provider_id, the validated provider_health, and recovery_handoff=confirmed, including an out-of-order terminal whose health projection is sequence-stale. Only a fresh unavailable observation lowers the generation-scoped runtime overlay. The token proves reception, lease binding, and local-fence handoff only; it is never recovery_eligible and never authorizes retry.
• Every supported OpenAI Chat/Responses normalized or tunnel request enters one request-local StreamGate runtime, which is the sole liveness owner even when configured semantic filtering is disabled. That runtime may consume the confirmed handoff as a raw-free response_stalled provider error while its endpoint adapters preserve the disabled-semantic native status, headers, JSON/SSE/tunnel order, validation, usage, cancellation, and terminal behavior. It retains only the stable failure code, confirmed-handoff token, and available|unavailable|unknown health classification; Node/provider messages and arbitrary metadata are not copied. Exact replay additionally requires the existing uncommitted, uncancelled, side-effect-safe, snapshot-backed, shared-budget gate. A confirmed old terminal closes its Edge transport without another CancelRun; pool re-admission consumes the provider once as AvoidProviderID, with same-provider fallback only for exact available evidence.
• The runtime overlay is keyed by (node_id, connection_generation, provider_id) and remains separate from configuration health. It excludes the provider from effective admission and projects it unavailable in status snapshots. Recovery requires a later CAPABILITIES result for the same current adapter/target mapping with strictly higher sequence and exact normalized available; malformed, ambiguous, stale-generation, unknown, and unavailable results are no-ops.

Health probe contract

The execution package owns the stable, fail-closed probe outcome vocabulary consumed by Node terminal assembly. It is the typed three-way boundary between an inconclusive probe and a definitive provider-health classification; nothing else maps provider probe results to health.

• ProviderHealth is the stable normalized value: request_stalled, provider_unhealthy, or health_unknown (fail-closed default).
• LivenessClassification is the stable observable category a probe outcome reduces through: available, unavailable, timeout, error, unsupported, unknown, and identity_mismatch.
• ProbeOutcome is the typed, target-aware input; ClassifyProbeOutcome reduces it to a classification and NormalizeProbeOutcome maps it to health. The mapping is exactly: available → request_stalled; a validated matching unavailable result → provider_unhealthy; every error, timeout, unsupported adapter, unknown status, empty/mismatched adapter or target, and instance mismatch → health_unknown.
• A returned error takes precedence over any reported status, so endpoint construction, request/network, non-success HTTP, and decode failures can never be confused with a positive exact-target-absent result.
• The Node probe coordinator (ProbeHealth) roots its own five-second bounded context from the background, re-checks that deadline/cancellation after the probe returns, validates exact adapter and target identity (including a pinned instance key when set), and feeds only the typed normalizer. It never copies arbitrary provider metadata.
• ResolveProbeFunc returns nil for an adapter that does not implement ProviderProber; a nil hook makes ProbeHealth fail closed to health_unknown without invoking any endpoint.

Probe completion is evidence only. The probe itself must never reset original request progress, change the attempt fence, authorize retry, sequence the watchdog terminal, directly mutate the Edge overlay, or infer recovery. Node owns the stall-terminal join and the connection-scoped health_observation_seq. Edge owns reception-generation and immutable-lease validation, the separate runtime overlay, candidate exclusion, snapshot projection, and exact later CAPABILITIES recovery. The ingress recovery host remains the sole owner of commit, cancellation, side-effect, budget, candidate, and replay eligibility decisions.

Prohibited ownership

The package must not own interactive shells, persistent processes, terminal emulation, working-directory mutation, resumable conversations, local quota probing, or arbitrary host command execution. It must not import application-internal packages or generated transport types.

Operational evidence projections

The Node and Edge owners expose bounded operational projections derived exclusively from the established stall terminal, health-overlay, and recovery decisions documented above. These projections never widen the Node↔Edge wire protocol: they carry no new frame, field, ordering rule, or retry semantic, and they are emitted only after the authoritative decision is finalized.

Node stall observations (owner: Node process-global)

• iop_node_response_stalls_total (counter): labels execution_path, provider_health, liveness_classification, attempt_fence. Every claimed stall increments exactly one series.
• iop_node_response_stall_duration_seconds (histogram): same four labels. Samples the idle duration in seconds.
• Dedicated structured log node_response_stall_observation: fields execution_path, provider_health, liveness_classification, attempt_fence, idle_duration_ms.
• Label values are closed and low-cardinality: execution_path ∈ {normalized, provider_tunnel, unknown}; provider_health ∈ {available, unavailable, unknown}; liveness_classification ∈ {request_stalled, provider_unhealthy, health_unknown}; attempt_fence ∈ {confirmed, unconfirmed, unknown}.
• Prohibited from metric labels and general logs: raw prompt/response, credential, caller metadata, recovery_eligible. High-cardinality inputs normalize to unknown.
• Observer failure is fire-and-forget and never suppresses the terminal.
• Source: apps/node/internal/node/liveness_observability.go; test: apps/node/internal/node/liveness_observability_test.go::TestNodeLivenessObservability.

Edge provider-health overlay observations (owner: Edge service queue process-global)

• iop_edge_provider_health_evidence_total (counter): labels source, evidence_health, decision. Records authoritative overlay decisions.
• iop_edge_provider_health_transitions_total (counter): labels from_health, to_health. Records overlay state transitions.
• Dedicated structured log edge_provider_health_observation: fields source, evidence_health, decision, from_health, to_health, state_changed.
• Label values are closed: source ∈ {stall, probe, unknown}; evidence_health ∈ {available, unavailable, unknown}; decision ∈ {applied, rejected_stale, rejected_binding, rejected_ambiguous, inconclusive}; from_health/to_health ∈ {available, unavailable, unknown}.
• Prohibited from metric labels and general logs: provider, node, run, session, adapter, target, payload, or credential values.
• Edge delivery is synchronous after decision/release/pump and after the queue lock is released; observer latency can delay handler return but cannot retain the lock or change the finalized transition.
• Source: apps/edge/internal/service/provider_health_observability.go; test: apps/edge/internal/service/provider_health_observability_test.go::TestProviderHealthObservability and TestProviderHealthObservabilityDoesNotExposeSentinels.

Edge OpenAI recovery observations (owner: Edge OpenAI server request-local wrapper with process-global collectors)

• iop_edge_liveness_recovery_eligibility_total (counter): labels execution_path, provider_health, commit_state, eligibility. Records eligibility decisions per liveness cycle.
• iop_edge_liveness_recovery_results_total (counter): labels execution_path, provider_health, recovery_result. Records at most one final result per liveness cycle.
• Dedicated structured log edge_liveness_recovery_observation: fields phase, execution_path, provider_health, commit_state, eligibility, recovery_result.
• Label values are closed: execution_path ∈ {normalized, provider_tunnel, unknown}; provider_health ∈ {available, unavailable, unknown}; commit_state ∈ {transport_uncommitted, stream_open, terminal_committed, unknown}; eligibility ∈ {eligible, no_owner, post_commit, unconfirmed_fence, caller_cancelled, tool_side_effect, budget_exhausted, no_candidate, same_provider_forbidden, other}; recovery_result ∈ {redispatched, plan_rejected, abort_failed, rebuild_failed, dispatch_failed, not_selected, terminal, other}.
• Prohibited from metric labels and general logs: correlation, attempt, run, session, model, provider, node, plan, shared_attempt_id, credential, or slot identifiers.
• Each request owns one fresh wrapper; the collectors are process-global and registered once at package init.
• phase is the bounded request-local cycle phase: idle before any eligible observation, eligible_pending after an eligible eligibility decision until the cycle resolves (redispatched, plan_rejected, abort_failed, rebuild_failed, dispatch_failed, not_selected, or terminal). Only these two values appear in the lifecycle; every other row carries one of them.
• Empty eligibility and recovery_result rows belong to the lifecycle transitions that do not record a metric row: private filter rows that are not filter_evaluated, a second eligibility while eligible_pending, provider errors the liveness filter did not treat as a stall, and non-ExactReplay recovery observations that fall outside the private cycle. They are documented here so the safe-log field vocabulary is complete and not read as implying a missing classification.
• Current immutable observations yield provider_health=unknown because the predecessor's private filter_evaluated observation does not carry provider health — health lives only in the request-local recovery state bridge, never in the immutable timeline. The closed classifier reserves available and unavailable for future health-bearing observations without claiming either is currently emitted.
• Source: apps/edge/internal/openai/liveness_recovery_observability.go; test: apps/edge/internal/openai/liveness_recovery_observability_test.go::TestOpenAILivenessObservationSink and TestOpenAILivenessRecoveryObservability.

Fresh health recovery in provider snapshots

A recovered provider appears in the existing Edge provider snapshot overlay as status=available, health=available, with effective capacity restored to configured values. The snapshot reflects the same (node_id, connection_generation, provider_id) key used by the runtime overlay. A newer connection generation does not inherit the old overlay.

Leakage boundary

Operational projections exclude raw payloads, credentials, caller-controlled identities, and any unbounded identifier from metric labels and general structured logs. The exclusion applies to metric labels and general logs only; valid typed terminal metadata (e.g. run_id, adapter, target on the allowlisted stall metadata map) remains on the wire as already required by the typed terminal contract.

Verification

• go test -count=1 ./packages/go/execution
• go test -race -count=1 ./packages/go/execution
• go vet ./packages/go/execution