iop/agent-spec/runtime/edge-node-execution.md

spec_doc_type	spec_id	status	source_evidence
spec	runtime/edge-node-execution	구현됨	type path notes contract agent-contract/inner/execution-runtime.md Host-neutral provider execution primitives type path notes contract agent-contract/inner/edge-node-runtime-wire.md Edge-Node registration, execution, tunnel, cancellation, command, and refresh wire type path notes code packages/go/execution/types.go Provider execution and event types type path notes code packages/go/execution/liveness.go Response-stall timeout default, validation, and RuntimeEvent/ProviderTunnelFrame activity classifiers type path notes code apps/node/internal/node/runtime_bridge.go Protobuf-to-execution translation with raw stall timeout validation before router/provider invocation type path notes code apps/edge/internal/transport/server.go Edge-side tunnel-tolerant heartbeat and disconnect supervision type path notes code apps/node/internal/transport/client.go Node-side tunnel-tolerant heartbeat and reconnect transport type path notes code apps/edge/internal/service/provider_tunnel.go Provider selection, credential binding validation, reception-aware terminal handoff, lease acquisition, and pre-send fencing type path notes code apps/edge/internal/service/model_queue_release.go Immutable lease validation, generation/sequence-fenced runtime health overlay, recovery handoff annotation, and exactly-once release type path notes code apps/edge/internal/service/node_command.go CAPABILITIES dispatch identity retention and exact available recovery evidence application type path notes code apps/node/internal/node/tunnel_handler.go Provider tunnel handling and recipient-sealed credential lease consumption type path notes code apps/node/internal/node/liveness_watchdog.go Shared normalized/tunnel stall coordination, close-grace ownership, serialized emission fencing, bounded probe/fence join, and connection-scoped observation sequencing type path notes code apps/node/internal/node/health_probe.go Bounded independent exact-target health probe coordinator consumed by the stall terminal join type path notes code apps/node/internal/transport/session.go Connection-scoped monotonic health-observation sequence source type path notes code packages/go/credentiallease/envelope.go Signed scope validation, recipient sealing, expiry, replay, and exact binding verification type path notes test apps/node/internal/node/command_test.go Closed provider commands plus fail-closed exact CAPABILITIES health and Session sequence regressions type path notes test apps/edge/internal/service/provider_health_overlay_test.go S04 binding, stale evidence, normalized/tunnel release races, overlay projection, and CAPABILITIES recovery evidence type path notes test apps/edge/internal/openai/stream_gate_stall_recovery_test.go S05 always-owned OpenAI recovery, new attempt/provider selection, shared budget, old-transport close, and guard terminals type path notes test apps/edge/internal/transport/heartbeat_test.go Edge heartbeat liveness profile regression type path notes test apps/node/internal/transport/heartbeat_test.go Node heartbeat liveness and idle-connection regressions type path notes test apps/node/internal/node/liveness_watchdog_test.go Manual-clock S01/S02 threshold, progress, terminal, close-grace, ownership, metadata, and late-output evidence type path notes test apps/node/internal/node/provider_tunnel_test.go Credential preflight admission release regression type path notes test apps/node/internal/transport/session_test.go Run and tunnel handler lifetime cancellation on disconnect type path notes code apps/node/internal/node/liveness_observability.go Node stall counter/histogram and dedicated structured log with closed label values and raw-payload exclusion type path notes test apps/node/internal/node/liveness_observability_test.go Deterministic S06 Node stall observation regression with closed label values type path notes code apps/edge/internal/service/provider_health_observability.go Edge overlay evidence/transition counters and dedicated structured log with closed label values and identity exclusion type path notes test apps/edge/internal/service/provider_health_observability_test.go Deterministic S06 Edge overlay observation regression including sentinel exclusion via TestProviderHealthObservabilityDoesNotExposeSentinels type path notes code apps/edge/internal/openai/liveness_recovery_observability.go Edge OpenAI eligibility/results counters and dedicated structured log with closed label values and identifier exclusion type path notes test apps/edge/internal/openai/liveness_recovery_observability_test.go Deterministic S06 OpenAI recovery observation regression with closed label values

Edge-Node Provider Execution

목적

Edge owns provider selection, queue admission, leases, and connection-generation fencing. Node owns local provider adapters and executes normalized runs or provider HTTP tunnels after a ready handshake.

The shared packages/go/execution package contains provider lifecycle, registry, execution events, failures, cancellation, usage, and optional tunnel/command primitives. It does not manage host programs or durable conversation state.

기능 목록

기능	설명
register/readiness	등록된 Node의 현재 connection이 readiness를 완료한 뒤에만 dispatch한다.
normalized execution	adapter + target으로 provider 실행을 선택하고 ordered RunEvent stream을 반환한다.
provider raw tunnel	선택된 provider의 HTTP/SSE를 ProviderTunnelRequest/ProviderTunnelFrame으로 relay하며 순서와 단일 terminal outcome을 보장한다.
response-stall activity contract	선택된 provider의 response-stall timeout을 normalized/tunnel request에 보존한다. Node는 wire zero를 300000ms로 해석하고 invalid raw value를 adapter 호출 전에 거부한다. Runtime event의 terminal type은 payload/usage보다 우선하며 non-terminal usage는 progress다.
Node stall watchdog	Node가 normalized run과 raw tunnel에 하나의 activity watchdog을 적용한다. progress만 timer를 reset하며, stall은 response_stalled terminal 하나와 Node-owned safe metadata를 만들어 normalized RunEvent와 raw ProviderTunnelFrame wire의 optional typed ExecutionFailure 필드에 싣는다. stall claim 뒤에는 bounded close grace fence와 독립 exact-target health probe를 직렬 확장 없이 join한다. close grace 안에 provider return이 확인된 경우만 Retryable capability hint를 준다.
Node health evidence join	stall terminal에 three-way health evidence를 싣는다: provider_health status와 liveness_classification normalization이 available/request_stalled, unavailable/provider_unhealthy, unknown/health_unknown 쌍으로 fail-closed된다. probe 성공은 progress reset·fence 변경·retry authority가 아니며 late output은 fenced 상태를 유지한다.
health observation sequence	transport Session이 connection-scoped monotonic health_observation_seq를 소유한다. 새 connection은 0에서 시작해 첫 finalized observation이 1이며, 같은 connection의 normalized/tunnel observation이 source를 공유해 동시에도 유일 증가값을 받는다. internal/unbound 경로는 key를 생략한다.
Edge terminal health handoff	Edge validates authoritative reception node/generation plus the immutable provider/adapter/target lease before applying typed stall evidence. Every validated current bound stall receives provider_id, validated health, and recovery_handoff=confirmed, while only fresh unavailable evidence lowers a separate runtime overlay; the token never grants replay eligibility. Every valid current terminal still releases its lease exactly once.
CAPABILITIES recovery	Node runs the same bounded exact-target ProbeHealth and returns stable adapter/target/status plus the next Session sequence. Edge recovers exactly one matching current-generation unavailable provider only from a strictly newer available result; malformed, ambiguous, stale, unknown, and unavailable responses are no-ops.
recovery candidate preference	ProviderPoolDispatchRequest carries AvoidProviderID and AllowAvoidedProviderFallback. Every admission (initial and queued re-resolution) prefers a runtime-eligible alternate over the avoided provider; only the explicit fallback flag (derived from exact probe-backed available evidence) permits re-selecting the avoided provider when no alternate exists. Zero values preserve current selection. This is selection policy only: no retry loop, slot reservation, priority change, persistence, or retry counter.
OpenAI typed-stall consumption	Every supported Chat/Responses normalized or tunnel request has one unconditional runtime liveness owner, independent of configured semantic activation. It converts only the Edge-confirmed typed stall handoff into a raw-free StreamGate event, owns pre-commit eligibility, and closes the already fenced old transport before re-admission; Node does not grant replay authority.
tunnel-tolerant liveness	Edge와 Node는 30초 heartbeat interval과 45초 response wait를 공통으로 사용해 긴 prompt prefill이나 streaming backpressure 중의 정상 connection을 조기에 끊지 않는다.
reconnect/generation fencing	현재 connection이 종료되면 해당 generation만 fence하고 Node supervisor가 reconnect한다. Heartbeat wait를 넘긴 경우의 close reason은 heartbeat_timeout이다.
cancellation/command	run_id로 현재 run만 취소하며 command는 capabilities, transport status, Ollama API tunnel로 제한한다.
managed credential lease	Edge가 principal·route·slot·profile·target·Node·revision·generation을 binding한 sealed lease를 발급하고 Node가 capacity admission 후 provider 실행 직전에만 연다.

범위

• session_id는 event와 command result의 opaque correlation일 뿐이며 같은 값을 재사용해도 모든 run은 독립적이다.
• provider usage, capacity, queue pressure, lifecycle, reconnect, tool calling은 Edge-Node 실행 경로에서 계속 지원한다.
• managed mode는 등록과 dispatch 전에 CA로 검증된 Edge/Node workload identity를 요구한다.
• revoked, disabled, expired, stale, replayed, wrong-recipient, mismatched lease는 provider나 credential fallback 없이 fail closed한다.

IOP no longer provides persistent shell sessions, terminal emulation, process resume, local working-directory execution context, arbitrary host commands, or local quota/status probing.

The current spec maps reviewed Node and Edge observability producers to S06 behavior and deterministic tests. Node exposes bounded stall counters/histograms and dedicated structured logs with closed label values and raw-payload exclusion. Edge service queue exposes bounded overlay evidence/transition counters and dedicated structured logs with closed label values and identity exclusion. Edge OpenAI server exposes bounded eligibility/results counters and dedicated structured logs with closed label values and identifier exclusion. All projections are local observations and do not widen the wire protocol.

주요 흐름

sequenceDiagram
    participant Edge
    participant Node
    participant Provider

    Node->>Edge: RegisterRequest
    Edge-->>Node: RegisterResponse + config
    Node->>Edge: NodeReadyRequest
    Edge-->>Node: NodeReadyResponse
    Edge->>Node: ProviderTunnelRequest
    Node->>Provider: HTTP/SSE request
    Provider-->>Node: status/header/body stream
    Node-->>Edge: ordered ProviderTunnelFrame stream
    Note over Edge,Node: heartbeat 30s interval / 45s wait
    alt heartbeat wait exceeded
        Edge--xNode: current generation fenced
        Node->>Edge: supervised reconnect
    end

계약

• Edge-Node wire: agent-contract/inner/edge-node-runtime-wire.md
• provider execution primitives: agent-contract/inner/execution-runtime.md

Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 liveness profile이다. response_stall_timeout_ms만 provider execution request wire에 추가되며 provider response shape은 바뀌지 않는다.

설정/데이터/이벤트

• Edge와 Node의 현재 heartbeat interval은 30초, response wait는 45초다.
• 이 값은 runtime YAML model config나 max_tokens/context 설정이 아니라 transport 구현 상수다.
• 45초 동안 heartbeat response가 없으면 current connection을 heartbeat_timeout으로 닫고 provider resource를 offline 처리한 뒤 reconnect/queue 재평가를 수행한다.
• response-stall timeout은 provider config가 source이며 winning candidate가 re-resolution된 뒤의 request까지 같은 effective value를 보존한다. request hard timeout, queue timeout, transport heartbeat, client response-idle timeout과 timer lifecycle은 별도 소유권이다.

검증

• go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/service
• go test -race -count=1 ./packages/go/execution ./apps/node/internal/node ./apps/edge/internal/service
• go test -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport
• go test -race -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport
• 실제 provider tunnel 검증은 5초를 넘는 긴 prefill과 streaming 응답 동안 Node가 connected/healthy를 유지하고, 응답이 정상 terminal을 반환하며, heartbeat_timeout이 발생하지 않는지 확인한다.
• go test -count=1 ./apps/node/internal/node -run '^TestNodeLivenessObservability' — deterministic Node stall observation with closed label values and raw-payload exclusion.
• go test -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability' — deterministic Edge overlay evidence/transition with closed label values and identity exclusion; TestProviderHealthObservabilityDoesNotExposeSentinels covers the sentinel/prohibited-value guard.
• go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAILivenessObservationSink|TestOpenAILivenessRecoveryObservability)$' — deterministic OpenAI recovery eligibility/results with closed label values and identifier exclusion.

한계와 주의사항

• 30/45초 liveness profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다.
• 45초를 넘겨 실제 heartbeat response가 없는 connection은 기존과 같이 오프라인 처리하고 reconnect한다.
• Node owns local detection, cancellation, emission fencing, confirmed/unconfirmed ownership close, exact-target probe joining, connection-scoped observation sequencing, and the bounded iop_node_response_stalls_total / iop_node_response_stall_duration_seconds / node_response_stall_observation projections with closed label values.
• Edge owns reception-generation and immutable-lease validation, the generation-scoped runtime health overlay, iop_edge_provider_health_evidence_total / iop_edge_provider_health_transitions_total / edge_provider_health_observation projections with closed label values, effective admission/snapshot projection, and exact later CAPABILITIES recovery.
• The always-owned supported OpenAI ingress runtime owns commit, cancellation, side-effect, snapshot, shared-budget, candidate, and replay decisions, and exposes iop_edge_liveness_recovery_eligibility_total / iop_edge_liveness_recovery_results_total / edge_liveness_recovery_observation projections with closed label values.
• Node retry and recovery_eligible remain prohibited. Hard deadline and connection disconnect continue to take precedence over a simultaneous stall timer.
• Operational projections never widen the wire protocol; they carry no new frame, field, ordering rule, or retry semantic.

변경 기록

• 2026-08-02: provider tunnel의 긴 prompt prefill과 streaming backpressure를 정상 traffic으로 허용하도록 Edge/Node heartbeat profile을 30초 interval/45초 wait로 복원한 현재 구현과 회귀 검증을 반영했다 (apps/edge/internal/transport/server.go, apps/node/internal/transport/client.go).
• 2026-08-04: provider response-stall timeout의 config validation, selected-candidate propagation, Node adapter-visible retention, and activity classification contract를 반영했다.
• 2026-08-04: Added the shared Node run/tunnel watchdog coordinator, serialized tunnel emission fence, pre-provider admission cleanup, disconnect-bound handler lifetime, and deterministic S01/S02 manual-clock evidence.
• 2026-08-04: Joined the bounded close-grace fence and the independent exact-target health probe into one stall terminal carrying three-way health evidence, and added the connection-scoped health_observation_seq sourced from the transport Session.
• 2026-08-05: Added authoritative Edge terminal handoff, immutable lease binding, generation/sequence-fenced runtime provider health, exactly-once normalized/tunnel release, and fail-closed Session-sequenced CAPABILITIES recovery without config-health mutation or replay authorization.
• 2026-08-05: Added runtime-local OpenAI consumption of confirmed typed stalls, including cancel-free old-transport close and provider-pool avoidance hints for ExactReplay.
• 2026-08-05: Made supported OpenAI Chat/Responses normalized and tunnel liveness ownership unconditional and added S05 recovery/guard evidence independent of semantic policy activation.
• 2026-08-06: Mapped reviewed Node, Edge overlay, and OpenAI recovery observability producers to S06 behavior with deterministic test evidence. Node exposes iop_node_response_stalls_total, iop_node_response_stall_duration_seconds, and node_response_stall_observation (source: apps/node/internal/node/liveness_observability.go; test: TestNodeLivenessObservability). Edge service queue exposes iop_edge_provider_health_evidence_total, iop_edge_provider_health_transitions_total, and edge_provider_health_observation (source: apps/edge/internal/service/provider_health_observability.go; test: TestProviderHealthObservability, TestProviderHealthObservabilityDoesNotExposeSentinels). Edge OpenAI server exposes iop_edge_liveness_recovery_eligibility_total, iop_edge_liveness_recovery_results_total, and edge_liveness_recovery_observation (source: apps/edge/internal/openai/liveness_recovery_observability.go; test: TestOpenAILivenessObservationSink, TestOpenAILivenessRecoveryObservability). All projections carry only closed, low-cardinality label values and exclude raw payloads, credentials, and unbounded identifiers from metric labels and general logs. The wire protocol is unchanged.