--- spec_doc_type: spec spec_id: runtime/edge-node-execution status: 구현됨 source_evidence: - type: contract path: agent-contract/inner/execution-runtime.md notes: Host-neutral provider execution primitives - type: contract path: agent-contract/inner/edge-node-runtime-wire.md notes: Edge-Node registration, execution, tunnel, cancellation, command, and refresh wire - type: code path: packages/go/execution/types.go notes: Provider execution and event types - type: code path: packages/go/execution/liveness.go notes: Response-stall timeout default, validation, and RuntimeEvent/ProviderTunnelFrame activity classifiers - type: code path: apps/node/internal/node/runtime_bridge.go notes: Protobuf-to-execution translation with raw stall timeout validation before router/provider invocation - type: code path: apps/edge/internal/transport/server.go notes: Edge-side tunnel-tolerant heartbeat and disconnect supervision - type: code path: apps/node/internal/transport/client.go notes: Node-side tunnel-tolerant heartbeat and reconnect transport - type: code path: apps/edge/internal/service/provider_tunnel.go notes: Provider selection, credential binding validation, reception-aware terminal handoff, lease acquisition, and pre-send fencing - type: code path: apps/edge/internal/service/model_queue_release.go notes: Immutable lease validation, generation/sequence-fenced runtime health overlay, recovery handoff annotation, and exactly-once release - type: code path: apps/edge/internal/service/node_command.go notes: CAPABILITIES dispatch identity retention and exact available recovery evidence application - type: code path: apps/node/internal/node/tunnel_handler.go notes: Provider tunnel handling and recipient-sealed credential lease consumption - type: code path: apps/node/internal/node/liveness_watchdog.go notes: Shared normalized/tunnel stall coordination, close-grace ownership, serialized emission fencing, bounded probe/fence join, and connection-scoped observation sequencing - type: code path: apps/node/internal/node/health_probe.go notes: Bounded independent exact-target health probe coordinator consumed by the stall terminal join - type: code path: apps/node/internal/transport/session.go notes: Connection-scoped monotonic health-observation sequence source - type: code path: packages/go/credentiallease/envelope.go notes: Signed scope validation, recipient sealing, expiry, replay, and exact binding verification - type: test path: apps/node/internal/node/command_test.go notes: Closed provider commands plus fail-closed exact CAPABILITIES health and Session sequence regressions - type: test path: apps/edge/internal/service/provider_health_overlay_test.go notes: S04 binding, stale evidence, normalized/tunnel release races, overlay projection, and CAPABILITIES recovery evidence - type: test path: apps/edge/internal/openai/stream_gate_stall_recovery_test.go notes: S05 always-owned OpenAI recovery, new attempt/provider selection, shared budget, old-transport close, and guard terminals - type: test path: apps/edge/internal/transport/heartbeat_test.go notes: Edge heartbeat liveness profile regression - type: test path: apps/node/internal/transport/heartbeat_test.go notes: Node heartbeat liveness and idle-connection regressions - type: test path: apps/node/internal/node/liveness_watchdog_test.go notes: Manual-clock S01/S02 threshold, progress, terminal, close-grace, ownership, metadata, and late-output evidence - type: test path: apps/node/internal/node/provider_tunnel_test.go notes: Credential preflight admission release regression - type: test path: apps/node/internal/transport/session_test.go notes: Run and tunnel handler lifetime cancellation on disconnect - type: code path: apps/node/internal/node/liveness_observability.go notes: Node stall counter/histogram and dedicated structured log with closed label values and raw-payload exclusion - type: test path: apps/node/internal/node/liveness_observability_test.go notes: Deterministic S06 Node stall observation regression with closed label values - type: code path: apps/edge/internal/service/provider_health_observability.go notes: Edge overlay evidence/transition counters and dedicated structured log with closed label values and identity exclusion - type: test path: apps/edge/internal/service/provider_health_observability_test.go notes: Deterministic S06 Edge overlay observation regression including sentinel exclusion via TestProviderHealthObservabilityDoesNotExposeSentinels - type: code path: apps/edge/internal/openai/liveness_recovery_observability.go notes: Edge OpenAI eligibility/results counters and dedicated structured log with closed label values and identifier exclusion - type: test path: apps/edge/internal/openai/liveness_recovery_observability_test.go notes: Deterministic S06 OpenAI recovery observation regression with closed label values --- # Edge-Node Provider Execution ## 목적 Edge owns provider selection, queue admission, leases, and connection-generation fencing. Node owns local provider adapters and executes normalized runs or provider HTTP tunnels after a ready handshake. The shared `packages/go/execution` package contains provider lifecycle, registry, execution events, failures, cancellation, usage, and optional tunnel/command primitives. It does not manage host programs or durable conversation state. ## 기능 목록 | 기능 | 설명 | |------|------| | register/readiness | 등록된 Node의 현재 connection이 readiness를 완료한 뒤에만 dispatch한다. | | normalized execution | `adapter + target`으로 provider 실행을 선택하고 ordered `RunEvent` stream을 반환한다. | | provider raw tunnel | 선택된 provider의 HTTP/SSE를 `ProviderTunnelRequest`/`ProviderTunnelFrame`으로 relay하며 순서와 단일 terminal outcome을 보장한다. | | response-stall activity contract | 선택된 provider의 response-stall timeout을 normalized/tunnel request에 보존한다. Node는 wire zero를 `300000ms`로 해석하고 invalid raw value를 adapter 호출 전에 거부한다. Runtime event의 terminal type은 payload/usage보다 우선하며 non-terminal usage는 progress다. | | Node stall watchdog | Node가 normalized run과 raw tunnel에 하나의 activity watchdog을 적용한다. progress만 timer를 reset하며, stall은 `response_stalled` terminal 하나와 Node-owned safe metadata를 만들어 normalized `RunEvent`와 raw `ProviderTunnelFrame` wire의 optional typed `ExecutionFailure` 필드에 싣는다. stall claim 뒤에는 bounded close grace fence와 독립 exact-target health probe를 직렬 확장 없이 join한다. close grace 안에 provider return이 확인된 경우만 `Retryable` capability hint를 준다. | | Node health evidence join | stall terminal에 three-way health evidence를 싣는다: `provider_health` status와 `liveness_classification` normalization이 `available`/`request_stalled`, `unavailable`/`provider_unhealthy`, `unknown`/`health_unknown` 쌍으로 fail-closed된다. probe 성공은 progress reset·fence 변경·retry authority가 아니며 late output은 fenced 상태를 유지한다. | | health observation sequence | transport Session이 connection-scoped monotonic `health_observation_seq`를 소유한다. 새 connection은 0에서 시작해 첫 finalized observation이 1이며, 같은 connection의 normalized/tunnel observation이 source를 공유해 동시에도 유일 증가값을 받는다. internal/unbound 경로는 key를 생략한다. | | Edge terminal health handoff | Edge validates authoritative reception node/generation plus the immutable provider/adapter/target lease before applying typed stall evidence. Every validated current bound stall receives `provider_id`, validated health, and `recovery_handoff=confirmed`, while only fresh unavailable evidence lowers a separate runtime overlay; the token never grants replay eligibility. Every valid current terminal still releases its lease exactly once. | | CAPABILITIES recovery | Node runs the same bounded exact-target `ProbeHealth` and returns stable adapter/target/status plus the next Session sequence. Edge recovers exactly one matching current-generation unavailable provider only from a strictly newer `available` result; malformed, ambiguous, stale, unknown, and unavailable responses are no-ops. | | recovery candidate preference | `ProviderPoolDispatchRequest` carries `AvoidProviderID` and `AllowAvoidedProviderFallback`. Every admission (initial and queued re-resolution) prefers a runtime-eligible alternate over the avoided provider; only the explicit fallback flag (derived from exact probe-backed `available` evidence) permits re-selecting the avoided provider when no alternate exists. Zero values preserve current selection. This is selection policy only: no retry loop, slot reservation, priority change, persistence, or retry counter. | | OpenAI typed-stall consumption | Every supported Chat/Responses normalized or tunnel request has one unconditional runtime liveness owner, independent of configured semantic activation. It converts only the Edge-confirmed typed stall handoff into a raw-free StreamGate event, owns pre-commit eligibility, and closes the already fenced old transport before re-admission; Node does not grant replay authority. | | tunnel-tolerant liveness | Edge와 Node는 30초 heartbeat interval과 45초 response wait를 공통으로 사용해 긴 prompt prefill이나 streaming backpressure 중의 정상 connection을 조기에 끊지 않는다. | | reconnect/generation fencing | 현재 connection이 종료되면 해당 generation만 fence하고 Node supervisor가 reconnect한다. Heartbeat wait를 넘긴 경우의 close reason은 `heartbeat_timeout`이다. | | cancellation/command | `run_id`로 현재 run만 취소하며 command는 capabilities, transport status, Ollama API tunnel로 제한한다. | | managed credential lease | Edge가 principal·route·slot·profile·target·Node·revision·generation을 binding한 sealed lease를 발급하고 Node가 capacity admission 후 provider 실행 직전에만 연다. | ## 범위 - `session_id`는 event와 command result의 opaque correlation일 뿐이며 같은 값을 재사용해도 모든 run은 독립적이다. - provider usage, capacity, queue pressure, lifecycle, reconnect, tool calling은 Edge-Node 실행 경로에서 계속 지원한다. - managed mode는 등록과 dispatch 전에 CA로 검증된 Edge/Node workload identity를 요구한다. - revoked, disabled, expired, stale, replayed, wrong-recipient, mismatched lease는 provider나 credential fallback 없이 fail closed한다. IOP no longer provides persistent shell sessions, terminal emulation, process resume, local working-directory execution context, arbitrary host commands, or local quota/status probing. The current spec maps reviewed Node and Edge observability producers to S06 behavior and deterministic tests. Node exposes bounded stall counters/histograms and dedicated structured logs with closed label values and raw-payload exclusion. Edge service queue exposes bounded overlay evidence/transition counters and dedicated structured logs with closed label values and identity exclusion. Edge OpenAI server exposes bounded eligibility/results counters and dedicated structured logs with closed label values and identifier exclusion. All projections are local observations and do not widen the wire protocol. ## 주요 흐름 ```mermaid sequenceDiagram participant Edge participant Node participant Provider Node->>Edge: RegisterRequest Edge-->>Node: RegisterResponse + config Node->>Edge: NodeReadyRequest Edge-->>Node: NodeReadyResponse Edge->>Node: ProviderTunnelRequest Node->>Provider: HTTP/SSE request Provider-->>Node: status/header/body stream Node-->>Edge: ordered ProviderTunnelFrame stream Note over Edge,Node: heartbeat 30s interval / 45s wait alt heartbeat wait exceeded Edge--xNode: current generation fenced Node->>Edge: supervised reconnect end ``` ## 계약 - Edge-Node wire: `agent-contract/inner/edge-node-runtime-wire.md` - provider execution primitives: `agent-contract/inner/execution-runtime.md` Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 liveness profile이다. `response_stall_timeout_ms`만 provider execution request wire에 추가되며 provider response shape은 바뀌지 않는다. ## 설정/데이터/이벤트 - Edge와 Node의 현재 heartbeat interval은 30초, response wait는 45초다. - 이 값은 runtime YAML model config나 `max_tokens`/context 설정이 아니라 transport 구현 상수다. - 45초 동안 heartbeat response가 없으면 current connection을 `heartbeat_timeout`으로 닫고 provider resource를 offline 처리한 뒤 reconnect/queue 재평가를 수행한다. - response-stall timeout은 provider config가 source이며 winning candidate가 re-resolution된 뒤의 request까지 같은 effective value를 보존한다. request hard timeout, queue timeout, transport heartbeat, client response-idle timeout과 timer lifecycle은 별도 소유권이다. ## 검증 - `go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/service` - `go test -race -count=1 ./packages/go/execution ./apps/node/internal/node ./apps/edge/internal/service` - `go test -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport` - `go test -race -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport` - 실제 provider tunnel 검증은 5초를 넘는 긴 prefill과 streaming 응답 동안 Node가 connected/healthy를 유지하고, 응답이 정상 terminal을 반환하며, `heartbeat_timeout`이 발생하지 않는지 확인한다. - `go test -count=1 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — deterministic Node stall observation with closed label values and raw-payload exclusion. - `go test -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — deterministic Edge overlay evidence/transition with closed label values and identity exclusion; `TestProviderHealthObservabilityDoesNotExposeSentinels` covers the sentinel/prohibited-value guard. - `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAILivenessObservationSink|TestOpenAILivenessRecoveryObservability)$'` — deterministic OpenAI recovery eligibility/results with closed label values and identifier exclusion. ## 한계와 주의사항 - 30/45초 liveness profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다. - 45초를 넘겨 실제 heartbeat response가 없는 connection은 기존과 같이 오프라인 처리하고 reconnect한다. - Node owns local detection, cancellation, emission fencing, confirmed/unconfirmed ownership close, exact-target probe joining, connection-scoped observation sequencing, and the bounded `iop_node_response_stalls_total` / `iop_node_response_stall_duration_seconds` / `node_response_stall_observation` projections with closed label values. - Edge owns reception-generation and immutable-lease validation, the generation-scoped runtime health overlay, `iop_edge_provider_health_evidence_total` / `iop_edge_provider_health_transitions_total` / `edge_provider_health_observation` projections with closed label values, effective admission/snapshot projection, and exact later CAPABILITIES recovery. - The always-owned supported OpenAI ingress runtime owns commit, cancellation, side-effect, snapshot, shared-budget, candidate, and replay decisions, and exposes `iop_edge_liveness_recovery_eligibility_total` / `iop_edge_liveness_recovery_results_total` / `edge_liveness_recovery_observation` projections with closed label values. - Node retry and `recovery_eligible` remain prohibited. Hard deadline and connection disconnect continue to take precedence over a simultaneous stall timer. - Operational projections never widen the wire protocol; they carry no new frame, field, ordering rule, or retry semantic. ## 변경 기록 - 2026-08-02: provider tunnel의 긴 prompt prefill과 streaming backpressure를 정상 traffic으로 허용하도록 Edge/Node heartbeat profile을 30초 interval/45초 wait로 복원한 현재 구현과 회귀 검증을 반영했다 (`apps/edge/internal/transport/server.go`, `apps/node/internal/transport/client.go`). - 2026-08-04: provider response-stall timeout의 config validation, selected-candidate propagation, Node adapter-visible retention, and activity classification contract를 반영했다. - 2026-08-04: Added the shared Node run/tunnel watchdog coordinator, serialized tunnel emission fence, pre-provider admission cleanup, disconnect-bound handler lifetime, and deterministic S01/S02 manual-clock evidence. - 2026-08-04: Joined the bounded close-grace fence and the independent exact-target health probe into one stall terminal carrying three-way health evidence, and added the connection-scoped `health_observation_seq` sourced from the transport Session. - 2026-08-05: Added authoritative Edge terminal handoff, immutable lease binding, generation/sequence-fenced runtime provider health, exactly-once normalized/tunnel release, and fail-closed Session-sequenced CAPABILITIES recovery without config-health mutation or replay authorization. - 2026-08-05: Added runtime-local OpenAI consumption of confirmed typed stalls, including cancel-free old-transport close and provider-pool avoidance hints for ExactReplay. - 2026-08-05: Made supported OpenAI Chat/Responses normalized and tunnel liveness ownership unconditional and added S05 recovery/guard evidence independent of semantic policy activation. - 2026-08-06: Mapped reviewed Node, Edge overlay, and OpenAI recovery observability producers to S06 behavior with deterministic test evidence. Node exposes `iop_node_response_stalls_total`, `iop_node_response_stall_duration_seconds`, and `node_response_stall_observation` (source: `apps/node/internal/node/liveness_observability.go`; test: `TestNodeLivenessObservability`). Edge service queue exposes `iop_edge_provider_health_evidence_total`, `iop_edge_provider_health_transitions_total`, and `edge_provider_health_observation` (source: `apps/edge/internal/service/provider_health_observability.go`; test: `TestProviderHealthObservability`, `TestProviderHealthObservabilityDoesNotExposeSentinels`). Edge OpenAI server exposes `iop_edge_liveness_recovery_eligibility_total`, `iop_edge_liveness_recovery_results_total`, and `edge_liveness_recovery_observation` (source: `apps/edge/internal/openai/liveness_recovery_observability.go`; test: `TestOpenAILivenessObservationSink`, `TestOpenAILivenessRecoveryObservability`). All projections carry only closed, low-cardinality label values and exclude raw payloads, credentials, and unbounded identifiers from metric labels and general logs. The wire protocol is unchanged.