iop/agent-spec/runtime/edge-node-execution.md
toki fef1f7a9dc feat(liveness): provider 실행 stall 관측을 구현한다
Node의 provider progress 기반 stall timeout, watchdog fencing과 bounded health probe evidence를 실행 경로에 반영한다. Edge-Node 계약과 구현 스펙, 테스트 및 Milestone 완료 evidence를 현재 상태와 맞춘다.
2026-08-05 09:45:14 +09:00

11 KiB

spec_doc_type spec_id status source_evidence
spec runtime/edge-node-execution 구현됨
type path notes
contract agent-contract/inner/execution-runtime.md Host-neutral provider execution primitives
type path notes
contract agent-contract/inner/edge-node-runtime-wire.md Edge-Node registration, execution, tunnel, cancellation, command, and refresh wire
type path notes
code packages/go/execution/types.go Provider execution and event types
type path notes
code packages/go/execution/liveness.go Response-stall timeout default, validation, and RuntimeEvent/ProviderTunnelFrame activity classifiers
type path notes
code apps/node/internal/node/runtime_bridge.go Protobuf-to-execution translation with raw stall timeout validation before router/provider invocation
type path notes
code apps/edge/internal/transport/server.go Edge-side tunnel-tolerant heartbeat and disconnect supervision
type path notes
code apps/node/internal/transport/client.go Node-side tunnel-tolerant heartbeat and reconnect transport
type path notes
code apps/edge/internal/service/provider_tunnel.go Provider selection, credential binding validation, lease acquisition, and pre-send fencing
type path notes
code apps/node/internal/node/tunnel_handler.go Provider tunnel handling and recipient-sealed credential lease consumption
type path notes
code apps/node/internal/node/liveness_watchdog.go Shared normalized/tunnel stall coordination, close-grace ownership, serialized emission fencing, bounded probe/fence join, and connection-scoped observation sequencing
type path notes
code apps/node/internal/node/health_probe.go Bounded independent exact-target health probe coordinator consumed by the stall terminal join
type path notes
code apps/node/internal/transport/session.go Connection-scoped monotonic health-observation sequence source
type path notes
code packages/go/credentiallease/envelope.go Signed scope validation, recipient sealing, expiry, replay, and exact binding verification
type path notes
test apps/node/internal/node/command_test.go Closed provider commands, correlation, and cancellation regressions
type path notes
test apps/edge/internal/transport/heartbeat_test.go Edge heartbeat liveness profile regression
type path notes
test apps/node/internal/transport/heartbeat_test.go Node heartbeat liveness and idle-connection regressions
type path notes
test apps/node/internal/node/liveness_watchdog_test.go Manual-clock S01/S02 threshold, progress, terminal, close-grace, ownership, metadata, and late-output evidence
type path notes
test apps/node/internal/node/provider_tunnel_test.go Credential preflight admission release regression
type path notes
test apps/node/internal/transport/session_test.go Run and tunnel handler lifetime cancellation on disconnect

Edge-Node Provider Execution

목적

Edge owns provider selection, queue admission, leases, and connection-generation fencing. Node owns local provider adapters and executes normalized runs or provider HTTP tunnels after a ready handshake.

The shared packages/go/execution package contains provider lifecycle, registry, execution events, failures, cancellation, usage, and optional tunnel/command primitives. It does not manage host programs or durable conversation state.

기능 목록

기능 설명
register/readiness 등록된 Node의 현재 connection이 readiness를 완료한 뒤에만 dispatch한다.
normalized execution adapter + target으로 provider 실행을 선택하고 ordered RunEvent stream을 반환한다.
provider raw tunnel 선택된 provider의 HTTP/SSE를 ProviderTunnelRequest/ProviderTunnelFrame으로 relay하며 순서와 단일 terminal outcome을 보장한다.
response-stall activity contract 선택된 provider의 response-stall timeout을 normalized/tunnel request에 보존한다. Node는 wire zero를 300000ms로 해석하고 invalid raw value를 adapter 호출 전에 거부한다. Runtime event의 terminal type은 payload/usage보다 우선하며 non-terminal usage는 progress다.
Node stall watchdog Node가 normalized run과 raw tunnel에 하나의 activity watchdog을 적용한다. progress만 timer를 reset하며, stall은 response_stalled terminal 하나와 Node-owned safe metadata를 만든다. stall claim 뒤에는 bounded close grace fence와 독립 exact-target health probe를 직렬 확장 없이 join한다. close grace 안에 provider return이 확인된 경우만 Retryable capability hint를 준다.
Node health evidence join stall terminal에 three-way health evidence를 싣는다: provider_health status와 liveness_classification normalization이 available/request_stalled, unavailable/provider_unhealthy, unknown/health_unknown 쌍으로 fail-closed된다. probe 성공은 progress reset·fence 변경·retry authority가 아니며 late output은 fenced 상태를 유지한다.
health observation sequence transport Session이 connection-scoped monotonic health_observation_seq를 소유한다. 새 connection은 0에서 시작해 첫 finalized observation이 1이며, 같은 connection의 normalized/tunnel observation이 source를 공유해 동시에도 유일 증가값을 받는다. internal/unbound 경로는 key를 생략한다.
tunnel-tolerant liveness Edge와 Node는 30초 heartbeat interval과 45초 response wait를 공통으로 사용해 긴 prompt prefill이나 streaming backpressure 중의 정상 connection을 조기에 끊지 않는다.
reconnect/generation fencing 현재 connection이 종료되면 해당 generation만 fence하고 Node supervisor가 reconnect한다. Heartbeat wait를 넘긴 경우의 close reason은 heartbeat_timeout이다.
cancellation/command run_id로 현재 run만 취소하며 command는 capabilities, transport status, Ollama API tunnel로 제한한다.
managed credential lease Edge가 principal·route·slot·profile·target·Node·revision·generation을 binding한 sealed lease를 발급하고 Node가 capacity admission 후 provider 실행 직전에만 연다.

범위

  • session_id는 event와 command result의 opaque correlation일 뿐이며 같은 값을 재사용해도 모든 run은 독립적이다.
  • provider usage, capacity, queue pressure, lifecycle, reconnect, tool calling은 Edge-Node 실행 경로에서 계속 지원한다.
  • managed mode는 등록과 dispatch 전에 CA로 검증된 Edge/Node workload identity를 요구한다.
  • revoked, disabled, expired, stale, replayed, wrong-recipient, mismatched lease는 provider나 credential fallback 없이 fail closed한다.

IOP no longer provides persistent shell sessions, terminal emulation, process resume, local working-directory execution context, arbitrary host commands, or local quota/status probing.

주요 흐름

sequenceDiagram
    participant Edge
    participant Node
    participant Provider

    Node->>Edge: RegisterRequest
    Edge-->>Node: RegisterResponse + config
    Node->>Edge: NodeReadyRequest
    Edge-->>Node: NodeReadyResponse
    Edge->>Node: ProviderTunnelRequest
    Node->>Provider: HTTP/SSE request
    Provider-->>Node: status/header/body stream
    Node-->>Edge: ordered ProviderTunnelFrame stream
    Note over Edge,Node: heartbeat 30s interval / 45s wait
    alt heartbeat wait exceeded
        Edge--xNode: current generation fenced
        Node->>Edge: supervised reconnect
    end

계약

  • Edge-Node wire: agent-contract/inner/edge-node-runtime-wire.md
  • provider execution primitives: agent-contract/inner/execution-runtime.md

Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 liveness profile이다. response_stall_timeout_ms만 provider execution request wire에 추가되며 provider response shape은 바뀌지 않는다.

설정/데이터/이벤트

  • Edge와 Node의 현재 heartbeat interval은 30초, response wait는 45초다.
  • 이 값은 runtime YAML model config나 max_tokens/context 설정이 아니라 transport 구현 상수다.
  • 45초 동안 heartbeat response가 없으면 current connection을 heartbeat_timeout으로 닫고 provider resource를 offline 처리한 뒤 reconnect/queue 재평가를 수행한다.
  • response-stall timeout은 provider config가 source이며 winning candidate가 re-resolution된 뒤의 request까지 같은 effective value를 보존한다. request hard timeout, queue timeout, transport heartbeat, client response-idle timeout과 timer lifecycle은 별도 소유권이다.

검증

  • go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/service
  • go test -race -count=1 ./packages/go/execution ./apps/node/internal/node ./apps/edge/internal/service
  • go test -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport
  • go test -race -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport
  • 실제 provider tunnel 검증은 5초를 넘는 긴 prefill과 streaming 응답 동안 Node가 connected/healthy를 유지하고, 응답이 정상 terminal을 반환하며, heartbeat_timeout이 발생하지 않는지 확인한다.

한계와 주의사항

  • 30/45초 liveness profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다.
  • 45초를 넘겨 실제 heartbeat response가 없는 connection은 기존과 같이 오프라인 처리하고 reconnect한다.
  • Node watchdog은 local detection, cancellation, emission fence, confirmed/unconfirmed ownership close, 그리고 stall terminal에 대한 exact-target health probe join과 connection-scoped observation sequencing을 소유한다. Edge reception-generation binding, stale-observation validation, Edge health overlay, Node retry, recovery_eligible, recovery, candidate selection은 이 slice 밖의 후속 작업으로 남는다. Hard deadline and connection disconnect continue to take precedence over a simultaneous stall timer.

변경 기록

  • 2026-08-02: provider tunnel의 긴 prompt prefill과 streaming backpressure를 정상 traffic으로 허용하도록 Edge/Node heartbeat profile을 30초 interval/45초 wait로 복원한 현재 구현과 회귀 검증을 반영했다 (apps/edge/internal/transport/server.go, apps/node/internal/transport/client.go).
  • 2026-08-04: provider response-stall timeout의 config validation, selected-candidate propagation, Node adapter-visible retention, and activity classification contract를 반영했다.
  • 2026-08-04: Added the shared Node run/tunnel watchdog coordinator, serialized tunnel emission fence, pre-provider admission cleanup, disconnect-bound handler lifetime, and deterministic S01/S02 manual-clock evidence. Provider health probing and Edge-owned recovery remain future slices.
  • 2026-08-04: Joined the bounded close-grace fence and the independent exact-target health probe into one stall terminal carrying three-way health evidence, and added the connection-scoped health_observation_seq sourced from the transport Session. Edge reception-generation binding, stale validation, Edge health overlay, recovery, and candidate selection remain future slices.