toki
dc9a9a8c59
승인된 execution preset을 Edge 조정 경계와 Node workspace/tool 실행 경계로 연결해 단일 요청 수명주기와 관측 계약을 일관되게 처리한다.
305 lines
35 KiB
Markdown
305 lines
35 KiB
Markdown
---
|
|
spec_doc_type: spec
|
|
spec_id: runtime/edge-node-execution
|
|
status: 구현됨
|
|
source_evidence:
|
|
- type: contract
|
|
path: agent-contract/inner/execution-runtime.md
|
|
notes: Host-neutral provider execution primitives
|
|
- type: contract
|
|
path: agent-contract/inner/edge-node-runtime-wire.md
|
|
notes: Edge-Node registration, execution, tunnel, cancellation, command, and refresh wire
|
|
- type: code
|
|
path: packages/go/execution/types.go
|
|
notes: Provider execution and event types
|
|
- type: code
|
|
path: packages/go/execution/liveness.go
|
|
notes: Response-stall timeout default, validation, and RuntimeEvent/ProviderTunnelFrame activity classifiers
|
|
- type: code
|
|
path: apps/node/internal/node/runtime_bridge.go
|
|
notes: Protobuf-to-execution translation with raw stall timeout validation before router/provider invocation
|
|
- type: code
|
|
path: apps/edge/internal/transport/server.go
|
|
notes: Edge-side tunnel-tolerant heartbeat and disconnect supervision
|
|
- type: code
|
|
path: apps/node/internal/transport/client.go
|
|
notes: Node-side tunnel-tolerant heartbeat and reconnect transport
|
|
- type: code
|
|
path: apps/edge/internal/service/provider_tunnel.go
|
|
notes: Provider selection, credential binding validation, reception-aware terminal handoff, lease acquisition, and pre-send fencing
|
|
- type: code
|
|
path: apps/edge/internal/service/model_queue_release.go
|
|
notes: Immutable lease validation, generation/sequence-fenced runtime health overlay, recovery handoff annotation, and exactly-once release
|
|
- type: code
|
|
path: apps/edge/internal/service/node_command.go
|
|
notes: CAPABILITIES dispatch identity retention and exact available recovery evidence application
|
|
- type: code
|
|
path: apps/node/internal/node/tunnel_handler.go
|
|
notes: Provider tunnel handling and recipient-sealed credential lease consumption
|
|
- type: code
|
|
path: apps/node/internal/node/liveness_watchdog.go
|
|
notes: Shared normalized/tunnel stall coordination, close-grace ownership, serialized emission fencing, bounded probe/fence join, and connection-scoped observation sequencing
|
|
- type: code
|
|
path: apps/node/internal/node/health_probe.go
|
|
notes: Bounded independent exact-target health probe coordinator consumed by the stall terminal join
|
|
- type: code
|
|
path: apps/node/internal/transport/session.go
|
|
notes: Connection-scoped monotonic health-observation sequence source
|
|
- type: code
|
|
path: packages/go/credentiallease/envelope.go
|
|
notes: Signed scope validation, recipient sealing, expiry, replay, and exact binding verification
|
|
- type: test
|
|
path: apps/node/internal/node/command_test.go
|
|
notes: Closed provider commands plus fail-closed exact CAPABILITIES health and Session sequence regressions
|
|
- type: test
|
|
path: apps/edge/internal/service/provider_health_overlay_test.go
|
|
notes: S04 binding, stale evidence, normalized/tunnel release races, overlay projection, and CAPABILITIES recovery evidence
|
|
- type: test
|
|
path: apps/edge/internal/openai/stream_gate_stall_recovery_test.go
|
|
notes: S05 always-owned OpenAI recovery, new attempt/provider selection, shared budget, old-transport close, and guard terminals
|
|
- type: test
|
|
path: apps/edge/internal/transport/heartbeat_test.go
|
|
notes: Edge heartbeat liveness profile regression
|
|
- type: test
|
|
path: apps/node/internal/transport/heartbeat_test.go
|
|
notes: Node heartbeat liveness and idle-connection regressions
|
|
- type: test
|
|
path: apps/node/internal/node/liveness_watchdog_test.go
|
|
notes: Manual-clock S01/S02 threshold, progress, terminal, close-grace, ownership, metadata, and late-output evidence
|
|
- type: test
|
|
path: apps/node/internal/node/provider_tunnel_test.go
|
|
notes: Credential preflight admission release regression
|
|
- type: test
|
|
path: apps/node/internal/transport/session_test.go
|
|
notes: Run and tunnel handler lifetime cancellation on disconnect
|
|
- type: code
|
|
path: apps/edge/internal/service/single_request_workspace.go
|
|
notes: Exact configured workspace owner and ready-generation admission projection
|
|
- type: code
|
|
path: apps/edge/internal/service/workspace_wire.go
|
|
notes: Exact-generation dispatch plus frozen request-authority construction and stable failure translation
|
|
- type: code
|
|
path: apps/edge/internal/service/single_request_tool_types.go
|
|
notes: Closed internal workspace schemas, strict decoding, defensive copies, and raw-free typed result projection
|
|
- type: code
|
|
path: apps/edge/internal/service/single_request_tool_loop.go
|
|
notes: Request-local ordered tool continuation, saved-stage correlation, immutable budgets, and cancellation ownership
|
|
- type: test
|
|
path: apps/edge/internal/service/single_request_tool_loop_test.go
|
|
notes: Ordered multi-tool wire evidence plus identity, capability, stale result, budget, deadline, and cancel failures
|
|
- type: code
|
|
path: apps/node/internal/transport/session.go
|
|
notes: Optional workspace handler registration that preserves legacy provider Handler compatibility
|
|
- type: code
|
|
path: apps/node/internal/workspace/runtime.go
|
|
notes: Darwin-only immutable catalog, opened root authority, operation-aware limits, immutable request-authority copy, and lifecycle ownership
|
|
- type: code
|
|
path: apps/node/internal/workspace/file_executor.go
|
|
notes: Capability-gated bounded batch listing, descriptor-relative structured write, and non-recursive delete
|
|
- type: test
|
|
path: apps/node/internal/workspace/file_executor_test.go
|
|
notes: Reserved namespace, no-effect symlink/parent/device rejection, bounded listing, atomicity, special-file, and concurrency regressions
|
|
- type: code
|
|
path: apps/node/internal/workspace/command_executor.go
|
|
notes: Exact command-template lookup, minimal allowlisted environment, shared output cap, active-command identity, and terminal result ownership
|
|
- type: code
|
|
path: apps/node/internal/workspace/command_process_unix.go
|
|
notes: Darwin/Linux inherited-root fchdir/exec shim and process-group termination
|
|
- type: code
|
|
path: apps/node/internal/workspace/cleanup.go
|
|
notes: Exactly-once request cleanup ownership, process cancellation and wait, bounded result cache, and internal artifact inventory
|
|
- type: code
|
|
path: apps/node/internal/workspace/cleanup_path_unix.go
|
|
notes: No-follow request namespace creation, descriptor enumeration, identity validation, and deepest-first non-recursive removal
|
|
- type: test
|
|
path: apps/node/internal/workspace/cleanup_test.go
|
|
notes: Cleanup races, process groups, timeout, unsafe entry refusal, identity and device mismatch, user result preservation, and request isolation
|
|
- type: test
|
|
path: apps/node/internal/workspace/command_executor_test.go
|
|
notes: Success, non-zero exit, timeout, context/explicit cancel, child process group, shared output, environment, request isolation, and renamed-root identity evidence
|
|
- type: test
|
|
path: apps/node/internal/node/workspace_handler_test.go
|
|
notes: Typed command/cancel mapping, duplicate cancel, not-found, and raw-free stable error evidence
|
|
- type: test
|
|
path: apps/edge/internal/service/single_request_workspace_test.go
|
|
notes: Workspace admission rejection, effective-limit, refresh, and generation-fence regressions
|
|
- type: test
|
|
path: apps/edge/internal/service/workspace_wire_test.go
|
|
notes: Frozen open authority, typed workspace round trips, cancellation, and stale-generation no-reselection regressions
|
|
- type: test
|
|
path: apps/edge/internal/service/single_request_cleanup_test.go
|
|
notes: Cleanup-before-terminal ordering, success failure conversion, cancellation category preservation, write failure, unopened workspace, and exactly-once terminal races
|
|
- type: test
|
|
path: apps/node/internal/bootstrap/workspace_runtime_test.go
|
|
notes: Path-free startup failure, handler-before-ready composition, and registry/workspace/session/store close-order regressions
|
|
- type: code
|
|
path: apps/node/internal/node/liveness_observability.go
|
|
notes: Node stall counter/histogram and dedicated structured log with closed label values and raw-payload exclusion
|
|
- type: test
|
|
path: apps/node/internal/node/liveness_observability_test.go
|
|
notes: Deterministic S06 Node stall observation regression with closed label values
|
|
- type: code
|
|
path: apps/edge/internal/service/provider_health_observability.go
|
|
notes: Edge overlay evidence/transition counters and dedicated structured log with closed label values and identity exclusion
|
|
- type: test
|
|
path: apps/edge/internal/service/provider_health_observability_test.go
|
|
notes: Deterministic S06 Edge overlay observation regression including sentinel exclusion via TestProviderHealthObservabilityDoesNotExposeSentinels
|
|
- type: code
|
|
path: apps/edge/internal/openai/liveness_recovery_observability.go
|
|
notes: Edge OpenAI eligibility/results counters and dedicated structured log with closed label values and identifier exclusion
|
|
- type: test
|
|
path: apps/edge/internal/openai/liveness_recovery_observability_test.go
|
|
notes: Deterministic S06 OpenAI recovery observation regression with closed label values
|
|
---
|
|
|
|
# Edge-Node Provider Execution
|
|
|
|
## 목적
|
|
|
|
Edge owns provider selection, queue admission, leases, and connection-generation fencing. Node owns local provider adapters and executes normalized runs or provider HTTP tunnels after a ready handshake.
|
|
|
|
The shared `packages/go/execution` package contains provider lifecycle, registry, execution events, failures, cancellation, usage, and optional tunnel/command primitives. It does not manage host programs or durable conversation state.
|
|
|
|
## 기능 목록
|
|
|
|
| 기능 | 설명 |
|
|
|------|------|
|
|
| register/readiness | 등록된 Node의 현재 connection이 readiness를 완료한 뒤에만 dispatch한다. |
|
|
| normalized execution | `adapter + target`으로 provider 실행을 선택하고 ordered `RunEvent` stream을 반환한다. |
|
|
| single-request coordinator | Immutable admission과 closed stage envelope을 service-owned state graph (`accepted`, `planning`, `working`, `reviewing`, `repairing`, `internal_tool`, `finalizing`, `completed`, `failed`, `cancelled`)로 처리한다. An internal tool result can resume only its saved stage. After a successful workspace open, every terminal path waits for one cleanup before the finalizing candidate can reach surface acknowledgement. |
|
|
| single-request observation evidence | Stage-pure timing, tool/cleanup/total counts, cardinality-bounded labels, Node logs, and raw-free correlation are documented for the single-request path. `iop_anthropic_single_request_ingress_total` is strictly unlabeled. Internal tool names, raw arguments, and private results are absent from public output and log projections. Actual Claude/Mac timing evidence is explicitly deferred to `claude-smoke` (SDD S12); deterministic coordinator/tool-loop tests do not imply external qualification. |
|
|
| workspace admission | An opaque `workspace_ref` resolves only through the configured Node catalog. Edge freezes the exact configured owner, dispatch-ready connection generation, closed operation/command/environment-name capabilities, and effective limits before executor startup; unavailable, foreign, pending, malformed, and stale candidates fail closed without fallback or reselection. |
|
|
| workspace runtime wire | The dedicated `WorkspaceOpen`/`Tool`/`Cancel`/`Cleanup` request-response families carry immutable coordinator identities and closed status/error codes. Edge overwrites open capabilities with frozen request authority; Node copies only catalog-subset operations/command ids and equal-or-lower effective limits. |
|
|
| workspace tool executor | A validated Darwin Node catalog owns opened root and directory handles. Go 1.24-compatible no-follow file primitives provide bounded read, bounded list, structured write, and non-recursive delete. Exact operator-owned command templates run through an inherited-root `fchdir`/`exec` shim with minimal allowlisted environment, shared stdout/stderr bounds, process-group timeout/cancel, and stable typed results. |
|
|
| internal workspace tool loop | The service decodes only `workspace_read`, `workspace_list`, `workspace_write`, `workspace_delete`, and `workspace_command`, opens the admitted workspace once, dispatches one call at a time on the frozen generation, and delivers one deep-copied typed result to the emitting executor continuation. Unique request/stage/tool correlation, per-stage iteration/output/deadline limits, request wall-clock budget, and typed cancel fail closed without external continuation or reselection. |
|
|
| request-owned cleanup | Node creates and inventories only `.iop/job/<request_id>` internal state, cancels and waits for all active command groups, validates the exact tree without following entries, and removes matching artifacts deepest-first with non-recursive descriptor operations. Symlinks, special files, foreign devices, identity replacements, and unowned entries fail closed. User results and sibling request state are preserved. Concurrent cleanup callers receive one bounded cached typed result. |
|
|
| provider raw tunnel | 선택된 provider의 HTTP/SSE를 `ProviderTunnelRequest`/`ProviderTunnelFrame`으로 relay하며 순서와 단일 terminal outcome을 보장한다. |
|
|
| response-stall activity contract | 선택된 provider의 response-stall timeout을 normalized/tunnel request에 보존한다. Node는 wire zero를 `300000ms`로 해석하고 invalid raw value를 adapter 호출 전에 거부한다. Runtime event의 terminal type은 payload/usage보다 우선하며 non-terminal usage는 progress다. |
|
|
| Node stall watchdog | Node가 normalized run과 raw tunnel에 하나의 activity watchdog을 적용한다. progress만 timer를 reset하며, stall은 `response_stalled` terminal 하나와 Node-owned safe metadata를 만들어 normalized `RunEvent`와 raw `ProviderTunnelFrame` wire의 optional typed `ExecutionFailure` 필드에 싣는다. stall claim 뒤에는 bounded close grace fence와 독립 exact-target health probe를 직렬 확장 없이 join한다. close grace 안에 provider return이 확인된 경우만 `Retryable` capability hint를 준다. |
|
|
| Node health evidence join | stall terminal에 three-way health evidence를 싣는다: `provider_health` status와 `liveness_classification` normalization이 `available`/`request_stalled`, `unavailable`/`provider_unhealthy`, `unknown`/`health_unknown` 쌍으로 fail-closed된다. probe 성공은 progress reset·fence 변경·retry authority가 아니며 late output은 fenced 상태를 유지한다. |
|
|
| health observation sequence | transport Session이 connection-scoped monotonic `health_observation_seq`를 소유한다. 새 connection은 0에서 시작해 첫 finalized observation이 1이며, 같은 connection의 normalized/tunnel observation이 source를 공유해 동시에도 유일 증가값을 받는다. internal/unbound 경로는 key를 생략한다. |
|
|
| Edge terminal health handoff | Edge validates authoritative reception node/generation plus the immutable provider/adapter/target lease before applying typed stall evidence. Every validated current bound stall receives `provider_id`, validated health, and `recovery_handoff=confirmed`, while only fresh unavailable evidence lowers a separate runtime overlay; the token never grants replay eligibility. Every valid current terminal still releases its lease exactly once. |
|
|
| CAPABILITIES recovery | Node runs the same bounded exact-target `ProbeHealth` and returns stable adapter/target/status plus the next Session sequence. Edge recovers exactly one matching current-generation unavailable provider only from a strictly newer `available` result; malformed, ambiguous, stale, unknown, and unavailable responses are no-ops. |
|
|
| recovery candidate preference | `ProviderPoolDispatchRequest` carries `AvoidProviderID` and `AllowAvoidedProviderFallback`. Every admission (initial and queued re-resolution) prefers a runtime-eligible alternate over the avoided provider; only the explicit fallback flag (derived from exact probe-backed `available` evidence) permits re-selecting the avoided provider when no alternate exists. Zero values preserve current selection. This is selection policy only: no retry loop, slot reservation, priority change, persistence, or retry counter. |
|
|
| OpenAI typed-stall consumption | Every supported Chat/Responses normalized or tunnel request has one unconditional runtime liveness owner, independent of configured semantic activation. It converts only the Edge-confirmed typed stall handoff into a raw-free StreamGate event, owns pre-commit eligibility, and closes the already fenced old transport before re-admission; Node does not grant replay authority. |
|
|
| tunnel-tolerant liveness | Edge와 Node는 30초 heartbeat interval과 45초 response wait를 공통으로 사용해 긴 prompt prefill이나 streaming backpressure 중의 정상 connection을 조기에 끊지 않는다. |
|
|
| reconnect/generation fencing | 현재 connection이 종료되면 해당 generation만 fence하고 Node supervisor가 reconnect한다. Heartbeat wait를 넘긴 경우의 close reason은 `heartbeat_timeout`이다. |
|
|
| cancellation/command | `run_id`로 현재 run만 취소하며 command는 capabilities, transport status, Ollama API tunnel로 제한한다. |
|
|
| managed credential lease | Edge가 principal·route·slot·profile·target·Node·revision·generation을 binding한 sealed lease를 발급하고 Node가 capacity admission 후 provider 실행 직전에만 연다. |
|
|
|
|
## 범위
|
|
|
|
- `session_id`는 event와 command result의 opaque correlation일 뿐이며 같은 값을 재사용해도 모든 run은 독립적이다.
|
|
- provider usage, capacity, queue pressure, lifecycle, reconnect, tool calling은 Edge-Node 실행 경로에서 계속 지원한다.
|
|
- single-request coordinator owns the service-level workspace admission described above as well as executor envelope privacy and the service-owned state graph. It exposes no workspace root, command executable/template/arguments, or environment values to the coordinator-facing binding.
|
|
- The request-local internal tool loop is implemented between the coordinator and the dedicated workspace wire. Strict decode and capability checks happen before wire effects; Node results are accepted only for the one pending call and return only bounded typed fields to the same optional executor continuation. Repeated or stale identities, malformed/denied calls, exhausted immutable budgets, and cancellation terminate internally without selecting another Node or involving the HTTP caller.
|
|
- The Node-private workspace request/result wire is implemented, including catalog delivery, parser registration, optional handler behavior, stable typed failures, generation-fenced dispatch, context-cancel propagation, and request cleanup. The Node validates the Darwin catalog before ready, installs the workspace handler before ready, and cleans active requests before closing workspace authority ahead of session/store teardown. Request authority is immutable and request-local. File operations reserve `.iop`, reject symlink/mount/replaced-parent/special-file paths before effects, process bounded list batches with deterministic truncation, and use a same-parent structured write. Command execution resolves only admitted ids to fixed templates, enters the already-opened root descriptor through `fchdir`, provides only allowlisted environment entries, shares one output cap across drained stdout/stderr, and owns the complete process group through exit, timeout, context cancel, exact request/tool cancel, or request cleanup.
|
|
- managed mode는 등록과 dispatch 전에 CA로 검증된 Edge/Node workload identity를 요구한다.
|
|
- revoked, disabled, expired, stale, replayed, wrong-recipient, mismatched lease는 provider나 credential fallback 없이 fail closed한다.
|
|
|
|
IOP no longer provides persistent shell sessions, terminal emulation, process resume, local working-directory execution context, arbitrary host commands, or local quota/status probing.
|
|
|
|
The current spec maps reviewed Node and Edge observability producers to S06 behavior and deterministic tests. Node exposes bounded stall counters/histograms and dedicated structured logs with closed label values and raw-payload exclusion. Edge service queue exposes bounded overlay evidence/transition counters and dedicated structured logs with closed label values and identity exclusion. Edge OpenAI server exposes bounded eligibility/results counters and dedicated structured logs with closed label values and identifier exclusion. All projections are local observations and do not widen the wire protocol.
|
|
|
|
Single-request lifecycle observation evidence links ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, and raw-free correlation for one real POST. `iop_anthropic_single_request_ingress_total` is strictly unlabeled (no request_id, stage_id, provider identity, content, or workspace reference). Internal tool names, raw arguments, and private results are absent from public output and log projections. Stage-pure timing, cardinality-bounded labels, and privacy semantics are documented here. Actual Claude/Mac timing evidence is explicitly deferred to `claude-smoke` (SDD S12); deterministic coordinator/tool-loop tests do not imply external qualification.
|
|
|
|
## 주요 흐름
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant Edge
|
|
participant Node
|
|
participant Provider
|
|
|
|
Node->>Edge: RegisterRequest
|
|
Edge-->>Node: RegisterResponse + config
|
|
Node->>Edge: NodeReadyRequest
|
|
Edge-->>Node: NodeReadyResponse
|
|
opt admitted single-request internal workspace call
|
|
Edge->>Node: WorkspaceOpenRequest once (frozen generation)
|
|
Node-->>Edge: WorkspaceOpenResponse
|
|
loop one ordered pending call
|
|
Edge->>Node: WorkspaceToolRequest(request, stage, tool)
|
|
Node-->>Edge: bounded typed WorkspaceToolResponse
|
|
end
|
|
Edge->>Node: WorkspaceCleanupRequest once before terminal commit
|
|
Node->>Node: cancel/wait request process groups and validate inventory
|
|
Node-->>Edge: typed WorkspaceCleanupResponse
|
|
Note over Edge: expose finalizing only after successful cleanup
|
|
Note over Edge: observation: ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation
|
|
end
|
|
Edge->>Node: ProviderTunnelRequest
|
|
Node->>Provider: HTTP/SSE request
|
|
Provider-->>Node: status/header/body stream
|
|
Node-->>Edge: ordered ProviderTunnelFrame stream
|
|
Note over Edge,Node: heartbeat 30s interval / 45s wait
|
|
alt heartbeat wait exceeded
|
|
Edge--xNode: current generation fenced
|
|
Node->>Edge: supervised reconnect
|
|
end
|
|
```
|
|
|
|
## 계약
|
|
|
|
- Edge-Node wire: `agent-contract/inner/edge-node-runtime-wire.md`
|
|
- provider execution primitives: `agent-contract/inner/execution-runtime.md`
|
|
|
|
Heartbeat interval/wait는 protobuf field가 아닌 양쪽 transport 구현의 liveness profile이다. `response_stall_timeout_ms`만 provider execution request wire에 추가되며 provider response shape은 바뀌지 않는다.
|
|
|
|
## 설정/데이터/이벤트
|
|
|
|
- Edge와 Node의 현재 heartbeat interval은 30초, response wait는 45초다.
|
|
- 이 값은 runtime YAML model config나 `max_tokens`/context 설정이 아니라 transport 구현 상수다.
|
|
- 45초 동안 heartbeat response가 없으면 current connection을 `heartbeat_timeout`으로 닫고 provider resource를 offline 처리한 뒤 reconnect/queue 재평가를 수행한다.
|
|
- response-stall timeout은 provider config가 source이며 winning candidate가 re-resolution된 뒤의 request까지 같은 effective value를 보존한다. request hard timeout, queue timeout, transport heartbeat, client response-idle timeout과 timer lifecycle은 별도 소유권이다.
|
|
|
|
## 검증
|
|
|
|
- `go test -count=1 ./packages/go/execution ./apps/node/... ./apps/edge/internal/service`
|
|
- `go test -race -count=1 ./packages/go/execution ./apps/node/internal/node ./apps/edge/internal/service`
|
|
- `go test -race -count=1 ./apps/edge/internal/node -run 'TestRegistryReadyOwnerSnapshot'`
|
|
- `go test -race -count=1 ./apps/edge/internal/service -run 'TestSingleRequestWorkspace'`
|
|
- `go test -race -count=1 ./apps/edge/internal/service -run 'Test(InternalWorkspaceTool|SingleRequestInternalToolLoop)'`
|
|
- `go test -race -count=1 ./apps/edge/internal/node ./apps/edge/internal/service ./apps/edge/internal/transport ./apps/node/internal/transport -run 'Test(BuildConfigPayload.*Workspace|WorkspaceWire|NodeParserMapWorkspace|SessionWorkspace|EdgeParserMapWorkspace)'`
|
|
- `go test -race -count=1 ./apps/node/internal/workspace -run 'Test(CommandExecutor|WorkspaceCommandHelperProcess)'`
|
|
- `go test -race -count=1 ./apps/node/internal/node -run 'TestNodeWorkspace(Command|Cancel)'`
|
|
- `go test -race -count=1 ./apps/node/internal/workspace -run 'TestWorkspaceCleanup'`
|
|
- `go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/service -run 'Test(NodeWorkspaceCleanup|SingleRequestCleanup)'`
|
|
- `go test -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport`
|
|
- `go test -race -count=1 ./apps/node/internal/transport ./apps/edge/internal/transport`
|
|
- 실제 provider tunnel 검증은 5초를 넘는 긴 prefill과 streaming 응답 동안 Node가 connected/healthy를 유지하고, 응답이 정상 terminal을 반환하며, `heartbeat_timeout`이 발생하지 않는지 확인한다.
|
|
- `go test -count=1 ./apps/node/internal/node -run '^TestNodeLivenessObservability'` — deterministic Node stall observation with closed label values and raw-payload exclusion.
|
|
- `go test -count=1 ./apps/edge/internal/service -run '^TestProviderHealthObservability'` — deterministic Edge overlay evidence/transition with closed label values and identity exclusion; `TestProviderHealthObservabilityDoesNotExposeSentinels` covers the sentinel/prohibited-value guard.
|
|
- `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAILivenessObservationSink|TestOpenAILivenessRecoveryObservability)$'` — deterministic OpenAI recovery eligibility/results with closed label values and identifier exclusion.
|
|
- `go test -count=1 ./apps/edge/internal/openai -run 'TestAnthropicSingleRequestObservation'` — deterministic single-request observation evidence: ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation, and unlabeled metric assertion.
|
|
|
|
## 한계와 주의사항
|
|
|
|
- 30/45초 liveness profile은 provider 응답 token 상한이나 model context window를 늘리지 않는다. 요청 중단 원인 판정 시 model 설정과 transport disconnect를 별도로 확인한다.
|
|
- 45초를 넘겨 실제 heartbeat response가 없는 connection은 기존과 같이 오프라인 처리하고 reconnect한다.
|
|
- Node owns local detection, cancellation, emission fencing, confirmed/unconfirmed ownership close, exact-target probe joining, connection-scoped observation sequencing, and the bounded `iop_node_response_stalls_total` / `iop_node_response_stall_duration_seconds` / `node_response_stall_observation` projections with closed label values.
|
|
- Edge owns reception-generation and immutable-lease validation, the generation-scoped runtime health overlay, `iop_edge_provider_health_evidence_total` / `iop_edge_provider_health_transitions_total` / `edge_provider_health_observation` projections with closed label values, effective admission/snapshot projection, and exact later CAPABILITIES recovery.
|
|
- The always-owned supported OpenAI ingress runtime owns commit, cancellation, side-effect, snapshot, shared-budget, candidate, and replay decisions, and exposes `iop_edge_liveness_recovery_eligibility_total` / `iop_edge_liveness_recovery_results_total` / `edge_liveness_recovery_observation` projections with closed label values.
|
|
- Node retry and `recovery_eligible` remain prohibited. Hard deadline and connection disconnect continue to take precedence over a simultaneous stall timer.
|
|
- Operational projections never widen the wire protocol; they carry no new frame, field, ordering rule, or retry semantic.
|
|
- Workspace admission and the private wire both fence the exact ready connection generation. The wire never exposes workspace fields through provider `RunRequest`, `NodeCommand`, or public API output. The executor exposes no caller access to `.iop`; only request-owned internal runtime code can derive and inventory `.iop/job/<request_id>`. Structured write input is required for WRITE, while legacy content-only input remains rejected. COMMAND is non-interactive and has no shell, PTY, arbitrary argv, ambient environment, path-based cwd lookup, or persistent process ownership. Cleanup never rolls back or deletes user-requested workspace results.
|
|
- The service-owned internal loop does not implement provider-specific plan/work/review prompts or repair policy. Those drivers and actual Claude qualification remain separate work even though canonical Node tool continuation and cleanup ordering are implemented.
|
|
- Single-request observation evidence (ingress=1, request-total=1, terminal=1, stage/tool/cleanup counts, raw-free correlation) is documented and tested. `iop_anthropic_single_request_ingress_total` is strictly unlabeled. Internal tool names, raw arguments, and private results are absent from public output and log projections. Actual Claude/Mac timing evidence is explicitly deferred to `claude-smoke` (SDD S12); deterministic coordinator/tool-loop tests do not imply external qualification.
|
|
|
|
## 변경 기록
|
|
|
|
- 2026-08-02: provider tunnel의 긴 prompt prefill과 streaming backpressure를 정상 traffic으로 허용하도록 Edge/Node heartbeat profile을 30초 interval/45초 wait로 복원한 현재 구현과 회귀 검증을 반영했다 (`apps/edge/internal/transport/server.go`, `apps/node/internal/transport/client.go`).
|
|
- 2026-08-04: provider response-stall timeout의 config validation, selected-candidate propagation, Node adapter-visible retention, and activity classification contract를 반영했다.
|
|
- 2026-08-04: Added the shared Node run/tunnel watchdog coordinator, serialized tunnel emission fence, pre-provider admission cleanup, disconnect-bound handler lifetime, and deterministic S01/S02 manual-clock evidence.
|
|
- 2026-08-04: Joined the bounded close-grace fence and the independent exact-target health probe into one stall terminal carrying three-way health evidence, and added the connection-scoped `health_observation_seq` sourced from the transport Session.
|
|
- 2026-08-05: Added authoritative Edge terminal handoff, immutable lease binding, generation/sequence-fenced runtime provider health, exactly-once normalized/tunnel release, and fail-closed Session-sequenced CAPABILITIES recovery without config-health mutation or replay authorization.
|
|
- 2026-08-05: Added runtime-local OpenAI consumption of confirmed typed stalls, including cancel-free old-transport close and provider-pool avoidance hints for ExactReplay.
|
|
- 2026-08-05: Made supported OpenAI Chat/Responses normalized and tunnel liveness ownership unconditional and added S05 recovery/guard evidence independent of semantic policy activation.
|
|
- 2026-08-06: Mapped reviewed Node, Edge overlay, and OpenAI recovery observability producers to S06 behavior with deterministic test evidence. Node exposes `iop_node_response_stalls_total`, `iop_node_response_stall_duration_seconds`, and `node_response_stall_observation` (source: `apps/node/internal/node/liveness_observability.go`; test: `TestNodeLivenessObservability`). Edge service queue exposes `iop_edge_provider_health_evidence_total`, `iop_edge_provider_health_transitions_total`, and `edge_provider_health_observation` (source: `apps/edge/internal/service/provider_health_observability.go`; test: `TestProviderHealthObservability`, `TestProviderHealthObservabilityDoesNotExposeSentinels`). Edge OpenAI server exposes `iop_edge_liveness_recovery_eligibility_total`, `iop_edge_liveness_recovery_results_total`, and `edge_liveness_recovery_observation` (source: `apps/edge/internal/openai/liveness_recovery_observability.go`; test: `TestOpenAILivenessObservationSink`, `TestOpenAILivenessRecoveryObservability`). All projections carry only closed, low-cardinality label values and exclude raw payloads, credentials, and unbounded identifiers from metric labels and general logs. The wire protocol is unchanged.
|
|
- 2026-08-06: Added implemented Edge workspace admission for single-request execution: an opaque `workspace_ref` binds to one configured ready Node generation and a closed capability projection before executor startup. Admission rejects unavailable, foreign, pending, malformed, and stale candidates without fallback or reselection; Node-private wire, executor, filesystem path, and symlink enforcement remain deferred.
|
|
- 2026-08-06: Added the dedicated Edge-Node workspace wire. `NodeConfigPayload` now delivers the approved catalog; `WorkspaceOpen`/`Tool`/`Cancel`/`Cleanup` messages have closed typed outcomes, immutable coordinator identities, parser registration, and an optional Node handler. Edge dispatch is generation-fenced and context cancellation sends one typed cancel. Node filesystem and process execution are intentionally deferred.
|
|
- 2026-08-06: Completed the reviewed workspace file boundary repair. Edge now sends only frozen request authority, Node admits immutable catalog subsets/lower limits, and structured write reaches the file executor while legacy incomplete input remains rejected. The Go 1.24-compatible descriptor-relative no-follow write path validates before effects, bounded list processing retains fixed state, startup errors are path-free, and composition proves handler-before-ready plus workspace-before-session/store teardown. Command execution/cancellation and cleanup remain deferred.
|
|
- 2026-08-07: Implemented exact-template workspace COMMAND and typed cancellation. The Node uses an inherited-root `fchdir`/`exec` shim, minimal allowlisted environment, a shared draining stdout/stderr cap, and one process-group result owner across exit, timeout, context cancel, and exact request/tool cancel. Focused race tests cover non-zero exit, output overflow, descendant termination, cross-request isolation, and configured-root rename/replacement. Artifact cleanup remains deferred.
|
|
- 2026-08-07: Implemented the coordinator-owned internal workspace tool loop with closed strict schemas, one-time exact-generation open, ordered pending-call correlation, deep-copied raw-free continuation results, immutable iteration/output/deadline budgets, typed cancellation, and real one-POST multi-tool privacy evidence.
|
|
- 2026-08-07: Added request-owned workspace cleanup. Node inventories its exact internal request namespace and artifacts, cancels and waits for all request command groups, refuses unowned, symlink, special-file, identity, and filesystem-boundary mismatches, and removes only validated entries with no-follow non-recursive descriptor operations. Edge gates every opened-workspace terminal path on one typed cleanup before finalizing acknowledgement; cleanup failure converts pending success while preserving existing failure or cancellation categories.
|
|
- 2026-08-08: Synchronized single-request lifecycle observation evidence: stage-pure timing (planning/working/reviewing/repairing/finalizing/completed/failed/cancelled), tool/cleanup/total counts, cardinality-bounded labels, Node logs, and raw-free correlation. External Claude/Mac timing evidence is explicitly deferred to `claude-smoke` (SDD S12). Deterministic coordinator/tool-loop tests cover the full single-request path without implying external qualification.
|