test(openai): 관리형 용량 검증을 보강한다
Ornith 실환경 동시성 및 스트림 terminal 증적을 자동 검증한다.\n배포 스킬과 dev 테스트 문서, 로컬 캐시 ignore 정책을 함께 동기화한다.
This commit is contained in:
parent
2fcc1093c7
commit
abecea20a9
14 changed files with 2997 additions and 14 deletions
3
.gitignore
vendored
3
.gitignore
vendored
|
|
@ -13,6 +13,9 @@ agent-test/runs/
|
|||
/iop.db
|
||||
/*.log
|
||||
/.cache/
|
||||
/.gocache/
|
||||
/.local/
|
||||
/.tmp/
|
||||
**/__pycache__/
|
||||
*.py[cod]
|
||||
/build/
|
||||
|
|
|
|||
|
|
@ -140,19 +140,19 @@ dev-runtime provider pool을 `dev` 기준 git-flow release로 배포한다. `dev
|
|||
- 각 node의 `provider_snapshots`에서 provider `id`, `capacity`, `in_flight`, `queued`, `health`, `served_models`를 확인한다.
|
||||
- `/v1/models`가 대상 model alias를 노출하는지 확인한다.
|
||||
|
||||
9. **OpenAI-compatible capacity smoke**
|
||||
- `/v1/responses`와 `/v1/chat/completions`를 각각 검증한다. legacy `/v1/completions`는 구현되어 있지 않으면 실패로 보지 않는다.
|
||||
- 표준 부하 프롬프트는 짧은 토큰 응답을 요구하지 않는다. 700~1200 token 수준의 구조화된 답변을 유도해 요청이 동시에 관측될 시간을 만든다.
|
||||
- endpoint별로 선택한 model group의 총 provider capacity + 1개 요청을 동시에 보낸다. 현재 Laguna `laguna-s:2.1`은 GX10 capacity `4`이므로 5개, Ornith `ornith:35b`는 OneXPlayer `3` + RTX5090 `1`이므로 5개, Qwen `qwen3.6:35b`는 mac-mlx-vllm `2`이므로 3개 동시 호출이다.
|
||||
- 환경 문서에 더 작은 aggregate smoke 기준이 남아 있더라도, 배포 완료 판정은 이 capacity+1 기준을 우선한다.
|
||||
- 요청 실행 중 Control Plane status를 반복 polling하여 대상 provider들의 `in_flight` 합이 총 capacity에 도달하고 `queued` 합이 1 이상이 되는 순간을 증거로 남긴다.
|
||||
- 각 provider의 `in_flight`가 자기 capacity를 넘지 않고, 적어도 한 번은 기대 capacity까지 차는지 확인한다.
|
||||
- 모든 요청 완료 후 같은 status에서 대상 provider들의 `in_flight=0`, `queued=0`으로 돌아오는지 확인한다.
|
||||
- Qwen과 Laguna reasoning/thinking 텍스트는 정상 응답으로 허용한다. Laguna think smoke는 같은 요청의 Pi `high`에서 `thinking_start`/`thinking_delta`/`thinking_end`, `off`에서 thinking event 0개와 최종 text를 대조한다. agentic smoke는 tool-call 전후 reasoning, tool result, 최종 text를 모두 확인한다. exact-output match를 smoke 성공 기준으로 삼지 않는다.
|
||||
- Pi/Cline형 agent/tool-call 경계를 검증할 때는 forced tool call, auto tool call, streaming `delta.tool_calls`, multi-turn tool result 후 최종 답변을 provider direct와 Edge OpenAI-compatible 경로에서 나눠 확인한다. raw native marker나 reasoning text가 assistant content로 새면 해당 model/runtime의 parser/template profile 미확정으로 보고한다.
|
||||
9. **Run the OpenAI-compatible capacity smoke**
|
||||
- Qualify `/v1/responses` and `/v1/chat/completions` separately. Do not fail solely because the unimplemented legacy `/v1/completions` endpoint is absent.
|
||||
- Use a short request that asks for a structured 700-1200-token answer and explicitly bounds provider-native thinking. Compute the emitted JSON Unicode rune count, `runes/4 + runes/16` estimate, and context class. Do not reuse long-context or repetition fixtures as a normal-capacity oracle.
|
||||
- In managed mode, authenticate `GET /v1/credentials/routes` with the same principal token used at OpenAI ingress. Require one active exact route alias, then intersect its `resource_selector`, profile, and upstream model with one healthy connected provider snapshot.
|
||||
- Run `scripts/e2e-openai-managed-capacity-smoke.sh` for one endpoint and route at a time. Require the emitted request to classify as `normal`, then send the selected provider's `capacity` plus one request. If it classifies as `long`, fail this normal-capacity gate and use the separate long-context admission smoke with `long_context_capacity`; never add capacity from a provider excluded by the authenticated route.
|
||||
- Keep current projected Ornith cases separate: qualify `ornith:35b` only through `onexplayer-lemonade`, and qualify `ornith-fast` only through `rtx5090-lemonade`. Run Chat and Responses independently for each alias. Do not mutate routes, capacities, or timeout values to make the smoke pass.
|
||||
- Require HTTP 200 for every request, exactly one endpoint-native success terminal and one `[DONE]` per stream, selected-provider peak `in_flight` equal to eligible capacity, `queued >= 1`, and selected final counters `0/0`. Treat a missing route match, changed capacity, wrong context class, missing status sample, or incomplete terminal as a fail-closed release blocker.
|
||||
- Every invocation must allocate a distinct mode-0700 directory under ignored `agent-test/runs/**`. Accept only request/result/status files bound to that invocation's manifest and timestamps. Preserve raw routes, request bodies, response bodies, curl errors, token/header values, route ids, slot ids, prompts, and output only inside that directory; retain only the script's allowlisted sanitized summary as task or tracked evidence.
|
||||
- For unprojected or legacy provider pools, retain the existing model-group capacity rule. Current baselines are Laguna `laguna-s:2.1` on GX10 capacity `4` (five requests) and Qwen `qwen3.6:35b` on mac-mlx-vllm capacity `2` (three requests).
|
||||
- Allow Qwen and Laguna reasoning/thinking text as normal output. For Laguna, compare Pi `high` thinking events with `off` zero-thinking events and final text. For agent/tool-call qualification, test forced, automatic, streaming, and multi-turn tool calls separately from capacity qualification.
|
||||
|
||||
10. **git-flow release finish와 tag 반영**
|
||||
- 빌드 전·후 테스트, 배포 후 연결 검증, `/v1/responses`와 `/v1/chat/completions` capacity smoke가 모두 성공했는지 다시 확인한다.
|
||||
- 빌드 전·후 테스트, 배포 후 연결 검증, `/v1/responses`와 `/v1/chat/completions` capacity smoke가 모두 성공했는지 다시 확인한다. Managed route가 있으면 `scripts/e2e-openai-managed-capacity-smoke.sh`의 네 Ornith route/endpoint case와 current-run provenance gate가 모두 통과해야 한다.
|
||||
- 하나라도 실패했거나 필수 검증이 실행되지 않았으면 finish하지 않고 `release/dev-<count>` branch를 유지한다.
|
||||
- 현재 release HEAD가 `DEPLOY_SHA`와 같은지 확인한다. 달라졌으면 배포 산출물과 source가 달라진 것이므로 finish하지 않는다.
|
||||
- finish 직전에 `git fetch origin dev main --tags`를 다시 실행하고 `origin/dev=DEV_BASE_SHA`, `origin/main=MAIN_BASE_SHA`, remote tag 없음이 모두 유지되는지 확인한다. 하나라도 달라졌으면 finish하지 않고 release branch를 유지한다.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,125 @@
|
|||
evidence_schema=iop.ornith_session_stall_timeout_order.v2
|
||||
observed_date=2026-08-13
|
||||
environment=dev-runtime
|
||||
release=dev-974
|
||||
qualification_source_sha=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
qualification_source_ref=release/dev-974
|
||||
qualification_source_clean=true
|
||||
qualification_origin_dev_match=true
|
||||
build.edge.sha256=25e097be4aee972cc62821889f8dd697b7d0f8de712973d169449b2271a39096
|
||||
build.node_darwin_arm64.sha256=c24a3b0e16bc7ab807c00c465a47f9caccd35c5575d264d0c4ad10b9ad422f01
|
||||
build.node_linux_arm64.sha256=759eb31f8c8752925e54f61fdeb4f9366bd39e37ea50815e79328c3be86ee4e7
|
||||
build.node_windows_amd64.sha256=f04f33f17be8a4fdc29ef0e4b705953d561478a75f00948d3fdfc13fee8ed572
|
||||
config.onexplayer-lemonade.response_stall_timeout_ms=120000
|
||||
config.onexplayer-lemonade.capacity=3
|
||||
config.rtx5090-lemonade.response_stall_timeout_ms=120000
|
||||
config.rtx5090-lemonade.capacity=1
|
||||
config.long_context_threshold_tokens=30000
|
||||
config.external_caller_boundary_ms=180000
|
||||
config.edge_request_hard_timeout_ms=3600000
|
||||
timeout_order_valid=true
|
||||
restart_required_paths.count=2
|
||||
restart_required_paths.exact_ornith_only=true
|
||||
runtime.ports_open=18082,18083,18084,19093,19101
|
||||
runtime.connected_node_count=4
|
||||
runtime.ornith_providers_healthy=true
|
||||
runtime.ornith_recovered_in_flight=0
|
||||
runtime.ornith_recovered_queued=0
|
||||
direct.onexplayer-lemonade.http_status=200
|
||||
direct.onexplayer-lemonade.duration_ms=111232
|
||||
direct.onexplayer-lemonade.finish_count=1
|
||||
direct.onexplayer-lemonade.done_count=1
|
||||
direct.onexplayer-lemonade.error_count=0
|
||||
direct.onexplayer-lemonade.outcome=normal_terminal
|
||||
direct.rtx5090-lemonade.http_status=200
|
||||
direct.rtx5090-lemonade.duration_ms=7710
|
||||
direct.rtx5090-lemonade.finish_count=1
|
||||
direct.rtx5090-lemonade.done_count=1
|
||||
direct.rtx5090-lemonade.error_count=0
|
||||
direct.rtx5090-lemonade.outcome=normal_terminal
|
||||
stall_reproduction_status=not_reproduced
|
||||
managed.script_sha256=bc0dcf5aa8acc6a10942a6d48e4b3248547004df04d0a292562aea2ca465a903
|
||||
managed.provenance=current-run-manifest
|
||||
managed.ornith_35b.chat.run_id=1786601559737433000-4ba66d5c48201ad1
|
||||
managed.ornith_35b.chat.selected_provider=onexplayer-lemonade
|
||||
managed.ornith_35b.chat.context_class=normal
|
||||
managed.ornith_35b.chat.request_runes=431
|
||||
managed.ornith_35b.chat.estimated_input_tokens=133
|
||||
managed.ornith_35b.chat.eligible_capacity=3
|
||||
managed.ornith_35b.chat.request_count=4
|
||||
managed.ornith_35b.chat.http_200_count=4
|
||||
managed.ornith_35b.chat.finish_count=4
|
||||
managed.ornith_35b.chat.done_count=4
|
||||
managed.ornith_35b.chat.error_count=0
|
||||
managed.ornith_35b.chat.duration_ms_min=35049
|
||||
managed.ornith_35b.chat.duration_ms_max=60703
|
||||
managed.ornith_35b.chat.selected_peak_in_flight=3
|
||||
managed.ornith_35b.chat.selected_max_queued=1
|
||||
managed.ornith_35b.chat.selected_final_in_flight=0
|
||||
managed.ornith_35b.chat.selected_final_queued=0
|
||||
managed.ornith_35b.chat.outcome=pass
|
||||
managed.ornith_35b.responses.run_id=1786601621300814000-c4d9c6c134fe368b
|
||||
managed.ornith_35b.responses.selected_provider=onexplayer-lemonade
|
||||
managed.ornith_35b.responses.context_class=normal
|
||||
managed.ornith_35b.responses.request_runes=407
|
||||
managed.ornith_35b.responses.estimated_input_tokens=126
|
||||
managed.ornith_35b.responses.eligible_capacity=3
|
||||
managed.ornith_35b.responses.request_count=4
|
||||
managed.ornith_35b.responses.http_200_count=4
|
||||
managed.ornith_35b.responses.completed_count=4
|
||||
managed.ornith_35b.responses.done_count=4
|
||||
managed.ornith_35b.responses.error_count=0
|
||||
managed.ornith_35b.responses.duration_ms_min=35451
|
||||
managed.ornith_35b.responses.duration_ms_max=57073
|
||||
managed.ornith_35b.responses.selected_peak_in_flight=3
|
||||
managed.ornith_35b.responses.selected_max_queued=1
|
||||
managed.ornith_35b.responses.selected_final_in_flight=0
|
||||
managed.ornith_35b.responses.selected_final_queued=0
|
||||
managed.ornith_35b.responses.outcome=pass
|
||||
managed.ornith_fast.chat.run_id=1786601679237576000-53b4cfe86fa3dc40
|
||||
managed.ornith_fast.chat.selected_provider=rtx5090-lemonade
|
||||
managed.ornith_fast.chat.context_class=normal
|
||||
managed.ornith_fast.chat.request_runes=432
|
||||
managed.ornith_fast.chat.estimated_input_tokens=135
|
||||
managed.ornith_fast.chat.eligible_capacity=1
|
||||
managed.ornith_fast.chat.request_count=2
|
||||
managed.ornith_fast.chat.http_200_count=2
|
||||
managed.ornith_fast.chat.finish_count=2
|
||||
managed.ornith_fast.chat.done_count=2
|
||||
managed.ornith_fast.chat.error_count=0
|
||||
managed.ornith_fast.chat.duration_ms_min=3402
|
||||
managed.ornith_fast.chat.duration_ms_max=6602
|
||||
managed.ornith_fast.chat.selected_peak_in_flight=1
|
||||
managed.ornith_fast.chat.selected_max_queued=1
|
||||
managed.ornith_fast.chat.selected_final_in_flight=0
|
||||
managed.ornith_fast.chat.selected_final_queued=0
|
||||
managed.ornith_fast.chat.outcome=pass
|
||||
managed.ornith_fast.responses.run_id=1786601686638286000-825708e93ce3450d
|
||||
managed.ornith_fast.responses.selected_provider=rtx5090-lemonade
|
||||
managed.ornith_fast.responses.context_class=normal
|
||||
managed.ornith_fast.responses.request_runes=408
|
||||
managed.ornith_fast.responses.estimated_input_tokens=127
|
||||
managed.ornith_fast.responses.eligible_capacity=1
|
||||
managed.ornith_fast.responses.request_count=2
|
||||
managed.ornith_fast.responses.http_200_count=2
|
||||
managed.ornith_fast.responses.completed_count=2
|
||||
managed.ornith_fast.responses.done_count=2
|
||||
managed.ornith_fast.responses.error_count=0
|
||||
managed.ornith_fast.responses.duration_ms_min=3223
|
||||
managed.ornith_fast.responses.duration_ms_max=6399
|
||||
managed.ornith_fast.responses.selected_peak_in_flight=1
|
||||
managed.ornith_fast.responses.selected_max_queued=1
|
||||
managed.ornith_fast.responses.selected_final_in_flight=0
|
||||
managed.ornith_fast.responses.selected_final_queued=0
|
||||
managed.ornith_fast.responses.outcome=pass
|
||||
edge.auth.active_control_plane_token_count=1
|
||||
edge.auth.preflight_http_status=200
|
||||
edge.auth.preflight_model_count=9
|
||||
raw_material_in_tracked_evidence=false
|
||||
release_finish=done
|
||||
release_tag=dev-974
|
||||
release_tag_tree=d46c7e3057bafbeedc1ddb10d2619feb3c6eb2db
|
||||
release_main_sha=fd41adac777f7c7faea0cdd27acbe2817890abf8
|
||||
release_dev_sha=019500a5d65117bfd115e0c33601ceef3461ee52
|
||||
release_atomic_push=pass
|
||||
release_remote_branch_deleted=true
|
||||
|
|
@ -0,0 +1,249 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=0 tag=TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Code Review Reference - TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-13
|
||||
task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness, plan=0, tag=TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Dependency evidence: `agent-task/archive/2026/07/m-openai-compatible-output-validation-filters/03_repeat_guard_s07_remote_evidence/complete.log` is `PASS` and satisfies predecessor index `03` for this `04+03` packet.
|
||||
- That completion proves the previous repeat-guard remote evidence and single-terminal validation passed for milestone task `repeat-guard`.
|
||||
- It does not prove the Ornith logical-finish-without-`[DONE]`/EOF/`END` stall shape or the dev timeout-order rollout; those remain owned by this packet.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-{review_lane}-{review_grade}.md` → `code_review_{review_lane}_{review_grade}_{review_log_number}.log` and `PLAN-{build_lane}-{build_grade}.md` → `plan_{build_lane}_{build_grade}_{plan_log_number}.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/{task_name}/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-<milestone-slug>`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| TEST-1 — Logical finish followed by transport stall | [x] |
|
||||
| TEST-2 — Deterministic Lemonade adapter-to-Edge smoke | [ ] |
|
||||
| TEST-3 — Dev Ornith timeout order and sanitized evidence | [ ] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [TEST-1] Add exact Node and Edge logical-finish-without-transport-terminal regressions while preserving the existing terminal contract.
|
||||
- [ ] [TEST-2] Extend the fake Lemonade end-to-end smoke with a bounded hang-after-finish fixture and sanitized single-terminal assertions.
|
||||
- [ ] [TEST-3] Align both dev Ornith provider response-stall timeouts, perform a same-ref rollout, update the runbook, and capture sanitized live evidence.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_{review_lane}_{review_grade}_{review_log_number}.log`.
|
||||
- [x] Archive active `PLAN-*-G??.md` to `plan_{build_lane}_{build_grade}_{plan_log_number}.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/{task_name}/` to `agent-task/archive/YYYY/MM/{task_name}/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-<milestone-slug>`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/{task_group}/` or verify it was kept due to remaining siblings/files.
|
||||
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- TEST-1 completed as planned. TEST-2's deterministic adapter-to-Edge fixture reproduced an actual existing runtime defect instead of passing as a missing-regression addition. Per the plan's explicit stop condition, no product runtime fix, TEST-3 documentation/config change, remote clean-sync/rebuild/deploy/restart, or live Ornith evidence collection was attempted.
|
||||
- Full local and race qualification was not run after the TEST-2 blocker. Focused TEST-1 regressions, script syntax, and `git diff --check` were rerun so the completed bounded test work remains reviewable without presenting the packet as qualified.
|
||||
- The planned inventory query binary could not execute from `/tmp` because that mount is non-executable (`Permission denied`). The same read-only inventory query was rebuilt under `.tmp/task-g09/` and confirmed the two intended Ornith providers; this did not change repository state or the TEST-3 stop decision.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Kept `finish_reason` as provider wire/protocol state. The new Node test drives response-start plus a logical-finish BODY and proves only watchdog expiry creates the typed transport failure; late usage and `END` remain fenced.
|
||||
- Added the Edge logical-finish matrix with a held pre-commit branch and an already-open post-commit branch. It asserts one alternate replay before commit, no replay after commit, hidden pending finish wire, no raw failure leakage, and exactly one sanitized terminal on the endpoint-aware path.
|
||||
- Strengthened the normal Chat codec control with usage on the logical finish frame and an assertion that no Core terminal exists before `[DONE]`/physical finish.
|
||||
- The fake Lemonade fixture uses request-context cancellation instead of an unbounded sleep, a 200 ms test-only provider stall timeout, count-based terminal assertions, post-cancel health control, and explicit fake-process reap. Its observed failure is kept as the regression/blocker rather than widening this test packet into an unreviewed production change.
|
||||
- Exact blocker/root cause: with `openai.stream_evidence_gate.enabled` omitted/false, `buildOpenAITunnelStreamGateRuntime` selects `newOpenAITunnelEventSource` rather than `newOpenAITunnelEndpointEventSource` (`apps/edge/internal/openai/stream_gate_runtime.go`). Node correctly emits a typed `response_stalled` after 200 ms, but the tunnel sink's codec endpoint remains unbound. After the already-released logical finish opens the response, `openAITunnelReleaseSink.CommitTerminal` only writes the sanitized SSE error plus `[DONE]` when `endpointIsChat()` is true (`apps/edge/internal/openai/stream_gate_release_sink.go`), so it returns with no client terminal. Resume condition: a separately reviewed runtime fix must preserve endpoint-aware terminal rendering in semantic-disabled supported Chat traffic, after which this packet can rerun TEST-2 and only then proceed to TEST-3.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- [ ] Confirm `finish_reason` remains wire/protocol state and does not create a Core terminal before `[DONE]` or tunnel `END`.
|
||||
- [ ] Confirm Node expiry emits exactly one typed `response_stalled` and fences late usage/`END`.
|
||||
- [ ] Confirm Edge pre-commit recovery replays once and exposes no pending/raw bytes; post-commit recovery emits one sanitized error plus `[DONE]` without replay.
|
||||
- [ ] Confirm the fake Lemonade fixture is bounded, cancels cleanly, and retains the exact normal finish/usage/`[DONE]` control.
|
||||
- [ ] Confirm both current Ornith provider entries use `response_stall_timeout_ms: 120000` with no route/capacity change and the Edge request timeout remains longer than the caller boundary.
|
||||
- [ ] Confirm remote source/build/runtime identities match one clean latest `origin/dev` ref after full rebuild/restart.
|
||||
- [ ] Confirm tracked evidence is sanitized and raw prompt/output exists only in ignored artifacts.
|
||||
- [ ] Confirm only files in the plan's `Modified Files Summary` changed.
|
||||
|
||||
## Verification Results
|
||||
|
||||
### TEST-1 focused regressions
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAITunnelCodecTerminalWire)$'
|
||||
```
|
||||
|
||||
Fresh resumed-worker result on 2026-08-13 (exit 0 for each):
|
||||
|
||||
```text
|
||||
ok iop/apps/node/internal/node 0.055s
|
||||
ok iop/apps/edge/internal/openai 0.037s
|
||||
```
|
||||
|
||||
Additional local checks (exit 0):
|
||||
|
||||
```text
|
||||
bash -n scripts/e2e-openai-lemonade.sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### TEST-2 deterministic end-to-end smoke
|
||||
|
||||
```bash
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
```
|
||||
|
||||
status=BLOCKED — the deterministic TEST-2 fixture exposed the existing semantic-disabled Chat terminal-rendering source defect described below.
|
||||
|
||||
Blocked, reproduced again by the resumed worker after the prior two runs (fresh exit 1). Sanitized fresh output:
|
||||
|
||||
```text
|
||||
[openai-lemonade] hang fixture terminal counts prefix=1 finish=1 run_error=0 done=0
|
||||
```
|
||||
|
||||
The prior ignored run artifacts confirmed, without copying raw bodies into tracked evidence, that Node observed `execution_path=provider_tunnel`, `attempt_fence=confirmed`, `idle_duration_ms=200`, and returned `provider response stalled`. The fake provider also observed request-context cancellation. The fresh count-only replay again showed that the client received neither the expected sanitized `run_error` nor `[DONE]`, matching the semantic-disabled endpoint-binding root cause recorded above.
|
||||
|
||||
Resume condition: Complete a separately reviewed runtime fix that preserves endpoint-aware Chat terminal rendering when semantic validation is disabled, then rerun TEST-2 successfully before starting TEST-3.
|
||||
|
||||
### TEST-3 remote rollout and live evidence
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-list --left-right --count HEAD...origin/dev && go version'\'''
|
||||
```
|
||||
|
||||
Not run. TEST-2 exposed the actual source defect above, so the plan's stop condition prohibited starting the TEST-3 remote rollout. No dev config, process, binary, port, tracked runbook, or evidence log was changed. Resume only after the reviewed runtime fix makes TEST-2 pass locally.
|
||||
|
||||
### Full local qualification
|
||||
|
||||
```bash
|
||||
go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Not run as a completion qualification because TEST-2 is blocked by the source defect. `git diff --check` did pass. Intentional tracked changes at handoff:
|
||||
|
||||
```text
|
||||
M apps/edge/internal/openai/stream_gate_pipeline_test.go
|
||||
M apps/edge/internal/openai/stream_gate_stall_recovery_test.go
|
||||
M apps/node/internal/node/liveness_watchdog_test.go
|
||||
M scripts/e2e-openai-lemonade.sh
|
||||
?? agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/
|
||||
```
|
||||
|
||||
No TEST-3 documentation, dev runtime config, or tracked remote-evidence file was created. Active PLAN and CODE_REVIEW files remain in place as required.
|
||||
|
||||
### Reviewer verification (fresh, 2026-08-13)
|
||||
|
||||
The reviewer reran the applicable local commands from the active plan rather than relying on the implementation handoff. Focused regressions, script syntax, package qualification, race qualification, and diff hygiene all passed:
|
||||
|
||||
```text
|
||||
$ go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
ok iop/apps/node/internal/node 0.057s
|
||||
|
||||
$ go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAITunnelCodecTerminalWire)$'
|
||||
ok iop/apps/edge/internal/openai 0.054s
|
||||
|
||||
$ bash -n scripts/e2e-openai-lemonade.sh
|
||||
# exit 0, no output
|
||||
|
||||
$ go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
ok iop/packages/go/execution 0.035s
|
||||
ok iop/packages/go/config 0.197s
|
||||
ok iop/apps/node/internal/node 1.044s
|
||||
ok iop/apps/edge/internal/openai 8.575s
|
||||
|
||||
$ go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
ok iop/apps/node/internal/node 3.733s
|
||||
ok iop/apps/edge/internal/openai 13.410s
|
||||
|
||||
$ git diff --check
|
||||
# exit 0, no output
|
||||
```
|
||||
|
||||
The required deterministic adapter-to-Edge smoke still failed and reproduced the implementation handoff exactly:
|
||||
|
||||
```text
|
||||
$ IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
[openai-lemonade] hang fixture terminal counts prefix=1 finish=1 run_error=0 done=0
|
||||
# exit 1
|
||||
```
|
||||
|
||||
The reviewer also completed a read-only declared-runner preflight. `/Users/toki/agent-work/iop-dev` is clean on `release/dev-971` at `003398a149c1433cb21bc8aa2720bde272b87373`, equal to `origin/dev` with divergence `0 0`; Go is `1.26.3 darwin/arm64`, git-flow is `1.12.3`, the Edge/Node binaries under `build/dev-runtime/bin/` report source `003398a149c1`, and ports `18082`, `18083`, `18084`, `19093`, and `19101` are listening. This proves the TEST-3 route is available but does not satisfy its rollout/evidence acceptance criteria.
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- **Overall Verdict:** FAIL
|
||||
- **Dimension Assessment:**
|
||||
- Correctness: Fail — the deterministic semantic-disabled Chat tunnel path opens the stream but emits no sanitized terminal after the typed stall.
|
||||
- Completeness: Fail — TEST-2 does not pass and TEST-3 rollout, runbook, and tracked evidence were not completed.
|
||||
- Test coverage: Fail — the new semantic-enabled logical-finish matrix passes, but it does not cover the semantic-disabled product path that the fake smoke exercises; streaming Responses has the same endpoint/sink split without an exact post-commit regression.
|
||||
- API contract: Fail — the always-owned runtime does not preserve the required single sanitized terminal when semantic filtering is disabled.
|
||||
- Code quality: Pass — the added test/script changes are bounded and contain no unrelated product changes or debug residue.
|
||||
- Implementation deviation: Pass — implementation obeyed the explicit stop condition instead of widening the original test-only plan into an unreviewed runtime change.
|
||||
- Verification trust: Pass — fresh reviewer execution reproduced the recorded failure, and the implementation did not claim TEST-2/TEST-3 completion.
|
||||
- Spec conformance: Fail — the `repeat-guard,ops-evidence` SDD evidence contribution cannot close while terminal liveness and dev operational evidence remain incomplete.
|
||||
- **Findings:**
|
||||
- **Required R1 — Semantic-disabled provider-tunnel stalls can leave an already-open OpenAI-compatible stream unterminated.**
|
||||
- **Evidence:** Fresh `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh` exits 1 with `prefix=1 finish=1 run_error=0 done=0`. In `apps/edge/internal/openai/stream_gate_runtime.go`, both Chat pool tunnel construction and generic tunnel initial/recovery construction choose `newOpenAITunnelEventSource` when semantic filtering is disabled, so `openAITunnelCodecState.endpoint` remains unbound. `openAITunnelReleaseSink.CommitTerminal` emits a post-commit Chat `run_error` plus `[DONE]` only when `endpointIsChat()` is true. `apps/edge/internal/openai/responses_stream_gate.go` also selects the endpoint-unaware source and generic composite tunnel sink for semantic-disabled streaming Responses. This contradicts `agent-contract/outer/openai-compatible-api.md:120-126` and `agent-contract/inner/execution-runtime.md:52`.
|
||||
- **Root Cause:** Endpoint framing was incorrectly coupled to semantic-filter activation. Disabling configured semantic filters bypasses the endpoint codec and, for streaming Responses, its endpoint-native terminal sink, even though the private typed-stall registration and request-local StreamGate remain active. Once a prefix is committed, Core commits an error terminal but the selected raw sink lacks the endpoint identity/renderer needed to serialize it.
|
||||
- **Selected Fix:** In `apps/edge/internal/openai/stream_gate_runtime.go`, always construct `newOpenAITunnelEndpointEventSource` for supported Chat/Responses tunnel attempts (initial and recovery), independent of `semanticEnabled`; keep that flag limited to semantic policy and compatibility decisions. For direct streaming Responses, select the existing `newOpenAIResponsesPoolReleaseSink` and its codec state instead of the generic raw tunnel sink. In `apps/edge/internal/openai/responses_stream_gate.go`, likewise always use the Responses endpoint source and use `newOpenAIResponsesPoolReleaseSink` for streaming provider-pool Responses regardless of semantic-filter activation. This reuses the existing endpoint-native sink that observes raw sequence state, preserves successful provider wire byte-for-byte, and writes one sanitized Responses error plus `[DONE]` after commit. Extend `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` with semantic-disabled Chat and Responses pre/post-commit typed-stall cases, including direct Responses, and retain the existing disabled-semantic byte-order compatibility controls. Acceptance requires the focused tests, full package tests, race tests, and fake Lemonade smoke to exit 0 with exact one-terminal/no-leakage assertions.
|
||||
- **Required R2 — The required dev timeout-order rollout and durable operational evidence are absent.**
|
||||
- **Evidence:** TEST-3 remains unchecked, `docs/edge-local-dev-guide.md` has no Ornith response-stall ownership section, and `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log` does not exist. Fresh read-only preflight proves the declared dev runner is reachable, clean, synchronized at `003398a149c1`, built from that same source, and has all required ports listening; the omission is therefore the planned TEST-2 stop condition, not an unavailable execution route.
|
||||
- **Root Cause:** The original plan correctly prohibited TEST-3 after TEST-2 exposed a product defect. Consequently no dev config/restart, direct/Edge Ornith smoke, runbook update, or sanitized evidence could be accepted in this loop.
|
||||
- **Selected Fix:** After R1 and all local qualification commands pass, execute the existing TEST-3 procedure exactly: set `response_stall_timeout_ms: 120000` only for `onexplayer-lemonade` and `rtx5090-lemonade`, preserve routing/capacity, keep the Edge request timeout above the approximately 180-second caller boundary, perform the full `dev-runtime-deploy` clean same-ref config-check/rebuild/deploy/restart/identity procedure, run the direct-provider and Edge approximately 41k-token controls for both providers, update `docs/edge-local-dev-guide.md`, and write the exact sanitized evidence log above. Record `not_reproduced` when appropriate; never copy prompt/output/token/credential material into tracked artifacts. Acceptance requires same-ref identity, healthy declared ports/nodes, the timeout inequality, one terminal per control, and the tracked raw-free evidence file.
|
||||
- **Routing Signals:** `review_rework_count=1`, `evidence_integrity_failure=false`
|
||||
- **Next Step:** Archive this FAIL pair and materialize the prepared `REVIEW_TEST` follow-up PLAN/CODE_REVIEW pair containing direct fixes R1 and R2; do not write `complete.log` or update the roadmap.
|
||||
|
|
@ -0,0 +1,383 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=1 tag=REVIEW_TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Code Review Reference - REVIEW_TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-13
|
||||
task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness, plan=1, tag=REVIEW_TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G09_0.log`.
|
||||
- Prior review: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G09_0.log`, verdict `FAIL`, `Required R1` semantic-disabled Chat/Responses tunnel terminal ownership and `Required R2` unfinished dev timeout-order evidence; no Suggested or Nit findings.
|
||||
- Fresh reviewer evidence: focused Node/Edge tests, full package tests, race tests, script syntax, and `git diff --check` passed; `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh` failed with `prefix=1 finish=1 run_error=0 done=0`.
|
||||
- Dev preflight: `/Users/toki/agent-work/iop-dev` was clean on `release/dev-971` at `003398a149c1433cb21bc8aa2720bde272b87373`, equal to `origin/dev`; Edge/Node binaries reported source `003398a149c1`, and ports `18082`, `18083`, `18084`, `19093`, `19101` were listening.
|
||||
- Roadmap carryover: this packet still contributes only `repeat-guard,ops-evidence`; it does not complete or update the Milestone directly.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_1.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_1.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-openai-compatible-output-validation-filters`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|------|---------|
|
||||
| REVIEW_TEST-1 — Endpoint-aware semantic-disabled terminal ownership | [x] |
|
||||
| REVIEW_TEST-2 — Deterministic adapter composition and local qualification | [x] |
|
||||
| REVIEW_TEST-3 — Dev Ornith timeout order and sanitized evidence | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [REVIEW_TEST-1] Restore endpoint-aware semantic-disabled Chat/Responses tunnel terminal behavior and add exact pre/post-commit regressions.
|
||||
- [x] [REVIEW_TEST-2] Rerun the bounded fake Lemonade adapter-to-Edge smoke plus focused, full, race, syntax, and diff qualification.
|
||||
- [x] [REVIEW_TEST-3] After local qualification, apply the two-provider dev timeout order, perform same-ref rollout/health checks, update the runbook, and capture sanitized Ornith evidence.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [x] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_1.log`.
|
||||
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G10_1.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [ ] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [ ] If PASS, move active task directory `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/` to `agent-task/archive/YYYY/MM/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/` and update this checklist at the final archive path.
|
||||
- [ ] If PASS and task group is `m-openai-compatible-output-validation-filters`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [ ] If PASS for split work, remove empty active parent `agent-task/m-openai-compatible-output-validation-filters/` or verify it was kept due to remaining siblings/files.
|
||||
- [x] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- REVIEW_TEST-1/2 구현과 runbook 갱신은 현재 source commit `2fcc1093c7629ab94b460d083520ffd9c8064815`에 이미 포함되어 있어 선택된 구현을 재작성하지 않고 source/blame와 fresh 검증으로 확인했다.
|
||||
- 원격 순차 테스트의 첫 zsh wrapper는 테스트 실행 전에 parse error가 났고, 같은 package 목록을 ignored `remote-sequential-tests.zsh`로 실행해 build 전/후 각각 `packages=49 failed=0`을 확인했다.
|
||||
- build 완료 뒤 artifact summary의 `awk '$1'`가 zsh positional parameter로 해석되어 summary만 실패했다. artifact 자체는 완성돼 있었고 shell-safe identity 명령으로 네 hash/source를 재확인했다.
|
||||
- Windows Node 재기동 wrapper는 `Win32_Process.Create`가 반환한 `cmd.exe` PID와 자식 `iop-node.exe` PID를 동일하다고 가정해 실패처럼 보였다. 실제 process/query, installed hash, 상태 snapshot으로 OneXPlayer/RTX5090 Node가 같은 candidate로 실행 중임을 확인했다.
|
||||
- Windows ACL smoke 첫 호출은 PowerShell의 dotted `-test` flag 전달이 분리되어 테스트 실행 전에 실패했다. `cmd.exe /c` 인용으로 재실행해 두 Windows host에서 요구 테스트를 통과시켰다.
|
||||
- 첫 Edge Chat smoke는 잘못된 `/v1/chat-completions` 경로로 404였다. 올바른 `/v1/chat/completions`로 수정해 재시도했으나 Chat/Responses 모두 인증 admission 전 401이었다.
|
||||
- 실행 중 Control Plane의 활성 token digest와 runner/local 보호 SOPS 및 Pi auth/model token scalar를 원문 노출 없이 비교했으나 일치 0개였다. runner의 shell/agent history와 ignored run 후보에서도 일치 0개였다. 새 token 발급/회전은 계획된 runtime rollout보다 넓은 인증 상태 변경이므로 수행하지 않았다. resume condition은 활성 digest와 일치하는 보호된 raw token 제공 또는 token rotation 명시 승인이다.
|
||||
- 2026-08-13 재개 진단에서 보호 SOPS의 기존 token은 static `openai-principal-tokens.yaml` active entry와 digest가 일치했지만 `/v1/models`는 계속 401이었다. Managed mode는 static mapping이 아니라 fresh Control Plane projection만 인증 source로 사용하며, Control Plane durable store의 active digest 1건과 보호 SOPS 전체의 일치 건수는 다시 0건이었다. 따라서 blocker를 static token 부재가 아니라 active Control Plane projected digest와 일치하는 보호된 plaintext 부재로 좁혔다.
|
||||
- 승인된 protected wrapper의 활성 token을 명령행·로그·파일에 남기지 않고 stdin으로 1회 주입해 인증 preflight(`/v1/models` HTTP 200, model count 9)를 통과시켰다. 그 뒤 direct control은 반복하지 않고 Edge Chat/Responses capacity+1만 1회 실행했다.
|
||||
- 인증 후 capacity smoke는 Chat/Responses 각 5건 모두 180초 caller boundary에서 `curl rc=28`, terminal 0건으로 실패했다. Chat 2건은 HTTP 200 스트림을 열고 약 289–301 KB를 받았지만 끝나지 않았고, 나머지 8건은 header 전 대기였다. 두 endpoint 모두 OneXPlayer peak `in_flight=2`, `queued=3`를 관측했고 최종 `0/0`, healthy/available로 회복했다.
|
||||
- 필수 Edge capacity smoke가 실제 runtime terminal 수렴 실패로 미완료여서 `dev-runtime-deploy` stop condition에 따라 release finish/tag/atomic push를 실행하지 않고 `release/dev-974`를 유지했다.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- semantic filter enablement와 endpoint codec/terminal sink 선택을 분리하고, Chat/Responses의 성공 wire는 그대로 유지하면서 typed stall만 endpoint-native 단일 terminal로 수렴시켰다.
|
||||
- dev rollout은 `release/dev-974`의 단일 SHA에서 Edge, macOS/Linux ARM64/Windows AMD64 Node를 모두 rebuild/restart했다. Ornith capacity/priority/routes는 변경하지 않고 두 provider의 `response_stall_timeout_ms`만 `120000`으로 적용했다.
|
||||
- timeout ownership은 Node `120000 ms` < 외부 caller 약 `180000 ms` < Edge hard timeout `3600000 ms` 순서로 유지했다.
|
||||
- tracked 증빙에는 source/build/config identity, monotonic duration, terminal count, outcome과 blocker만 기록하고 raw prompt/output/token/credential은 ignored run 경로에만 두었다.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- [ ] Confirm endpoint codec binding is independent of semantic filter enablement on every selected Chat/Responses initial and recovery tunnel path.
|
||||
- [ ] Confirm semantic-disabled successful Chat/Responses status, headers, JSON/SSE wire order, usage, and single provider terminal remain byte-identical.
|
||||
- [ ] Confirm pre-commit typed stalls replay at most once under the existing guards and post-commit Chat/Responses emit exactly one sanitized endpoint-native terminal with no raw provider data.
|
||||
- [ ] Confirm direct streaming Responses and provider-pool streaming Responses both close after an already-committed typed stall.
|
||||
- [ ] Confirm `finish_reason`/`response.completed` stays protocol wire state until `[DONE]` or physical `END`.
|
||||
- [ ] Confirm the fake Lemonade fixture remains bounded, receives one `run_error` and one `[DONE]`, cancels/reaps cleanly, and passes a post-cancel control.
|
||||
- [ ] Confirm both current Ornith provider entries use `response_stall_timeout_ms: 120000` with no route/capacity change and Edge timeout remains above the caller boundary.
|
||||
- [ ] Confirm dev source/build/runtime identities match one clean release ref, required ports/nodes/providers are healthy, and tracked evidence is sanitized.
|
||||
- [ ] Confirm only files in `Modified Files Summary` changed.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr for every command. If output is too long, save it under an ignored task-specific run directory and record the exact command/path; never reconstruct output.
|
||||
|
||||
### REVIEW_TEST-1 focused terminal regressions
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire)$'
|
||||
```
|
||||
|
||||
출력(exit 0):
|
||||
|
||||
```text
|
||||
ok iop/apps/edge/internal/openai 0.077s
|
||||
```
|
||||
|
||||
### REVIEW_TEST-2 deterministic fake composition
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-lemonade.sh
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
```
|
||||
|
||||
모두 exit 0. 실제 출력:
|
||||
|
||||
```text
|
||||
bash -n: no output
|
||||
ok iop/apps/node/internal/node 0.047s
|
||||
[openai-lemonade] OpenAI-compatible Lemonade serving test PASSED (mode=fake).
|
||||
```
|
||||
|
||||
### REVIEW_TEST-3 dev preflight, rollout, and live evidence
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev && git rev-list --left-right --count HEAD...origin/dev && go version && git flow version | head -n 1'\'''
|
||||
```
|
||||
|
||||
Record the exact sanitized inventory, clean-sync/release, sequential tests, build hashes/source ids, config check/dry-run, deploy/restart, port/node/provider snapshot, direct/Edge smoke, capacity, and release-finish output required by `dev-runtime-deploy`. Raw prompt/output/token/credential artifacts stay under ignored `agent-test/runs/**`.
|
||||
|
||||
상세 sanitized 로그는 `.local/task-runs/stream-terminal-liveness/`에 보존했다. raw HTTP body는 runner의 ignored `agent-test/runs/stream-terminal-liveness-dev-974/`에만 있다.
|
||||
|
||||
Preflight/release identity:
|
||||
|
||||
```text
|
||||
branch=release/dev-974
|
||||
head=origin/dev=origin/release/dev-974=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
divergence=0 0; shallow=false
|
||||
origin/main=0c1e9e6a3f181eaebe59f888043fadc37385741d
|
||||
go=go1.26.3 darwin/arm64; git-flow=1.12.3 AVH
|
||||
remote dev-974 tag count=0
|
||||
```
|
||||
|
||||
Config/rebuild/test:
|
||||
|
||||
```text
|
||||
config check: OK
|
||||
dry-run: restart_required; exact paths=onexplayer-lemonade.response_stall_timeout_ms,rtx5090-lemonade.response_stall_timeout_ms; 300000 -> 120000
|
||||
pre-build sequential tests: packages=49 failed=0
|
||||
post-build sequential tests: packages=49 failed=0
|
||||
edge sha256=25e097be4aee972cc62821889f8dd697b7d0f8de712973d169449b2271a39096
|
||||
node darwin/arm64 sha256=c24a3b0e16bc7ab807c00c465a47f9caccd35c5575d264d0c4ad10b9ad422f01
|
||||
node linux/arm64 sha256=759eb31f8c8752925e54f61fdeb4f9366bd39e37ea50815e79328c3be86ee4e7
|
||||
node windows/amd64 sha256=f04f33f17be8a4fdc29ef0e4b705953d561478a75f00948d3fdfc13fee8ed572
|
||||
all artifact vcs.revision=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
Windows ACL smoke: required tests PASS on OneXPlayer and RTX5090
|
||||
```
|
||||
|
||||
Deployment/health/config identity:
|
||||
|
||||
```text
|
||||
Edge + mac-codex + GX10 + OneXPlayer + RTX5090 restarted from the same candidate
|
||||
ports 18082,18083,18084,19093,19101=open
|
||||
connected node count=4
|
||||
onexplayer-lemonade capacity=3 priority=2 timeout=120000 health=healthy status=available in_flight=0 queued=0
|
||||
rtx5090-lemonade capacity=1 priority=0 timeout=120000 health=healthy status=available in_flight=0 queued=0
|
||||
Edge request hard timeout=3600000 (> caller boundary 180000)
|
||||
post-restart refresh dry-run: applied, no changes
|
||||
RTX manual toggle/profile/listener/Edge connection=true; IOP Startup/Run/Task/Service entries not created
|
||||
```
|
||||
|
||||
약 41k-token direct controls:
|
||||
|
||||
```text
|
||||
onexplayer-lemonade: http=200 duration_ms=111232 finish=1 done=1 error=0 outcome=normal_terminal
|
||||
rtx5090-lemonade: http=200 duration_ms=7710 finish=1 done=1 error=0 outcome=normal_terminal
|
||||
stall_reproduction_status=not_reproduced
|
||||
```
|
||||
|
||||
Blocker와 stop condition:
|
||||
|
||||
```text
|
||||
protected active-token preflight: /v1/models http=200 model_count=9
|
||||
Chat capacity+1: requests=5, caller timeouts=5, terminal=0, http=200,200,000,000,000
|
||||
Responses capacity+1: requests=5, caller timeouts=5, terminal=0, http=000,000,000,000,000
|
||||
both endpoints: onexplayer peak in_flight=2 queued=3; final in_flight=0 queued=0 healthy/available
|
||||
resume_condition=fix or replan admission/terminal convergence, then repeat one bounded capacity smoke
|
||||
capacity smoke=failed; release finish/tag/atomic push=not run; release/dev-974 retained
|
||||
```
|
||||
|
||||
REVIEW_TEST-3의 구현 작업과 stop condition 처리는 완료됐다. 배포, same-ref identity, 두 provider 직접 control, timeout ordering, runbook과 raw-free evidence는 완료됐다. 이후 활성 token을 안전하게 stdin으로 주입해 인증 blocker를 제거하고 필수 capacity smoke를 실제 실행했지만, capacity+1 장문 요청 10건 모두 caller boundary 전에 endpoint terminal을 내지 못했다. 따라서 acceptance의 capacity smoke는 미충족이며, 이를 성공 전제로 하는 release finish/tag/atomic push도 `dev-runtime-deploy` stop condition에 따라 실행하지 않았다. 구현 체크는 이 실패를 성공으로 간주하지 않고, 계획된 배포·검증·중단·증거 기록까지 수행했음을 뜻한다.
|
||||
|
||||
2026-08-13 최신 read-only 재확인:
|
||||
|
||||
```text
|
||||
release/dev-974 head=origin/dev=origin/release/dev-974=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
remote checkout clean; divergence=0 0; remote tag count=0
|
||||
ports 18082,18083,18084,19093,19101=open; Edge health http=200
|
||||
connected node count=4
|
||||
onexplayer-lemonade capacity=3 health=healthy status=available in_flight=0 queued=0
|
||||
rtx5090-lemonade capacity=1 health=healthy status=available in_flight=0 queued=0
|
||||
protected static token digest match=true; managed /v1/models http=401
|
||||
Control Plane active token count=1; active digest vs protected SOPS match count=0
|
||||
```
|
||||
|
||||
위 read-only 재확인의 401은 stale SOPS 후보에 대한 과거 진단이다. 이후 protected active-token preflight와 실제 capacity smoke 결과가 최신 판정이며, 인증 blocker는 해소됐다. 현재 blocker는 180초 안에 terminal로 수렴하지 않는 capacity+1 장문 runtime이다.
|
||||
|
||||
Tracked raw-free evidence: `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log`.
|
||||
|
||||
### Full local qualification
|
||||
|
||||
```bash
|
||||
go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
모두 exit 0. 실제 출력:
|
||||
|
||||
```text
|
||||
ok iop/packages/go/execution 0.010s
|
||||
ok iop/packages/go/config 0.164s
|
||||
ok iop/apps/node/internal/node 0.984s
|
||||
ok iop/apps/edge/internal/openai 8.471s
|
||||
ok iop/apps/node/internal/node 3.605s # race
|
||||
ok iop/apps/edge/internal/openai 12.919s # race
|
||||
git diff --check: no output
|
||||
git status: existing untracked .gocache/, .local/, active agent-task plus this evidence file; no unrelated tracked modification
|
||||
```
|
||||
|
||||
2026-08-13 resume fresh rerun도 모두 exit 0이며 전체 stdout/stderr는 ignored `.local/task-runs/stream-terminal-liveness/resume-a03-local-verification.log`에 보존했다:
|
||||
|
||||
```text
|
||||
focused Edge terminal regressions: ok 0.068s
|
||||
Node watchdog regression: ok 0.041s
|
||||
bash syntax: no output
|
||||
fake Lemonade: PASSED (mode=fake)
|
||||
selected full packages: execution 0.006s, config 0.206s, node 0.988s, edge 8.425s
|
||||
race: node 3.644s, edge 12.664s
|
||||
git diff --check: no output
|
||||
```
|
||||
|
||||
2026-08-13 최종 worker 재검증도 모두 exit 0이며 전체 stdout/stderr는 ignored `.local/task-runs/stream-terminal-liveness/resume-a05-final-verification.log`에 보존했다:
|
||||
|
||||
```text
|
||||
Node watchdog regression: ok 0.032s
|
||||
focused Edge terminal regressions: ok 0.057s
|
||||
bash syntax: no output
|
||||
fake Lemonade: PASSED (mode=fake)
|
||||
selected full packages: execution 0.014s, config 0.217s, node 1.078s, edge 8.513s
|
||||
race: node 3.693s, edge 13.139s
|
||||
git diff --check: no output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---------|-------|------|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Reviewer Fresh Verification — 2026-08-13
|
||||
|
||||
### Focused and full local qualification
|
||||
|
||||
```text
|
||||
$ go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
ok iop/apps/node/internal/node 0.076s
|
||||
$ go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'
|
||||
ok iop/apps/edge/internal/openai 0.049s
|
||||
$ bash -n scripts/e2e-openai-lemonade.sh
|
||||
(no output)
|
||||
$ IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
[openai-lemonade] OpenAI-compatible Lemonade serving test PASSED (mode=fake).
|
||||
$ go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
ok iop/packages/go/execution 0.009s
|
||||
ok iop/packages/go/config 0.193s
|
||||
ok iop/apps/node/internal/node 1.038s
|
||||
ok iop/apps/edge/internal/openai 8.457s
|
||||
$ go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
ok iop/apps/node/internal/node 3.688s
|
||||
ok iop/apps/edge/internal/openai 12.751s
|
||||
$ git diff --check
|
||||
(no output)
|
||||
```
|
||||
|
||||
Full stdout/stderr is retained outside the repository at `/tmp/iop-review-stream-terminal.uzTeHn/focused.log` and `/tmp/iop-review-full.SjSkTi/full.log`.
|
||||
|
||||
### Dev route, policy, and artifact provenance
|
||||
|
||||
The first two read-only SSH probes failed before reading state because of shell quoting and missing remote PyYAML. The corrected Ruby YAML plus read-only SQLite probe exited 0 and printed only sanitized configuration/route facts:
|
||||
|
||||
```text
|
||||
branch=release/dev-974
|
||||
head=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
origin_dev=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
dirty_count=0
|
||||
model=ornith:35b default_max_tokens=32768 min_max_tokens=16384 providers=onexplayer-lemonade,rtx5090-lemonade
|
||||
model=ornith-fast default_max_tokens=32768 min_max_tokens=16384 providers=rtx5090-lemonade
|
||||
long_context_threshold_tokens=30000
|
||||
provider=onexplayer-lemonade capacity=3 long_context_capacity=2 priority=2 response_stall_timeout_ms=120000
|
||||
provider=rtx5090-lemonade capacity=1 long_context_capacity=1 priority=0 response_stall_timeout_ms=120000
|
||||
active_route_alias=ornith-fast resource_selector=rtx5090-lemonade
|
||||
active_route_alias=ornith:35b resource_selector=onexplayer-lemonade
|
||||
```
|
||||
|
||||
The live wrapper used `"evidence " * 41000`, which is about 369,000 runes and exceeds the configured 30,000-token long-context threshold under `runes/4 + runes/16`. It also sent five requests through the managed `ornith:35b` route even though `apps/edge/internal/openai/principal_routes.go:291-305` restricts that route to OneXPlayer. The observed OneXPlayer `in_flight=2`, `queued=3` is therefore the expected long-context-slot result, not evidence that aggregate normal capacity 4 was exercised. Chat `max_tokens=900` is raised to the model's `min_max_tokens=16384` by `apps/edge/internal/openai/chat_policy.go:60-80`; the two HTTP 200 streams were still producing data at the caller deadline, so Node's idle-stall watchdog had no reason to fire.
|
||||
|
||||
The remote raw directory also contains mixed-generation files. Current timeout stderr files have a newer mtime than eight 59-byte SSE bodies left by the earlier 401 run, while `safe/summary.json` counted those stale bodies as the current `http=000` cases. The fixed run root therefore does not prove one-run body provenance. Sanitized output is retained at `/tmp/iop-review-dev-route-preflight.log` and `/tmp/iop-review-dev-artifact-preflight.log`; no token or credential value was read or printed.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- **Overall Verdict:** FAIL
|
||||
|
||||
### Dimension Assessment
|
||||
|
||||
| Dimension | Result | Assessment |
|
||||
|---|---|---|
|
||||
| Correctness | Pass | Endpoint-aware semantic-disabled Chat/Responses terminal ownership is implemented and passes focused, composition, full, and race verification. |
|
||||
| Completeness | Fail | The required dev Chat/Responses capacity smoke did not succeed, so release finish/tag/atomic push correctly remain unrun. |
|
||||
| Test coverage | Fail | Deterministic runtime regressions pass, but the required route-realistic dev capacity/terminal qualification has not produced passing evidence. |
|
||||
| API contract | Pass | Successful endpoint-native wire compatibility and one sanitized typed-stall terminal remain covered. |
|
||||
| Code quality | Pass | No debug/dead-code/TODO or diff-check issue was found in the runtime fix. |
|
||||
| Implementation deviation | Fail | The live smoke combined a long-context input, a caller output request rewritten to 16,384 tokens, and an aggregate-capacity assumption contradicted by the active managed route selector. |
|
||||
| Verification trust | Fail | A fixed artifact directory mixed prior 401 response bodies with the current timeout run, and the recorded `approx_input_tokens=41000` does not match the repository estimator. |
|
||||
| Spec conformance | Fail | SDD S07/ops-evidence still lacks a successful capacity+1, terminal-complete, raw-free dev result and the release completion it gates. |
|
||||
|
||||
### Findings
|
||||
|
||||
- **Required R1 — The dev capacity smoke targets capacity that the authenticated route cannot reach.**
|
||||
- **Evidence:** Reviewer read-only preflight shows `ornith:35b -> resource_selector=onexplayer-lemonade`, while the smoke sends five long-context requests and expects OneXPlayer 3 + RTX5090 1. `principal_routes.go:291-305` excludes every provider other than the resolved selector. The configured long slots are OneXPlayer 2 and RTX5090 1, and the observed selected route peak is exactly OneXPlayer `2/3` in-flight/queued. `chat_policy.go:60-80` also raises Chat 900 to 16,384 output tokens, while the two open streams continued producing bytes through 180 seconds.
|
||||
- **Root Cause:** The project deployment/test contract and task-local wrapper treat a catalog model group's physical providers as one caller-reachable pool without first intersecting them with the active managed principal route. The same wrapper also conflates a 41k-token liveness control with the shorter normal-capacity smoke and assumes the caller deadline is an idle-stall oracle.
|
||||
- **Selected Fix:** Update `agent-ops/skills/project/dev-runtime-deploy/SKILL.md` and `agent-test/dev/edge-smoke.md` so managed capacity is derived from the authenticated route's eligible provider selector; add `scripts/e2e-openai-managed-capacity-smoke.sh` that runs Chat and Responses separately for one route/provider with `eligible capacity + 1`, a short input that requests a 700–1200-token structured answer, bounded thinking, exact endpoint terminal validation, selected-provider peak/queue/final recovery, and no runtime route/capacity mutation. Document this distinction in `docs/edge-local-dev-guide.md`. Rerun `ornith:35b` against OneXPlayer and `ornith-fast` against RTX5090 using their actual projected aliases, then finish the retained release only after both endpoint matrices pass.
|
||||
- **Disposition:** direct-fix.
|
||||
- **Acceptance:** `bash -n scripts/e2e-openai-managed-capacity-smoke.sh`; `./scripts/e2e-openai-managed-capacity-smoke.sh --self-test`; `python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy`; route-aware dev Chat/Responses runs show every request HTTP 200 with exactly one success terminal, selected-provider peak equal to eligible capacity, queue at least 1, non-selected provider excluded from the claimed total, and final selected counters 0/0.
|
||||
|
||||
- **Required R2 — The live evidence directory does not provide single-run provenance.**
|
||||
- **Evidence:** Eight `edge-*.sse` files are 59-byte bodies with older mtimes from the prior 401 attempt, whereas current `*.stderr` files have the later timeout mtime. The latest summary reports those bytes under current `rc=28/http=000`. It also labels the `"evidence " * 41000` payload as about 41k input tokens although the repository estimator classifies roughly 369,000 runes above 100k estimated tokens.
|
||||
- **Root Cause:** The wrapper reuses `agent-test/runs/stream-terminal-liveness-dev-974`, does not create an immutable unique run directory, and summarizes any pre-existing response file when curl times out before opening a new response body. Request-shape metadata is a handwritten label instead of being derived from the actual request using the repository estimator contract.
|
||||
- **Selected Fix:** Make the new managed-capacity smoke create a mode-0700 unique run directory for every invocation, remove/refuse pre-existing per-case targets before curl, bind every summary row to the current run id and request/result mtimes, derive rune/estimated-token/context-class metadata from the emitted request, and fail closed on missing/current-run-mismatched bodies. Update the tracked evidence log only from the new sanitized summary; retain raw bodies exclusively under the unique ignored run directory.
|
||||
- **Disposition:** direct-fix.
|
||||
- **Acceptance:** the script self-test rejects a stale-body fixture and proves unique-run isolation; the dev summaries contain one run id, current request/result provenance, computed input estimate/context class, no raw/secret fields, and no stale 59-byte body attribution; `git diff --check` passes.
|
||||
|
||||
### Routing Signals
|
||||
|
||||
- `review_rework_count=2`
|
||||
- `evidence_integrity_failure=true`
|
||||
|
||||
### Next Step
|
||||
|
||||
Invoke the plan skill in `prepare-follow-up` mode with Required R1/R2 exactly as closed above, archive this pair, and materialize the freshly routed `REVIEW_REVIEW_TEST` follow-up pair. Do not write `complete.log` or update the Milestone.
|
||||
|
|
@ -0,0 +1,384 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=2 tag=REVIEW_REVIEW_TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Code Review Reference - REVIEW_REVIEW_TEST
|
||||
|
||||
> **[IMPLEMENTING AGENT — READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `Implementation Checklist`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> Execute the plan's selected root cause, scope, files, and dependency decisions as written. Do not choose another owner, narrow/expand the write boundary, or replace a fix with another verification attempt.
|
||||
> If implementation is blocked, record the exact blocker, attempted commands/output, and resume condition only in implementation-owned evidence fields.
|
||||
> Do not ask the user directly, present choices, call user-input tools, create control-plane stop files, or classify the next state.
|
||||
> Finalization (`Code Review Result`, log rename, `complete.log`, archive moves, `Review-Only Checklist`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## Overview
|
||||
|
||||
date=2026-08-13
|
||||
task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness, plan=2, tag=REVIEW_REVIEW_TEST
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G10_1.log`.
|
||||
- Prior review: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G10_1.log`, verdict `FAIL`, with `Required R1` for route-ineligible aggregate capacity and `Required R2` for mixed-generation artifact provenance; no Suggested or Nit findings.
|
||||
- Fresh reviewer evidence: local Node/Edge focused, fake Lemonade, selected full-package, race, syntax, and diff checks passed. Read-only dev preflight showed `ornith:35b -> onexplayer-lemonade`, `ornith-fast -> rtx5090-lemonade`, normal capacities `3/1`, long-context capacities `2/1`, and a 30,000-token long-context threshold.
|
||||
- Failure evidence: the five-request `ornith:35b` run used a roughly 369,000-rune input, was classified long by the repository estimator, reached OneXPlayer `in_flight=2, queued=3`, and never exercised RTX5090. Eight response bodies summarized as current results were older 59-byte artifacts from a prior 401 attempt.
|
||||
- Roadmap carryover: this packet still contributes only `repeat-guard,ops-evidence`; PASS is contribution evidence and does not directly complete or update the Milestone.
|
||||
|
||||
## For the Review Agent
|
||||
|
||||
> **[REVIEW AGENT ONLY]** The finalization steps below are review-agent only. Implementing agents must not execute this section.
|
||||
|
||||
Compare implementation of each item against source files. Run the applicable verification commands directly and record fresh output in `Verification Results`; implementation-owned output is handoff evidence, not a substitute for reviewer verification. If implementation is present, repair missing or stale verification output instead of failing solely for insufficient recorded evidence. When verification exposes a defect, collect the necessary data, determine the exact root cause, and select one concrete fix before generating the follow-up plan; never delegate investigation or remedy selection to the worker.
|
||||
Review completion means the following steps are finished:
|
||||
|
||||
1. Append verdict and `review_rework_count` / `evidence_integrity_failure` routing signals.
|
||||
2. Archive `CODE_REVIEW-cloud-G10.md` → `code_review_cloud_G10_2.log` and `PLAN-cloud-G10.md` → `plan_cloud_G10_2.log`.
|
||||
3. If PASS, write `complete.log` and move active task directory to `agent-task/archive/YYYY/MM/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/`. If WARN/FAIL, fully write the next filesystem state required by the code-review skill.
|
||||
4. If PASS and task group is `m-openai-compatible-output-validation-filters`, preserve the first-line `milestone-task` metadata in `complete.log` and report it for the runtime aggregation event. Roadmap state evaluation belongs to `sync-milestone-workstate`.
|
||||
5. Check applicable `Review-Only Checklist` items at the final `.log` location before reporting.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Item Completion
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| REVIEW_REVIEW_TEST-1 — Managed route-qualified capacity contract | [x] |
|
||||
| REVIEW_REVIEW_TEST-2 — Deterministic managed-capacity and provenance oracle | [x] |
|
||||
| REVIEW_REVIEW_TEST-3 — Route-realistic dev evidence and retained release gate | [x] |
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [REVIEW_REVIEW_TEST-1] Correct managed-capacity ownership in the project deployment skill, dev Edge smoke profile, and local guide.
|
||||
- [x] [REVIEW_REVIEW_TEST-2] Add the route-aware Chat/Responses capacity smoke with deterministic route, terminal, provenance, and stale-artifact self-tests.
|
||||
- [x] [REVIEW_REVIEW_TEST-3] Run both projected Ornith route matrices, replace disputed evidence from the sanitized current-run summaries, and finish the retained release only if every gate passes.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
## Review-Only Checklist
|
||||
|
||||
> **[REVIEW AGENT ONLY]** This checklist is used only by the review agent.
|
||||
> Implementing agents must not modify or check this section.
|
||||
|
||||
- [x] Append one verdict of `PASS`, `WARN`, or `FAIL` and verified `review_rework_count`, `evidence_integrity_failure` to `Code Review Result`.
|
||||
- [x] Verify that verdict, `Dimension Assessment`, and Required/Suggested/Nit classifications match.
|
||||
- [x] Run applicable required verification and record fresh command/output; repair reviewer-reconstructable evidence gaps instead of forwarding them to another plan.
|
||||
- [ ] For every Required/Suggested finding, record reviewer-collected `Evidence`, exact `Root Cause`, and one `Selected Fix` with affected files/symbols/tests and acceptance commands before creating a follow-up plan.
|
||||
- [x] Archive active `CODE_REVIEW-*-G??.md` to `code_review_cloud_G10_2.log`.
|
||||
- [x] Archive active `PLAN-*-G??.md` to `plan_cloud_G10_2.log`.
|
||||
- [x] Verify that the Agent-Ops managed block in `.gitignore` unignores `agent-task/**/*.md` and `agent-task/**/*.log` and ignores `agent-roadmap/current.md`.
|
||||
- [x] If PASS, write `complete.log` based on `agent-ops/skills/common/code-review/templates/complete-log-template.md` and leave no active `.md` files.
|
||||
- [x] If PASS, move active task directory `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/` to `agent-task/archive/YYYY/MM/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/` and update this checklist at the final archive path.
|
||||
- [x] If PASS and task group is `m-openai-compatible-output-validation-filters`, preserve and report `milestone-task` metadata for runtime aggregation, without modifying roadmap or directly calling `update-roadmap`.
|
||||
- [x] If PASS for split work, remove empty active parent `agent-task/m-openai-compatible-output-validation-filters/` or verify it was kept due to remaining siblings/files.
|
||||
- [ ] If WARN/FAIL, write the next filesystem state matching code-review verdict and do not write `complete.log`.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
- No implementation scope deviation was made. The previously unavailable active principal token was supplied through the already approved protected wrapper and stdin handoff; it was not printed or stored in tracked evidence.
|
||||
- The first passing live matrix exposed a macOS cross-process monotonic-clock epoch incompatibility in the informational duration field. The runner was corrected to use the shared wall-clock epoch, its local and remote self-tests and SHA identity were re-proved, and all four cases were rerun. The terminal, capacity, queue, recovery, and provenance gates were unchanged.
|
||||
- `shellcheck` is not installed in the local environment, so the selected `bash -n`, local self-test, and remote macOS Bash 3.2 self-test were used for the shell surface. `shellcheck` was not a plan acceptance command.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Each live invocation owns one authenticated route alias and one endpoint. Route resolution requires exactly one active alias and joins `resource_selector`, profile, and upstream model to exactly one healthy connected provider snapshot.
|
||||
- Capacity comes from only the selected provider and the computed request class: `capacity` for `normal`, `long_context_capacity` for `long`. The normal live path rejects a computed long request before dispatch.
|
||||
- Every invocation creates a unique mode-0700 ignored run directory and binds request/result/status files to an immutable run id, manifest, digest, and creation boundary. Terminal parsing consumes only manifest-owned response files and fails on stale, missing, or foreign-run paths.
|
||||
- The sanitized summary uses a fixed allowlist of run id, script hash, route alias, selected provider, endpoint, computed request shape/class, terminal counts, durations, selected peak/queue/final counters, provenance, and outcome. Raw routes, token/header values, route/slot ids, prompts, requests, response bodies, and model output remain in ignored run material.
|
||||
- The disputed v1 tracked evidence was replaced by a raw-free v2 summary only after the corrected runner produced four passing current-run summaries. Release finish and atomic push ran only after the final retained-release ref gate passed.
|
||||
|
||||
## Reviewer Checkpoints
|
||||
|
||||
- [x] Confirm managed capacity is derived from the authenticated route selector and only the selected provider's capacity for the computed context class.
|
||||
- [x] Confirm current projected aliases map exactly to `ornith:35b/onexplayer-lemonade` and `ornith-fast/rtx5090-lemonade` without route or capacity mutation.
|
||||
- [x] Confirm the request fixture is short, computed as `normal`, requests a 700-1200-token structured answer, and bounds provider-native thinking without relying on the caller max-token field as the duration oracle.
|
||||
- [x] Confirm Chat and Responses run separately with selected eligible capacity+1, exact endpoint success terminals, selected peak equal to capacity, queue at least one, and selected final counters `0/0`.
|
||||
- [x] Confirm non-selected providers are excluded from the claimed capacity total.
|
||||
- [x] Confirm every invocation creates a distinct mode-0700 run directory and every summary row is bound to its current run manifest, request/result path, and mtime.
|
||||
- [x] Confirm the self-test rejects route mismatch, aggregate-capacity claims, duplicate/missing terminals, stale/missing/foreign-run bodies, and non-current provenance.
|
||||
- [x] Confirm sanitized summaries and tracked evidence contain no token/header, route slot/id, prompt, response body, output, or credential material.
|
||||
- [x] Confirm the disputed 41k-token label, terminal-zero capacity claim, and fixed raw-artifact path were not retained as successful evidence.
|
||||
- [x] Confirm the retained release was finished only if every local/live/ref gate passed and no partial refs were pushed on failure.
|
||||
- [x] Confirm only files in `Modified Files Summary` changed.
|
||||
|
||||
## Verification Results
|
||||
|
||||
Record actual stdout/stderr for every command. If output is too long, save it under an ignored task-specific run directory and record the exact command/path; never reconstruct output.
|
||||
|
||||
### REVIEW_REVIEW_TEST-1 managed route-qualified contract
|
||||
|
||||
```bash
|
||||
python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
rg --sort path -n 'managed|resource_selector|eligible capacity|e2e-openai-managed-capacity-smoke' agent-ops/skills/project/dev-runtime-deploy/SKILL.md agent-test/dev/edge-smoke.md docs/edge-local-dev-guide.md
|
||||
```
|
||||
|
||||
Exit `0`:
|
||||
|
||||
Exact output: `.local/task-runs/stream-terminal-liveness/plan2-contract.log` (`sha256=6c249186293aac6795b1402dd6dec1f88f9709d9ee9d1b1ed85f005029dd6a66`).
|
||||
|
||||
```text
|
||||
Skill is valid!
|
||||
agent-ops/skills/project/dev-runtime-deploy/SKILL.md:146: In managed mode, authenticate GET /v1/credentials/routes ... intersect its resource_selector, profile, and upstream model ...
|
||||
agent-ops/skills/project/dev-runtime-deploy/SKILL.md:147: Run scripts/e2e-openai-managed-capacity-smoke.sh ... selected provider's eligible capacity plus one ... never add a provider excluded by that route.
|
||||
agent-ops/skills/project/dev-runtime-deploy/SKILL.md:149: Require HTTP 200 ... exactly one endpoint-native success terminal and one [DONE] ... final counters 0/0.
|
||||
agent-test/dev/edge-smoke.md:153: dev-runtime managed capacity smoke는 scripts/e2e-openai-managed-capacity-smoke.sh를 사용한다 ... ornith:35b는 onexplayer-lemonade=3, ornith-fast는 rtx5090-lemonade=1 ...
|
||||
agent-test/dev/edge-smoke.md:154: emitted request JSON ... runes/4 + runes/16 ... normal qualification은 capacity ... long_context_capacity ...
|
||||
agent-test/dev/edge-smoke.md:157: HTTP 200 ... endpoint-native success terminal 정확히 1개, [DONE] 정확히 1개 ...
|
||||
agent-test/dev/edge-smoke.md:158: mode 0700 unique directory ... stale/missing/foreign-run body는 실패한다.
|
||||
docs/edge-local-dev-guide.md:170: Managed mode의 capacity는 ... 모든 provider의 합이 아닙니다 ... resource_selector ...
|
||||
docs/edge-local-dev-guide.md:172: scripts/e2e-openai-managed-capacity-smoke.sh ... capacity + 1 ...
|
||||
docs/edge-local-dev-guide.md:174: terminal ... peak ... queue ... final in_flight=0/queued=0 ... fail-closed ...
|
||||
```
|
||||
|
||||
### REVIEW_REVIEW_TEST-2 deterministic smoke and provenance self-test
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
```
|
||||
|
||||
Both commands exited `0`. Self-test output:
|
||||
|
||||
Exact output: `.local/task-runs/stream-terminal-liveness/plan2-self-test.log` (`sha256=a1e6efa43684683bc9b0299a20418e6e1e273d4a02e9c2034b5d49de572e06af`).
|
||||
|
||||
```text
|
||||
[managed-capacity-smoke] self-test unique_run_directories=true mode=0700
|
||||
[managed-capacity-smoke] self-test rejected=route-selector-mismatch
|
||||
[managed-capacity-smoke] self-test rejected=aggregate-capacity-claim
|
||||
[managed-capacity-smoke] self-test route_exclusion=true normal_capacity=3 long_capacity=2
|
||||
[managed-capacity-smoke] self-test rejected=duplicate-terminal
|
||||
[managed-capacity-smoke] self-test rejected=missing-terminal
|
||||
[managed-capacity-smoke] self-test exact_chat_and_responses_terminals=true
|
||||
[managed-capacity-smoke] self-test rejected=stale-body
|
||||
[managed-capacity-smoke] self-test rejected=missing-body
|
||||
[managed-capacity-smoke] self-test rejected=foreign-run-body
|
||||
[managed-capacity-smoke] self-test current_manifest_acceptance=true stale_missing_foreign_rejected=true
|
||||
[managed-capacity-smoke] SELF_TEST_PASS
|
||||
```
|
||||
|
||||
The corrected copied script passed under remote macOS Bash 3.2. Its local and remote SHA-256 both equal:
|
||||
|
||||
```text
|
||||
bc0dcf5aa8acc6a10942a6d48e4b3248547004df04d0a292562aea2ca465a903
|
||||
```
|
||||
|
||||
### REVIEW_REVIEW_TEST-3 dev route matrices and retained release
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev && git rev-parse origin/release/dev-974 && git rev-list --left-right --count HEAD...origin/dev && git tag -l dev-974 && go version && git flow version | head -n 1'\'''
|
||||
```
|
||||
|
||||
Run the new smoke through the authorized protected-token wrapper for `ornith:35b/onexplayer-lemonade` and `ornith-fast/rtx5090-lemonade`, each with Chat and Responses. Record exact sanitized summaries and release finish/tag/atomic-push output or the exact blocker and retained-release state.
|
||||
|
||||
Retained-release preflight exited `0`:
|
||||
|
||||
```text
|
||||
## release/dev-974...origin/release/dev-974
|
||||
HEAD=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
origin/dev=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
origin/release/dev-974=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
HEAD...origin/dev=0 0
|
||||
local dev-974 tag count=0
|
||||
remote dev-974 tag count=0
|
||||
go version go1.26.3 darwin/arm64
|
||||
git-flow 1.12.3 (AVH Edition)
|
||||
```
|
||||
|
||||
The approved local `/tmp/iop-bench-13-env` wrapper supplied the existing active principal token over SSH protected stdin to the remote runner without printing it; the wrapper is intentionally not copied to the remote host. Corrected-run sanitized summaries:
|
||||
|
||||
```text
|
||||
ornith:35b chat: run_id=1786601559737433000-4ba66d5c48201ad1 provider=onexplayer-lemonade class=normal estimate=133 capacity=3 requests=4 HTTP200=4 finish=4 done=4 errors=0 duration_ms=35049..60703 peak=3 max_queued=1 final=0/0 provenance=current-run-manifest outcome=pass
|
||||
ornith:35b responses: run_id=1786601621300814000-c4d9c6c134fe368b provider=onexplayer-lemonade class=normal estimate=126 capacity=3 requests=4 HTTP200=4 completed=4 done=4 errors=0 duration_ms=35451..57073 peak=3 max_queued=1 final=0/0 provenance=current-run-manifest outcome=pass
|
||||
ornith-fast chat: run_id=1786601679237576000-53b4cfe86fa3dc40 provider=rtx5090-lemonade class=normal estimate=135 capacity=1 requests=2 HTTP200=2 finish=2 done=2 errors=0 duration_ms=3402..6602 peak=1 max_queued=1 final=0/0 provenance=current-run-manifest outcome=pass
|
||||
ornith-fast responses: run_id=1786601686638286000-825708e93ce3450d provider=rtx5090-lemonade class=normal estimate=127 capacity=1 requests=2 HTTP200=2 completed=2 done=2 errors=0 duration_ms=3223..6399 peak=1 max_queued=1 final=0/0 provenance=current-run-manifest outcome=pass
|
||||
script_sha256=bc0dcf5aa8acc6a10942a6d48e4b3248547004df04d0a292562aea2ca465a903
|
||||
```
|
||||
|
||||
The v2 tracked evidence redaction check exited `0` and rejected token/header, raw artifact, prompt/body, route-id, and slot-id fields. Its SHA-256 is `965407296852ee25a3ccc5e2069e21faff8386da92e911dd80682d24476a12e7`.
|
||||
|
||||
Final retained-release gate and finish:
|
||||
|
||||
```text
|
||||
candidate=2fcc1093c7629ab94b460d083520ffd9c8064815
|
||||
candidate_tree=d46c7e3057bafbeedc1ddb10d2619feb3c6eb2db
|
||||
pre_finish origin/dev=2fcc1093c7629ab94b460d083520ffd9c8064815 origin/main=0c1e9e6a3f181eaebe59f888043fadc37385741d tag_count=0 release_remote_count=1 tracked_status=clean
|
||||
git flow release finish --keepremote -m "Release dev-974" dev-974: PASS
|
||||
git push --atomic origin main dev dev-974 :release/dev-974: PASS
|
||||
final main=origin/main=fd41adac777f7c7faea0cdd27acbe2817890abf8
|
||||
final dev=origin/dev=019500a5d65117bfd115e0c33601ceef3461ee52
|
||||
final tag_object=4d2596e3779027ba88457c44fcc9bc91134cf5e9 tag_tree=d46c7e3057bafbeedc1ddb10d2619feb3c6eb2db
|
||||
final local_release_count=0 remote_release_count=0 tracked_status=clean
|
||||
```
|
||||
|
||||
### Final local qualification
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
All executable local qualification commands exited `0`. Exact ignored output paths and SHA-256:
|
||||
|
||||
```text
|
||||
.local/task-runs/stream-terminal-liveness/plan2-self-test.log a1e6efa43684683bc9b0299a20418e6e1e273d4a02e9c2034b5d49de572e06af
|
||||
.local/task-runs/stream-terminal-liveness/plan2-contract.log 6c249186293aac6795b1402dd6dec1f88f9709d9ee9d1b1ed85f005029dd6a66
|
||||
.local/task-runs/stream-terminal-liveness/plan2-node.log d55dbf369fb2761e401bf0f0ac53d6f10093f5e081cb88657c86a07c4c20106c
|
||||
.local/task-runs/stream-terminal-liveness/plan2-edge.log 8dfa7d08ab348b15aae58561ac98f688af78fb68c0e230aae46f70e206f3ad68
|
||||
.local/task-runs/stream-terminal-liveness/plan2-lemonade.log 4bfc218d83fb5cd90819978c9bfee79d35d8178cdf2c9c62df40718aec514a80
|
||||
.local/task-runs/stream-terminal-liveness/plan2-diff-check.log e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
|
||||
.local/task-runs/stream-terminal-liveness/plan2-status.log 1d6c83b3b8470ad2f6d386802f20957c2c5d520d5020e5d206d4edfd4fdc863c
|
||||
```
|
||||
|
||||
Short output:
|
||||
|
||||
```text
|
||||
bash -n: PASS
|
||||
self-test: SELF_TEST_PASS
|
||||
skill validation: Skill is valid!
|
||||
ok iop/apps/node/internal/node 0.141s
|
||||
ok iop/apps/edge/internal/openai 0.148s
|
||||
[openai-lemonade] OpenAI-compatible Lemonade serving test PASSED (mode=fake).
|
||||
git diff --check: PASS
|
||||
## feature/openai-compatible-stream-terminal-liveness...origin/feature/openai-compatible-stream-terminal-liveness [ahead 12]
|
||||
M agent-ops/skills/project/dev-runtime-deploy/SKILL.md
|
||||
M agent-test/dev/edge-smoke.md
|
||||
M docs/edge-local-dev-guide.md
|
||||
?? agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log
|
||||
?? agent-task/m-openai-compatible-output-validation-filters/
|
||||
?? scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
```
|
||||
|
||||
`.gocache/` and `.local/` are pre-existing ignored workspace material and were not added to the plan write set. The evidence path is now the v2 sanitized current-run summary produced after all four live cases passed.
|
||||
|
||||
### Resumed worker verification
|
||||
|
||||
The resumed worker reran the deterministic gates before the authorized live handoff became available:
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
git diff --check
|
||||
```
|
||||
|
||||
All corrected commands exited `0`:
|
||||
|
||||
```text
|
||||
[managed-capacity-smoke] SELF_TEST_PASS
|
||||
Skill is valid!
|
||||
ok iop/apps/node/internal/node
|
||||
ok iop/apps/edge/internal/openai
|
||||
[openai-lemonade] OpenAI-compatible Lemonade serving test PASSED (mode=fake).
|
||||
git diff --check: PASS
|
||||
```
|
||||
|
||||
The implementation was subsequently resumed through the approved protected wrapper; the successful live and release evidence above supersedes that temporary external-blocker handoff.
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT — BEFORE SAVING] Have you filled in every implementation-owned section?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## Section Ownership
|
||||
|
||||
| Section | Owner | Note |
|
||||
|---|---|---|
|
||||
| Header comment, Overview, Review Agent Instructions | Fixed at stub creation | Implementing agent must not modify or execute these (archive, complete.log, and task-directory archive move are review-agent only) |
|
||||
| Archive Evidence Snapshot | Fixed at stub creation from plan when present | Implementing agent uses it as default prior-loop context; read only the specific archive files cited there when more detail is required |
|
||||
| Implementation Item Completion (item names) | Fixed at stub creation | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Implementation Checklist (item text/order) | Fixed at stub creation from plan | Implementing agent checks `[ ]` → `[x]` only |
|
||||
| Review-Only Checklist | Review agent only | Implementing agent must not modify or check this section |
|
||||
| Deviations from Plan, Key Design Decisions | Implementing agent | Replace placeholder text with actual content |
|
||||
| Reviewer Checkpoints | Fixed at stub creation | Pre-filled from plan |
|
||||
| Verification Results (section headings + commands) | Implementing agent, then review agent | Implementing agent records initial output; review agent reruns applicable commands and may fill, replace, or append fresh verified output before verdict. Implementing-agent command changes require a `Deviations from Plan` entry |
|
||||
| Code Review Result | Review agent appends | Not included in stub |
|
||||
|
||||
## Reviewer Fresh Verification — 2026-08-13
|
||||
|
||||
### Local acceptance rerun
|
||||
|
||||
The reviewer reran the plan's executable local gates. Full stdout/stderr was captured at `/tmp/iop-code-review-plan2.LIOA8I/reviewer-local.log` with SHA-256 `38abbfc9eada6b0f2ea760896d41e94b37a2b9f83cac7f7b34d1dd2b7aaa5a82`.
|
||||
|
||||
```text
|
||||
$ bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
PASS
|
||||
$ ./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
[managed-capacity-smoke] self-test rejected=route-selector-mismatch
|
||||
[managed-capacity-smoke] self-test rejected=aggregate-capacity-claim
|
||||
[managed-capacity-smoke] self-test exact_chat_and_responses_terminals=true
|
||||
[managed-capacity-smoke] self-test rejected=stale-body
|
||||
[managed-capacity-smoke] self-test rejected=missing-body
|
||||
[managed-capacity-smoke] self-test rejected=foreign-run-body
|
||||
[managed-capacity-smoke] SELF_TEST_PASS
|
||||
$ python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
Skill is valid!
|
||||
$ go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
ok iop/apps/node/internal/node 0.059s
|
||||
$ go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'
|
||||
ok iop/apps/edge/internal/openai 0.044s
|
||||
$ IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
[openai-lemonade] OpenAI-compatible Lemonade serving test PASSED (mode=fake).
|
||||
$ git diff --check
|
||||
PASS
|
||||
```
|
||||
|
||||
### Remote current-run and release validation
|
||||
|
||||
The reviewer independently opened the four ignored remote run manifests and sanitized summaries without printing request/response bodies or credential material. For every case, the route alias resolved to one active route; `resource_selector`, non-empty profile, upstream model, connected provider snapshot, current-run owner, mtime, request/result digest, endpoint terminal, eligible capacity, peak, queue, and final recovery all matched.
|
||||
|
||||
```text
|
||||
run=1786601559737433000-4ba66d5c48201ad1 endpoint=chat provider=onexplayer-lemonade profile=present upstream_match=true provenance=digest+mtime+owner terminal=pass capacity=3 peak=3 queue=1 final=0/0
|
||||
run=1786601621300814000-c4d9c6c134fe368b endpoint=responses provider=onexplayer-lemonade profile=present upstream_match=true provenance=digest+mtime+owner terminal=pass capacity=3 peak=3 queue=1 final=0/0
|
||||
run=1786601679237576000-53b4cfe86fa3dc40 endpoint=chat provider=rtx5090-lemonade profile=present upstream_match=true provenance=digest+mtime+owner terminal=pass capacity=1 peak=1 queue=1 final=0/0
|
||||
run=1786601686638286000-825708e93ce3450d endpoint=responses provider=rtx5090-lemonade profile=present upstream_match=true provenance=digest+mtime+owner terminal=pass capacity=1 peak=1 queue=1 final=0/0
|
||||
REMOTE_MANIFEST_VALIDATION_PASS
|
||||
```
|
||||
|
||||
The copied remote runner and local source both hash to `bc0dcf5aa8acc6a10942a6d48e4b3248547004df04d0a292562aea2ca465a903`. Remote `dev` is clean at `019500a5d65117bfd115e0c33601ceef3461ee52`, equal to `origin/dev`; `origin/main` is `fd41adac777f7c7faea0cdd27acbe2817890abf8`; `dev-974^{tree}` is the candidate tree `d46c7e3057bafbeedc1ddb10d2619feb3c6eb2db`; the remote tag exists exactly once; `release/dev-974` no longer exists; `config check` passes; ports `18082,18083,18084,19093,19101` are open.
|
||||
|
||||
### Evidence projection check and reviewer repair
|
||||
|
||||
The reviewer parsed the tracked v2 evidence as unique key/value rows and asserted all four route/endpoint counts, terminal counts, selected capacities, peaks, queue evidence, final recovery, script hash, raw-material exclusion marker, release finish, atomic push, and release-branch deletion. The check passed. Two non-behavioral Nits were repaired directly: the implementation had translated the existing `dev-runtime-deploy` frontmatter description and `목적` heading/body into English, and one procedure line implied that the normal-only managed script accepted long requests. The unrelated language drift was restored, and the procedure now fails the normal gate on a long classification and points to the separate long-context smoke.
|
||||
|
||||
## Code Review Result
|
||||
|
||||
- **Overall Verdict:** PASS
|
||||
|
||||
### Dimension Assessment
|
||||
|
||||
| Dimension | Result | Assessment |
|
||||
|---|---|---|
|
||||
| Correctness | Pass | Managed capacity is computed from one authenticated route selector and its selected provider; all four live route/endpoint cases satisfy exact capacity, queue, terminal, and recovery gates. |
|
||||
| Completeness | Pass | The route-qualified contract, deterministic oracle, v2 evidence replacement, retained release finish, tag, atomic push, and release cleanup are complete. |
|
||||
| Test coverage | Pass | Self-test negatives cover route mismatch, aggregate capacity, terminal duplication/absence, and stale/missing/foreign provenance; focused Node/Edge and fake Lemonade regressions pass. |
|
||||
| API contract | Pass | Chat and Responses preserve their endpoint-native success terminal plus one `[DONE]`, while managed route/profile/upstream binding remains authenticated and caller-neutral. |
|
||||
| Code quality | Pass | Shell syntax, skill validation, diff check, bounded secret cleanup, unique 0700 run ownership, digest/mtime checks, and allowlisted summaries pass. |
|
||||
| Implementation deviation | Pass | The macOS elapsed-clock correction preserved the selected gates; the only unrelated language drift was directly repaired as a Nit. |
|
||||
| Verification trust | Pass | Reviewer-local reruns and independent remote manifest/digest/ref validation agree with the recorded results; no contradictory evidence remains. |
|
||||
| Spec conformance | Pass | The contribution evidence satisfies the selected S03/S04/S07 terminal and raw-free operational gates for `repeat-guard,ops-evidence`; Milestone completion remains deferred to runtime aggregation. |
|
||||
|
||||
### Findings
|
||||
|
||||
- Required: None.
|
||||
- Suggested: None.
|
||||
- Nit (repaired): Restored the pre-existing Korean frontmatter description and `목적` text in `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`; no behavior or acceptance criterion changed.
|
||||
- Nit (repaired): Aligned the deployment procedure with the implemented normal-only managed smoke and directed long requests to the separate long-context admission smoke.
|
||||
|
||||
### Routing Signals
|
||||
|
||||
- `review_rework_count=2`
|
||||
- `evidence_integrity_failure=false`
|
||||
|
||||
### Next Step
|
||||
|
||||
Archive the active plan/review pair, write `complete.log`, move this split task under the current month archive, and emit the `m-openai-compatible-output-validation-filters` completion metadata for runtime aggregation. Do not modify the roadmap directly.
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=2 tag=REVIEW_REVIEW_TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Complete - m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-08-13
|
||||
|
||||
## 요약
|
||||
|
||||
3회의 plan/review 루프 끝에 managed route-qualified capacity smoke, current-run provenance, 네 개 Ornith live matrix, raw-free v2 evidence, `dev-974` retained release를 모두 검증하고 최종 PASS로 종료했다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_cloud_G09_0.log` | `code_review_cloud_G09_0.log` | FAIL | semantic-disabled Chat/Responses tunnel terminal 소유권과 dev timeout-order evidence 미충족 |
|
||||
| `plan_cloud_G10_1.log` | `code_review_cloud_G10_1.log` | FAIL | managed route가 접근할 수 없는 aggregate capacity와 mixed-generation 아티팩트 provenance 발견 |
|
||||
| `plan_cloud_G10_2.log` | `code_review_cloud_G10_2.log` | PASS | route-qualified 네 개 live case, manifest/digest/mtime 소유권, raw-free evidence, release finish/tag/atomic push 검증 |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- Managed principal route의 `resource_selector`, profile, upstream model을 하나의 healthy provider snapshot과 교차해 eligible capacity를 계산하는 Chat/Responses smoke를 추가했다.
|
||||
- 각 invocation을 고유한 mode-0700 run directory, immutable run id, manifest, request/result digest, dispatch mtime, result owner에 묶고 stale/missing/foreign-run body를 fail-closed로 처리했다.
|
||||
- `ornith:35b`/OneXPlayer와 `ornith-fast`/RTX5090의 Chat/Responses 네 케이스를 분리 실행해 HTTP 200, exact terminal, peak=capacity, queue>=1, final 0/0을 확인했다.
|
||||
- 분쟁된 v1 evidence를 current-run sanitized v2 evidence로 교체하고 `dev-974` release finish, tag, atomic push, remote release branch 삭제를 완료했다.
|
||||
- 리뷰 중 발견한 `dev-runtime-deploy` frontmatter/목적의 불필요한 영문 번역과 normal-only 스크립트의 long-context 처리 오해 가능성을 문서 Nit로 복구했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `bash -n scripts/e2e-openai-managed-capacity-smoke.sh` - PASS.
|
||||
- `./scripts/e2e-openai-managed-capacity-smoke.sh --self-test` - PASS; route/aggregate/terminal/provenance negative fixture가 모두 reject됨.
|
||||
- `python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy` - PASS; `Skill is valid!`.
|
||||
- `go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'` - PASS; `ok iop/apps/node/internal/node 0.059s`.
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'` - PASS; `ok iop/apps/edge/internal/openai 0.044s`.
|
||||
- `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh` - PASS.
|
||||
- 원격 네 run manifest/digest/mtime/owner/route/profile/upstream/terminal/capacity 독립 검증 - PASS; `REMOTE_MANIFEST_VALIDATION_PASS`.
|
||||
- 원격 ref/runtime 검증 - PASS; clean `dev=origin/dev`, `dev-974` tag 1개, remote release branch 0개, `config check` PASS, 포트 `18082,18083,18084,19093,19101` open.
|
||||
- tracked v2 evidence schema/redaction/matrix/release assertion - PASS.
|
||||
- `git diff --check` - PASS.
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- 없음
|
||||
|
|
@ -0,0 +1,245 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=0 tag=TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# OpenAI-compatible stream terminal liveness and timeout ownership
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement the deterministic terminal-liveness regressions and the dev timeout-order rollout exactly as written. Preserve the existing protocol contract: an OpenAI-compatible `finish_reason` is releasable wire state, while `[DONE]`, provider EOF, or tunnel `END` remains the transport terminal. Do not synthesize a Core terminal from a normal `finish_reason`, change routing, or add another terminal grace mechanism. Run every verification command, fill the implementation-owned sections of `CODE_REVIEW-cloud-G09.md` with actual output, keep both active files in place, and report ready for review. If blocked, record only the exact blocker, attempted command/output, and resume condition in the review evidence; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
`/config/workspace/iop/HANDOFF-ORNITH-SESSION-STALL.md` records a real Ornith request in which Pi was terminated after roughly 180 seconds of silence with exit 143. The request had already produced provider content but no observed Node tunnel `END`, provider terminal, or Core terminal. The current IOP defaults allow a Node response stall to wait 300 seconds and an Edge request to wait 3600 seconds, so the external 180-second caller can terminate before IOP owns the failure and recovery boundary.
|
||||
|
||||
The current implementation already has typed `response_stalled` watchdog handling, Edge pre-commit recovery, post-commit sanitized termination, exact replay gates, and single-terminal fencing. The gap is narrower: the actual logical-finish-without-transport-terminal shape is not represented by a deterministic Node/Edge end-to-end regression, and the dev Ornith provider timeout order is not aligned with the 180-second caller boundary.
|
||||
|
||||
The active milestone `openai-compatible-output-validation-filters` already owns repeat-guard idle behavior, tunnel/codec terminal integrity, Ornith live smoke, and operational evidence. This task therefore remains a split subtask of existing milestone `[output-01]`; it does not create or promote a second milestone.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Dependency evidence: `agent-task/archive/2026/07/m-openai-compatible-output-validation-filters/03_repeat_guard_s07_remote_evidence/complete.log` is `PASS` and satisfies predecessor index `03` for this `04+03` packet.
|
||||
- That completion proves the previous repeat-guard remote evidence and single-terminal validation passed for milestone task `repeat-guard`.
|
||||
- It does not prove the Ornith logical-finish-without-`[DONE]`/EOF/`END` stall shape or the dev timeout-order rollout; those remain owned by this packet.
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. `03_repeat_guard_s07_remote_evidence` is complete at the exact archive path above; do not reread sibling archive tasks.
|
||||
2. Complete TEST-1 and TEST-2 locally before changing or restarting any dev runtime.
|
||||
3. TEST-3 may start only after local regressions pass and the remote workspace is clean, synchronized to the selected `origin/dev` release ref, and rebuilt from that same ref.
|
||||
4. The deterministic tests are mandatory even if the real external stall cannot be reproduced. A live `not_reproduced` result is evidence, not a substitute for TEST-1 or TEST-2.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `apps/node/internal/adapters/openai_compat/provider_tunnel.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_test.go`
|
||||
- `apps/edge/internal/openai/stream_gate_tunnel_codec.go`
|
||||
- `apps/edge/internal/openai/stream_gate_pipeline_test.go`
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
|
||||
- `packages/go/execution/liveness.go`
|
||||
- `scripts/e2e-openai-lemonade.sh`
|
||||
- `docs/edge-local-dev-guide.md`
|
||||
- `agent-spec/index.md`
|
||||
- `agent-spec/openai/stream-evidence-gate.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-spec/runtime/provider-pool-config-refresh.md`
|
||||
- `agent-spec/openai/openai-compatible-surface.md`
|
||||
- `agent-contract/index.md`
|
||||
- `agent-contract/outer/openai-api.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/openai-compatible-output-validation-filters.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/dev/rules.md`
|
||||
- applicable local/dev Edge, Node, platform-common, and testing profiles
|
||||
- applicable Edge, Node, platform-common, and testing domain rules
|
||||
- `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`
|
||||
- `/config/workspace/iop/HANDOFF-ORNITH-SESSION-STALL.md`
|
||||
- `agent-task/archive/2026/07/m-openai-compatible-output-validation-filters/03_repeat_guard_s07_remote_evidence/complete.log`
|
||||
|
||||
### SDD and Contract Criteria
|
||||
|
||||
- SDD status is approved and unlocked. This packet maps to milestone tasks `repeat-guard,ops-evidence` and preserves the approved evidence and single-terminal gates.
|
||||
- Supported Chat/Responses traffic keeps StreamGate as the liveness owner even when semantic validation filters are disabled.
|
||||
- `finish_reason=length` managed continuation is an approved semantic case, but normal `finish_reason=stop` is not a transport terminal and must not be promoted to one by this task.
|
||||
- The typed `response_stalled` error, Node watchdog, Edge validated recovery, exact replay boundary, post-commit sanitized error, and terminal fence are existing contract behavior. This task adds missing shape coverage and operational timeout ordering; it does not redesign those contracts.
|
||||
- Current inventory intentionally routes `ornith:35b` to `onexplayer-lemonade` and `rtx5090-lemonade`. No provider catalog, route, or capacity change is in scope.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Branch: `feature/openai-compatible-stream-terminal-liveness`, created with Git Flow from synchronized `dev` and published to `origin/feature/openai-compatible-stream-terminal-liveness` at base `841511472a62ec20d79eac5f800180d1de34b541` before this plan was written.
|
||||
- Fresh local preflight passed for the focused Node watchdog tests, Edge terminal/stall tests, and `git diff --check` with Go `1.26.2 linux/arm64`.
|
||||
- The current Node adapter relays `BODY` until provider EOF and then emits `END`; the watchdog counts response-start/body/usage as progress and treats `END` or error as terminal.
|
||||
- The current Edge codec stages `finish_reason` as wire data and creates a Core terminal only for `[DONE]` or tunnel `END`. Existing tests cover the general stall-recovery matrix but not the exact finish-frame-followed-by-missing-terminal shape.
|
||||
- The fake Lemonade smoke covers normal `finish_reason`/usage/`[DONE]`; it has no deterministic hang-after-finish fixture.
|
||||
- Confidence is high that the 300-second IOP response-stall default loses ownership to the 180-second caller. Confidence is moderate that this was the sole real-world trigger because no provider EOF/Node `END` trace was captured; TEST-1/2 close the deterministic boundary while TEST-3 collects sanitized live evidence.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/workdir: `ssh toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, using login zsh so `/opt/homebrew/bin/git-flow` and the declared dev toolchain are available.
|
||||
- Read-only inspection found the checkout on `dev`, clean but three commits behind `origin/dev`; the deployed Edge binary source identity did not match the checkout. Do not reuse that checkout or binary.
|
||||
- Before rollout, fetch and cleanly synchronize to the selected latest `origin/dev` release ref, record branch/HEAD/dirty/ancestry state, and follow the project `dev-runtime-deploy` skill for a full same-ref rebuild, config check, deploy, restart, and source/build identity proof.
|
||||
- Confirm ports `18082`, `18083`, `18084`, `19093`, and `19101`, connected Nodes, expected provider snapshots, and the exact deployed Edge/Node build identities before live smoke.
|
||||
- Current `build/dev-runtime/edge.yaml` omits `response_stall_timeout_ms`, so the 300-second default is active. The rollout must explicitly set `120000` for both dev Ornith Lemonade provider entries and preserve an Edge request timeout longer than that value.
|
||||
- Record only sanitized timing/outcome evidence in tracked files. Raw prompts, provider output, tokens, credentials, and full request/response bodies stay in ignored run artifacts and must never enter the plan, review file, evidence log, or terminal transcript.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Node: no exact regression proves that a body containing a logical finish but lacking provider EOF/tunnel `END` expires once as typed `response_stalled` and fences late terminal frames.
|
||||
- Edge: no exact matrix drives that shape through both pre-commit replay eligibility and post-commit sanitized termination while asserting no raw leakage and one Core terminal.
|
||||
- Adapter-to-runtime smoke: no deterministic fake provider flushes a final content/finish frame and then remains open until cancellation.
|
||||
- Operations: no tracked guide/evidence establishes `response_stall_timeout_ms < caller kill timeout < edge request timeout` for the current Ornith dev provider pool.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- `apps/node/internal/adapters/openai_compat/provider_tunnel.go`: provider body relay and EOF-to-`END` emission; observe through tests, do not change terminal semantics.
|
||||
- `packages/go/execution/liveness.go`: typed liveness vocabulary and terminal/progress classification; reuse without adding a new error type.
|
||||
- `apps/edge/internal/openai/stream_gate_tunnel_codec.go`: `finish_reason`, `[DONE]`, usage, `END`, and single-terminal handling; preserve implementation and strengthen assertions.
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: pre/post-commit recovery matrix owner.
|
||||
- `apps/edge/internal/openai/stream_gate_pipeline_test.go`: exact wire order and logical-vs-transport terminal assertions.
|
||||
- `scripts/e2e-openai-lemonade.sh`: deterministic adapter → Node watchdog → Edge runtime fixture owner.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
Keep one bounded test-and-operations packet. The local regression proves the exact event shape, and the timeout rollout proves the same liveness owner wins before the external caller. Splitting them would leave either an unqualified operational change or a regression that does not resolve the observed timeout order. No product-source or contract migration is necessary.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- In scope: Node/Edge tests, fake Lemonade end-to-end fixture, dev timeout-order documentation, same-ref dev rollout, and sanitized evidence.
|
||||
- Out of scope: provider routing/capacity changes, new fallback policy, Core/outer API schema changes, synthesizing terminal state from `finish_reason`, production deployment, Chronos/Pi timeout modification, raw prompt/output retention, and edits to common Agent-Ops rules or skills.
|
||||
- If TEST-1 or TEST-2 reveals an actual source defect rather than a missing regression, stop and record the blocker. Do not expand this test plan into an unreviewed runtime fix.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=first-pass`; build and review closures are true with no capability gap.
|
||||
- Finalizer: `finalize-task-policy.sh pair`.
|
||||
- Build: `grade-boundary/cloud/G09`, catalog `worker/cloud/G09`, filename `PLAN-cloud-G09.md`.
|
||||
- Review: `official-review/cloud/G09`, catalog `review/cloud/G09`, filename `CODE_REVIEW-cloud-G09.md`.
|
||||
- Grade scores: build `2/2/1/2/2`, review `2/2/1/2/2` for scope/state/blast/evidence/verification.
|
||||
- `large_indivisible_context=false`; positive loop risks are `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, and `variant_product` (5).
|
||||
- Recovery signals: `review_rework_count=0`, `evidence_integrity_failure=false`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [TEST-1] Add exact Node and Edge logical-finish-without-transport-terminal regressions while preserving the existing terminal contract.
|
||||
- [ ] [TEST-2] Extend the fake Lemonade end-to-end smoke with a bounded hang-after-finish fixture and sanitized single-terminal assertions.
|
||||
- [ ] [TEST-3] Align both dev Ornith provider response-stall timeouts, perform a same-ref rollout, update the runbook, and capture sanitized live evidence.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [TEST-1] Logical finish followed by transport stall
|
||||
|
||||
**Problem:** Current unit coverage proves general stall recovery and normal terminal ordering but does not reproduce the observed sequence: response body and logical `finish_reason`, followed by silence with no `[DONE]`, provider EOF, tunnel `END`, or Core terminal.
|
||||
|
||||
**Solution:** Add a Node watchdog regression named `TestTunnelWatchdogFinishFrameWithoutEndStallsOnce`. Feed response-start and a body frame containing a logical finish, withhold `END`, expire the watchdog, and assert exactly one typed `response_stalled`; late usage/`END` must be fenced and must not create a second terminal. Add an Edge table regression named `TestOpenAIStallAfterLogicalFinishMatrix` with:
|
||||
|
||||
- pre-commit/under-threshold repeat-guard state: pending finish wire remains hidden, one eligible alternate replay occurs, and only the recovered output plus one terminal is exposed;
|
||||
- post-commit state: no replay occurs, the committed stream emits exactly one sanitized SSE error followed by `[DONE]`, pending state is cleaned up, and no provider/raw diagnostic bytes leak;
|
||||
- success control: content/finish, usage, `[DONE]`, and physical `END` preserve exact byte order and produce one Core terminal;
|
||||
- contract assertion: a finish frame alone does not create a Core terminal before `[DONE]` or `END`.
|
||||
|
||||
Do not modify the watchdog, codec, adapter, StreamGate, or liveness source unless a follow-up reviewed fix is created.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `apps/node/internal/node/liveness_watchdog_test.go`: add the finish-without-`END` expiry/fence regression.
|
||||
- [ ] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: add pre/post-commit exact-shape recovery cases.
|
||||
- [ ] `apps/edge/internal/openai/stream_gate_pipeline_test.go`: add logical-vs-transport terminal and exact success-order assertions.
|
||||
- [ ] `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md`: record fresh focused/full/race outputs.
|
||||
|
||||
**Test Strategy:** Use fake clocks/channels and existing test helpers only. Assert typed codes, replay count, byte order, pending cleanup, raw-data absence, and exact terminal counts; do not use wall-clock sleeps or network access.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAITunnelCodecTerminalWire)$'
|
||||
```
|
||||
|
||||
Expected: both commands exit 0; the missing transport terminal becomes one typed stall, the Edge branch matches commit state, and normal terminal order remains unchanged.
|
||||
|
||||
### [TEST-2] Deterministic Lemonade adapter-to-Edge smoke
|
||||
|
||||
**Problem:** The existing fake Lemonade smoke always completes normally, so it cannot prove the real adapter/watchdog/Edge composition for a provider that flushes a logical finish and then leaves the response open.
|
||||
|
||||
**Solution:** Add a bounded fixture request recognized only by the fake provider. It writes content plus `finish_reason`, flushes, omits `[DONE]` and EOF, and blocks until request-context cancellation. Generate a test-only provider config with a short `response_stall_timeout_ms` such as 200 ms. Drive the request through the actual OpenAI-compatible provider adapter, Node watchdog, and Edge runtime; permit only the expected typed stall in the script's failure scan and continue rejecting every other error marker.
|
||||
|
||||
Assert one sanitized client terminal, no duplicate error/`[DONE]`, no raw provider diagnostic leakage, prompt handling after cancellation, and clean process teardown. Retain the existing normal content/finish/usage/`[DONE]` control and exact ordering assertions.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `scripts/e2e-openai-lemonade.sh`: add the bounded hang-after-finish fixture, test-only timeout, expected-error allowlist, exact assertions, and cleanup.
|
||||
- [ ] `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md`: record the fake-mode output without request or response bodies.
|
||||
|
||||
**Test Strategy:** Run fake mode locally; use deterministic synchronization and the request context rather than an unbounded sleep. The script must fail on duplicate terminals, timeout-owner mismatch, unexpected error types, leaked raw diagnostics, or a surviving fake-provider process.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
```
|
||||
|
||||
Expected: exit 0; the normal control is unchanged and the stall fixture ends once through IOP's typed/sanitized path.
|
||||
|
||||
### [TEST-3] Dev Ornith timeout order and sanitized evidence
|
||||
|
||||
**Problem:** Both current Ornith Lemonade provider entries inherit a 300-second Node response-stall timeout, while the observed caller is killed after about 180 seconds. The stale remote checkout/binary also prevents trustworthy live attribution.
|
||||
|
||||
**Solution:** Document and apply this dev-only ordering for both `onexplayer-lemonade` and `rtx5090-lemonade` Ornith provider entries:
|
||||
|
||||
```text
|
||||
response_stall_timeout_ms = 120000
|
||||
caller kill timeout = approximately 180000
|
||||
edge request timeout > 180000
|
||||
```
|
||||
|
||||
Follow `dev-runtime-deploy` from a clean latest `origin/dev` ref using login zsh. Run config check/dry-run, require the expected restart-required classification, rebuild/deploy/restart all applicable same-ref binaries, and re-prove source/build/runtime identities before smoke. Do not change route selection.
|
||||
|
||||
Run a direct-provider control and an Edge request with the same approximately 41k-token input shape and a caller boundary equivalent to the observed 180 seconds. For each provider selected by the current inventory, capture sanitized monotonic timings for response start, last body, logical finish, `[DONE]`/EOF/`END`/Core terminal when observable, timeout owner, recovery decision, and final client outcome. A post-commit stall must emit IOP's explicit sanitized terminal before 180 seconds; an eligible pre-commit stall must perform one exact replay. A normal success must preserve finish/usage/`[DONE]` order and one terminal. If the real provider stall is not reproduced, record `not_reproduced` with the successful control and timeout configuration; do not manufacture a failure or weaken TEST-1/2.
|
||||
|
||||
Write only a raw-free summary to the tracked evidence log. Keep the request, raw output, tokens, provider payloads, and any sensitive runtime details under an ignored run-artifact directory.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `docs/edge-local-dev-guide.md`: document response-stall ownership, safe timeout ordering, config validation/restart requirement, and raw-evidence handling.
|
||||
- [ ] `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log`: record sanitized source/build/config identities, provider/result matrix, monotonic timing summary, terminal counts, and reproduction status.
|
||||
- [ ] `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md`: record rollout commands/results and the exact tracked evidence path without secrets.
|
||||
|
||||
**Test Strategy:** Treat dev as external integration evidence after deterministic local qualification. Validate both configuration and runtime identity before requests, use the current provider mapping without substitution, and retain a successful normal-stream control beside any stall result.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-list --left-right --count HEAD...origin/dev && go version'\'''
|
||||
```
|
||||
|
||||
Then execute the exact config-check, build, deploy, restart, health, direct-provider, and Edge smoke commands required by `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`, recording sanitized output in the review file and evidence log.
|
||||
|
||||
Expected: the remote checkout and deployed artifacts share one clean release ref; both Ornith providers report 120000 ms response-stall ownership; health is restored; normal controls have one terminal; any stall is owned by IOP before the caller's 180-second boundary.
|
||||
|
||||
## Final Verification
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAITunnelCodecTerminalWire)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: all tests and smoke commands exit 0, race tests report no race, diff check is silent, and only the declared plan/review/implementation/evidence changes are present. TEST-3 additionally requires the same-ref remote identity, health, timeout-order, single-terminal, and sanitized-evidence criteria above.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `apps/node/internal/node/liveness_watchdog_test.go` | TEST-1 |
|
||||
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | TEST-1 |
|
||||
| `apps/edge/internal/openai/stream_gate_pipeline_test.go` | TEST-1 |
|
||||
| `scripts/e2e-openai-lemonade.sh` | TEST-2 |
|
||||
| `docs/edge-local-dev-guide.md` | TEST-3 |
|
||||
| `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log` | TEST-3 |
|
||||
| `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md` | TEST-1, TEST-2, TEST-3 |
|
||||
|
|
@ -0,0 +1,321 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=1 tag=REVIEW_TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Restore semantic-disabled stream terminals and finish timeout evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement the reviewer-selected R1 runtime fix and R2 rollout exactly as written. Do not investigate another cause, choose another sink/source design, narrow the Chat/Responses matrix, or change provider routing/capacity. Run every verification command, fill the implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with actual output, keep both active files in place, and report ready for review. If blocked, record only the exact blocker, attempted commands/output, and resume condition in implementation-owned evidence fields; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The first review reproduced the fake Lemonade logical-finish stall with one visible prefix and finish frame but no `run_error` or `[DONE]`. The request-local StreamGate and typed stall remain active when semantic filters are disabled, but tunnel endpoint framing and terminal serialization were incorrectly gated by the semantic switch. The original TEST-3 dev evidence was correctly stopped after that defect surfaced and must resume only after the selected runtime fix passes locally.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G09_0.log`.
|
||||
- Prior review: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G09_0.log`, verdict `FAIL`, `Required R1` semantic-disabled Chat/Responses tunnel terminal ownership and `Required R2` unfinished dev timeout-order evidence; no Suggested or Nit findings.
|
||||
- Fresh reviewer evidence: focused Node/Edge tests, full package tests, race tests, script syntax, and `git diff --check` passed; `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh` failed with `prefix=1 finish=1 run_error=0 done=0`.
|
||||
- Dev preflight: `/Users/toki/agent-work/iop-dev` was clean on `release/dev-971` at `003398a149c1433cb21bc8aa2720bde272b87373`, equal to `origin/dev`; Edge/Node binaries reported source `003398a149c1`, and ports `18082`, `18083`, `18084`, `19093`, `19101` were listening.
|
||||
- Roadmap carryover: this packet still contributes only `repeat-guard,ops-evidence`; it does not complete or update the Milestone directly.
|
||||
|
||||
## Finding Resolution Map
|
||||
|
||||
| Finding | Reviewer evidence | Exact root cause | Selected fix | Mode | Changed precondition | Acceptance commands |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Required R1 | Fake Lemonade smoke exits 1 with `prefix=1 finish=1 run_error=0 done=0`; semantic-disabled branches at `stream_gate_runtime.go:1241-1246,1836-1841,1867-1872` and `responses_stream_gate.go:1180-1185,1279-1286` bypass endpoint framing/terminal sinks. | Endpoint framing was coupled to semantic-filter activation. The Core commits a typed stall terminal, but an already-open raw sink lacks endpoint identity or a Responses renderer. | Always use endpoint event sources for supported tunnels; select the existing Responses pool sink/codec state for direct and provider-pool streaming Responses independent of semantic filters; add semantic-disabled Chat/Responses pre/post-commit and direct Responses regressions while preserving exact success wire. | direct-fix | The fake smoke and new regressions now exercise the fixed endpoint-aware product path instead of the unchanged broken branch. | `go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire)$'`; `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh`; full/race commands below. |
|
||||
| Required R2 | TEST-3 was not run and its guide/evidence files are absent; read-only dev preflight proves the declared runner and current services are reachable. | The prior plan's correct stop condition prohibited rollout after TEST-2 exposed R1. | Only after R1 local qualification passes, apply `response_stall_timeout_ms: 120000` to the two Ornith providers, execute the project dev-runtime same-ref validation/rebuild/deploy/restart path without route/capacity changes, run direct and Edge controls, update the guide, and write the raw-free evidence log. | direct-fix | R1 makes local TEST-2 pass, removing the stop condition before external rollout/evidence. | Remote preflight/config/deploy/health/identity commands from `dev-runtime-deploy`; direct/Edge Ornith controls; tracked evidence checks and final commands below. |
|
||||
|
||||
## Dependencies and Execution Order
|
||||
|
||||
1. Archived predecessor `03_repeat_guard_s07_remote_evidence` remains satisfied by the prior packet evidence; do not search sibling archives.
|
||||
2. Complete REVIEW_TEST-1, then pass every REVIEW_TEST-2 local/fake/full/race check before changing dev runtime state.
|
||||
3. Start REVIEW_TEST-3 only after the local stop condition is cleared. Follow `dev-runtime-deploy`; do not deploy an arbitrary dirty or non-`dev` source ref. If its clean-source/release preconditions are not satisfied, record the exact state and resume condition rather than weakening same-ref identity.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `apps/edge/internal/openai/stream_gate_runtime.go`
|
||||
- `apps/edge/internal/openai/responses_stream_gate.go`
|
||||
- `apps/edge/internal/openai/stream_gate_release_sink.go`
|
||||
- `apps/edge/internal/openai/stream_gate_tunnel_codec.go`
|
||||
- `apps/edge/internal/openai/provider_tunnel.go`
|
||||
- `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`
|
||||
- `apps/edge/internal/openai/stream_gate_pipeline_test.go`
|
||||
- `apps/node/internal/node/liveness_watchdog_test.go`
|
||||
- `scripts/e2e-openai-lemonade.sh`
|
||||
- `docs/edge-local-dev-guide.md`
|
||||
- `agent-spec/runtime/stream-evidence-gate.md`
|
||||
- `agent-spec/runtime/edge-node-execution.md`
|
||||
- `agent-spec/runtime/provider-pool-config-refresh.md`
|
||||
- `agent-spec/input/openai-compatible-surface.md`
|
||||
- `agent-contract/outer/openai-compatible-api.md`
|
||||
- `agent-contract/inner/execution-runtime.md`
|
||||
- `agent-contract/inner/edge-node-runtime-wire.md`
|
||||
- `agent-contract/inner/edge-config-runtime-refresh.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/openai-compatible-output-validation-filters.md`
|
||||
- `agent-test/local/rules.md`
|
||||
- `agent-test/local/edge-smoke.md`
|
||||
- `agent-test/local/node-smoke.md`
|
||||
- `agent-test/local/platform-common-smoke.md`
|
||||
- `agent-test/local/testing-smoke.md`
|
||||
- `agent-test/dev/rules.md`
|
||||
- `agent-test/dev/edge-smoke.md`
|
||||
- `agent-test/dev/node-smoke.md`
|
||||
- `agent-ops/rules/project/domain/edge/rules.md`
|
||||
- `agent-ops/rules/project/domain/node/rules.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`
|
||||
- `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G09_0.log`
|
||||
- `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G09_0.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`; status approved, lock released.
|
||||
- Milestone contribution: `repeat-guard,ops-evidence`.
|
||||
- Acceptance scenarios: S03/S04 require stream-open recovery/safe-stop to converge on one endpoint terminal; S07 requires deterministic generic smoke plus sanitized Ornith dev evidence and permits `not_reproduced` only beside deterministic coverage.
|
||||
- Evidence Map: S03/S04 drive the semantic-disabled pre/post-commit single-terminal regressions and retained `[DONE]`/side-effect boundaries. S07 drives the fake adapter smoke, two-provider dev timeout ordering, raw-free guide/evidence artifact, and live result matrix. These rows define REVIEW_TEST-1 through REVIEW_TEST-3 and the final verification.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Handoff source: the archived G09 review contains implementation output plus fresh reviewer reruns. Repository-native source/contract reads independently confirm the same root cause and selected fix.
|
||||
- Exact local failure: `IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh` exits 1 with `run_error=0 done=0`; focused/full/race tests otherwise pass.
|
||||
- Contract criteria: `agent-contract/outer/openai-compatible-api.md:120-126` and `agent-contract/inner/execution-runtime.md:52` require one always-owned runtime, semantic-disabled native compatibility, private typed-stall ownership, and one sanitized terminal.
|
||||
- Constraints: no synthesized Core terminal from `finish_reason`; no route/capacity change; no raw prompt/output/tool/auth in tracked artifacts; no common Agent-Ops edits; config timeout change remains restart-required.
|
||||
- Confidence: high. The deterministic smoke, source branches, codec binding, and sink predicates form one directly observed causal chain.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/workdir: `ssh toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, login zsh.
|
||||
- Observed state: clean `release/dev-971`, HEAD/origin-dev `003398a149c1433cb21bc8aa2720bde272b87373`, divergence `0 0`, Go `1.26.3 darwin/arm64`, git-flow `1.12.3`.
|
||||
- Artifacts: `build/dev-runtime/bin/edge` and `build/dev-runtime/bin/iop-node` report module source `003398a149c1`; config is `build/dev-runtime/edge.yaml`.
|
||||
- Runtime: ports `18082`, `18083`, `18084`, `19093`, and `19101` listen. Re-prove branch/ref/artifact/process identity immediately before deployment; this snapshot is evidence, not permission to reuse a stale candidate.
|
||||
- Setup/rebuild: follow `agent-ops/skills/project/dev-runtime-deploy/SKILL.md` from clean synchronized `dev`, reuse only the matching release state allowed by that skill, run sequential full tests, build all four binaries from one `DEPLOY_SHA`, require expected `restart_required`, restart Edge then Nodes, and verify connected provider snapshots before smoke.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- Current `TestOpenAIStallAfterLogicalFinishMatrix` sets semantic filtering true only; add exact false cases.
|
||||
- Current semantic-disabled compatibility tests prove successful byte order but not typed-stall pre/post-commit termination.
|
||||
- Streaming Responses semantic-disabled provider-pool and direct tunnel post-commit failure serialization need explicit regressions.
|
||||
- The fake Lemonade composition already supplies the exact Chat regression and must become green without weakening its assertions.
|
||||
- Dev timeout ordering, same-ref rollout identity, and raw-free Ornith outcome evidence remain absent.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No symbol rename/removal is selected.
|
||||
- Change sites: `newOpenAIChatAttemptEventSourceFactory`, `buildOpenAITunnelStreamGateRuntime`, `runOpenAITunnelStreamGate`, `buildOpenAIResponsesStreamGateRuntimeFromAttempt`, and `runOpenAIResponsesStreamGateAttempt`.
|
||||
- Existing callers remain unchanged: direct Chat/Responses tunnel paths in `provider_tunnel.go`, provider-pool Chat in `stream_gate_runtime.go`, and provider-pool Responses in `responses_stream_gate.go`.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
Keep one follow-up packet. Endpoint-source selection and endpoint-native terminal rendering form one correctness invariant across direct/pool and Chat/Responses variants; local tests and fake smoke must close before the dev timeout rollout can safely supply the required operational evidence. Splitting R2 now would recreate the stop-condition gap and cannot independently PASS.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- In scope: the five selected OpenAI runtime/test/script files, the two inherited completed regression files, the dev guide, exact evidence log, and active review evidence.
|
||||
- Out of scope: Node watchdog/adapter behavior changes, new liveness error types, Core filter/recovery algorithms, public/wire/config schema changes, provider routing/capacity, production deployment, caller timeout changes, and common Agent-Ops files.
|
||||
- `stream_gate_release_sink.go` and `stream_gate_tunnel_codec.go` are read-only: the existing Responses pool sink already preserves raw success wire, tracks its endpoint-native sequence state, and serializes the required error terminal.
|
||||
- Existing TEST-1 modifications in `liveness_watchdog_test.go` and `stream_gate_pipeline_test.go` are retained and reverified, not redesigned.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; all build/review scope, context, verification, evidence, ownership, and decision closures are true; no capability gap.
|
||||
- Finalizer: `finalize-task-policy.sh pair`.
|
||||
- Build: scores `2/2/2/2/2`, grade `G10`, route basis `grade-boundary`, lane `cloud`, catalog `worker/cloud/G10`, filename `PLAN-cloud-G10.md`.
|
||||
- Review: scores `2/2/2/2/2`, route `official-review/cloud/G10`, catalog `review/cloud/G10`, filename `CODE_REVIEW-cloud-G10.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks are `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, and `variant_product` (5).
|
||||
- Recovery signals: `review_rework_count=1`, `evidence_integrity_failure=false`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [x] [REVIEW_TEST-1] Restore endpoint-aware semantic-disabled Chat/Responses tunnel terminal behavior and add exact pre/post-commit regressions.
|
||||
- [x] [REVIEW_TEST-2] Rerun the bounded fake Lemonade adapter-to-Edge smoke plus focused, full, race, syntax, and diff qualification.
|
||||
- [x] [REVIEW_TEST-3] After local qualification, apply the two-provider dev timeout order, perform same-ref rollout/health checks, update the runbook, and capture sanitized Ornith evidence.
|
||||
- [x] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REVIEW_TEST-1] Endpoint-aware semantic-disabled terminal ownership
|
||||
|
||||
**Problem:** `apps/edge/internal/openai/stream_gate_runtime.go:1241-1246,1836-1841,1867-1872` and `apps/edge/internal/openai/responses_stream_gate.go:1180-1185` select an endpoint-unaware source when semantic filtering is disabled. Generic direct Responses and `apps/edge/internal/openai/responses_stream_gate.go:1279-1286` also select a raw tunnel sink instead of the existing Responses-native streaming sink. The private typed stall therefore reaches Core but can leave the client stream unterminated.
|
||||
|
||||
**Solution:** Decouple endpoint parsing/rendering from semantic-filter policy. Always bind the known supported endpoint codec, retain exact raw wire on success, and give both Chat and Responses a single sanitized post-commit terminal.
|
||||
|
||||
Before (`stream_gate_runtime.go:1836-1841`):
|
||||
|
||||
```go
|
||||
var src *openAITunnelEventSource
|
||||
if semanticEnabled {
|
||||
src = newOpenAITunnelEndpointEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler, req.endpoint, state)
|
||||
} else {
|
||||
src = newOpenAITunnelEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler, state)
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
src := newOpenAITunnelEndpointEventSource(
|
||||
transport.tunnel.Stream(), transport.tunnel.WaitTimeout(),
|
||||
rewriter, assembler, req.endpoint, state,
|
||||
)
|
||||
```
|
||||
|
||||
Apply the same unconditional endpoint-source construction at the Chat pool attempt factory, generic tunnel initial attempt, and Responses attempt factory. Do not remove `semanticEnabled`; keep it only where semantic registry, compatibility messages, and graceful-close behavior genuinely differ.
|
||||
|
||||
Before (`responses_stream_gate.go:1279-1286`):
|
||||
|
||||
```go
|
||||
if dc.responsesRequestContext.envelope.Stream {
|
||||
if s.streamGateSemanticEnabled() {
|
||||
sink = newOpenAIResponsesPoolReleaseSink(w, holder, selector)
|
||||
} else {
|
||||
flusher, _ := w.(http.Flusher)
|
||||
sink = newOpenAICompositeReleaseSink(selector, normalized, newOpenAITunnelReleaseSink(w, flusher))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```go
|
||||
if dc.responsesRequestContext.envelope.Stream {
|
||||
sink = newOpenAIResponsesPoolReleaseSink(w, holder, selector)
|
||||
}
|
||||
```
|
||||
|
||||
Before (`stream_gate_runtime.go:1949-1956`):
|
||||
|
||||
```go
|
||||
var sink *openAITunnelReleaseSink
|
||||
if req.stream {
|
||||
sink = newOpenAITunnelReleaseSink(w, flusher)
|
||||
} else {
|
||||
sink = newOpenAIBufferedTunnelReleaseSink(w, flusher, req.requestModel)
|
||||
}
|
||||
```
|
||||
|
||||
After: use `openAIStreamGateSink` and select `newOpenAIResponsesPoolReleaseSink` with a tunnel selector when `req.stream && req.endpoint == openAIRebuildEndpointResponses`; keep the existing tunnel sinks for Chat and non-stream requests. In `buildOpenAITunnelStreamGateRuntime`, resolve the Responses pool codec via `openAIResponsesTunnelCodecStateForSink`, set its usage holder once, and bind every initial/recovery attempt from the actual transport dispatch before events flow. This existing sink owns raw Responses sequence observation, successful raw terminal release, and one sanitized error plus `[DONE]`. Do not add a second error renderer, replay staged provider terminal wire on failure, expose raw failure data, or change pre-commit provider-error/status behavior.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [x] `apps/edge/internal/openai/stream_gate_runtime.go`: make known endpoint codec construction independent of semantic filters for Chat/generic tunnel initial and recovery attempts.
|
||||
- [x] `apps/edge/internal/openai/responses_stream_gate.go`: make Responses tunnel sources endpoint-aware and streaming pool sink endpoint-native regardless of semantic filters.
|
||||
- [x] `apps/edge/internal/openai/stream_gate_stall_recovery_test.go`: add `TestOpenAISemanticGateDisabledStallTerminalMatrix` covering Chat/Responses pool pre/post-commit ownership, no replay after commit, raw leakage absence, and exact terminal counts.
|
||||
- [x] `apps/edge/internal/openai/stream_gate_pipeline_test.go`: preserve inherited codec assertions and add the direct Responses runtime terminal case beside the existing generic tunnel lifecycle fixtures.
|
||||
|
||||
**Test Strategy:** Add deterministic scripted tunnel frames and typed `confirmedStallFailure`; no wall-clock sleep or network. Pre-commit cases assert one allowed replay and hidden rejected wire. Post-commit Chat asserts one `run_error` and `[DONE]`; Responses asserts one `type:error`, one `[DONE]`, no `response.completed`, no raw provider message/metadata, and one transport close. Retain `TestOpenAISemanticGateDisabledCompatibility` byte-for-byte Chat/Responses success assertions as the compatibility oracle.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire)$'
|
||||
```
|
||||
|
||||
Expected: exit 0; semantic true/false supported paths converge on one safe terminal, successful disabled-semantic tunnel bytes remain exact, and `finish_reason` alone never becomes a Core terminal.
|
||||
|
||||
### [REVIEW_TEST-2] Deterministic adapter composition and local qualification
|
||||
|
||||
**Problem:** `scripts/e2e-openai-lemonade.sh:399-405` correctly observes one prefix/finish but zero error/done on the current source. The existing TEST-1 regressions pass because their logical-finish matrix enables semantic filtering.
|
||||
|
||||
**Solution:** Keep the bounded fixture, 200 ms Node timeout, request-context cancellation, raw-leak guard, post-cancel health control, and exact count assertions unchanged in meaning. After REVIEW_TEST-1, rerun it through the actual fake provider → adapter → Node watchdog → Edge runtime. Only make script changes if required to keep the already-selected deterministic oracle stable; do not allowlist missing terminals or relax counts.
|
||||
|
||||
Before (`scripts/e2e-openai-lemonade.sh:399-405`):
|
||||
|
||||
```bash
|
||||
HANG_ERROR_COUNT=$(grep -c '"type":"run_error"' "$HANG_OUT" || true)
|
||||
HANG_DONE_COUNT=$(grep -c '^data: \[DONE\]$' "$HANG_OUT" || true)
|
||||
if [ "$HANG_PREFIX_COUNT" -ne 1 ] || [ "$HANG_FINISH_COUNT" -ne 1 ] || [ "$HANG_ERROR_COUNT" -ne 1 ] || [ "$HANG_DONE_COUNT" -ne 1 ]; then
|
||||
echo "[openai-lemonade] hang fixture terminal counts prefix=$HANG_PREFIX_COUNT finish=$HANG_FINISH_COUNT run_error=$HANG_ERROR_COUNT done=$HANG_DONE_COUNT"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
After: retain the same exact `1/1/1/1` contract and make no assertion downgrade. The runtime fix must make it pass.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [x] `scripts/e2e-openai-lemonade.sh`: retain the exact bounded semantic-disabled Chat oracle; adjust only deterministic fixture mechanics if required by the selected runtime fix.
|
||||
- [x] `apps/node/internal/node/liveness_watchdog_test.go`: preserve and rerun the completed finish-without-END fence regression.
|
||||
- [x] `apps/edge/internal/openai/stream_gate_pipeline_test.go`: preserve and rerun the completed logical-vs-transport terminal/byte-order regression alongside REVIEW_TEST-1's direct Responses case.
|
||||
|
||||
**Test Strategy:** The fake smoke is the product composition regression. Focused unit tests isolate endpoint/source/sink behavior; full and race suites catch shared state, codec queue, terminal fencing, and compatibility regressions. Fresh `-count=1` output is mandatory; cached results are not acceptable.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-lemonade.sh
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Expected: every command exits 0; the fake fixture reports no count mismatch, cancels and reaps cleanly, normal post-cancel traffic succeeds, package/race tests pass, and diff check is silent.
|
||||
|
||||
### [REVIEW_TEST-3] Dev Ornith timeout order and sanitized evidence
|
||||
|
||||
**Problem:** The declared dev config still needs an explicit 120-second Node response-stall boundary for both Ornith providers, and neither the guide nor the 2026-08-13 raw-free evidence artifact exists. The external caller boundary is approximately 180 seconds, so the default 300 seconds does not preserve IOP timeout ownership.
|
||||
|
||||
**Solution:** Only after REVIEW_TEST-2 passes, follow `dev-runtime-deploy` from clean synchronized `dev`. Change the ignored runtime config values only for `onexplayer-lemonade` and `rtx5090-lemonade`, preserve their current capacity/priority/routes, and ensure the Edge request hard timeout remains above 180 seconds. `response_stall_timeout_ms` is restart-required: run config check and dry-run, perform the full same-ref rebuild/restart and identity/health proof, then execute direct-provider and Edge approximately 41k-token controls for both providers. Record only sanitized timings/outcomes and `not_reproduced` when applicable.
|
||||
|
||||
Before (`docs/edge-local-dev-guide.md:70-110`): the provider-operation section documents model/provider baselines but not response-stall timeout ownership.
|
||||
|
||||
After: add one concise subsection specifying:
|
||||
|
||||
```text
|
||||
Node provider response_stall_timeout_ms = 120000
|
||||
external caller boundary ≈ 180000
|
||||
Edge request hard timeout > 180000
|
||||
```
|
||||
|
||||
Document that the config change requires restart, same-ref identity must be re-proved, and tracked evidence excludes prompt/output/token/credential material.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [x] `docs/edge-local-dev-guide.md`: document timeout ownership, restart validation, same-ref proof, and raw-evidence policy.
|
||||
- [x] `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log`: record sanitized source/build/config identities, two-provider direct/Edge result matrix, monotonic timing summary, terminal counts, timeout owner/recovery decision, and reproduction status.
|
||||
- [x] `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md`: record exact sanitized rollout and smoke stdout/stderr or ignored artifact paths.
|
||||
|
||||
**Test Strategy:** This is required dev integration evidence, not a substitute for deterministic tests. Apply the current inventory mapping (`onexplayer-lemonade` capacity 3, `rtx5090-lemonade` capacity 1) without substitution. Preserve raw request/response material only under ignored `agent-test/runs/**`; tracked output contains safe refs, timeout values, monotonic durations, counts, and outcome labels only.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev && git rev-list --left-right --count HEAD...origin/dev && go version && git flow version | head -n 1'\'''
|
||||
```
|
||||
|
||||
Then execute the exact inventory, clean-sync/release, sequential full-test, four-binary rebuild, `config check`, `config refresh --help`, expected `config refresh --mode dry-run` restart-required, Edge/Node deployment/restart, port/node/provider snapshot, direct-provider, Edge Chat/Responses, capacity, and release-finish commands required by `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`. Do not finish or push the release unless every skill-mandated test/capacity smoke succeeds. Record sanitized actual output in the review and evidence file.
|
||||
|
||||
Expected: one clean selected release source owns all deployed binaries; both Ornith provider entries report `120000`; Edge timeout is above 180 seconds; ports/nodes/provider snapshots recover; every normal control terminates once; a reproduced stall is owned by IOP before the caller boundary or is recorded as `not_reproduced`; tracked evidence contains no raw or secret material.
|
||||
|
||||
## Final Verification
|
||||
|
||||
```bash
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
go test -count=1 ./packages/go/execution ./packages/go/config ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
go test -race -count=1 ./apps/node/internal/node ./apps/edge/internal/openai
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: all local commands exit 0 with fresh output; only files listed below and the active task artifacts are changed. REVIEW_TEST-3 additionally satisfies the same-ref deployment, expected restart, health, timeout inequality, exact terminal count, and raw-free evidence criteria above.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `apps/edge/internal/openai/stream_gate_runtime.go` | REVIEW_TEST-1 |
|
||||
| `apps/edge/internal/openai/responses_stream_gate.go` | REVIEW_TEST-1 |
|
||||
| `apps/edge/internal/openai/stream_gate_stall_recovery_test.go` | REVIEW_TEST-1 |
|
||||
| `apps/node/internal/node/liveness_watchdog_test.go` | REVIEW_TEST-2 |
|
||||
| `apps/edge/internal/openai/stream_gate_pipeline_test.go` | REVIEW_TEST-1, REVIEW_TEST-2 |
|
||||
| `scripts/e2e-openai-lemonade.sh` | REVIEW_TEST-2 |
|
||||
| `docs/edge-local-dev-guide.md` | REVIEW_TEST-3 |
|
||||
| `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log` | REVIEW_TEST-3 |
|
||||
| `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md` | REVIEW_TEST-1, REVIEW_TEST-2, REVIEW_TEST-3 |
|
||||
|
|
@ -0,0 +1,241 @@
|
|||
<!-- task=m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness plan=2 tag=REVIEW_REVIEW_TEST milestone-task=repeat-guard,ops-evidence -->
|
||||
|
||||
# Route-aware managed capacity smoke and single-run evidence
|
||||
|
||||
## For the Implementing Agent
|
||||
|
||||
Implement the reviewer-selected R1/R2 fixes exactly as written. Keep product routing, provider capacities, timeout values, and OpenAI-compatible runtime behavior unchanged. Add the route-aware smoke and its deterministic self-test, update the project deployment/test contracts and guide, then run the authorized dev verification against the retained release candidate. Run every verification command, fill implementation-owned sections of `CODE_REVIEW-cloud-G10.md` with actual output, keep both active files in place, and report ready for review. If blocked, record only the exact blocker, attempted command/output, and resume condition in the review evidence; do not ask the user, call user-input tools, create control-plane stop files, classify the next state, archive logs, or write `complete.log`.
|
||||
|
||||
## Background
|
||||
|
||||
The second review proved the runtime terminal fix locally, but the dev capacity qualification measured an aggregate catalog capacity that the authenticated managed route could not reach and mixed prior response bodies into the current run summary. A route-qualified smoke must derive eligible capacity from the authenticated route selector, keep normal-capacity and long-context checks distinct, and bind every summarized artifact to one unique invocation. The retained release may finish only after both projected Ornith aliases pass Chat and Responses with exact terminal and recovery evidence.
|
||||
|
||||
## Archive Evidence Snapshot
|
||||
|
||||
- Prior plan: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G10_1.log`.
|
||||
- Prior review: `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G10_1.log`, verdict `FAIL`, with `Required R1` for route-ineligible aggregate capacity and `Required R2` for mixed-generation artifact provenance; no Suggested or Nit findings.
|
||||
- Fresh reviewer evidence: local Node/Edge focused, fake Lemonade, selected full-package, race, syntax, and diff checks passed. Read-only dev preflight showed `ornith:35b -> onexplayer-lemonade`, `ornith-fast -> rtx5090-lemonade`, normal capacities `3/1`, long-context capacities `2/1`, and a 30,000-token long-context threshold.
|
||||
- Failure evidence: the five-request `ornith:35b` run used a roughly 369,000-rune input, was classified long by the repository estimator, reached OneXPlayer `in_flight=2, queued=3`, and never exercised RTX5090. Eight response bodies summarized as current results were older 59-byte artifacts from a prior 401 attempt.
|
||||
- Roadmap carryover: this packet still contributes only `repeat-guard,ops-evidence`; PASS is contribution evidence and does not directly complete or update the Milestone.
|
||||
|
||||
## Finding Resolution Map
|
||||
|
||||
| Finding | Reviewer evidence | Root cause | Selected fix | Mode | Changed precondition | Acceptance commands |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Required R1 | The active managed routes resolve `ornith:35b` only to `onexplayer-lemonade` and `ornith-fast` only to `rtx5090-lemonade`; `principal_routes.go:291-305` excludes non-selected providers. The prior long request reached OneXPlayer's two long slots, while Chat policy raised the caller's 900-token cap to the catalog minimum. | Deployment/test guidance treated a catalog model group's physical providers as one caller-reachable pool without intersecting the authenticated route selector, and reused a long liveness request as a normal-capacity oracle. | Make the project deployment skill and dev Edge profile derive managed capacity from the authenticated route selector and selected provider snapshot. Add one smoke that runs Chat and Responses separately with a short 700-1200-token structured-answer request, bounded thinking, `eligible capacity + 1`, exact endpoint terminal checks, selected-provider peak/queue/final recovery, and no route/capacity mutation. Document the distinction and run `ornith:35b`/OneXPlayer and `ornith-fast`/RTX5090 separately. | direct-fix | Each live case is now bound to one authenticated route, one selected provider, the actual normal context class, and only that provider's eligible capacity. | `bash -n scripts/e2e-openai-managed-capacity-smoke.sh`; `./scripts/e2e-openai-managed-capacity-smoke.sh --self-test`; `python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy`; both live route matrices return HTTP 200, exact success terminals, peak equal to selected eligible capacity, queue at least one, and final selected counters `0/0`. |
|
||||
| Required R2 | The fixed raw directory retained older 59-byte SSE bodies while newer curl stderr represented the current timeouts; the summary paired them. The handwritten `approx_input_tokens=41000` label contradicted the actual emitted payload and repository estimator. | The task-local wrapper reused one directory, summarized paths outside a current-run manifest, and used a handwritten request-size label. | Give every invocation a mode-0700 unique run directory and run id. Create fresh per-case paths only inside it, bind request/result mtimes and the summary row to the run manifest, derive rune count, `runes/4 + runes/16` estimate, and context class from the emitted request, and fail closed on missing or mismatched current-run bodies. Generate the tracked evidence only from the new sanitized summary. | direct-fix | A stale body, wrong run id, non-current mtime, missing result, handwritten estimate, or unsafe summary field now fails before evidence is accepted. | The self-test rejects a stale-body fixture and proves two invocations use distinct 0700 directories; live summaries carry one run id with computed request shape and current result provenance; `git diff --check` passes. |
|
||||
|
||||
## Analysis
|
||||
|
||||
### Files Read
|
||||
|
||||
- `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/plan_cloud_G10_1.log`
|
||||
- `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/code_review_cloud_G10_1.log`
|
||||
- `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`
|
||||
- `agent-test/dev/rules.md`
|
||||
- `agent-test/dev/edge-smoke.md`
|
||||
- `agent-test/dev/testing-smoke.md`
|
||||
- `agent-ops/rules/project/domain/testing/rules.md`
|
||||
- `scripts/e2e-provider-capacity-smoke.sh`
|
||||
- `scripts/e2e-long-context-admission-smoke.sh`
|
||||
- `apps/control-plane/cmd/control-plane/credential_http_handlers.go`
|
||||
- `apps/control-plane/internal/credentialops/service.go`
|
||||
- `apps/edge/internal/openai/principal_routes.go`
|
||||
- `apps/edge/internal/openai/chat_policy.go`
|
||||
- `apps/edge/internal/openai/input_estimator.go`
|
||||
- `apps/edge/internal/openai/responses_types.go`
|
||||
- `apps/edge/internal/openai/chat_handler.go`
|
||||
- `apps/edge/internal/openai/responses_handler.go`
|
||||
- `docs/edge-local-dev-guide.md`
|
||||
- `agent-roadmap/phase/knowledge-tool-optimization-extension/milestones/openai-compatible-output-validation-filters.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`
|
||||
- `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log`
|
||||
|
||||
### SDD Criteria
|
||||
|
||||
- SDD: `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/SDD.md`, status `[승인됨]`, lock `해제`.
|
||||
- Milestone contribution ids remain exactly `repeat-guard,ops-evidence`.
|
||||
- Acceptance Scenario S07 and its Evidence Map row require caller-neutral dev `ornith:35b` capacity+1 streaming evidence, a repeat fingerprint/decision or `not_reproduced`, raw request/output/auth exclusion, and a generic deterministic fixture that the live result cannot replace.
|
||||
- This follow-up does not change S07's product behavior. It repairs the live oracle so route eligibility, normal versus long admission, exact terminal counts, and raw-free one-run provenance are trustworthy before the retained release and task evidence can close.
|
||||
|
||||
### Verification Context
|
||||
|
||||
- Handoff source: the archived G10 review contains fresh local verification plus a read-only dev route/config/artifact preflight. Repository source confirms the route predicate, output-token floor, input estimator, and credential route list shape used by the selected fix.
|
||||
- Existing local behavior: focused terminal regressions, the fake Lemonade composition, selected full package tests, and race tests pass at source `2fcc1093c7629ab94b460d083520ffd9c8064815`.
|
||||
- Current dev candidate: `/Users/toki/agent-work/iop-dev`, clean `release/dev-974`, `HEAD=origin/dev=origin/release/dev-974=2fcc1093c7629ab94b460d083520ffd9c8064815`, no `dev-974` tag, same-ref Edge/Node artifacts, ports `18082,18083,18084,19093,19101`, four connected Nodes, and healthy Ornith providers recovered to `0/0`.
|
||||
- Route source: authenticated `GET /v1/credentials/routes` returns secret-blind route DTOs including alias and `resource_selector`; the smoke must use the same protected principal token as OpenAI ingress, never print it, and require one active exact alias whose selector equals the requested provider. If the credential HTTPS route is not directly reachable from the smoke host, an authorized protected wrapper may capture that same authenticated JSON into the unique run directory before the smoke, but an unscoped catalog/config guess is invalid.
|
||||
- Request shape: build the actual Chat and Responses JSON in the unique run directory, count its Unicode runes, compute `runes/4 + runes/16`, and classify it against the runtime threshold. The selected capacity comes from `capacity` only when this computed class is `normal`; a long request must use `long_context_capacity` and must never satisfy this normal-capacity packet.
|
||||
- Constraints: Bash must remain compatible with the remote macOS Bash 3.2 baseline; token/header values, prompt text, response bodies, route slot identifiers, and credential material stay out of stdout, tracked evidence, and task files. Raw request/response files remain only under ignored `agent-test/runs/**`.
|
||||
- Confidence: high. The previous failure is explained by exact route, admission, estimator, policy, and file-mtime evidence, and the selected self-test can reproduce both oracle defects without network access.
|
||||
|
||||
#### External Verification Preflight
|
||||
|
||||
- Runner/workdir: `ssh toki@toki-labs.com`, `/Users/toki/agent-work/iop-dev`, login zsh for the declared toolchain; the new script may be copied to a mode-0700 ignored runner path with its SHA-256 recorded because the runtime candidate itself remains `dev-974`.
|
||||
- Re-prove before live traffic: branch/HEAD/origin divergence/dirty state; retained release branch/tag state; `go version`; `git flow version`; script SHA-256; Edge/Node build revisions and hashes; ports; four connected Nodes; route list response; model discovery; provider capacities/long capacities/health; `long_context_threshold_tokens`; and final initial `in_flight=0, queued=0`.
|
||||
- Authorized secret route: use the already approved protected active-token wrapper or stdin handoff. The token may be exported only in the child process environment and must be unset after the command. Do not print commands with expanded headers.
|
||||
- Live cases: run Chat then Responses for `ornith:35b` with expected selector `onexplayer-lemonade` and separately for `ornith-fast` with `rtx5090-lemonade`. Every case must be normal context, send selected capacity+1 concurrent requests, poll status at 50-100ms, validate every HTTP/SSE result, and wait for selected-provider recovery.
|
||||
- Release action: if all four endpoint/route cases pass and the retained release/source/build/origin preconditions remain unchanged, resume the existing `dev-runtime-deploy` finish/tag/atomic-push step. If any case or precondition fails, retain the release branch and do not finish or push.
|
||||
|
||||
### Test Coverage Gaps
|
||||
|
||||
- No current script validates managed route selector eligibility before calculating capacity.
|
||||
- No deterministic self-test rejects catalog aggregate capacity for a selector-pinned route.
|
||||
- No current live smoke enforces per-invocation unique evidence directories and current-run request/result provenance.
|
||||
- No current self-test rejects a stale response body with a current curl result.
|
||||
- No accepted dev matrix separately proves Chat and Responses terminal completion for both current Ornith projected aliases.
|
||||
|
||||
### Symbol References
|
||||
|
||||
- No production symbol is renamed or removed.
|
||||
- New script functions should own route resolution, provider snapshot normalization, request-shape calculation, unique-run allocation, endpoint request creation, SSE terminal parsing, peak polling, final recovery, sanitized summary emission, and self-test fixtures. Do not add a product API or import the smoke into runtime code.
|
||||
|
||||
### Split Judgment
|
||||
|
||||
Keep one follow-up packet. The project skill/profile corrections are unsafe without the executable route/provenance oracle, and the tracked evidence/release decision is invalid until that same oracle passes live. Splitting documentation, script, and rollout would permit an intermediate state that repeats either R1 or R2 and cannot independently PASS.
|
||||
|
||||
### Scope Rationale
|
||||
|
||||
- In scope: `dev-runtime-deploy`, dev Edge smoke criteria, one new managed-capacity smoke and self-test, the local guide, the exact existing sanitized evidence log, live validation of the retained release, and active review evidence.
|
||||
- Out of scope: `apps/**` and `packages/**` production behavior, public/inner contract schema, provider catalog or route mutation, capacity/timeout changes, new credentials, Control Plane database mutation, new Agent-Ops common rules/skills, long-context repeat reproduction redesign, and production deployment.
|
||||
- Existing `scripts/e2e-provider-capacity-smoke.sh` and `scripts/e2e-long-context-admission-smoke.sh` are read-only pattern sources. Do not retrofit the local shared-provider fixture or the separate long-context scenario into this managed-route smoke.
|
||||
|
||||
### Final Routing
|
||||
|
||||
- `evaluation_mode=isolated-reassessment`; build/review scope, context, verification, evidence, ownership, and decision closures are true; no capability gap.
|
||||
- Finalizer: `finalize-task-policy.sh pair`.
|
||||
- Build: scores `2/2/2/2/2`, grade `G10`, base/final route basis `grade-boundary`, lane `cloud`, catalog `worker/cloud/G10`, filename `PLAN-cloud-G10.md`.
|
||||
- Review: scores `2/2/2/2/2`, route `official-review/cloud/G10`, catalog `review/cloud/G10`, filename `CODE_REVIEW-cloud-G10.md`.
|
||||
- `large_indivisible_context=false`; positive loop risks are `temporal_state`, `concurrent_consistency`, `boundary_contract`, `structured_interpretation`, and `variant_product` (5).
|
||||
- Recovery signals: `review_rework_count=2`, `evidence_integrity_failure=true`; both boundaries match, while the already-cloud grade basis remains `grade-boundary`.
|
||||
|
||||
## Implementation Checklist
|
||||
|
||||
- [ ] [REVIEW_REVIEW_TEST-1] Correct managed-capacity ownership in the project deployment skill, dev Edge smoke profile, and local guide.
|
||||
- [ ] [REVIEW_REVIEW_TEST-2] Add the route-aware Chat/Responses capacity smoke with deterministic route, terminal, provenance, and stale-artifact self-tests.
|
||||
- [ ] [REVIEW_REVIEW_TEST-3] Run both projected Ornith route matrices, replace disputed evidence from the sanitized current-run summaries, and finish the retained release only if every gate passes.
|
||||
- [ ] Fill implementation-owned sections in CODE_REVIEW-*-G??.md with actual implementation notes and verification output.
|
||||
|
||||
### [REVIEW_REVIEW_TEST-1] Managed route-qualified capacity contract
|
||||
|
||||
**Problem:** `agent-ops/skills/project/dev-runtime-deploy/SKILL.md:143-150` and `agent-test/dev/edge-smoke.md:151-156` calculate capacity from all providers in a model catalog group. In managed mode, `apps/edge/internal/openai/principal_routes.go:291-305` constrains dispatch to the authenticated route selector, so the aggregate can include unreachable resources.
|
||||
|
||||
**Solution:** Replace model-group aggregate guidance with one explicit rule: resolve the authenticated route alias first, intersect its `resource_selector`, profile, and upstream model with current provider snapshots, classify the emitted request as normal or long, and use only the selected eligible provider's matching capacity. Require `scripts/e2e-openai-managed-capacity-smoke.sh` for managed Chat/Responses deployment qualification. Keep unprojected/legacy pools on their existing pool-capacity rule. Update the guide to distinguish short normal-capacity verification from the independent long-context/repeat smoke.
|
||||
|
||||
Before (`agent-ops/skills/project/dev-runtime-deploy/SKILL.md:143-150`):
|
||||
|
||||
```markdown
|
||||
- endpoint별로 선택한 model group의 총 provider capacity + 1개 요청을 동시에 보낸다.
|
||||
- 요청 실행 중 ... 대상 provider들의 in_flight 합이 총 capacity에 도달하고 queued 합이 1 이상 ...
|
||||
```
|
||||
|
||||
After:
|
||||
|
||||
```markdown
|
||||
- Managed mode resolves the authenticated route alias and resource selector before capacity calculation.
|
||||
- Run selected-provider eligible capacity + 1 for one endpoint and route at a time; never add a provider excluded by that route.
|
||||
- Compute request context class from the emitted request and use capacity or long_context_capacity accordingly.
|
||||
```
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-ops/skills/project/dev-runtime-deploy/SKILL.md`: define managed route projection, selected-provider capacity, script invocation, and fail-closed release gate.
|
||||
- [ ] `agent-test/dev/edge-smoke.md`: replace the Ornith aggregate baseline with exact projected alias/provider cases and separate normal/long acceptance.
|
||||
- [ ] `docs/edge-local-dev-guide.md`: document route-aware capacity ownership, unique-run evidence, terminal criteria, and raw-material boundary.
|
||||
|
||||
**Test Strategy:** Validate the project skill structure and use the new script self-test as the executable oracle for the documented mapping. Do not add product Go tests because no runtime behavior changes.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
rg --sort path -n 'managed|resource_selector|eligible capacity|e2e-openai-managed-capacity-smoke' agent-ops/skills/project/dev-runtime-deploy/SKILL.md agent-test/dev/edge-smoke.md docs/edge-local-dev-guide.md
|
||||
```
|
||||
|
||||
Expected: validation exits 0; the three documents consistently require route-qualified capacity and do not claim `ornith:35b` can aggregate OneXPlayer and RTX5090 under the current projected selectors.
|
||||
|
||||
### [REVIEW_REVIEW_TEST-2] Deterministic managed-capacity and provenance oracle
|
||||
|
||||
**Problem:** No tracked smoke owns the selected route/provider eligibility or exact Chat/Responses terminal checks. The prior task-local wrapper reused one output directory, so current curl metadata could be paired with a stale response body.
|
||||
|
||||
**Solution:** Add `scripts/e2e-openai-managed-capacity-smoke.sh` with live and `--self-test` modes. Live mode must:
|
||||
|
||||
1. create a `mktemp -d` child under ignored `agent-test/runs/**`, set and verify mode `0700`, assign one immutable run id, and refuse caller-selected existing run directories;
|
||||
2. obtain the secret-blind authenticated route list with the protected principal token, require one exact alias/selector match, and join it to one healthy current provider snapshot;
|
||||
3. build the actual short Chat/Responses request bodies inside the run directory, compute Unicode runes, `runes/4 + runes/16`, and context class, require `normal`, and choose the selected provider's normal `capacity`;
|
||||
4. launch `capacity + 1` streaming requests for one endpoint/route, poll status at 50-100ms, and record selected-provider peak/queue plus final recovery without adding non-selected capacity;
|
||||
5. bind every case to a current-run manifest entry and fresh request/result/curl-status paths, reject missing files or mtimes before request creation, and parse only manifest-owned results;
|
||||
6. require every Chat result to have HTTP 200, one finish terminal, one `[DONE]`, and no error; require every Responses result to have HTTP 200, one `response.completed`, one `[DONE]`, and no error;
|
||||
7. write a sanitized summary with run id, script hash, route alias, selected provider, endpoint, request count, computed request shape/class, durations, terminal counts, peak/queue/final counters, and outcome only. Never copy raw request/response, token/header, slot/route id, prompt, or output into the summary.
|
||||
|
||||
`--self-test` must use local fixtures only. It must prove route selector exclusion, normal/long capacity choice, Chat and Responses exact terminals, duplicate/missing terminal rejection, two distinct 0700 run directories, current-run manifest/mtime acceptance, and stale/missing/foreign-run body rejection. Keep macOS Bash 3.2 compatibility and use Python 3 only for deterministic JSON/rune/provenance parsing where shell parsing would be unsafe.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `scripts/e2e-openai-managed-capacity-smoke.sh`: implement the live driver, safe summary, and complete deterministic self-test.
|
||||
|
||||
**Test Strategy:** `bash -n` checks the shell surface. `--self-test` is the required regression and must execute all positive and negative fixtures without network, credentials, or repository writes outside a temporary directory.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
```
|
||||
|
||||
Expected: both commands exit 0; self-test output reports every route/terminal/provenance negative rejected, unique mode-0700 directories, and no surviving repository artifact.
|
||||
|
||||
### [REVIEW_REVIEW_TEST-3] Route-realistic dev evidence and retained release gate
|
||||
|
||||
**Problem:** `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log:29-69` contains a handwritten 41k-token label, aggregate-capacity interpretation, terminal-zero results, and a fixed raw directory that the reviewer proved contained mixed generations. Those rows cannot qualify S07 or the retained release.
|
||||
|
||||
**Solution:** After local gates pass, re-prove the retained release and runtime identities. Execute the new smoke with the authorized protected token for these exact matrices, sequentially:
|
||||
|
||||
- `ornith:35b`, selector/provider `onexplayer-lemonade`, Chat then Responses, request count `3 + 1`;
|
||||
- `ornith-fast`, selector/provider `rtx5090-lemonade`, Chat then Responses, request count `1 + 1`.
|
||||
|
||||
Use the script's short normal request and current-run summaries; do not reuse the 369k-rune liveness payload. Require every request to complete before the bounded caller deadline with exact endpoint terminals, selected peak equal to capacity, queue at least one, and selected final counters `0/0`. Replace the disputed request-shape/capacity/artifact rows in the tracked evidence with a v2 sanitized summary derived only from these successful run summaries. Preserve previously proven source/build/timeout/direct-control facts only when their meaning remains accurate, and keep `stall_reproduction_status=not_reproduced` distinct from the normal-capacity result. If all gates and retained-release ref checks pass, execute the already-defined release finish/tag/atomic push; otherwise record the exact blocker and leave the release unfinished.
|
||||
|
||||
**Modified Files and Checklist:**
|
||||
|
||||
- [ ] `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log`: replace disputed rows with one-run, route-qualified, computed-shape, terminal-complete v2 evidence.
|
||||
- [ ] `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md`: record actual local/live/release output or exact ignored sanitized artifact paths.
|
||||
|
||||
**Test Strategy:** Treat the four live cases as mandatory integration evidence and keep the deterministic self-test as the provenance oracle. Do not declare `not_reproduced`, counter recovery alone, or release retention as a PASS substitute.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
ssh toki@toki-labs.com '/bin/zsh -lc '\''cd /Users/toki/agent-work/iop-dev && git status --short --branch && git rev-parse HEAD && git rev-parse origin/dev && git rev-parse origin/release/dev-974 && git rev-list --left-right --count HEAD...origin/dev && git tag -l dev-974 && go version && git flow version | head -n 1'\'''
|
||||
```
|
||||
|
||||
Then run the new smoke through the authorized protected-token wrapper for the exact `ornith:35b/onexplayer-lemonade` and `ornith-fast/rtx5090-lemonade` Chat/Responses matrices. Save raw bodies only in the script-created ignored run directories and save sanitized summaries in the implementation-owned review evidence. Execute the retained release finish/tag/atomic-push sequence only after all four matrices and final preconditions pass.
|
||||
|
||||
Expected: source/build/runtime identity remains one retained candidate; all four route/endpoint cases pass with exact terminals, selected-provider peak/queue/final recovery and no aggregate misattribution; tracked evidence is raw-free and current-run bound; release finish is either fully atomic and verified or not run with one exact blocker.
|
||||
|
||||
## Final Verification
|
||||
|
||||
```bash
|
||||
bash -n scripts/e2e-openai-managed-capacity-smoke.sh
|
||||
./scripts/e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
python3 /config/.codex/skills/.system/skill-creator/scripts/quick_validate.py agent-ops/skills/project/dev-runtime-deploy
|
||||
go test -count=1 ./apps/node/internal/node -run '^TestTunnelWatchdogFinishFrameWithoutEndStallsOnce$'
|
||||
go test -count=1 ./apps/edge/internal/openai -run '^(TestOpenAIStallAfterLogicalFinishMatrix|TestOpenAISemanticGateDisabledStallTerminalMatrix|TestOpenAISemanticGateDisabledCompatibility|TestOpenAITunnelCodecTerminalWire|TestOpenAIDirectResponsesSemanticDisabledStallTerminal)$'
|
||||
IOP_LEMONADE_MODE=fake ./scripts/e2e-openai-lemonade.sh
|
||||
git diff --check
|
||||
git status --short --branch
|
||||
```
|
||||
|
||||
Expected: every local command exits 0 with fresh output; only the files below and active task artifacts change. External verification additionally passes the four exact route/endpoint cases, tracked evidence provenance/redaction checks, retained-release ref gate, and atomic finish or preserves one exact blocker without partial refs.
|
||||
|
||||
After completing all code changes, fill implementation-owned sections in `CODE_REVIEW-*-G??.md`.
|
||||
|
||||
## Modified Files Summary
|
||||
|
||||
| File | Items |
|
||||
|---|---|
|
||||
| `agent-ops/skills/project/dev-runtime-deploy/SKILL.md` | REVIEW_REVIEW_TEST-1 |
|
||||
| `agent-test/dev/edge-smoke.md` | REVIEW_REVIEW_TEST-1 |
|
||||
| `docs/edge-local-dev-guide.md` | REVIEW_REVIEW_TEST-1 |
|
||||
| `scripts/e2e-openai-managed-capacity-smoke.sh` | REVIEW_REVIEW_TEST-2 |
|
||||
| `agent-roadmap/sdd/knowledge-tool-optimization-extension/openai-compatible-output-validation-filters/evidence/2026-08-13-ornith-session-stall-timeout-order.log` | REVIEW_REVIEW_TEST-3 |
|
||||
| `agent-task/m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md` | REVIEW_REVIEW_TEST-1, REVIEW_REVIEW_TEST-2, REVIEW_REVIEW_TEST-3 |
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
# Milestone Work Log
|
||||
|
||||
> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.
|
||||
|
||||
| seq | time | event | task | loop | role | attempt | model | result | locator |
|
||||
|---:|---|---|---|---:|---|---:|---|---|---|
|
||||
| 1 | 26-08-13 09:51:41 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T095141+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a00/locator.json |
|
||||
| 2 | 26-08-13 10:16:28 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 0 | codex/gpt-5.6-sol xhigh | failed:generic-error:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T095141+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a00/locator.json |
|
||||
| 3 | 26-08-13 10:16:31 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 1 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T101630+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a01/locator.json |
|
||||
| 4 | 26-08-13 10:25:11 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 1 | codex/gpt-5.6-sol xhigh | failed:generic-error:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T101630+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a01/locator.json |
|
||||
| 5 | 26-08-13 10:25:15 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 2 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T102515+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a02/locator.json |
|
||||
| 6 | 26-08-13 10:31:25 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G09.md | 0 | worker | 2 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T102515+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__worker__a02/locator.json |
|
||||
| 7 | 26-08-13 10:31:25 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T103125+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a00/locator.json |
|
||||
| 8 | 26-08-13 11:17:48 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G09.md | 0 | review | 0 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T103125+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a00/locator.json |
|
||||
| 9 | 26-08-13 11:18:12 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T111812+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a00/locator.json |
|
||||
| 10 | 26-08-13 11:52:01 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 0 | codex/gpt-5.6-sol xhigh | failed:cancelled | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T111812+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a00/locator.json |
|
||||
| 11 | 26-08-13 11:52:30 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 1 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T115230+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a01/locator.json |
|
||||
| 12 | 26-08-13 11:56:51 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 1 | codex/gpt-5.6-sol xhigh | failed:cancelled | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T115230+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a01/locator.json |
|
||||
| 13 | 26-08-13 11:57:14 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 2 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T115714+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a02/locator.json |
|
||||
| 14 | 26-08-13 12:17:18 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 3 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T121718+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a03/locator.json |
|
||||
| 15 | 26-08-13 13:04:05 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 3 | codex/gpt-5.6-sol xhigh | failed:generic-error:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T121718+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a03/locator.json |
|
||||
| 16 | 26-08-13 13:04:07 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 4 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T130407+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a04/locator.json |
|
||||
| 17 | 26-08-13 13:09:32 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 4 | codex/gpt-5.6-sol xhigh | failed:generic-error:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T130407+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a04/locator.json |
|
||||
| 18 | 26-08-13 13:09:36 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 5 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T130936+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a05/locator.json |
|
||||
| 19 | 26-08-13 13:24:50 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 5 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T130936+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a05/locator.json |
|
||||
| 20 | 26-08-13 13:24:50 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 1 | review | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T132450+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__review__a00/locator.json |
|
||||
| 21 | 26-08-13 14:01:04 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 1 | review | 0 | codex/gpt-5.6-sol xhigh | failed:session-stall:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T132450+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__review__a00/locator.json |
|
||||
| 22 | 26-08-13 14:01:06 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 0 | review | 1 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T140106+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a01/locator.json |
|
||||
| 23 | 26-08-13 14:17:35 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 0 | review | 1 | codex/gpt-5.6-sol xhigh | failed:session-stall:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T140106+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a01/locator.json |
|
||||
| 24 | 26-08-13 14:17:39 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 0 | review | 2 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T141739+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a02/locator.json |
|
||||
| 25 | 26-08-13 14:31:33 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 0 | review | 2 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T141739+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p0__review__a02/locator.json |
|
||||
| 26 | 26-08-13 14:32:00 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T143200+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a00/locator.json |
|
||||
| 27 | 26-08-13 14:44:54 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 0 | codex/gpt-5.6-sol xhigh | failed:session-stall:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T143200+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a00/locator.json |
|
||||
| 28 | 26-08-13 14:44:56 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 1 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T144456+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a01/locator.json |
|
||||
| 29 | 26-08-13 15:10:07 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 1 | codex/gpt-5.6-sol xhigh | failed:generic-error:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T144456+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a01/locator.json |
|
||||
| 30 | 26-08-13 15:10:11 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 2 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T151011+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a02/locator.json |
|
||||
| 31 | 26-08-13 15:20:32 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 2 | worker | 2 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T151011+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__worker__a02/locator.json |
|
||||
| 32 | 26-08-13 15:20:33 KST | START | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 2 | review | 0 | codex/gpt-5.6-sol xhigh | running | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T152033+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__review__a00/locator.json |
|
||||
| 33 | 26-08-13 15:37:51 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/CODE_REVIEW-cloud-G10.md | 2 | review | 0 | codex/gpt-5.6-sol xhigh | succeeded:0 | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T152033+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p2__review__a00/locator.json |
|
||||
| 34 | 26-08-13 15:37:51 KST | FINISH | m-openai-compatible-output-validation-filters/04+03_stream_terminal_liveness/PLAN-cloud-G10.md | 1 | worker | 2 | codex/gpt-5.6-sol xhigh | reconciled:verified-complete-archive | /config/workspace/iop-s1/.git/agent-task-dispatcher/runs/20260813T115714+0900__m-openai-compatible-output-validation-filters__04__03_stream_terminal_liveness__p1__worker__a02/locator.json |
|
||||
|
|
@ -3,7 +3,7 @@ test_env: dev
|
|||
test_profile: edge-smoke
|
||||
domain: edge
|
||||
verification_type: smoke
|
||||
last_rule_updated_at: 2026-08-06
|
||||
last_rule_updated_at: 2026-08-13
|
||||
---
|
||||
|
||||
# edge-smoke dev 테스트
|
||||
|
|
@ -150,10 +150,12 @@ GX10 Laguna 공식 vLLM baseline은 `temperature=0.7`, `top_p=0.95`, model `gene
|
|||
- raw text tool-call boundary smoke: Pi/Cline형 `tools[]` 요청에서 응답 body와 SSE delta 어디에도 `<tool_call>`, `{{`, `<|mask_end|>` 원문이 성공 content로 남지 않는다. 요청 `tools[]`에 있는 valid raw text tool-call은 `message.tool_calls`(또는 stream `delta.tool_calls`)와 `finish_reason: "tool_calls"`로 정규화되고, unknown tool hallucination이나 malformed 블록은 성공 content가 아니라 non-stream `tool_validation_error`(HTTP 502) 또는 SSE `tool_validation_error` 이벤트로 끝난다. non-stream/strict buffered stream은 bounded retry 후 차단을 확인한다. 세부 payload와 계약은 `docs/edge-local-dev-guide.md`의 Raw text tool-call boundary smoke와 `agent-contract/outer/openai-compatible-api.md`를 따른다.
|
||||
- raw boundary smoke evidence는 tracked 문서가 아니라 ignored run 위치(`agent-test/runs/**`)나 code-review output path에 저장하고, 저장물에도 token 원문을 남기지 않는다.
|
||||
- provider-pool dispatch는 `in_flight >= capacity`인 provider를 후보에서 제외하고, 남은 후보 중 가장 낮은 `in_flight` 레벨을 먼저 선택한다. 같은 `in_flight` 레벨 안에서는 낮은 priority 값과 rotation으로 분산한다.
|
||||
- dev-runtime capacity smoke는 model group별로 분리한다. Laguna `laguna-s:2.1`은 `gx10-vllm=4`이므로 5개 동시 요청에서 `in_flight=4`, `queued>=1`; Ornith `ornith:35b`는 `onexplayer-lemonade=3` + `rtx5090-lemonade=1`이므로 5개 요청에서 `in_flight=4`, `queued>=1`; Qwen `qwen3.6:35b`는 `mac-mlx-vllm=2`이므로 3개 요청에서 `in_flight=2`, `queued>=1`을 기준으로 한다. `ornith:35b`와 `ornith-fast`는 provider-owned shared capacity 구현 전까지 model group별 capacity를 독립 집계하므로 같은 smoke에서 두 alias를 섞지 않는다.
|
||||
- dev-runtime managed capacity smoke는 `scripts/e2e-openai-managed-capacity-smoke.sh`를 사용한다. 같은 principal token으로 authenticated route alias를 먼저 조회하고, exact active route의 `resource_selector`, profile, upstream model과 일치하는 healthy provider snapshot 하나만 eligible pool로 본다. `ornith:35b`는 `onexplayer-lemonade=3`, `ornith-fast`는 `rtx5090-lemonade=1`로 각각 분리하며, 현재 projected route에서 제외된 두 provider의 capacity를 더하지 않는다. Chat/Responses를 alias별로 따로 실행해 각각 4개/2개 요청에서 selected `in_flight=3`/`1`, `queued>=1`을 확인한다.
|
||||
- managed smoke는 실제 emitted request JSON의 Unicode rune 수와 `runes/4 + runes/16` estimate를 계산하고 `long_context_threshold_tokens`와 대조한다. normal qualification은 `capacity`, 별도 long-context 시나리오는 `long_context_capacity`를 사용한다. 장문 반복/liveness payload로 normal capacity를 증명하거나 normal 요청으로 long slot을 증명하지 않는다. unprojected/legacy Laguna `laguna-s:2.1`과 Qwen `qwen3.6:35b`는 기존 model-group capacity `4`/`2` 기준을 유지한다.
|
||||
- capacity smoke 완료 후 대상 provider의 `in_flight=0`, `queued=0` 회복을 확인한다.
|
||||
- long-context admission 시나리오(normal 10-way, mixed long/normal, all-long-slot-full)와 최종 회복 근거는 `agent-test/dev/long-context-admission-smoke.md`와 `scripts/e2e-long-context-admission-smoke.sh`를 사용한다.
|
||||
- `/v1/responses`와 `/v1/chat/completions`는 각각 700~1200 token 수준의 구조화된 응답을 유도하고 50~100ms 간격으로 status를 polling한다. 각 요청의 HTTP 200, target provider별 capacity 미초과, 합산 peak와 queue, 최종 0 회복을 별도 증거로 남긴다.
|
||||
- `/v1/responses`와 `/v1/chat/completions`는 각각 짧은 입력에서 700~1200 token 수준의 구조화된 응답을 유도하고 provider-native thinking을 제한하며 50~100ms 간격으로 status를 polling한다. 각 요청의 HTTP 200, endpoint-native success terminal 정확히 1개, `[DONE]` 정확히 1개, selected provider의 eligible capacity 미초과, exact peak와 queue, 최종 0 회복을 별도 증거로 남긴다.
|
||||
- managed smoke invocation마다 ignored `agent-test/runs/**` 아래 mode `0700` unique directory와 immutable run id를 생성한다. manifest가 소유하고 현재 run/dispatch 이후 생성된 request/result/status만 판정하며 stale/missing/foreign-run body는 실패한다. tracked/review evidence에는 sanitized summary의 run id, script hash, alias, selected provider, endpoint, computed shape/class, terminal count, peak/queue/final counter, outcome만 허용하고 token/header, route/slot id, prompt, response body와 출력은 기록하지 않는다.
|
||||
- Qwen provider-pool smoke는 thinking/reasoning 텍스트가 포함될 수 있다. 추론 출력 자체를 실패로 보지 말고 HTTP 성공, model alias, final marker 포함 여부, provider node log/run count 증가로 판정한다. 응답 전체가 특정 token과 정확히 같은지 비교하는 strict exact-match는 이 profile의 기본 판정으로 쓰지 않는다.
|
||||
- Qwen provider를 agent/tool-call 용도로 검증할 때는 일반 chat smoke와 별도로 forced tool call, auto tool call, streaming `delta.tool_calls`, multi-turn tool result 후 최종 답변을 확인한다. raw native marker나 reasoning text가 assistant content로 새면 해당 model/runtime의 parser/template profile 미확정으로 보고한다.
|
||||
- Qwen runtime에는 Qwen 전용 parser/template 검증값만 사용한다. dev-corp Gemma 계열의 `tool_call_parser=gemma4`, `reasoning_parser=gemma4`, Gemma4 chat template/profile을 Qwen provider에 복사하지 않는다.
|
||||
|
|
|
|||
|
|
@ -165,6 +165,16 @@ http://<edge-host>:18081/v1
|
|||
- `think=false` 또는 `reasoning_effort=none`은 hide-only 옵션이 아니라 thinking disable 요청이다. Provider-native field가 있는 경우 해당 field를 우선 사용해 passthrough 보존을 검증한다.
|
||||
- Provider-pool passthrough 파라미터의 세부 계약과 금지/허용 범위는 `agent-contract/outer/openai-compatible-api.md`를 기준으로 한다.
|
||||
|
||||
### Managed route-qualified capacity smoke
|
||||
|
||||
Managed mode의 capacity는 전역 model catalog나 같은 upstream model을 제공하는 모든 provider의 합이 아닙니다. 먼저 OpenAI ingress와 같은 principal token으로 active route alias를 조회하고, route의 `resource_selector`, profile, upstream model과 일치하는 현재 healthy provider snapshot만 eligible capacity로 계산합니다. 현재 dev projection에서는 `ornith:35b`가 `onexplayer-lemonade`, `ornith-fast`가 `rtx5090-lemonade`에 각각 고정되므로 두 capacity를 합치거나 alias를 한 batch에 섞지 않습니다.
|
||||
|
||||
정상 capacity 검증은 `scripts/e2e-openai-managed-capacity-smoke.sh`로 Chat과 Responses를 한 endpoint·route씩 실행합니다. 스크립트는 실제 emitted JSON의 Unicode rune 수와 `runes/4 + runes/16` estimate를 계산해 `normal`임을 확인하고, selected provider의 `capacity + 1`만 전송합니다. 별도 long-context/repeat smoke는 `long_context_capacity`를 사용하며 normal-capacity 완료 근거를 대체하지 않습니다.
|
||||
|
||||
성공 조건은 모든 요청 HTTP 200, Chat의 finish terminal과 Responses의 `response.completed` 각각 정확히 1개, stream별 `[DONE]` 정확히 1개, selected provider peak가 eligible capacity와 같고 queue가 1 이상인 상태, 최종 `in_flight=0`/`queued=0`입니다. route mismatch, context class mismatch, non-selected capacity 포함, terminal 누락/중복, status 관측 누락은 fail-closed입니다.
|
||||
|
||||
각 invocation은 ignored `agent-test/runs/**` 아래 mode `0700` unique directory를 생성하고 current-run manifest가 소유한 request/result/status만 판정합니다. Raw route DTO, token/header, route/slot id, prompt, request/response body와 모델 출력은 해당 디렉터리 밖으로 복사하지 않습니다. tracked evidence와 code review에는 allowlist된 sanitized summary의 run id, script hash, route alias, selected provider, endpoint, computed request shape/class, terminal count, peak/queue/final counter와 outcome만 남깁니다.
|
||||
|
||||
예시 (dev-corp `gemma4:26b` provider-pool non-stream 측정, think 생략):
|
||||
|
||||
아래 `18081` 포트는 local Edge 예시다. dev-corp public smoke에서는 base URL을 `https://digitalplatform.iop.ai.kr/v1`로 바꾼다. Direct Edge listener `http://digitalplatform.iop.ai.kr:18086/v1`도 동작하지만 사용자-facing 기본값은 포트 없는 public URL이다.
|
||||
|
|
|
|||
932
scripts/e2e-openai-managed-capacity-smoke.sh
Executable file
932
scripts/e2e-openai-managed-capacity-smoke.sh
Executable file
|
|
@ -0,0 +1,932 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Route-qualified managed OpenAI capacity smoke.
|
||||
#
|
||||
# One live invocation owns exactly one public route and one endpoint. It resolves
|
||||
# that route with the same principal token used at ingress, selects only the
|
||||
# route-qualified provider snapshot, sends selected capacity + 1 requests, and
|
||||
# retains raw material only under the ignored agent-test/runs tree. The emitted
|
||||
# summary is deliberately secret- and payload-free.
|
||||
|
||||
SCRIPT_PATH="${BASH_SOURCE[0]}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "$SCRIPT_PATH")" && pwd -P)"
|
||||
if [ -d "$SCRIPT_DIR/../agent-test" ]; then
|
||||
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||
else
|
||||
REPO_ROOT="$(pwd -P)"
|
||||
fi
|
||||
|
||||
SELF_TEST=0
|
||||
MODEL=""
|
||||
PROVIDER=""
|
||||
ENDPOINT=""
|
||||
BASE_URL="https://toki-labs.com:18083/v1"
|
||||
ROUTES_URL="https://toki-labs.com:18001/v1/credentials/routes"
|
||||
STATUS_URL="https://toki-labs.com:18001/edges/edge-toki-labs-dev/status"
|
||||
CA_FILE=""
|
||||
RUNS_ROOT="$REPO_ROOT/agent-test/runs"
|
||||
LONG_CONTEXT_THRESHOLD_TOKENS=""
|
||||
CALLER_TIMEOUT_SECONDS=180
|
||||
POLL_INTERVAL_SECONDS=0.075
|
||||
RECOVERY_TIMEOUT_SECONDS=120
|
||||
RESOLVE_HOST=""
|
||||
RESOLVE_IP=""
|
||||
TOKEN_FROM_STDIN=0
|
||||
RUN_DIR=""
|
||||
TOKEN=""
|
||||
AUTH_HEADER_FILE=""
|
||||
SELF_TEST_TMP_ROOT=""
|
||||
|
||||
log() {
|
||||
printf '[managed-capacity-smoke] %s\n' "$*"
|
||||
}
|
||||
|
||||
die() {
|
||||
log "ERROR: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
cleanup_live_secret() {
|
||||
unset TOKEN 2>/dev/null || true
|
||||
TOKEN=""
|
||||
if [ -n "${AUTH_HEADER_FILE:-}" ]; then
|
||||
rm -f "$AUTH_HEADER_FILE"
|
||||
AUTH_HEADER_FILE=""
|
||||
fi
|
||||
}
|
||||
|
||||
usage() {
|
||||
sed -n '3,43p' "$0"
|
||||
cat <<'EOF'
|
||||
|
||||
Live usage:
|
||||
e2e-openai-managed-capacity-smoke.sh \
|
||||
--model <public-route-alias> --provider <resource-selector> \
|
||||
--endpoint <chat|responses> --long-context-threshold-tokens <n> \
|
||||
--ca-file <ca.pem> --token-stdin [--resolve-host <host> --resolve-ip <ip>]
|
||||
|
||||
Self-test:
|
||||
e2e-openai-managed-capacity-smoke.sh --self-test
|
||||
|
||||
The live token is read as one line from stdin and is never printed. Raw route,
|
||||
request, response, curl, and status files remain in a new mode-0700 directory
|
||||
under agent-test/runs. Each invocation handles exactly one route and endpoint.
|
||||
EOF
|
||||
}
|
||||
|
||||
while [ "$#" -gt 0 ]; do
|
||||
case "$1" in
|
||||
--self-test)
|
||||
SELF_TEST=1
|
||||
shift
|
||||
;;
|
||||
--model)
|
||||
[ "$#" -ge 2 ] || die "--model requires a value"
|
||||
MODEL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--provider)
|
||||
[ "$#" -ge 2 ] || die "--provider requires a value"
|
||||
PROVIDER="$2"
|
||||
shift 2
|
||||
;;
|
||||
--endpoint)
|
||||
[ "$#" -ge 2 ] || die "--endpoint requires a value"
|
||||
ENDPOINT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--base-url)
|
||||
[ "$#" -ge 2 ] || die "--base-url requires a value"
|
||||
BASE_URL="${2%/}"
|
||||
shift 2
|
||||
;;
|
||||
--routes-url)
|
||||
[ "$#" -ge 2 ] || die "--routes-url requires a value"
|
||||
ROUTES_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--status-url)
|
||||
[ "$#" -ge 2 ] || die "--status-url requires a value"
|
||||
STATUS_URL="$2"
|
||||
shift 2
|
||||
;;
|
||||
--ca-file)
|
||||
[ "$#" -ge 2 ] || die "--ca-file requires a value"
|
||||
CA_FILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--runs-root)
|
||||
[ "$#" -ge 2 ] || die "--runs-root requires a value"
|
||||
RUNS_ROOT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--long-context-threshold-tokens)
|
||||
[ "$#" -ge 2 ] || die "--long-context-threshold-tokens requires a value"
|
||||
LONG_CONTEXT_THRESHOLD_TOKENS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--caller-timeout-seconds)
|
||||
[ "$#" -ge 2 ] || die "--caller-timeout-seconds requires a value"
|
||||
CALLER_TIMEOUT_SECONDS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--recovery-timeout-seconds)
|
||||
[ "$#" -ge 2 ] || die "--recovery-timeout-seconds requires a value"
|
||||
RECOVERY_TIMEOUT_SECONDS="$2"
|
||||
shift 2
|
||||
;;
|
||||
--resolve-host)
|
||||
[ "$#" -ge 2 ] || die "--resolve-host requires a value"
|
||||
RESOLVE_HOST="$2"
|
||||
shift 2
|
||||
;;
|
||||
--resolve-ip)
|
||||
[ "$#" -ge 2 ] || die "--resolve-ip requires a value"
|
||||
RESOLVE_IP="$2"
|
||||
shift 2
|
||||
;;
|
||||
--token-stdin)
|
||||
TOKEN_FROM_STDIN=1
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
die "unknown argument: $1"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
for required in python3 curl; do
|
||||
command -v "$required" >/dev/null 2>&1 || die "$required is required"
|
||||
done
|
||||
|
||||
script_sha256() {
|
||||
python3 - "$SCRIPT_PATH" <<'PY'
|
||||
import hashlib
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
print(hashlib.sha256(pathlib.Path(sys.argv[1]).read_bytes()).hexdigest())
|
||||
PY
|
||||
}
|
||||
|
||||
allocate_run_dir() {
|
||||
local root="$1"
|
||||
local directory mode
|
||||
umask 077
|
||||
mkdir -p "$root"
|
||||
directory="$(mktemp -d "$root/openai-managed-capacity.XXXXXX")"
|
||||
chmod 700 "$directory"
|
||||
mode="$(python3 - "$directory" <<'PY'
|
||||
import os
|
||||
import stat
|
||||
import sys
|
||||
|
||||
print(oct(stat.S_IMODE(os.stat(sys.argv[1]).st_mode))[2:].zfill(4))
|
||||
PY
|
||||
)"
|
||||
[ "$mode" = "0700" ] || die "run directory mode is $mode, want 0700"
|
||||
printf '%s\n' "$directory"
|
||||
}
|
||||
|
||||
resolve_route_capacity() {
|
||||
local routes_file="$1"
|
||||
local status_file="$2"
|
||||
local model="$3"
|
||||
local provider="$4"
|
||||
local context_class="$5"
|
||||
local claimed_capacity="${6:-}"
|
||||
python3 - "$routes_file" "$status_file" "$model" "$provider" "$context_class" "$claimed_capacity" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
routes_path, status_path, model, expected_provider, context_class, claimed = sys.argv[1:]
|
||||
|
||||
def field(obj, *names):
|
||||
for name in names:
|
||||
if name in obj:
|
||||
return obj[name]
|
||||
return None
|
||||
|
||||
routes_doc = json.load(open(routes_path, encoding="utf-8"))
|
||||
if isinstance(routes_doc, dict):
|
||||
routes = field(routes_doc, "routes", "Routes", "data", "Data")
|
||||
else:
|
||||
routes = routes_doc
|
||||
if not isinstance(routes, list):
|
||||
raise SystemExit("route collection is not an array")
|
||||
|
||||
matches = []
|
||||
for route in routes:
|
||||
if not isinstance(route, dict):
|
||||
continue
|
||||
alias = str(field(route, "alias", "Alias") or "").strip()
|
||||
status = str(field(route, "status", "Status") or "").strip().lower()
|
||||
if alias == model and status == "active":
|
||||
matches.append(route)
|
||||
if len(matches) != 1:
|
||||
raise SystemExit(f"active exact route count is {len(matches)}, want 1")
|
||||
|
||||
route = matches[0]
|
||||
selector = str(field(route, "resource_selector", "ResourceSelector") or "").strip()
|
||||
profile = str(field(route, "profile_id", "ProfileID") or "").strip()
|
||||
upstream = str(field(route, "upstream_model", "UpstreamModel") or "").strip()
|
||||
if selector != expected_provider:
|
||||
raise SystemExit("route selector does not match expected provider")
|
||||
if not profile or not upstream:
|
||||
raise SystemExit("route profile or upstream model is empty")
|
||||
|
||||
status_doc = json.load(open(status_path, encoding="utf-8"))
|
||||
snapshots = []
|
||||
for node in status_doc.get("nodes") or status_doc.get("Nodes") or []:
|
||||
if not isinstance(node, dict) or field(node, "connected", "Connected") is not True:
|
||||
continue
|
||||
for snapshot in field(node, "provider_snapshots", "ProviderSnapshots") or []:
|
||||
if not isinstance(snapshot, dict):
|
||||
continue
|
||||
provider_id = str(field(snapshot, "id", "ID", "adapter", "Adapter") or "").strip()
|
||||
if provider_id == selector:
|
||||
snapshots.append(snapshot)
|
||||
if len(snapshots) != 1:
|
||||
raise SystemExit(f"connected selected-provider snapshot count is {len(snapshots)}, want 1")
|
||||
|
||||
snapshot = snapshots[0]
|
||||
health = str(field(snapshot, "health", "Health") or "").strip().lower()
|
||||
status = str(field(snapshot, "status", "Status") or "").strip().lower()
|
||||
if health not in {"healthy", "available"} or status != "available":
|
||||
raise SystemExit("selected provider is not healthy and available")
|
||||
served = field(snapshot, "served_models", "ServedModels") or []
|
||||
if upstream not in served:
|
||||
raise SystemExit("route upstream model is absent from selected provider snapshot")
|
||||
if context_class == "normal":
|
||||
capacity = int(field(snapshot, "capacity", "Capacity") or 0)
|
||||
elif context_class == "long":
|
||||
capacity = int(field(snapshot, "long_context_capacity", "LongContextCapacity") or 0)
|
||||
else:
|
||||
raise SystemExit("context class must be normal or long")
|
||||
if capacity <= 0:
|
||||
raise SystemExit("selected eligible capacity is not positive")
|
||||
if claimed and int(claimed) != capacity:
|
||||
raise SystemExit("claimed capacity includes route-ineligible resources")
|
||||
|
||||
print(json.dumps({
|
||||
"route_alias": model,
|
||||
"selected_provider": selector,
|
||||
"profile_present": True,
|
||||
"upstream_model_match": True,
|
||||
"context_class": context_class,
|
||||
"eligible_capacity": capacity,
|
||||
}, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
}
|
||||
|
||||
parse_terminal() {
|
||||
local endpoint="$1"
|
||||
local body_file="$2"
|
||||
python3 - "$endpoint" "$body_file" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
endpoint, path = sys.argv[1:]
|
||||
data = pathlib.Path(path).read_text(encoding="utf-8", errors="strict")
|
||||
done = finish = completed = errors = 0
|
||||
data_events = []
|
||||
for raw_line in data.splitlines():
|
||||
if not raw_line.startswith("data:"):
|
||||
continue
|
||||
payload = raw_line[5:].strip()
|
||||
data_events.append(payload)
|
||||
if payload == "[DONE]":
|
||||
done += 1
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(payload)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise SystemExit(f"invalid SSE JSON: {exc}")
|
||||
if not isinstance(obj, dict):
|
||||
raise SystemExit("SSE data payload is not an object")
|
||||
event_type = str(obj.get("type") or "")
|
||||
if event_type == "response.completed":
|
||||
completed += 1
|
||||
if event_type in {"error", "run_error", "response.failed", "response.incomplete"} or obj.get("error"):
|
||||
errors += 1
|
||||
choices = obj.get("choices") or []
|
||||
if not isinstance(choices, list):
|
||||
raise SystemExit("choices is not an array")
|
||||
for choice in choices:
|
||||
if isinstance(choice, dict) and choice.get("finish_reason") is not None:
|
||||
finish += 1
|
||||
if not data_events or data_events[-1] != "[DONE]":
|
||||
raise SystemExit("[DONE] is missing or is not the final data event")
|
||||
if done != 1 or errors != 0:
|
||||
raise SystemExit(f"terminal counts invalid: done={done} errors={errors}")
|
||||
if endpoint == "chat":
|
||||
if finish != 1 or completed != 0:
|
||||
raise SystemExit(f"chat terminal counts invalid: finish={finish} completed={completed}")
|
||||
elif endpoint == "responses":
|
||||
if completed != 1 or finish != 0:
|
||||
raise SystemExit(f"responses terminal counts invalid: completed={completed} finish={finish}")
|
||||
else:
|
||||
raise SystemExit("unsupported endpoint")
|
||||
print(json.dumps({"done_count": done, "finish_count": finish, "completed_count": completed, "error_count": errors}, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
}
|
||||
|
||||
validate_provenance() {
|
||||
local run_dir="$1"
|
||||
local manifest="$2"
|
||||
python3 - "$run_dir" "$manifest" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
run_dir = pathlib.Path(sys.argv[1]).resolve(strict=True)
|
||||
manifest_path = pathlib.Path(sys.argv[2]).resolve(strict=True)
|
||||
if manifest_path.parent != run_dir:
|
||||
raise SystemExit("manifest is not owned by the current run directory")
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
run_id = str(manifest.get("run_id") or "")
|
||||
started_ns = int(manifest.get("run_started_ns") or 0)
|
||||
dispatch_ns = int(manifest.get("dispatch_started_ns") or 0)
|
||||
if not run_id or started_ns <= 0 or dispatch_ns <= started_ns:
|
||||
raise SystemExit("manifest run identity or timestamps are invalid")
|
||||
owner = json.loads((run_dir / "run-owner.json").read_text(encoding="utf-8"))
|
||||
if owner.get("run_id") != run_id or int(owner.get("run_started_ns") or 0) != started_ns:
|
||||
raise SystemExit("run owner does not match manifest")
|
||||
cases = manifest.get("cases")
|
||||
if not isinstance(cases, list) or not cases:
|
||||
raise SystemExit("manifest contains no cases")
|
||||
|
||||
seen = set()
|
||||
for case in cases:
|
||||
case_id = str(case.get("case_id") or "")
|
||||
if not case_id or case_id in seen:
|
||||
raise SystemExit("case identity is empty or duplicated")
|
||||
seen.add(case_id)
|
||||
for key in ("request", "result", "http_code", "curl_rc", "duration"):
|
||||
raw = str(case.get(key) or "")
|
||||
path = (run_dir / raw).resolve(strict=True)
|
||||
if run_dir not in path.parents:
|
||||
raise SystemExit(f"{case_id} {key} is owned by another run")
|
||||
stat = path.stat()
|
||||
if key == "request":
|
||||
if stat.st_mtime_ns < started_ns:
|
||||
raise SystemExit(f"{case_id} request predates current run")
|
||||
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
if digest != case.get("request_sha256"):
|
||||
raise SystemExit(f"{case_id} request digest mismatch")
|
||||
else:
|
||||
if stat.st_mtime_ns < dispatch_ns:
|
||||
raise SystemExit(f"{case_id} {key} predates current dispatch")
|
||||
result = (run_dir / str(case["result"])).resolve(strict=True)
|
||||
result_digest = hashlib.sha256(result.read_bytes()).hexdigest()
|
||||
ownership = json.loads((run_dir / str(case.get("result_owner") or "")).read_text(encoding="utf-8"))
|
||||
if ownership.get("run_id") != run_id or ownership.get("case_id") != case_id:
|
||||
raise SystemExit(f"{case_id} result owner is foreign")
|
||||
if ownership.get("result_sha256") != result_digest:
|
||||
raise SystemExit(f"{case_id} result digest mismatch")
|
||||
print(json.dumps({"run_id": run_id, "case_count": len(cases), "provenance": "current-run"}, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
}
|
||||
|
||||
write_self_test_manifest() {
|
||||
local run_dir="$1"
|
||||
local run_id="$2"
|
||||
local endpoint="$3"
|
||||
local body_file="$4"
|
||||
local result_file="$5"
|
||||
python3 - "$run_dir" "$run_id" "$endpoint" "$body_file" "$result_file" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import time
|
||||
import sys
|
||||
|
||||
root = pathlib.Path(sys.argv[1]).resolve()
|
||||
run_id, endpoint = sys.argv[2:4]
|
||||
request = pathlib.Path(sys.argv[4]).resolve()
|
||||
result = pathlib.Path(sys.argv[5]).resolve()
|
||||
started = min(request.stat().st_mtime_ns, result.stat().st_mtime_ns) - 1_000_000
|
||||
dispatch = result.stat().st_mtime_ns - 1
|
||||
(root / "run-owner.json").write_text(json.dumps({"run_id": run_id, "run_started_ns": started}), encoding="utf-8")
|
||||
for name, value in (("http.code", "200\n"), ("curl.rc", "0\n"), ("duration.json", '{"duration_ms":1}\n')):
|
||||
(root / name).write_text(value, encoding="utf-8")
|
||||
owner_path = root / "result-owner.json"
|
||||
owner_path.write_text(json.dumps({
|
||||
"run_id": run_id,
|
||||
"case_id": "case-1",
|
||||
"result_sha256": hashlib.sha256(result.read_bytes()).hexdigest(),
|
||||
}), encoding="utf-8")
|
||||
manifest = {
|
||||
"run_id": run_id,
|
||||
"run_started_ns": started,
|
||||
"dispatch_started_ns": dispatch,
|
||||
"endpoint": endpoint,
|
||||
"cases": [{
|
||||
"case_id": "case-1",
|
||||
"request": request.relative_to(root).as_posix(),
|
||||
"request_sha256": hashlib.sha256(request.read_bytes()).hexdigest(),
|
||||
"result": result.relative_to(root).as_posix(),
|
||||
"result_owner": owner_path.relative_to(root).as_posix(),
|
||||
"http_code": "http.code",
|
||||
"curl_rc": "curl.rc",
|
||||
"duration": "duration.json",
|
||||
}],
|
||||
}
|
||||
(root / "manifest.json").write_text(json.dumps(manifest, indent=2), encoding="utf-8")
|
||||
PY
|
||||
}
|
||||
|
||||
expect_failure() {
|
||||
local label="$1"
|
||||
shift
|
||||
if "$@" >/dev/null 2>&1; then
|
||||
die "self-test negative was accepted: $label"
|
||||
fi
|
||||
log "self-test rejected=$label"
|
||||
}
|
||||
|
||||
run_self_test() {
|
||||
local tmp_root run_a run_b mode_a mode_b
|
||||
tmp_root="$(mktemp -d "${TMPDIR:-/tmp}/iop-managed-capacity-self-test.XXXXXX")"
|
||||
SELF_TEST_TMP_ROOT="$tmp_root"
|
||||
trap 'if [ -n "${SELF_TEST_TMP_ROOT:-}" ]; then rm -rf "$SELF_TEST_TMP_ROOT"; fi' EXIT
|
||||
run_a="$(allocate_run_dir "$tmp_root")"
|
||||
run_b="$(allocate_run_dir "$tmp_root")"
|
||||
[ "$run_a" != "$run_b" ] || die "self-test run directories are not unique"
|
||||
mode_a="$(python3 -c 'import os,stat,sys; print(oct(stat.S_IMODE(os.stat(sys.argv[1]).st_mode))[2:].zfill(4))' "$run_a")"
|
||||
mode_b="$(python3 -c 'import os,stat,sys; print(oct(stat.S_IMODE(os.stat(sys.argv[1]).st_mode))[2:].zfill(4))' "$run_b")"
|
||||
[ "$mode_a" = "0700" ] && [ "$mode_b" = "0700" ] || die "self-test run directory mode mismatch"
|
||||
log "self-test unique_run_directories=true mode=0700"
|
||||
|
||||
python3 - "$run_a" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
routes = [{
|
||||
"Alias": "ornith:35b", "ProfileID": "openai", "UpstreamModel": "ornith-upstream",
|
||||
"ResourceSelector": "onexplayer-lemonade", "Status": "active",
|
||||
}]
|
||||
status = {"nodes": [{"connected": True, "provider_snapshots": [
|
||||
{"id": "onexplayer-lemonade", "status": "available", "health": "healthy", "capacity": 3,
|
||||
"long_context_capacity": 2, "in_flight": 0, "queued": 0, "served_models": ["ornith-upstream"]},
|
||||
{"id": "rtx5090-lemonade", "status": "available", "health": "healthy", "capacity": 1,
|
||||
"long_context_capacity": 1, "in_flight": 0, "queued": 0, "served_models": ["ornith-upstream"]},
|
||||
]}]}
|
||||
(root / "routes.json").write_text(json.dumps(routes), encoding="utf-8")
|
||||
(root / "status.json").write_text(json.dumps(status), encoding="utf-8")
|
||||
(root / "request.json").write_text('{"model":"ornith:35b"}', encoding="utf-8")
|
||||
(root / "chat.sse").write_text('data: {"choices":[{"finish_reason":"stop"}]}\n\ndata: [DONE]\n\n', encoding="utf-8")
|
||||
(root / "responses.sse").write_text('data: {"type":"response.completed"}\n\ndata: [DONE]\n\n', encoding="utf-8")
|
||||
(root / "chat-duplicate.sse").write_text('data: {"choices":[{"finish_reason":"stop"}]}\n\ndata: [DONE]\n\ndata: [DONE]\n\n', encoding="utf-8")
|
||||
(root / "responses-missing.sse").write_text('data: {"type":"response.output_text.done"}\n\ndata: [DONE]\n\n', encoding="utf-8")
|
||||
PY
|
||||
local normal_json long_json
|
||||
normal_json="$(resolve_route_capacity "$run_a/routes.json" "$run_a/status.json" "ornith:35b" "onexplayer-lemonade" normal 3)"
|
||||
long_json="$(resolve_route_capacity "$run_a/routes.json" "$run_a/status.json" "ornith:35b" "onexplayer-lemonade" long 2)"
|
||||
python3 - "$normal_json" "$long_json" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
normal, long = map(json.loads, sys.argv[1:])
|
||||
assert normal["eligible_capacity"] == 3
|
||||
assert long["eligible_capacity"] == 2
|
||||
PY
|
||||
expect_failure route-selector-mismatch resolve_route_capacity "$run_a/routes.json" "$run_a/status.json" "ornith:35b" "rtx5090-lemonade" normal
|
||||
expect_failure aggregate-capacity-claim resolve_route_capacity "$run_a/routes.json" "$run_a/status.json" "ornith:35b" "onexplayer-lemonade" normal 4
|
||||
log "self-test route_exclusion=true normal_capacity=3 long_capacity=2"
|
||||
|
||||
parse_terminal chat "$run_a/chat.sse" >/dev/null
|
||||
parse_terminal responses "$run_a/responses.sse" >/dev/null
|
||||
expect_failure duplicate-terminal parse_terminal chat "$run_a/chat-duplicate.sse"
|
||||
expect_failure missing-terminal parse_terminal responses "$run_a/responses-missing.sse"
|
||||
log "self-test exact_chat_and_responses_terminals=true"
|
||||
|
||||
write_self_test_manifest "$run_a" self-test-a chat "$run_a/request.json" "$run_a/chat.sse"
|
||||
validate_provenance "$run_a" "$run_a/manifest.json" >/dev/null
|
||||
python3 - "$run_a" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
manifest = json.loads((root / "manifest.json").read_text())
|
||||
old = manifest["dispatch_started_ns"] - 1_000_000_000
|
||||
os.utime(root / "chat.sse", ns=(old, old))
|
||||
PY
|
||||
expect_failure stale-body validate_provenance "$run_a" "$run_a/manifest.json"
|
||||
|
||||
python3 - "$run_a" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
manifest = json.loads((root / "manifest.json").read_text())
|
||||
manifest["cases"][0]["result"] = "missing.sse"
|
||||
(root / "manifest-missing.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
PY
|
||||
expect_failure missing-body validate_provenance "$run_a" "$run_a/manifest-missing.json"
|
||||
|
||||
python3 - "$run_a" "$run_b" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
root, other = map(pathlib.Path, sys.argv[1:])
|
||||
(other / "foreign.sse").write_text('data: [DONE]\n\n', encoding="utf-8")
|
||||
manifest = json.loads((root / "manifest.json").read_text())
|
||||
manifest["cases"][0]["result"] = str((other / "foreign.sse").resolve())
|
||||
(root / "manifest-foreign.json").write_text(json.dumps(manifest), encoding="utf-8")
|
||||
PY
|
||||
expect_failure foreign-run-body validate_provenance "$run_a" "$run_a/manifest-foreign.json"
|
||||
log "self-test current_manifest_acceptance=true stale_missing_foreign_rejected=true"
|
||||
log "SELF_TEST_PASS"
|
||||
}
|
||||
|
||||
if [ "$SELF_TEST" -eq 1 ]; then
|
||||
run_self_test
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[ -n "$MODEL" ] || die "--model is required"
|
||||
[ -n "$PROVIDER" ] || die "--provider is required"
|
||||
case "$ENDPOINT" in chat|responses) ;; *) die "--endpoint must be chat or responses" ;; esac
|
||||
case "$LONG_CONTEXT_THRESHOLD_TOKENS" in ''|*[!0-9]*) die "--long-context-threshold-tokens must be a positive integer" ;; esac
|
||||
[ "$LONG_CONTEXT_THRESHOLD_TOKENS" -gt 0 ] || die "--long-context-threshold-tokens must be positive"
|
||||
case "$CALLER_TIMEOUT_SECONDS" in ''|*[!0-9]*) die "--caller-timeout-seconds must be a positive integer" ;; esac
|
||||
case "$RECOVERY_TIMEOUT_SECONDS" in ''|*[!0-9]*) die "--recovery-timeout-seconds must be a positive integer" ;; esac
|
||||
[ "$TOKEN_FROM_STDIN" -eq 1 ] || die "--token-stdin is required for live mode"
|
||||
[ -n "$CA_FILE" ] && [ -f "$CA_FILE" ] || die "--ca-file must name a readable file"
|
||||
if { [ -n "$RESOLVE_HOST" ] && [ -z "$RESOLVE_IP" ]; } || { [ -z "$RESOLVE_HOST" ] && [ -n "$RESOLVE_IP" ]; }; then
|
||||
die "--resolve-host and --resolve-ip must be supplied together"
|
||||
fi
|
||||
IFS= read -r TOKEN || die "failed to read principal token from stdin"
|
||||
[ -n "$TOKEN" ] || die "principal token is empty"
|
||||
|
||||
RUN_DIR="$(allocate_run_dir "$RUNS_ROOT")"
|
||||
RUN_ID="$(python3 - <<'PY'
|
||||
import secrets
|
||||
import time
|
||||
print(f"{time.time_ns()}-{secrets.token_hex(8)}")
|
||||
PY
|
||||
)"
|
||||
RUN_STARTED_NS="$(python3 -c 'import time; print(time.time_ns())')"
|
||||
SCRIPT_SHA256="$(script_sha256)"
|
||||
python3 - "$RUN_DIR/run-owner.json" "$RUN_ID" "$RUN_STARTED_NS" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
path, run_id, started = sys.argv[1:]
|
||||
pathlib.Path(path).write_text(json.dumps({"run_id": run_id, "run_started_ns": int(started)}), encoding="utf-8")
|
||||
PY
|
||||
AUTH_HEADER_FILE="$RUN_DIR/auth-header"
|
||||
printf 'Authorization: Bearer %s\n' "$TOKEN" >"$AUTH_HEADER_FILE"
|
||||
chmod 600 "$AUTH_HEADER_FILE"
|
||||
unset TOKEN
|
||||
TOKEN=""
|
||||
trap cleanup_live_secret EXIT
|
||||
trap 'cleanup_live_secret; exit 1' HUP INT TERM
|
||||
|
||||
CURL_COMMON=(--noproxy '*' --connect-timeout 10 --cacert "$CA_FILE")
|
||||
CURL_RESOLVE=()
|
||||
if [ -n "$RESOLVE_HOST" ]; then
|
||||
while IFS= read -r resolve; do
|
||||
[ -n "$resolve" ] && CURL_RESOLVE+=(--resolve "$resolve")
|
||||
done < <(python3 - "$RESOLVE_HOST" "$RESOLVE_IP" "$BASE_URL" "$ROUTES_URL" "$STATUS_URL" <<'PY'
|
||||
import sys
|
||||
import urllib.parse
|
||||
host, ip = sys.argv[1:3]
|
||||
seen = set()
|
||||
for value in sys.argv[3:]:
|
||||
parsed = urllib.parse.urlparse(value)
|
||||
port = parsed.port or (443 if parsed.scheme == "https" else 80)
|
||||
item = f"{host}:{port}:{ip}"
|
||||
if item not in seen:
|
||||
print(item)
|
||||
seen.add(item)
|
||||
PY
|
||||
)
|
||||
fi
|
||||
|
||||
curl "${CURL_COMMON[@]}" ${CURL_RESOLVE[@]+"${CURL_RESOLVE[@]}"} --max-time 15 -fsS \
|
||||
-H "@$AUTH_HEADER_FILE" "$ROUTES_URL" >"$RUN_DIR/routes.json"
|
||||
curl "${CURL_COMMON[@]}" ${CURL_RESOLVE[@]+"${CURL_RESOLVE[@]}"} --max-time 10 -fsS \
|
||||
"$STATUS_URL" >"$RUN_DIR/status-initial.json"
|
||||
|
||||
REQUEST_INFO="$(python3 - "$RUN_DIR" "$MODEL" "$ENDPOINT" "$LONG_CONTEXT_THRESHOLD_TOKENS" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
model, endpoint, threshold = sys.argv[2], sys.argv[3], int(sys.argv[4])
|
||||
prompt = (
|
||||
"Write a generic capacity qualification report of approximately 850 tokens. "
|
||||
"Use exactly ten numbered sections with short headings and concrete but non-sensitive operational observations. "
|
||||
"Do not use tools or hidden reasoning. End with the marker MANAGED_CAPACITY_COMPLETE."
|
||||
)
|
||||
common = {
|
||||
"model": model,
|
||||
"stream": True,
|
||||
"temperature": 0,
|
||||
"chat_template_kwargs": {"enable_thinking": False},
|
||||
}
|
||||
if endpoint == "chat":
|
||||
body = dict(common, messages=[{"role": "user", "content": prompt}], max_tokens=1200)
|
||||
else:
|
||||
body = dict(common, input=prompt, max_output_tokens=1200)
|
||||
encoded = json.dumps(body, ensure_ascii=False, separators=(",", ":"))
|
||||
runes = len(encoded)
|
||||
estimate = runes // 4 + runes // 16
|
||||
context_class = "long" if estimate >= threshold else "normal"
|
||||
path = root / "request-template.json"
|
||||
path.write_text(encoded, encoding="utf-8")
|
||||
print(json.dumps({
|
||||
"request_runes": runes,
|
||||
"estimated_input_tokens": max(1, estimate),
|
||||
"context_class": context_class,
|
||||
"request_sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
}, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
)"
|
||||
CONTEXT_CLASS="$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["context_class"])' "$REQUEST_INFO")"
|
||||
[ "$CONTEXT_CLASS" = "normal" ] || die "managed normal-capacity smoke emitted a $CONTEXT_CLASS request"
|
||||
ROUTE_INFO="$(resolve_route_capacity "$RUN_DIR/routes.json" "$RUN_DIR/status-initial.json" "$MODEL" "$PROVIDER" "$CONTEXT_CLASS")"
|
||||
ELIGIBLE_CAPACITY="$(python3 -c 'import json,sys; print(json.loads(sys.argv[1])["eligible_capacity"])' "$ROUTE_INFO")"
|
||||
REQUEST_COUNT=$((ELIGIBLE_CAPACITY + 1))
|
||||
|
||||
python3 - "$RUN_DIR" "$RUN_ID" "$RUN_STARTED_NS" "$MODEL" "$PROVIDER" "$ENDPOINT" "$REQUEST_COUNT" "$REQUEST_INFO" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
run_id, started, model, provider, endpoint = sys.argv[2:7]
|
||||
request_count = int(sys.argv[7])
|
||||
request_info = json.loads(sys.argv[8])
|
||||
template = root / "request-template.json"
|
||||
cases = []
|
||||
for index in range(1, request_count + 1):
|
||||
case_id = f"case-{index}"
|
||||
case_dir = root / case_id
|
||||
case_dir.mkdir(mode=0o700)
|
||||
request = case_dir / "request.json"
|
||||
shutil.copyfile(template, request)
|
||||
cases.append({
|
||||
"case_id": case_id,
|
||||
"request": request.relative_to(root).as_posix(),
|
||||
"request_sha256": hashlib.sha256(request.read_bytes()).hexdigest(),
|
||||
"result": f"{case_id}/response.sse",
|
||||
"result_owner": f"{case_id}/result-owner.json",
|
||||
"http_code": f"{case_id}/http.code",
|
||||
"curl_rc": f"{case_id}/curl.rc",
|
||||
"duration": f"{case_id}/duration.json",
|
||||
})
|
||||
manifest = {
|
||||
"run_id": run_id,
|
||||
"run_started_ns": int(started),
|
||||
"dispatch_started_ns": 0,
|
||||
"route_alias": model,
|
||||
"selected_provider": provider,
|
||||
"endpoint": endpoint,
|
||||
"request_shape": request_info,
|
||||
"cases": cases,
|
||||
}
|
||||
(root / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True), encoding="utf-8")
|
||||
PY
|
||||
|
||||
DISPATCH_STARTED_NS="$(python3 -c 'import time; print(time.time_ns())')"
|
||||
python3 - "$RUN_DIR/manifest.json" "$DISPATCH_STARTED_NS" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
path = pathlib.Path(sys.argv[1])
|
||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
||||
manifest["dispatch_started_ns"] = int(sys.argv[2])
|
||||
path.write_text(json.dumps(manifest, indent=2, sort_keys=True), encoding="utf-8")
|
||||
PY
|
||||
|
||||
if [ "$ENDPOINT" = "chat" ]; then
|
||||
API_URL="$BASE_URL/chat/completions"
|
||||
else
|
||||
API_URL="$BASE_URL/responses"
|
||||
fi
|
||||
|
||||
run_case() {
|
||||
local index="$1"
|
||||
local case_id="case-$index"
|
||||
local case_dir="$RUN_DIR/$case_id"
|
||||
local start_ns end_ns duration_ms rc http
|
||||
for path in response.sse http.code curl.rc curl.stderr duration.json result-owner.json; do
|
||||
[ ! -e "$case_dir/$path" ] || return 91
|
||||
done
|
||||
# Each timestamp is collected by a separate Python process. On macOS,
|
||||
# monotonic clock epochs are not guaranteed to be comparable across those
|
||||
# processes, so use the shared wall-clock epoch for the elapsed-time field.
|
||||
start_ns="$(python3 -c 'import time; print(time.time_ns())')"
|
||||
set +e
|
||||
http="$(curl "${CURL_COMMON[@]}" ${CURL_RESOLVE[@]+"${CURL_RESOLVE[@]}"} --max-time "$CALLER_TIMEOUT_SECONDS" \
|
||||
--no-buffer -sS -o "$case_dir/response.sse" -w '%{http_code}' \
|
||||
-H "@$AUTH_HEADER_FILE" -H 'Content-Type: application/json' \
|
||||
--data-binary "@$case_dir/request.json" "$API_URL" 2>"$case_dir/curl.stderr")"
|
||||
rc=$?
|
||||
set -e
|
||||
end_ns="$(python3 -c 'import time; print(time.time_ns())')"
|
||||
duration_ms=$(((end_ns - start_ns) / 1000000))
|
||||
printf '%s\n' "$http" >"$case_dir/http.code"
|
||||
printf '%s\n' "$rc" >"$case_dir/curl.rc"
|
||||
printf '{"duration_ms":%s}\n' "$duration_ms" >"$case_dir/duration.json"
|
||||
python3 - "$case_dir/result-owner.json" "$RUN_ID" "$case_id" "$case_dir/response.sse" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
owner, run_id, case_id, result = sys.argv[1:]
|
||||
pathlib.Path(owner).write_text(json.dumps({
|
||||
"run_id": run_id,
|
||||
"case_id": case_id,
|
||||
"result_sha256": hashlib.sha256(pathlib.Path(result).read_bytes()).hexdigest(),
|
||||
}), encoding="utf-8")
|
||||
PY
|
||||
}
|
||||
|
||||
fetch_status() {
|
||||
local output="$1"
|
||||
curl "${CURL_COMMON[@]}" ${CURL_RESOLVE[@]+"${CURL_RESOLVE[@]}"} --max-time 10 -fsS "$STATUS_URL" >"$output"
|
||||
}
|
||||
|
||||
append_selected_observation() {
|
||||
local snapshot="$1"
|
||||
python3 - "$snapshot" "$PROVIDER" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
doc = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||
provider_id = sys.argv[2]
|
||||
matches = []
|
||||
for node in doc.get("nodes") or []:
|
||||
if node.get("connected") is not True:
|
||||
continue
|
||||
for snapshot in node.get("provider_snapshots") or []:
|
||||
if (snapshot.get("id") or snapshot.get("adapter")) == provider_id:
|
||||
matches.append(snapshot)
|
||||
if len(matches) != 1:
|
||||
raise SystemExit("selected provider observation is not unique")
|
||||
s = matches[0]
|
||||
print(json.dumps({
|
||||
"capacity": int(s.get("capacity") or 0),
|
||||
"in_flight": int(s.get("in_flight") or 0),
|
||||
"queued": int(s.get("queued") or 0),
|
||||
"health": str(s.get("health") or ""),
|
||||
"status": str(s.get("status") or ""),
|
||||
}, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
}
|
||||
|
||||
pids=()
|
||||
index=1
|
||||
while [ "$index" -le "$REQUEST_COUNT" ]; do
|
||||
run_case "$index" &
|
||||
pids+=("$!")
|
||||
index=$((index + 1))
|
||||
done
|
||||
|
||||
sample=0
|
||||
while :; do
|
||||
alive=0
|
||||
for pid in "${pids[@]}"; do
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
alive=1
|
||||
fi
|
||||
done
|
||||
snapshot="$RUN_DIR/status-$sample.json"
|
||||
if fetch_status "$snapshot"; then
|
||||
append_selected_observation "$snapshot" >>"$RUN_DIR/status-observations.ndjson"
|
||||
else
|
||||
printf 'status_fetch_failed sample=%s\n' "$sample" >>"$RUN_DIR/status-errors.log"
|
||||
fi
|
||||
sample=$((sample + 1))
|
||||
[ "$alive" -eq 1 ] || break
|
||||
sleep "$POLL_INTERVAL_SECONDS"
|
||||
done
|
||||
for pid in "${pids[@]}"; do
|
||||
if wait "$pid"; then
|
||||
:
|
||||
else
|
||||
:
|
||||
fi
|
||||
done
|
||||
cleanup_live_secret
|
||||
|
||||
recovery_deadline=$((SECONDS + RECOVERY_TIMEOUT_SECONDS))
|
||||
while :; do
|
||||
snapshot="$RUN_DIR/status-recovery-$sample.json"
|
||||
if fetch_status "$snapshot"; then
|
||||
observation="$(append_selected_observation "$snapshot")"
|
||||
printf '%s\n' "$observation" >>"$RUN_DIR/status-observations.ndjson"
|
||||
if python3 - "$observation" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
item = json.loads(sys.argv[1])
|
||||
raise SystemExit(0 if item["in_flight"] == 0 and item["queued"] == 0 and item["health"] in {"healthy", "available"} and item["status"] == "available" else 1)
|
||||
PY
|
||||
then
|
||||
break
|
||||
fi
|
||||
fi
|
||||
[ "$SECONDS" -lt "$recovery_deadline" ] || die "selected provider did not recover before the bounded deadline"
|
||||
sample=$((sample + 1))
|
||||
sleep "$POLL_INTERVAL_SECONDS"
|
||||
done
|
||||
|
||||
validate_provenance "$RUN_DIR" "$RUN_DIR/manifest.json" >"$RUN_DIR/provenance-summary.json"
|
||||
index=1
|
||||
while [ "$index" -le "$REQUEST_COUNT" ]; do
|
||||
parse_terminal "$ENDPOINT" "$RUN_DIR/case-$index/response.sse" >"$RUN_DIR/case-$index/terminal-summary.json"
|
||||
index=$((index + 1))
|
||||
done
|
||||
|
||||
python3 - "$RUN_DIR" "$SCRIPT_SHA256" "$ROUTE_INFO" "$REQUEST_INFO" "$ELIGIBLE_CAPACITY" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
root = pathlib.Path(sys.argv[1])
|
||||
script_sha = sys.argv[2]
|
||||
route = json.loads(sys.argv[3])
|
||||
shape = json.loads(sys.argv[4])
|
||||
capacity = int(sys.argv[5])
|
||||
manifest = json.loads((root / "manifest.json").read_text(encoding="utf-8"))
|
||||
observations = [json.loads(line) for line in (root / "status-observations.ndjson").read_text(encoding="utf-8").splitlines() if line]
|
||||
if not observations:
|
||||
raise SystemExit("no selected-provider observations were captured")
|
||||
if any(item["capacity"] != capacity for item in observations):
|
||||
raise SystemExit("selected provider capacity changed during the invocation")
|
||||
peak = max(item["in_flight"] for item in observations)
|
||||
max_queue = max(item["queued"] for item in observations)
|
||||
if peak != capacity:
|
||||
raise SystemExit(f"selected provider peak={peak}, want capacity={capacity}")
|
||||
if max_queue < 1:
|
||||
raise SystemExit("selected provider queue was not observed")
|
||||
if any(item["in_flight"] > capacity for item in observations):
|
||||
raise SystemExit("selected provider exceeded eligible capacity")
|
||||
final = observations[-1]
|
||||
if final["in_flight"] != 0 or final["queued"] != 0:
|
||||
raise SystemExit("selected provider final counters did not recover")
|
||||
|
||||
http_statuses = []
|
||||
durations = []
|
||||
done = finish = completed = errors = 0
|
||||
for case in manifest["cases"]:
|
||||
case_root = root / case["case_id"]
|
||||
rc = int((case_root / "curl.rc").read_text().strip())
|
||||
http = (case_root / "http.code").read_text().strip()
|
||||
if rc != 0 or http != "200":
|
||||
raise SystemExit(f"{case['case_id']} curl/http failed")
|
||||
http_statuses.append(int(http))
|
||||
durations.append(int(json.loads((case_root / "duration.json").read_text())["duration_ms"]))
|
||||
terminal = json.loads((case_root / "terminal-summary.json").read_text())
|
||||
done += terminal["done_count"]
|
||||
finish += terminal["finish_count"]
|
||||
completed += terminal["completed_count"]
|
||||
errors += terminal["error_count"]
|
||||
|
||||
summary = {
|
||||
"evidence_schema": "iop.openai_managed_capacity_smoke.v1",
|
||||
"run_id": manifest["run_id"],
|
||||
"script_sha256": script_sha,
|
||||
"route_alias": route["route_alias"],
|
||||
"selected_provider": route["selected_provider"],
|
||||
"endpoint": manifest["endpoint"],
|
||||
"request_count": len(manifest["cases"]),
|
||||
"request_runes": shape["request_runes"],
|
||||
"estimated_input_tokens": shape["estimated_input_tokens"],
|
||||
"context_class": shape["context_class"],
|
||||
"eligible_capacity": capacity,
|
||||
"http_200_count": sum(1 for value in http_statuses if value == 200),
|
||||
"duration_ms_min": min(durations),
|
||||
"duration_ms_max": max(durations),
|
||||
"done_count": done,
|
||||
"finish_count": finish,
|
||||
"completed_count": completed,
|
||||
"error_count": errors,
|
||||
"selected_peak_in_flight": peak,
|
||||
"selected_max_queued": max_queue,
|
||||
"selected_final_in_flight": final["in_flight"],
|
||||
"selected_final_queued": final["queued"],
|
||||
"provenance": "current-run-manifest",
|
||||
"outcome": "pass",
|
||||
}
|
||||
(root / "sanitized-summary.json").write_text(json.dumps(summary, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
print(json.dumps(summary, separators=(",", ":"), sort_keys=True))
|
||||
PY
|
||||
log "PASS evidence_dir=$RUN_DIR"
|
||||
Loading…
Reference in a new issue