sync: backpressure cancel code review cleanup and test fixes
This commit is contained in:
parent
893aad6edb
commit
3348b64284
8 changed files with 1080 additions and 159 deletions
|
|
@ -0,0 +1,208 @@
|
|||
<!-- task=m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel plan=0 tag=FLOWCTRL -->
|
||||
|
||||
# Code Review Reference - FLOWCTRL
|
||||
|
||||
> **[IMPLEMENTING AGENT - READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `구현 체크리스트`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked by a selected Milestone `구현 잠금 > 결정 필요` item, fill `사용자 리뷰 요청` with linked evidence and stop with active files in place; code-review decides whether to write `USER_REVIEW.md`. Environment/secret/service blockers, generic scope changes, repeated failures, and evidence gaps that a follow-up agent can close are normal follow-up issues, not user-review blockers by themselves.
|
||||
> Do not ask the user directly, present choices in chat, or call `request_user_input` during implementation; record only Milestone lock decisions in `사용자 리뷰 요청` and stop for code-review.
|
||||
> Finalization (`코드리뷰 결과`, log rename, `complete.log`, archive moves, `코드리뷰 전용 체크리스트`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## 개요
|
||||
|
||||
date=2026-07-08
|
||||
task=m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel, plan=0, tag=FLOWCTRL
|
||||
|
||||
## Roadmap Targets
|
||||
|
||||
- Milestone: `agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md`
|
||||
- Milestone link: [Milestone 문서](agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md)
|
||||
- Task ids:
|
||||
- `backpressure-cancel`: tunnel path는 기존 socket 위에서 ordered, lossless, backpressure-aware stream을 유지하고 cancel/control frame을 지연시키지 않는다.
|
||||
- Completion mode: check-on-pass
|
||||
|
||||
## 이 파일을 읽는 리뷰 에이전트에게
|
||||
|
||||
> **[REVIEW AGENT ONLY]** 아래 종결 절차는 코드리뷰 에이전트 전용이다. 구현 에이전트는 이 섹션을 실행하지 않는다.
|
||||
|
||||
각 항목의 구현을 실제 소스 파일과 대조하고, `검증 결과` 섹션의 출력이 코드와 일치하는지 확인하세요.
|
||||
리뷰 완료는 아래 순서까지 끝난 상태를 의미합니다.
|
||||
|
||||
1. 판정을 append한다.
|
||||
2. `CODE_REVIEW-local-G08.md` -> `code_review_local_G08_N.log`, `PLAN-local-G08.md` -> `plan_local_G08_M.log`로 아카이브한다.
|
||||
3. PASS이면 `complete.log` 작성 후 active task 디렉터리를 `agent-task/archive/YYYY/MM/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`로 이동한다. WARN/FAIL이면 user-review gate를 확인한 뒤 다음 active plan/review 파일 또는 `USER_REVIEW.md`를 작성한다.
|
||||
4. PASS이고 task group이 `m-<milestone-slug>`이면 완료 이벤트 메타데이터를 보고한다. roadmap 상태 체크와 `update-roadmap` 호출은 런타임 책임이다.
|
||||
5. 적용 가능한 `코드리뷰 전용 체크리스트` 항목을 최종 `.log` 위치에서 체크한 뒤 보고한다.
|
||||
|
||||
---
|
||||
|
||||
## 구현 항목별 완료 여부
|
||||
|
||||
| 항목 | 완료 여부 |
|
||||
|------|---------|
|
||||
| [FLOWCTRL-1] Edge HTTP Tunnel Cancel And Ordering | [x] |
|
||||
| [FLOWCTRL-2] Service Tunnel Backpressure | [x] |
|
||||
| [FLOWCTRL-3] Node Provider Cancellation | [x] |
|
||||
|
||||
## 구현 체크리스트
|
||||
|
||||
- [x] `writeProviderTunnelResponse`, `writeProviderTunnelSidebandStream`, `writeProviderTunnelSidebandResponse`가 caller disconnect, write failure, timeout에서 exactly-once cancel/close를 수행하는지 보강한다.
|
||||
- [x] `providerTunnelRouter`와 `openProviderTunnel`의 full-buffer/backpressure와 terminal release 동작을 focused service test로 검증한다.
|
||||
- [x] `openai_compat`와 `vllm` tunnel provider cancellation/body error tests가 ordered ERROR/END semantics와 upstream request cancellation을 충분히 검증하도록 보강한다.
|
||||
- [x] slow client/write failure/provider error fixture에서 chunk drop/reorder가 없고 cancel/control frame이 지연되지 않는지 검증한다.
|
||||
- [x] `go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service`를 통과시킨다.
|
||||
- [x] `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm`를 통과시킨다.
|
||||
- [x] `go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service`를 통과시킨다.
|
||||
- [x] `git diff --check`를 통과시킨다.
|
||||
- [x] CODE_REVIEW-*-G??.md의 구현 에이전트 소유 섹션을 실제 구현 내용과 검증 출력으로 채운다. 이 항목이 완료되기 전에는 구현이 완료된 것이 아니다.
|
||||
|
||||
## 코드리뷰 전용 체크리스트
|
||||
|
||||
> **[REVIEW AGENT ONLY]** 이 체크리스트는 코드리뷰 에이전트만 사용한다.
|
||||
> 구현 에이전트는 이 섹션을 수정하거나 체크하지 않는다.
|
||||
|
||||
- [x] `코드리뷰 결과`에 `PASS`, `WARN`, `FAIL` 중 하나의 판정을 append한다.
|
||||
- [x] 판정과 `차원별 평가`, Required/Suggested/Nit 분류가 서로 일치한다.
|
||||
- [x] active `CODE_REVIEW-*-G??.md`를 `code_review_local_G08_N.log`로 아카이브한다.
|
||||
- [x] active `PLAN-*-G??.md`를 `plan_local_G08_M.log`로 아카이브한다.
|
||||
- [x] `.gitignore`의 Agent-Ops 관리 block이 `agent-task/**/*.md`와 `agent-task/**/*.log`를 unignore하고 `agent-roadmap/current.md`를 ignore하는지 확인한다.
|
||||
- [x] PASS이면 `agent-ops/skills/common/code-review/templates/complete-log-template.md` 기준으로 `complete.log`를 작성하고 active `.md` 파일을 남기지 않는다.
|
||||
- [x] PASS이면 active task 디렉터리 `agent-task/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`를 `agent-task/archive/YYYY/MM/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`로 이동하고 최종 archive 경로에서 이 체크리스트를 갱신한다.
|
||||
- [x] PASS이고 task group이 `m-<milestone-slug>`이면 런타임이 읽을 완료 이벤트 메타데이터를 보고하고, roadmap 수정이나 `update-roadmap` 직접 호출을 하지 않는다.
|
||||
- [x] PASS split 작업이면 이동 후 빈 active parent `agent-task/m-openai-compatible-raw-tunnel-sideband-passthrough/`를 제거하거나, 남은 sibling/file이 있어 유지했다고 확인한다.
|
||||
- [ ] WARN/FAIL이고 user-review gate가 트리거되지 않았으면 다음 active `PLAN-{build_lane}-GNN.md`와 `CODE_REVIEW-{review_lane}-GNN.md`를 작성하고 `complete.log`를 작성하지 않는다.
|
||||
- [ ] USER_REVIEW이면 `agent-ops/skills/common/code-review/templates/user-review-template.md` 기준으로 `USER_REVIEW.md`를 작성하고 active `PLAN-*.md`, `CODE_REVIEW-*.md`, `complete.log`를 남기지 않는다.
|
||||
- [ ] USER_REVIEW가 연결된 Milestone 결정으로 완료/PASS 해소되면 `USER_REVIEW.md`를 해소 상태로 갱신하고 `complete.log`를 작성한 뒤 task directory를 archive로 이동한다.
|
||||
|
||||
## 계획 대비 변경 사항
|
||||
|
||||
- `apps/edge/internal/openai/stream.go`와 `apps/edge/internal/service/run_dispatch.go`는 수정하지 않았다. 계획은 "필요 시" exactly-once close/cancel helper와 testability helper 도입을 허용했는데, 분석 결과 pure passthrough와 streaming sideband writer의 context cancel / wait timeout / body write failure 경로는 모두 상호 배타적으로 즉시 return하여 이미 cancel이 정확히 1회만 전파되고, non-streaming sideband writer는 provider body를 끝까지 버퍼링한 뒤 envelope를 쓰는 구조라 중간 body write failure 경로 대신 context cancel / wait timeout에서 cancel이 전파된다. `openProviderTunnel`의 slot release도 `releaseOnce(sync.Once)` + `inflightByRun` delete로 END/ERROR/Close 조합에서 구조적으로 exactly-once였다. 따라서 프로덕션 변경 없이 이 동작을 테스트로 고정했다.
|
||||
- `providerTunnelRouter` backpressure 테스트는 `run_dispatch.go`에 helper를 추가하는 대신 기존 `subscribe(tunnelID, buffer)`의 buffer 파라미터를 작은 값으로 직접 사용해 full-buffer blocking을 결정적으로 재현했다 (internal test 패키지라 접근 가능).
|
||||
- 계획의 수정 파일 요약에 있던 `apps/edge/internal/openai/stream.go`, `apps/edge/internal/service/run_dispatch.go`는 위 사유로 실제 diff에 포함되지 않는다. 테스트 파일 4개만 변경했다.
|
||||
|
||||
## 주요 설계 결정
|
||||
|
||||
- **Edge HTTP write failure fixture**: `httptest.ResponseRecorder`는 Write 실패를 만들 수 없어 `flakyResponseWriter` test double을 `server_test.go`에 추가했다. `failAfterWrites` 회 성공 후 모든 Write가 실패하고, `writeDelay`로 slow client를 에뮬레이트한다. Write 호출 단위 카운트라 sideband route observation write(1회)와 provider body write를 구분해 실패 지점을 정확히 지정할 수 있다.
|
||||
- **Tunnel wait timeout fixture**: `fakeRunService.tunnelWaitTimeout`/`fakeTunnelHandle.waitTimeout` 필드를 추가해 기본 5s wait timeout을 테스트에서 50ms로 줄였다. 기존 fixture 사용처는 zero-value로 기존 동작(5s)을 유지한다.
|
||||
- **Exactly-once cancel 검증 방식**: write failure 후 END 포함 추가 frame이 채널에 남아 있는 fixture로 handler를 돌리고, `cancelCallsSnapshot()`이 정확히 1개의 CancelRun만 기록하며 실패 이후 body가 더 이상 쓰이지 않음을 단언했다. provider ERROR terminal에는 CancelRun 0회(불필요한 cancel 미발생)를 함께 고정했다.
|
||||
- **Service backpressure 검증 방식**: buffer 2 subscriber에 frame 6개를 route하는 producer goroutine이 100ms 안에 끝나지 않음(=drop이 아니라 block)을 확인한 뒤, drain 시 sequence 0..5가 순서대로 전부 도착하고 producer가 해제되는 것을 단언했다. `unsubscribe`(handle Close 경로)가 blocked route를 해제하는 케이스와, capacity 1 provider slot이 END+Close 중복 종료 후에도 정확히 1회 release되어 다음 tunnel이 즉시 admit되는 케이스를 별도 테스트로 고정했다.
|
||||
- **Node adapter ordered semantics helper**: 두 adapter 테스트 패키지에 동일한 `assertOrderedTunnelFrames` helper를 추가해 contiguous sequence(0..N, drop/reorder 없음)와 "terminal END/ERROR는 정확히 1개이며 마지막 frame"을 공통 단언했다. 기존 cancel 테스트에는 terminal ERROR 메시지가 `context canceled`를 포함하는지와 mid frame이 모두 BODY인지 단언을 추가했다.
|
||||
- **Edge write failure -> upstream cancel end-to-end (Node 측)**: `failingTunnelSink`(N frame 후 emit 실패)를 추가해 Edge socket write 실패 시 adapter가 emit error를 즉시 반환하고 추가 frame 없이 relay를 멈추며, provider handler가 upstream request teardown을 관측하는 것을 `openai_compat`/`vllm` 양쪽에서 검증했다. Edge 측 write failure -> CancelRun은 FLOWCTRL-1 테스트가 담당하여 두 조각이 S09 end-to-end를 구성한다.
|
||||
- adapter 테스트 fixture는 기존 관례대로 패키지별 중복 정의를 유지했다 (두 테스트 패키지는 이미 `fakeTunnelSink`를 각자 소유).
|
||||
|
||||
## 사용자 리뷰 요청
|
||||
|
||||
_기본값은 `없음`이다. 구현 중 새 결정이 필요해 보여도 직접 질문하거나 선택지를 제시하거나 `request_user_input`을 호출하지 않는다. 이 섹션은 선택된 Milestone `구현 잠금 > 결정 필요` 항목이 실구현을 차단할 때만 채운다. 외부 환경/secret/서비스 준비, 검증 증거 공백, 반복 실패, 일반 범위 조정은 사용자 리뷰 요청이 아니며 `검증 결과`, `계획 대비 변경 사항`, 또는 code-review의 일반 follow-up plan으로 처리한다._
|
||||
|
||||
- 상태: 없음
|
||||
- 사유 유형: 없음
|
||||
- 연결 대상: 없음
|
||||
- 결정 필요: 없음
|
||||
- 차단 근거: 없음
|
||||
- 실행한 검증/명령: 없음
|
||||
- 자동 후속 불가 이유: 없음
|
||||
- 재개 조건: 없음
|
||||
|
||||
## 리뷰어를 위한 체크포인트
|
||||
|
||||
- Slow/write-failure/provider-error tests가 S09 Evidence Map에 직접 대응하는지 확인한다.
|
||||
- Tunnel router full-buffer behavior가 drop이 아니라 block/release임을 검증하는지 확인한다.
|
||||
- CancelRun이 caller disconnect/write failure/timeout에서 중복 없이 전파되는지 확인한다.
|
||||
|
||||
## 검증 결과
|
||||
|
||||
### FLOWCTRL-1 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/openai -run 'TestChatCompletionsPassthrough.*(Cancel|WriteFailure|ProviderError|ByteIdentity)|TestChatCompletionsPassthroughSideband.*(Cancel|WriteFailure)' -v
|
||||
--- PASS: TestChatCompletionsPassthroughNonStreamingByteIdentity (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughStreamingByteIdentity (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughProviderErrorStatusRelayed (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughContextCancelSendsCancelRun (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughWriteFailureSendsCancelRunOnce (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughWaitTimeoutSendsCancelRun (0.05s)
|
||||
--- PASS: TestChatCompletionsPassthroughProviderErrorAfterStartStopsBodyWrites (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughSlowClientByteIdentity (0.13s)
|
||||
--- PASS: TestChatCompletionsPassthroughSidebandStreamWriteFailureSendsCancelRunOnce (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughSidebandStreamContextCancelSendsCancelRun (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughSidebandNonStreamingContextCancelSendsCancelRun (0.00s)
|
||||
--- PASS: TestChatCompletionsPassthroughSidebandNonStreamingTimeoutSendsCancelRun (0.05s)
|
||||
ok iop/apps/edge/internal/openai 0.251s
|
||||
```
|
||||
|
||||
### FLOWCTRL-2 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/service -run 'TestProviderTunnel.*(Backpressure|Release|Close)' -v
|
||||
--- PASS: TestProviderTunnelRouterBackpressureBlocksWithoutDrop (0.10s)
|
||||
--- PASS: TestProviderTunnelRouterCloseUnblocksBlockedRoute (0.10s)
|
||||
--- PASS: TestProviderTunnelReleaseExactlyOnceAcrossEndAndClose (0.01s)
|
||||
ok iop/apps/edge/internal/service 0.222s
|
||||
```
|
||||
|
||||
### FLOWCTRL-3 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm -run 'Test(OpenAICompat|Vllm)TunnelProvider' -v
|
||||
--- PASS: TestOpenAICompatTunnelProvider (0.00s)
|
||||
--- PASS: TestOpenAICompatTunnelProvider_Cancel (0.10s)
|
||||
--- PASS: TestOpenAICompatTunnelProvider_BodyReadError (0.00s)
|
||||
--- PASS: TestOpenAICompatTunnelProvider_SinkWriteFailureStopsUpstreamRelay (0.01s)
|
||||
ok iop/apps/node/internal/adapters/openai_compat 0.120s
|
||||
--- PASS: TestVllmTunnelProvider (0.00s)
|
||||
--- PASS: TestVllmTunnelProvider_Cancel (0.10s)
|
||||
--- PASS: TestVllmTunnelProvider_BodyReadError (0.00s)
|
||||
--- PASS: TestVllmTunnelProvider_SinkWriteFailureStopsUpstreamRelay (0.01s)
|
||||
--- PASS: TestVllmTunnelProvider_GemmaContentToolCallRawBytes (0.00s)
|
||||
ok iop/apps/node/internal/adapters/vllm 0.120s
|
||||
```
|
||||
|
||||
### 최종 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service
|
||||
ok iop/apps/edge/internal/openai 1.827s
|
||||
ok iop/apps/edge/internal/service 0.910s
|
||||
|
||||
$ go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm
|
||||
ok iop/apps/node/internal/adapters/openai_compat 0.127s
|
||||
ok iop/apps/node/internal/adapters/vllm 0.121s
|
||||
|
||||
$ go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service
|
||||
ok iop/apps/edge/internal/openai 2.983s
|
||||
ok iop/apps/edge/internal/service 1.924s
|
||||
|
||||
$ git diff --check
|
||||
(출력 없음, exit 0)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT - BEFORE SAVING] Have you filled in every implementation-owned section: completion table, implementation checklist, changes from plan, design decisions, and verification output?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
||||
## 코드리뷰 결과
|
||||
|
||||
- 리뷰 일시: 2026-07-08
|
||||
- 종합 판정: PASS
|
||||
- 차원별 평가:
|
||||
- Correctness: Pass
|
||||
- Completeness: Pass
|
||||
- Test coverage: Pass
|
||||
- API contract: Pass
|
||||
- Code quality: Pass
|
||||
- Implementation deviation: Pass
|
||||
- Verification trust: Pass
|
||||
- Spec conformance: Pass
|
||||
- 발견된 문제: 없음
|
||||
- 검증 재실행:
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run 'TestChatCompletionsPassthrough.*(Cancel|WriteFailure|ProviderError|ByteIdentity)|TestChatCompletionsPassthroughSideband.*(Cancel|WriteFailure)' -v`: PASS
|
||||
- `go test -count=1 ./apps/edge/internal/service -run 'TestProviderTunnel.*(Backpressure|Release|Close)' -v`: PASS
|
||||
- `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm -run 'Test(OpenAICompat|Vllm)TunnelProvider' -v`: PASS
|
||||
- `go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service`: PASS
|
||||
- `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm`: PASS
|
||||
- `go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service`: PASS
|
||||
- `git diff --check`: PASS
|
||||
- SDD Evidence Map:
|
||||
- S09 `backpressure-cancel`: slow client byte identity, write failure cancel, provider error truncation, service router backpressure/release, adapter cancel/body-error/sink-failure tests가 Roadmap Target evidence를 충족한다.
|
||||
- 다음 단계: PASS 종결. active plan/review를 `.log`로 아카이브하고 `complete.log`를 작성한 뒤 task directory를 archive로 이동한다.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Complete - m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel
|
||||
|
||||
## 완료 일시
|
||||
|
||||
2026-07-08
|
||||
|
||||
## 요약
|
||||
|
||||
FLOWCTRL plan 0은 1회 리뷰 루프에서 PASS로 종결됐다. S09 `backpressure-cancel` evidence로 Edge HTTP tunnel cancel/order, service tunnel backpressure/release, Node provider cancellation/body-error/sink-failure 테스트가 추가 및 검증됐다.
|
||||
|
||||
## 루프 이력
|
||||
|
||||
| Plan | Review | Verdict | 메모 |
|
||||
|------|--------|---------|------|
|
||||
| `plan_local_G08_0.log` | `code_review_local_G08_0.log` | PASS | Required/Suggested 없음. SDD S09 Evidence Map 충족 확인. |
|
||||
|
||||
## 구현/정리 내용
|
||||
|
||||
- `apps/edge/internal/openai/server_test.go`에 passthrough/sideband tunnel write failure, timeout, provider error, slow client, context cancel fixture를 추가했다.
|
||||
- `apps/edge/internal/service/run_dispatch_internal_test.go`에 provider tunnel router full-buffer backpressure, close unblock, END+Close release exactly-once 테스트를 추가했다.
|
||||
- `apps/node/internal/adapters/openai_compat/openai_compat_test.go`와 `apps/node/internal/adapters/vllm/vllm_test.go`에 ordered terminal semantics, cancellation error, body read error, sink write failure upstream teardown 검증을 보강했다.
|
||||
|
||||
## 최종 검증
|
||||
|
||||
- `go test -count=1 ./apps/edge/internal/openai -run 'TestChatCompletionsPassthrough.*(Cancel|WriteFailure|ProviderError|ByteIdentity)|TestChatCompletionsPassthroughSideband.*(Cancel|WriteFailure)' -v` - PASS; targeted Edge OpenAI tunnel tests passed.
|
||||
- `go test -count=1 ./apps/edge/internal/service -run 'TestProviderTunnel.*(Backpressure|Release|Close)' -v` - PASS; targeted service tunnel tests passed.
|
||||
- `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm -run 'Test(OpenAICompat|Vllm)TunnelProvider' -v` - PASS; targeted Node adapter tunnel tests passed.
|
||||
- `go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service` - PASS; `ok iop/apps/edge/internal/openai 1.821s`, `ok iop/apps/edge/internal/service 0.902s`.
|
||||
- `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm` - PASS; `ok iop/apps/node/internal/adapters/openai_compat 0.123s`, `ok iop/apps/node/internal/adapters/vllm 0.123s`.
|
||||
- `go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service` - PASS; `ok iop/apps/edge/internal/openai 2.966s`, `ok iop/apps/edge/internal/service 1.945s`.
|
||||
- `git diff --check` - PASS; no output.
|
||||
|
||||
## Roadmap Completion
|
||||
|
||||
- Milestone: `agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md`
|
||||
- Milestone link: [Milestone 문서](agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md)
|
||||
- Completed task ids:
|
||||
- `backpressure-cancel`: PASS; evidence=`agent-task/archive/2026/07/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/plan_local_G08_0.log`, `agent-task/archive/2026/07/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/code_review_local_G08_0.log`; verification=`go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service`, `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm`, `go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service`, `git diff --check`
|
||||
- Not completed task ids: 없음
|
||||
|
||||
## 잔여 Nit
|
||||
|
||||
- 없음
|
||||
|
||||
## 후속 작업
|
||||
|
||||
- 없음
|
||||
|
|
@ -1,138 +0,0 @@
|
|||
<!-- task=m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel plan=0 tag=FLOWCTRL -->
|
||||
|
||||
# Code Review Reference - FLOWCTRL
|
||||
|
||||
> **[IMPLEMENTING AGENT - READ FIRST] Filling in this file is the mandatory final step of implementation.**
|
||||
> The task is NOT complete until every implementation-owned section below is filled in.
|
||||
> Complete the `구현 체크리스트`; the final checklist item is mandatory before saving.
|
||||
> Fill implementation-owned sections, then stop with active files in place and report ready for review.
|
||||
> If implementation is blocked by a selected Milestone `구현 잠금 > 결정 필요` item, fill `사용자 리뷰 요청` with linked evidence and stop with active files in place; code-review decides whether to write `USER_REVIEW.md`. Environment/secret/service blockers, generic scope changes, repeated failures, and evidence gaps that a follow-up agent can close are normal follow-up issues, not user-review blockers by themselves.
|
||||
> Do not ask the user directly, present choices in chat, or call `request_user_input` during implementation; record only Milestone lock decisions in `사용자 리뷰 요청` and stop for code-review.
|
||||
> Finalization (`코드리뷰 결과`, log rename, `complete.log`, archive moves, `코드리뷰 전용 체크리스트`) is review-agent-only, even after compaction/resume.
|
||||
> Follow the ownership table at the bottom of this file for which sections you own.
|
||||
|
||||
## 개요
|
||||
|
||||
date=2026-07-08
|
||||
task=m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel, plan=0, tag=FLOWCTRL
|
||||
|
||||
## Roadmap Targets
|
||||
|
||||
- Milestone: `agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md`
|
||||
- Milestone link: [Milestone 문서](agent-roadmap/phase/routing-policy-model-orchestration/milestones/openai-compatible-raw-tunnel-sideband-passthrough.md)
|
||||
- Task ids:
|
||||
- `backpressure-cancel`: tunnel path는 기존 socket 위에서 ordered, lossless, backpressure-aware stream을 유지하고 cancel/control frame을 지연시키지 않는다.
|
||||
- Completion mode: check-on-pass
|
||||
|
||||
## 이 파일을 읽는 리뷰 에이전트에게
|
||||
|
||||
> **[REVIEW AGENT ONLY]** 아래 종결 절차는 코드리뷰 에이전트 전용이다. 구현 에이전트는 이 섹션을 실행하지 않는다.
|
||||
|
||||
각 항목의 구현을 실제 소스 파일과 대조하고, `검증 결과` 섹션의 출력이 코드와 일치하는지 확인하세요.
|
||||
리뷰 완료는 아래 순서까지 끝난 상태를 의미합니다.
|
||||
|
||||
1. 판정을 append한다.
|
||||
2. `CODE_REVIEW-local-G08.md` -> `code_review_local_G08_N.log`, `PLAN-local-G08.md` -> `plan_local_G08_M.log`로 아카이브한다.
|
||||
3. PASS이면 `complete.log` 작성 후 active task 디렉터리를 `agent-task/archive/YYYY/MM/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`로 이동한다. WARN/FAIL이면 user-review gate를 확인한 뒤 다음 active plan/review 파일 또는 `USER_REVIEW.md`를 작성한다.
|
||||
4. PASS이고 task group이 `m-<milestone-slug>`이면 완료 이벤트 메타데이터를 보고한다. roadmap 상태 체크와 `update-roadmap` 호출은 런타임 책임이다.
|
||||
5. 적용 가능한 `코드리뷰 전용 체크리스트` 항목을 최종 `.log` 위치에서 체크한 뒤 보고한다.
|
||||
|
||||
---
|
||||
|
||||
## 구현 항목별 완료 여부
|
||||
|
||||
| 항목 | 완료 여부 |
|
||||
|------|---------|
|
||||
| [FLOWCTRL-1] Edge HTTP Tunnel Cancel And Ordering | [ ] |
|
||||
| [FLOWCTRL-2] Service Tunnel Backpressure | [ ] |
|
||||
| [FLOWCTRL-3] Node Provider Cancellation | [ ] |
|
||||
|
||||
## 구현 체크리스트
|
||||
|
||||
- [ ] `writeProviderTunnelResponse`, `writeProviderTunnelSidebandStream`, `writeProviderTunnelSidebandResponse`가 caller disconnect, write failure, timeout에서 exactly-once cancel/close를 수행하는지 보강한다.
|
||||
- [ ] `providerTunnelRouter`와 `openProviderTunnel`의 full-buffer/backpressure와 terminal release 동작을 focused service test로 검증한다.
|
||||
- [ ] `openai_compat`와 `vllm` tunnel provider cancellation/body error tests가 ordered ERROR/END semantics와 upstream request cancellation을 충분히 검증하도록 보강한다.
|
||||
- [ ] slow client/write failure/provider error fixture에서 chunk drop/reorder가 없고 cancel/control frame이 지연되지 않는지 검증한다.
|
||||
- [ ] `go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service`를 통과시킨다.
|
||||
- [ ] `go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm`를 통과시킨다.
|
||||
- [ ] `go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service`를 통과시킨다.
|
||||
- [ ] `git diff --check`를 통과시킨다.
|
||||
- [ ] CODE_REVIEW-*-G??.md의 구현 에이전트 소유 섹션을 실제 구현 내용과 검증 출력으로 채운다. 이 항목이 완료되기 전에는 구현이 완료된 것이 아니다.
|
||||
|
||||
## 코드리뷰 전용 체크리스트
|
||||
|
||||
> **[REVIEW AGENT ONLY]** 이 체크리스트는 코드리뷰 에이전트만 사용한다.
|
||||
> 구현 에이전트는 이 섹션을 수정하거나 체크하지 않는다.
|
||||
|
||||
- [ ] `코드리뷰 결과`에 `PASS`, `WARN`, `FAIL` 중 하나의 판정을 append한다.
|
||||
- [ ] 판정과 `차원별 평가`, Required/Suggested/Nit 분류가 서로 일치한다.
|
||||
- [ ] active `CODE_REVIEW-*-G??.md`를 `code_review_local_G08_N.log`로 아카이브한다.
|
||||
- [ ] active `PLAN-*-G??.md`를 `plan_local_G08_M.log`로 아카이브한다.
|
||||
- [ ] `.gitignore`의 Agent-Ops 관리 block이 `agent-task/**/*.md`와 `agent-task/**/*.log`를 unignore하고 `agent-roadmap/current.md`를 ignore하는지 확인한다.
|
||||
- [ ] PASS이면 `agent-ops/skills/common/code-review/templates/complete-log-template.md` 기준으로 `complete.log`를 작성하고 active `.md` 파일을 남기지 않는다.
|
||||
- [ ] PASS이면 active task 디렉터리 `agent-task/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`를 `agent-task/archive/YYYY/MM/m-openai-compatible-raw-tunnel-sideband-passthrough/05_backpressure_cancel/`로 이동하고 최종 archive 경로에서 이 체크리스트를 갱신한다.
|
||||
- [ ] PASS이고 task group이 `m-<milestone-slug>`이면 런타임이 읽을 완료 이벤트 메타데이터를 보고하고, roadmap 수정이나 `update-roadmap` 직접 호출을 하지 않는다.
|
||||
- [ ] PASS split 작업이면 이동 후 빈 active parent `agent-task/m-openai-compatible-raw-tunnel-sideband-passthrough/`를 제거하거나, 남은 sibling/file이 있어 유지했다고 확인한다.
|
||||
- [ ] WARN/FAIL이고 user-review gate가 트리거되지 않았으면 다음 active `PLAN-{build_lane}-GNN.md`와 `CODE_REVIEW-{review_lane}-GNN.md`를 작성하고 `complete.log`를 작성하지 않는다.
|
||||
- [ ] USER_REVIEW이면 `agent-ops/skills/common/code-review/templates/user-review-template.md` 기준으로 `USER_REVIEW.md`를 작성하고 active `PLAN-*.md`, `CODE_REVIEW-*.md`, `complete.log`를 남기지 않는다.
|
||||
- [ ] USER_REVIEW가 연결된 Milestone 결정으로 완료/PASS 해소되면 `USER_REVIEW.md`를 해소 상태로 갱신하고 `complete.log`를 작성한 뒤 task directory를 archive로 이동한다.
|
||||
|
||||
## 계획 대비 변경 사항
|
||||
|
||||
_구현 에이전트가 계획과 다르게 구현한 부분을 이유와 함께 기록한다._
|
||||
|
||||
## 주요 설계 결정
|
||||
|
||||
_구현 에이전트가 주요 설계 결정 사항을 기록한다._
|
||||
|
||||
## 사용자 리뷰 요청
|
||||
|
||||
_기본값은 `없음`이다. 구현 중 새 결정이 필요해 보여도 직접 질문하거나 선택지를 제시하거나 `request_user_input`을 호출하지 않는다. 이 섹션은 선택된 Milestone `구현 잠금 > 결정 필요` 항목이 실구현을 차단할 때만 채운다. 외부 환경/secret/서비스 준비, 검증 증거 공백, 반복 실패, 일반 범위 조정은 사용자 리뷰 요청이 아니며 `검증 결과`, `계획 대비 변경 사항`, 또는 code-review의 일반 follow-up plan으로 처리한다._
|
||||
|
||||
- 상태: 없음
|
||||
- 사유 유형: 없음
|
||||
- 연결 대상: 없음
|
||||
- 결정 필요: 없음
|
||||
- 차단 근거: 없음
|
||||
- 실행한 검증/명령: 없음
|
||||
- 자동 후속 불가 이유: 없음
|
||||
- 재개 조건: 없음
|
||||
|
||||
## 리뷰어를 위한 체크포인트
|
||||
|
||||
- Slow/write-failure/provider-error tests가 S09 Evidence Map에 직접 대응하는지 확인한다.
|
||||
- Tunnel router full-buffer behavior가 drop이 아니라 block/release임을 검증하는지 확인한다.
|
||||
- CancelRun이 caller disconnect/write failure/timeout에서 중복 없이 전파되는지 확인한다.
|
||||
|
||||
## 검증 결과
|
||||
|
||||
_구현 에이전트가 각 중간 검증 및 최종 검증 명령 실행 후 출력을 여기에 붙여 넣는다._
|
||||
|
||||
### FLOWCTRL-1 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/openai -run 'TestChatCompletionsPassthrough.*(Cancel|WriteFailure|ProviderError|ByteIdentity)|TestChatCompletionsPassthroughSideband.*(Cancel|WriteFailure)'
|
||||
```
|
||||
|
||||
### FLOWCTRL-2 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/service -run 'TestProviderTunnel.*(Backpressure|Release|Close)'
|
||||
```
|
||||
|
||||
### FLOWCTRL-3 중간 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm -run 'Test(OpenAICompat|Vllm)TunnelProvider'
|
||||
```
|
||||
|
||||
### 최종 검증
|
||||
```bash
|
||||
$ go test -count=1 ./apps/edge/internal/openai ./apps/edge/internal/service
|
||||
$ go test -count=1 ./apps/node/internal/adapters/openai_compat ./apps/node/internal/adapters/vllm
|
||||
$ go test -count=1 -race ./apps/edge/internal/openai ./apps/edge/internal/service
|
||||
$ git diff --check
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
> **[IMPLEMENTING AGENT - BEFORE SAVING] Have you filled in every implementation-owned section: completion table, implementation checklist, changes from plan, design decisions, and verification output?**
|
||||
> If anything is blank, go back and fill it in before saving this file.
|
||||
> Leave review-agent-only sections unchanged.
|
||||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
|
|
@ -48,19 +49,28 @@ type fakeRunService struct {
|
|||
tunnelErr error
|
||||
tunnelReqs []edgeservice.SubmitProviderTunnelRequest
|
||||
tunnelBodies [][]byte
|
||||
// tunnelWaitTimeout overrides the tunnel handle wait timeout for
|
||||
// timeout-path fixtures; zero keeps the 5s default.
|
||||
tunnelWaitTimeout time.Duration
|
||||
}
|
||||
|
||||
type fakeTunnelHandle struct {
|
||||
dispatch edgeservice.RunDispatch
|
||||
frames chan *iop.ProviderTunnelFrame
|
||||
dispatch edgeservice.RunDispatch
|
||||
frames chan *iop.ProviderTunnelFrame
|
||||
waitTimeout time.Duration
|
||||
}
|
||||
|
||||
func (h *fakeTunnelHandle) Dispatch() edgeservice.RunDispatch { return h.dispatch }
|
||||
func (h *fakeTunnelHandle) Stream() edgeservice.ProviderTunnelStream {
|
||||
return edgeservice.ProviderTunnelStream{Frames: h.frames}
|
||||
}
|
||||
func (h *fakeTunnelHandle) Close() {}
|
||||
func (h *fakeTunnelHandle) WaitTimeout() time.Duration { return 5 * time.Second }
|
||||
func (h *fakeTunnelHandle) Close() {}
|
||||
func (h *fakeTunnelHandle) WaitTimeout() time.Duration {
|
||||
if h.waitTimeout > 0 {
|
||||
return h.waitTimeout
|
||||
}
|
||||
return 5 * time.Second
|
||||
}
|
||||
|
||||
func (s *fakeRunService) SubmitProviderTunnel(_ context.Context, req edgeservice.SubmitProviderTunnelRequest) (edgeservice.ProviderTunnelResult, error) {
|
||||
s.submitMu.Lock()
|
||||
|
|
@ -103,7 +113,8 @@ func (s *fakeRunService) SubmitProviderTunnel(_ context.Context, req edgeservice
|
|||
SessionID: req.SessionID,
|
||||
TimeoutSec: 5,
|
||||
},
|
||||
frames: frames,
|
||||
frames: frames,
|
||||
waitTimeout: s.tunnelWaitTimeout,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
|
@ -4828,6 +4839,362 @@ func TestChatCompletionsPassthroughContextCancelSendsCancelRun(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// flakyResponseWriter is a tunnel-path ResponseWriter double: it accepts
|
||||
// failAfterWrites successful Write calls, then fails every subsequent Write,
|
||||
// optionally sleeping writeDelay per call to emulate a slow client.
|
||||
type flakyResponseWriter struct {
|
||||
mu sync.Mutex
|
||||
header http.Header
|
||||
status int
|
||||
body bytes.Buffer
|
||||
failAfterWrites int // -1: never fail
|
||||
writeDelay time.Duration
|
||||
writes int
|
||||
failedWrites int
|
||||
}
|
||||
|
||||
func newFlakyResponseWriter(failAfterWrites int) *flakyResponseWriter {
|
||||
return &flakyResponseWriter{header: make(http.Header), failAfterWrites: failAfterWrites}
|
||||
}
|
||||
|
||||
func (w *flakyResponseWriter) Header() http.Header { return w.header }
|
||||
|
||||
func (w *flakyResponseWriter) WriteHeader(code int) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.status == 0 {
|
||||
w.status = code
|
||||
}
|
||||
}
|
||||
|
||||
func (w *flakyResponseWriter) Write(p []byte) (int, error) {
|
||||
if w.writeDelay > 0 {
|
||||
time.Sleep(w.writeDelay)
|
||||
}
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
if w.status == 0 {
|
||||
w.status = http.StatusOK
|
||||
}
|
||||
if w.failAfterWrites >= 0 && w.writes >= w.failAfterWrites {
|
||||
w.failedWrites++
|
||||
return 0, errors.New("client connection write failed")
|
||||
}
|
||||
w.writes++
|
||||
w.body.Write(p)
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (w *flakyResponseWriter) Flush() {}
|
||||
|
||||
func (w *flakyResponseWriter) snapshot() (status int, body string, failedWrites int) {
|
||||
w.mu.Lock()
|
||||
defer w.mu.Unlock()
|
||||
return w.status, w.body.String(), w.failedWrites
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughWriteFailureSendsCancelRunOnce verifies S09
|
||||
// for a caller write failure after response start: exactly one CancelRun is
|
||||
// propagated to the Node, the ordered prefix written before the failure is
|
||||
// preserved, and no tunnel bytes are written after the failed write even
|
||||
// though more frames (including END) are available.
|
||||
func TestChatCompletionsPassthroughWriteFailureSendsCancelRunOnce(t *testing.T) {
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 5)
|
||||
frames <- &iop.ProviderTunnelFrame{
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START,
|
||||
StatusCode: 200,
|
||||
Headers: map[string]string{"Content-Type": "text/event-stream"},
|
||||
}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: chunk-1\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: chunk-2\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: chunk-3\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true}
|
||||
close(frames)
|
||||
srv, fake := chatSidebandServer(frames)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"stream":true,
|
||||
"messages":[{"role":"user","content":"hello"}]
|
||||
}`))
|
||||
w := newFlakyResponseWriter(1) // first body write succeeds, second fails
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
status, body, failedWrites := w.snapshot()
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("status: got %d body=%s", status, body)
|
||||
}
|
||||
if body != "data: chunk-1\n\n" {
|
||||
t.Fatalf("body after write failure must stay the ordered prefix:\n got: %q\nwant: %q", body, "data: chunk-1\n\n")
|
||||
}
|
||||
if failedWrites == 0 {
|
||||
t.Fatal("fixture did not exercise the write failure path")
|
||||
}
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call on write failure, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
if calls[0].RunID != "run-tunnel" || calls[0].NodeRef != "node-1" {
|
||||
t.Fatalf("unexpected cancel request: %+v", calls[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughWaitTimeoutSendsCancelRun verifies the tunnel
|
||||
// wait timeout before response start returns a 502 and propagates exactly one
|
||||
// CancelRun to the Node.
|
||||
func TestChatCompletionsPassthroughWaitTimeoutSendsCancelRun(t *testing.T) {
|
||||
fake := &fakeRunService{
|
||||
tunnelFrames: make(chan *iop.ProviderTunnelFrame),
|
||||
tunnelWaitTimeout: 50 * time.Millisecond,
|
||||
}
|
||||
srv := NewServer(config.EdgeOpenAIConf{}, fake, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{
|
||||
{ID: "pool-model", Providers: map[string]string{"prov-1": "served-model"}},
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"messages":[{"role":"user","content":"hello"}]
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
if w.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status: got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "run timed out") {
|
||||
t.Fatalf("timeout error not surfaced: %s", w.Body.String())
|
||||
}
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call on wait timeout, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
if calls[0].RunID != "run-tunnel" {
|
||||
t.Fatalf("unexpected cancel request: %+v", calls[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughProviderErrorAfterStartStopsBodyWrites
|
||||
// verifies an ERROR frame after response start truncates the stream: bytes
|
||||
// written before the error are preserved in order, later frames are never
|
||||
// written, and no spurious CancelRun is sent for a provider-terminal failure.
|
||||
func TestChatCompletionsPassthroughProviderErrorAfterStartStopsBodyWrites(t *testing.T) {
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 4)
|
||||
frames <- &iop.ProviderTunnelFrame{
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START,
|
||||
StatusCode: 200,
|
||||
Headers: map[string]string{"Content-Type": "text/event-stream"},
|
||||
}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: before-error\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_ERROR, Error: "provider stream failed"}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: after-error-must-not-write\n\n")}
|
||||
close(frames)
|
||||
srv, fake := chatSidebandServer(frames)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"stream":true,
|
||||
"messages":[{"role":"user","content":"hello"}]
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("status already committed before the error must stay 200, got %d", w.Code)
|
||||
}
|
||||
if got := w.Body.String(); got != "data: before-error\n\n" {
|
||||
t.Fatalf("provider error must truncate after the ordered prefix:\n got: %q\nwant: %q", got, "data: before-error\n\n")
|
||||
}
|
||||
if calls := fake.cancelCallsSnapshot(); len(calls) != 0 {
|
||||
t.Fatalf("provider-terminal error must not trigger CancelRun, got %+v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughSlowClientByteIdentity verifies a slow caller
|
||||
// applies backpressure without corrupting the relay: every body frame reaches
|
||||
// the caller exactly once, in order, byte-identically, and a slow-but-alive
|
||||
// client never triggers CancelRun.
|
||||
func TestChatCompletionsPassthroughSlowClientByteIdentity(t *testing.T) {
|
||||
const chunkCount = 40
|
||||
var want strings.Builder
|
||||
frames := make(chan *iop.ProviderTunnelFrame, chunkCount+2)
|
||||
frames <- &iop.ProviderTunnelFrame{
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START,
|
||||
StatusCode: 200,
|
||||
Headers: map[string]string{"Content-Type": "text/event-stream"},
|
||||
}
|
||||
for i := 0; i < chunkCount; i++ {
|
||||
chunk := fmt.Sprintf("data: chunk-%02d\n\n", i)
|
||||
want.WriteString(chunk)
|
||||
frames <- &iop.ProviderTunnelFrame{
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY,
|
||||
Sequence: int64(i + 1),
|
||||
Body: []byte(chunk),
|
||||
}
|
||||
}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true}
|
||||
close(frames)
|
||||
srv, fake := chatSidebandServer(frames)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"stream":true,
|
||||
"messages":[{"role":"user","content":"hello"}]
|
||||
}`))
|
||||
w := newFlakyResponseWriter(-1)
|
||||
w.writeDelay = 2 * time.Millisecond
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
status, body, _ := w.snapshot()
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("status: got %d", status)
|
||||
}
|
||||
if body != want.String() {
|
||||
t.Fatalf("slow client stream not byte-identical/ordered:\n got: %q\nwant: %q", body, want.String())
|
||||
}
|
||||
if calls := fake.cancelCallsSnapshot(); len(calls) != 0 {
|
||||
t.Fatalf("slow but alive client must not trigger CancelRun, got %+v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughSidebandStreamWriteFailureSendsCancelRunOnce
|
||||
// verifies the sideband streaming writer propagates exactly one CancelRun when
|
||||
// a provider body write fails mid-stream and stops relaying afterwards.
|
||||
func TestChatCompletionsPassthroughSidebandStreamWriteFailureSendsCancelRunOnce(t *testing.T) {
|
||||
frames := make(chan *iop.ProviderTunnelFrame, 5)
|
||||
frames <- &iop.ProviderTunnelFrame{
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_RESPONSE_START,
|
||||
StatusCode: 200,
|
||||
Headers: map[string]string{"Content-Type": "text/event-stream"},
|
||||
}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: chunk-1\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY, Body: []byte("data: chunk-2\n\n")}
|
||||
frames <- &iop.ProviderTunnelFrame{Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END, End: true}
|
||||
close(frames)
|
||||
srv, fake := chatSidebandServer(frames)
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"stream":true,
|
||||
"messages":[{"role":"user","content":"hello"}],
|
||||
"metadata":{"iop_response_mode":"passthrough+sideband"}
|
||||
}`))
|
||||
// Write 1 is the route sideband event, write 2 the first provider chunk;
|
||||
// the second provider chunk fails.
|
||||
w := newFlakyResponseWriter(2)
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
status, body, failedWrites := w.snapshot()
|
||||
if status != http.StatusOK {
|
||||
t.Fatalf("status: got %d body=%s", status, body)
|
||||
}
|
||||
if failedWrites == 0 {
|
||||
t.Fatal("fixture did not exercise the sideband write failure path")
|
||||
}
|
||||
if !strings.Contains(body, "data: chunk-1\n\n") || strings.Contains(body, "chunk-2") {
|
||||
t.Fatalf("sideband write failure must truncate after the ordered prefix: %q", body)
|
||||
}
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call on sideband write failure, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
if calls[0].RunID != "run-tunnel" || calls[0].NodeRef != "node-1" {
|
||||
t.Fatalf("unexpected cancel request: %+v", calls[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughSidebandStreamContextCancelSendsCancelRun
|
||||
// verifies caller disconnect on the sideband streaming surface propagates
|
||||
// exactly one CancelRun while the tunnel is still in flight.
|
||||
func TestChatCompletionsPassthroughSidebandStreamContextCancelSendsCancelRun(t *testing.T) {
|
||||
fake := &fakeRunService{tunnelFrames: make(chan *iop.ProviderTunnelFrame)}
|
||||
srv := NewServer(config.EdgeOpenAIConf{}, fake, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{
|
||||
{ID: "pool-model", Providers: map[string]string{"prov-1": "served-model"}},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"stream":true,
|
||||
"messages":[{"role":"user","content":"hello"}],
|
||||
"metadata":{"iop_response_mode":"passthrough+sideband"}
|
||||
}`)).WithContext(ctx)
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
if calls[0].RunID != "run-tunnel" || calls[0].NodeRef != "node-1" {
|
||||
t.Fatalf("unexpected cancel request: %+v", calls[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughSidebandNonStreamingContextCancelSendsCancelRun
|
||||
// verifies caller disconnect on the buffered sideband surface propagates
|
||||
// exactly one CancelRun while the tunnel is still in flight.
|
||||
func TestChatCompletionsPassthroughSidebandNonStreamingContextCancelSendsCancelRun(t *testing.T) {
|
||||
fake := &fakeRunService{tunnelFrames: make(chan *iop.ProviderTunnelFrame)}
|
||||
srv := NewServer(config.EdgeOpenAIConf{}, fake, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{
|
||||
{ID: "pool-model", Providers: map[string]string{"prov-1": "served-model"}},
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"messages":[{"role":"user","content":"hello"}],
|
||||
"metadata":{"iop_response_mode":"passthrough+sideband"}
|
||||
}`)).WithContext(ctx)
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
if calls[0].RunID != "run-tunnel" || calls[0].NodeRef != "node-1" {
|
||||
t.Fatalf("unexpected cancel request: %+v", calls[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestChatCompletionsPassthroughSidebandNonStreamingTimeoutSendsCancelRun
|
||||
// verifies the buffered sideband surface converts a tunnel wait timeout into a
|
||||
// 502 and exactly one CancelRun.
|
||||
func TestChatCompletionsPassthroughSidebandNonStreamingTimeoutSendsCancelRun(t *testing.T) {
|
||||
fake := &fakeRunService{
|
||||
tunnelFrames: make(chan *iop.ProviderTunnelFrame),
|
||||
tunnelWaitTimeout: 50 * time.Millisecond,
|
||||
}
|
||||
srv := NewServer(config.EdgeOpenAIConf{}, fake, nil)
|
||||
srv.SetModelCatalog([]config.ModelCatalogEntry{
|
||||
{ID: "pool-model", Providers: map[string]string{"prov-1": "served-model"}},
|
||||
})
|
||||
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", strings.NewReader(`{
|
||||
"model":"pool-model",
|
||||
"messages":[{"role":"user","content":"hello"}],
|
||||
"metadata":{"iop_response_mode":"passthrough+sideband"}
|
||||
}`))
|
||||
w := httptest.NewRecorder()
|
||||
srv.handleChatCompletions(w, req)
|
||||
|
||||
if w.Code != http.StatusBadGateway {
|
||||
t.Fatalf("status: got %d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if !strings.Contains(w.Body.String(), "run timed out") {
|
||||
t.Fatalf("timeout error not surfaced: %s", w.Body.String())
|
||||
}
|
||||
calls := fake.cancelCallsSnapshot()
|
||||
if len(calls) != 1 {
|
||||
t.Fatalf("expected exactly 1 CancelRun call on sideband timeout, got %d: %+v", len(calls), calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChatCompletionsProviderPoolAppliesGenerationPolicy(t *testing.T) {
|
||||
fake := &fakeRunService{events: bufferedRunEvents(
|
||||
&iop.RunEvent{Type: "delta", Delta: "ok"},
|
||||
|
|
|
|||
|
|
@ -278,6 +278,160 @@ func TestSubmitProviderTunnelCloseReleasesSlot(t *testing.T) {
|
|||
"admission slot not released after Close")
|
||||
}
|
||||
|
||||
// TestProviderTunnelRouterBackpressureBlocksWithoutDrop verifies the router
|
||||
// blocks the producer while the subscriber buffer is full instead of dropping
|
||||
// frames, and delivers every frame in order once the subscriber drains.
|
||||
func TestProviderTunnelRouterBackpressureBlocksWithoutDrop(t *testing.T) {
|
||||
svc := New(edgenode.NewRegistry(), nil)
|
||||
ch, unsub := svc.tunnels.subscribe("tunnel-bp", 2)
|
||||
defer unsub()
|
||||
|
||||
const total = 6
|
||||
routed := make(chan struct{})
|
||||
go func() {
|
||||
defer close(routed)
|
||||
for i := 0; i < total; i++ {
|
||||
svc.tunnels.route(&iop.ProviderTunnelFrame{
|
||||
TunnelId: "tunnel-bp",
|
||||
Sequence: int64(i),
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY,
|
||||
})
|
||||
}
|
||||
}()
|
||||
|
||||
// With buffer 2 the producer cannot finish all routes before the
|
||||
// subscriber drains: full buffer must block, not drop.
|
||||
select {
|
||||
case <-routed:
|
||||
t.Fatal("route calls finished while the subscriber buffer was full; expected blocking backpressure")
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
|
||||
for i := 0; i < total; i++ {
|
||||
select {
|
||||
case f := <-ch:
|
||||
if f.GetSequence() != int64(i) {
|
||||
t.Fatalf("frame %d: got sequence %d; frames must stay ordered and lossless under backpressure", i, f.GetSequence())
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatalf("frame %d not delivered after draining", i)
|
||||
}
|
||||
}
|
||||
|
||||
select {
|
||||
case <-routed:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("producer still blocked after all frames were drained")
|
||||
}
|
||||
}
|
||||
|
||||
// TestProviderTunnelRouterCloseUnblocksBlockedRoute verifies unsubscribe (the
|
||||
// handle Close/cancel path) releases a route call blocked on a full buffer, so
|
||||
// an abandoned tunnel can never wedge the transport frame reader.
|
||||
func TestProviderTunnelRouterCloseUnblocksBlockedRoute(t *testing.T) {
|
||||
svc := New(edgenode.NewRegistry(), nil)
|
||||
_, unsub := svc.tunnels.subscribe("tunnel-close", 1)
|
||||
|
||||
frame := func(seq int64) *iop.ProviderTunnelFrame {
|
||||
return &iop.ProviderTunnelFrame{
|
||||
TunnelId: "tunnel-close",
|
||||
Sequence: seq,
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_BODY,
|
||||
}
|
||||
}
|
||||
|
||||
blocked := make(chan struct{})
|
||||
go func() {
|
||||
defer close(blocked)
|
||||
svc.tunnels.route(frame(0)) // fills the buffer
|
||||
svc.tunnels.route(frame(1)) // blocks until close releases it
|
||||
}()
|
||||
|
||||
select {
|
||||
case <-blocked:
|
||||
t.Fatal("second route must block on the full subscriber buffer")
|
||||
case <-time.After(100 * time.Millisecond):
|
||||
}
|
||||
|
||||
unsub()
|
||||
|
||||
select {
|
||||
case <-blocked:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("unsubscribe did not release the blocked route call")
|
||||
}
|
||||
|
||||
// A late frame for the closed tunnel is dropped without blocking.
|
||||
svc.tunnels.route(frame(2))
|
||||
}
|
||||
|
||||
// TestProviderTunnelReleaseExactlyOnceAcrossEndAndClose verifies the tracked
|
||||
// provider-pool slot (capacity 1) is released exactly once when both the
|
||||
// terminal END frame and handle Close fire, and that the freed slot admits the
|
||||
// next tunnel immediately instead of queueing.
|
||||
func TestProviderTunnelReleaseExactlyOnceAcrossEndAndClose(t *testing.T) {
|
||||
env := newProviderTunnelTestEnv(t)
|
||||
svc := env.svc
|
||||
|
||||
submit := func(runID string) ProviderTunnelResult {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
handle, err := svc.SubmitProviderTunnel(ctx, SubmitProviderTunnelRequest{
|
||||
RunID: runID,
|
||||
ModelGroupKey: "qwen3.6:35b",
|
||||
Method: "POST",
|
||||
Path: "/v1/chat/completions",
|
||||
Body: []byte(`{"model":"qwen3.6:35b"}`),
|
||||
ProviderPool: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("SubmitProviderTunnel(%s): %v", runID, err)
|
||||
}
|
||||
return handle
|
||||
}
|
||||
|
||||
handle1 := submit("run-release-1")
|
||||
waitForCondition(t, func() bool {
|
||||
captured := env.capturedRequest()
|
||||
return captured != nil && captured.GetRunId() == "run-release-1"
|
||||
}, "fake node did not receive the first ProviderTunnelRequest")
|
||||
captured := env.capturedRequest()
|
||||
|
||||
svc.RouteProviderTunnelFrame(&iop.ProviderTunnelFrame{
|
||||
RunId: captured.GetRunId(),
|
||||
TunnelId: captured.GetTunnelId(),
|
||||
Kind: iop.ProviderTunnelFrameKind_PROVIDER_TUNNEL_FRAME_KIND_END,
|
||||
End: true,
|
||||
})
|
||||
stream := handle1.Stream()
|
||||
for {
|
||||
if _, open := <-stream.Frames; !open {
|
||||
break
|
||||
}
|
||||
}
|
||||
waitForCondition(t, func() bool { return inflightRunCount(svc.queue) == 0 },
|
||||
"admission slot not released after END frame")
|
||||
|
||||
// Close after the terminal frame must stay a no-op release.
|
||||
handle1.Close()
|
||||
if got := inflightRunCount(svc.queue); got != 0 {
|
||||
t.Fatalf("inflight count after END+Close: got %d, want 0", got)
|
||||
}
|
||||
|
||||
// The capacity-1 slot is free again: the next tunnel is admitted without
|
||||
// waiting on the queue (a leaked slot would block this submit).
|
||||
handle2 := submit("run-release-2")
|
||||
defer handle2.Close()
|
||||
waitForCondition(t, func() bool {
|
||||
captured := env.capturedRequest()
|
||||
return captured != nil && captured.GetRunId() == "run-release-2"
|
||||
}, "fake node did not receive the second ProviderTunnelRequest after slot release")
|
||||
if got := inflightRunCount(svc.queue); got != 1 {
|
||||
t.Fatalf("inflight count after second dispatch: got %d, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSubmitProviderTunnelErrorFrameReleasesSlot verifies the ERROR frame is
|
||||
// terminal and releases the provider-pool slot like END.
|
||||
func TestSubmitProviderTunnelErrorFrameReleasesSlot(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
|
|
@ -1113,6 +1114,56 @@ func (s *fakeTunnelSink) all() []runtime.ProviderTunnelFrame {
|
|||
return append([]runtime.ProviderTunnelFrame(nil), s.frames...)
|
||||
}
|
||||
|
||||
// failingTunnelSink emulates the Edge socket write failing after failAfter
|
||||
// successful frames, as when the Edge relay aborts an abandoned tunnel.
|
||||
type failingTunnelSink struct {
|
||||
mu sync.Mutex
|
||||
frames []runtime.ProviderTunnelFrame
|
||||
failAfter int
|
||||
}
|
||||
|
||||
func (s *failingTunnelSink) EmitTunnelFrame(_ context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(s.frames) >= s.failAfter {
|
||||
return errors.New("edge socket write failed")
|
||||
}
|
||||
s.frames = append(s.frames, frame)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *failingTunnelSink) all() []runtime.ProviderTunnelFrame {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]runtime.ProviderTunnelFrame(nil), s.frames...)
|
||||
}
|
||||
|
||||
// assertOrderedTunnelFrames verifies S09 frame semantics: contiguous sequences
|
||||
// starting at 0 (no drop/reorder) and exactly one terminal END/ERROR frame,
|
||||
// which must be the last frame emitted.
|
||||
func assertOrderedTunnelFrames(t *testing.T, frames []runtime.ProviderTunnelFrame) {
|
||||
t.Helper()
|
||||
if len(frames) == 0 {
|
||||
t.Fatal("no tunnel frames emitted")
|
||||
}
|
||||
terminals := 0
|
||||
for i, f := range frames {
|
||||
if f.Sequence != int64(i) {
|
||||
t.Errorf("frame %d: sequence %d, want %d (ordered, lossless)", i, f.Sequence, i)
|
||||
}
|
||||
switch f.Kind {
|
||||
case runtime.ProviderTunnelFrameKindEnd, runtime.ProviderTunnelFrameKindError:
|
||||
terminals++
|
||||
if i != len(frames)-1 {
|
||||
t.Errorf("terminal %s frame at index %d must be the last of %d frames", f.Kind, i, len(frames))
|
||||
}
|
||||
}
|
||||
}
|
||||
if terminals != 1 {
|
||||
t.Errorf("expected exactly one terminal END/ERROR frame, got %d", terminals)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatTunnelProvider(t *testing.T) {
|
||||
expectedBody := "data: {\"choices\":[{\"delta\":{\"reasoning_content\":\"think Qwen\"}}]}\n\ndata: [DONE]\n\n"
|
||||
|
||||
|
|
@ -1195,6 +1246,7 @@ func TestOpenAICompatTunnelProvider(t *testing.T) {
|
|||
if endFrame.Sequence != lastSeq+1 {
|
||||
t.Errorf("expected seq %d, got %d", lastSeq+1, endFrame.Sequence)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
}
|
||||
|
||||
func TestOpenAICompatTunnelProvider_Cancel(t *testing.T) {
|
||||
|
|
@ -1265,18 +1317,26 @@ func TestOpenAICompatTunnelProvider_Cancel(t *testing.T) {
|
|||
if len(frames) < 2 {
|
||||
t.Fatalf("expected at least 2 frames, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
|
||||
if frames[0].Kind != runtime.ProviderTunnelFrameKindResponseStart {
|
||||
t.Errorf("expected first frame RESPONSE_START, got %s", frames[0].Kind)
|
||||
}
|
||||
for i, f := range frames[1 : len(frames)-1] {
|
||||
if f.Kind != runtime.ProviderTunnelFrameKindBody {
|
||||
t.Errorf("mid frame %d: expected BODY before the terminal ERROR, got %s", i+1, f.Kind)
|
||||
}
|
||||
}
|
||||
lastFrame := frames[len(frames)-1]
|
||||
if lastFrame.Kind != runtime.ProviderTunnelFrameKindError {
|
||||
t.Errorf("expected last frame to be ERROR, got %s", lastFrame.Kind)
|
||||
}
|
||||
expectedSeq := int64(len(frames) - 1)
|
||||
if lastFrame.Sequence != expectedSeq {
|
||||
t.Errorf("expected error frame sequence %d, got %d", expectedSeq, lastFrame.Sequence)
|
||||
}
|
||||
if lastFrame.Error == "" {
|
||||
t.Error("expected non-empty error message in frame")
|
||||
}
|
||||
if !strings.Contains(lastFrame.Error, "context canceled") {
|
||||
t.Errorf("expected cancellation error in terminal frame, got %q", lastFrame.Error)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAICompatTunnelProvider_BodyReadError(t *testing.T) {
|
||||
|
|
@ -1318,16 +1378,97 @@ func TestOpenAICompatTunnelProvider_BodyReadError(t *testing.T) {
|
|||
if len(frames) < 2 {
|
||||
t.Fatalf("expected at least 2 frames, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
|
||||
lastFrame := frames[len(frames)-1]
|
||||
if lastFrame.Kind != runtime.ProviderTunnelFrameKindError {
|
||||
t.Errorf("expected last frame to be ERROR, got %s", lastFrame.Kind)
|
||||
}
|
||||
expectedSeq := int64(len(frames) - 1)
|
||||
if lastFrame.Sequence != expectedSeq {
|
||||
t.Errorf("expected error frame sequence %d, got %d", expectedSeq, lastFrame.Sequence)
|
||||
}
|
||||
if !strings.Contains(lastFrame.Error, "read response body") && !strings.Contains(lastFrame.Error, "EOF") {
|
||||
t.Errorf("expected body read error msg, got %q", lastFrame.Error)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAICompatTunnelProvider_SinkWriteFailureStopsUpstreamRelay covers the
|
||||
// Edge-write-failure leg of S09: when EmitTunnelFrame fails mid-stream the
|
||||
// adapter stops relaying immediately, returns the emit error without appending
|
||||
// further frames, and the upstream provider request is torn down.
|
||||
func TestOpenAICompatTunnelProvider_SinkWriteFailureStopsUpstreamRelay(t *testing.T) {
|
||||
handlerObservedTeardown := make(chan struct{})
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
flusher, ok := w.(http.Flusher)
|
||||
for {
|
||||
_, err := w.Write([]byte("data: chunk\n\n"))
|
||||
if err != nil {
|
||||
close(handlerObservedTeardown)
|
||||
return
|
||||
}
|
||||
if ok {
|
||||
flusher.Flush()
|
||||
}
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
close(handlerObservedTeardown)
|
||||
return
|
||||
case <-time.After(10 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
adapter := New(config.OpenAICompatConf{
|
||||
Endpoint: server.URL,
|
||||
}, zap.NewNop())
|
||||
|
||||
// RESPONSE_START and the first BODY frame succeed; the next emit fails as
|
||||
// if the Edge socket write broke after the caller went away.
|
||||
sink := &failingTunnelSink{failAfter: 2}
|
||||
req := runtime.ProviderTunnelRequest{
|
||||
RunID: "run-1",
|
||||
TunnelID: "tunnel-1",
|
||||
Method: "POST",
|
||||
Path: "/v1/chat/completions",
|
||||
Body: []byte(`{"prompt":"hi"}`),
|
||||
}
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
errCh <- adapter.TunnelProvider(context.Background(), req, sink)
|
||||
}()
|
||||
|
||||
var tunnelErr error
|
||||
select {
|
||||
case tunnelErr = <-errCh:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("timeout waiting for TunnelProvider to return after sink write failure")
|
||||
}
|
||||
if tunnelErr == nil || !strings.Contains(tunnelErr.Error(), "edge socket write failed") {
|
||||
t.Fatalf("expected the sink emit error to propagate, got %v", tunnelErr)
|
||||
}
|
||||
|
||||
select {
|
||||
case <-handlerObservedTeardown:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("provider handler did not observe upstream request teardown")
|
||||
}
|
||||
|
||||
frames := sink.all()
|
||||
if len(frames) != 2 {
|
||||
t.Fatalf("expected relay to stop at 2 frames after emit failure, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
if frames[0].Kind != runtime.ProviderTunnelFrameKindResponseStart {
|
||||
t.Errorf("expected first frame RESPONSE_START, got %s", frames[0].Kind)
|
||||
}
|
||||
if frames[1].Kind != runtime.ProviderTunnelFrameKindBody {
|
||||
t.Errorf("expected second frame BODY, got %s", frames[1].Kind)
|
||||
}
|
||||
for i, f := range frames {
|
||||
if f.Sequence != int64(i) {
|
||||
t.Errorf("frame %d: sequence %d, want %d", i, f.Sequence, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
|
|
@ -492,6 +493,56 @@ func (s *fakeTunnelSink) all() []runtime.ProviderTunnelFrame {
|
|||
return append([]runtime.ProviderTunnelFrame(nil), s.frames...)
|
||||
}
|
||||
|
||||
// failingTunnelSink emulates the Edge socket write failing after failAfter
|
||||
// successful frames, as when the Edge relay aborts an abandoned tunnel.
|
||||
type failingTunnelSink struct {
|
||||
mu sync.Mutex
|
||||
frames []runtime.ProviderTunnelFrame
|
||||
failAfter int
|
||||
}
|
||||
|
||||
func (s *failingTunnelSink) EmitTunnelFrame(_ context.Context, frame runtime.ProviderTunnelFrame) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(s.frames) >= s.failAfter {
|
||||
return errors.New("edge socket write failed")
|
||||
}
|
||||
s.frames = append(s.frames, frame)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *failingTunnelSink) all() []runtime.ProviderTunnelFrame {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]runtime.ProviderTunnelFrame(nil), s.frames...)
|
||||
}
|
||||
|
||||
// assertOrderedTunnelFrames verifies S09 frame semantics: contiguous sequences
|
||||
// starting at 0 (no drop/reorder) and exactly one terminal END/ERROR frame,
|
||||
// which must be the last frame emitted.
|
||||
func assertOrderedTunnelFrames(t *testing.T, frames []runtime.ProviderTunnelFrame) {
|
||||
t.Helper()
|
||||
if len(frames) == 0 {
|
||||
t.Fatal("no tunnel frames emitted")
|
||||
}
|
||||
terminals := 0
|
||||
for i, f := range frames {
|
||||
if f.Sequence != int64(i) {
|
||||
t.Errorf("frame %d: sequence %d, want %d (ordered, lossless)", i, f.Sequence, i)
|
||||
}
|
||||
switch f.Kind {
|
||||
case runtime.ProviderTunnelFrameKindEnd, runtime.ProviderTunnelFrameKindError:
|
||||
terminals++
|
||||
if i != len(frames)-1 {
|
||||
t.Errorf("terminal %s frame at index %d must be the last of %d frames", f.Kind, i, len(frames))
|
||||
}
|
||||
}
|
||||
}
|
||||
if terminals != 1 {
|
||||
t.Errorf("expected exactly one terminal END/ERROR frame, got %d", terminals)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVllmTunnelProvider(t *testing.T) {
|
||||
expectedBody := "data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\ndata: [DONE]\n\n"
|
||||
|
||||
|
|
@ -570,6 +621,7 @@ func TestVllmTunnelProvider(t *testing.T) {
|
|||
if endFrame.Sequence != lastSeq+1 {
|
||||
t.Errorf("expected seq %d, got %d", lastSeq+1, endFrame.Sequence)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
}
|
||||
|
||||
func TestVllmTunnelProvider_Cancel(t *testing.T) {
|
||||
|
|
@ -640,18 +692,26 @@ func TestVllmTunnelProvider_Cancel(t *testing.T) {
|
|||
if len(frames) < 2 {
|
||||
t.Fatalf("expected at least 2 frames, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
|
||||
if frames[0].Kind != runtime.ProviderTunnelFrameKindResponseStart {
|
||||
t.Errorf("expected first frame RESPONSE_START, got %s", frames[0].Kind)
|
||||
}
|
||||
for i, f := range frames[1 : len(frames)-1] {
|
||||
if f.Kind != runtime.ProviderTunnelFrameKindBody {
|
||||
t.Errorf("mid frame %d: expected BODY before the terminal ERROR, got %s", i+1, f.Kind)
|
||||
}
|
||||
}
|
||||
lastFrame := frames[len(frames)-1]
|
||||
if lastFrame.Kind != runtime.ProviderTunnelFrameKindError {
|
||||
t.Errorf("expected last frame to be ERROR, got %s", lastFrame.Kind)
|
||||
}
|
||||
expectedSeq := int64(len(frames) - 1)
|
||||
if lastFrame.Sequence != expectedSeq {
|
||||
t.Errorf("expected error frame sequence %d, got %d", expectedSeq, lastFrame.Sequence)
|
||||
}
|
||||
if lastFrame.Error == "" {
|
||||
t.Error("expected non-empty error message in frame")
|
||||
}
|
||||
if !strings.Contains(lastFrame.Error, "context canceled") {
|
||||
t.Errorf("expected cancellation error in terminal frame, got %q", lastFrame.Error)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVllmTunnelProvider_BodyReadError(t *testing.T) {
|
||||
|
|
@ -693,20 +753,101 @@ func TestVllmTunnelProvider_BodyReadError(t *testing.T) {
|
|||
if len(frames) < 2 {
|
||||
t.Fatalf("expected at least 2 frames, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
|
||||
lastFrame := frames[len(frames)-1]
|
||||
if lastFrame.Kind != runtime.ProviderTunnelFrameKindError {
|
||||
t.Errorf("expected last frame to be ERROR, got %s", lastFrame.Kind)
|
||||
}
|
||||
expectedSeq := int64(len(frames) - 1)
|
||||
if lastFrame.Sequence != expectedSeq {
|
||||
t.Errorf("expected error frame sequence %d, got %d", expectedSeq, lastFrame.Sequence)
|
||||
}
|
||||
if !strings.Contains(lastFrame.Error, "read response body") && !strings.Contains(lastFrame.Error, "EOF") {
|
||||
t.Errorf("expected body read error msg, got %q", lastFrame.Error)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVllmTunnelProvider_SinkWriteFailureStopsUpstreamRelay covers the
|
||||
// Edge-write-failure leg of S09: when EmitTunnelFrame fails mid-stream the
|
||||
// adapter stops relaying immediately, returns the emit error without appending
|
||||
// further frames, and the upstream provider request is torn down.
|
||||
func TestVllmTunnelProvider_SinkWriteFailureStopsUpstreamRelay(t *testing.T) {
|
||||
handlerObservedTeardown := make(chan struct{})
|
||||
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
|
||||
flusher, ok := w.(http.Flusher)
|
||||
for {
|
||||
_, err := w.Write([]byte("data: chunk\n\n"))
|
||||
if err != nil {
|
||||
close(handlerObservedTeardown)
|
||||
return
|
||||
}
|
||||
if ok {
|
||||
flusher.Flush()
|
||||
}
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
close(handlerObservedTeardown)
|
||||
return
|
||||
case <-time.After(10 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
adapter := New(config.VllmConf{
|
||||
Endpoint: server.URL,
|
||||
}, zap.NewNop())
|
||||
|
||||
// RESPONSE_START and the first BODY frame succeed; the next emit fails as
|
||||
// if the Edge socket write broke after the caller went away.
|
||||
sink := &failingTunnelSink{failAfter: 2}
|
||||
req := runtime.ProviderTunnelRequest{
|
||||
RunID: "run-1",
|
||||
TunnelID: "tunnel-1",
|
||||
Method: "POST",
|
||||
Path: "/v1/chat/completions",
|
||||
Body: []byte(`{"prompt":"hi"}`),
|
||||
}
|
||||
|
||||
errCh := make(chan error, 1)
|
||||
go func() {
|
||||
errCh <- adapter.TunnelProvider(context.Background(), req, sink)
|
||||
}()
|
||||
|
||||
var tunnelErr error
|
||||
select {
|
||||
case tunnelErr = <-errCh:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("timeout waiting for TunnelProvider to return after sink write failure")
|
||||
}
|
||||
if tunnelErr == nil || !strings.Contains(tunnelErr.Error(), "edge socket write failed") {
|
||||
t.Fatalf("expected the sink emit error to propagate, got %v", tunnelErr)
|
||||
}
|
||||
|
||||
select {
|
||||
case <-handlerObservedTeardown:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("provider handler did not observe upstream request teardown")
|
||||
}
|
||||
|
||||
frames := sink.all()
|
||||
if len(frames) != 2 {
|
||||
t.Fatalf("expected relay to stop at 2 frames after emit failure, got %d: %+v", len(frames), frames)
|
||||
}
|
||||
if frames[0].Kind != runtime.ProviderTunnelFrameKindResponseStart {
|
||||
t.Errorf("expected first frame RESPONSE_START, got %s", frames[0].Kind)
|
||||
}
|
||||
if frames[1].Kind != runtime.ProviderTunnelFrameKindBody {
|
||||
t.Errorf("expected second frame BODY, got %s", frames[1].Kind)
|
||||
}
|
||||
for i, f := range frames {
|
||||
if f.Sequence != int64(i) {
|
||||
t.Errorf("frame %d: sequence %d, want %d", i, f.Sequence, i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestVllmTunnelProvider_GemmaContentToolCallRawBytes(t *testing.T) {
|
||||
expectedBody := "data: {\"choices\":[{\"delta\":{\"content\":\"Gemma response \"}}]}\n\n" +
|
||||
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_gemma_1\",\"type\":\"function\",\"function\":{\"name\":\"run_commands\",\"arguments\":\"{\\\"commands\\\": \"}}]}}]}\n\n" +
|
||||
|
|
@ -784,4 +925,5 @@ func TestVllmTunnelProvider_GemmaContentToolCallRawBytes(t *testing.T) {
|
|||
if endFrame.Sequence != lastSeq+1 {
|
||||
t.Errorf("expected seq %d, got %d", lastSeq+1, endFrame.Sequence)
|
||||
}
|
||||
assertOrderedTunnelFrames(t, frames)
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue