feat(stream-gate): 응답 스트림 검증 경로를 확장한다
응답 출력의 검증·필터링·큐 제어와 운영 계약을 함께 정비해 provider 출력 검증 흐름의 일관성과 복구 가능성을 높인다.
This commit is contained in:
commit
33d13afab5
45 changed files with 11929 additions and 326 deletions
|
|
@ -33,8 +33,9 @@ tracked config에는 public 예시와 기본 구조만 두고, 실제 endpoint/c
|
|||
|
||||
- `openai.principal_tokens[]`는 raw token을 저장하지 않고 hash/reference로 principal 매핑을 관리한다. 각 entry는 `token_ref` (non-empty, unique), `token_hash_sha256` (64-char hex, duplicate hash rejection), `principal_ref` (non-empty), optional `principal_alias` 필드를 갖는다. 여러 entry가 같은 `principal_ref`와 `principal_alias`를 공유할 수 있으며, 이때 `token_ref`가 앱/통합/용도별 사용량 분해 기준이 된다. tracked config에는 raw token을 저장하지 않고 hash/reference만 둔다.
|
||||
- `openai.provider_auth`는 request-time raw provider token forwarding rule이다. `enabled=false`가 기본이며 raw token 값은 저장하지 않는다. `enabled=true`이고 header fields가 생략되면 `from_header=X-IOP-Provider-Authorization`, `target_header=Authorization`, `scheme=Bearer`, `required=true`로 해석한다.
|
||||
- `openai.stream_evidence_gate`는 request-local Recovery Coordinator 기본값·절대 상한·ingress snapshot 제한 설정이다. `enabled`는 normalized live-SSE chat completion, provider tunnel passthrough, provider-pool dispatch, tool-validation recovery를 `packages/go/streamgate` request runtime이 소유하도록 라우팅할지 여부이며 omitted 기본값 false(legacy eager-write path와 legacy tool-validation retry loop를 그대로 유지)이다. `max_request_fault_recovery`는 요청당 전체 fault recovery 상한(`0..3`, omitted 기본값 3, explicit 0은 모든 fault recovery 비활성화)이다. `max_strategy_fault_recovery`는 fault strategy(exact_replay/continuation_repair/schema_repair)별 상한(`0..max_request_fault_recovery`, omitted 기본값은 effective request total 상속, explicit 0은 해당 strategy 비활성화)이며 request-start 시점에 immutable runtime option snapshot으로 각 fault strategy에 동일하게 적용된다. `max_ingress_snapshot_bytes`는 ingress snapshot 바이트 상한(`1..16777216` [16 MiB], omitted/0 기본값 16 MiB)이다.
|
||||
- `openai.stream_evidence_gate`는 request-local Recovery Coordinator 기본값·절대 상한·ingress snapshot 제한 설정이다. `enabled`는 지원되는 Chat Completions, normalized Responses, provider tunnel passthrough, provider-pool dispatch, tool-validation recovery를 `packages/go/streamgate` request runtime이 소유하도록 라우팅할지 여부이며 omitted 기본값 false(legacy eager-write path와 legacy tool-validation retry loop를 그대로 유지)이다. `max_request_fault_recovery`는 요청당 전체 fault recovery 상한(`0..3`, omitted 기본값 3, explicit 0은 모든 fault recovery 비활성화)이다. `max_strategy_fault_recovery`는 fault strategy(exact_replay/continuation_repair/schema_repair)별 상한(`0..max_request_fault_recovery`, omitted 기본값은 effective request total 상속, explicit 0은 해당 strategy 비활성화)이며 request-start 시점에 immutable runtime option snapshot으로 각 fault strategy에 동일하게 적용된다. `max_ingress_snapshot_bytes`는 ingress snapshot 바이트 상한(`1..16777216` [16 MiB], omitted/0 기본값 16 MiB)이다. `environment`는 request-start selector snapshot이며 `dev|dev-corp`만 허용하고 omitted 기본값은 `dev`다. `filters[]`는 unique `filter` (`repeat_guard|schema_gate|provider_error`) policy이다. `enabled` omitted=true, `enforcement` omitted=`blocking`, `capability` omitted=`output.<filter>`, `hold_evidence_runes` omitted=500, `timeout_ms` omitted=5000으로 정규화하며 selector는 `environment|model_group|model|provider`로만 filter enablement/enforcement를 보정한다. base-disabled filter도 registry snapshot에 남아 더 구체적인 selector가 활성화할 수 있고, 실제 target에서 활성화된 `blocking` filter만 provider capability admission에 참여한다. `observe_only`는 evidence를 만들지만 admission을 막지 않는다. `repeat_guard` uses the configured rune bound for active request-local history/current-stream inspection and stores only bounded fingerprints, counts, and offsets in its semantic snapshot and observations. `schema_gate` and `provider_error` remain lifecycle foundations until their matcher Tasks; an unmatched provider error never creates exact replay. Config accepts no caller/agent selector.
|
||||
- `openai.stream_evidence_gate` 설정은 request-start 시점에 snapshot으로 고정되며 in-flight request의 실행 중 refresh 영향에서 격리된다 (generation isolation). 새 generation의 설정은 이후 시작되는 새 request에만 적용된다.
|
||||
- The request-start `models[].context_window_tokens` snapshot is the resume builder's target context bound. Each Chat/Responses runtime shares one request-local content/reasoning recorder across its initial and recovery event sources. A continuation rebuild uses only that recorder and the fixed directive; unknown or exceeded context rejects the rebuild before re-admission. An omitted caller temperature selects `0.2`, `0.4`, then `0.6` by continuation strategy attempt, while an explicit value is preserved. Recorder state and its raw values remain request-local, are consumed once per attempt, and are never added to config refresh state or observations. Repeat history and counters are pinned to the same request-start config generation and are not refreshable TTL/session state.
|
||||
- `openai` deep diff는 restart-required로 분류한다. `openai.principal_tokens[]` 및 `openai.stream_evidence_gate` 변경은 restart-required classifier에 포함된다.
|
||||
- `openai.model_routes[]`는 외부 OpenAI-compatible `model` id를 내부 `adapter + target` route로 매핑하는 compatibility catalog다.
|
||||
- `long_context_threshold_tokens`는 Edge root의 입력 토큰 추정 기준 long-context 분류 threshold다. 기본값은 `100000`이며 0 이하 값은 config load에서 거부한다.
|
||||
|
|
|
|||
|
|
@ -85,11 +85,17 @@ Edge는 이 값을 provider tunnel request의 `openai.provider_auth.target_heade
|
|||
|
||||
Chat Completions와 Responses ingress에는 configured request snapshot 상한이 body 첫 read 전에 적용된다. body 또는 typed semantic view가 상한을 넘거나 rebuild peak 회계가 실패하면 provider admission 없이 HTTP `413`, `error.type="invalid_request_error"` 한 번으로 종료한다. 이 오류의 `message`는 내부 byte 수, snapshot reference, Core 오류 이름을 노출하지 않는다. 기존 public error body는 계속 `error.type`과 `error.message`만 가지며 size/trace/causes 같은 필드를 추가하지 않는다.
|
||||
|
||||
위 bounded ingress/size 오류 호환성은 활성 계약이다. `openai.stream_evidence_gate.enabled=true`이면 지원되는 Chat Completions와 streaming provider-tunnel 경로가 [완료된 Stream Evidence Gate Core Milestone](../../agent-roadmap/archive/phase/knowledge-tool-optimization-extension/milestones/stream-evidence-gate-core.md)의 request-local runtime을 사용한다. runtime은 response status/header와 opening event를 첫 safe release까지 보류하고, filter 결과를 모두 모은 뒤 release, terminal 또는 bounded recovery 중 하나만 실행한다. 기본값 `false`에서는 기존 compatibility 경로를 유지한다.
|
||||
위 bounded ingress/size 오류 호환성은 활성 계약이다. `openai.stream_evidence_gate.enabled=true`이면 지원되는 Chat Completions, normalized Responses, provider-tunnel 경로가 [완료된 Stream Evidence Gate Core Milestone](../../agent-roadmap/archive/phase/knowledge-tool-optimization-extension/milestones/stream-evidence-gate-core.md)의 request-local runtime을 사용한다. runtime은 response status/header와 opening event를 첫 safe release까지 보류하고, filter 결과를 모두 모은 뒤 release, terminal 또는 bounded recovery 중 하나만 실행한다. 기본값 `false`에서는 기존 compatibility 경로를 유지한다.
|
||||
|
||||
복구 요청 조립 또는 dispatch가 실패하면 endpoint별 오류 하나만 보낸다. 내부 원인 사슬은 raw stack trace, provider endpoint/body, user prompt, output/reasoning 원문, tool args/result, 인증 정보를 포함하지 않으며 외부 JSON/SSE에 `causes`, `stack`, `trace` 같은 확장 필드로 노출하지 않는다.
|
||||
|
||||
Core 활성화만으로 반복, missing tool-call, schema 같은 semantic filter가 자동 활성화되지는 않는다. 현재 production registry는 공통 mechanics와 적용 가능한 request-local tool validation을 제공하며, 추가 semantic filter와 corrective directive는 각 소비 Milestone이 등록한다.
|
||||
Core activation does not automatically enable a semantic detector. Only `repeat_guard`, `schema_gate`, and `provider_error` explicitly present in `openai.stream_evidence_gate.filters[]` enter the request-start registry; `schema_gate` participates only when `metadata.scheme` is present. Filter selection depends on endpoint, environment, model group/model, actual provider, and execution path, never on a caller, SDK, or agent product name.
|
||||
|
||||
When a selected continuation plan addresses the request-local recovery source, the endpoint Rebuilder constructs a new request from recorded assistant content/reasoning and the fixed English resume directive only. It never copies caller messages, Responses `input`, or caller `instructions`: Chat uses an assistant message followed by the fixed directive, while Responses uses assistant output/reasoning items plus that directive as `instructions`. The raw recorded values are preserved byte-for-byte except for the selected content or reasoning byte cursor that excludes the repeated tail. If the caller omitted `temperature`, continuation attempts use `0.2`, `0.4`, and `0.6` in strategy-attempt order; an explicit caller temperature is preserved. A missing model context window, or a rebuilt prompt plus the fixed completion reserve above that window, fails closed before any replacement dispatch or recovery-budget consumption. This builder does not invoke a translator, local model, or `RecoveryPlanPreparer`.
|
||||
|
||||
`repeat_guard` inspects only the current request's endpoint-native history and current provider stream. Chat reads role-separated `content` and the plain `reasoning_content`, `reasoning`, and `reasoning_text` aliases; Responses reads its own message/reasoning/function-call item shapes. A user occurrence excludes the same assistant anchor. Missing reasoning history remains zero occurrences: Edge does not infer a session, TTL, or lineage. Signed, encrypted, unknown, final-content, tool-argument, and tool-result values are never sanitation targets or observation payloads. Completed identical action/result fingerprints establish no progress; a changed completed result is progress, while a different action alone is not. Tool release or a side-effect boundary disables automatic continuation.
|
||||
|
||||
차단(`blocking`) filter가 실제 target에 적용되면 해당 provider는 policy capability를 광고해야 한다. 후보 모두가 capability를 만족하지 않으면 Edge는 provider dispatch 전에 OpenAI-compatible HTTP `400`과 `error.type="invalid_request_error"`로 종료한다. `observe_only`와 disabled filter는 candidate admission을 막지 않는다. response start/opening event는 blocking filter의 all-complete 결과 전에는 commit하지 않는다. `repeat_guard` actively returns sanitized pass, safe-stop, or continuation decisions from the configured Unicode rolling window (500 runes by default) and committed look-behind. A continuation keeps the already released prefix, removes the repeated pending tail, suppresses one byte-identical replacement opening/prefix, and emits one final endpoint terminal marker. `schema_gate` and `provider_error` remain lifecycle foundations until their matcher Tasks are implemented; an unmatched provider error never creates exact replay.
|
||||
|
||||
## Responses API
|
||||
|
||||
|
|
|
|||
|
|
@ -4844,6 +4844,84 @@ def work_log_event_cells(line: str) -> list[str] | None:
|
|||
return cells if len(cells) == 9 else None
|
||||
|
||||
|
||||
def merge_work_log_sources(sources: set[Path]) -> str:
|
||||
"""Merge duplicate dispatcher timelines without losing conflicting rows."""
|
||||
allowed_metadata = {
|
||||
"# Milestone Work Log",
|
||||
"## Dispatcher Timeline",
|
||||
"> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.",
|
||||
"> Dispatcher-owned. Workers and reviewers do not edit this section.",
|
||||
"| seq | time | event | task | role | attempt | model | result | locator |",
|
||||
"|---:|---|---|---|---|---:|---|---|---|",
|
||||
}
|
||||
unique_rows: dict[tuple[str, str, str, str, str], tuple[str, ...]] = {}
|
||||
ordered_rows: list[tuple[str, int, int, list[str]]] = []
|
||||
|
||||
for source_index, source in enumerate(sorted(sources)):
|
||||
try:
|
||||
lines = source.read_text(
|
||||
encoding="utf-8",
|
||||
errors="replace",
|
||||
).splitlines()
|
||||
except OSError as exc:
|
||||
raise ValueError(
|
||||
f"WORK_LOG source를 읽을 수 없다: source={source} error={exc}"
|
||||
) from exc
|
||||
for line_number, line in enumerate(lines, start=1):
|
||||
cells = work_log_event_cells(line)
|
||||
if cells is None:
|
||||
if not line.strip() or line.strip() in allowed_metadata:
|
||||
continue
|
||||
raise ValueError(
|
||||
"WORK_LOG 병합 대상에 안전하게 보존할 수 없는 내용이 있다: "
|
||||
f"source={source} line={line_number}"
|
||||
)
|
||||
if cells[0] == "seq" or cells[0].startswith("---"):
|
||||
continue
|
||||
if cells[2] not in {"START", "FINISH"}:
|
||||
raise ValueError(
|
||||
"WORK_LOG 병합 대상에 지원하지 않는 event가 있다: "
|
||||
f"source={source} line={line_number} event={cells[2]}"
|
||||
)
|
||||
try:
|
||||
sequence = int(cells[0])
|
||||
int(cells[5])
|
||||
except ValueError as exc:
|
||||
raise ValueError(
|
||||
"WORK_LOG 병합 대상의 sequence 또는 attempt가 유효하지 않다: "
|
||||
f"source={source} line={line_number}"
|
||||
) from exc
|
||||
key = (cells[2], cells[3], cells[4], cells[5], cells[8])
|
||||
fingerprint = tuple(cells[1:])
|
||||
previous = unique_rows.get(key)
|
||||
if previous is not None:
|
||||
if previous != fingerprint:
|
||||
raise ValueError(
|
||||
"WORK_LOG 병합 충돌: "
|
||||
f"event={cells[2]} task={cells[3]} role={cells[4]} "
|
||||
f"attempt={cells[5]} locator={cells[8]}"
|
||||
)
|
||||
continue
|
||||
unique_rows[key] = fingerprint
|
||||
ordered_rows.append((cells[1], source_index, sequence, cells))
|
||||
|
||||
if not ordered_rows:
|
||||
raise ValueError("WORK_LOG 병합 대상에 timeline row가 없다")
|
||||
|
||||
header = (
|
||||
"# Milestone Work Log\n\n"
|
||||
"> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n"
|
||||
"| seq | time | event | task | role | attempt | model | result | locator |\n"
|
||||
"|---:|---|---|---|---|---:|---|---|---|\n"
|
||||
)
|
||||
rendered_rows: list[str] = []
|
||||
for sequence, (_, _, _, cells) in enumerate(sorted(ordered_rows), start=1):
|
||||
escaped = [str(value).replace("|", r"\|").replace("\n", " ") for value in cells]
|
||||
escaped[0] = str(sequence)
|
||||
rendered_rows.append("| " + " | ".join(escaped) + " |\n")
|
||||
return header + "".join(rendered_rows)
|
||||
|
||||
|
||||
def unfinished_work_log_attempts(path: Path) -> list[dict[str, Any]]:
|
||||
"""Return START rows that have no matching FINISH row."""
|
||||
try:
|
||||
|
|
@ -5018,12 +5096,6 @@ def archive_completed_group_work_logs(
|
|||
}
|
||||
if not sources:
|
||||
continue
|
||||
if len(sources) != 1:
|
||||
errors[task_group] = (
|
||||
"active/archive WORK_LOG source가 여러 개라 자동 병합할 수 없다: "
|
||||
+ ",".join(str(path) for path in sorted(sources))
|
||||
)
|
||||
continue
|
||||
target_directory = completed_group_archive_directory(
|
||||
task_group,
|
||||
task_names,
|
||||
|
|
@ -5043,19 +5115,45 @@ def archive_completed_group_work_logs(
|
|||
if destination.exists():
|
||||
errors[task_group] = (
|
||||
"WORK_LOG archive destination이 이미 존재한다: "
|
||||
f"source={source} destination={destination}"
|
||||
"sources="
|
||||
+ ",".join(str(path) for path in sorted(sources))
|
||||
+ f" destination={destination}"
|
||||
)
|
||||
continue
|
||||
source = next(iter(sources))
|
||||
temporary = destination.with_name(destination.name + ".tmp")
|
||||
try:
|
||||
close_unfinished_work_log_attempts(source)
|
||||
source.replace(destination)
|
||||
except OSError as exc:
|
||||
if len(sources) == 1:
|
||||
close_unfinished_work_log_attempts(source)
|
||||
source.replace(destination)
|
||||
else:
|
||||
if temporary.exists():
|
||||
raise OSError(
|
||||
"WORK_LOG archive temporary destination이 이미 존재한다: "
|
||||
f"temporary={temporary}"
|
||||
)
|
||||
temporary.write_text(
|
||||
merge_work_log_sources(sources),
|
||||
encoding="utf-8",
|
||||
)
|
||||
close_unfinished_work_log_attempts(temporary)
|
||||
temporary.replace(destination)
|
||||
for merged_source in sources:
|
||||
merged_source.unlink()
|
||||
except (OSError, ValueError) as exc:
|
||||
if temporary.exists():
|
||||
try:
|
||||
temporary.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
errors[task_group] = (
|
||||
f"WORK_LOG archive 실패: source={source} "
|
||||
"WORK_LOG archive 실패: sources="
|
||||
+ ",".join(str(path) for path in sorted(sources))
|
||||
+ " "
|
||||
f"destination={destination} error={exc}"
|
||||
)
|
||||
continue
|
||||
if source == active_source:
|
||||
if active_source in sources:
|
||||
for task_name in sorted(task_names, reverse=True):
|
||||
if "/" not in task_name:
|
||||
continue
|
||||
|
|
|
|||
|
|
@ -5629,6 +5629,100 @@ class WorkLogArchiveTest(unittest.TestCase):
|
|||
"legacy timeline\n",
|
||||
)
|
||||
|
||||
def test_merges_active_and_archived_work_logs_after_review_move(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
workspace = Path(temporary)
|
||||
active = workspace / "agent-task" / "single" / dispatch.WORK_LOG_NAME
|
||||
archive = self.complete_archive(workspace, "single")
|
||||
legacy = archive / dispatch.WORK_LOG_NAME
|
||||
locator = workspace / "runs" / "review" / "locator.json"
|
||||
locator_text = str(locator.resolve())
|
||||
legacy.parent.mkdir(parents=True, exist_ok=True)
|
||||
legacy.write_text(
|
||||
"# Milestone Work Log\n\n"
|
||||
"> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n"
|
||||
"| seq | time | event | task | role | attempt | model | result | locator |\n"
|
||||
"|---:|---|---|---|---|---:|---|---|---|\n"
|
||||
f"| 1 | 26-07-30 06:34:00 | START | single | review | 0 | codex | running | {locator_text} |\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
active.parent.mkdir(parents=True, exist_ok=True)
|
||||
active.write_text(
|
||||
"# Milestone Work Log\n\n"
|
||||
"> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n"
|
||||
"| seq | time | event | task | role | attempt | model | result | locator |\n"
|
||||
"|---:|---|---|---|---|---:|---|---|---|\n"
|
||||
f"| 1 | 26-07-30 06:43:00 | FINISH | single | review | 0 | codex | succeeded:0 | {locator_text} |\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
archived, errors = dispatch.archive_completed_group_work_logs(
|
||||
workspace,
|
||||
{"single"},
|
||||
{"single": str(archive)},
|
||||
set(),
|
||||
)
|
||||
|
||||
destination = archive / "work_log_0.log"
|
||||
self.assertEqual(errors, {})
|
||||
self.assertEqual(
|
||||
archived,
|
||||
{"single": str(destination.resolve())},
|
||||
)
|
||||
self.assertFalse(active.exists())
|
||||
self.assertFalse(legacy.exists())
|
||||
self.assertEqual(
|
||||
len(
|
||||
[
|
||||
line
|
||||
for line in destination.read_text(encoding="utf-8").splitlines()
|
||||
if (cells := dispatch.work_log_event_cells(line))
|
||||
and cells[2] in {"START", "FINISH"}
|
||||
]
|
||||
),
|
||||
2,
|
||||
)
|
||||
self.assertEqual(dispatch.unfinished_work_log_attempts(destination), [])
|
||||
|
||||
def test_preserves_sources_when_work_log_merge_rows_conflict(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
workspace = Path(temporary)
|
||||
active = workspace / "agent-task" / "single" / dispatch.WORK_LOG_NAME
|
||||
archive = self.complete_archive(workspace, "single")
|
||||
legacy = archive / dispatch.WORK_LOG_NAME
|
||||
locator = workspace / "runs" / "worker" / "locator.json"
|
||||
locator_text = str(locator.resolve())
|
||||
header = (
|
||||
"# Milestone Work Log\n\n"
|
||||
"> Dispatcher-owned execution timeline. Workers and reviewers do not edit this file.\n\n"
|
||||
"| seq | time | event | task | role | attempt | model | result | locator |\n"
|
||||
"|---:|---|---|---|---|---:|---|---|---|\n"
|
||||
)
|
||||
legacy.write_text(
|
||||
header
|
||||
+ f"| 1 | 26-07-30 06:34:00 | START | single | worker | 0 | agy | running | {locator_text} |\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
active.parent.mkdir(parents=True, exist_ok=True)
|
||||
active.write_text(
|
||||
header
|
||||
+ f"| 1 | 26-07-30 06:35:00 | START | single | worker | 0 | agy | running | {locator_text} |\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
archived, errors = dispatch.archive_completed_group_work_logs(
|
||||
workspace,
|
||||
{"single"},
|
||||
{"single": str(archive)},
|
||||
set(),
|
||||
)
|
||||
|
||||
self.assertEqual(archived, {})
|
||||
self.assertIn("WORK_LOG 병합 충돌", errors["single"])
|
||||
self.assertTrue(active.exists())
|
||||
self.assertTrue(legacy.exists())
|
||||
self.assertFalse((archive / "work_log_0.log").exists())
|
||||
|
||||
def test_archive_failure_preserves_source_and_reports_retryable_error(self):
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
workspace = Path(temporary)
|
||||
|
|
|
|||
|
|
@ -34,9 +34,9 @@ Phase를 가로지르는 실제 다음 작업 선택은 [전역 마일스톤 실
|
|||
- 경로: [stream-evidence-gate-core](../../archive/phase/knowledge-tool-optimization-extension/milestones/stream-evidence-gate-core.md)
|
||||
- 요약: codec의 response-start/event를 첫 safe release까지 stage하고 500-rune rolling, bounded terminal/fragment hold, pre-read 기본값/절대 상한 16 MiB raw-canonical ingress snapshot과 request-snapshot 기반 Filter Registry를 제공한다. Gate Coordinator가 single-flight all-complete evaluation/commit을, RecoveryPlan Coordinator와 host adapter가 strategy별 budget과 최초 실행 제외 기본값/절대 상한 3회의 request 전체 cap 아래 abort·optional one-shot plan prepare·lossless rebuild·cycle별 single re-admission을 담당한다.
|
||||
|
||||
- [계획] OpenAI-compatible 출력 검증 필터
|
||||
- [진행중] OpenAI-compatible 출력 검증 필터
|
||||
- 경로: [openai-compatible-output-validation-filters](milestones/openai-compatible-output-validation-filters.md)
|
||||
- 요약: OpenAI-compatible Chat Completions와 Responses provider stream의 반복, assistant-history anchor, 동일 tool/action, schema/provider error를 caller-neutral하게 판정하는 Core `Filter` 구현체를 제공한다. filter는 model/provider별 on/off와 semantic decision/RecoveryIntent만 소유하고, 병렬 평가·all-complete arbitration·retry budget·request rebuild/re-admission은 Stream Evidence Gate Core의 공통 Coordinator를 소비한다.
|
||||
- 요약: 실제 의미 필터 전에 local/dev deterministic diagnostic mock으로 실제 codec/Core/Arbiter/recovery/ReleaseSink의 pass·observe-only·blocking recovery와 raw-free timeline을 관측하는 smoke를 선행한다. 이후 OpenAI-compatible Chat Completions와 Responses provider stream의 반복, assistant-history anchor, 동일 tool/action, schema/provider error를 caller-neutral하게 판정하는 Core `Filter` 구현체를 제공한다. filter는 model/provider별 on/off와 semantic decision/RecoveryIntent만 소유하고, 병렬 평가·all-complete arbitration·retry budget·request rebuild/re-admission은 Stream Evidence Gate Core의 공통 Coordinator를 소비한다.
|
||||
|
||||
- [계획] OpenAI-compatible Incomplete Tool Call Syntax Gate
|
||||
- 경로: [openai-compatible-incomplete-tool-call-syntax-gate](milestones/openai-compatible-incomplete-tool-call-syntax-gate.md)
|
||||
|
|
|
|||
|
|
@ -9,10 +9,11 @@
|
|||
|
||||
OpenAI-compatible Chat Completions와 Responses provider 경로에서 모델 출력/행동 이상을 caller 종류와 무관하게 request history와 provider response stream으로 감지하고, 사용자 경험을 해치지 않는 방식으로 관찰/보정/중단/재시도/검증한다.
|
||||
반복 루프는 단일 stream content 반복, request history에 누적된 assistant-only anchor 반복, 동일 tool/action 반복을 모두 포함한다. 단일 stream content 반복은 streaming passthrough를 유지한 채 upstream만 교체해 continuation repair로 이어 쓴다. assistant history anchor는 endpoint codec이 보존한 표준 role/channel provenance를 기준으로 탐지하고, caller가 reasoning history를 재전송하지 않거나 conversation identity가 없으면 존재하지 않는 cross-request state를 추론하지 않는다. D01은 반복된 plain reasoning만 sanitize/live dedupe하고 no-progress·tool 미release·side-effect 비해당일 때 원문 safe prefix를 보존한 단계 복구를 허용하도록 확정됐다. D05는 언어 판별·번역 모델 호출을 제거하고 고정 영어 지시문으로 복구 요청을 직접 조립하도록 확정됐다. tool/action 반복은 `tool name + normalized args` fingerprint와 완료된 이전 tool result의 동일/no-progress 신호를 기준으로 감지해 side-effect 안전성이 없으면 repair 대신 안전 중단한다. `metadata.scheme` JSON 출력 계약은 검증 전 downstream content streaming을 막는 `contract_schema` 경로로 처리한다.
|
||||
실제 repeat/schema/provider-error 의미 필터 구현에 앞서 deterministic diagnostic filter로 pass, observe-only violation, blocking violation과 단일 recovery를 실제 codec/Core/Arbiter/ReleaseSink 경로에서 한 명령으로 관측하는 smoke를 선행 gate로 둔다.
|
||||
|
||||
## 상태
|
||||
|
||||
[계획]
|
||||
[진행중]
|
||||
|
||||
## 승격 조건
|
||||
|
||||
|
|
@ -49,18 +50,25 @@ OpenAI-compatible Chat Completions와 Responses provider 경로에서 모델 출
|
|||
- `metadata.scheme` JSON schema 계약 수신, 마지막 user message prompt append, hard bound가 있는 `terminal_gate` validation, schema 위반 시 bounded retry
|
||||
- `passthrough`, `passthrough_guarded`, `contract_schema` 내부 response path 구분과 실행 로그/관측 기준. 이 이름들은 caller가 지정하는 공개 request field가 아니라 IOP 내부 경로/로그 기준이다.
|
||||
- provider-pool의 raw tunnel/normalized RunEvent 실행 경로는 유지하되 두 path의 endpoint codec이 같은 Core event 계약으로 수렴하고, CLI adapter protocol 변경은 분리하는 책임 경계
|
||||
- local/dev에서만 명시적으로 활성화되는 deterministic diagnostic filter와 provider stream fixture. mock은 판정만 제어하고 Chat/Responses codec, Stream Evidence Gate Core, all-complete Arbiter, Recovery Coordinator, 실제 ReleaseSink와 raw-free `FilterObservation` sink는 production 구현을 그대로 사용한다.
|
||||
|
||||
## 기능
|
||||
|
||||
### Epic: [stream-gate-observability] Observable Stream Gate Diagnostic
|
||||
|
||||
실제 의미 필터 구현 전에 공통 stream pipeline의 지연 평가·release·recovery를 운영자가 반복 관측할 수 있는 선행 smoke를 묶는다.
|
||||
|
||||
- [x] [observable-core-smoke] 한 명령으로 실행 가능한 local/dev diagnostic smoke가 deterministic provider stream과 diagnostic `Filter` mock을 실제 Chat/Responses codec, Stream Evidence Gate Core, all-complete Arbiter, Recovery Coordinator, `ReleaseSink`, raw-free `FilterObservation` sink에 연결한다. mock은 `pass`, `observe_only` violation, pre-release `blocking` violation 뒤 recovery 성공의 판정만 제어하고, 외부 caller request로 활성화할 수 없으며 production 기본 Registry에는 등록하지 않는다. 검증: 모든 provider batch가 평가 전 stage되고, pass와 observe-only는 `response_staged -> filter_evaluation_started -> filter_evaluated -> arbitration_decided -> release_committed -> terminal_committed`, blocking은 `response_staged -> filter_evaluation_started -> filter_evaluated -> arbitration_decided -> recovery_plan_selected -> recovery_attempt_aborted -> recovery_rebuilt -> recovery_dispatched` 뒤 새 attempt의 stage/evaluate/release/terminal 순서를 구조화된 결과로 보여 준다. stable correlation, attempt 전환, 출력·terminal 중복 부재와 raw prompt/output/tool args/result/auth 미기록까지 확인한 뒤에만 `repeat-guard`, `schema-contract`, `provider-error-retry` 의미 필터 구현을 시작한다.
|
||||
|
||||
### Epic: [output-filter] Output Filter Runtime Foundation
|
||||
|
||||
OpenAI-compatible 출력 필터의 계약, 공통 pipeline, endpoint codec, Stream Evidence Gate 채택과 적용 정책 기반을 묶는다.
|
||||
|
||||
- [ ] [contract-doc] OpenAI-compatible 계약 문서와 구현 타입이 `/v1/chat/completions`와 `/v1/responses`의 `metadata.scheme`, 내부 `contract_schema`/`passthrough_guarded`/provider-error-retry path, provider-pool tunnel/normalized RunEvent 실행 경계, caller-neutral 반복 guard 입력, bounded raw-canonical request snapshot과 conversation identity 부재 시 degraded behavior를 설명한다. initial ingress overflow는 HTTP 413 `invalid_request_error`, internal rebuild overflow는 commit state에 맞는 recovery terminal error로 구분한다. raw tunnel provider를 normalized 실행으로 강제하지 않지만 두 path는 codec 뒤 같은 Core event/recovery 계약을 사용한다. 구현 시 [openai-compatible-api.md](../../../../agent-contract/outer/openai-compatible-api.md)와 [edge-config-runtime-refresh.md](../../../../agent-contract/inner/edge-config-runtime-refresh.md)를 코드/Go handler·config type 테스트와 함께 갱신하고 caller 제품명을 runtime 조건으로 사용하지 않는다.
|
||||
- [ ] [filter-pipeline] repeat/schema/provider-error filter가 Core Go `Filter` interface로 등록된다. trigger-ready filter는 동일 immutable batch에서 병렬 평가되고 terminal trigger 미충족 schema는 blocking deferred, error event가 없는 provider-error는 nonblocking not-applicable outcome으로 complete set에 포함된다. repeat는 rolling, schema는 bounded terminal, provider-error는 hold 없는 event subscription을 선언하고 enforcement/failure mode는 Registry가 소유한다. 검증: `go test ./apps/edge/internal/openai -count=1`에서 evaluated/deferred/not-applicable 구분, no deferred-pass, all-complete/failure policy와 direct-submit 금지가 통과한다.
|
||||
- [ ] [stream-gate-adoption] Chat Completions와 Responses codec이 response status/header/opening event까지 normalized event로 만들고, Edge adapter가 Core `AttemptDispatcher`/`AttemptController`/`ReleaseSink`를 구현한다. 두 handler의 unbounded `io.ReadAll` 앞에 host pre-read limit를 적용하고 limit+1 overflow를 413으로 구분한다. SnapshotBuilder/Rebuilder는 typed view 추가 직후와 rebuild 임시 output 할당 전후의 current peak retained bytes를 다시 확인하고 종료/cancel/overflow에서 object를 release한다. provider response-start와 normalized live SSE role chunk는 첫 safe release 전 commit/flush하지 않으며 hop-by-hop/변환 전 `Content-Length`는 전달하지 않는다. Core가 rolling/terminal/fragment hold, all-complete, idle no-release와 single action을 소유한다. 검증: Chat/Responses raw-body limit-1/limit/limit+1, exact-limit body+typed-view overflow, rebuild 전후 peak/no full-read overflow/release, tunnel/normalized 두 path의 eager header/role 없음, header allowlist/content-length 제거, staged 500 retry, backpressure, path switch와 single terminal fixture가 통과한다.
|
||||
- [ ] [responses-codec] `/v1/responses`의 input item history와 response-start/text/reasoning/function-call/terminal event를 normalized event와 repair input으로 변환하고, Core의 기본값/절대 상한 16 MiB `max_ingress_snapshot_bytes` 안에서 raw body 하나를 canonical source로 unknown caller item/field를 보존하는 bounded lossless Responses `RequestRebuilder`를 제공한다. raw parser와 serializer는 Chat과 분리하고 concrete model/auth rewrite는 dispatcher admission에 둔다. 검증: Responses raw body round-trip/unknown field, stream item split, staged opening event, reasoning/encrypted reasoning 보존, function-call, terminal/error, path-switch recovery가 Chat과 같은 semantic decision/plan을 내고 endpoint shape를 유지하며 retained/rebuild limit 초과는 no-dispatch로 끝난다.
|
||||
- [ ] [filter-policy] filter enable/disable, `blocking|observe_only`, hold mode/bound를 environment(`dev`, `dev-corp`)와 model group/model/provider/protocol capability별로 평가한다. request 시작 시 config generation과 required capability를 고정해 schema 같은 필수 filter를 지원하지 않는 provider 후보는 admission 전에 제외하고, actual target별 active set은 attempt마다 같은 snapshot으로 다시 resolve한다. 검증: Chat/Responses와 qwen/gemma/ornith fixture에서 policy precedence, mid-request reload 격리, provider 전환 re-resolution, required no-candidate 400, disabled/duplicate filter 미평가와 caller-neutral 분기가 통과한다.
|
||||
- [x] [contract-doc] OpenAI-compatible 계약 문서와 구현 타입이 `/v1/chat/completions`와 `/v1/responses`의 `metadata.scheme`, 내부 `contract_schema`/`passthrough_guarded`/provider-error-retry path, provider-pool tunnel/normalized RunEvent 실행 경계, caller-neutral 반복 guard 입력, bounded raw-canonical request snapshot과 conversation identity 부재 시 degraded behavior를 설명한다. initial ingress overflow는 HTTP 413 `invalid_request_error`, internal rebuild overflow는 commit state에 맞는 recovery terminal error로 구분한다. raw tunnel provider를 normalized 실행으로 강제하지 않지만 두 path는 codec 뒤 같은 Core event/recovery 계약을 사용한다. 구현 시 [openai-compatible-api.md](../../../../agent-contract/outer/openai-compatible-api.md)와 [edge-config-runtime-refresh.md](../../../../agent-contract/inner/edge-config-runtime-refresh.md)를 코드/Go handler·config type 테스트와 함께 갱신하고 caller 제품명을 runtime 조건으로 사용하지 않는다.
|
||||
- [x] [filter-pipeline] repeat/schema/provider-error filter가 Core Go `Filter` interface로 등록된다. trigger-ready filter는 동일 immutable batch에서 병렬 평가되고 terminal trigger 미충족 schema는 blocking deferred, error event가 없는 provider-error는 nonblocking not-applicable outcome으로 complete set에 포함된다. repeat는 rolling, schema는 bounded terminal, provider-error는 hold 없는 event subscription을 선언하고 enforcement/failure mode는 Registry가 소유한다. 검증: `go test ./apps/edge/internal/openai -count=1`에서 evaluated/deferred/not-applicable 구분, no deferred-pass, all-complete/failure policy와 direct-submit 금지가 통과한다.
|
||||
- [x] [stream-gate-adoption] Chat Completions와 Responses codec이 response status/header/opening event까지 normalized event로 만들고, Edge adapter가 Core `AttemptDispatcher`/`AttemptController`/`ReleaseSink`를 구현한다. 두 handler의 unbounded `io.ReadAll` 앞에 host pre-read limit를 적용하고 limit+1 overflow를 413으로 구분한다. SnapshotBuilder/Rebuilder는 typed view 추가 직후와 rebuild 임시 output 할당 전후의 current peak retained bytes를 다시 확인하고 종료/cancel/overflow에서 object를 release한다. provider response-start와 normalized live SSE role chunk는 첫 safe release 전 commit/flush하지 않으며 hop-by-hop/변환 전 `Content-Length`는 전달하지 않는다. Core가 rolling/terminal/fragment hold, all-complete, idle no-release와 single action을 소유한다. 검증: Chat/Responses raw-body limit-1/limit/limit+1, exact-limit body+typed-view overflow, rebuild 전후 peak/no full-read overflow/release, tunnel/normalized 두 path의 eager header/role 없음, header allowlist/content-length 제거, staged 500 retry, backpressure, path switch와 single terminal fixture가 통과한다.
|
||||
- [x] [responses-codec] `/v1/responses`의 input item history와 response-start/text/reasoning/function-call/terminal event를 normalized event와 repair input으로 변환하고, Core의 기본값/절대 상한 16 MiB `max_ingress_snapshot_bytes` 안에서 raw body 하나를 canonical source로 unknown caller item/field를 보존하는 bounded lossless Responses `RequestRebuilder`를 제공한다. raw parser와 serializer는 Chat과 분리하고 concrete model/auth rewrite는 dispatcher admission에 둔다. 검증: Responses raw body round-trip/unknown field, stream item split, staged opening event, reasoning/encrypted reasoning 보존, function-call, terminal/error, path-switch recovery가 Chat과 같은 semantic decision/plan을 내고 endpoint shape를 유지하며 retained/rebuild limit 초과는 no-dispatch로 끝난다.
|
||||
- [x] [filter-policy] filter enable/disable, `blocking|observe_only`, hold mode/bound를 environment(`dev`, `dev-corp`)와 model group/model/provider/protocol capability별로 평가한다. request 시작 시 config generation과 required capability를 고정해 schema 같은 필수 filter를 지원하지 않는 provider 후보는 admission 전에 제외하고, actual target별 active set은 attempt마다 같은 snapshot으로 다시 resolve한다. 검증: Chat/Responses와 qwen/gemma/ornith fixture에서 policy precedence, mid-request reload 격리, provider 전환 re-resolution, required no-candidate 400, disabled/duplicate filter 미평가와 caller-neutral 분기가 통과한다.
|
||||
|
||||
### Epic: [output-filter-recovery] Output Filter Recovery and Evidence
|
||||
|
||||
|
|
@ -103,11 +111,13 @@ OpenAI-compatible 출력 필터의 계약, 공통 pipeline, endpoint codec, Stre
|
|||
- assistant final `content`를 history anchor fingerprint만으로 조용히 삭제하는 방식
|
||||
- schema 계약이 있는 요청에서 검증 전 content delta를 사용자에게 먼저 노출하는 방식
|
||||
- validator 모델을 이용한 자연어/tool-call 판정 gate
|
||||
- diagnostic filter를 production 기본 Registry에 등록하거나 OpenAI-compatible public request field로 선택·활성화하는 기능
|
||||
- 누적 요청 컨텍스트 최적화, 장기 RAG, advisor/Context Hook, workflow routing 정책
|
||||
- 오류 사건의 cross-request 저장·중복 count, 연결 저장소 분석, 범용 수정 제안, 프로젝트별 마일스톤/작업 문서 생성, 사용자 승인, 격리 수정·테스트·독립 검토, 변경 요청·병합·배포·재발 관찰을 수행하는 별도 플랫폼 구현
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 표준선(선택): 첫 구현 단위는 `observable-core-smoke`다. diagnostic mock은 filter 판정만 결정론적으로 바꾸고 실제 codec/Core/Arbiter/recovery/ReleaseSink/observation 경로는 대체하지 않는다. pass·observe-only·blocking recovery의 구조화된 timeline과 출력/terminal 단일성·raw-free invariant가 관측되기 전에는 실제 의미 필터 구현으로 넘어가지 않는다.
|
||||
- 관련 경로: `packages/go/streamgate`, `apps/edge/internal/openai`, `apps/edge/internal/service`, `apps/node/internal/adapters/openai_compat`, `apps/node/internal/runtime`, `packages/go/config`, [openai-compatible-api.md](../../../../agent-contract/outer/openai-compatible-api.md), [Stream Evidence Gate Core](stream-evidence-gate-core.md)
|
||||
- 표준선(선택): `provider_length_gate`는 provider가 한 attempt의 output cap에 도달한 terminal reason만 caller-neutral하게 판정한다. managed profile에서는 그 terminal을 외부 오류로 전달하지 않고 Core의 stream-open continuation을 요청한다. provider attempt cap은 작은 운영 단위로 두고, IOP의 논리 trajectory는 original request와 assistant prefix를 조립한 뒤 측정한 rebuilt prompt와 reserve를 뺀 context window 여유, 그리고 caller가 명시한 논리 output cap까지만 확장한다. provider attempt cap은 이와 별개인 내부 운영 단위다. provider가 assistant prefill을 지원하면 우선 사용하고, 지원하지 않으면 고정 internal continuation directive로 같은 assistant prefix를 재구성한다. 이는 일반 오류 재시도와 다른 progress continuation이므로 Core의 fault recovery 3회 cap을 소비하지 않으며, context 여유·취소·완성 tool call·side effect·미완성 fragment 실패에서는 final terminal로 수렴한다. context 여유 또는 caller logical cap 소진의 경우에만 endpoint가 지원하는 logical `length` terminal과 단일 `[DONE]`을 한 번 전달하며, attempt 중간 `length`는 전달하지 않는다.
|
||||
- 표준선(선택): 반복루프 필터는 `rolling_window` passthrough consumer이며 이미 흘린 정상 prefix를 버리지 않는다. 반복 감지 시 continuation directive/온도 후보/반복 span만 반환하고, Core가 committed look-behind/release cursor를 보존해 current attempt abort, 고정 영어 지시문을 포함한 endpoint별 rebuild와 cycle별 single re-admission을 수행한다. 새 attempt의 response-start/role과 이미 보낸 prefix는 downstream에 중복하지 않는다.
|
||||
|
|
|
|||
|
|
@ -11,23 +11,23 @@
|
|||
현재 Python 감시·dispatcher와 Node CLI runtime의 전체 동등성을 단일 Go CLI Provider·AgentTaskManager 및 독립 `iop-agent` binary로 이전한다.
|
||||
|
||||
3. [OpenAI-compatible 출력 검증 필터](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-output-validation-filters.md)
|
||||
OpenAI-compatible single-stream 반복과 incoming request history에 누적된 assistant 반복, JSON contract 검증/repair 경로를 안정화한다.
|
||||
실제 의미 필터 전에 deterministic diagnostic mock으로 실제 Stream Evidence Gate의 pass·observe-only·blocking recovery를 관측하는 smoke를 통과시키고, OpenAI-compatible single-stream 반복과 incoming request history에 누적된 assistant 반복, JSON contract 검증/repair 경로를 안정화한다.
|
||||
|
||||
4. [Provider-Device-Model Qualification 리포트와 Lifecycle 관리](phase/operational-observability-provider-management/milestones/provider-device-model-qualification-report.md)
|
||||
provider/device/model별 compatibility, performance, quality, lifecycle 리포트 경계를 정리한다.
|
||||
|
||||
5. [OpenAI-compatible Incomplete Tool Call Syntax Gate](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-incomplete-tool-call-syntax-gate.md)
|
||||
4. [OpenAI-compatible Incomplete Tool Call Syntax Gate](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-incomplete-tool-call-syntax-gate.md)
|
||||
terminal provider 응답의 incomplete tool-call syntax를 deterministic하게 판정한다.
|
||||
|
||||
6. [OpenAI-compatible Runtime Output Integrity Filter](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-runtime-output-integrity-filter.md)
|
||||
5. [OpenAI-compatible Runtime Output Integrity Filter](phase/knowledge-tool-optimization-extension/milestones/openai-compatible-runtime-output-integrity-filter.md)
|
||||
terminal output invariant와 공통 filter/retry pipeline을 정의한다.
|
||||
|
||||
7. [LLM 판별 기반 Missing Tool Call 재시도 Gate](phase/knowledge-tool-optimization-extension/milestones/llm-judged-missing-tool-call-retry-gate.md)
|
||||
6. [LLM 판별 기반 Missing Tool Call 재시도 Gate](phase/knowledge-tool-optimization-extension/milestones/llm-judged-missing-tool-call-retry-gate.md)
|
||||
tool 사용 의도 누락 케이스를 LLM judge와 buffered retry 후보로 검토한다.
|
||||
|
||||
8. [Tool Call 판정 모델 Gate 리뷰](phase/knowledge-tool-optimization-extension/milestones/tool-call-validator-model-gate-review.md)
|
||||
7. [Tool Call 판정 모델 Gate 리뷰](phase/knowledge-tool-optimization-extension/milestones/tool-call-validator-model-gate-review.md)
|
||||
schema만으로 어려운 tool-call 후보에 validator 모델을 쓸지 검토한다.
|
||||
|
||||
8. [Provider 부하 메트릭과 Live Queue Dashboard](phase/operational-observability-provider-management/milestones/provider-load-metrics-queue-dashboard.md)
|
||||
Edge provider-pool의 capacity, in-flight, queued와 queue wait를 Prometheus/Grafana로 관측해 provider별 live 부하와 적체·회복을 분석한다.
|
||||
|
||||
9. [Pi CLI Provider Integration](phase/automation-runtime-bridge/milestones/pi-cli-provider-integration.md)
|
||||
Pi를 Node CLI provider 실행 후보에 추가하고 OpenAI-compatible route smoke로 안정화한다.
|
||||
|
||||
|
|
@ -64,26 +64,29 @@
|
|||
20. [Provider Runtime 설정과 모델 획득 오케스트레이션](phase/operational-observability-provider-management/milestones/provider-runtime-model-acquisition-orchestration.md)
|
||||
provider runtime launch/profile, model download/cache/verification 경계를 스케치한다.
|
||||
|
||||
21. [Provider 입력 컨텍스트 선택과 축소](phase/knowledge-tool-optimization-extension/milestones/request-context-assembly-optimization.md)
|
||||
21. [Provider-Device-Model Qualification 리포트와 Lifecycle 관리](phase/operational-observability-provider-management/milestones/provider-device-model-qualification-report.md)
|
||||
provider/device/model별 compatibility, performance, quality, lifecycle 리포트 경계를 정리한다.
|
||||
|
||||
22. [Provider 입력 컨텍스트 선택과 축소](phase/knowledge-tool-optimization-extension/milestones/request-context-assembly-optimization.md)
|
||||
provider dispatch 전에 무관한 과거 요청-답변 단위를 제거하고, 유지한 답변·tool/search 결과 안에서도 필요한 문단·코드 블록·구간만 남기는 입력 context 최적화를 스케치한다.
|
||||
|
||||
22. [장기 기억과 RAG 업데이트 사이클 (2차)](phase/knowledge-tool-optimization-extension/milestones/long-term-memory-rag-second-wave.md)
|
||||
23. [장기 기억과 RAG 업데이트 사이클 (2차)](phase/knowledge-tool-optimization-extension/milestones/long-term-memory-rag-second-wave.md)
|
||||
repo 장기 기억, RAG 저장소, update cycle, MCP 기반 context 절약 후보를 스케치한다.
|
||||
|
||||
23. [Advisor와 Context Hook 확장 (2차)](phase/knowledge-tool-optimization-extension/milestones/advisor-context-hook-second-wave.md)
|
||||
24. [Advisor와 Context Hook 확장 (2차)](phase/knowledge-tool-optimization-extension/milestones/advisor-context-hook-second-wave.md)
|
||||
advisor 역할과 여러 기능을 실행 흐름에 연결하는 Context Hook 경계를 스케치한다.
|
||||
|
||||
24. [oto 자동화 스케줄러와 CI-CD 연동 (2차)](phase/automation-runtime-bridge/milestones/oto-automation-scheduler-second-wave.md)
|
||||
25. [oto 자동화 스케줄러와 CI-CD 연동 (2차)](phase/automation-runtime-bridge/milestones/oto-automation-scheduler-second-wave.md)
|
||||
oto 기반 자동화, scheduler, CI-CD 연동 후보를 스케치한다.
|
||||
|
||||
25. [Flutter Desktop Control UI](phase/automation-runtime-bridge/milestones/flutter-desktop-control-ui.md)
|
||||
26. [Flutter Desktop Control UI](phase/automation-runtime-bridge/milestones/flutter-desktop-control-ui.md)
|
||||
`iop-agent` local proto-socket을 소비해 YAML 전체 설정과 project·실행·오류·로그를 관리하는 macOS Flutter 설정·운영 UI를 제공한다.
|
||||
|
||||
26. [Unity 3D Desktop Character](phase/automation-runtime-bridge/milestones/unity-3d-desktop-character.md)
|
||||
27. [Unity 3D Desktop Character](phase/automation-runtime-bridge/milestones/unity-3d-desktop-character.md)
|
||||
같은 local proto-socket을 독립적으로 소비하고 작업 상태를 투명 배경 3D 캐릭터와 animation으로 표현하는 macOS Unity client를 제공한다.
|
||||
|
||||
27. [IOP Hot Path One-shot 실행 경로](phase/knowledge-tool-optimization-extension/milestones/iop-hot-path-one-shot-execution.md)
|
||||
28. [IOP Hot Path One-shot 실행 경로](phase/knowledge-tool-optimization-extension/milestones/iop-hot-path-one-shot-execution.md)
|
||||
외부 `model=iop` 요청을 Gemini 3.6 Flash와 RTX 5090 `ornith-fast`의 bounded one-shot 경로로 처리해 최대 속도와 실사용 품질의 균형을 맞춘다.
|
||||
|
||||
28. [Node Provider 실행 Liveness 관측과 안전 복구](phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md)
|
||||
29. [Node Provider 실행 Liveness 관측과 안전 복구](phase/operational-observability-provider-management/milestones/node-provider-execution-liveness-recovery.md)
|
||||
Node가 5분간 provider 진행이 없는 request를 health와 분리 판정하고 local attempt를 fence한 뒤 기존 recovery owner가 안전한 요청만 공통 budget 안에서 재실행한다.
|
||||
|
|
|
|||
|
|
@ -50,6 +50,7 @@
|
|||
| Shared Replay Base | [Tool Call Runtime 검증 재시도 MVP](../../../archive/phase/knowledge-tool-optimization-extension/milestones/tool-call-runtime-validation-retry.md) | 응답 방출 전 bounded exact replay하는 기존 실행 기반. Stream Evidence Gate Core의 RecoveryPlan Coordinator가 이 경로와 attempt counter를 공통 recovery pipeline으로 흡수하고 provider-error/tool-validation filter는 intent만 제공한다. |
|
||||
| External Provider | OpenAI-compatible provider pool | provider 원본 요청/응답은 IOP 필터 정책에 따라 upstream abort/retry 대상이 된다. |
|
||||
| User Decision | 현재 사용자 요청 및 [user_review_0.log](user_review_0.log) | caller-neutral OpenAI-compatible filter, 별도 sanitized evidence log, `filters[] = [{ code, message }]`, 기존 Tool Call Runtime validation과의 common exact-replay 재사용, 기본 500-rune evidence pending-tail hold, normalized event/evidence tail/release commit/terminal sequence를 [Stream Evidence Gate Core](../stream-evidence-gate-core/SDD.md)의 공통 책임으로 분리하는 결정, D01 history mutation/repair와 `[0.2, 0.4, 0.6]` 온도 단계 복구, D02의 Chat Completions·Responses 동시 적용 및 endpoint별 codec 분리, D03의 기본 원문 기록 `on`과 설정 기반 `off` 전환, D04의 최초 실행 제외 공통 exact-replay 최대 3회와 commit 뒤 no-replay는 확정됐다. 재시도 provider 선택은 기존 provider-pool admission 정책을 따르며 filter가 강제하지 않는다. D05는 언어 판별·번역·로컬 모델 호출을 모두 제거하고 고정 영어 지시문을 직접 사용하는 것으로 확정됐다. 실제 재작업 요청은 반복 구간을 제외한 모델의 content와 think/reasoning 원문을 channel별로 구분해 고정 지시문과 사용하고 원래 사용자 요청·message는 넣지 않는다. D06은 cross-request 오류 사건의 안전한 지문/중복 count, 연결 소스 분석, 중립 수정 제안, 프로젝트별 작업 문서, 사용자 승인, 격리 수정·테스트·독립 검토, 변경 요청·병합·배포·재발 확인을 별도 범용 플랫폼으로 프로젝트화하고 이 Milestone은 raw-free event 방출까지만 맡는 것으로 확정됐다. |
|
||||
| Diagnostic Smoke Decision | 현재 사용자 요청 | 실제 의미 필터보다 먼저 local/dev 전용 deterministic diagnostic `Filter` mock으로 pass, observe-only violation, pre-release blocking violation과 단일 recovery를 관측한다. mock은 판정만 제어하고 Chat/Responses codec, Core, Arbiter, Recovery Coordinator, ReleaseSink, raw-free observation sink는 실제 구현을 사용한다. 외부 caller가 활성화할 수 없고 production 기본 Registry에는 등록하지 않는다. |
|
||||
| Failure Contract | [openai-compatible-api.md](../../../../agent-contract/outer/openai-compatible-api.md), [Stream Evidence Gate Core SDD](../stream-evidence-gate-core/SDD.md) | 내부는 최대 4단계의 raw-free `FailureCauseChain`을 보존하고, endpoint host는 HTTP/Chat SSE/Responses에 endpoint별 단일 terminal 오류만 직렬화한다. |
|
||||
| Managed Length Decision | 현재 사용자 요청 | managed profile은 provider의 작은 attempt `max_tokens`를 사용한다. output-cap `length`는 중간 terminal로 내보내지 않고 original request와 channel별 assistant prefix로 다음 attempt를 rebuild하며, original request와 assistant prefix를 조립한 뒤 측정한 rebuilt prompt token 및 reserve로 계산한 context-window 여유가 논리 trajectory의 유일한 장문 상한이다. prefix는 rebuilt prompt에 이미 포함되므로 별도의 누적 output과 이중 계상하지 않는다. fault recovery 3회 cap과 별도다. caller가 명시한 output cap은 provider attempt cap으로 취급하지 않고 논리 요청 전체에 한 번만 적용한다. |
|
||||
|
||||
|
|
@ -99,6 +100,7 @@
|
|||
- `tool_validation_error`, `schema_validation_error`, `guard_error`: 복구 불가 또는 retry exhausted terminal error.
|
||||
- 내부 filter interface/policy:
|
||||
- 구현은 [Stream Evidence Gate Core](../stream-evidence-gate-core/SDD.md)의 Go `Filter` interface와 shared helper를 사용한다. 각 filter는 stable ID, applicability, hold requirement, context-aware synchronous pure evaluation, sanitized evidence와 선택적 RecoveryIntent만 제공하며 error/cancel/deadline은 Core fail policy로 전달한다.
|
||||
- diagnostic `Filter` mock은 local/dev smoke의 명시적 test seam에서만 등록하고 deterministic decision만 반환한다. 외부 request/config field로 선택할 수 없고 production 기본 Registry에는 포함하지 않으며, codec/Core/Arbiter/recovery/ReleaseSink/observation sink는 mock으로 대체하지 않는다.
|
||||
- repeat/schema/provider-error처럼 사용자 출력 안전성에 직접 관여하는 filter는 기본 `blocking`, dev-corp 사전 관찰용 action rule은 명시적 `observe_only`로 등록할 수 있다. blocking error/deadline은 fatal, observe-only error/deadline은 `observe_error`로 정규화하며 어느 경우에도 silent pass하지 않는다.
|
||||
- Chat/Responses codec은 response-start를 포함한 normalized event와 lossless `RequestRebuilder`를 제공하고 Edge adapter는 Core `AttemptDispatcher`/`AttemptController`/`ReleaseSink`를 구현한다. Core는 hold, all-complete, commit, recovery budget/abort/rebuild 호출/dispatch를 담당하고 filter는 반복/schema/provider-error 의미와 typed intent만 담당한다.
|
||||
- 기본 repeat는 `stream_hold.evidence_runes=500` rolling window, schema는 explicit bounded terminal gate, tool fragment는 fragment gate다. terminal gate 외 기본 경로는 전체 응답을 모으지 않으며 response status/header와 opening role/event도 첫 safe release까지 stage한다. 시간 경과는 release 조건이 아니다.
|
||||
|
|
@ -169,6 +171,7 @@
|
|||
| S20 | `resume-notice-builder` | D01 continuation repair가 허용됐고 content와 think/reasoning에 반복 전 모델 출력이 있다 | all-complete Arbiter가 plan 하나를 선택하고 current attempt ownership을 종료한 뒤 endpoint Rebuilder가 복구 요청을 조립한다 | 사용자 요청·message를 넣지 않고 두 channel의 원문을 구분해 고정 영어 지시문과 직접 조립한다. 출력은 요약·절단·재작성하지 않으며 문맥 한도를 넘으면 dispatch하지 않는다. 언어 판별·번역·로컬 모델·`RecoveryPlanPreparer` 호출은 없고, 실제 outbound recovery dispatch에만 budget을 소비한다. Chat Completions와 Responses가 각 endpoint shape를 보존한다 |
|
||||
| S21 | `stream-gate-adoption` | Chat/Responses ingress raw body가 limit-1, limit, limit+1이거나 typed view/rebuild를 포함한 current peak retained bytes가 limit을 넘는다 | endpoint host가 body를 읽기 전에 overflow를 판정하고 SnapshotBuilder/Rebuilder가 typed view 추가 직후와 rebuild 할당 전후에 다시 계상한다 | limit-1/limit은 pre-read body gate를 통과하고 limit+1은 즉시 거부된다. 이후 initial retained overflow는 raw snapshot을 release한 HTTP 413 `invalid_request_error`, rebuild overflow는 commit state에 맞는 terminal recovery error 하나로 끝난다. 어느 overflow도 provider dispatch/recovery budget 소비/raw request 로그를 만들지 않는다 |
|
||||
| S22 | `length-continuation` | managed profile의 Chat/Responses provider가 16K attempt cap에서 두 번 이상 `length` terminal을 반환하고 safe prefix가 release됐으며 rebuilt prompt/context/reserve와 (설정된 경우) caller logical cap allowance가 남아 있다 | `provider_length_gate`가 `managed_continuation` intent를 반환하고 Rebuilder가 original request와 channel별 assistant prefix를 다음 attempt로 조립한다 | 중간 `length`/`[DONE]`, response-start/role/prefix 중복 없이 같은 stream을 이어 간다. fault recovery 3회 cap은 소비하지 않으며 context 또는 caller logical cap exhaustion 때만 logical `length` terminal과 종료 marker를 한 번 보내고, complete tool boundary, cancel 또는 lossless fragment serialization 불가에서는 endpoint별 final terminal 하나로 수렴한다 |
|
||||
| S23 | `observable-core-smoke` | local/dev diagnostic smoke가 deterministic Chat/Responses provider stream과 `pass`, `observe_only` violation, pre-release `blocking` violation을 반환하는 diagnostic `Filter` mock을 사용한다 | 한 명령으로 실제 codec, Core, all-complete Arbiter, Recovery Coordinator, ReleaseSink와 observation sink를 통과시킨다 | 모든 batch가 평가 전에 stage된다. pass와 observe-only는 stage/evaluate/arbitrate/release/single-terminal로 수렴하고 observe-only violation은 출력 차단 없이 관측된다. blocking은 current attempt를 abort하고 recovery를 한 번 dispatch한 뒤 새 attempt의 출력과 terminal만 한 번 전달한다. 전체 timeline의 correlation은 안정적이고 attempt 전환은 명시되며 raw prompt/output/tool args/result/auth는 구조화된 결과나 일반 로그에 남지 않는다 |
|
||||
|
||||
## Evidence Map
|
||||
|
||||
|
|
@ -196,6 +199,7 @@
|
|||
| S20 | all-complete/abort-before-build, content·think/reasoning channel provenance, exact fixed-English directive, user request/message exclusion, context overflow no-dispatch, Chat/Responses rebuild fixture | `agent-task/m-openai-compatible-output-validation-filters/...` | `Roadmap Completion`에 `resume-notice-builder`, 고정 문구 일치/no translator·local-model·preparer call/no summary·truncation·rewrite/actual dispatch-only budget/endpoint shape assertion |
|
||||
| S21 | Chat/Responses raw-body limit-1/limit/limit+1 pre-read, exact-limit body+typed-view overflow, no-full-read overflow, rebuild pre/post-allocation peak and release fixtures | `agent-task/m-openai-compatible-output-validation-filters/...` | `Roadmap Completion`에 `stream-gate-adoption`, body-gate-vs-total-retained 구분/initial 413/rebuild commit-aware terminal/no-dispatch/no-budget/no-raw-log assertion |
|
||||
| S22 | 16K output-cap `length` 2회 이상, original request + channel prefix/prefill rebuild, same-stream cursor/opening/prefix suppression, context/reserve 또는 caller logical cap exhaustion과 complete tool-call, lossless fragment serialization 가능/불가, cancel fixture | `agent-task/m-openai-compatible-output-validation-filters/...` | `Roadmap Completion`에 `length-continuation`, fault cap과 trajectory budget 분리·no summary/truncation/new user message·single final terminal assertion |
|
||||
| S23 | 한 명령 local/dev diagnostic smoke, deterministic pass/observe-only/blocking provider fixtures, Chat/Responses 실제 codec/Core/Arbiter/recovery/ReleaseSink 연결, ordered raw-free observation timeline | `agent-task/m-openai-compatible-output-validation-filters/...` | `Roadmap Completion`에 `observable-core-smoke`, 모든 batch의 stage-before-evaluate/release, observe-only nonblocking, single abort/recovery, stable correlation/attempt switch, output·terminal 중복 부재와 prompt/output/tool/auth 미기록 assertion |
|
||||
|
||||
## Cross-repo Dependencies
|
||||
|
||||
|
|
@ -233,9 +237,11 @@
|
|||
- 2026-07-23: 사용자의 최종 승인을 반영해 SDD 상태를 `[승인됨]`, SDD 잠금을 `해제`로 전환하고 사용자 리뷰를 [user_review_0.log](user_review_0.log)로 보존했다.
|
||||
- 2026-07-24: 사용자가 provider의 큰 `max_tokens`가 local scheduling·동시성에 주는 부작용을 작은 attempt cap과 IOP managed continuation으로 분리하도록 확정했다. `length` 중간 terminal은 숨기고 원본 요청과 channel별 assistant prefix를 보존해 context-window/reserve가 허용하는 논리 trajectory를 같은 stream에 이어 간다. 이는 fault recovery 3회 cap과 별도이며 요약·문장 경계 절단·새 user message 방식은 사용하지 않는다.
|
||||
- 2026-07-25: 재검증에서 rebuilt prompt가 assistant prefix를 이미 포함하므로 누적 output 이중 계상을 제거했다. provider attempt cap은 내부 운영 단위로만 쓰고 caller 명시 output cap은 논리 요청 전체에 한 번 적용한다. context 또는 caller logical cap 소진은 중간 provider terminal이 아닌 endpoint-native logical `length` terminal 한 번으로 표현하며, 미완성 tool fragment는 lossless serializer가 있는 경우에만 내부 continuation prefix로 사용한다.
|
||||
- 2026-07-28: 사용자가 실제 의미 필터 구현 전에 pipeline 자체를 관측·검증할 deterministic diagnostic mock smoke를 선행 조건으로 확정했다. pass, observe-only violation, blocking violation 뒤 단일 recovery가 실제 codec/Core/Arbiter/ReleaseSink와 raw-free observation 경로를 통과해야 하며 production 기본 등록과 caller 활성화는 금지한다.
|
||||
|
||||
## 작업 컨텍스트
|
||||
|
||||
- 표준선: 첫 구현 단위는 `observable-core-smoke`다. diagnostic mock은 결정론적 filter decision만 제공하고 실제 host codec/Core/Arbiter/recovery/ReleaseSink/observation sink를 대체하지 않는다. pass·observe-only·blocking recovery timeline, stable correlation/attempt switch, 출력·terminal 단일성과 raw-free invariant를 한 명령의 구조화된 smoke 결과로 확인하기 전에는 `repeat-guard`, `schema-contract`, `provider-error-retry` 구현으로 넘어가지 않는다.
|
||||
- 표준선: OpenAI-compatible provider 출력 검증 host adapter는 Edge OpenAI handler/stream bridge에 두고 transport-agnostic Core는 `packages/go/streamgate`에 둔다. normalized 실행 path를 tunnel path로 강제 전환하지 않지만 두 path의 provider output은 endpoint codec 뒤 같은 Core event contract를 사용할 수 있다.
|
||||
- 표준선: `provider_length_gate`는 provider의 output-cap terminal만 caller-neutral하게 판정한다. managed profile은 작은 attempt cap으로 dispatch하고, original request와 content/think/reasoning assistant prefix를 endpoint별 shape로 rebuild한다. prefill capability가 있으면 우선 사용하며 없으면 고정 internal continuation directive만 더한다. Core의 `ManagedTrajectoryBudget`은 rebuilt prompt token과 reserve를 뺀 실제 context-window 여유 및 caller가 명시한 논리 output cap을 장문 상한으로 사용하며, prefix를 누적 output으로 이중 계상하지 않고 fault recovery 3회 cap과 분리한다. provider attempt cap은 내부 운영 단위다.
|
||||
- 표준선: Chat/Responses codec은 response-start 포함 raw parser와 lossless `RequestRebuilder`를, Edge는 `AttemptDispatcher`/`AttemptController`/`ReleaseSink`를 제공한다. Core가 active filter single-flight fan-out/all-complete, deterministic action, commit, budget/abort/rebuild 호출/single re-admission을 담당한다.
|
||||
|
|
|
|||
|
|
@ -18,6 +18,12 @@ source_evidence:
|
|||
- type: code
|
||||
path: apps/edge/internal/openai/stream_gate_runtime.go
|
||||
notes: runtime-enabled Chat과 provider tunnel response lifecycle
|
||||
- type: code
|
||||
path: apps/edge/internal/openai/chat_decode.go
|
||||
notes: Chat role/content/reasoning-alias repeat history decoder
|
||||
- type: code
|
||||
path: apps/edge/internal/openai/responses_decode.go
|
||||
notes: Responses message/reasoning/function-call repeat history decoder
|
||||
- type: code
|
||||
path: apps/edge/internal/openai/normalized_sse.go
|
||||
notes: normalized Chat Completions SSE stream과 terminal event 처리
|
||||
|
|
@ -81,6 +87,8 @@ Edge가 OpenAI-compatible HTTP 요청을 받아 내부 `adapter + target` 실행
|
|||
| metadata/workspace 처리 | `metadata.workspace`는 `RunRequest.workspace`로 분리하고, 일반 metadata는 최대 16개 string key/value만 허용한다. |
|
||||
| Chat Completions | `/v1/chat/completions`는 non-streaming과 streaming SSE를 지원한다. |
|
||||
| bounded ingress와 Stream Evidence Gate | Chat/Responses body를 첫 read 전에 최대 16 MiB로 제한한다. `openai.stream_evidence_gate.enabled=true`인 지원 경로는 response-start staging, filter arbitration, bounded recovery와 단일 terminal을 `runtime/stream-evidence-gate`에 위임한다. |
|
||||
| repeat-resume request shape | A selected continuation uses only request-local assistant content/reasoning plus a fixed English directive. Chat emits assistant provenance followed by the directive; Responses emits assistant output/reasoning items and places the directive in `instructions`. Caller messages, `input`, and original `instructions` are excluded. |
|
||||
| repeat history boundary | Chat and Responses use separate endpoint decoders to create a bounded raw-free role/channel/action snapshot from the current request only. User occurrences exclude assistant anchors; missing reasoning does not infer lineage or TTL state. |
|
||||
| model-driven response path | request `model`이 가리키는 provider capability가 provider raw tunnel 또는 normalized RunEvent path를 결정한다. caller metadata는 route나 response shape를 선택하지 않는다. |
|
||||
| provider raw passthrough | `passthrough`는 provider status/header/body bytes를 기존 Edge-Node tunnel로 relay하고 pure response body에 IOP 확장 envelope를 섞지 않는다. |
|
||||
| provider-native field 보존 | provider raw tunnel route는 `model` served target rewrite와 auth/header 처리 외에 selected provider가 지원하는 OpenAI-compatible 표준 field와 provider extension field를 보존한다. |
|
||||
|
|
@ -134,6 +142,8 @@ sequenceDiagram
|
|||
|
||||
- `configs/edge.yaml`의 `openai` 섹션이 listener, bearer token, legacy adapter/target, model routes, strict output을 제공한다.
|
||||
- `openai.stream_evidence_gate`는 기본 비활성이고, recovery cap 0..3과 16 MiB 이하 ingress snapshot 상한을 설정한다. 변경은 현재 restart-required다.
|
||||
- When `repeat_guard` is configured, Chat accepts plain `content`, `reasoning_content`, `reasoning`, and `reasoning_text` provenance for fingerprinting; Responses accepts its own text/reasoning/function-call item provenance. Signed, encrypted, and unknown values are canonical-only and never sanitation or observation payloads.
|
||||
- Completed action/result fingerprints provide the only request-history progress boundary. An identical consecutive action/result is no-progress; a changed completed result is progress, while a different action alone is insufficient. No caller product, session metadata, inferred TTL, or cross-request cache participates.
|
||||
- top-level `models[]`가 있으면 OpenAI model list와 provider-pool dispatch에서 legacy route보다 우선한다.
|
||||
- provider-pool model group은 capacity + priority + availability 기준으로 provider candidate를 먼저 선택하고, 선택된 provider가 OpenAI-compatible 호출 방식을 지원하면 raw tunnel passthrough로 dispatch한다. Ollama/CLI/native provider가 선택되면 normalized `RunRequest` path로 dispatch한다.
|
||||
- provider capacity와 long-context slot은 model alias별이 아니라 `node_id + provider_id`별로 공유한다. queue pending 상한과 timeout은 Edge root `provider_pool` policy이며, lease 반환·refresh·disconnect/reconnect가 모든 model group waiter를 global enqueue 순서로 재평가한다.
|
||||
|
|
@ -164,6 +174,7 @@ sequenceDiagram
|
|||
|
||||
- normalized(non-provider) `/v1/responses`는 non-streaming string input만 지원한다. provider model group route의 `/v1/responses`는 raw passthrough로 streaming과 Codex/unknown field를 그대로 provider에 전달한다.
|
||||
- Stream Evidence Gate 활성화만으로 반복, missing tool-call, schema 같은 semantic filter가 자동 활성화되지는 않는다. 해당 mechanics와 현재 지원 경로는 `agent-spec/runtime/stream-evidence-gate.md`를 따른다.
|
||||
- A repeat-resume rebuild requires the request-start model catalog context window. Unknown or insufficient context fails before a replacement dispatch, preserving the recovery budget; it does not use a translator, local model, or `RecoveryPlanPreparer`.
|
||||
- `/v1/completions`는 제공하지 않는다.
|
||||
- OpenAI-compatible request에 provider/Ollama 전용 root field를 추가하지 않는다.
|
||||
- workspace는 prompt 본문에 섞지 않고 metadata에서 분리한다.
|
||||
|
|
|
|||
|
|
@ -97,7 +97,7 @@ Edge 설정에서 provider-pool이 어떻게 모델 실행 후보를 고르고,
|
|||
| long-context admission | estimated input token이 threshold 이상이면 `context_class=long`으로 분류하고, provider long slot이 있으면 일반 capacity slot과 함께 점유한다. |
|
||||
| config refresh dry-run/apply | loopback admin HTTP `POST /refresh`가 candidate config를 dry-run 또는 apply한다. |
|
||||
| refresh classification | listener, Edge identity, bootstrap path, adapter structural 변경 등은 restart-required로 분류한다. |
|
||||
| Stream Evidence Gate config | `openai.stream_evidence_gate`는 runtime 활성화, request-total/strategy fault recovery cap과 ingress snapshot 상한을 제공하며 현재 restart-required로 분류된다. |
|
||||
| Stream Evidence Gate config | `openai.stream_evidence_gate` provides runtime activation, request-total/strategy fault recovery caps, ingress snapshot bounds, and per-filter capability/enforcement/Unicode hold policy; it is currently restart-required. |
|
||||
| mutable apply | 적용 가능한 변경은 Edge `Cfg`, `NodeStore`, service/input model catalog, OpenAI long-context threshold를 copy-on-write로 교체한다. |
|
||||
| Node config refresh push | 변경이 있으면 Edge가 dispatch-ready Node에 node-specific `NodeConfigRefreshRequest`를 push한다. accepted지만 pending인 Node는 register response config를 적용한 뒤 ready가 될 때까지 push 대상이 아니다. |
|
||||
| Node registry swap | Node는 refresh payload로 새 adapter registry를 만들고 router registry를 swap한다. old registry stop은 active run이 있으면 drain 이후로 지연한다. |
|
||||
|
|
@ -145,6 +145,8 @@ sequenceDiagram
|
|||
|
||||
- `long_context_threshold_tokens` 기본 예시는 `100000`이고 0 이하 값은 config load에서 거부된다.
|
||||
- `openai.stream_evidence_gate.enabled` 기본값은 `false`다. request fault recovery는 0..3, strategy cap은 request-total 이하, ingress snapshot은 1..16777216 bytes이며 설정 변경은 restart-required다.
|
||||
- `filters[].hold_evidence_runes` is bounded `1..65536` and defaults to 500. For `repeat_guard` it controls the Unicode pending/look-behind evidence window, not a time-based release or a cross-request retention period.
|
||||
- Blocking repeat capability admission is re-resolved for the actual provider/path while the request-start filter policy, history snapshot, recovery ordinals, and temperature candidate order remain generation-stable across provider switches.
|
||||
- `provider_pool.max_queue`는 0/생략 시 기본값 `16`, `queue_timeout_ms`는 생략 시 `30000`이고 명시적 0은 timeout 없음이다. canonical root key가 없을 때만 서로 같은 legacy provider queue pair를 승격하며 값이 다르면 load를 거부한다.
|
||||
- `nodes[].providers[].capacity`와 `long_context_capacity`는 provider resource 속성이고 같은 provider를 공유하는 model alias가 합산 점유한다. `total_context_tokens`는 runtime ledger가 아니라 `context_window_tokens * long_context_capacity` 정적 validation 값이다.
|
||||
- provider `enabled=false`는 dispatch pool에서 제외하지만 adapter process lifecycle 변경을 의미하지 않는다.
|
||||
|
|
|
|||
|
|
@ -27,6 +27,9 @@ source_evidence:
|
|||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_vertical_slice_test.go
|
||||
notes: Edge 실제 admission과 full-cycle 검증
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/stream_gate_pipeline_test.go
|
||||
notes: Chat/Responses tunnel의 exact-wire terminal, split tool identity, non-2xx lifecycle 검증
|
||||
- type: test
|
||||
path: apps/edge/internal/openai/filter_observation_sink_test.go
|
||||
notes: raw-free observation allowlist와 correlation 검증
|
||||
|
|
@ -42,18 +45,20 @@ codec이 정규화한 provider event를 downstream에 쓰기 전에 evidence와
|
|||
|
||||
| 기능 | 설명 |
|
||||
|------|------|
|
||||
| normalized event contract | response-start, text/reasoning, tool-call, terminal, provider-error event와 base disposition을 transport와 분리한다. |
|
||||
| normalized event contract | response-start, text/reasoning, tool-call, terminal, provider-error event를 transport와 분리하고, Chat/Responses tunnel의 caller wire는 semantic evidence와 별도 queue에서 원본 순서로 release한다. |
|
||||
| evidence hold | 기본 500 Unicode rune rolling window와 bounded terminal/fragment gate를 사용하며 안전한 prefix만 release한다. |
|
||||
| staged commit | status/header/opening event를 첫 safe release까지 보류하고 `transport_uncommitted`, `stream_open`, `terminal_committed` 상태에서 terminal을 한 번만 commit한다. |
|
||||
| filter registry와 arbitration | request 시작의 registry/config generation을 고정하고 attempt별 active filter를 다시 해석한 뒤 병렬 결과를 모두 모아 deterministic action 하나를 선택한다. |
|
||||
| bounded recovery | exact replay, continuation repair, schema repair는 request-total·strategy별 fault cap 안에서 실행하며 managed continuation은 별도 trajectory budget을 사용한다. |
|
||||
| request rebuild | 기본값이자 절대 상한 16 MiB의 ingress snapshot에서 OpenAI JSON unknown field를 보존하고 필요한 typed subtree만 bounded patch한다. |
|
||||
| repeat-resume builder | A selected continuation plan can consume one request-local content/reasoning snapshot and build endpoint-native Chat or Responses resume input with the fixed English directive, without caller history or another model call. |
|
||||
| active repeat guard | Request-local Chat/Responses history fingerprints, a Unicode rolling pending window, and committed look-behind produce sanitized pass, continuation, repeated-action safe-stop, or side-effect fatal decisions. |
|
||||
| host re-admission | 현재 provider ownership을 닫은 뒤 optional one-shot prepare, rebuild, budget consume, 단일 dispatch 순서로 새 actual model/provider/path binding을 설치한다. |
|
||||
| raw-free observation | request correlation, attempt/epoch, filter/rule, decision, recovery와 bounded sanitized cause/evidence만 timeline sink로 보낸다. |
|
||||
|
||||
## 범위
|
||||
|
||||
- 포함: transport-agnostic Core, OpenAI-compatible Chat runtime, provider-pool mixed path, Chat/Responses streaming provider tunnel, request-local ingress/rebuild와 Edge observation sink.
|
||||
- 포함: transport-agnostic Core, OpenAI-compatible Chat와 normalized non-stream Responses runtime, provider-pool mixed path, Chat/Responses streaming provider tunnel, request-local ingress/rebuild와 Edge observation sink.
|
||||
- 제외: 반복·missing tool-call·schema 같은 semantic detector 자체, provider/model 선택 알고리즘, raw parser, cross-request 저장과 범용 오류 수정 workflow.
|
||||
|
||||
## 주요 흐름
|
||||
|
|
@ -93,21 +98,27 @@ sequenceDiagram
|
|||
- `max_request_fault_recovery`는 0..3, `max_strategy_fault_recovery`는 0..request-total이고 생략 시 request-total을 상속한다.
|
||||
- `max_ingress_snapshot_bytes`는 1..16777216이며 생략 시 16 MiB다. raw body limit은 첫 read 전에 적용되고 canonical body, typed view와 rebuild peak가 같은 request-local ledger에 포함된다.
|
||||
- Stream Evidence Gate 설정 변경은 현재 restart-required다. request가 시작된 뒤 config/registry snapshot은 바뀌지 않는다.
|
||||
- production Core registry는 공통 Noop filter와 적용 가능한 request-local tool validation을 포함한다. 추가 semantic filter는 별도 소비 Milestone이 등록한다.
|
||||
- The production Core registry includes the common Noop filter, configured active `repeat_guard`, schema/provider-error lifecycle foundations, and applicable request-local tool validation. Repeat detection uses the configured 500-rune default, never time-based release, and returns a continuation only before a tool/side-effect boundary. Provider-error still records unmatched errors as pass until its matcher Task.
|
||||
- Resume recording is bounded by the ingress snapshot limit and is reset for every attempt. The Rebuilder consumes it once after the owning attempt is aborted. It uses the request-start model catalog context window and fails before dispatch when the window is unknown or the rebuilt prompt plus its completion reserve does not fit.
|
||||
- A repeat continuation cursor is a UTF-8 byte boundary for content or reasoning. Already committed look-behind fixes the cursor at the released channel boundary; the pending duplicate is discarded, and a byte-identical replacement prefix is suppressed once. Omitted temperature uses `0.2`, `0.4`, and `0.6` by strategy attempt; explicit temperature is preserved.
|
||||
|
||||
## 검증
|
||||
|
||||
- `make proto` - generated protobuf가 현재 schema와 일치해야 한다.
|
||||
- `go test -race -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./packages/go/config` - Core, Edge vertical cycle, request rebuild, observation과 config 경계가 통과해야 한다.
|
||||
- `go test -race -count=1 ./packages/go/streamgate ./apps/edge/internal/openai ./packages/go/config` - Core, Edge vertical cycle, Chat/Responses tunnel exact-wire terminal·split tool identity·non-2xx lifecycle, request rebuild, observation과 config 경계가 통과해야 한다.
|
||||
- `git diff --check` - 문서와 코드 diff에 공백 오류가 없어야 한다.
|
||||
|
||||
## 한계와 주의사항
|
||||
|
||||
- normalized `/v1/responses`는 현재 streaming을 지원하지 않으며 해당 non-stream path는 Stream Evidence Gate runtime을 사용하지 않는다.
|
||||
- normalized `/v1/responses`는 streaming을 지원하지 않지만 gate가 활성화되면 request-local Stream Evidence Gate runtime을 사용한다. 지원되는 Chat/Responses provider tunnel도 protocol finish와 transport terminal을 분리해 trailing wire를 한 번 release한다.
|
||||
- direct provider tunnel의 non-stream response는 기존 buffered passthrough 경로를 유지한다. ingress 상한은 runtime 활성 여부와 무관하게 적용된다.
|
||||
- Core 활성화만으로 후속 semantic filter가 자동 활성화되지는 않는다.
|
||||
- The repeat detector remains a separately configured filter. The implemented builder is only the request-local continuation seam; it does not translate, summarize, or use a local model or `RecoveryPlanPreparer`.
|
||||
- observation은 저장소가 아니라 event envelope이며 보존·조회 정책은 host observability sink가 소유한다.
|
||||
|
||||
## 변경 기록
|
||||
|
||||
- 2026-07-28: Stream Evidence Gate Core 완료 감사에서 현재 Core, Edge wiring, 계약과 테스트 근거로 targeted spec 생성.
|
||||
- 2026-07-28: Chat/Responses tunnel의 terminal wire queue, split tool identity와 non-2xx provider-error lifecycle 근거로 normalized Responses runtime 범위와 foundation 한계를 현재 구현에 맞췄다.
|
||||
- 2026-07-28: Added the request-local Chat/Responses repeat-resume builder, its bounded recorder lifecycle, fixed directive, caller-history exclusion, and context-window fail-closed boundary.
|
||||
- 2026-07-29: Activated request-local history/current-stream repeat detection, Unicode safe cursors, no-progress action safe-stop, one-shot prefix suppression, and continuation temperature candidates.
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@ import (
|
|||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
func decodeChatCompletionRequest(dec *json.Decoder, req *chatCompletionRequest) error {
|
||||
|
|
@ -112,6 +114,121 @@ func decodeChatCompletionRequestLenient(dec *json.Decoder, req *chatCompletionRe
|
|||
return nil
|
||||
}
|
||||
|
||||
// decodeOpenAIChatRepeatHistory builds the Chat-specific, raw-free repeat
|
||||
// history view. Only plain role text, the documented reasoning aliases, and
|
||||
// completed tool call/result pairs participate. Signed, encrypted, malformed,
|
||||
// and unknown fields remain in the canonical ingress snapshot but are ignored
|
||||
// by this semantic decoder.
|
||||
func decodeOpenAIChatRepeatHistory(rawBody []byte) (openAIRepeatHistorySnapshot, error) {
|
||||
var root struct {
|
||||
Messages []json.RawMessage `json:"messages"`
|
||||
}
|
||||
if err := json.Unmarshal(rawBody, &root); err != nil {
|
||||
return openAIRepeatHistorySnapshot{}, fmt.Errorf("decode Chat repeat history: %w", err)
|
||||
}
|
||||
builder := newOpenAIRepeatHistoryBuilder()
|
||||
pendingActions := make(map[string]streamgate.FixedFingerprint)
|
||||
for _, rawMessage := range root.Messages {
|
||||
var message map[string]json.RawMessage
|
||||
if err := json.Unmarshal(rawMessage, &message); err != nil {
|
||||
continue
|
||||
}
|
||||
var role string
|
||||
if err := json.Unmarshal(message["role"], &role); err != nil {
|
||||
continue
|
||||
}
|
||||
role = strings.ToLower(strings.TrimSpace(role))
|
||||
for _, text := range openAIRepeatPlainTextParts(message["content"]) {
|
||||
builder.recordText(role, openAIRepeatChannelContent, text)
|
||||
}
|
||||
for _, alias := range []string{"reasoning_content", "reasoning", "reasoning_text"} {
|
||||
var reasoning string
|
||||
if err := json.Unmarshal(message[alias], &reasoning); err == nil {
|
||||
builder.recordText(role, openAIRepeatChannelReasoning, reasoning)
|
||||
}
|
||||
}
|
||||
if role == "assistant" {
|
||||
var calls []json.RawMessage
|
||||
if json.Unmarshal(message["tool_calls"], &calls) == nil {
|
||||
for _, rawCall := range calls {
|
||||
var call struct {
|
||||
ID string `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Function struct {
|
||||
Name string `json:"name"`
|
||||
Arguments json.RawMessage `json:"arguments"`
|
||||
} `json:"function"`
|
||||
}
|
||||
if json.Unmarshal(rawCall, &call) != nil {
|
||||
continue
|
||||
}
|
||||
if fingerprint, ok := openAIRepeatActionFingerprint(
|
||||
call.Function.Name,
|
||||
openAIRepeatArgumentsValue(call.Function.Arguments),
|
||||
); ok && strings.TrimSpace(call.ID) != "" {
|
||||
pendingActions[call.ID] = fingerprint
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if role == "tool" {
|
||||
var callID string
|
||||
if json.Unmarshal(message["tool_call_id"], &callID) != nil {
|
||||
continue
|
||||
}
|
||||
action, ok := pendingActions[callID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
result, ok := openAIRepeatTextFingerprint(strings.Join(openAIRepeatPlainTextParts(message["content"]), "\n"))
|
||||
if ok {
|
||||
builder.recordCompletedAction(action, result)
|
||||
}
|
||||
}
|
||||
}
|
||||
return builder.snapshot(), nil
|
||||
}
|
||||
|
||||
func openAIRepeatArgumentsValue(raw json.RawMessage) json.RawMessage {
|
||||
var value string
|
||||
if json.Unmarshal(raw, &value) == nil {
|
||||
return json.RawMessage(value)
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
func openAIRepeatPlainTextParts(raw json.RawMessage) []string {
|
||||
var plain string
|
||||
if json.Unmarshal(raw, &plain) == nil {
|
||||
return []string{plain}
|
||||
}
|
||||
var parts []json.RawMessage
|
||||
if json.Unmarshal(raw, &parts) != nil {
|
||||
return nil
|
||||
}
|
||||
values := make([]string, 0, len(parts))
|
||||
for _, rawPart := range parts {
|
||||
var part map[string]json.RawMessage
|
||||
if json.Unmarshal(rawPart, &part) != nil {
|
||||
continue
|
||||
}
|
||||
var partType string
|
||||
if json.Unmarshal(part["type"], &partType) != nil {
|
||||
continue
|
||||
}
|
||||
switch partType {
|
||||
case "text", "input_text", "output_text":
|
||||
default:
|
||||
continue
|
||||
}
|
||||
var text string
|
||||
if json.Unmarshal(part["text"], &text) == nil {
|
||||
values = append(values, text)
|
||||
}
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
func promptFromMessages(messages []chatMessage) string {
|
||||
var b strings.Builder
|
||||
for _, msg := range messages {
|
||||
|
|
|
|||
|
|
@ -249,6 +249,20 @@ func (s *Server) handleChatCompletionsProviderPool(w http.ResponseWriter, dc *ch
|
|||
},
|
||||
}
|
||||
|
||||
if s.streamGateEnabled() {
|
||||
fctx, err := s.openAIChatOutputFilterContext(dc)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
predicate, err := openAIStreamGateCandidatePredicate(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
poolReq.AcceptCandidate = predicate
|
||||
}
|
||||
|
||||
// Pre-dispatch provider auth header injection. Runs inside SubmitProviderPool
|
||||
// BEFORE buildProviderTunnelRequest and the Node Send step, so the auth
|
||||
// header lands on the actual wire request. On failure the slot is released
|
||||
|
|
@ -289,6 +303,10 @@ func (s *Server) handleChatCompletionsProviderPool(w http.ResponseWriter, dc *ch
|
|||
writeError(w, http.StatusBadRequest, "invalid_request_error", "provider auth token is required")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, edgeservice.ErrProviderPoolCandidateRejected) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", openAIStreamGateCandidateRejectedMessage)
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadGateway, "node_dispatch_error", err.Error())
|
||||
return
|
||||
}
|
||||
|
|
@ -328,10 +346,11 @@ func (s *Server) handleChatCompletionsProviderPool(w http.ResponseWriter, dc *ch
|
|||
// metadata, and input of the original dispatch.
|
||||
poolDC := dc.withRetrySubmit(func(ctx context.Context, retryReq edgeservice.SubmitRunRequest) (any, error) {
|
||||
return s.service.SubmitProviderPool(ctx, edgeservice.ProviderPoolDispatchRequest{
|
||||
Run: retryReq,
|
||||
Tunnel: poolReq.Tunnel,
|
||||
PrepareTunnel: poolReq.PrepareTunnel,
|
||||
PrepareRun: poolReq.PrepareRun,
|
||||
Run: retryReq,
|
||||
Tunnel: poolReq.Tunnel,
|
||||
PrepareTunnel: poolReq.PrepareTunnel,
|
||||
PrepareRun: poolReq.PrepareRun,
|
||||
AcceptCandidate: poolReq.AcceptCandidate,
|
||||
})
|
||||
})
|
||||
if req.Stream {
|
||||
|
|
|
|||
|
|
@ -128,12 +128,19 @@ type responsesDispatchContext struct {
|
|||
outputPolicy strictOutputPolicy
|
||||
runMetadata map[string]string
|
||||
submitReq edgeservice.SubmitRunRequest
|
||||
poolDispatch *edgeservice.ProviderPoolDispatchRequest
|
||||
}
|
||||
|
||||
func (dc *responsesDispatchContext) usageLabels(s *Server, responseMode string) usageLabels {
|
||||
return s.usageLabelsFor(dc.r.Context(), strings.TrimSpace(dc.req.Model), dc.endpoint, responseMode)
|
||||
}
|
||||
|
||||
func (dc *responsesDispatchContext) withPoolDispatch(pool edgeservice.ProviderPoolDispatchRequest) *responsesDispatchContext {
|
||||
derived := *dc
|
||||
derived.poolDispatch = &pool
|
||||
return &derived
|
||||
}
|
||||
|
||||
// withRetrySubmit derives a context bound to a retry function. The
|
||||
// provider-pool retry closure can only be built once the pool request exists,
|
||||
// so the pool path derives a new context rather than mutating this one.
|
||||
|
|
|
|||
|
|
@ -17,9 +17,254 @@ const (
|
|||
openAIRebuildFamily = "openai.json"
|
||||
)
|
||||
|
||||
// openAIRepeatResumeDirective is the fixed English continuation instruction the
|
||||
// resume-notice builder writes when a repeat-loop continuation plan is selected.
|
||||
// It is byte-stable: recovery never rewrites, summarizes, or translates it, and
|
||||
// it is the only non-model text added to a resume body.
|
||||
const openAIRepeatResumeDirective = "The previous model output was stopped after a repetition loop was detected. Continue from the provided content and reasoning output without repeating already generated text."
|
||||
|
||||
// openAIRebuildContextReserveTokens is the completion-token headroom reserved on
|
||||
// top of the measured resume prompt when comparing against the target model's
|
||||
// context window. A resume body whose prompt plus this reserve exceeds the
|
||||
// window (or whose window is unknown) is refused before any dispatch.
|
||||
const openAIRebuildContextReserveTokens = 256
|
||||
|
||||
var openAIRebuiltSequence atomic.Uint64
|
||||
|
||||
var errOpenAIRecoveryPatchDuplicate = errors.New("duplicate OpenAI recovery patch")
|
||||
var (
|
||||
errOpenAIRecoveryPatchDuplicate = errors.New("duplicate OpenAI recovery patch")
|
||||
errOpenAIRebuildContextOverflow = errors.New("OpenAI resume request exceeds the target model context window")
|
||||
errOpenAIRecoverySourceUnavailable = errors.New("OpenAI recovery source snapshot is unavailable")
|
||||
errOpenAIRecoverySourceCursorRange = errors.New("OpenAI recovery source cursor is out of range")
|
||||
)
|
||||
|
||||
// openAIRecoverySourceStore is the request-local recorder of the current
|
||||
// attempt's model output. It captures text and reasoning deltas in event order
|
||||
// per channel behind a bounded byte cap and exposes only a stable snapshot ref
|
||||
// and cursor to filters, never the raw output. The Rebuilder consumes a snapshot
|
||||
// once, after the attempt that produced it is aborted, to assemble an
|
||||
// endpoint-specific continuation body. Its ref is the ingress recovery ref so a
|
||||
// continuation directive can address it without leaking model output.
|
||||
type openAIRecoverySourceStore struct {
|
||||
mu sync.Mutex
|
||||
ref string
|
||||
maxBytes int
|
||||
content []byte
|
||||
reasoning []byte
|
||||
// suppressContent/suppressReasoning are one-attempt known-prefix guards.
|
||||
// consume installs the safe continuation prefix and the replacement event
|
||||
// source removes it only when the provider echoes it byte-for-byte.
|
||||
suppressContent []byte
|
||||
suppressReasoning []byte
|
||||
overflow bool
|
||||
consumed bool
|
||||
closed bool
|
||||
}
|
||||
|
||||
func newOpenAIRecoverySourceStore(ingress *openAIIngressSnapshot) *openAIRecoverySourceStore {
|
||||
store := &openAIRecoverySourceStore{}
|
||||
if ref, err := ingress.recoveryRef(); err == nil {
|
||||
store.ref = ref.SnapshotRef()
|
||||
store.maxBytes = int(ref.MaxBytes())
|
||||
}
|
||||
return store
|
||||
}
|
||||
|
||||
func (s *openAIRecoverySourceStore) snapshotRef() string {
|
||||
if s == nil {
|
||||
return ""
|
||||
}
|
||||
return s.ref
|
||||
}
|
||||
|
||||
func (s *openAIRecoverySourceStore) recordText(text string) { s.record(true, text) }
|
||||
func (s *openAIRecoverySourceStore) recordReasoning(text string) { s.record(false, text) }
|
||||
|
||||
func (s *openAIRecoverySourceStore) record(isContent bool, text string) {
|
||||
if s == nil || text == "" {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.closed {
|
||||
return
|
||||
}
|
||||
if s.maxBytes > 0 && len(s.content)+len(s.reasoning)+len(text) > s.maxBytes {
|
||||
s.overflow = true
|
||||
return
|
||||
}
|
||||
if isContent {
|
||||
s.content = append(s.content, text...)
|
||||
} else {
|
||||
s.reasoning = append(s.reasoning, text...)
|
||||
}
|
||||
}
|
||||
|
||||
// resetAttempt clears the recorded output at the start of a new attempt so a
|
||||
// snapshot only ever contains the single attempt that produced it.
|
||||
func (s *openAIRecoverySourceStore) resetAttempt() {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.closed {
|
||||
return
|
||||
}
|
||||
s.content = s.content[:0]
|
||||
s.reasoning = s.reasoning[:0]
|
||||
s.overflow = false
|
||||
s.consumed = false
|
||||
}
|
||||
|
||||
// consume returns the recorded channels with the repeated tail removed. A
|
||||
// cursor at or below content length addresses the content channel. A cursor
|
||||
// above content length uses content-length+1 as a reasoning-channel sentinel
|
||||
// and addresses the reasoning prefix after it. This keeps the public directive
|
||||
// raw-free and single-field while preserving a rune-boundary byte cursor for
|
||||
// either model-output channel.
|
||||
func (s *openAIRecoverySourceStore) consume(cursor int) (string, string, error) {
|
||||
if s == nil {
|
||||
return "", "", streamgate.ErrIngressSnapshotClosed
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.closed {
|
||||
return "", "", streamgate.ErrIngressSnapshotClosed
|
||||
}
|
||||
if s.consumed {
|
||||
return "", "", errOpenAIRecoverySourceUnavailable
|
||||
}
|
||||
if s.overflow {
|
||||
return "", "", errOpenAIRebuildContextOverflow
|
||||
}
|
||||
if len(s.content)+len(s.reasoning) == 0 {
|
||||
return "", "", errOpenAIRecoverySourceUnavailable
|
||||
}
|
||||
if cursor < 0 {
|
||||
return "", "", errOpenAIRecoverySourceCursorRange
|
||||
}
|
||||
contentEnd, reasoningEnd := len(s.content), len(s.reasoning)
|
||||
if cursor <= len(s.content) {
|
||||
contentEnd = cursor
|
||||
} else {
|
||||
reasoningEnd = cursor - len(s.content) - 1
|
||||
if reasoningEnd < 0 || reasoningEnd > len(s.reasoning) {
|
||||
return "", "", errOpenAIRecoverySourceCursorRange
|
||||
}
|
||||
}
|
||||
s.consumed = true
|
||||
content := string(s.content[:contentEnd])
|
||||
reasoning := string(s.reasoning[:reasoningEnd])
|
||||
s.suppressContent = append(s.suppressContent[:0], content...)
|
||||
s.suppressReasoning = append(s.suppressReasoning[:0], reasoning...)
|
||||
return content, reasoning, nil
|
||||
}
|
||||
|
||||
func (s *openAIRecoverySourceStore) suppressKnownPrefix(isContent bool, value string) string {
|
||||
if s == nil || value == "" {
|
||||
return value
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
var expected *[]byte
|
||||
if isContent {
|
||||
expected = &s.suppressContent
|
||||
} else {
|
||||
expected = &s.suppressReasoning
|
||||
}
|
||||
if len(*expected) == 0 {
|
||||
return value
|
||||
}
|
||||
incoming := []byte(value)
|
||||
switch {
|
||||
case len(incoming) <= len(*expected) && string((*expected)[:len(incoming)]) == value:
|
||||
*expected = (*expected)[len(incoming):]
|
||||
return ""
|
||||
case len(incoming) > len(*expected) && string(incoming[:len(*expected)]) == string(*expected):
|
||||
value = string(incoming[len(*expected):])
|
||||
*expected = nil
|
||||
return value
|
||||
default:
|
||||
// The replacement started with novel output. Disable suppression
|
||||
// immediately so a coincidental later substring is never removed.
|
||||
*expected = nil
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
func (s *openAIRecoverySourceStore) close() {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.closed {
|
||||
return
|
||||
}
|
||||
s.closed = true
|
||||
s.content = nil
|
||||
s.reasoning = nil
|
||||
s.suppressContent = nil
|
||||
s.suppressReasoning = nil
|
||||
}
|
||||
|
||||
// openAIRecoverySourceEventSource wraps a NormalizedEventSource so every text and
|
||||
// reasoning delta the Core reads is recorded into the request-local source store
|
||||
// before it reaches any filter. It resets the store on each response start so
|
||||
// each attempt records only its own output. It is otherwise a transparent
|
||||
// passthrough and never mutates events.
|
||||
type openAIRecoverySourceEventSource struct {
|
||||
inner streamgate.NormalizedEventSource
|
||||
store *openAIRecoverySourceStore
|
||||
}
|
||||
|
||||
func newOpenAIRecoverySourceEventSource(inner streamgate.NormalizedEventSource, store *openAIRecoverySourceStore) streamgate.NormalizedEventSource {
|
||||
if store == nil {
|
||||
return inner
|
||||
}
|
||||
return &openAIRecoverySourceEventSource{inner: inner, store: store}
|
||||
}
|
||||
|
||||
func (s *openAIRecoverySourceEventSource) NextEvent(ctx context.Context) (streamgate.NormalizedEvent, error) {
|
||||
for {
|
||||
ev, err := s.inner.NextEvent(ctx)
|
||||
if err != nil {
|
||||
return ev, err
|
||||
}
|
||||
switch ev.Kind() {
|
||||
case streamgate.EventKindResponseStart:
|
||||
s.store.resetAttempt()
|
||||
case streamgate.EventKindTextDelta:
|
||||
if value, valueErr := ev.AsTextDelta(); valueErr == nil {
|
||||
value = s.store.suppressKnownPrefix(true, value)
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
s.store.recordText(value)
|
||||
ev, err = streamgate.NewTextDeltaEvent(ev.Channel(), value, ev.Timestamp())
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
}
|
||||
case streamgate.EventKindReasoningDelta:
|
||||
if value, valueErr := ev.AsReasoningDelta(); valueErr == nil {
|
||||
value = s.store.suppressKnownPrefix(false, value)
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
s.store.recordReasoning(value)
|
||||
ev, err = streamgate.NewReasoningDeltaEvent(ev.Channel(), value, ev.Timestamp())
|
||||
if err != nil {
|
||||
return streamgate.NormalizedEvent{}, err
|
||||
}
|
||||
}
|
||||
}
|
||||
return ev, err
|
||||
}
|
||||
}
|
||||
|
||||
var _ streamgate.NormalizedEventSource = (*openAIRecoverySourceEventSource)(nil)
|
||||
|
||||
type openAIRecoveryPatchEntry struct {
|
||||
cursor int
|
||||
|
|
@ -331,19 +576,25 @@ func (s *openAIRebuiltRequestStore) close() {
|
|||
// reservation; nil receivers return ErrIngressSnapshotClosed instead of
|
||||
// panicking.
|
||||
type openAIRequestRebuilder struct {
|
||||
mu sync.Mutex
|
||||
cond *sync.Cond
|
||||
inFlight int
|
||||
closeWaiters int
|
||||
closed bool
|
||||
closeDone bool
|
||||
ingress *openAIIngressSnapshot
|
||||
endpoint string
|
||||
patches *openAIRecoveryPatchStore
|
||||
rebuilt *openAIRebuiltRequestStore
|
||||
mu sync.Mutex
|
||||
cond *sync.Cond
|
||||
inFlight int
|
||||
closeWaiters int
|
||||
closed bool
|
||||
closeDone bool
|
||||
ingress *openAIIngressSnapshot
|
||||
endpoint string
|
||||
recoverySource *openAIRecoverySourceStore
|
||||
contextWindowTokens int
|
||||
patches *openAIRecoveryPatchStore
|
||||
rebuilt *openAIRebuiltRequestStore
|
||||
}
|
||||
|
||||
func newOpenAIRequestRebuilder(ingress *openAIIngressSnapshot, endpoint string) (*openAIRequestRebuilder, error) {
|
||||
// newOpenAIRequestRebuilder binds the request-local ingress, the optional model
|
||||
// output recovery source, and the target model's context window (0 when
|
||||
// unknown) for one HTTP request. The recovery source powers the S20 resume
|
||||
// builder; a nil source keeps only the exact/schema/patch continuation paths.
|
||||
func newOpenAIRequestRebuilder(ingress *openAIIngressSnapshot, endpoint string, recoverySource *openAIRecoverySourceStore, contextWindowTokens int) (*openAIRequestRebuilder, error) {
|
||||
switch endpoint {
|
||||
case openAIRebuildEndpointChat, openAIRebuildEndpointResponses:
|
||||
default:
|
||||
|
|
@ -353,10 +604,12 @@ func newOpenAIRequestRebuilder(ingress *openAIIngressSnapshot, endpoint string)
|
|||
return nil, streamgate.ErrIngressSnapshotClosed
|
||||
}
|
||||
r := &openAIRequestRebuilder{
|
||||
ingress: ingress,
|
||||
endpoint: endpoint,
|
||||
patches: newOpenAIRecoveryPatchStore(ingress),
|
||||
rebuilt: newOpenAIRebuiltRequestStore(),
|
||||
ingress: ingress,
|
||||
endpoint: endpoint,
|
||||
recoverySource: recoverySource,
|
||||
contextWindowTokens: contextWindowTokens,
|
||||
patches: newOpenAIRecoveryPatchStore(ingress),
|
||||
rebuilt: newOpenAIRebuiltRequestStore(),
|
||||
}
|
||||
r.cond = sync.NewCond(&r.mu)
|
||||
return r, nil
|
||||
|
|
@ -387,6 +640,7 @@ func (r *openAIRequestRebuilder) Close() {
|
|||
}
|
||||
r.rebuilt.close()
|
||||
r.patches.close()
|
||||
r.recoverySource.close()
|
||||
r.closeDone = true
|
||||
r.cond.Broadcast()
|
||||
r.mu.Unlock()
|
||||
|
|
@ -437,6 +691,15 @@ func (r *openAIRequestRebuilder) RebuildRequest(ctx context.Context, snapshotRef
|
|||
}
|
||||
|
||||
directive := plan.Directive()
|
||||
// S20 resume builder: a continuation directive that addresses the
|
||||
// request-local recovery source is assembled from recorded model output and
|
||||
// the fixed English directive, without any caller history or extra model
|
||||
// call. Continuation directives that address a pre-computed patch keep the
|
||||
// existing patch-store path below.
|
||||
if directive.Kind() == streamgate.RecoveryDirectiveKindContinuation &&
|
||||
r.recoverySource != nil && directive.SnapshotRef() == r.recoverySource.snapshotRef() {
|
||||
return r.rebuildResumeContinuation(ctx, plan, directive)
|
||||
}
|
||||
var patchEntry *openAIRecoveryPatchEntry
|
||||
switch directive.Kind() {
|
||||
case streamgate.RecoveryDirectiveKindExact:
|
||||
|
|
@ -523,6 +786,164 @@ func (r *openAIRequestRebuilder) RebuildRequest(ctx context.Context, snapshotRef
|
|||
return draft, nil
|
||||
}
|
||||
|
||||
// rebuildResumeContinuation assembles an endpoint-specific continuation body
|
||||
// from the request-local recovery source snapshot and the fixed English resume
|
||||
// directive. It consumes the snapshot once (so it only runs after the aborted
|
||||
// attempt recorded output), refuses to build when the target context window is
|
||||
// unknown or the resume prompt plus reserve would exceed it, and only then
|
||||
// reserves, commits, and leases the rebuilt body. A context-window refusal
|
||||
// returns before any lease is created so no dispatch or recovery budget is
|
||||
// consumed.
|
||||
func (r *openAIRequestRebuilder) rebuildResumeContinuation(ctx context.Context, plan streamgate.RecoveryPlan, directive streamgate.RecoveryDirective) (streamgate.RebuiltRequestDraft, error) {
|
||||
content, reasoning, err := r.recoverySource.consume(directive.Cursor())
|
||||
if err != nil {
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
output, err := r.buildResumeBody(content, reasoning, plan)
|
||||
if err != nil {
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
rebuiltPromptTokens := estimateInputTokensBytes(output, nil, nil, nil)
|
||||
if r.contextWindowTokens <= 0 || rebuiltPromptTokens+openAIRebuildContextReserveTokens > r.contextWindowTokens {
|
||||
return streamgate.RebuiltRequestDraft{}, errOpenAIRebuildContextOverflow
|
||||
}
|
||||
|
||||
requestRef := fmt.Sprintf("openai.rebuilt.%d", openAIRebuiltSequence.Add(1))
|
||||
lease := &openAIRebuiltLease{ingress: r.ingress}
|
||||
guard, err := r.ingress.reserveRebuild(int64(len(output)))
|
||||
if err != nil {
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
rebuilt, err := guard.CommitOwnedTyped(openAIRebuiltBodyViewName, output)
|
||||
if err != nil {
|
||||
guard.Close()
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
lease.guard = guard
|
||||
lease.rebuilt = rebuilt
|
||||
lease.bodyAlias = output
|
||||
accessor := rebuilt.Accessor()
|
||||
retained := uint64(accessor.RetainedBytes())
|
||||
maxBytes := uint64(accessor.MaxBytes())
|
||||
peak := retained
|
||||
if current, refErr := r.ingress.recoveryRef(); refErr == nil && current.PeakBytes() > peak {
|
||||
peak = current.PeakBytes()
|
||||
}
|
||||
|
||||
draft, err := streamgate.NewRebuiltRequestDraftWithIdempotency(
|
||||
plan.PlanID(), plan.IdempotencyKey(), requestRef, r.endpoint, openAIRebuildFamily,
|
||||
retained, peak, maxBytes, rebuiltPromptTokens, plan.RequiredCapabilities(),
|
||||
)
|
||||
if err != nil {
|
||||
lease.release()
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
if err := r.rebuilt.put(requestRef, lease); err != nil {
|
||||
lease.release()
|
||||
return streamgate.RebuiltRequestDraft{}, err
|
||||
}
|
||||
return draft, nil
|
||||
}
|
||||
|
||||
// buildResumeBody serializes a fresh endpoint-native continuation request that
|
||||
// carries only the recorded assistant content/reasoning provenance and the
|
||||
// fixed resume directive. It drops the caller's original messages/input/
|
||||
// instructions and never summarizes, truncates (beyond the content cursor
|
||||
// already applied), or rewrites the recorded output.
|
||||
func (r *openAIRequestRebuilder) buildResumeBody(content, reasoning string, plan streamgate.RecoveryPlan) ([]byte, error) {
|
||||
model, stream, ingressTemperature, err := r.ingressResumeHead()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
temperature := continuationTemperature(plan.StrategyAttempt(), ingressTemperature)
|
||||
if r.endpoint == openAIRebuildEndpointResponses {
|
||||
return buildOpenAIResponsesResumeBody(model, content, reasoning, temperature)
|
||||
}
|
||||
return buildOpenAIChatResumeBody(model, stream, content, reasoning, temperature)
|
||||
}
|
||||
|
||||
// ingressResumeHead reads only the target model, stream flag, and caller
|
||||
// temperature from the canonical body. No caller message or input is copied.
|
||||
func (r *openAIRequestRebuilder) ingressResumeHead() (string, bool, *float64, error) {
|
||||
body, err := r.ingress.canonicalBody()
|
||||
if err != nil {
|
||||
return "", false, nil, err
|
||||
}
|
||||
var head struct {
|
||||
Model string `json:"model"`
|
||||
Stream bool `json:"stream"`
|
||||
Temperature *float64 `json:"temperature"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &head); err != nil {
|
||||
return "", false, nil, fmt.Errorf("decode OpenAI resume ingress head: %w", err)
|
||||
}
|
||||
return head.Model, head.Stream, head.Temperature, nil
|
||||
}
|
||||
|
||||
func continuationTemperature(strategyAttempt int, ingress *float64) *float64 {
|
||||
if ingress != nil {
|
||||
value := *ingress
|
||||
return &value
|
||||
}
|
||||
candidates := [...]float64{0.2, 0.4, 0.6}
|
||||
index := strategyAttempt - 1
|
||||
if index < 0 {
|
||||
index = 0
|
||||
}
|
||||
if index >= len(candidates) {
|
||||
index = len(candidates) - 1
|
||||
}
|
||||
value := candidates[index]
|
||||
return &value
|
||||
}
|
||||
|
||||
func buildOpenAIChatResumeBody(model string, stream bool, content, reasoning string, temperature *float64) ([]byte, error) {
|
||||
assistant := map[string]any{"role": "assistant", "content": content}
|
||||
if reasoning != "" {
|
||||
assistant["reasoning_content"] = reasoning
|
||||
}
|
||||
body := map[string]any{
|
||||
"model": model,
|
||||
"stream": stream,
|
||||
"messages": []any{
|
||||
assistant,
|
||||
map[string]any{"role": "user", "content": openAIRepeatResumeDirective},
|
||||
},
|
||||
}
|
||||
if temperature != nil {
|
||||
body["temperature"] = *temperature
|
||||
}
|
||||
return json.Marshal(body)
|
||||
}
|
||||
|
||||
func buildOpenAIResponsesResumeBody(model, content, reasoning string, temperature *float64) ([]byte, error) {
|
||||
input := make([]any, 0, 2)
|
||||
if reasoning != "" {
|
||||
input = append(input, map[string]any{
|
||||
"type": "reasoning",
|
||||
"content": []any{map[string]any{"type": "reasoning_text", "text": reasoning}},
|
||||
})
|
||||
}
|
||||
input = append(input, map[string]any{
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"content": []any{map[string]any{"type": "output_text", "text": content}},
|
||||
})
|
||||
body := map[string]any{
|
||||
"model": model,
|
||||
"stream": false,
|
||||
"instructions": openAIRepeatResumeDirective,
|
||||
"input": input,
|
||||
}
|
||||
if temperature != nil {
|
||||
body["temperature"] = *temperature
|
||||
}
|
||||
return json.Marshal(body)
|
||||
}
|
||||
|
||||
func (r *openAIRequestRebuilder) leaveCloseWaiter() {
|
||||
r.mu.Lock()
|
||||
r.closeWaiters--
|
||||
|
|
|
|||
|
|
@ -6,13 +6,78 @@ import (
|
|||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
type openAIResumeTestEventSource struct{}
|
||||
|
||||
func (openAIResumeTestEventSource) NextEvent(context.Context) (streamgate.NormalizedEvent, error) {
|
||||
return streamgate.NormalizedEvent{}, io.EOF
|
||||
}
|
||||
|
||||
type openAIResumeTrace struct {
|
||||
steps []string
|
||||
}
|
||||
|
||||
func (t *openAIResumeTrace) record(step string) { t.steps = append(t.steps, step) }
|
||||
func (t *openAIResumeTrace) snapshot() []string { return append([]string(nil), t.steps...) }
|
||||
|
||||
type openAIResumeTestController struct {
|
||||
calls int
|
||||
trace *openAIResumeTrace
|
||||
}
|
||||
|
||||
func (c *openAIResumeTestController) AbortAttempt(context.Context) error {
|
||||
c.calls++
|
||||
if c.trace != nil {
|
||||
c.trace.record("abort")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type openAIResumeTracingRebuilder struct {
|
||||
inner streamgate.RequestRebuilder
|
||||
trace *openAIResumeTrace
|
||||
}
|
||||
|
||||
func (r openAIResumeTracingRebuilder) RebuildRequest(ctx context.Context, snapshot streamgate.RecoveryRequestSnapshotRef, plan streamgate.RecoveryPlan) (streamgate.RebuiltRequestDraft, error) {
|
||||
r.trace.record("rebuild")
|
||||
return r.inner.RebuildRequest(ctx, snapshot, plan)
|
||||
}
|
||||
|
||||
type openAIResumeTracingDispatcher struct {
|
||||
inner streamgate.AttemptDispatcher
|
||||
trace *openAIResumeTrace
|
||||
}
|
||||
|
||||
func (d openAIResumeTracingDispatcher) DispatchAttempt(ctx context.Context, request streamgate.RebuiltRequest) (streamgate.AttemptBinding, error) {
|
||||
d.trace.record("dispatch")
|
||||
return d.inner.DispatchAttempt(ctx, request)
|
||||
}
|
||||
|
||||
type openAIResumeNoDispatch struct{ calls int }
|
||||
|
||||
func (d *openAIResumeNoDispatch) DispatchAttempt(context.Context, streamgate.RebuiltRequest) (streamgate.AttemptBinding, error) {
|
||||
d.calls++
|
||||
return streamgate.AttemptBinding{}, errors.New("context-overflow recovery must not dispatch")
|
||||
}
|
||||
|
||||
type openAIResumeRecordingPreparer struct{ calls int }
|
||||
|
||||
func (p *openAIResumeRecordingPreparer) PrepareRecoveryPlan(context.Context, streamgate.RecoveryPlan, streamgate.RecoveryPreparationSnapshot) (streamgate.RecoveryDirective, error) {
|
||||
p.calls++
|
||||
return streamgate.RecoveryDirective{}, errors.New("resume continuation must not invoke a preparer")
|
||||
}
|
||||
|
||||
func mustOpenAIRecoveryPlan(t *testing.T, id string, strategy streamgate.RecoveryStrategy, directive streamgate.RecoveryDirective) streamgate.RecoveryPlan {
|
||||
t.Helper()
|
||||
intent, err := streamgate.NewRecoveryIntent(strategy, directive, "openai.recovery", 10)
|
||||
|
|
@ -57,7 +122,7 @@ func newOpenAIRebuilderFixture(t *testing.T, endpoint string, body []byte, maxBy
|
|||
t.Fatalf("buildOpenAIIngressSnapshot: %v", err)
|
||||
}
|
||||
t.Cleanup(ingress.Close)
|
||||
rebuilder, err := newOpenAIRequestRebuilder(ingress, endpoint)
|
||||
rebuilder, err := newOpenAIRequestRebuilder(ingress, endpoint, nil, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("newOpenAIRequestRebuilder: %v", err)
|
||||
}
|
||||
|
|
@ -69,6 +134,461 @@ func newOpenAIRebuilderFixture(t *testing.T, endpoint string, body []byte, maxBy
|
|||
return ingress, rebuilder, ref
|
||||
}
|
||||
|
||||
func newOpenAIResumeRebuilderFixture(t *testing.T, endpoint string, body []byte, contextWindowTokens int) (*openAIIngressSnapshot, *openAIRequestRebuilder, *openAIRecoverySourceStore, streamgate.RecoveryRequestSnapshotRef) {
|
||||
t.Helper()
|
||||
ingress, err := buildOpenAIIngressSnapshot(8192, body, json.RawMessage(body))
|
||||
if err != nil {
|
||||
t.Fatalf("buildOpenAIIngressSnapshot: %v", err)
|
||||
}
|
||||
t.Cleanup(ingress.Close)
|
||||
source := newOpenAIRecoverySourceStore(ingress)
|
||||
rebuilder, err := newOpenAIRequestRebuilder(ingress, endpoint, source, contextWindowTokens)
|
||||
if err != nil {
|
||||
t.Fatalf("newOpenAIRequestRebuilder: %v", err)
|
||||
}
|
||||
t.Cleanup(rebuilder.Close)
|
||||
ref, err := ingress.recoveryRef()
|
||||
if err != nil {
|
||||
t.Fatalf("recoveryRef: %v", err)
|
||||
}
|
||||
return ingress, rebuilder, source, ref
|
||||
}
|
||||
|
||||
func TestOpenAIRequestRebuilderBuildsChatRepeatResume(t *testing.T) {
|
||||
const callerSentinel = "CALLER_CHAT_HISTORY_MUST_NOT_BE_COPIED"
|
||||
const content = "MODEL_CONTENT_PREFIX__REPEATED_TAIL"
|
||||
const reasoning = "MODEL_REASONING_SENTINEL"
|
||||
body := []byte(`{"model":"served-chat","stream":true,"messages":[{"role":"user","content":"` + callerSentinel + `"}],"tools":[{"type":"function"}]}`)
|
||||
_, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointChat, body, 4096)
|
||||
source.recordText(content)
|
||||
source.recordReasoning(reasoning)
|
||||
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len("MODEL_CONTENT_PREFIX__"), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
draft, err := rebuilder.RebuildRequest(context.Background(), ref, mustOpenAIRecoveryPlan(t, "plan.chat.resume", streamgate.RecoveryStrategyContinuationRepair, directive))
|
||||
if err != nil {
|
||||
t.Fatalf("RebuildRequest: %v", err)
|
||||
}
|
||||
lease, err := rebuilder.RebuiltStore().take(draft.RequestRef())
|
||||
if err != nil {
|
||||
t.Fatalf("take: %v", err)
|
||||
}
|
||||
defer lease.release()
|
||||
got, err := lease.body()
|
||||
if err != nil {
|
||||
t.Fatalf("body: %v", err)
|
||||
}
|
||||
if bytes.Contains(got, []byte(callerSentinel)) || bytes.Contains(got, []byte(`"tools"`)) {
|
||||
t.Fatalf("chat resume copied caller history: %s", got)
|
||||
}
|
||||
var rebuilt struct {
|
||||
Model string `json:"model"`
|
||||
Stream bool `json:"stream"`
|
||||
Messages []struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
ReasoningContent string `json:"reasoning_content"`
|
||||
} `json:"messages"`
|
||||
}
|
||||
if err := json.Unmarshal(got, &rebuilt); err != nil {
|
||||
t.Fatalf("decode resume body: %v", err)
|
||||
}
|
||||
if rebuilt.Model != "served-chat" || !rebuilt.Stream || len(rebuilt.Messages) != 2 {
|
||||
t.Fatalf("chat resume envelope = %#v", rebuilt)
|
||||
}
|
||||
if got := rebuilt.Messages[0]; got.Role != "assistant" || got.Content != "MODEL_CONTENT_PREFIX__" || got.ReasoningContent != reasoning {
|
||||
t.Fatalf("chat assistant provenance = %#v", got)
|
||||
}
|
||||
if got := rebuilt.Messages[1]; got.Role != "user" || got.Content != openAIRepeatResumeDirective {
|
||||
t.Fatalf("chat resume directive = %#v", got)
|
||||
}
|
||||
if _, err := rebuilder.RebuildRequest(context.Background(), ref, mustOpenAIRecoveryPlan(t, "plan.chat.resume.second", streamgate.RecoveryStrategyContinuationRepair, directive)); !errors.Is(err, errOpenAIRecoverySourceUnavailable) {
|
||||
t.Fatalf("second resume error = %v, want one-shot source error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRequestRebuilderBuildsResponsesRepeatResume(t *testing.T) {
|
||||
const callerInput = "CALLER_RESPONSES_INPUT_MUST_NOT_BE_COPIED"
|
||||
const callerInstructions = "CALLER_INSTRUCTIONS_MUST_NOT_BE_COPIED"
|
||||
const content = "MODEL_OUTPUT_SENTINEL"
|
||||
const reasoning = "MODEL_THINK_SENTINEL"
|
||||
body := []byte(`{"model":"served-responses","stream":false,"instructions":"` + callerInstructions + `","input":"` + callerInput + `"}`)
|
||||
_, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointResponses, body, 4096)
|
||||
source.recordText(content)
|
||||
source.recordReasoning(reasoning)
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len(content), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
draft, err := rebuilder.RebuildRequest(context.Background(), ref, mustOpenAIRecoveryPlan(t, "plan.responses.resume", streamgate.RecoveryStrategyContinuationRepair, directive))
|
||||
if err != nil {
|
||||
t.Fatalf("RebuildRequest: %v", err)
|
||||
}
|
||||
lease, err := rebuilder.RebuiltStore().take(draft.RequestRef())
|
||||
if err != nil {
|
||||
t.Fatalf("take: %v", err)
|
||||
}
|
||||
defer lease.release()
|
||||
got, err := lease.body()
|
||||
if err != nil {
|
||||
t.Fatalf("body: %v", err)
|
||||
}
|
||||
if bytes.Contains(got, []byte(callerInput)) || bytes.Contains(got, []byte(callerInstructions)) {
|
||||
t.Fatalf("responses resume copied caller input or instructions: %s", got)
|
||||
}
|
||||
var rebuilt struct {
|
||||
Model string `json:"model"`
|
||||
Stream bool `json:"stream"`
|
||||
Instructions string `json:"instructions"`
|
||||
Input []struct {
|
||||
Type string `json:"type"`
|
||||
Role string `json:"role"`
|
||||
Content []struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
} `json:"input"`
|
||||
}
|
||||
if err := json.Unmarshal(got, &rebuilt); err != nil {
|
||||
t.Fatalf("decode resume body: %v", err)
|
||||
}
|
||||
if rebuilt.Model != "served-responses" || rebuilt.Stream || rebuilt.Instructions != openAIRepeatResumeDirective || len(rebuilt.Input) != 2 {
|
||||
t.Fatalf("responses resume envelope = %#v", rebuilt)
|
||||
}
|
||||
if got := rebuilt.Input[0]; got.Type != "reasoning" || len(got.Content) != 1 || got.Content[0].Type != "reasoning_text" || got.Content[0].Text != reasoning {
|
||||
t.Fatalf("responses reasoning provenance = %#v", got)
|
||||
}
|
||||
if got := rebuilt.Input[1]; got.Type != "message" || got.Role != "assistant" || len(got.Content) != 1 || got.Content[0].Type != "output_text" || got.Content[0].Text != content {
|
||||
t.Fatalf("responses assistant provenance = %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIResponsesRepeatResumeAdmissionAcceptsBuilderBody(t *testing.T) {
|
||||
const callerInput = "CALLER_RESPONSES_INPUT_MUST_NOT_BE_COPIED"
|
||||
const callerInstructions = "CALLER_INSTRUCTIONS_MUST_NOT_BE_COPIED"
|
||||
const content = "MODEL_OUTPUT_SENTINEL"
|
||||
const reasoning = "MODEL_THINK_SENTINEL"
|
||||
body := []byte(`{"model":"served-responses","stream":false,"instructions":"` + callerInstructions + `","input":"` + callerInput + `"}`)
|
||||
_, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointResponses, body, 4096)
|
||||
source.recordText(content)
|
||||
source.recordReasoning(reasoning)
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len(content), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
draft, err := rebuilder.RebuildRequest(context.Background(), ref, mustOpenAIRecoveryPlan(t, "plan.responses.resume.admission", streamgate.RecoveryStrategyContinuationRepair, directive))
|
||||
if err != nil {
|
||||
t.Fatalf("RebuildRequest: %v", err)
|
||||
}
|
||||
lease, err := rebuilder.RebuiltStore().take(draft.RequestRef())
|
||||
if err != nil {
|
||||
t.Fatalf("take: %v", err)
|
||||
}
|
||||
defer lease.release()
|
||||
resumeBody, err := lease.body()
|
||||
if err != nil {
|
||||
t.Fatalf("body: %v", err)
|
||||
}
|
||||
|
||||
srv := NewServer(config.EdgeOpenAIConf{Adapter: "ollama", Target: "served", SessionID: "cli"}, &fakeRunService{}, nil)
|
||||
base := newTestRequestContext(t, routeDispatch{Adapter: "ollama", Target: "served", SessionID: "cli"}, body)
|
||||
base.endpoint = usageEndpointResponses
|
||||
requestCtx := &responsesRequestContext{openAIRequestContext: base, envelope: responsesEnvelope{Model: "served-responses"}}
|
||||
initial, err := srv.newResponsesDispatchContext(requestCtx, responsesRequest{Model: "served-responses", Input: json.RawMessage(`"initial"`)})
|
||||
if err != nil {
|
||||
t.Fatalf("newResponsesDispatchContext: %v", err)
|
||||
}
|
||||
state := &openAIResponsesAttemptContext{}
|
||||
admission, err := newOpenAIResponsesRecoveryAdmissionBuilder(srv, initial, state)(context.Background(), streamgate.RebuiltRequest{}, resumeBody)
|
||||
if err != nil {
|
||||
t.Fatalf("resume admission: %v", err)
|
||||
}
|
||||
if admission.kind != openAIAdmissionRun || admission.run.Prompt == "" {
|
||||
t.Fatalf("resume admission = %#v, want normalized run", admission)
|
||||
}
|
||||
got := state.get()
|
||||
if got == nil {
|
||||
t.Fatal("resume admission did not retain dispatch context")
|
||||
}
|
||||
if strings.Contains(got.prompt, callerInput) || strings.Contains(got.prompt, callerInstructions) {
|
||||
t.Fatalf("resume prompt copied caller history: %q", got.prompt)
|
||||
}
|
||||
for _, want := range []string{openAIRepeatResumeDirective, content, reasoning} {
|
||||
if !strings.Contains(got.prompt, want) {
|
||||
t.Fatalf("resume prompt %q does not preserve %q", got.prompt, want)
|
||||
}
|
||||
}
|
||||
if gotValue, ok := got.input["responses_resume_content"].(string); !ok || gotValue != content {
|
||||
t.Fatalf("resume content input = %#v", got.input["responses_resume_content"])
|
||||
}
|
||||
if gotValue, ok := got.input["responses_resume_reasoning"].(string); !ok || gotValue != reasoning {
|
||||
t.Fatalf("resume reasoning input = %#v", got.input["responses_resume_reasoning"])
|
||||
}
|
||||
|
||||
if err := decodeResponsesRequest(json.NewDecoder(bytes.NewReader(resumeBody)), &responsesRequest{}); err != nil {
|
||||
// Strict decoding is intentionally structural; public string-only input
|
||||
// is enforced at dispatch-context construction below.
|
||||
t.Fatalf("strict request decode unexpectedly failed: %v", err)
|
||||
}
|
||||
var public responsesRequest
|
||||
if err := decodeResponsesRequest(json.NewDecoder(bytes.NewReader(resumeBody)), &public); err != nil {
|
||||
t.Fatalf("public decode: %v", err)
|
||||
}
|
||||
if _, err := srv.newResponsesDispatchContext(requestCtx, public); err == nil {
|
||||
t.Fatal("public Responses array input was admitted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRecoverySourceStoreLifecycle(t *testing.T) {
|
||||
ingress, err := buildOpenAIIngressSnapshot(4096, []byte(`{"model":"m","messages":[]}`), json.RawMessage(`{"model":"m","messages":[]}`))
|
||||
if err != nil {
|
||||
t.Fatalf("buildOpenAIIngressSnapshot: %v", err)
|
||||
}
|
||||
defer ingress.Close()
|
||||
source := newOpenAIRecoverySourceStore(ingress)
|
||||
source.recordText("first")
|
||||
source.recordReasoning("think")
|
||||
content, reasoning, err := source.consume(5)
|
||||
if err != nil || content != "first" || reasoning != "think" {
|
||||
t.Fatalf("first consume = (%q, %q, %v)", content, reasoning, err)
|
||||
}
|
||||
if _, _, err := source.consume(0); !errors.Is(err, errOpenAIRecoverySourceUnavailable) {
|
||||
t.Fatalf("second consume error = %v, want unavailable", err)
|
||||
}
|
||||
source.resetAttempt()
|
||||
source.recordText("next")
|
||||
content, reasoning, err = source.consume(4)
|
||||
if err != nil || content != "next" || reasoning != "" {
|
||||
t.Fatalf("reset consume = (%q, %q, %v)", content, reasoning, err)
|
||||
}
|
||||
source.close()
|
||||
if _, _, err := source.consume(0); !errors.Is(err, streamgate.ErrIngressSnapshotClosed) {
|
||||
t.Fatalf("closed consume error = %v, want ErrIngressSnapshotClosed", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatGuardReasoningCursor(t *testing.T) {
|
||||
ingress, err := buildOpenAIIngressSnapshot(
|
||||
4096,
|
||||
[]byte(`{"model":"m","messages":[]}`),
|
||||
json.RawMessage(`{"model":"m","messages":[]}`),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("buildOpenAIIngressSnapshot: %v", err)
|
||||
}
|
||||
defer ingress.Close()
|
||||
source := newOpenAIRecoverySourceStore(ingress)
|
||||
source.recordText("content-prefix")
|
||||
source.recordReasoning("reasoning-prefix-repeated-tail")
|
||||
cursor := len("content-prefix") + 1 + len("reasoning-prefix-")
|
||||
content, reasoning, err := source.consume(cursor)
|
||||
if err != nil {
|
||||
t.Fatalf("consume reasoning cursor: %v", err)
|
||||
}
|
||||
if content != "content-prefix" || reasoning != "reasoning-prefix-" {
|
||||
t.Fatalf("reasoning cursor result = (%q, %q)", content, reasoning)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRepeatGuardKnownPrefixSuppression(t *testing.T) {
|
||||
ingress, err := buildOpenAIIngressSnapshot(
|
||||
4096,
|
||||
[]byte(`{"model":"m","messages":[]}`),
|
||||
json.RawMessage(`{"model":"m","messages":[]}`),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("buildOpenAIIngressSnapshot: %v", err)
|
||||
}
|
||||
defer ingress.Close()
|
||||
source := newOpenAIRecoverySourceStore(ingress)
|
||||
source.recordText("safe-prefix")
|
||||
if _, _, err := source.consume(len("safe-prefix")); err != nil {
|
||||
t.Fatalf("consume: %v", err)
|
||||
}
|
||||
source.resetAttempt()
|
||||
if got := source.suppressKnownPrefix(true, "safe-"); got != "" {
|
||||
t.Fatalf("first echoed fragment = %q, want suppressed", got)
|
||||
}
|
||||
if got := source.suppressKnownPrefix(true, "prefixnovel"); got != "novel" {
|
||||
t.Fatalf("second echoed fragment = %q, want novel suffix", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRequestRebuilderRepeatResumeContextOverflow(t *testing.T) {
|
||||
body := []byte(`{"model":"served-chat","messages":[{"role":"user","content":"caller"}]}`)
|
||||
ingress, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointChat, body, 1)
|
||||
source.recordText("model output")
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len("model output"), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
_, err = rebuilder.RebuildRequest(context.Background(), ref, mustOpenAIRecoveryPlan(t, "plan.resume.overflow", streamgate.RecoveryStrategyContinuationRepair, directive))
|
||||
if !errors.Is(err, errOpenAIRebuildContextOverflow) {
|
||||
t.Fatalf("RebuildRequest error = %v, want context overflow", err)
|
||||
}
|
||||
if got := len(rebuilder.RebuiltStore().leases); got != 0 {
|
||||
t.Fatalf("overflow retained %d dispatchable requests", got)
|
||||
}
|
||||
accessor, err := ingress.accessor()
|
||||
if err != nil {
|
||||
t.Fatalf("ingress accessor: %v", err)
|
||||
}
|
||||
if got := accessor.ReservedTempBytes(); got != 0 {
|
||||
t.Fatalf("overflow reserved bytes = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAIRepeatResumeContextOverflowPreservesBudget pins the host half of
|
||||
// the coordinator contract: continuation rebuild refusal creates no dispatchable
|
||||
// request lease. Recovery usage is consumed only immediately before an outbound
|
||||
// dispatcher call, so this fail-closed path leaves the fault budget at zero.
|
||||
func TestOpenAIRepeatResumeContextOverflowPreservesBudget(t *testing.T) {
|
||||
body := []byte(`{"model":"served-chat","messages":[{"role":"user","content":"caller"}]}`)
|
||||
_, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointChat, body, 1)
|
||||
source.recordText("model output")
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len("model output"), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
intent, err := streamgate.NewRecoveryIntent(streamgate.RecoveryStrategyContinuationRepair, directive, "repeat_detected", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryIntent: %v", err)
|
||||
}
|
||||
arbitration, err := streamgate.NewArbitrationResult(streamgate.ArbitrationActionRecover, streamgate.BaseDispositionTerminalErrorCandidate, "repeat_guard", "repeat_detected", &intent, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewArbitrationResult: %v", err)
|
||||
}
|
||||
policy, err := streamgate.NewRecoveryPolicySnapshot(3, map[streamgate.RecoveryStrategy]int{streamgate.RecoveryStrategyContinuationRepair: 3})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryPolicySnapshot: %v", err)
|
||||
}
|
||||
usage, err := streamgate.NewRecoveryUsageSnapshot(0, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryUsageSnapshot: %v", err)
|
||||
}
|
||||
trace := &openAIResumeTrace{}
|
||||
controller := &openAIResumeTestController{trace: trace}
|
||||
binding, err := streamgate.NewAttemptBinding("attempt.initial", "served-chat", "provider", "normalized", openAIResumeTestEventSource{}, controller)
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptBinding: %v", err)
|
||||
}
|
||||
dispatcher := &openAIResumeNoDispatch{}
|
||||
coordinator, err := streamgate.NewRecoveryCoordinator(streamgate.RecoveryCoordinatorOptions{
|
||||
Policy: policy,
|
||||
Usage: usage,
|
||||
RequestSnapshot: ref,
|
||||
CurrentBinding: binding,
|
||||
Rebuilder: openAIResumeTracingRebuilder{inner: rebuilder, trace: trace},
|
||||
Dispatcher: dispatcher,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryCoordinator: %v", err)
|
||||
}
|
||||
result, err := coordinator.Execute(context.Background(), streamgate.RecoveryCycleInput{
|
||||
Arbitration: arbitration, PlanID: "plan.resume.budget", IdempotencyKey: "plan.resume.budget:key", ConsumerID: "openai.edge",
|
||||
CommitState: streamgate.CommitStateStreamOpen,
|
||||
})
|
||||
if !errors.Is(err, streamgate.ErrRecoveryRebuildFailed) {
|
||||
t.Fatalf("Execute error = %v, want ErrRecoveryRebuildFailed", err)
|
||||
}
|
||||
if controller.calls != 1 || dispatcher.calls != 0 {
|
||||
t.Fatalf("context refusal lifecycle = aborts:%d dispatches:%d, want 1/0", controller.calls, dispatcher.calls)
|
||||
}
|
||||
if got, want := trace.snapshot(), []string{"abort", "rebuild"}; !slices.Equal(got, want) {
|
||||
t.Fatalf("context refusal order = %v, want %v", got, want)
|
||||
}
|
||||
if got := coordinator.UsageSnapshot().RequestFaultRecoveries(); got != 0 {
|
||||
t.Fatalf("coordinator fault usage = %d, want 0", got)
|
||||
}
|
||||
if got := result.UsageSnapshot().RequestFaultRecoveries(); got != 0 {
|
||||
t.Fatalf("result fault usage = %d, want 0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatResumeDoesNotUsePreparer(t *testing.T) {
|
||||
body := []byte(`{"model":"served-responses","input":"CALLER_INPUT_MUST_NOT_BE_COPIED"}`)
|
||||
_, rebuilder, source, ref := newOpenAIResumeRebuilderFixture(t, openAIRebuildEndpointResponses, body, 4096)
|
||||
source.recordText("safe prefix repeated tail")
|
||||
|
||||
fake := &fakeRunService{eventRuns: []chan *iop.RunEvent{bufferedRunEvents(&iop.RunEvent{Type: "complete"})}}
|
||||
srv := NewServer(config.EdgeOpenAIConf{Adapter: "ollama", Target: "served", SessionID: "cli"}, fake, nil)
|
||||
base := newTestRequestContext(t, routeDispatch{Adapter: "ollama", Target: "served", SessionID: "cli"}, body)
|
||||
base.endpoint = usageEndpointResponses
|
||||
requestCtx := &responsesRequestContext{openAIRequestContext: base, envelope: responsesEnvelope{Model: "served-responses"}}
|
||||
dc, err := srv.newResponsesDispatchContext(requestCtx, responsesRequest{Model: "served-responses", Input: json.RawMessage(`"CALLER_INPUT_MUST_NOT_BE_COPIED"`)})
|
||||
if err != nil {
|
||||
t.Fatalf("newResponsesDispatchContext: %v", err)
|
||||
}
|
||||
state := &openAIResponsesAttemptContext{}
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(srv.service, rebuilder.RebuiltStore(), newOpenAIResponsesRecoveryAdmissionBuilder(srv, dc, state), func(transport openAIAttemptTransport) (streamgate.NormalizedEventSource, error) {
|
||||
if transport.run == nil || state.get() == nil {
|
||||
return nil, errors.New("resume was not admitted")
|
||||
}
|
||||
return newOpenAIResponsesEventSource(state.get(), transport.run, &openAIResponsesResultHolder{}, nil), nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("newOpenAIAttemptDispatcher: %v", err)
|
||||
}
|
||||
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(len("safe prefix "), source.snapshotRef())
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryDirectiveContinuation: %v", err)
|
||||
}
|
||||
intent, err := streamgate.NewRecoveryIntent(streamgate.RecoveryStrategyContinuationRepair, directive, "repeat_detected", 10)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryIntent: %v", err)
|
||||
}
|
||||
arbitration, err := streamgate.NewArbitrationResult(streamgate.ArbitrationActionRecover, streamgate.BaseDispositionTerminalErrorCandidate, "repeat_guard", "repeat_detected", &intent, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewArbitrationResult: %v", err)
|
||||
}
|
||||
policy, err := streamgate.NewRecoveryPolicySnapshot(3, map[streamgate.RecoveryStrategy]int{streamgate.RecoveryStrategyContinuationRepair: 3})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryPolicySnapshot: %v", err)
|
||||
}
|
||||
usage, err := streamgate.NewRecoveryUsageSnapshot(0, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryUsageSnapshot: %v", err)
|
||||
}
|
||||
trace := &openAIResumeTrace{}
|
||||
controller := &openAIResumeTestController{trace: trace}
|
||||
binding, err := streamgate.NewAttemptBinding("attempt.initial", "served-responses", "provider", "normalized", openAIResumeTestEventSource{}, controller)
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptBinding: %v", err)
|
||||
}
|
||||
preparer := &openAIResumeRecordingPreparer{}
|
||||
coordinator, err := streamgate.NewRecoveryCoordinator(streamgate.RecoveryCoordinatorOptions{
|
||||
Policy: policy, Usage: usage, RequestSnapshot: ref, CurrentBinding: binding,
|
||||
Rebuilder: openAIResumeTracingRebuilder{inner: rebuilder, trace: trace},
|
||||
Dispatcher: openAIResumeTracingDispatcher{inner: dispatcher, trace: trace},
|
||||
Preparers: map[string]streamgate.RecoveryPlanPreparer{"recording": preparer}, PreparationTimeout: time.Second,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewRecoveryCoordinator: %v", err)
|
||||
}
|
||||
result, err := coordinator.Execute(context.Background(), streamgate.RecoveryCycleInput{
|
||||
Arbitration: arbitration, PlanID: "plan.resume.no-preparer", IdempotencyKey: "plan.resume.no-preparer:key", ConsumerID: "openai.edge",
|
||||
CommitState: streamgate.CommitStateStreamOpen,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Execute: %v", err)
|
||||
}
|
||||
if preparer.calls != 0 {
|
||||
t.Fatalf("continuation invoked preparer %d times, want 0", preparer.calls)
|
||||
}
|
||||
if controller.calls != 1 || len(fake.reqsSnapshot()) != 1 {
|
||||
t.Fatalf("lifecycle aborts=%d dispatches=%d, want one abort then one dispatch", controller.calls, len(fake.reqsSnapshot()))
|
||||
}
|
||||
if got, want := trace.snapshot(), []string{"abort", "rebuild", "dispatch"}; !slices.Equal(got, want) {
|
||||
t.Fatalf("recovery order = %v, want %v", got, want)
|
||||
}
|
||||
if got := result.UsageSnapshot().RequestFaultRecoveries(); got != 1 {
|
||||
t.Fatalf("fault usage after actual dispatch = %d, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRequestRebuilderExactByteIdentity(t *testing.T) {
|
||||
body := []byte("{\n \"unknown\" : [1, 2], \"model\" : \"alias\", \"messages\" : []\n}\n")
|
||||
_, rebuilder, ref := newOpenAIRebuilderFixture(t, openAIRebuildEndpointChat, body, 4096)
|
||||
|
|
@ -142,6 +662,63 @@ func TestOpenAIRequestRebuilderResponsesSchemaPatch(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestOpenAIResponsesCodecEndpointDistinctUnknownPreservation proves the S18
|
||||
// Responses lossless codec: a Responses rebuild patches the top-level "input"
|
||||
// field while preserving unknown top-level fields and unknown/encrypted input
|
||||
// items, and the same patch shape on the Chat endpoint targets "messages"
|
||||
// instead (endpoint-specific, no shared parser).
|
||||
func TestOpenAIResponsesCodecEndpointDistinctUnknownPreservation(t *testing.T) {
|
||||
respBody := []byte(`{ "model" : "alias", "input" : [ {"type":"reasoning","encrypted":"zzz"}, {"type":"message","role":"user","content":"old"} ], "custom" : [3, 2, 1], "future" : {"x":true} }`)
|
||||
_, respRebuilder, respRef := newOpenAIRebuilderFixture(t, openAIRebuildEndpointResponses, respBody, 8192)
|
||||
respPatch := json.RawMessage(`[{"type":"message","role":"user","content":"fixed"}]`)
|
||||
if err := respRebuilder.PatchStore().PutContinuation("snapshot.resp", 5, respPatch); err != nil {
|
||||
t.Fatalf("PutContinuation(responses): %v", err)
|
||||
}
|
||||
respDirective, _ := streamgate.NewRecoveryDirectiveContinuation(5, "snapshot.resp")
|
||||
respPlan := mustOpenAIRecoveryPlan(t, "plan.resp", streamgate.RecoveryStrategyContinuationRepair, respDirective)
|
||||
respDraft, err := respRebuilder.RebuildRequest(context.Background(), respRef, respPlan)
|
||||
if err != nil {
|
||||
t.Fatalf("RebuildRequest(responses): %v", err)
|
||||
}
|
||||
respLease, _ := respRebuilder.RebuiltStore().take(respDraft.RequestRef())
|
||||
defer respLease.release()
|
||||
respGot, _ := respLease.body()
|
||||
wantResp := bytes.Replace(respBody, []byte(`[ {"type":"reasoning","encrypted":"zzz"}, {"type":"message","role":"user","content":"old"} ]`), respPatch, 1)
|
||||
if !bytes.Equal(respGot, wantResp) {
|
||||
t.Fatalf("responses rebuild changed non-target bytes:\n got=%s\nwant=%s", respGot, wantResp)
|
||||
}
|
||||
for _, keep := range []string{`"custom" : [3, 2, 1]`, `"future" : {"x":true}`} {
|
||||
if !bytes.Contains(respGot, []byte(keep)) {
|
||||
t.Errorf("responses rebuild dropped unknown top-level field %q", keep)
|
||||
}
|
||||
}
|
||||
|
||||
// Same patch shape on the Chat endpoint targets "messages", not "input";
|
||||
// a Chat body's "input" field is an unknown field and must be preserved.
|
||||
chatBody := []byte(`{ "model" : "alias", "messages" : [ {"role":"user","content":"old"} ], "input" : "unknown-to-chat" }`)
|
||||
_, chatRebuilder, chatRef := newOpenAIRebuilderFixture(t, openAIRebuildEndpointChat, chatBody, 8192)
|
||||
chatPatch := json.RawMessage(`[{"role":"user","content":"fixed"}]`)
|
||||
if err := chatRebuilder.PatchStore().PutContinuation("snapshot.chat", 5, chatPatch); err != nil {
|
||||
t.Fatalf("PutContinuation(chat): %v", err)
|
||||
}
|
||||
chatDirective, _ := streamgate.NewRecoveryDirectiveContinuation(5, "snapshot.chat")
|
||||
chatPlan := mustOpenAIRecoveryPlan(t, "plan.chat", streamgate.RecoveryStrategyContinuationRepair, chatDirective)
|
||||
chatDraft, err := chatRebuilder.RebuildRequest(context.Background(), chatRef, chatPlan)
|
||||
if err != nil {
|
||||
t.Fatalf("RebuildRequest(chat): %v", err)
|
||||
}
|
||||
chatLease, _ := chatRebuilder.RebuiltStore().take(chatDraft.RequestRef())
|
||||
defer chatLease.release()
|
||||
chatGot, _ := chatLease.body()
|
||||
wantChat := bytes.Replace(chatBody, []byte(`[ {"role":"user","content":"old"} ]`), chatPatch, 1)
|
||||
if !bytes.Equal(chatGot, wantChat) {
|
||||
t.Fatalf("chat rebuild changed non-target bytes:\n got=%s\nwant=%s", chatGot, wantChat)
|
||||
}
|
||||
if !bytes.Contains(chatGot, []byte(`"input" : "unknown-to-chat"`)) {
|
||||
t.Errorf("chat rebuild dropped its unknown top-level \"input\" field")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRequestRebuilderPatchPlusOutputPeakOverflow(t *testing.T) {
|
||||
body := []byte(`{"model":"m","messages":[]}`)
|
||||
patch := json.RawMessage(`[{"role":"user","content":"larger"}]`)
|
||||
|
|
|
|||
|
|
@ -25,12 +25,12 @@ func (s *Server) tunnelChatCompletionPassthrough(w http.ResponseWriter, dc *chat
|
|||
}
|
||||
metricLabels := dc.usageLabels(s, responseModePassthrough)
|
||||
|
||||
// Runtime-enabled streaming: the Core request runtime owns
|
||||
// Runtime-enabled tunnel: the Core request runtime owns
|
||||
// response-start staging and commits status/header only at first safe
|
||||
// release. Non-streaming tunnel passthrough is unaffected: it has no
|
||||
// eager-commit-before-evidence problem since the body is already fully
|
||||
// buffered before any write.
|
||||
if s.streamGateEnabled() && dc.req.Stream {
|
||||
if s.streamGateEnabled() {
|
||||
s.runOpenAITunnelStreamGate(w, dc.r, s.openAIChatTunnelStreamGateRequest(dc), handle, metricLabels)
|
||||
return
|
||||
}
|
||||
|
|
@ -54,8 +54,10 @@ func (s *Server) openAIChatTunnelStreamGateRequest(dc *chatDispatchContext) open
|
|||
endpoint: openAIRebuildEndpointChat,
|
||||
method: http.MethodPost,
|
||||
path: "/v1/chat/completions",
|
||||
stream: dc.req.Stream,
|
||||
modelGroupKey: strings.TrimSpace(dc.req.Model),
|
||||
metadata: metadata,
|
||||
hasScheme: chatRequestHasSchemeMetadata(dc.req.Metadata),
|
||||
estimate: dc.estimate,
|
||||
contextClass: dc.contextClass,
|
||||
requestModel: dc.req.Model,
|
||||
|
|
@ -90,8 +92,10 @@ func (s *Server) openAIResponsesPoolTunnelStreamGateRequest(
|
|||
endpoint: openAIRebuildEndpointResponses,
|
||||
method: http.MethodPost,
|
||||
path: "/v1/responses",
|
||||
stream: requestCtx.envelope.Stream,
|
||||
modelGroupKey: strings.TrimSpace(requestCtx.envelope.Model),
|
||||
metadata: metadata,
|
||||
hasScheme: chatRequestHasSchemeMetadata(requestCtx.envelope.Metadata),
|
||||
estimate: requestCtx.estimate,
|
||||
contextClass: requestCtx.contextClass,
|
||||
// requestModel stays empty: Responses passthrough prefers
|
||||
|
|
@ -477,20 +481,22 @@ func (s *Server) tunnelResponsesPassthrough(w http.ResponseWriter, requestCtx *r
|
|||
zap.String("queue_reason", handle.Dispatch().QueueReason),
|
||||
)
|
||||
|
||||
// Runtime-enabled streaming: the Core request runtime owns
|
||||
// Runtime-enabled tunnel: the Core request runtime owns
|
||||
// response-start staging and commits status/header only at first safe
|
||||
// release. requestModel stays empty so the recovery rewrite path never
|
||||
// touches the provider-echoed model, matching legacy Responses
|
||||
// passthrough behavior.
|
||||
if s.streamGateEnabled() && requestCtx.envelope.Stream {
|
||||
if s.streamGateEnabled() {
|
||||
streamGateReq := openAITunnelStreamGateRequest{
|
||||
route: requestCtx.route,
|
||||
ingress: requestCtx.ingress,
|
||||
endpoint: openAIRebuildEndpointResponses,
|
||||
method: http.MethodPost,
|
||||
path: "/v1/responses",
|
||||
stream: requestCtx.envelope.Stream,
|
||||
modelGroupKey: strings.TrimSpace(requestCtx.envelope.Model),
|
||||
metadata: metadata,
|
||||
hasScheme: chatRequestHasSchemeMetadata(requestCtx.envelope.Metadata),
|
||||
estimate: requestCtx.estimate,
|
||||
contextClass: requestCtx.contextClass,
|
||||
requestModel: "",
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ import (
|
|||
|
||||
func (s *Server) completeResponse(w http.ResponseWriter, dc *responsesDispatchContext, handle edgeservice.RunResult) {
|
||||
metricLabels := dc.usageLabels(s, responseModeNormalized)
|
||||
text, reasoning, _, _, usage, _, err := collectRunResult(dc.r.Context(), handle.Stream(), handle.WaitTimeout())
|
||||
text, reasoning, _, toolCalls, usage, _, err := collectRunResult(dc.r.Context(), handle.Stream(), handle.WaitTimeout())
|
||||
if err != nil {
|
||||
s.cancelRunOnHTTPGiveUp(handle.Dispatch(), err)
|
||||
emitUsageMetrics(metricLabels, usageStatusForError(err), usageObservation{})
|
||||
|
|
@ -39,14 +39,7 @@ func (s *Server) completeResponse(w http.ResponseWriter, dc *responsesDispatchCo
|
|||
CreatedAt: time.Now().Unix(),
|
||||
Model: responseModel(dc.req.Model, handle.Dispatch().Target),
|
||||
OutputText: text,
|
||||
Output: []responsesOutputItem{{
|
||||
Type: "message",
|
||||
Role: "assistant",
|
||||
Content: []responsesContentItem{{
|
||||
Type: "output_text",
|
||||
Text: text,
|
||||
}},
|
||||
}},
|
||||
Usage: u,
|
||||
Output: responsesOutputItems(text, toolCalls),
|
||||
Usage: u,
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@ import (
|
|||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
func decodeResponsesRequest(dec *json.Decoder, req *responsesRequest) error {
|
||||
|
|
@ -38,6 +40,239 @@ func decodeResponsesRequest(dec *json.Decoder, req *responsesRequest) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// decodeOpenAIResponsesRepeatHistory is intentionally independent from the
|
||||
// Chat messages decoder. It reads only plain Responses input provenance and
|
||||
// reduces it to bounded fingerprints; encrypted and unknown input items remain
|
||||
// canonical-only data and never become mutation candidates.
|
||||
func decodeOpenAIResponsesRepeatHistory(rawBody []byte) (openAIRepeatHistorySnapshot, error) {
|
||||
var root struct {
|
||||
Input json.RawMessage `json:"input"`
|
||||
}
|
||||
if err := json.Unmarshal(rawBody, &root); err != nil {
|
||||
return openAIRepeatHistorySnapshot{}, fmt.Errorf("decode Responses repeat history: %w", err)
|
||||
}
|
||||
builder := newOpenAIRepeatHistoryBuilder()
|
||||
var inputString string
|
||||
if json.Unmarshal(root.Input, &inputString) == nil {
|
||||
builder.recordText("user", openAIRepeatChannelContent, inputString)
|
||||
return builder.snapshot(), nil
|
||||
}
|
||||
var items []json.RawMessage
|
||||
if json.Unmarshal(root.Input, &items) != nil {
|
||||
return builder.snapshot(), nil
|
||||
}
|
||||
pendingActions := make(map[string]streamgate.FixedFingerprint)
|
||||
for _, rawItem := range items {
|
||||
var item map[string]json.RawMessage
|
||||
if json.Unmarshal(rawItem, &item) != nil {
|
||||
continue
|
||||
}
|
||||
var itemType string
|
||||
if json.Unmarshal(item["type"], &itemType) != nil {
|
||||
continue
|
||||
}
|
||||
switch itemType {
|
||||
case "message":
|
||||
var role string
|
||||
if json.Unmarshal(item["role"], &role) != nil {
|
||||
continue
|
||||
}
|
||||
for _, text := range openAIRepeatResponsesTextParts(item["content"], "input_text", "output_text", "text") {
|
||||
builder.recordText(role, openAIRepeatChannelContent, text)
|
||||
}
|
||||
case "reasoning":
|
||||
for _, text := range openAIRepeatResponsesTextParts(item["content"], "reasoning_text") {
|
||||
builder.recordText("assistant", openAIRepeatChannelReasoning, text)
|
||||
}
|
||||
case "function_call":
|
||||
var callID, name string
|
||||
var arguments json.RawMessage
|
||||
_ = json.Unmarshal(item["call_id"], &callID)
|
||||
if callID == "" {
|
||||
_ = json.Unmarshal(item["id"], &callID)
|
||||
}
|
||||
_ = json.Unmarshal(item["name"], &name)
|
||||
arguments = openAIRepeatArgumentsValue(item["arguments"])
|
||||
if fingerprint, ok := openAIRepeatActionFingerprint(name, arguments); ok && strings.TrimSpace(callID) != "" {
|
||||
pendingActions[callID] = fingerprint
|
||||
}
|
||||
case "function_call_output", "function_call_error":
|
||||
var callID string
|
||||
_ = json.Unmarshal(item["call_id"], &callID)
|
||||
action, ok := pendingActions[callID]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
var result string
|
||||
if json.Unmarshal(item["output"], &result) != nil {
|
||||
_ = json.Unmarshal(item["error"], &result)
|
||||
}
|
||||
if fingerprint, ok := openAIRepeatTextFingerprint(result); ok {
|
||||
builder.recordCompletedAction(action, fingerprint)
|
||||
}
|
||||
}
|
||||
}
|
||||
return builder.snapshot(), nil
|
||||
}
|
||||
|
||||
func openAIRepeatResponsesTextParts(raw json.RawMessage, allowedTypes ...string) []string {
|
||||
allowed := make(map[string]struct{}, len(allowedTypes))
|
||||
for _, value := range allowedTypes {
|
||||
allowed[value] = struct{}{}
|
||||
}
|
||||
var parts []json.RawMessage
|
||||
if json.Unmarshal(raw, &parts) != nil {
|
||||
return nil
|
||||
}
|
||||
values := make([]string, 0, len(parts))
|
||||
for _, rawPart := range parts {
|
||||
var part map[string]json.RawMessage
|
||||
if json.Unmarshal(rawPart, &part) != nil {
|
||||
continue
|
||||
}
|
||||
var partType, text string
|
||||
if json.Unmarshal(part["type"], &partType) != nil {
|
||||
continue
|
||||
}
|
||||
if _, ok := allowed[partType]; !ok {
|
||||
continue
|
||||
}
|
||||
if json.Unmarshal(part["text"], &text) == nil {
|
||||
values = append(values, text)
|
||||
}
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
// decodeOpenAIResponsesResumeRequest accepts only the private recovery body
|
||||
// emitted by buildOpenAIResponsesResumeBody. It must never be used for public
|
||||
// /v1/responses ingress: public arrays remain rejected by parseResponsesInput.
|
||||
//
|
||||
// The decoder is intentionally exact. It admits only a non-streaming model,
|
||||
// the fixed resume directive, and either an assistant output item or a
|
||||
// reasoning item immediately followed by that assistant output item. It does
|
||||
// not trim, normalize, or reinterpret recorded text.
|
||||
func decodeOpenAIResponsesResumeRequest(body []byte) (responsesResumeRequest, error) {
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.Unmarshal(body, &raw); err != nil {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
if len(raw) < 4 || len(raw) > 5 {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
for key := range raw {
|
||||
switch key {
|
||||
case "model", "stream", "instructions", "input", "temperature":
|
||||
default:
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
}
|
||||
|
||||
var model string
|
||||
if err := json.Unmarshal(raw["model"], &model); err != nil || strings.TrimSpace(model) == "" {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
var stream bool
|
||||
if err := json.Unmarshal(raw["stream"], &stream); err != nil || stream {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
var instructions string
|
||||
if err := json.Unmarshal(raw["instructions"], &instructions); err != nil || instructions != openAIRepeatResumeDirective {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
var temperature *float64
|
||||
if rawTemperature, ok := raw["temperature"]; ok {
|
||||
var value float64
|
||||
if err := json.Unmarshal(rawTemperature, &value); err != nil || value < 0 || value > 2 {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
temperature = &value
|
||||
}
|
||||
|
||||
var items []json.RawMessage
|
||||
if err := json.Unmarshal(raw["input"], &items); err != nil || len(items) < 1 || len(items) > 2 {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
|
||||
decodeTextItem := func(rawItem json.RawMessage, wantType string) (string, error) {
|
||||
var item map[string]json.RawMessage
|
||||
if err := json.Unmarshal(rawItem, &item); err != nil {
|
||||
return "", err
|
||||
}
|
||||
wantItemFields := 2
|
||||
if wantType == "message" {
|
||||
wantItemFields = 3
|
||||
}
|
||||
if len(item) != wantItemFields {
|
||||
return "", fmt.Errorf("unexpected item fields")
|
||||
}
|
||||
var itemType string
|
||||
if err := json.Unmarshal(item["type"], &itemType); err != nil || itemType != wantType {
|
||||
return "", fmt.Errorf("unexpected item type")
|
||||
}
|
||||
if wantType == "message" {
|
||||
var role string
|
||||
if err := json.Unmarshal(item["role"], &role); err != nil || role != "assistant" {
|
||||
return "", fmt.Errorf("unexpected item role")
|
||||
}
|
||||
}
|
||||
var content []json.RawMessage
|
||||
if err := json.Unmarshal(item["content"], &content); err != nil || len(content) != 1 {
|
||||
return "", fmt.Errorf("invalid item content")
|
||||
}
|
||||
var part map[string]json.RawMessage
|
||||
if err := json.Unmarshal(content[0], &part); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(part) != 2 {
|
||||
return "", fmt.Errorf("unexpected content fields")
|
||||
}
|
||||
partType := "output_text"
|
||||
if wantType == "reasoning" {
|
||||
partType = "reasoning_text"
|
||||
}
|
||||
var gotPartType, text string
|
||||
if err := json.Unmarshal(part["type"], &gotPartType); err != nil || gotPartType != partType {
|
||||
return "", fmt.Errorf("unexpected content type")
|
||||
}
|
||||
if err := json.Unmarshal(part["text"], &text); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return text, nil
|
||||
}
|
||||
|
||||
var content, reasoning string
|
||||
if len(items) == 2 {
|
||||
var err error
|
||||
reasoning, err = decodeTextItem(items[0], "reasoning")
|
||||
if err != nil {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
content, err = decodeTextItem(items[1], "message")
|
||||
if err != nil {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
} else {
|
||||
var err error
|
||||
content, err = decodeTextItem(items[0], "message")
|
||||
if err != nil {
|
||||
return responsesResumeRequest{}, fmt.Errorf("invalid OpenAI Responses resume request")
|
||||
}
|
||||
}
|
||||
|
||||
return responsesResumeRequest{
|
||||
Request: responsesRequest{
|
||||
Model: model,
|
||||
Input: raw["input"],
|
||||
Instructions: instructions,
|
||||
Stream: false,
|
||||
Temperature: temperature,
|
||||
},
|
||||
Content: content,
|
||||
Reasoning: reasoning,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// decodeResponsesEnvelope leniently extracts only the routing-relevant fields
|
||||
// (model, metadata, stream, background) from a /v1/responses request body. It
|
||||
// does not reject unknown fields so the provider tunnel passthrough can forward
|
||||
|
|
|
|||
|
|
@ -132,7 +132,6 @@ func (s *Server) handleResponses(w http.ResponseWriter, r *http.Request) {
|
|||
writeError(w, http.StatusBadGateway, "node_dispatch_error", err.Error())
|
||||
return
|
||||
}
|
||||
defer handle.Close()
|
||||
|
||||
s.logger.Info("openai responses dispatch",
|
||||
zap.String("run_id", handle.Dispatch().RunID),
|
||||
|
|
@ -148,6 +147,11 @@ func (s *Server) handleResponses(w http.ResponseWriter, r *http.Request) {
|
|||
zap.String("queue_reason", handle.Dispatch().QueueReason),
|
||||
)
|
||||
|
||||
if s.streamGateEnabled() {
|
||||
s.runOpenAIResponsesStreamGate(w, dc, handle)
|
||||
return
|
||||
}
|
||||
defer handle.Close()
|
||||
s.completeResponse(w, dc, handle)
|
||||
}
|
||||
|
||||
|
|
@ -202,6 +206,29 @@ func (s *Server) newResponsesDispatchContext(requestCtx *responsesRequestContext
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.newResponsesDispatchContextFromInput(requestCtx, req, inputStr, "", "", false)
|
||||
}
|
||||
|
||||
// newResponsesResumeDispatchContext constructs a normalized run from the
|
||||
// already-validated private continuation shape. Public request parsing cannot
|
||||
// enter this path, so string-only /v1/responses ingress remains unchanged.
|
||||
func (s *Server) newResponsesResumeDispatchContext(requestCtx *responsesRequestContext, resume responsesResumeRequest) (*responsesDispatchContext, error) {
|
||||
resumeInput := resume.Content
|
||||
if resume.Reasoning != "" {
|
||||
// Keep both channels byte-for-byte while using only the deterministic
|
||||
// separator already defined by buildResponsesPrompt.
|
||||
resumeInput = buildResponsesPrompt(resume.Reasoning, resume.Content)
|
||||
}
|
||||
return s.newResponsesDispatchContextFromInput(requestCtx, resume.Request, resumeInput, resume.Content, resume.Reasoning, true)
|
||||
}
|
||||
|
||||
func (s *Server) newResponsesDispatchContextFromInput(requestCtx *responsesRequestContext, req responsesRequest, inputStr, resumeContent, resumeReasoning string, isResume bool) (*responsesDispatchContext, error) {
|
||||
if req.Stream {
|
||||
return nil, fmt.Errorf("streaming is not supported for /v1/responses")
|
||||
}
|
||||
if req.Background {
|
||||
return nil, fmt.Errorf("background is not supported for /v1/responses")
|
||||
}
|
||||
prompt := buildResponsesPrompt(req.Instructions, inputStr)
|
||||
outputPolicy := s.resolveOutputPolicy(prompt)
|
||||
if instruction := strictOutputContractInstruction(outputPolicy); instruction != "" {
|
||||
|
|
@ -219,6 +246,12 @@ func (s *Server) newResponsesDispatchContext(requestCtx *responsesRequestContext
|
|||
runMetadata["openai_stream"] = fmt.Sprintf("%t", req.Stream)
|
||||
runMetadata["strict_output"] = fmt.Sprintf("%t", outputPolicy.Strict)
|
||||
input := map[string]any{"prompt": prompt}
|
||||
if isResume {
|
||||
// Preserve recovery channel provenance for the normalized execution
|
||||
// path without exposing it in the public request contract.
|
||||
input["responses_resume_content"] = resumeContent
|
||||
input["responses_resume_reasoning"] = resumeReasoning
|
||||
}
|
||||
if defaultThinkingTokenBudget > 0 {
|
||||
input["think"] = true
|
||||
input["thinking_token_budget"] = defaultThinkingTokenBudget
|
||||
|
|
@ -335,6 +368,20 @@ func (s *Server) handleResponsesProviderPool(w http.ResponseWriter, requestCtx *
|
|||
Tunnel: baseTunnel,
|
||||
}
|
||||
|
||||
if s.streamGateEnabled() {
|
||||
fctx, err := s.openAIResponsesOutputFilterContext(requestCtx)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
predicate, err := openAIStreamGateCandidatePredicate(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "run_error", "stream gate runtime unavailable")
|
||||
return
|
||||
}
|
||||
poolReq.AcceptCandidate = predicate
|
||||
}
|
||||
|
||||
// Pre-dispatch provider auth header injection. Runs inside SubmitProviderPool
|
||||
// BEFORE buildProviderTunnelRequest and the Node Send step, so the auth
|
||||
// header lands on the actual wire request. On failure the slot is released
|
||||
|
|
@ -404,6 +451,10 @@ func (s *Server) handleResponsesProviderPool(w http.ResponseWriter, requestCtx *
|
|||
writeError(w, http.StatusBadRequest, "invalid_request_error", "provider auth token is required")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, edgeservice.ErrProviderPoolCandidateRejected) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", openAIStreamGateCandidateRejectedMessage)
|
||||
return
|
||||
}
|
||||
if isValidationError(err) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
|
||||
return
|
||||
|
|
@ -434,11 +485,12 @@ func (s *Server) handleResponsesProviderPool(w http.ResponseWriter, requestCtx *
|
|||
// an error and no tunnel handle exists. Provider bytes are relayed as
|
||||
// pure passthrough; caller metadata never selects a sideband surface.
|
||||
metricLabels := s.usageLabelsFor(r.Context(), strings.TrimSpace(env.Model), usageEndpointResponses, responseModePassthrough)
|
||||
// Runtime-enabled streaming: the Core request runtime owns response-start
|
||||
// staging, and every recovery re-enters SubmitProviderPool through the
|
||||
// same runtime instead of pinning the initially selected candidate.
|
||||
if s.streamGateEnabled() && env.Stream {
|
||||
s.runOpenAITunnelStreamGate(w, r, s.openAIResponsesPoolTunnelStreamGateRequest(requestCtx, poolReq, runMeta), result.Tunnel, metricLabels)
|
||||
// Runtime-enabled: the Core request runtime owns response-start staging,
|
||||
// and every recovery re-enters SubmitProviderPool through the
|
||||
// Responses-specific runtime instead of pinning the initially selected
|
||||
// candidate or reusing the caller-derived normalized context.
|
||||
if s.streamGateEnabled() {
|
||||
s.runOpenAIResponsesPoolStreamGate(w, requestCtx, poolReq, result.Tunnel)
|
||||
return
|
||||
}
|
||||
s.writeProviderTunnelResponse(w, r, result.Tunnel, env.Stream, env.Model, metricLabels)
|
||||
|
|
@ -459,6 +511,11 @@ func (s *Server) handleResponsesProviderPool(w http.ResponseWriter, requestCtx *
|
|||
}
|
||||
// Relay the prepared normalized context so strict-output XML wrapping
|
||||
// and the exact derived metadata survive the provider-pool path.
|
||||
if s.streamGateEnabled() {
|
||||
s.runOpenAIResponsesStreamGate(w, preparedDispatch.withPoolDispatch(poolReq), handle)
|
||||
return
|
||||
}
|
||||
defer handle.Close()
|
||||
s.completeResponse(w, preparedDispatch, handle)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -271,3 +271,50 @@ func TestResponsesStrictOutputNormalizesAgentResponse(t *testing.T) {
|
|||
t.Fatalf("output[0].content[0].text wrong: %+v", resp.Output)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponsesStreamGateNormalizedHandlerUsesSingleTerminalBarrier(t *testing.T) {
|
||||
fake := &fakeRunService{events: bufferedRunEvents(
|
||||
&iop.RunEvent{Type: "reasoning_delta", Delta: "private reasoning"},
|
||||
&iop.RunEvent{Type: "delta", Delta: "hello"},
|
||||
&iop.RunEvent{Type: "complete", Metadata: map[string]string{
|
||||
runtimeMetadataOpenAIToolCalls: `[{"id":"call-1","type":"function","function":{"name":"lookup","arguments":"{}"}}]`,
|
||||
}},
|
||||
)}
|
||||
srv := NewServer(config.EdgeOpenAIConf{
|
||||
Adapter: "ollama", Target: "served-model",
|
||||
StreamEvidenceGate: config.StreamEvidenceGateConf{
|
||||
Enabled: true,
|
||||
Filters: []config.StreamGateFilterPolicyConf{
|
||||
{Filter: config.StreamGateFilterRepeatGuard},
|
||||
{Filter: config.StreamGateFilterSchemaGate},
|
||||
{Filter: config.StreamGateFilterProviderError},
|
||||
},
|
||||
},
|
||||
}, fake, nil)
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/responses", strings.NewReader(`{
|
||||
"model":"client-model",
|
||||
"input":"say hello",
|
||||
"metadata":{"scheme":"object"}
|
||||
}`))
|
||||
w := newRecordingResponseWriter()
|
||||
srv.routes().ServeHTTP(w, req)
|
||||
if w.code != http.StatusOK {
|
||||
t.Fatalf("status=%d body=%s", w.code, w.body.String())
|
||||
}
|
||||
if w.headerCallCount() != 1 {
|
||||
t.Fatalf("WriteHeader calls=%d, want one terminal commit", w.headerCallCount())
|
||||
}
|
||||
var response responsesResponse
|
||||
if err := json.Unmarshal([]byte(w.body.String()), &response); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if response.OutputText != "hello" {
|
||||
t.Fatalf("output_text=%q", response.OutputText)
|
||||
}
|
||||
if len(response.Output) != 2 || response.Output[1].Type != "function_call" || response.Output[1].Name != "lookup" {
|
||||
t.Fatalf("responses output shape=%+v", response.Output)
|
||||
}
|
||||
if strings.Count(w.body.String(), `"object":"response"`) != 1 {
|
||||
t.Fatalf("response opening count body=%s", w.body.String())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
1266
apps/edge/internal/openai/responses_stream_gate.go
Normal file
1266
apps/edge/internal/openai/responses_stream_gate.go
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -18,6 +18,17 @@ type responsesRequest struct {
|
|||
TopP *float64 `json:"top_p,omitempty"`
|
||||
}
|
||||
|
||||
// responsesResumeRequest is the private, endpoint-native continuation shape
|
||||
// emitted by buildOpenAIResponsesResumeBody. It is deliberately separate from
|
||||
// responsesRequest: public normalized ingress continues to accept string input
|
||||
// only, while a recovery attempt may carry the already-validated assistant
|
||||
// output items needed to resume an aborted provider attempt.
|
||||
type responsesResumeRequest struct {
|
||||
Request responsesRequest
|
||||
Content string
|
||||
Reasoning string
|
||||
}
|
||||
|
||||
// responsesEnvelope holds the routing-relevant fields decoded leniently from a
|
||||
// /v1/responses request body before strict normalization. Unknown fields are
|
||||
// ignored so the provider tunnel passthrough can forward Codex/Responses
|
||||
|
|
@ -50,9 +61,13 @@ type responsesResponse struct {
|
|||
}
|
||||
|
||||
type responsesOutputItem struct {
|
||||
Type string `json:"type"`
|
||||
Role string `json:"role"`
|
||||
Content []responsesContentItem `json:"content"`
|
||||
Type string `json:"type"`
|
||||
Role string `json:"role,omitempty"`
|
||||
Content []responsesContentItem `json:"content,omitempty"`
|
||||
ID string `json:"id,omitempty"`
|
||||
CallID string `json:"call_id,omitempty"`
|
||||
Name string `json:"name,omitempty"`
|
||||
Arguments string `json:"arguments,omitempty"`
|
||||
}
|
||||
|
||||
type responsesContentItem struct {
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package openai
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
|
|
@ -62,6 +63,35 @@ type openAIAttemptTransport struct {
|
|||
|
||||
type openAIAttemptEventSourceFactory func(openAIAttemptTransport) (streamgate.NormalizedEventSource, error)
|
||||
|
||||
// openAIRecoveryAdmissionState carries only the sanitized classification of a
|
||||
// recovery dispatch failure from the AttemptDispatcher to the release sink.
|
||||
// Core intentionally hides raw host dispatcher errors, while the OpenAI host
|
||||
// still has to preserve the public admission contract: an all-candidates
|
||||
// capability rejection is a pre-dispatch 400 on initial, queued, and recovery
|
||||
// admission alike.
|
||||
type openAIRecoveryAdmissionState struct {
|
||||
mu sync.Mutex
|
||||
candidateRejected bool
|
||||
}
|
||||
|
||||
func (s *openAIRecoveryAdmissionState) record(err error) {
|
||||
if s == nil || !errors.Is(err, edgeservice.ErrProviderPoolCandidateRejected) {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.candidateRejected = true
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAIRecoveryAdmissionState) rejected() bool {
|
||||
if s == nil {
|
||||
return false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.candidateRejected
|
||||
}
|
||||
|
||||
// openAIAttemptDispatcher adapts the three existing Edge admission surfaces
|
||||
// to Core AttemptDispatcher. Provider/model/path values are never accepted
|
||||
// from the rebuilder; they come exclusively from RunDispatch after admission.
|
||||
|
|
@ -70,6 +100,7 @@ type openAIAttemptDispatcher struct {
|
|||
store *openAIRebuiltRequestStore
|
||||
build openAIAttemptAdmissionBuilder
|
||||
eventSource openAIAttemptEventSourceFactory
|
||||
state *openAIRecoveryAdmissionState
|
||||
}
|
||||
|
||||
func newOpenAIAttemptDispatcher(
|
||||
|
|
@ -81,7 +112,14 @@ func newOpenAIAttemptDispatcher(
|
|||
if service == nil || store == nil || build == nil || eventSource == nil {
|
||||
return nil, fmt.Errorf("OpenAI attempt dispatcher dependencies are required")
|
||||
}
|
||||
return &openAIAttemptDispatcher{service: service, store: store, build: build, eventSource: eventSource}, nil
|
||||
return &openAIAttemptDispatcher{
|
||||
service: service, store: store, build: build, eventSource: eventSource,
|
||||
state: &openAIRecoveryAdmissionState{},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (d *openAIAttemptDispatcher) admissionState() *openAIRecoveryAdmissionState {
|
||||
return d.state
|
||||
}
|
||||
|
||||
func (d *openAIAttemptDispatcher) DispatchAttempt(ctx context.Context, request streamgate.RebuiltRequest) (streamgate.AttemptBinding, error) {
|
||||
|
|
@ -116,6 +154,7 @@ func (d *openAIAttemptDispatcher) DispatchAttempt(ctx context.Context, request s
|
|||
|
||||
transport, dispatch, closeTransport, err := d.dispatch(ctx, admission)
|
||||
if err != nil {
|
||||
d.state.record(err)
|
||||
return streamgate.AttemptBinding{}, err
|
||||
}
|
||||
transportOwned := true
|
||||
|
|
|
|||
691
apps/edge/internal/openai/stream_gate_filters.go
Normal file
691
apps/edge/internal/openai/stream_gate_filters.go
Normal file
|
|
@ -0,0 +1,691 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
// This file defines the caller-neutral, Edge-owned semantic output-validation
|
||||
// filters that consume the Stream Evidence Gate Core. Each filter reads only the
|
||||
// immutable FilterContext and EvidenceBatch and returns a sanitized decision plus
|
||||
// an optional typed RecoveryIntent. Core owns hold, all-complete arbitration,
|
||||
// commit, rebuild, and the recovery budget; these filters own only the semantic
|
||||
// judgement and the intent shape.
|
||||
//
|
||||
// The three kinds map directly onto the three Core hold shapes the pipeline must
|
||||
// exercise (SDD S02/S14):
|
||||
//
|
||||
// - repeat_guard: rolling-window participant -> ready/evaluated each epoch.
|
||||
// - schema_gate: terminal-gate participant -> blocking-deferred until the
|
||||
// terminal trigger, then evaluated.
|
||||
// - provider_error: error-event-only participant -> not-applicable on a clean
|
||||
// epoch and observed-unmatched pass on a provider error.
|
||||
// Matcher and recovery intent construction remain follow-up
|
||||
// work; this foundation never creates an exact replay.
|
||||
//
|
||||
// Full repeat detection/repair (repeat-guard Task) and JSON-schema validation
|
||||
// (schema-contract Task) are out of this milestone Task's scope; here the repeat
|
||||
// and schema filters are the rolling/terminal pipeline participants that produce
|
||||
// the correct outcome lifecycle and evidence.
|
||||
|
||||
const (
|
||||
openAIRepeatGuardFilterID = "openai.repeat_guard"
|
||||
openAIRepeatGuardRuleID = "openai.repeat_guard.rolling"
|
||||
openAIRepeatActionGuardFilterID = "openai.repeat_guard.action"
|
||||
openAIRepeatActionGuardRuleID = "openai.repeat_guard.action.terminal"
|
||||
openAISchemaGateFilterID = "openai.schema_gate"
|
||||
openAISchemaGateRuleID = "openai.schema_gate.terminal"
|
||||
openAIProviderErrorFilterID = "openai.provider_error"
|
||||
openAIProviderErrorRuleID = "openai.provider_error.foundation"
|
||||
|
||||
openAIOutputFilterConsumerID = "openai.output_filters"
|
||||
)
|
||||
|
||||
// openAIOutputFilterKind identifies the semantic behavior of an output filter.
|
||||
type openAIOutputFilterKind string
|
||||
|
||||
const (
|
||||
openAIOutputFilterRepeatGuard openAIOutputFilterKind = config.StreamGateFilterRepeatGuard
|
||||
openAIOutputFilterRepeatActionGuard openAIOutputFilterKind = "repeat_action_guard"
|
||||
openAIOutputFilterSchemaGate openAIOutputFilterKind = config.StreamGateFilterSchemaGate
|
||||
openAIOutputFilterProviderError openAIOutputFilterKind = config.StreamGateFilterProviderError
|
||||
)
|
||||
|
||||
// openAIOutputFilter is the single semantic output-validation filter type. Its
|
||||
// kind selects the hold shape (rolling/terminal/none) and the decision it makes.
|
||||
// It is request-local: requestRef and priority are captured at construction so a
|
||||
// provider_error violation intent always carries a priority that matches its
|
||||
// resolved registration priority.
|
||||
type openAIOutputFilter struct {
|
||||
streamgate.FilterBase
|
||||
kind openAIOutputFilterKind
|
||||
ruleID string
|
||||
channel string
|
||||
holdRunes int
|
||||
priority int
|
||||
requestRef string
|
||||
history openAIRepeatHistorySnapshot
|
||||
|
||||
repeatMu sync.Mutex
|
||||
repeatAttemptID string
|
||||
releasedContentBytes int
|
||||
releasedReasoningBytes int
|
||||
}
|
||||
|
||||
// newOpenAIOutputFilter constructs one request-local semantic participant.
|
||||
// history is optional for direct unit fixtures; production always supplies the
|
||||
// endpoint decoder's immutable raw-free snapshot.
|
||||
func newOpenAIOutputFilter(kind openAIOutputFilterKind, holdRunes, priority int, requestRef string, history ...openAIRepeatHistorySnapshot) (*openAIOutputFilter, error) {
|
||||
var id, rule string
|
||||
switch kind {
|
||||
case openAIOutputFilterRepeatGuard:
|
||||
id, rule = openAIRepeatGuardFilterID, openAIRepeatGuardRuleID
|
||||
case openAIOutputFilterRepeatActionGuard:
|
||||
id, rule = openAIRepeatActionGuardFilterID, openAIRepeatActionGuardRuleID
|
||||
case openAIOutputFilterSchemaGate:
|
||||
id, rule = openAISchemaGateFilterID, openAISchemaGateRuleID
|
||||
case openAIOutputFilterProviderError:
|
||||
id, rule = openAIProviderErrorFilterID, openAIProviderErrorRuleID
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported openai output filter kind %q", kind)
|
||||
}
|
||||
if holdRunes <= 0 {
|
||||
holdRunes = config.DefaultStreamGateFilterHoldEvidenceRunes
|
||||
}
|
||||
if priority < 0 {
|
||||
return nil, fmt.Errorf("openai output filter priority must be non-negative")
|
||||
}
|
||||
base, err := streamgate.NewFilterBase(id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
filter := &openAIOutputFilter{
|
||||
FilterBase: base,
|
||||
kind: kind,
|
||||
ruleID: rule,
|
||||
channel: streamGateChannelDefault,
|
||||
holdRunes: holdRunes,
|
||||
priority: priority,
|
||||
requestRef: requestRef,
|
||||
}
|
||||
if len(history) > 0 {
|
||||
filter.history = history[0]
|
||||
}
|
||||
return filter, nil
|
||||
}
|
||||
|
||||
// Applies is execution-path-neutral because endpoint codecs expose the same
|
||||
// semantic event kinds for normalized and provider-tunnel attempts.
|
||||
func (f *openAIOutputFilter) Applies(streamgate.FilterContext) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// HoldRequirement selects the Core hold shape for this filter kind.
|
||||
func (f *openAIOutputFilter) HoldRequirement(streamgate.FilterContext) streamgate.FilterHoldRequirement {
|
||||
switch f.kind {
|
||||
case openAIOutputFilterRepeatGuard:
|
||||
req, err := streamgate.NewFilterHoldRequirementRolling(
|
||||
f.channel,
|
||||
[]streamgate.EventKind{
|
||||
streamgate.EventKindTextDelta,
|
||||
streamgate.EventKindReasoningDelta,
|
||||
},
|
||||
f.holdRunes,
|
||||
)
|
||||
if err != nil {
|
||||
req, _ = streamgate.NewFilterHoldRequirementRolling(f.channel, []streamgate.EventKind{streamgate.EventKindTextDelta}, f.holdRunes)
|
||||
}
|
||||
return req
|
||||
case openAIOutputFilterRepeatActionGuard:
|
||||
req, err := streamgate.NewFilterHoldRequirementTerminalGate(
|
||||
f.channel,
|
||||
[]streamgate.EventKind{
|
||||
streamgate.EventKindToolCallFragment,
|
||||
streamgate.EventKindTerminal,
|
||||
},
|
||||
streamgate.EventKindTerminal,
|
||||
)
|
||||
if err != nil {
|
||||
req, _ = streamgate.NewFilterHoldRequirementTerminalGate(
|
||||
f.channel,
|
||||
[]streamgate.EventKind{streamgate.EventKindToolCallFragment},
|
||||
streamgate.EventKindTerminal,
|
||||
)
|
||||
}
|
||||
return req
|
||||
case openAIOutputFilterSchemaGate:
|
||||
req, err := streamgate.NewFilterHoldRequirementTerminalGate(
|
||||
f.channel,
|
||||
[]streamgate.EventKind{streamgate.EventKindTextDelta, streamgate.EventKindTerminal},
|
||||
streamgate.EventKindTerminal,
|
||||
)
|
||||
if err != nil {
|
||||
req, _ = streamgate.NewFilterHoldRequirementTerminalGate(f.channel, []streamgate.EventKind{streamgate.EventKindTextDelta}, streamgate.EventKindTerminal)
|
||||
}
|
||||
return req
|
||||
default: // provider_error: none mode is trigger-ready only on provider_error.
|
||||
req, err := streamgate.NewFilterHoldRequirementNone(
|
||||
f.channel,
|
||||
[]streamgate.EventKind{streamgate.EventKindProviderError},
|
||||
)
|
||||
if err != nil {
|
||||
req, _ = streamgate.NewFilterHoldRequirementNone(f.channel, []streamgate.EventKind{streamgate.EventKindProviderError})
|
||||
}
|
||||
return req
|
||||
}
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) Evaluate(ctx context.Context, fctx streamgate.FilterContext, batch streamgate.EvidenceBatch) (streamgate.FilterDecision, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
if f.kind == openAIOutputFilterRepeatGuard {
|
||||
return f.evaluateRepeatGuard(fctx, batch)
|
||||
}
|
||||
if f.kind == openAIOutputFilterRepeatActionGuard {
|
||||
return f.evaluateRepeatActionGuard(batch)
|
||||
}
|
||||
events := batch.Events()
|
||||
kind := streamgate.EventKindTextDelta
|
||||
if len(events) > 0 {
|
||||
kind = events[len(events)-1].Kind()
|
||||
}
|
||||
ts := batch.CapturedAt()
|
||||
if ts.IsZero() {
|
||||
ts = time.Now()
|
||||
}
|
||||
|
||||
descriptor := "repeat_rolling_clear"
|
||||
switch f.kind {
|
||||
case openAIOutputFilterSchemaGate:
|
||||
descriptor = "schema_terminal_clear"
|
||||
case openAIOutputFilterProviderError:
|
||||
if batchHasProviderError(batch) {
|
||||
descriptor = "provider_error_observed_unmatched"
|
||||
} else {
|
||||
descriptor = "provider_error_absent"
|
||||
}
|
||||
}
|
||||
evidence, err := streamgate.NewSanitizedEvidence(
|
||||
kind, f.channel, f.ruleID, descriptor,
|
||||
openAIOutputFilterFingerprint(f.ruleID, descriptor), len(events), 0,
|
||||
streamgate.FilterOutcomeKindEvaluated, ts,
|
||||
)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
return streamgate.NewFilterDecision(streamgate.FilterDecisionKindPass, openAIOutputFilterConsumerID, f.ID(), f.ruleID, evidence, nil)
|
||||
}
|
||||
|
||||
type openAIRepeatTextMatch struct {
|
||||
channel openAIRepeatChannel
|
||||
cursorBytes int
|
||||
count int
|
||||
fingerprint streamgate.FixedFingerprint
|
||||
descriptor string
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) evaluateRepeatGuard(fctx streamgate.FilterContext, batch streamgate.EvidenceBatch) (streamgate.FilterDecision, error) {
|
||||
f.repeatMu.Lock()
|
||||
defer f.repeatMu.Unlock()
|
||||
if f.repeatAttemptID != fctx.AttemptID() {
|
||||
f.repeatAttemptID = fctx.AttemptID()
|
||||
f.releasedContentBytes = 0
|
||||
f.releasedReasoningBytes = 0
|
||||
}
|
||||
|
||||
pending := batch.ChannelPending()[f.channel]
|
||||
lookBehind := batch.CommittedLookBehind()[f.channel]
|
||||
events := batch.Events()
|
||||
kind := streamgate.EventKindTextDelta
|
||||
if len(events) > 0 {
|
||||
kind = events[len(events)-1].Kind()
|
||||
}
|
||||
ts := batch.CapturedAt()
|
||||
if ts.IsZero() {
|
||||
ts = time.Now()
|
||||
}
|
||||
|
||||
contentLookBehind, reasoningLookBehind, releasedTool := openAIRepeatEventText(lookBehind)
|
||||
contentPending, reasoningPending, pendingTool := openAIRepeatEventText(pending)
|
||||
contentTail := contentLookBehind + contentPending
|
||||
reasoningTail := reasoningLookBehind + reasoningPending
|
||||
contentBase := f.releasedContentBytes - len(contentLookBehind)
|
||||
reasoningBase := f.releasedReasoningBytes - len(reasoningLookBehind)
|
||||
if contentBase < 0 {
|
||||
contentBase = 0
|
||||
}
|
||||
if reasoningBase < 0 {
|
||||
reasoningBase = 0
|
||||
}
|
||||
|
||||
match, matched := f.findRepeatTextMatch(
|
||||
openAIRepeatChannelContent,
|
||||
contentTail,
|
||||
contentBase,
|
||||
f.releasedContentBytes+len(contentPending),
|
||||
)
|
||||
if !matched {
|
||||
match, matched = f.findRepeatTextMatch(
|
||||
openAIRepeatChannelReasoning,
|
||||
reasoningTail,
|
||||
reasoningBase,
|
||||
f.releasedContentBytes+len(contentPending),
|
||||
)
|
||||
}
|
||||
if matched {
|
||||
// A rolling suffix match that uses committed look-behind must resume at
|
||||
// the released boundary. The bounded suffix can otherwise identify a
|
||||
// later overlapping period inside pending bytes and duplicate part of
|
||||
// the safe prefix. A history anchor keeps its exact pending prefix when
|
||||
// it starts after the boundary, but cannot retract committed bytes.
|
||||
if match.channel == openAIRepeatChannelContent &&
|
||||
(len(contentLookBehind) > 0 &&
|
||||
match.descriptor == "repeat_content_detected" ||
|
||||
match.cursorBytes < f.releasedContentBytes) {
|
||||
match.cursorBytes = f.releasedContentBytes
|
||||
}
|
||||
reasoningReleasedBoundary := f.releasedContentBytes + len(contentPending) + 1 + f.releasedReasoningBytes
|
||||
if match.channel == openAIRepeatChannelReasoning &&
|
||||
(len(reasoningLookBehind) > 0 &&
|
||||
match.descriptor == "repeat_content_detected" ||
|
||||
match.cursorBytes < reasoningReleasedBoundary) {
|
||||
match.cursorBytes = reasoningReleasedBoundary
|
||||
}
|
||||
if fctx.HasToolSideEffect() || releasedTool || pendingTool {
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindFatal,
|
||||
kind,
|
||||
"repeat_after_tool_boundary",
|
||||
match.fingerprint,
|
||||
match.count,
|
||||
match.cursorBytes,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
if f.requestRef == "" {
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindFatal,
|
||||
kind,
|
||||
"repeat_recovery_source_unavailable",
|
||||
match.fingerprint,
|
||||
match.count,
|
||||
match.cursorBytes,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
directive, err := streamgate.NewRecoveryDirectiveContinuation(match.cursorBytes, f.requestRef)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
intent, err := streamgate.NewRecoveryIntent(
|
||||
streamgate.RecoveryStrategyContinuationRepair,
|
||||
directive,
|
||||
"repeat_content_detected",
|
||||
f.priority,
|
||||
)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindViolation,
|
||||
kind,
|
||||
match.descriptor,
|
||||
match.fingerprint,
|
||||
match.count,
|
||||
match.cursorBytes,
|
||||
&intent,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
|
||||
// A successful epoch releases every pending event captured by the Core.
|
||||
// Tracking only byte counts keeps the next committed-look-behind cursor
|
||||
// absolute without retaining any model output beyond the Core's bounded tail.
|
||||
if len(pending) > 0 {
|
||||
f.releasedContentBytes += len(contentPending)
|
||||
f.releasedReasoningBytes += len(reasoningPending)
|
||||
}
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindPass,
|
||||
kind,
|
||||
"repeat_rolling_clear",
|
||||
openAIOutputFilterFingerprint(f.ruleID, "repeat_rolling_clear"),
|
||||
len(events),
|
||||
0,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
|
||||
type openAIRepeatPendingAction struct {
|
||||
id string
|
||||
name string
|
||||
arguments []byte
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) evaluateRepeatActionGuard(batch streamgate.EvidenceBatch) (streamgate.FilterDecision, error) {
|
||||
ts := batch.CapturedAt()
|
||||
if ts.IsZero() {
|
||||
ts = time.Now()
|
||||
}
|
||||
|
||||
actions := make(map[string]*openAIRepeatPendingAction)
|
||||
order := make([]string, 0)
|
||||
for _, ev := range batch.ChannelPending()[f.channel] {
|
||||
if ev.Kind() != streamgate.EventKindToolCallFragment {
|
||||
continue
|
||||
}
|
||||
call, err := ev.AsToolCallFragment()
|
||||
if err != nil {
|
||||
return f.repeatActionFatal("action_fragment_invalid", "", ts)
|
||||
}
|
||||
id := call.ID
|
||||
if id == "" {
|
||||
return f.repeatActionFatal("action_call_id_missing", "", ts)
|
||||
}
|
||||
action := actions[id]
|
||||
if action == nil {
|
||||
action = &openAIRepeatPendingAction{id: id}
|
||||
actions[id] = action
|
||||
order = append(order, id)
|
||||
}
|
||||
if call.Name != "" {
|
||||
if action.name != "" && action.name != call.Name {
|
||||
return f.repeatActionFatal("action_name_conflict", id, ts)
|
||||
}
|
||||
action.name = call.Name
|
||||
}
|
||||
action.arguments = append(action.arguments, call.Arguments...)
|
||||
}
|
||||
|
||||
for _, id := range order {
|
||||
action := actions[id]
|
||||
if action.name == "" {
|
||||
return f.repeatActionFatal("action_name_missing", id, ts)
|
||||
}
|
||||
if len(action.arguments) == 0 || !json.Valid(action.arguments) {
|
||||
return f.repeatActionFatal("action_arguments_incomplete", id, ts)
|
||||
}
|
||||
fingerprint, ok := openAIRepeatActionFingerprint(action.name, json.RawMessage(action.arguments))
|
||||
if !ok {
|
||||
return f.repeatActionFatal("action_fingerprint_unavailable", id, ts)
|
||||
}
|
||||
if count, repeated := f.history.repeatedAction(fingerprint); repeated {
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindFatal,
|
||||
streamgate.EventKindToolCallFragment,
|
||||
"repeated_action_no_progress",
|
||||
fingerprint,
|
||||
count+1,
|
||||
0,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
kind := streamgate.EventKindTerminal
|
||||
if len(order) > 0 {
|
||||
kind = streamgate.EventKindToolCallFragment
|
||||
}
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindPass,
|
||||
kind,
|
||||
"repeat_action_clear",
|
||||
openAIOutputFilterFingerprint(f.ruleID, "repeat_action_clear"),
|
||||
len(order),
|
||||
0,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) repeatActionFatal(descriptor, callID string, ts time.Time) (streamgate.FilterDecision, error) {
|
||||
fingerprint := openAIOutputFilterFingerprint(f.ruleID, descriptor)
|
||||
if callID != "" {
|
||||
fingerprint = streamgate.FixedFingerprint(sha256.Sum256([]byte(f.ruleID + "\x00" + descriptor + "\x00" + callID)))
|
||||
}
|
||||
return f.repeatDecision(
|
||||
streamgate.FilterDecisionKindFatal,
|
||||
streamgate.EventKindToolCallFragment,
|
||||
descriptor,
|
||||
fingerprint,
|
||||
1,
|
||||
0,
|
||||
nil,
|
||||
ts,
|
||||
)
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) findRepeatTextMatch(
|
||||
channel openAIRepeatChannel,
|
||||
tail string,
|
||||
baseBytes int,
|
||||
contentBytesAtViolation int,
|
||||
) (openAIRepeatTextMatch, bool) {
|
||||
if tail == "" {
|
||||
return openAIRepeatTextMatch{}, false
|
||||
}
|
||||
if !f.history.completedProgress() {
|
||||
if candidate, fingerprint, sourceStart, matched := f.findAssistantHistoryAnchor(channel, tail); matched {
|
||||
cursor := baseBytes + sourceStart
|
||||
if channel == openAIRepeatChannelReasoning {
|
||||
cursor = contentBytesAtViolation + 1 + cursor
|
||||
}
|
||||
return openAIRepeatTextMatch{
|
||||
channel: channel, cursorBytes: cursor, count: candidate.count + 1,
|
||||
fingerprint: fingerprint, descriptor: "assistant_history_anchor_repeat",
|
||||
}, true
|
||||
}
|
||||
}
|
||||
|
||||
runes := []rune(tail)
|
||||
minimum := f.holdRunes / 8
|
||||
if minimum < 32 {
|
||||
minimum = 32
|
||||
}
|
||||
if minimum > 128 {
|
||||
minimum = 128
|
||||
}
|
||||
start, count, ok := openAIRepeatedSuffix(runes, minimum)
|
||||
if !ok {
|
||||
return openAIRepeatTextMatch{}, false
|
||||
}
|
||||
repeated := string(runes[start:])
|
||||
fingerprint, ok := openAIRepeatTextFingerprint(repeated)
|
||||
if !ok {
|
||||
return openAIRepeatTextMatch{}, false
|
||||
}
|
||||
cursor := baseBytes + len(string(runes[:start]))
|
||||
if channel == openAIRepeatChannelReasoning {
|
||||
cursor = contentBytesAtViolation + 1 + cursor
|
||||
}
|
||||
return openAIRepeatTextMatch{
|
||||
channel: channel, cursorBytes: cursor, count: count,
|
||||
fingerprint: fingerprint, descriptor: "repeat_content_detected",
|
||||
}, true
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) findAssistantHistoryAnchor(
|
||||
channel openAIRepeatChannel,
|
||||
tail string,
|
||||
) (openAIRepeatHistoryCandidate, streamgate.FixedFingerprint, int, bool) {
|
||||
candidates := f.history.assistantCandidates[channel]
|
||||
if len(candidates) == 0 {
|
||||
return openAIRepeatHistoryCandidate{}, streamgate.FixedFingerprint{}, 0, false
|
||||
}
|
||||
normalized, sourceByteOffsets := normalizeOpenAIRepeatTail(tail)
|
||||
if len(normalized) == 0 {
|
||||
return openAIRepeatHistoryCandidate{}, streamgate.FixedFingerprint{}, 0, false
|
||||
}
|
||||
|
||||
bestStart := -1
|
||||
bestLength := -1
|
||||
var bestCandidate openAIRepeatHistoryCandidate
|
||||
var bestFingerprint streamgate.FixedFingerprint
|
||||
for fingerprint, candidate := range candidates {
|
||||
length := candidate.normalizedRunes
|
||||
if length <= 0 || length > len(normalized) {
|
||||
continue
|
||||
}
|
||||
for start := 0; start+length <= len(normalized); start++ {
|
||||
windowFingerprint := streamgate.FixedFingerprint(sha256.Sum256([]byte(string(normalized[start : start+length]))))
|
||||
if windowFingerprint != fingerprint {
|
||||
continue
|
||||
}
|
||||
sourceStart := sourceByteOffsets[start]
|
||||
if bestStart < 0 || sourceStart < bestStart ||
|
||||
(sourceStart == bestStart && length > bestLength) {
|
||||
bestStart = sourceStart
|
||||
bestLength = length
|
||||
bestCandidate = candidate
|
||||
bestFingerprint = fingerprint
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if bestStart < 0 {
|
||||
return openAIRepeatHistoryCandidate{}, streamgate.FixedFingerprint{}, 0, false
|
||||
}
|
||||
return bestCandidate, bestFingerprint, bestStart, true
|
||||
}
|
||||
|
||||
// normalizeOpenAIRepeatTail mirrors strings.Fields normalization while retaining
|
||||
// only transient source byte offsets. The returned offsets let a normalized
|
||||
// candidate window produce an original UTF-8-safe continuation cursor without
|
||||
// retaining any request-history text.
|
||||
func normalizeOpenAIRepeatTail(value string) ([]rune, []int) {
|
||||
normalized := make([]rune, 0, len([]rune(value)))
|
||||
sourceByteOffsets := make([]int, 0, cap(normalized))
|
||||
inField := false
|
||||
for byteOffset, r := range value {
|
||||
if unicode.IsSpace(r) {
|
||||
inField = false
|
||||
continue
|
||||
}
|
||||
if !inField && len(normalized) > 0 {
|
||||
normalized = append(normalized, ' ')
|
||||
sourceByteOffsets = append(sourceByteOffsets, byteOffset)
|
||||
}
|
||||
normalized = append(normalized, r)
|
||||
sourceByteOffsets = append(sourceByteOffsets, byteOffset)
|
||||
inField = true
|
||||
}
|
||||
return normalized, sourceByteOffsets
|
||||
}
|
||||
|
||||
// openAIRepeatedSuffix finds a rune-safe duplicated suffix. It compares the
|
||||
// current suffix with an earlier suffix at a non-overlapping period, extending
|
||||
// backwards to the start of the observed duplicate. The returned start is the
|
||||
// safe continuation cursor: everything before it is preserved.
|
||||
func openAIRepeatedSuffix(runes []rune, minimum int) (int, int, bool) {
|
||||
if minimum <= 0 || len(runes) < minimum*2 {
|
||||
return 0, 0, false
|
||||
}
|
||||
bestStart, bestCount := 0, 0
|
||||
for period := minimum; period <= len(runes)-minimum; period++ {
|
||||
match := 0
|
||||
for match < period && match < len(runes)-period {
|
||||
left := len(runes) - period - 1 - match
|
||||
right := len(runes) - 1 - match
|
||||
if left < 0 || runes[left] != runes[right] {
|
||||
break
|
||||
}
|
||||
match++
|
||||
}
|
||||
if match < minimum {
|
||||
continue
|
||||
}
|
||||
start := len(runes) - match
|
||||
if match > bestCount || (match == bestCount && start < bestStart) {
|
||||
bestStart, bestCount = start, match
|
||||
}
|
||||
}
|
||||
if bestCount < minimum {
|
||||
return 0, 0, false
|
||||
}
|
||||
return bestStart, 2, true
|
||||
}
|
||||
|
||||
func openAIRepeatEventText(events []streamgate.NormalizedEvent) (string, string, bool) {
|
||||
var content, reasoning string
|
||||
hasTool := false
|
||||
for _, ev := range events {
|
||||
switch ev.Kind() {
|
||||
case streamgate.EventKindTextDelta:
|
||||
if value, err := ev.AsTextDelta(); err == nil {
|
||||
content += value
|
||||
}
|
||||
case streamgate.EventKindReasoningDelta:
|
||||
if value, err := ev.AsReasoningDelta(); err == nil {
|
||||
reasoning += value
|
||||
}
|
||||
case streamgate.EventKindToolCallFragment:
|
||||
hasTool = true
|
||||
}
|
||||
}
|
||||
return content, reasoning, hasTool
|
||||
}
|
||||
|
||||
func (f *openAIOutputFilter) repeatDecision(
|
||||
decisionKind streamgate.FilterDecisionKind,
|
||||
eventKind streamgate.EventKind,
|
||||
descriptor string,
|
||||
fingerprint streamgate.FixedFingerprint,
|
||||
count, offset int,
|
||||
intent *streamgate.RecoveryIntent,
|
||||
ts time.Time,
|
||||
) (streamgate.FilterDecision, error) {
|
||||
evidence, err := streamgate.NewSanitizedEvidence(
|
||||
eventKind,
|
||||
f.channel,
|
||||
f.ruleID,
|
||||
descriptor,
|
||||
fingerprint,
|
||||
count,
|
||||
offset,
|
||||
streamgate.FilterOutcomeKindEvaluated,
|
||||
ts,
|
||||
)
|
||||
if err != nil {
|
||||
return streamgate.FilterDecision{}, err
|
||||
}
|
||||
return streamgate.NewFilterDecision(
|
||||
decisionKind,
|
||||
openAIOutputFilterConsumerID,
|
||||
f.ID(),
|
||||
f.ruleID,
|
||||
evidence,
|
||||
intent,
|
||||
)
|
||||
}
|
||||
|
||||
// batchHasProviderError reports whether the terminal batch carries a provider
|
||||
// error event.
|
||||
func batchHasProviderError(batch streamgate.EvidenceBatch) bool {
|
||||
for _, ev := range batch.Events() {
|
||||
if ev.Kind() == streamgate.EventKindProviderError {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// openAIOutputFilterFingerprint derives a stable, raw-free fingerprint from the
|
||||
// rule id and a sanitized descriptor so evidence carries no provider text.
|
||||
func openAIOutputFilterFingerprint(ruleID, descriptor string) streamgate.FixedFingerprint {
|
||||
return streamgate.FixedFingerprint(sha256.Sum256([]byte(ruleID + "\x00" + descriptor)))
|
||||
}
|
||||
|
||||
var _ streamgate.Filter = (*openAIOutputFilter)(nil)
|
||||
1039
apps/edge/internal/openai/stream_gate_filters_test.go
Normal file
1039
apps/edge/internal/openai/stream_gate_filters_test.go
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -11,6 +11,7 @@ import (
|
|||
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
func TestReadOpenAIIngressBodyBoundary(t *testing.T) {
|
||||
|
|
@ -137,3 +138,43 @@ func TestChatIngressTypedOverflowDoesNotDispatch(t *testing.T) {
|
|||
t.Fatal("typed-view overflow reached provider dispatch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponsesIngressBoundaryAndTypedOverflowZeroDispatch(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
size int
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "limit-minus-one", size: 7},
|
||||
{name: "limit", size: 8},
|
||||
{name: "limit-plus-one", size: 9, wantErr: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
request := httptest.NewRequest(http.MethodPost, "/v1/responses", bytes.NewReader(bytes.Repeat([]byte("x"), tc.size)))
|
||||
_, err := readOpenAIIngressBody(httptest.NewRecorder(), request, 8)
|
||||
if errors.Is(err, errOpenAIIngressTooLarge) != tc.wantErr {
|
||||
t.Fatalf("error=%v wantErr=%t", err, tc.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
body := `{"model":"m","input":"x"}`
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
limit int
|
||||
}{
|
||||
{name: "raw overflow", limit: len(body) - 1},
|
||||
{name: "typed view overflow", limit: len(body)},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
service := &fakeRunService{events: bufferedRunEvents(&iop.RunEvent{Type: "complete"})}
|
||||
server := NewServer(config.EdgeOpenAIConf{Adapter: "ollama", Target: "served", StreamEvidenceGate: config.StreamEvidenceGateConf{MaxIngressSnapshotBytes: tc.limit}}, service, nil)
|
||||
request := httptest.NewRequest(http.MethodPost, "/v1/responses", strings.NewReader(body))
|
||||
recorder := httptest.NewRecorder()
|
||||
server.handleResponses(recorder, request)
|
||||
if recorder.Code != http.StatusRequestEntityTooLarge || len(service.reqsSnapshot()) != 0 {
|
||||
t.Fatalf("status=%d dispatches=%d body=%s", recorder.Code, len(service.reqsSnapshot()), recorder.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
1040
apps/edge/internal/openai/stream_gate_pipeline_test.go
Normal file
1040
apps/edge/internal/openai/stream_gate_pipeline_test.go
Normal file
File diff suppressed because it is too large
Load diff
505
apps/edge/internal/openai/stream_gate_policy.go
Normal file
505
apps/edge/internal/openai/stream_gate_policy.go
Normal file
|
|
@ -0,0 +1,505 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
// chatRequestHasSchemeMetadata reports whether the request carries a non-empty
|
||||
// metadata.scheme output contract. Presence, not shape, is enough to decide
|
||||
// whether the schema_gate filter participates; JSON-schema validation itself is
|
||||
// the schema-contract Task's scope.
|
||||
func chatRequestHasSchemeMetadata(metadata json.RawMessage) bool {
|
||||
if len(metadata) == 0 {
|
||||
return false
|
||||
}
|
||||
var m map[string]json.RawMessage
|
||||
if err := json.Unmarshal(metadata, &m); err != nil {
|
||||
return false
|
||||
}
|
||||
raw, ok := m["scheme"]
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(raw))
|
||||
return trimmed != "" && trimmed != "null"
|
||||
}
|
||||
|
||||
// This file translates the request-stable stream_evidence_gate filter policy
|
||||
// (packages/go/config) into Core FilterRegistrations and FilterPolicyLayers, and
|
||||
// resolves required output-filter capabilities for provider admission. The Core
|
||||
// owns selector precedence, per-target Applies re-resolution, and eligibility;
|
||||
// this file only builds the generation-bound inputs and adapts the config
|
||||
// enums to the Core enums.
|
||||
|
||||
// openAIOutputFilterContext carries the immutable request-start facts used by
|
||||
// selector resolution and endpoint-specific filter participation.
|
||||
type openAIOutputFilterContext struct {
|
||||
environment string
|
||||
endpoint string
|
||||
modelGroup string
|
||||
hasScheme bool
|
||||
requestRef string
|
||||
history openAIRepeatHistorySnapshot
|
||||
}
|
||||
|
||||
const (
|
||||
openAIRepeatHistoryAnchorThreshold = 2
|
||||
openAIRepeatHistoryMaxFingerprints = 1024
|
||||
openAIRepeatHistoryMaxActions = 256
|
||||
)
|
||||
|
||||
type openAIRepeatChannel string
|
||||
|
||||
const (
|
||||
openAIRepeatChannelContent openAIRepeatChannel = "content"
|
||||
openAIRepeatChannelReasoning openAIRepeatChannel = "reasoning"
|
||||
)
|
||||
|
||||
type openAIRepeatHistoryAction struct {
|
||||
actionFingerprint streamgate.FixedFingerprint
|
||||
resultFingerprint streamgate.FixedFingerprint
|
||||
}
|
||||
|
||||
type openAIRepeatHistoryCandidate struct {
|
||||
count int
|
||||
normalizedRunes int
|
||||
}
|
||||
|
||||
// openAIRepeatHistorySnapshot is the immutable, raw-free semantic view passed
|
||||
// from an endpoint decoder to the request-local repeat filter. It intentionally
|
||||
// has no session, caller, TTL, raw text, tool arguments, or tool result fields.
|
||||
type openAIRepeatHistorySnapshot struct {
|
||||
assistantOccurrences map[openAIRepeatChannel]map[streamgate.FixedFingerprint]int
|
||||
userOccurrences map[streamgate.FixedFingerprint]int
|
||||
assistantCandidates map[openAIRepeatChannel]map[streamgate.FixedFingerprint]openAIRepeatHistoryCandidate
|
||||
repeatedActions map[streamgate.FixedFingerprint]int
|
||||
hasCompletedProgress bool
|
||||
hasTerminalProgress bool
|
||||
}
|
||||
|
||||
type openAIRepeatHistoryBuilder struct {
|
||||
assistantOccurrences map[openAIRepeatChannel]map[streamgate.FixedFingerprint]int
|
||||
userOccurrences map[streamgate.FixedFingerprint]int
|
||||
normalizedTextRunes map[streamgate.FixedFingerprint]int
|
||||
completedActions []openAIRepeatHistoryAction
|
||||
hasTerminalProgress bool
|
||||
}
|
||||
|
||||
func newOpenAIRepeatHistoryBuilder() *openAIRepeatHistoryBuilder {
|
||||
return &openAIRepeatHistoryBuilder{
|
||||
assistantOccurrences: map[openAIRepeatChannel]map[streamgate.FixedFingerprint]int{
|
||||
openAIRepeatChannelContent: {},
|
||||
openAIRepeatChannelReasoning: {},
|
||||
},
|
||||
userOccurrences: make(map[streamgate.FixedFingerprint]int),
|
||||
normalizedTextRunes: make(map[streamgate.FixedFingerprint]int),
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeOpenAIRepeatText(value string) string {
|
||||
return strings.Join(strings.Fields(value), " ")
|
||||
}
|
||||
|
||||
func openAIRepeatTextFingerprint(value string) (streamgate.FixedFingerprint, bool) {
|
||||
normalized := normalizeOpenAIRepeatText(value)
|
||||
if normalized == "" {
|
||||
return streamgate.FixedFingerprint{}, false
|
||||
}
|
||||
return streamgate.FixedFingerprint(sha256.Sum256([]byte(normalized))), true
|
||||
}
|
||||
|
||||
func (b *openAIRepeatHistoryBuilder) admitTextFingerprint(fingerprint streamgate.FixedFingerprint, normalizedRunes int) bool {
|
||||
if _, exists := b.normalizedTextRunes[fingerprint]; exists {
|
||||
return true
|
||||
}
|
||||
if len(b.normalizedTextRunes) >= openAIRepeatHistoryMaxFingerprints {
|
||||
return false
|
||||
}
|
||||
b.normalizedTextRunes[fingerprint] = normalizedRunes
|
||||
return true
|
||||
}
|
||||
|
||||
func openAIRepeatActionFingerprint(name string, arguments json.RawMessage) (streamgate.FixedFingerprint, bool) {
|
||||
name = normalizeOpenAIRepeatText(name)
|
||||
if name == "" {
|
||||
return streamgate.FixedFingerprint{}, false
|
||||
}
|
||||
var canonical any
|
||||
if len(arguments) > 0 && json.Unmarshal(arguments, &canonical) == nil {
|
||||
if encoded, err := json.Marshal(canonical); err == nil {
|
||||
arguments = encoded
|
||||
}
|
||||
}
|
||||
normalizedArgs := normalizeOpenAIRepeatText(string(arguments))
|
||||
return streamgate.FixedFingerprint(sha256.Sum256([]byte(name + "\x00" + normalizedArgs))), true
|
||||
}
|
||||
|
||||
func (b *openAIRepeatHistoryBuilder) recordText(role string, channel openAIRepeatChannel, value string) {
|
||||
role = strings.ToLower(strings.TrimSpace(role))
|
||||
if role != "assistant" && role != "user" {
|
||||
return
|
||||
}
|
||||
normalized := normalizeOpenAIRepeatText(value)
|
||||
if normalized == "" {
|
||||
return
|
||||
}
|
||||
fingerprint := streamgate.FixedFingerprint(sha256.Sum256([]byte(normalized)))
|
||||
if !b.admitTextFingerprint(fingerprint, utf8.RuneCountInString(normalized)) {
|
||||
return
|
||||
}
|
||||
switch role {
|
||||
case "assistant":
|
||||
if b.assistantOccurrences[channel] == nil {
|
||||
b.assistantOccurrences[channel] = make(map[streamgate.FixedFingerprint]int)
|
||||
}
|
||||
b.assistantOccurrences[channel][fingerprint]++
|
||||
case "user":
|
||||
b.userOccurrences[fingerprint]++
|
||||
}
|
||||
}
|
||||
|
||||
func (b *openAIRepeatHistoryBuilder) recordCompletedAction(action, result streamgate.FixedFingerprint) {
|
||||
if action == (streamgate.FixedFingerprint{}) || result == (streamgate.FixedFingerprint{}) {
|
||||
return
|
||||
}
|
||||
if len(b.completedActions) >= openAIRepeatHistoryMaxActions {
|
||||
return
|
||||
}
|
||||
b.completedActions = append(b.completedActions, openAIRepeatHistoryAction{
|
||||
actionFingerprint: action,
|
||||
resultFingerprint: result,
|
||||
})
|
||||
}
|
||||
|
||||
func (b *openAIRepeatHistoryBuilder) snapshot() openAIRepeatHistorySnapshot {
|
||||
snapshot := openAIRepeatHistorySnapshot{
|
||||
assistantOccurrences: map[openAIRepeatChannel]map[streamgate.FixedFingerprint]int{
|
||||
openAIRepeatChannelContent: {},
|
||||
openAIRepeatChannelReasoning: {},
|
||||
},
|
||||
userOccurrences: map[streamgate.FixedFingerprint]int{},
|
||||
assistantCandidates: map[openAIRepeatChannel]map[streamgate.FixedFingerprint]openAIRepeatHistoryCandidate{
|
||||
openAIRepeatChannelContent: {},
|
||||
openAIRepeatChannelReasoning: {},
|
||||
},
|
||||
repeatedActions: make(map[streamgate.FixedFingerprint]int),
|
||||
hasTerminalProgress: b.hasTerminalProgress,
|
||||
}
|
||||
for fingerprint, count := range b.userOccurrences {
|
||||
snapshot.userOccurrences[fingerprint] = count
|
||||
}
|
||||
totalAssistant := make(map[streamgate.FixedFingerprint]int)
|
||||
for channel, occurrences := range b.assistantOccurrences {
|
||||
for fingerprint, count := range occurrences {
|
||||
snapshot.assistantOccurrences[channel][fingerprint] = count
|
||||
totalAssistant[fingerprint] += count
|
||||
}
|
||||
}
|
||||
for channel, occurrences := range snapshot.assistantOccurrences {
|
||||
for fingerprint := range occurrences {
|
||||
if snapshot.userOccurrences[fingerprint] == 0 &&
|
||||
totalAssistant[fingerprint] >= openAIRepeatHistoryAnchorThreshold {
|
||||
snapshot.assistantCandidates[channel][fingerprint] = openAIRepeatHistoryCandidate{
|
||||
count: totalAssistant[fingerprint],
|
||||
normalizedRunes: b.normalizedTextRunes[fingerprint],
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(b.completedActions) >= 2 {
|
||||
previous := b.completedActions[len(b.completedActions)-2]
|
||||
current := b.completedActions[len(b.completedActions)-1]
|
||||
snapshot.hasCompletedProgress =
|
||||
previous.resultFingerprint != current.resultFingerprint
|
||||
if previous.actionFingerprint == current.actionFingerprint &&
|
||||
previous.resultFingerprint == current.resultFingerprint {
|
||||
snapshot.repeatedActions[current.actionFingerprint] = 1
|
||||
}
|
||||
}
|
||||
return snapshot
|
||||
}
|
||||
|
||||
func (s openAIRepeatHistorySnapshot) assistantCandidate(channel openAIRepeatChannel, fingerprint streamgate.FixedFingerprint) (int, bool) {
|
||||
candidate, ok := s.assistantCandidates[channel][fingerprint]
|
||||
return candidate.count, ok
|
||||
}
|
||||
|
||||
func (s openAIRepeatHistorySnapshot) assistantCandidateMetadata(channel openAIRepeatChannel, fingerprint streamgate.FixedFingerprint) (openAIRepeatHistoryCandidate, bool) {
|
||||
candidate, ok := s.assistantCandidates[channel][fingerprint]
|
||||
return candidate, ok
|
||||
}
|
||||
|
||||
func (s openAIRepeatHistorySnapshot) repeatedAction(fingerprint streamgate.FixedFingerprint) (int, bool) {
|
||||
count, ok := s.repeatedActions[fingerprint]
|
||||
return count, ok
|
||||
}
|
||||
|
||||
func (s openAIRepeatHistorySnapshot) completedProgress() bool {
|
||||
return s.hasCompletedProgress || s.hasTerminalProgress
|
||||
}
|
||||
|
||||
// streamgateFilterEnforcement adapts a config enforcement string to the Core
|
||||
// enforcement enum, defaulting to blocking for empty/unknown input (validation
|
||||
// rejects unknown values before this point).
|
||||
func streamgateFilterEnforcement(s string) streamgate.FilterEnforcement {
|
||||
if s == config.StreamGateFilterEnforcementObserveOnly {
|
||||
return streamgate.FilterEnforcementObserveOnly
|
||||
}
|
||||
return streamgate.FilterEnforcementBlocking
|
||||
}
|
||||
|
||||
// streamgateSelectorType adapts a config selector type to the Core selector
|
||||
// type. It returns false when the type is unknown.
|
||||
func streamgateSelectorType(s string) (streamgate.PolicySelectorType, bool) {
|
||||
switch s {
|
||||
case config.StreamGateFilterSelectorEnvironment:
|
||||
return streamgate.PolicySelectorEnvironment, true
|
||||
case config.StreamGateFilterSelectorModelGroup:
|
||||
return streamgate.PolicySelectorModelGroup, true
|
||||
case config.StreamGateFilterSelectorModel:
|
||||
return streamgate.PolicySelectorModel, true
|
||||
case config.StreamGateFilterSelectorProvider:
|
||||
return streamgate.PolicySelectorProvider, true
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// openAIOutputFilterRegistrations builds the Core registrations and policy
|
||||
// layers for the configured semantic output filters. A schema_gate filter is
|
||||
// only registered when the caller requested a schema contract; a request with
|
||||
// no scheme neither registers nor requires it. The returned slices are the
|
||||
// request-stable inputs to a generation-bound FilterRegistrySnapshot.
|
||||
func openAIOutputFilterRegistrations(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext) ([]streamgate.FilterRegistration, []streamgate.FilterPolicyLayer, error) {
|
||||
var (
|
||||
regs []streamgate.FilterRegistration
|
||||
policies []streamgate.FilterPolicyLayer
|
||||
)
|
||||
for _, fc := range gateCfg.Filters {
|
||||
if fc.Filter == config.StreamGateFilterSchemaGate && !fctx.hasScheme {
|
||||
continue
|
||||
}
|
||||
|
||||
filter, err := newOpenAIOutputFilter(
|
||||
openAIOutputFilterKind(fc.Filter),
|
||||
fc.EffectiveHoldEvidenceRunes(),
|
||||
fc.Priority,
|
||||
fctx.requestRef,
|
||||
fctx.history,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
filters := []*openAIOutputFilter{filter}
|
||||
if fc.Filter == config.StreamGateFilterRepeatGuard {
|
||||
actionFilter, actionErr := newOpenAIOutputFilter(
|
||||
openAIOutputFilterRepeatActionGuard,
|
||||
fc.EffectiveHoldEvidenceRunes(),
|
||||
fc.Priority,
|
||||
fctx.requestRef,
|
||||
fctx.history,
|
||||
)
|
||||
if actionErr != nil {
|
||||
return nil, nil, actionErr
|
||||
}
|
||||
filters = append(filters, actionFilter)
|
||||
}
|
||||
timeout := time.Duration(fc.EffectiveTimeoutMS()) * time.Millisecond
|
||||
for _, registeredFilter := range filters {
|
||||
reg, err := streamgate.NewFilterRegistration(
|
||||
registeredFilter, fc.EffectiveCapability(), fc.EffectiveEnabled(),
|
||||
streamgateFilterEnforcement(fc.EffectiveEnforcement()), timeout, fc.Priority,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
regs = append(regs, reg)
|
||||
|
||||
for i, sel := range fc.Selectors {
|
||||
selType, ok := streamgateSelectorType(sel.Type)
|
||||
if !ok {
|
||||
return nil, nil, fmt.Errorf("openai output filter %q selector %d unknown type %q", fc.Filter, i, sel.Type)
|
||||
}
|
||||
enabled := fc.EffectiveEnabled()
|
||||
if sel.Enabled != nil {
|
||||
enabled = *sel.Enabled
|
||||
}
|
||||
enforcement := fc.EffectiveEnforcement()
|
||||
if sel.Enforcement != "" {
|
||||
enforcement = sel.Enforcement
|
||||
}
|
||||
layer, err := streamgate.NewFilterPolicyLayer(
|
||||
registeredFilter.ID(), selType, sel.Key, enabled,
|
||||
streamgateFilterEnforcement(enforcement), timeout, fc.Priority,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
policies = append(policies, layer)
|
||||
}
|
||||
}
|
||||
}
|
||||
return regs, policies, nil
|
||||
}
|
||||
|
||||
// openAIOutputFilterRequestContext builds the caller-neutral RequestFilterContext
|
||||
// used to resolve required capabilities and eligibility. It intentionally
|
||||
// carries no caller/agent product name: the same payload resolves identically
|
||||
// for raw HTTP, OpenAI SDK, or any other caller.
|
||||
func openAIOutputFilterRequestContext(fctx openAIOutputFilterContext) (streamgate.RequestFilterContext, error) {
|
||||
return streamgate.NewRequestFilterContext(
|
||||
streamGateConfigGeneration,
|
||||
"admission",
|
||||
fctx.environment,
|
||||
fctx.endpoint,
|
||||
openAIRebuildFamily,
|
||||
"",
|
||||
streamgate.CommitStateTransportUncommitted,
|
||||
false,
|
||||
false,
|
||||
"",
|
||||
)
|
||||
}
|
||||
|
||||
// blockingCapabilitiesForTarget resolves effective policy at the actual target
|
||||
// and excludes observe-only filters from admission.
|
||||
func blockingCapabilitiesForTarget(reqSnap streamgate.RequestFilterSnapshot, target streamgate.AttemptTarget) ([]string, error) {
|
||||
resolved, err := reqSnap.ResolveAttempt(target)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
set := make(map[string]struct{}, len(resolved))
|
||||
for _, filter := range resolved {
|
||||
if filter.Enforcement() != streamgate.FilterEnforcementBlocking {
|
||||
continue
|
||||
}
|
||||
set[filter.Registration().RequiredCapabilityID()] = struct{}{}
|
||||
}
|
||||
required := make([]string, 0, len(set))
|
||||
for capability := range set {
|
||||
required = append(required, capability)
|
||||
}
|
||||
sort.Strings(required)
|
||||
return required, nil
|
||||
}
|
||||
|
||||
func candidateHasCapabilities(target streamgate.AttemptTarget, required []string) bool {
|
||||
for _, capability := range required {
|
||||
if !target.HasCapability(capability) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// openAIStreamGateRequiredCapabilities returns the union of output-filter
|
||||
// capabilities that the given candidate target must advertise. Only
|
||||
// enabled+applicable filters at the target contribute.
|
||||
func openAIStreamGateRequiredCapabilities(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext, target streamgate.AttemptTarget) ([]string, error) {
|
||||
regs, policies, err := openAIOutputFilterRegistrations(gateCfg, fctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
snap, err := streamgate.NewFilterRegistrySnapshot(streamGateConfigGeneration, regs, policies)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reqCtx, err := openAIOutputFilterRequestContext(fctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reqSnap, err := snap.BeginRequest(reqCtx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return blockingCapabilitiesForTarget(reqSnap, target)
|
||||
}
|
||||
|
||||
// errStreamGateRequiredCapabilityUnsupported is the admission-time error a
|
||||
// handler maps to an OpenAI-compatible invalid_request_error (400) when a
|
||||
// required output-filter capability is unsupported by every candidate provider,
|
||||
// before any provider dispatch or recovery budget is consumed.
|
||||
var errStreamGateRequiredCapabilityUnsupported = fmt.Errorf("stream gate: required output filter capability unsupported by all candidates")
|
||||
|
||||
// openAIStreamGateAdmitCandidates returns the subset of candidate targets whose
|
||||
// capability set covers every required output-filter capability resolved at each
|
||||
// candidate's own target context. It returns
|
||||
// errStreamGateRequiredCapabilityUnsupported when no candidate is eligible, so
|
||||
// the caller rejects the request with a pre-dispatch 400 instead of leaving a
|
||||
// required filter silently unenforced.
|
||||
func openAIStreamGateAdmitCandidates(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext, candidates []streamgate.AttemptTarget) ([]streamgate.AttemptTarget, error) {
|
||||
regs, policies, err := openAIOutputFilterRegistrations(gateCfg, fctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// No configured output filters => no required capability => every candidate
|
||||
// is admissible; preserve the existing provider-pool admission unchanged.
|
||||
if len(regs) == 0 {
|
||||
return candidates, nil
|
||||
}
|
||||
snap, err := streamgate.NewFilterRegistrySnapshot(streamGateConfigGeneration, regs, policies)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reqCtx, err := openAIOutputFilterRequestContext(fctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
reqSnap, err := snap.BeginRequest(reqCtx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
eligible := make([]streamgate.AttemptTarget, 0, len(candidates))
|
||||
for _, candidate := range candidates {
|
||||
required, err := blockingCapabilitiesForTarget(reqSnap, candidate)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if candidateHasCapabilities(candidate, required) {
|
||||
eligible = append(eligible, candidate)
|
||||
}
|
||||
}
|
||||
if len(eligible) == 0 {
|
||||
return nil, errStreamGateRequiredCapabilityUnsupported
|
||||
}
|
||||
return eligible, nil
|
||||
}
|
||||
|
||||
// openAIStreamGateCandidatePredicate turns one immutable request policy into a
|
||||
// provider-pool admission predicate. Service invokes it before the first
|
||||
// dispatch and on every recovery/queue re-resolution, so provider-specific
|
||||
// selectors are evaluated against the actual selected target rather than a
|
||||
// caller-supplied model or agent identity.
|
||||
func openAIStreamGateCandidatePredicate(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext) (edgeservice.ProviderPoolCandidatePredicate, error) {
|
||||
regs, _, err := openAIOutputFilterRegistrations(gateCfg, fctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(regs) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
return func(candidate edgeservice.ProviderPoolCandidate) bool {
|
||||
target, err := streamgate.NewAttemptTarget(
|
||||
fctx.modelGroup,
|
||||
candidate.ActualModel,
|
||||
candidate.ProviderID,
|
||||
candidate.ExecutionPath,
|
||||
candidate.LifecycleCapabilities,
|
||||
)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
eligible, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{target})
|
||||
return err == nil && len(eligible) == 1
|
||||
}, nil
|
||||
}
|
||||
579
apps/edge/internal/openai/stream_gate_policy_test.go
Normal file
579
apps/edge/internal/openai/stream_gate_policy_test.go
Normal file
|
|
@ -0,0 +1,579 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
func TestOpenAIRepeatHistoryChatAliases(t *testing.T) {
|
||||
const anchor = "assistant-only anchor"
|
||||
body := []byte(`{
|
||||
"model":"m",
|
||||
"messages":[
|
||||
{"role":"assistant","content":"assistant-only anchor"},
|
||||
{"role":"assistant","reasoning_content":"assistant-only anchor"},
|
||||
{"role":"assistant","reasoning":"assistant-only anchor"},
|
||||
{"role":"assistant","reasoning_text":"assistant-only anchor"},
|
||||
{"role":"assistant","encrypted_reasoning":"RAW_SECRET","signed_reasoning":"RAW_SIGNATURE"}
|
||||
]
|
||||
}`)
|
||||
history, err := decodeOpenAIChatRepeatHistory(body)
|
||||
if err != nil {
|
||||
t.Fatalf("decodeOpenAIChatRepeatHistory: %v", err)
|
||||
}
|
||||
fingerprint, ok := openAIRepeatTextFingerprint(anchor)
|
||||
if !ok {
|
||||
t.Fatal("anchor fingerprint was empty")
|
||||
}
|
||||
if count, ok := history.assistantCandidate(openAIRepeatChannelContent, fingerprint); !ok || count != 4 {
|
||||
t.Fatalf("content candidate = (%d, %v), want (4, true)", count, ok)
|
||||
}
|
||||
if count, ok := history.assistantCandidate(openAIRepeatChannelReasoning, fingerprint); !ok || count != 4 {
|
||||
t.Fatalf("reasoning candidate = (%d, %v), want (4, true)", count, ok)
|
||||
}
|
||||
for _, sentinel := range []string{"RAW_SECRET", "RAW_SIGNATURE"} {
|
||||
sentinelFingerprint, _ := openAIRepeatTextFingerprint(sentinel)
|
||||
if _, ok := history.assistantCandidate(openAIRepeatChannelReasoning, sentinelFingerprint); ok {
|
||||
t.Fatalf("unknown protected field %q became a reasoning candidate", sentinel)
|
||||
}
|
||||
}
|
||||
|
||||
userBody := []byte(`{"messages":[
|
||||
{"role":"user","content":"assistant-only anchor"},
|
||||
{"role":"assistant","reasoning_content":"assistant-only anchor"},
|
||||
{"role":"assistant","reasoning":"assistant-only anchor"}
|
||||
]}`)
|
||||
userHistory, err := decodeOpenAIChatRepeatHistory(userBody)
|
||||
if err != nil {
|
||||
t.Fatalf("decode user-exclusion history: %v", err)
|
||||
}
|
||||
if _, ok := userHistory.assistantCandidate(openAIRepeatChannelReasoning, fingerprint); ok {
|
||||
t.Fatal("user occurrence did not exclude the assistant anchor")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatHistoryAssistantCandidateMetadata(t *testing.T) {
|
||||
const anchor = "assistant\tanchor 한글"
|
||||
fingerprint, ok := openAIRepeatTextFingerprint(anchor)
|
||||
if !ok {
|
||||
t.Fatal("anchor fingerprint was empty")
|
||||
}
|
||||
|
||||
builder := newOpenAIRepeatHistoryBuilder()
|
||||
builder.recordText("assistant", openAIRepeatChannelContent, anchor)
|
||||
builder.recordText("assistant", openAIRepeatChannelReasoning, "assistant anchor 한글")
|
||||
history := builder.snapshot()
|
||||
for _, channel := range []openAIRepeatChannel{
|
||||
openAIRepeatChannelContent,
|
||||
openAIRepeatChannelReasoning,
|
||||
} {
|
||||
metadata, found := history.assistantCandidateMetadata(channel, fingerprint)
|
||||
if !found {
|
||||
t.Fatalf("%s candidate metadata is missing", channel)
|
||||
}
|
||||
if metadata.count != 2 || metadata.normalizedRunes != len([]rune("assistant anchor 한글")) {
|
||||
t.Fatalf("%s metadata = %+v", channel, metadata)
|
||||
}
|
||||
}
|
||||
|
||||
builder.recordText("user", openAIRepeatChannelContent, " assistant anchor\n한글 ")
|
||||
if _, found := builder.snapshot().assistantCandidateMetadata(openAIRepeatChannelContent, fingerprint); found {
|
||||
t.Fatal("normalized user occurrence did not exclude the assistant candidate")
|
||||
}
|
||||
|
||||
capped := newOpenAIRepeatHistoryBuilder()
|
||||
var firstFingerprint streamgate.FixedFingerprint
|
||||
for i := 0; i < openAIRepeatHistoryMaxFingerprints; i++ {
|
||||
value := fmt.Sprintf("bounded assistant anchor %04d", i)
|
||||
capped.recordText("assistant", openAIRepeatChannelContent, value)
|
||||
capped.recordText("assistant", openAIRepeatChannelContent, value)
|
||||
if i == 0 {
|
||||
firstFingerprint, _ = openAIRepeatTextFingerprint(value)
|
||||
}
|
||||
}
|
||||
capped.recordText("assistant", openAIRepeatChannelContent, "candidate beyond cap")
|
||||
capped.recordText("assistant", openAIRepeatChannelContent, "candidate beyond cap")
|
||||
cappedSnapshot := capped.snapshot()
|
||||
if got := len(cappedSnapshot.assistantCandidates[openAIRepeatChannelContent]); got != openAIRepeatHistoryMaxFingerprints {
|
||||
t.Fatalf("candidate cap = %d, want %d", got, openAIRepeatHistoryMaxFingerprints)
|
||||
}
|
||||
beyondFingerprint, _ := openAIRepeatTextFingerprint("candidate beyond cap")
|
||||
if _, found := cappedSnapshot.assistantCandidateMetadata(openAIRepeatChannelContent, beyondFingerprint); found {
|
||||
t.Fatal("candidate beyond the raw-free fingerprint cap was retained")
|
||||
}
|
||||
|
||||
// A user occurrence of an already admitted fingerprint must still be
|
||||
// recorded after the cap is full so provenance exclusion cannot be bypassed.
|
||||
capped.recordText("user", openAIRepeatChannelContent, "bounded assistant anchor 0000")
|
||||
if _, found := capped.snapshot().assistantCandidateMetadata(openAIRepeatChannelContent, firstFingerprint); found {
|
||||
t.Fatal("user exclusion was dropped after the fingerprint cap filled")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatHistoryResponsesProvenance(t *testing.T) {
|
||||
body := []byte(`{"model":"m","input":[
|
||||
{"type":"reasoning","content":[{"type":"reasoning_text","text":"reasoning anchor"},{"type":"encrypted_reasoning","text":"RAW_SECRET"}]},
|
||||
{"type":"reasoning","content":[{"type":"reasoning_text","text":"reasoning anchor"}]},
|
||||
{"type":"function_call","call_id":"call-1","name":"lookup","arguments":"{\"id\":1}"},
|
||||
{"type":"function_call_output","call_id":"call-1","output":"same result"},
|
||||
{"type":"function_call","call_id":"call-2","name":"lookup","arguments":"{\"id\":1}"},
|
||||
{"type":"function_call_output","call_id":"call-2","output":"same result"},
|
||||
{"type":"unknown","payload":"RAW_UNKNOWN"}
|
||||
]}`)
|
||||
history, err := decodeOpenAIResponsesRepeatHistory(body)
|
||||
if err != nil {
|
||||
t.Fatalf("decodeOpenAIResponsesRepeatHistory: %v", err)
|
||||
}
|
||||
anchorFingerprint, _ := openAIRepeatTextFingerprint("reasoning anchor")
|
||||
if count, ok := history.assistantCandidate(openAIRepeatChannelReasoning, anchorFingerprint); !ok || count != 2 {
|
||||
t.Fatalf("reasoning candidate = (%d, %v), want (2, true)", count, ok)
|
||||
}
|
||||
actionFingerprint, _ := openAIRepeatActionFingerprint("lookup", json.RawMessage(`{"id":1}`))
|
||||
if count, ok := history.repeatedAction(actionFingerprint); !ok || count != 1 {
|
||||
t.Fatalf("repeated action = (%d, %v), want (1, true)", count, ok)
|
||||
}
|
||||
for _, sentinel := range []string{"RAW_SECRET", "RAW_UNKNOWN"} {
|
||||
fingerprint, _ := openAIRepeatTextFingerprint(sentinel)
|
||||
if _, ok := history.assistantCandidate(openAIRepeatChannelReasoning, fingerprint); ok {
|
||||
t.Fatalf("protected Responses value %q became a candidate", sentinel)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatHistoryDoesNotInferLineage(t *testing.T) {
|
||||
for name, decode := range map[string]func([]byte) (openAIRepeatHistorySnapshot, error){
|
||||
"chat": decodeOpenAIChatRepeatHistory,
|
||||
"responses": decodeOpenAIResponsesRepeatHistory,
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
history, err := decode([]byte(`{"model":"m","metadata":{"session_id":"caller-session"},"messages":[],"input":"current user input"}`))
|
||||
if err != nil {
|
||||
t.Fatalf("decode history: %v", err)
|
||||
}
|
||||
if len(history.assistantCandidates[openAIRepeatChannelContent]) != 0 ||
|
||||
len(history.assistantCandidates[openAIRepeatChannelReasoning]) != 0 ||
|
||||
len(history.repeatedActions) != 0 {
|
||||
t.Fatalf("omitted history inferred lineage: %+v", history)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatHistoryProgressBoundary(t *testing.T) {
|
||||
noProgressBody := []byte(`{"messages":[
|
||||
{"role":"assistant","tool_calls":[{"id":"a","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"a","content":"same"},
|
||||
{"role":"assistant","tool_calls":[{"id":"b","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"b","content":"same"}
|
||||
]}`)
|
||||
noProgress, err := decodeOpenAIChatRepeatHistory(noProgressBody)
|
||||
if err != nil {
|
||||
t.Fatalf("decode no-progress history: %v", err)
|
||||
}
|
||||
actionFingerprint, _ := openAIRepeatActionFingerprint("lookup", json.RawMessage(`{"id":1}`))
|
||||
if _, ok := noProgress.repeatedAction(actionFingerprint); !ok {
|
||||
t.Fatal("identical completed action/result was not marked no-progress")
|
||||
}
|
||||
if noProgress.completedProgress() {
|
||||
t.Fatal("identical completed action/result was marked progress")
|
||||
}
|
||||
|
||||
progressBody := []byte(`{"messages":[
|
||||
{"role":"assistant","tool_calls":[{"id":"a","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"a","content":"first"},
|
||||
{"role":"assistant","tool_calls":[{"id":"b","type":"function","function":{"name":"other","arguments":"{\"id\":2}"}}]},
|
||||
{"role":"tool","tool_call_id":"b","content":"changed"}
|
||||
]}`)
|
||||
progress, err := decodeOpenAIChatRepeatHistory(progressBody)
|
||||
if err != nil {
|
||||
t.Fatalf("decode progress history: %v", err)
|
||||
}
|
||||
if !progress.completedProgress() {
|
||||
t.Fatal("changed completed result hash was not marked progress")
|
||||
}
|
||||
if len(progress.repeatedActions) != 0 {
|
||||
t.Fatalf("distinct completed action/result was marked repeated: %+v", progress.repeatedActions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIRepeatHistoryLatestPairWins(t *testing.T) {
|
||||
body := []byte(`{"messages":[
|
||||
{"role":"assistant","tool_calls":[{"id":"older-a","type":"function","function":{"name":"lookup","arguments":"{\"id\":0}"}}]},
|
||||
{"role":"tool","tool_call_id":"older-a","content":"older result"},
|
||||
{"role":"assistant","tool_calls":[{"id":"older-b","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"older-b","content":"changed result"},
|
||||
{"role":"assistant","tool_calls":[{"id":"latest-a","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"latest-a","content":"same latest failure"},
|
||||
{"role":"assistant","tool_calls":[{"id":"latest-b","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"latest-b","content":"same latest failure"}
|
||||
]}`)
|
||||
history, err := decodeOpenAIChatRepeatHistory(body)
|
||||
if err != nil {
|
||||
t.Fatalf("decode history: %v", err)
|
||||
}
|
||||
if history.completedProgress() {
|
||||
t.Fatal("latest identical action/result churn was hidden by older progress")
|
||||
}
|
||||
fingerprint, ok := openAIRepeatActionFingerprint("lookup", json.RawMessage(`{"id":1}`))
|
||||
if !ok {
|
||||
t.Fatal("build latest action fingerprint")
|
||||
}
|
||||
if _, repeated := history.repeatedAction(fingerprint); !repeated {
|
||||
t.Fatal("latest identical action/result pair is not blockable")
|
||||
}
|
||||
|
||||
changedLatest := []byte(`{"messages":[
|
||||
{"role":"assistant","tool_calls":[{"id":"a","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"a","content":"first"},
|
||||
{"role":"assistant","tool_calls":[{"id":"b","type":"function","function":{"name":"lookup","arguments":"{\"id\":1}"}}]},
|
||||
{"role":"tool","tool_call_id":"b","content":"changed"}
|
||||
]}`)
|
||||
progress, err := decodeOpenAIChatRepeatHistory(changedLatest)
|
||||
if err != nil {
|
||||
t.Fatalf("decode latest progress history: %v", err)
|
||||
}
|
||||
if !progress.completedProgress() {
|
||||
t.Fatal("latest changed result did not release the progress guard")
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedFilterIDs(resolved []streamgate.ResolvedFilter) map[string]bool {
|
||||
out := make(map[string]bool, len(resolved))
|
||||
for _, r := range resolved {
|
||||
out[r.FilterID()] = true
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestOpenAIStreamGateRequiredCapabilityAdmission verifies that a required
|
||||
// output-filter capability excludes candidates that do not advertise it, and
|
||||
// that a request whose only candidates all lack the capability is rejected
|
||||
// before dispatch (S02/S08 pre-admission 400).
|
||||
func TestOpenAIStreamGateRequiredCapabilityAdmission(t *testing.T) {
|
||||
gateCfg := config.StreamEvidenceGateConf{
|
||||
Enabled: true,
|
||||
Filters: []config.StreamGateFilterPolicyConf{
|
||||
{Filter: config.StreamGateFilterProviderError, Priority: 20},
|
||||
},
|
||||
}
|
||||
fctx := openAIOutputFilterContext{endpoint: openAIRebuildEndpointChat, requestRef: "openai.snap.1"}
|
||||
|
||||
capable, err := streamgate.NewAttemptTarget("client-model", "ornith:35b", "prov-a", "normalized", []string{"output.provider_error"})
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptTarget(capable): %v", err)
|
||||
}
|
||||
incapable, err := streamgate.NewAttemptTarget("client-model", "ornith:35b", "prov-b", "normalized", nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptTarget(incapable): %v", err)
|
||||
}
|
||||
|
||||
eligible, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{capable, incapable})
|
||||
if err != nil {
|
||||
t.Fatalf("admit(mixed): %v", err)
|
||||
}
|
||||
if len(eligible) != 1 || eligible[0].Provider() != "prov-a" {
|
||||
t.Fatalf("mixed admission = %+v, want only prov-a", eligible)
|
||||
}
|
||||
|
||||
if _, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{incapable}); !errors.Is(err, errStreamGateRequiredCapabilityUnsupported) {
|
||||
t.Fatalf("all-incapable admission err = %v, want errStreamGateRequiredCapabilityUnsupported", err)
|
||||
}
|
||||
|
||||
// No configured filters => no required capability => every candidate admitted.
|
||||
empty := config.StreamEvidenceGateConf{Enabled: true}
|
||||
admitted, err := openAIStreamGateAdmitCandidates(empty, fctx, []streamgate.AttemptTarget{incapable})
|
||||
if err != nil {
|
||||
t.Fatalf("admit(no filters): %v", err)
|
||||
}
|
||||
if len(admitted) != 1 {
|
||||
t.Fatalf("no-filter admission = %d, want 1 (unchanged)", len(admitted))
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAIStreamGatePolicySelectorPrecedence verifies that a provider selector
|
||||
// disabling a filter is re-resolved per attempt target: the same request snapshot
|
||||
// keeps the filter active for one provider and drops it for the disabled provider
|
||||
// (S08 provider switch).
|
||||
func TestOpenAIStreamGatePolicySelectorPrecedence(t *testing.T) {
|
||||
gateCfg := config.StreamEvidenceGateConf{
|
||||
Enabled: true,
|
||||
Filters: []config.StreamGateFilterPolicyConf{
|
||||
{
|
||||
Filter: config.StreamGateFilterProviderError,
|
||||
Priority: 20,
|
||||
Selectors: []config.StreamGateFilterSelectorConf{
|
||||
{Type: config.StreamGateFilterSelectorProvider, Key: "prov-disabled", Enabled: boolPtr(false)},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
fctx := openAIOutputFilterContext{endpoint: openAIRebuildEndpointChat, requestRef: "openai.snap.1"}
|
||||
reqSnap := beginOutputFilterRequest(t, gateCfg, fctx)
|
||||
|
||||
enabledTarget := attemptTarget(t, "prov-enabled")
|
||||
disabledTarget := attemptTarget(t, "prov-disabled")
|
||||
|
||||
enabledResolved, err := reqSnap.ResolveAttempt(enabledTarget)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt(enabled): %v", err)
|
||||
}
|
||||
if !resolvedFilterIDs(enabledResolved)[openAIProviderErrorFilterID] {
|
||||
t.Errorf("provider_error not active for prov-enabled")
|
||||
}
|
||||
|
||||
disabledResolved, err := reqSnap.ResolveAttempt(disabledTarget)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt(disabled): %v", err)
|
||||
}
|
||||
if resolvedFilterIDs(disabledResolved)[openAIProviderErrorFilterID] {
|
||||
t.Errorf("provider_error must be inactive for prov-disabled selector")
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAIStreamGateCallerNeutralResolution verifies resolution depends only on
|
||||
// protocol/model/provider/path facts, never on a caller product name (S13).
|
||||
// The three fixture labels identify the originating client only to the test;
|
||||
// the byte-identical protocol payload and every request/filter context passed to
|
||||
// production policy resolution deliberately carry no caller-name field.
|
||||
func TestOpenAIStreamGateCallerNeutralResolution(t *testing.T) {
|
||||
gateCfg := outputFilterGateCfg(config.StreamGateFilterEnforcementBlocking)
|
||||
const protocolPayload = `{"model":"qwen","messages":[{"role":"user","content":"hi"}],"metadata":{"scheme":{"type":"object"}},"stream":true}`
|
||||
fixtures := []struct {
|
||||
name string
|
||||
payload []byte
|
||||
}{
|
||||
{name: "raw HTTP", payload: []byte(protocolPayload)},
|
||||
{name: "OpenAI SDK", payload: []byte(protocolPayload)},
|
||||
{name: "Pi", payload: []byte(protocolPayload)},
|
||||
}
|
||||
var baseline string
|
||||
for _, fixture := range fixtures {
|
||||
t.Run(fixture.name, func(t *testing.T) {
|
||||
if strings.Contains(string(fixture.payload), "caller") || strings.Contains(string(fixture.payload), "agent") {
|
||||
t.Fatalf("fixture unexpectedly contains a caller-name field: %s", fixture.payload)
|
||||
}
|
||||
var req chatCompletionRequest
|
||||
if err := json.Unmarshal(fixture.payload, &req); err != nil {
|
||||
t.Fatalf("decode protocol payload: %v", err)
|
||||
}
|
||||
fctx := openAIOutputFilterContext{
|
||||
environment: config.StreamGateEnvironmentDev,
|
||||
endpoint: openAIRebuildEndpointChat,
|
||||
modelGroup: req.Model,
|
||||
hasScheme: chatRequestHasSchemeMetadata(req.Metadata),
|
||||
requestRef: "openai.snap.caller-neutral",
|
||||
}
|
||||
reqSnap := beginOutputFilterRequest(t, gateCfg, fctx)
|
||||
target, err := streamgate.NewAttemptTarget(req.Model, "qwen:latest", "prov-a", string(edgeservice.ProviderPoolPathNormalized), []string{
|
||||
"output.repeat_guard", "output.schema_gate", "output.provider_error",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptTarget: %v", err)
|
||||
}
|
||||
resolved, err := reqSnap.ResolveAttempt(target)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt: %v", err)
|
||||
}
|
||||
admitted, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{target})
|
||||
if err != nil || len(admitted) != 1 {
|
||||
t.Fatalf("admission decision=(%d,%v), want one admitted target", len(admitted), err)
|
||||
}
|
||||
parts := []string{fmt.Sprintf("path=%s;admitted=%d", target.ExecutionPath(), len(admitted))}
|
||||
for _, filter := range resolved {
|
||||
hold := filter.HoldRequirement()
|
||||
parts = append(parts, fmt.Sprintf("%s:%s:%d:%s:%d", filter.FilterID(), filter.Enforcement(), filter.Priority(), hold.Mode(), hold.EvidenceRunes()))
|
||||
}
|
||||
signature := strings.Join(parts, "|")
|
||||
if baseline == "" {
|
||||
baseline = signature
|
||||
} else if signature != baseline {
|
||||
t.Fatalf("caller-neutral path/threshold/decision changed:\n got %s\nwant %s", signature, baseline)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestOpenAIStreamGateConfigReloadIsolation verifies each request resolves against
|
||||
// the generation snapshot it began with: an enabled-filter generation keeps the
|
||||
// filter, a disabled-filter generation drops it, and a request context cannot
|
||||
// begin against a mismatched generation snapshot (S08 config reload isolation).
|
||||
func TestOpenAIStreamGateConfigReloadIsolation(t *testing.T) {
|
||||
enabledFilter, err := newOpenAIOutputFilter(openAIOutputFilterProviderError, 500, 20, "openai.snap.1")
|
||||
if err != nil {
|
||||
t.Fatalf("newOpenAIOutputFilter: %v", err)
|
||||
}
|
||||
enabledReg, err := streamgate.NewFilterRegistration(enabledFilter, "output.provider_error", true, streamgate.FilterEnforcementBlocking, streamGateFilterTimeout, 20)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFilterRegistration: %v", err)
|
||||
}
|
||||
snapGen1, err := streamgate.NewFilterRegistrySnapshot("edge.gen.1", []streamgate.FilterRegistration{enabledReg}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("snapshot gen1: %v", err)
|
||||
}
|
||||
snapGen2, err := streamgate.NewFilterRegistrySnapshot("edge.gen.2", nil, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("snapshot gen2: %v", err)
|
||||
}
|
||||
|
||||
reqGen1 := beginGenerationRequest(t, snapGen1, "edge.gen.1")
|
||||
reqGen2 := beginGenerationRequest(t, snapGen2, "edge.gen.2")
|
||||
target := attemptTarget(t, "prov-a")
|
||||
|
||||
gen1Resolved, err := reqGen1.ResolveAttempt(target)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt(gen1): %v", err)
|
||||
}
|
||||
if !resolvedFilterIDs(gen1Resolved)[openAIProviderErrorFilterID] {
|
||||
t.Errorf("gen1 request must keep its enabled provider_error filter")
|
||||
}
|
||||
gen2Resolved, err := reqGen2.ResolveAttempt(target)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt(gen2): %v", err)
|
||||
}
|
||||
if len(gen2Resolved) != 0 {
|
||||
t.Errorf("gen2 request resolved %d filters, want 0 (its own generation)", len(gen2Resolved))
|
||||
}
|
||||
|
||||
// A request context cannot begin against a mismatched generation snapshot.
|
||||
mismatchCtx, err := streamgate.NewRequestFilterContext(
|
||||
"edge.gen.2", "attempt.x", streamGateEnvironment, openAIRebuildEndpointChat,
|
||||
openAIRebuildFamily, "", streamgate.CommitStateTransportUncommitted, false, false, "",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRequestFilterContext(mismatch): %v", err)
|
||||
}
|
||||
if _, err := snapGen1.BeginRequest(mismatchCtx); err == nil {
|
||||
t.Errorf("BeginRequest with a mismatched generation succeeded, want generation isolation error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIStreamGatePolicyTargetMatrix(t *testing.T) {
|
||||
gateCfg := config.StreamEvidenceGateConf{
|
||||
Environment: config.StreamGateEnvironmentDevCorp,
|
||||
Filters: []config.StreamGateFilterPolicyConf{{
|
||||
Filter: config.StreamGateFilterProviderError,
|
||||
Enabled: boolPtr(false),
|
||||
Selectors: []config.StreamGateFilterSelectorConf{
|
||||
{Type: config.StreamGateFilterSelectorEnvironment, Key: config.StreamGateEnvironmentDevCorp, Enabled: boolPtr(true)},
|
||||
{Type: config.StreamGateFilterSelectorModelGroup, Key: "gemma", Enabled: boolPtr(false)},
|
||||
{Type: config.StreamGateFilterSelectorModel, Key: "ornith:35b", Enabled: boolPtr(true)},
|
||||
{Type: config.StreamGateFilterSelectorProvider, Key: "prov-off", Enabled: boolPtr(false)},
|
||||
},
|
||||
}},
|
||||
}
|
||||
fctx := openAIOutputFilterContext{
|
||||
environment: config.StreamGateEnvironmentDevCorp,
|
||||
endpoint: openAIRebuildEndpointChat,
|
||||
modelGroup: "qwen",
|
||||
requestRef: "openai.snap.matrix",
|
||||
}
|
||||
reqSnap := beginOutputFilterRequest(t, gateCfg, fctx)
|
||||
tests := []struct {
|
||||
name, group, model, provider string
|
||||
wantActive bool
|
||||
}{
|
||||
{name: "environment enables base-disabled", group: "qwen", model: "generic", provider: "prov-a", wantActive: true},
|
||||
{name: "model-group disables environment", group: "gemma", model: "generic", provider: "prov-a", wantActive: false},
|
||||
{name: "model overrides model-group", group: "gemma", model: "ornith:35b", provider: "prov-a", wantActive: true},
|
||||
{name: "provider overrides model", group: "gemma", model: "ornith:35b", provider: "prov-off", wantActive: false},
|
||||
}
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
target, err := streamgate.NewAttemptTarget(tc.group, tc.model, tc.provider, string(edgeservice.ProviderPoolPathNormalized), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptTarget: %v", err)
|
||||
}
|
||||
resolved, err := reqSnap.ResolveAttempt(target)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveAttempt: %v", err)
|
||||
}
|
||||
active := resolvedFilterIDs(resolved)[openAIProviderErrorFilterID]
|
||||
if active != tc.wantActive {
|
||||
t.Fatalf("active=%t, want %t", active, tc.wantActive)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenAIStreamGateObserveOnlyDoesNotGateAdmission(t *testing.T) {
|
||||
gateCfg := config.StreamEvidenceGateConf{
|
||||
Environment: config.StreamGateEnvironmentDev,
|
||||
Filters: []config.StreamGateFilterPolicyConf{{
|
||||
Filter: config.StreamGateFilterProviderError,
|
||||
Enforcement: config.StreamGateFilterEnforcementObserveOnly,
|
||||
Selectors: []config.StreamGateFilterSelectorConf{{
|
||||
Type: config.StreamGateFilterSelectorProvider, Key: "prov-block", Enforcement: config.StreamGateFilterEnforcementBlocking,
|
||||
}},
|
||||
}},
|
||||
}
|
||||
fctx := openAIOutputFilterContext{environment: config.StreamGateEnvironmentDev, endpoint: openAIRebuildEndpointChat, modelGroup: "qwen", requestRef: "openai.snap.observe"}
|
||||
observeTarget, _ := streamgate.NewAttemptTarget("qwen", "qwen:latest", "prov-observe", "normalized", nil)
|
||||
required, err := openAIStreamGateRequiredCapabilities(gateCfg, fctx, observeTarget)
|
||||
if err != nil {
|
||||
t.Fatalf("RequiredCapabilities(observe): %v", err)
|
||||
}
|
||||
if len(required) != 0 {
|
||||
t.Fatalf("observe-only required capabilities=%v, want none", required)
|
||||
}
|
||||
if admitted, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{observeTarget}); err != nil || len(admitted) != 1 {
|
||||
t.Fatalf("observe-only admission=(%d,%v), want admitted", len(admitted), err)
|
||||
}
|
||||
blockingTarget, _ := streamgate.NewAttemptTarget("qwen", "qwen:latest", "prov-block", "normalized", nil)
|
||||
if _, err := openAIStreamGateAdmitCandidates(gateCfg, fctx, []streamgate.AttemptTarget{blockingTarget}); !errors.Is(err, errStreamGateRequiredCapabilityUnsupported) {
|
||||
t.Fatalf("blocking selector err=%v, want capability rejection", err)
|
||||
}
|
||||
}
|
||||
|
||||
func beginOutputFilterRequest(t *testing.T, gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext) streamgate.RequestFilterSnapshot {
|
||||
t.Helper()
|
||||
regs, policies, err := openAIOutputFilterRegistrations(gateCfg, fctx)
|
||||
if err != nil {
|
||||
t.Fatalf("openAIOutputFilterRegistrations: %v", err)
|
||||
}
|
||||
snap, err := streamgate.NewFilterRegistrySnapshot(streamGateConfigGeneration, regs, policies)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFilterRegistrySnapshot: %v", err)
|
||||
}
|
||||
reqCtx, err := openAIOutputFilterRequestContext(fctx)
|
||||
if err != nil {
|
||||
t.Fatalf("openAIOutputFilterRequestContext: %v", err)
|
||||
}
|
||||
reqSnap, err := snap.BeginRequest(reqCtx)
|
||||
if err != nil {
|
||||
t.Fatalf("BeginRequest: %v", err)
|
||||
}
|
||||
return reqSnap
|
||||
}
|
||||
|
||||
func beginGenerationRequest(t *testing.T, snap streamgate.FilterRegistrySnapshot, generation string) streamgate.RequestFilterSnapshot {
|
||||
t.Helper()
|
||||
reqCtx, err := streamgate.NewRequestFilterContext(
|
||||
generation, "attempt.1", streamGateEnvironment, openAIRebuildEndpointChat,
|
||||
openAIRebuildFamily, "", streamgate.CommitStateTransportUncommitted, false, false, "",
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRequestFilterContext: %v", err)
|
||||
}
|
||||
reqSnap, err := snap.BeginRequest(reqCtx)
|
||||
if err != nil {
|
||||
t.Fatalf("BeginRequest: %v", err)
|
||||
}
|
||||
return reqSnap
|
||||
}
|
||||
|
||||
func attemptTarget(t *testing.T, provider string) streamgate.AttemptTarget {
|
||||
t.Helper()
|
||||
target, err := streamgate.NewAttemptTarget("client-model", "ornith:35b", provider, "normalized",
|
||||
[]string{"output.repeat_guard", "output.schema_gate", "output.provider_error"})
|
||||
if err != nil {
|
||||
t.Fatalf("NewAttemptTarget(%s): %v", provider, err)
|
||||
}
|
||||
return target
|
||||
}
|
||||
|
|
@ -11,6 +11,23 @@ import (
|
|||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
const openAIStreamGateCandidateRejectedMessage = "no provider supports the required output validation capability"
|
||||
|
||||
type openAIRecoveryAdmissionAwareSink interface {
|
||||
setRecoveryAdmissionState(*openAIRecoveryAdmissionState)
|
||||
}
|
||||
|
||||
// bindOpenAIRecoveryAdmissionState gives a release sink the sanitized result of
|
||||
// recovery re-admission. It never exposes the raw dispatcher error to Core,
|
||||
// observations, or the caller.
|
||||
func bindOpenAIRecoveryAdmissionState(sink streamgate.ReleaseSink, state *openAIRecoveryAdmissionState) {
|
||||
aware, ok := sink.(openAIRecoveryAdmissionAwareSink)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
aware.setRecoveryAdmissionState(state)
|
||||
}
|
||||
|
||||
// openAIStreamGateErrorMessage derives a caller-facing message from a Core
|
||||
// TerminalResult. Core external descriptors carry only sanitized stable
|
||||
// tokens (no raw provider text), so the code is the most specific safe value
|
||||
|
|
@ -26,6 +43,11 @@ func openAIStreamGateErrorMessage(tr streamgate.TerminalResult) string {
|
|||
return desc.Type()
|
||||
}
|
||||
|
||||
func isOpenAIProviderTunnelErrorTerminal(tr streamgate.TerminalResult) bool {
|
||||
desc := tr.ExternalDesc()
|
||||
return desc != nil && desc.Code() == streamGateErrorTunnelFailed
|
||||
}
|
||||
|
||||
// openAIChatSSEReleaseSink implements streamgate.ReleaseSink for the
|
||||
// normalized live-SSE chat completion path. It stages the HTTP status,
|
||||
// SSE headers, and the opening assistant role chunk behind the Core's first
|
||||
|
|
@ -33,11 +55,12 @@ func openAIStreamGateErrorMessage(tr streamgate.TerminalResult) string {
|
|||
// written, and it is only called by CommitBoundary once a release or a
|
||||
// success terminal is ready to commit.
|
||||
type openAIChatSSEReleaseSink struct {
|
||||
w http.ResponseWriter
|
||||
flusher http.Flusher
|
||||
id string
|
||||
created int64
|
||||
model string
|
||||
w http.ResponseWriter
|
||||
flusher http.Flusher
|
||||
id string
|
||||
created int64
|
||||
model string
|
||||
recoveryAdmission *openAIRecoveryAdmissionState
|
||||
|
||||
mu sync.Mutex
|
||||
wroteHeader bool
|
||||
|
|
@ -49,6 +72,12 @@ func newOpenAIChatSSEReleaseSink(w http.ResponseWriter, flusher http.Flusher, id
|
|||
return &openAIChatSSEReleaseSink{w: w, flusher: flusher, id: id, created: created, model: model}
|
||||
}
|
||||
|
||||
func (s *openAIChatSSEReleaseSink) setRecoveryAdmissionState(state *openAIRecoveryAdmissionState) {
|
||||
s.mu.Lock()
|
||||
s.recoveryAdmission = state
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
// terminalStatus reports whether the Core committed a terminal through this
|
||||
// sink and, if so, whether it was a success terminal. The host runner uses it
|
||||
// to derive the terminal usage-metric status as the single source of truth for
|
||||
|
|
@ -149,6 +178,11 @@ func (s *openAIChatSSEReleaseSink) CommitTerminal(ctx context.Context, tr stream
|
|||
}
|
||||
|
||||
message := openAIStreamGateErrorMessage(tr)
|
||||
if !s.wroteHeader && s.recoveryAdmission.rejected() {
|
||||
writeError(s.w, http.StatusBadRequest, "invalid_request_error", openAIStreamGateCandidateRejectedMessage)
|
||||
s.wroteHeader = true
|
||||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
}
|
||||
if !s.wroteHeader {
|
||||
writeError(s.w, http.StatusBadGateway, "run_error", message)
|
||||
s.wroteHeader = true
|
||||
|
|
@ -174,8 +208,10 @@ type openAITunnelReleaseSink struct {
|
|||
// and the caller-facing model echo rewrite runs exactly once at terminal.
|
||||
// A streaming attempt rewrites in provider byte order inside the event
|
||||
// source instead and writes each release straight through.
|
||||
buffered bool
|
||||
rewriter *providerModelRewriter
|
||||
buffered bool
|
||||
rewriter *providerModelRewriter
|
||||
codec *openAITunnelCodecState
|
||||
recoveryAdmission *openAIRecoveryAdmissionState
|
||||
|
||||
mu sync.Mutex
|
||||
wroteHeader bool
|
||||
|
|
@ -185,7 +221,27 @@ type openAITunnelReleaseSink struct {
|
|||
}
|
||||
|
||||
func newOpenAITunnelReleaseSink(w http.ResponseWriter, flusher http.Flusher) *openAITunnelReleaseSink {
|
||||
return &openAITunnelReleaseSink{w: w, flusher: flusher}
|
||||
return &openAITunnelReleaseSink{w: w, flusher: flusher, codec: &openAITunnelCodecState{}}
|
||||
}
|
||||
|
||||
func (s *openAITunnelReleaseSink) setRecoveryAdmissionState(state *openAIRecoveryAdmissionState) {
|
||||
s.mu.Lock()
|
||||
s.recoveryAdmission = state
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func openAITunnelCodecStateForSink(sink streamgate.ReleaseSink) *openAITunnelCodecState {
|
||||
switch typed := sink.(type) {
|
||||
case *openAITunnelReleaseSink:
|
||||
if typed.codec == nil {
|
||||
typed.codec = &openAITunnelCodecState{}
|
||||
}
|
||||
return typed.codec
|
||||
case *openAICompositeReleaseSink:
|
||||
return openAITunnelCodecStateForSink(typed.tunnel)
|
||||
default:
|
||||
return &openAITunnelCodecState{}
|
||||
}
|
||||
}
|
||||
|
||||
// newOpenAIBufferedTunnelReleaseSink builds the non-streaming passthrough sink.
|
||||
|
|
@ -232,20 +288,31 @@ func (s *openAITunnelReleaseSink) CommitResponseStart(ctx context.Context, rs st
|
|||
func (s *openAITunnelReleaseSink) Release(ctx context.Context, ev streamgate.ReleaseEvent) (streamgate.CommitState, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if ev.Kind() != streamgate.EventKindTextDelta {
|
||||
return streamgate.CommitStateStreamOpen, fmt.Errorf("openai stream gate: tunnel sink does not support release event kind %q", ev.Kind())
|
||||
}
|
||||
// Raw provider bytes travel through Core as an opaque text_delta payload;
|
||||
// Go strings are byte-exact so the round trip is lossless pure passthrough.
|
||||
text, err := ev.AsTextDelta()
|
||||
if err != nil {
|
||||
return streamgate.CommitStateStreamOpen, err
|
||||
var payload []byte
|
||||
if wire, ok := s.codec.popRelease(); ok {
|
||||
payload = wire
|
||||
} else {
|
||||
switch ev.Kind() {
|
||||
case streamgate.EventKindTextDelta:
|
||||
text, err := ev.AsTextDelta()
|
||||
if err != nil {
|
||||
return streamgate.CommitStateStreamOpen, err
|
||||
}
|
||||
payload = []byte(text)
|
||||
case streamgate.EventKindReasoningDelta, streamgate.EventKindToolCallFragment:
|
||||
return streamgate.CommitStateStreamOpen, fmt.Errorf("openai stream gate: tunnel codec lost wire payload for %q", ev.Kind())
|
||||
default:
|
||||
return streamgate.CommitStateStreamOpen, fmt.Errorf("openai stream gate: tunnel sink does not support release event kind %q", ev.Kind())
|
||||
}
|
||||
}
|
||||
if s.buffered {
|
||||
s.body = append(s.body, text...)
|
||||
s.body = append(s.body, payload...)
|
||||
return streamgate.CommitStateStreamOpen, nil
|
||||
}
|
||||
if _, err := s.w.Write([]byte(text)); err != nil {
|
||||
if len(payload) == 0 {
|
||||
return streamgate.CommitStateStreamOpen, nil
|
||||
}
|
||||
if _, err := s.w.Write(payload); err != nil {
|
||||
return streamgate.CommitStateStreamOpen, err
|
||||
}
|
||||
if s.flusher != nil {
|
||||
|
|
@ -259,6 +326,14 @@ func (s *openAITunnelReleaseSink) CommitTerminal(ctx context.Context, tr streamg
|
|||
defer s.mu.Unlock()
|
||||
s.terminalCommitted = true
|
||||
s.terminalSuccess = tr.Success()
|
||||
if payload, ok := s.codec.popTerminal(); ok && len(payload) > 0 && s.wroteHeader {
|
||||
if _, err := s.w.Write(payload); err != nil {
|
||||
return streamgate.CommitStateTerminalCommitted, err
|
||||
}
|
||||
if s.flusher != nil {
|
||||
s.flusher.Flush()
|
||||
}
|
||||
}
|
||||
if tr.Success() {
|
||||
if s.buffered {
|
||||
body := s.body
|
||||
|
|
@ -278,6 +353,33 @@ func (s *openAITunnelReleaseSink) CommitTerminal(ctx context.Context, tr streamg
|
|||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
}
|
||||
s.body = nil
|
||||
if !s.wroteHeader && s.recoveryAdmission.rejected() {
|
||||
writeError(s.w, http.StatusBadRequest, "invalid_request_error", openAIStreamGateCandidateRejectedMessage)
|
||||
s.wroteHeader = true
|
||||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
}
|
||||
if !s.wroteHeader && isOpenAIProviderTunnelErrorTerminal(tr) {
|
||||
if response, ok := s.codec.popErrorResponse(); ok {
|
||||
for key, value := range response.headers {
|
||||
s.w.Header().Set(key, value)
|
||||
}
|
||||
status := response.status
|
||||
if status == 0 {
|
||||
status = http.StatusBadGateway
|
||||
}
|
||||
s.w.WriteHeader(status)
|
||||
s.wroteHeader = true
|
||||
if len(response.body) > 0 {
|
||||
if _, err := s.w.Write(response.body); err != nil {
|
||||
return streamgate.CommitStateTerminalCommitted, err
|
||||
}
|
||||
}
|
||||
if s.flusher != nil {
|
||||
s.flusher.Flush()
|
||||
}
|
||||
return streamgate.CommitStateTerminalCommitted, nil
|
||||
}
|
||||
}
|
||||
if !s.wroteHeader {
|
||||
writeError(s.w, http.StatusBadGateway, "provider_tunnel_error", openAIStreamGateErrorMessage(tr))
|
||||
s.wroteHeader = true
|
||||
|
|
@ -359,6 +461,11 @@ type openAICompositeReleaseSink struct {
|
|||
frozen openAIStreamGateCodec
|
||||
}
|
||||
|
||||
func (s *openAICompositeReleaseSink) setRecoveryAdmissionState(state *openAIRecoveryAdmissionState) {
|
||||
bindOpenAIRecoveryAdmissionState(s.normalized, state)
|
||||
bindOpenAIRecoveryAdmissionState(s.tunnel, state)
|
||||
}
|
||||
|
||||
func newOpenAICompositeReleaseSink(selector *openAIStreamGateCodecSelector, normalized, tunnel openAIStreamGateSink) *openAICompositeReleaseSink {
|
||||
return &openAICompositeReleaseSink{selector: selector, normalized: normalized, tunnel: tunnel}
|
||||
}
|
||||
|
|
@ -439,8 +546,9 @@ type openAIBufferedChatReleaseSink struct {
|
|||
traceStream bool
|
||||
// stream selects the buffered SSE framing; false renders the non-stream
|
||||
// chat.completion JSON object.
|
||||
stream bool
|
||||
holder *openAIBufferedResultHolder
|
||||
stream bool
|
||||
holder *openAIBufferedResultHolder
|
||||
recoveryAdmission *openAIRecoveryAdmissionState
|
||||
|
||||
mu sync.Mutex
|
||||
wroteHeader bool
|
||||
|
|
@ -468,6 +576,12 @@ func newOpenAIBufferedChatReleaseSink(
|
|||
}
|
||||
}
|
||||
|
||||
func (s *openAIBufferedChatReleaseSink) setRecoveryAdmissionState(state *openAIRecoveryAdmissionState) {
|
||||
s.mu.Lock()
|
||||
s.recoveryAdmission = state
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAIBufferedChatReleaseSink) terminalStatus() (bool, bool) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
|
@ -535,6 +649,12 @@ func (s *openAIBufferedChatReleaseSink) renderSuccessLocked(result openAIBuffere
|
|||
// tool_validation_error with the last validation reason, and any other terminal
|
||||
// error keeps the run_error envelope.
|
||||
func (s *openAIBufferedChatReleaseSink) renderErrorLocked(tr streamgate.TerminalResult, result openAIBufferedAttemptResult, ok bool) {
|
||||
if !s.wroteHeader && s.recoveryAdmission.rejected() {
|
||||
writeError(s.w, http.StatusBadRequest, "invalid_request_error", openAIStreamGateCandidateRejectedMessage)
|
||||
s.wroteHeader = true
|
||||
return
|
||||
}
|
||||
|
||||
errType := "run_error"
|
||||
message := openAIStreamGateErrorMessage(tr)
|
||||
switch {
|
||||
|
|
|
|||
|
|
@ -13,13 +13,14 @@ import (
|
|||
|
||||
"go.uber.org/zap"
|
||||
edgeservice "iop/apps/edge/internal/service"
|
||||
"iop/packages/go/config"
|
||||
"iop/packages/go/streamgate"
|
||||
iop "iop/proto/gen/iop"
|
||||
)
|
||||
|
||||
const (
|
||||
streamGateChannelDefault = "default"
|
||||
streamGateEnvironment = "edge"
|
||||
streamGateEnvironment = config.StreamGateEnvironmentDev
|
||||
streamGateConfigGeneration = "edge.vertical-slice.1"
|
||||
streamGateNoopCapability = "core.always"
|
||||
streamGateFilterTimeout = 5 * time.Second
|
||||
|
|
@ -302,17 +303,17 @@ func sanitizedTunnelResponseHeaders(headers map[string]string) map[string]string
|
|||
return out
|
||||
}
|
||||
|
||||
// openAITunnelEventSource adapts a ProviderTunnelStream to
|
||||
// streamgate.NormalizedEventSource. Raw provider body bytes travel through
|
||||
// Core as opaque text_delta payloads (Go string<->[]byte round trips are
|
||||
// byte-exact) so pure passthrough is preserved end-to-end. The caller-facing
|
||||
// model echo rewrite runs before bytes enter Core so the transform is
|
||||
// evaluated exactly once, in original provider byte order.
|
||||
// openAITunnelEventSource adapts a ProviderTunnelStream to endpoint-semantic
|
||||
// Core events. The endpoint codec retains caller-facing provider frames in a
|
||||
// request-local release queue so Chat/Responses parsing and lossless passthrough
|
||||
// coexist. The model echo rewrite still runs exactly once in provider order.
|
||||
type openAITunnelEventSource struct {
|
||||
frames <-chan *iop.ProviderTunnelFrame
|
||||
waitTimeout time.Duration
|
||||
rewriter *providerModelRewriter
|
||||
assembler *providerChatAssembler
|
||||
frames <-chan *iop.ProviderTunnelFrame
|
||||
waitTimeout time.Duration
|
||||
rewriter *providerModelRewriter
|
||||
assembler *providerChatAssembler
|
||||
codec *openAITunnelEndpointCodec
|
||||
responseStatus int
|
||||
|
||||
mu sync.Mutex
|
||||
started bool
|
||||
|
|
@ -323,6 +324,12 @@ func newOpenAITunnelEventSource(stream edgeservice.ProviderTunnelStream, waitTim
|
|||
return &openAITunnelEventSource{frames: stream.Frames, waitTimeout: waitTimeout, rewriter: rewriter, assembler: assembler}
|
||||
}
|
||||
|
||||
func newOpenAITunnelEndpointEventSource(stream edgeservice.ProviderTunnelStream, waitTimeout time.Duration, rewriter *providerModelRewriter, assembler *providerChatAssembler, endpoint string, state *openAITunnelCodecState) *openAITunnelEventSource {
|
||||
source := newOpenAITunnelEventSource(stream, waitTimeout, rewriter, assembler)
|
||||
source.codec = newOpenAITunnelEndpointCodec(endpoint, state)
|
||||
return source
|
||||
}
|
||||
|
||||
func (s *openAITunnelEventSource) NextEvent(ctx context.Context) (streamgate.NormalizedEvent, error) {
|
||||
s.mu.Lock()
|
||||
if len(s.pending) > 0 {
|
||||
|
|
@ -387,7 +394,12 @@ func (s *openAITunnelEventSource) translateFrame(frame *iop.ProviderTunnelFrame)
|
|||
if status == 0 {
|
||||
status = http.StatusOK
|
||||
}
|
||||
ev, err := streamgate.NewResponseStartEvent(streamGateChannelDefault, status, sanitizedTunnelResponseHeaders(frame.GetHeaders()), time.Now())
|
||||
s.responseStatus = status
|
||||
headers := sanitizedTunnelResponseHeaders(frame.GetHeaders())
|
||||
if s.codec != nil && status >= http.StatusBadRequest {
|
||||
s.codec.state.stageErrorResponseStart(status, headers)
|
||||
}
|
||||
ev, err := streamgate.NewResponseStartEvent(streamGateChannelDefault, status, headers, time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -398,6 +410,13 @@ func (s *openAITunnelEventSource) translateFrame(frame *iop.ProviderTunnelFrame)
|
|||
if len(body) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
if s.codec != nil && s.responseStatus >= http.StatusBadRequest {
|
||||
// A non-2xx body is opaque provider wire even when it resembles a
|
||||
// successful Chat/Responses payload. It is committed only if this
|
||||
// attempt's provider-error terminal wins final arbitration.
|
||||
s.codec.state.appendErrorResponseWire(body)
|
||||
return nil, nil
|
||||
}
|
||||
if s.assembler != nil {
|
||||
s.assembler.Write(body)
|
||||
}
|
||||
|
|
@ -412,10 +431,18 @@ func (s *openAITunnelEventSource) translateFrame(frame *iop.ProviderTunnelFrame)
|
|||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
s.responseStatus = http.StatusOK
|
||||
}
|
||||
if len(rewritten) == 0 {
|
||||
return events, nil
|
||||
}
|
||||
if s.codec != nil {
|
||||
decoded, err := s.codec.decode(rewritten, false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return append(events, decoded...), nil
|
||||
}
|
||||
ev, err := streamgate.NewTextDeltaEvent(streamGateChannelDefault, string(rewritten), time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -443,15 +470,27 @@ func (s *openAITunnelEventSource) translateFrame(frame *iop.ProviderTunnelFrame)
|
|||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
s.responseStatus = http.StatusOK
|
||||
}
|
||||
var flushed []byte
|
||||
if s.rewriter != nil {
|
||||
if flushed := s.rewriter.FlushStream(); len(flushed) > 0 {
|
||||
ev, err := streamgate.NewTextDeltaEvent(streamGateChannelDefault, string(flushed), time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
flushed = s.rewriter.FlushStream()
|
||||
}
|
||||
if s.codec != nil {
|
||||
decoded, err := s.codec.finishTransport(flushed, s.responseStatus >= http.StatusBadRequest)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, decoded...)
|
||||
if len(decoded) > 0 {
|
||||
return events, nil
|
||||
}
|
||||
} else if len(flushed) > 0 {
|
||||
ev, err := streamgate.NewTextDeltaEvent(streamGateChannelDefault, string(flushed), time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
}
|
||||
term, err := streamgate.NewTerminalEvent(streamGateChannelDefault, time.Now())
|
||||
if err != nil {
|
||||
|
|
@ -488,6 +527,117 @@ func openAIStreamGateRegistrySnapshot() (streamgate.FilterRegistrySnapshot, erro
|
|||
return openAIStreamGateRegistrySnapshotWith()
|
||||
}
|
||||
|
||||
// openAIStreamGateRegistrySnapshotFor builds the production registry snapshot for
|
||||
// one request: the always-applicable Noop mechanics filter, the configured
|
||||
// semantic output filters (repeat/schema/provider-error) translated from the
|
||||
// stream_evidence_gate policy, plus any request-local extra registrations (e.g.
|
||||
// the tool-validation terminal gate). An empty Filters policy reduces to the
|
||||
// legacy Noop+extra set exactly, so the default production behavior is unchanged.
|
||||
func openAIStreamGateRegistrySnapshotFor(gateCfg config.StreamEvidenceGateConf, fctx openAIOutputFilterContext, extra ...streamgate.FilterRegistration) (streamgate.FilterRegistrySnapshot, error) {
|
||||
regs, err := openAIStreamGateNoopRegistrations()
|
||||
if err != nil {
|
||||
return streamgate.FilterRegistrySnapshot{}, err
|
||||
}
|
||||
outputRegs, policies, err := openAIOutputFilterRegistrations(gateCfg, fctx)
|
||||
if err != nil {
|
||||
return streamgate.FilterRegistrySnapshot{}, err
|
||||
}
|
||||
regs = append(regs, outputRegs...)
|
||||
regs = append(regs, extra...)
|
||||
return streamgate.NewFilterRegistrySnapshot(streamGateConfigGeneration, regs, policies)
|
||||
}
|
||||
|
||||
// streamGateConfig returns a copy of the request-stable stream-gate config the
|
||||
// request runtime pins at request start (generation isolation).
|
||||
func (s *Server) streamGateConfig() config.StreamEvidenceGateConf {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.cfg.StreamEvidenceGate
|
||||
}
|
||||
|
||||
// openAIChatOutputFilterContext resolves the immutable request-start selector,
|
||||
// model-group, endpoint, and schema facts for a chat request.
|
||||
func (s *Server) openAIChatOutputFilterContext(dc *chatDispatchContext) (openAIOutputFilterContext, error) {
|
||||
snapRef, err := dc.ingress.recoveryRef()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
body, err := dc.ingress.canonicalBody()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
history, err := decodeOpenAIChatRepeatHistory(body)
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
return openAIOutputFilterContext{
|
||||
environment: s.streamGateConfig().EffectiveEnvironment(),
|
||||
endpoint: openAIRebuildEndpointChat,
|
||||
modelGroup: strings.TrimSpace(dc.req.Model),
|
||||
hasScheme: chatRequestHasSchemeMetadata(dc.req.Metadata),
|
||||
requestRef: snapRef.SnapshotRef(),
|
||||
history: history,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// openAIResponsesOutputFilterContext resolves the same request-local policy
|
||||
// facts for a Responses request. The endpoint-specific lossless Rebuilder is
|
||||
// still the canonical snapshot owner; only the semantic filter selection is
|
||||
// shared with Chat.
|
||||
func (s *Server) openAIResponsesOutputFilterContext(requestCtx *responsesRequestContext) (openAIOutputFilterContext, error) {
|
||||
snapRef, err := requestCtx.ingress.recoveryRef()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
body, err := requestCtx.ingress.canonicalBody()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
history, err := decodeOpenAIResponsesRepeatHistory(body)
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
return openAIOutputFilterContext{
|
||||
environment: s.streamGateConfig().EffectiveEnvironment(),
|
||||
endpoint: openAIRebuildEndpointResponses,
|
||||
modelGroup: strings.TrimSpace(requestCtx.envelope.Model),
|
||||
hasScheme: chatRequestHasSchemeMetadata(requestCtx.envelope.Metadata),
|
||||
requestRef: snapRef.SnapshotRef(),
|
||||
history: history,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// openAITunnelOutputFilterContext preserves the same selector, model-group,
|
||||
// endpoint, and schema facts when the actual execution path is a tunnel.
|
||||
func (s *Server) openAITunnelOutputFilterContext(req openAITunnelStreamGateRequest) (openAIOutputFilterContext, error) {
|
||||
snapRef, err := req.ingress.recoveryRef()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
body, err := req.ingress.canonicalBody()
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
var history openAIRepeatHistorySnapshot
|
||||
switch req.endpoint {
|
||||
case openAIRebuildEndpointResponses:
|
||||
history, err = decodeOpenAIResponsesRepeatHistory(body)
|
||||
default:
|
||||
history, err = decodeOpenAIChatRepeatHistory(body)
|
||||
}
|
||||
if err != nil {
|
||||
return openAIOutputFilterContext{}, err
|
||||
}
|
||||
return openAIOutputFilterContext{
|
||||
environment: s.streamGateConfig().EffectiveEnvironment(),
|
||||
modelGroup: req.modelGroupKey,
|
||||
endpoint: req.endpoint,
|
||||
hasScheme: req.hasScheme,
|
||||
requestRef: snapRef.SnapshotRef(),
|
||||
history: history,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// openAIStreamGateRegistrySnapshotWith builds the production registry snapshot
|
||||
// plus request-local registrations. The only production request-local filter is
|
||||
// the tool-validation terminal gate, which needs this request's result holder;
|
||||
|
|
@ -650,10 +800,11 @@ func newOpenAIChatRecoveryAdmissionBuilder(s *Server, dc *chatDispatchContext, h
|
|||
return openAIAttemptAdmission{
|
||||
kind: openAIAdmissionPool,
|
||||
pool: edgeservice.ProviderPoolDispatchRequest{
|
||||
Run: runReq,
|
||||
Tunnel: tunnelReq,
|
||||
PrepareTunnel: poolTemplate.PrepareTunnel,
|
||||
PrepareRun: poolTemplate.PrepareRun,
|
||||
Run: runReq,
|
||||
Tunnel: tunnelReq,
|
||||
PrepareTunnel: poolTemplate.PrepareTunnel,
|
||||
PrepareRun: poolTemplate.PrepareRun,
|
||||
AcceptCandidate: poolTemplate.AcceptCandidate,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
|
@ -684,6 +835,10 @@ type openAIChatStreamGateConfig struct {
|
|||
selector *openAIStreamGateCodecSelector
|
||||
registry streamgate.FilterRegistrySnapshot
|
||||
holder *openAIBufferedResultHolder
|
||||
// recoverySource is the request-local model-output recorder shared by every
|
||||
// attempt's event source, so a repeat-loop continuation plan can be assembled
|
||||
// from the aborted attempt's own content/reasoning.
|
||||
recoverySource *openAIRecoverySourceStore
|
||||
// preparer and prepFactory are the optional one-shot host preparation seam
|
||||
// Core calls after attempt ownership is closed and before the rebuild. The
|
||||
// OpenAI surfaces have no production preparer in this slice, so both stay
|
||||
|
|
@ -704,6 +859,7 @@ func (s *Server) newOpenAIChatAttemptEventSourceFactory(
|
|||
usage *openAIStreamGateUsageHolder,
|
||||
) openAIAttemptEventSourceFactory {
|
||||
return func(transport openAIAttemptTransport) (streamgate.NormalizedEventSource, error) {
|
||||
var src streamgate.NormalizedEventSource
|
||||
switch transport.path {
|
||||
case openAIAdmissionRun:
|
||||
if transport.run == nil {
|
||||
|
|
@ -711,9 +867,10 @@ func (s *Server) newOpenAIChatAttemptEventSourceFactory(
|
|||
}
|
||||
cfg.selector.set(openAIStreamGateCodecNormalized)
|
||||
if cfg.mode == openAIChatGateModeBuffered {
|
||||
return newOpenAIBufferedChatEventSource(dc, transport.run, cfg.holder, usage), nil
|
||||
src = newOpenAIBufferedChatEventSource(dc, transport.run, cfg.holder, usage)
|
||||
} else {
|
||||
src = newOpenAIRunEventSource(transport.run.Stream(), transport.run.WaitTimeout(), usage)
|
||||
}
|
||||
return newOpenAIRunEventSource(transport.run.Stream(), transport.run.WaitTimeout(), usage), nil
|
||||
case openAIAdmissionTunnel:
|
||||
if transport.tunnel == nil {
|
||||
return nil, fmt.Errorf("openai stream gate: chat tunnel attempt is missing its tunnel transport")
|
||||
|
|
@ -723,11 +880,14 @@ func (s *Server) newOpenAIChatAttemptEventSourceFactory(
|
|||
// partial rewrite or usage state never bleeds into its replacement.
|
||||
assembler := &providerChatAssembler{streaming: dc.req.Stream}
|
||||
rewriter := newProviderModelRewriter(dc.req.Stream, dc.req.Model)
|
||||
src := newOpenAITunnelEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler)
|
||||
return &openAIStreamGateUsageTrackingTunnelSource{openAITunnelEventSource: src, usage: usage}, nil
|
||||
state := openAITunnelCodecStateForSink(cfg.sink)
|
||||
state.reset()
|
||||
tunnelSrc := newOpenAITunnelEndpointEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler, openAIRebuildEndpointChat, state)
|
||||
src = &openAIStreamGateUsageTrackingTunnelSource{openAITunnelEventSource: tunnelSrc, usage: usage}
|
||||
default:
|
||||
return nil, fmt.Errorf("openai stream gate: unsupported attempt transport path %q for chat completions", transport.path)
|
||||
}
|
||||
return newOpenAIRecoverySourceEventSource(src, cfg.recoverySource), nil
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -735,10 +895,26 @@ func (s *Server) newOpenAIChatAttemptEventSourceFactory(
|
|||
// for a runtime-enabled chat completion. The initial attempt binding wraps the
|
||||
// already-dispatched transport directly (no re-admission); the
|
||||
// dispatcher/rebuilder pair serves every subsequent recovery attempt.
|
||||
// openAIResumeContextWindowTokens returns the target model group's single-request
|
||||
// context-window contract from the provider-pool catalog, or 0 when the model is
|
||||
// unknown. The resume builder treats 0 as fail-closed.
|
||||
func (s *Server) openAIResumeContextWindowTokens(model string) int {
|
||||
if entry := s.findProviderPoolEntry(model); entry != nil {
|
||||
return entry.ContextWindowTokens
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (s *Server) buildOpenAIChatStreamGateRuntimeFor(dc *chatDispatchContext, cfg openAIChatStreamGateConfig) (*streamgate.RequestRuntime, *openAIStreamGateUsageHolder, error) {
|
||||
usage := &openAIStreamGateUsageHolder{}
|
||||
|
||||
rebuilder, err := newOpenAIRequestRebuilder(dc.ingress, openAIRebuildEndpointChat)
|
||||
// One request-local recorder is shared by the initial and every recovery
|
||||
// event source so a repeat-loop continuation is assembled from the aborted
|
||||
// attempt's own output. The rebuilder measures the resume body against the
|
||||
// target model context window and fails closed before dispatch on overflow.
|
||||
recoverySource := newOpenAIRecoverySourceStore(dc.ingress)
|
||||
cfg.recoverySource = recoverySource
|
||||
rebuilder, err := newOpenAIRequestRebuilder(dc.ingress, openAIRebuildEndpointChat, recoverySource, s.openAIResumeContextWindowTokens(dc.req.Model))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
@ -747,6 +923,7 @@ func (s *Server) buildOpenAIChatStreamGateRuntimeFor(dc *chatDispatchContext, cf
|
|||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
bindOpenAIRecoveryAdmissionState(cfg.sink, dispatcher.admissionState())
|
||||
|
||||
opts, err := s.streamGateRuntimeOptions()
|
||||
if err != nil {
|
||||
|
|
@ -777,7 +954,7 @@ func (s *Server) buildOpenAIChatStreamGateRuntimeFor(dc *chatDispatchContext, cf
|
|||
|
||||
requestID := openAIStreamGateSafeToken("req", dispatch.RunID)
|
||||
snapshot, err := streamgate.NewRequestRuntimeSnapshot(
|
||||
requestID, streamGateConfigGeneration, streamGateEnvironment, openAIRebuildEndpointChat, openAIRebuildFamily,
|
||||
requestID, streamGateConfigGeneration, s.streamGateConfig().EffectiveEnvironment(), openAIRebuildEndpointChat, openAIRebuildFamily,
|
||||
opts, cfg.registry, nil, snapRef, dispatcher, rebuilder, cfg.preparer, cfg.prepFactory, cfg.sink,
|
||||
)
|
||||
if err != nil {
|
||||
|
|
@ -886,7 +1063,15 @@ func (s *Server) runOpenAIChatStreamGate(w http.ResponseWriter, flusher http.Flu
|
|||
normalized := newOpenAIChatSSEReleaseSink(w, flusher, "chatcmpl-"+dispatch.RunID, time.Now().Unix(), responseModel(dc.req.Model, dispatch.Target))
|
||||
sink := s.openAIChatCompositeSink(w, flusher, dc, selector, normalized)
|
||||
|
||||
registry, err := openAIStreamGateRegistrySnapshot()
|
||||
fctx, err := s.openAIChatOutputFilterContext(dc)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate chat output filter context build failed", zap.Error(err))
|
||||
writeSSEErrorWithType(w, flusher, "run_error", "stream gate runtime unavailable")
|
||||
emitUsageMetrics(dc.usageLabels(s, responseModeNormalized), usageStatusError, usageObservation{})
|
||||
return
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate chat registry build failed", zap.Error(err))
|
||||
|
|
@ -977,7 +1162,11 @@ func (s *Server) openAIChatStreamGateRegistry(dc *chatDispatchContext, holder *o
|
|||
return streamgate.FilterRegistrySnapshot{}, err
|
||||
}
|
||||
extra = append(extra, extraFilters...)
|
||||
return openAIStreamGateRegistrySnapshotWith(extra...)
|
||||
fctx, err := s.openAIChatOutputFilterContext(dc)
|
||||
if err != nil {
|
||||
return streamgate.FilterRegistrySnapshot{}, err
|
||||
}
|
||||
return openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx, extra...)
|
||||
}
|
||||
|
||||
// runOpenAIChatPoolStreamGate drives a provider-pool chat completion through the
|
||||
|
|
@ -1093,8 +1282,10 @@ type openAITunnelStreamGateRequest struct {
|
|||
endpoint string // openAIRebuildEndpointChat or openAIRebuildEndpointResponses
|
||||
method string
|
||||
path string
|
||||
stream bool
|
||||
modelGroupKey string
|
||||
metadata map[string]string
|
||||
hasScheme bool
|
||||
estimate int
|
||||
contextClass string
|
||||
requestModel string // caller-facing model alias for echo rewrite; "" disables rewrite
|
||||
|
|
@ -1118,10 +1309,11 @@ func newOpenAITunnelRecoveryAdmissionBuilder(req openAITunnelStreamGateRequest)
|
|||
return openAIAttemptAdmission{
|
||||
kind: openAIAdmissionPool,
|
||||
pool: edgeservice.ProviderPoolDispatchRequest{
|
||||
Run: req.pool.Run,
|
||||
Tunnel: tunnelReq,
|
||||
PrepareTunnel: req.pool.PrepareTunnel,
|
||||
PrepareRun: req.pool.PrepareRun,
|
||||
Run: req.pool.Run,
|
||||
Tunnel: tunnelReq,
|
||||
PrepareTunnel: req.pool.PrepareTunnel,
|
||||
PrepareRun: req.pool.PrepareRun,
|
||||
AcceptCandidate: req.pool.AcceptCandidate,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
|
@ -1133,7 +1325,7 @@ func newOpenAITunnelRecoveryAdmissionBuilder(req openAITunnelStreamGateRequest)
|
|||
SessionID: req.route.SessionID,
|
||||
Method: req.method,
|
||||
Path: req.path,
|
||||
Stream: true,
|
||||
Stream: req.stream,
|
||||
TimeoutSec: req.route.TimeoutSec,
|
||||
MaxQueue: req.route.MaxQueue,
|
||||
QueueTimeoutMS: req.route.QueueTimeoutMS,
|
||||
|
|
@ -1160,7 +1352,8 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
) (*streamgate.RequestRuntime, *openAIStreamGateUsageHolder, error) {
|
||||
usage := &openAIStreamGateUsageHolder{}
|
||||
|
||||
rebuilder, err := newOpenAIRequestRebuilder(req.ingress, req.endpoint)
|
||||
recoverySource := newOpenAIRecoverySourceStore(req.ingress)
|
||||
rebuilder, err := newOpenAIRequestRebuilder(req.ingress, req.endpoint, recoverySource, s.openAIResumeContextWindowTokens(req.modelGroupKey))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
@ -1174,15 +1367,19 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
if transport.path != openAIAdmissionTunnel || transport.tunnel == nil {
|
||||
return nil, fmt.Errorf("openai stream gate: unsupported recovery transport path %q for provider tunnel", transport.path)
|
||||
}
|
||||
assembler := &providerChatAssembler{streaming: true}
|
||||
rewriter := newProviderModelRewriter(true, req.requestModel)
|
||||
src := newOpenAITunnelEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler)
|
||||
return &openAIStreamGateUsageTrackingTunnelSource{openAITunnelEventSource: src, usage: usage}, nil
|
||||
assembler := &providerChatAssembler{streaming: req.stream}
|
||||
rewriter := newProviderModelRewriter(req.stream, req.requestModel)
|
||||
state := openAITunnelCodecStateForSink(sink)
|
||||
state.reset()
|
||||
src := newOpenAITunnelEndpointEventSource(transport.tunnel.Stream(), transport.tunnel.WaitTimeout(), rewriter, assembler, req.endpoint, state)
|
||||
tracking := &openAIStreamGateUsageTrackingTunnelSource{openAITunnelEventSource: src, usage: usage}
|
||||
return newOpenAIRecoverySourceEventSource(tracking, recoverySource), nil
|
||||
}
|
||||
dispatcher, err := newOpenAIAttemptDispatcher(s.service, rebuilder.RebuiltStore(), build, eventSourceFactory)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
bindOpenAIRecoveryAdmissionState(sink, dispatcher.admissionState())
|
||||
|
||||
opts, err := s.streamGateRuntimeOptions()
|
||||
if err != nil {
|
||||
|
|
@ -1194,10 +1391,12 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
}
|
||||
|
||||
dispatch := handle.Dispatch()
|
||||
initialAssembler := &providerChatAssembler{streaming: true}
|
||||
initialRewriter := newProviderModelRewriter(true, req.requestModel)
|
||||
initialAssembler := &providerChatAssembler{streaming: req.stream}
|
||||
initialRewriter := newProviderModelRewriter(req.stream, req.requestModel)
|
||||
initialState := openAITunnelCodecStateForSink(sink)
|
||||
initialState.reset()
|
||||
initialSource := &openAIStreamGateUsageTrackingTunnelSource{
|
||||
openAITunnelEventSource: newOpenAITunnelEventSource(handle.Stream(), handle.WaitTimeout(), initialRewriter, initialAssembler),
|
||||
openAITunnelEventSource: newOpenAITunnelEndpointEventSource(handle.Stream(), handle.WaitTimeout(), initialRewriter, initialAssembler, req.endpoint, initialState),
|
||||
usage: usage,
|
||||
}
|
||||
initialController := &openAIAttemptController{service: s.service, dispatch: dispatch, closeTransport: handle.Close}
|
||||
|
|
@ -1206,7 +1405,7 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
actualOpenAIModel(dispatch),
|
||||
actualOpenAIProvider(dispatch),
|
||||
actualOpenAIExecutionPath(dispatch, openAIAdmissionTunnel),
|
||||
initialSource,
|
||||
newOpenAIRecoverySourceEventSource(initialSource, recoverySource),
|
||||
initialController,
|
||||
)
|
||||
if err != nil {
|
||||
|
|
@ -1215,7 +1414,7 @@ func (s *Server) buildOpenAITunnelStreamGateRuntime(
|
|||
|
||||
requestID := openAIStreamGateSafeToken("req", dispatch.RunID)
|
||||
snapshot, err := streamgate.NewRequestRuntimeSnapshot(
|
||||
requestID, streamGateConfigGeneration, streamGateEnvironment, req.endpoint, openAIRebuildFamily,
|
||||
requestID, streamGateConfigGeneration, s.streamGateConfig().EffectiveEnvironment(), req.endpoint, openAIRebuildFamily,
|
||||
opts, registry, nil, snapRef, dispatcher, rebuilder, nil, nil, sink,
|
||||
)
|
||||
if err != nil {
|
||||
|
|
@ -1254,13 +1453,26 @@ func (s *openAIStreamGateUsageTrackingTunnelSource) NextEvent(ctx context.Contex
|
|||
|
||||
var _ streamgate.NormalizedEventSource = (*openAIStreamGateUsageTrackingTunnelSource)(nil)
|
||||
|
||||
// runOpenAITunnelStreamGate drives streaming provider tunnel passthrough
|
||||
// through the Core request runtime.
|
||||
// runOpenAITunnelStreamGate drives streaming or buffered provider tunnel
|
||||
// passthrough through the Core request runtime.
|
||||
func (s *Server) runOpenAITunnelStreamGate(w http.ResponseWriter, r *http.Request, req openAITunnelStreamGateRequest, handle edgeservice.ProviderTunnelResult, metricLabels usageLabels) {
|
||||
flusher, _ := w.(http.Flusher)
|
||||
sink := newOpenAITunnelReleaseSink(w, flusher)
|
||||
var sink *openAITunnelReleaseSink
|
||||
if req.stream {
|
||||
sink = newOpenAITunnelReleaseSink(w, flusher)
|
||||
} else {
|
||||
sink = newOpenAIBufferedTunnelReleaseSink(w, flusher, req.requestModel)
|
||||
}
|
||||
|
||||
registry, err := openAIStreamGateRegistrySnapshot()
|
||||
fctx, err := s.openAITunnelOutputFilterContext(req)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate tunnel output filter context build failed", zap.Error(err))
|
||||
writeError(w, http.StatusInternalServerError, "provider_tunnel_error", "stream gate runtime unavailable")
|
||||
emitUsageMetrics(metricLabels, usageStatusError, usageObservation{})
|
||||
return
|
||||
}
|
||||
registry, err := openAIStreamGateRegistrySnapshotFor(s.streamGateConfig(), fctx)
|
||||
if err != nil {
|
||||
handle.Close()
|
||||
s.logger.Warn("openai stream gate tunnel registry build failed", zap.Error(err))
|
||||
|
|
|
|||
557
apps/edge/internal/openai/stream_gate_tunnel_codec.go
Normal file
557
apps/edge/internal/openai/stream_gate_tunnel_codec.go
Normal file
|
|
@ -0,0 +1,557 @@
|
|||
package openai
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"iop/packages/go/streamgate"
|
||||
)
|
||||
|
||||
// openAITunnelCodecState keeps caller-facing provider frames outside semantic
|
||||
// evidence while the Core holds parsed endpoint events. It is reset before each
|
||||
// recovery attempt, which is safe because path switches are allowed only before
|
||||
// any response bytes are committed.
|
||||
type openAITunnelCodecState struct {
|
||||
mu sync.Mutex
|
||||
releases [][]byte
|
||||
terminal []byte
|
||||
termSet bool
|
||||
errorResponse *openAITunnelErrorResponse
|
||||
}
|
||||
|
||||
type openAITunnelErrorResponse struct {
|
||||
status int
|
||||
headers map[string]string
|
||||
body []byte
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) reset() {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.releases = nil
|
||||
s.terminal = nil
|
||||
s.termSet = false
|
||||
s.errorResponse = nil
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) stageErrorResponseStart(status int, headers map[string]string) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.errorResponse = &openAITunnelErrorResponse{
|
||||
status: status,
|
||||
headers: cloneStringMap(headers),
|
||||
}
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) appendErrorResponseWire(payload []byte) {
|
||||
if s == nil || len(payload) == 0 {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
if s.errorResponse != nil {
|
||||
s.errorResponse.body = append(s.errorResponse.body, payload...)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) popErrorResponse() (openAITunnelErrorResponse, bool) {
|
||||
if s == nil {
|
||||
return openAITunnelErrorResponse{}, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.errorResponse == nil {
|
||||
return openAITunnelErrorResponse{}, false
|
||||
}
|
||||
response := openAITunnelErrorResponse{
|
||||
status: s.errorResponse.status,
|
||||
headers: cloneStringMap(s.errorResponse.headers),
|
||||
body: append([]byte(nil), s.errorResponse.body...),
|
||||
}
|
||||
s.errorResponse = nil
|
||||
return response, true
|
||||
}
|
||||
|
||||
func cloneStringMap(src map[string]string) map[string]string {
|
||||
if len(src) == 0 {
|
||||
return nil
|
||||
}
|
||||
dst := make(map[string]string, len(src))
|
||||
for key, value := range src {
|
||||
dst[key] = value
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) pushRelease(payload []byte) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.releases = append(s.releases, append([]byte(nil), payload...))
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) popRelease() ([]byte, bool) {
|
||||
if s == nil {
|
||||
return nil, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(s.releases) == 0 {
|
||||
return nil, false
|
||||
}
|
||||
payload := s.releases[0]
|
||||
s.releases = s.releases[1:]
|
||||
return append([]byte(nil), payload...), true
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) setTerminal(payload []byte) {
|
||||
if s == nil {
|
||||
return
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.terminal = append([]byte(nil), payload...)
|
||||
s.termSet = true
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *openAITunnelCodecState) popTerminal() ([]byte, bool) {
|
||||
if s == nil {
|
||||
return nil, false
|
||||
}
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if !s.termSet {
|
||||
return nil, false
|
||||
}
|
||||
payload := append([]byte(nil), s.terminal...)
|
||||
s.terminal = nil
|
||||
s.termSet = false
|
||||
return payload, true
|
||||
}
|
||||
|
||||
// openAITunnelEndpointCodec parses Chat Completions or Responses SSE frames
|
||||
// into semantic Core events while retaining each original frame for lossless
|
||||
// downstream release.
|
||||
type openAITunnelEndpointCodec struct {
|
||||
endpoint string
|
||||
state *openAITunnelCodecState
|
||||
pending []byte
|
||||
stagedWire []byte
|
||||
chatTools map[int]openAITunnelToolIdentity
|
||||
responseTools map[string]openAITunnelToolIdentity
|
||||
terminal bool
|
||||
}
|
||||
|
||||
type openAITunnelToolIdentity struct {
|
||||
id string
|
||||
name string
|
||||
}
|
||||
|
||||
func newOpenAITunnelEndpointCodec(endpoint string, state *openAITunnelCodecState) *openAITunnelEndpointCodec {
|
||||
if state == nil || (endpoint != openAIRebuildEndpointChat && endpoint != openAIRebuildEndpointResponses) {
|
||||
return nil
|
||||
}
|
||||
return &openAITunnelEndpointCodec{
|
||||
endpoint: endpoint,
|
||||
state: state,
|
||||
chatTools: make(map[int]openAITunnelToolIdentity),
|
||||
responseTools: make(map[string]openAITunnelToolIdentity),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) decode(body []byte, flush bool) ([]streamgate.NormalizedEvent, error) {
|
||||
if c == nil || c.terminal {
|
||||
return nil, nil
|
||||
}
|
||||
c.pending = append(c.pending, body...)
|
||||
var out []streamgate.NormalizedEvent
|
||||
for {
|
||||
frame, rest, ok := takeOpenAISSEFrame(c.pending)
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
c.pending = rest
|
||||
events, err := c.decodeFrame(frame)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, events...)
|
||||
if c.terminal {
|
||||
c.pending = nil
|
||||
return out, nil
|
||||
}
|
||||
}
|
||||
if flush && len(c.pending) > 0 {
|
||||
frame := append([]byte(nil), c.pending...)
|
||||
c.pending = nil
|
||||
events, err := c.decodeFrame(frame)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, events...)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func takeOpenAISSEFrame(buf []byte) (frame, rest []byte, ok bool) {
|
||||
lf := bytes.Index(buf, []byte("\n\n"))
|
||||
crlf := bytes.Index(buf, []byte("\r\n\r\n"))
|
||||
end := -1
|
||||
sepLen := 0
|
||||
if lf >= 0 {
|
||||
end, sepLen = lf, 2
|
||||
}
|
||||
if crlf >= 0 && (end < 0 || crlf < end) {
|
||||
end, sepLen = crlf, 4
|
||||
}
|
||||
if end < 0 {
|
||||
return nil, buf, false
|
||||
}
|
||||
frame = append([]byte(nil), buf[:end+sepLen]...)
|
||||
rest = append([]byte(nil), buf[end+sepLen:]...)
|
||||
return frame, rest, true
|
||||
}
|
||||
|
||||
func openAISSEData(frame []byte) string {
|
||||
normalized := strings.ReplaceAll(string(frame), "\r\n", "\n")
|
||||
var lines []string
|
||||
for _, line := range strings.Split(normalized, "\n") {
|
||||
if !strings.HasPrefix(line, "data:") {
|
||||
continue
|
||||
}
|
||||
lines = append(lines, strings.TrimSpace(strings.TrimPrefix(line, "data:")))
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) decodeFrame(frame []byte) ([]streamgate.NormalizedEvent, error) {
|
||||
data := openAISSEData(frame)
|
||||
if data == "" && json.Valid(bytes.TrimSpace(frame)) {
|
||||
data = string(bytes.TrimSpace(frame))
|
||||
}
|
||||
if strings.TrimSpace(data) == "[DONE]" {
|
||||
return c.finishTerminal(frame, false)
|
||||
}
|
||||
|
||||
var events []streamgate.NormalizedEvent
|
||||
var err error
|
||||
if c.endpoint == openAIRebuildEndpointResponses {
|
||||
events, err = c.decodeResponsesTunnelFrame(data)
|
||||
} else {
|
||||
events, err = c.decodeChatTunnelFrame(data)
|
||||
}
|
||||
if err == nil {
|
||||
// Continue below.
|
||||
} else {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
releaseAttached := false
|
||||
for _, event := range events {
|
||||
switch event.Kind() {
|
||||
case streamgate.EventKindTextDelta, streamgate.EventKindReasoningDelta, streamgate.EventKindToolCallFragment:
|
||||
if releaseAttached == false {
|
||||
payload := append(append([]byte(nil), c.stagedWire...), frame...)
|
||||
c.stagedWire = nil
|
||||
c.state.pushRelease(payload)
|
||||
releaseAttached = true
|
||||
} else {
|
||||
c.state.pushRelease(nil)
|
||||
}
|
||||
case streamgate.EventKindProviderError:
|
||||
if releaseAttached {
|
||||
return c.finishTerminal(nil, true)
|
||||
}
|
||||
return c.finishTerminal(frame, true)
|
||||
}
|
||||
}
|
||||
if releaseAttached == false {
|
||||
// Provider opening/metadata and protocol-level finish frames are wire-only:
|
||||
// retain them until the next semantic release or transport terminal rather
|
||||
// than manufacturing text evidence from their JSON payload.
|
||||
c.stagedWire = append(c.stagedWire, frame...)
|
||||
}
|
||||
return events, nil
|
||||
}
|
||||
|
||||
// finishTransport turns the physical END boundary into the only terminal when
|
||||
// no [DONE] marker already did so. A non-2xx response is a provider-error
|
||||
// lifecycle event even if its body was opaque JSON and therefore wire-only.
|
||||
func (c *openAITunnelEndpointCodec) finishTransport(body []byte, providerError bool) ([]streamgate.NormalizedEvent, error) {
|
||||
if c == nil || c.terminal {
|
||||
return nil, nil
|
||||
}
|
||||
events, err := c.decode(body, true)
|
||||
if err == nil && c.terminal == false {
|
||||
terminal, terminalErr := c.finishTerminal(nil, providerError)
|
||||
if terminalErr != nil {
|
||||
return nil, terminalErr
|
||||
}
|
||||
return append(events, terminal...), nil
|
||||
}
|
||||
return events, err
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) finishTerminal(frame []byte, providerError bool) ([]streamgate.NormalizedEvent, error) {
|
||||
if c.terminal {
|
||||
return nil, nil
|
||||
}
|
||||
payload := append(append([]byte(nil), c.stagedWire...), frame...)
|
||||
c.stagedWire = nil
|
||||
c.state.setTerminal(payload)
|
||||
c.terminal = true
|
||||
if providerError {
|
||||
ev, err := newOpenAIProviderErrorEvent(streamGateErrorTunnelFailed)
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
}
|
||||
ev, err := streamgate.NewTerminalEvent(streamGateChannelDefault, time.Now())
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) decodeChatTunnelFrame(data string) ([]streamgate.NormalizedEvent, error) {
|
||||
if strings.TrimSpace(data) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var payload struct {
|
||||
Choices []struct {
|
||||
Delta struct {
|
||||
Content string `json:"content"`
|
||||
Reasoning string `json:"reasoning"`
|
||||
ReasoningContent string `json:"reasoning_content"`
|
||||
ToolCalls []struct {
|
||||
Index int `json:"index"`
|
||||
ID string `json:"id"`
|
||||
Function struct {
|
||||
Name string `json:"name"`
|
||||
Arguments string `json:"arguments"`
|
||||
} `json:"function"`
|
||||
} `json:"tool_calls"`
|
||||
} `json:"delta"`
|
||||
Message struct {
|
||||
Content string `json:"content"`
|
||||
ReasoningContent string `json:"reasoning_content"`
|
||||
} `json:"message"`
|
||||
FinishReason *string `json:"finish_reason"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(data), &payload); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
var events []streamgate.NormalizedEvent
|
||||
for _, choice := range payload.Choices {
|
||||
content := choice.Delta.Content
|
||||
if content == "" {
|
||||
content = choice.Message.Content
|
||||
}
|
||||
if content != "" {
|
||||
ev, err := streamgate.NewTextDeltaEvent(streamGateChannelDefault, content, time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
}
|
||||
reasoning := choice.Delta.ReasoningContent
|
||||
if reasoning == "" {
|
||||
reasoning = choice.Delta.Reasoning
|
||||
}
|
||||
if reasoning == "" {
|
||||
reasoning = choice.Message.ReasoningContent
|
||||
}
|
||||
if reasoning != "" {
|
||||
ev, err := streamgate.NewReasoningDeltaEvent(streamGateChannelDefault, reasoning, time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
}
|
||||
for _, tool := range choice.Delta.ToolCalls {
|
||||
identity := c.chatTools[tool.Index]
|
||||
if tool.ID != "" {
|
||||
identity.id = tool.ID
|
||||
}
|
||||
if tool.Function.Name != "" {
|
||||
identity.name = tool.Function.Name
|
||||
}
|
||||
c.chatTools[tool.Index] = identity
|
||||
if tool.Function.Arguments == "" {
|
||||
continue
|
||||
}
|
||||
id := identity.id
|
||||
if id == "" {
|
||||
id = fmt.Sprintf("tool-%d", tool.Index)
|
||||
}
|
||||
name := identity.name
|
||||
if name == "" {
|
||||
name = "function"
|
||||
}
|
||||
ev, err := streamgate.NewToolCallFragmentEvent(streamGateChannelDefault, id, name, tool.Function.Arguments, time.Now())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
events = append(events, ev)
|
||||
}
|
||||
// finish_reason is endpoint protocol state, not the transport terminal.
|
||||
// Its frame remains in the release queue until [DONE] or END closes once.
|
||||
}
|
||||
return events, nil
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) rememberResponseTool(identity openAITunnelToolIdentity, keys ...string) {
|
||||
if identity.id == "" && identity.name == "" {
|
||||
return
|
||||
}
|
||||
for _, key := range keys {
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
current := c.responseTools[key]
|
||||
if identity.id != "" {
|
||||
current.id = identity.id
|
||||
}
|
||||
if identity.name != "" {
|
||||
current.name = identity.name
|
||||
}
|
||||
c.responseTools[key] = current
|
||||
}
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) responseTool(keys ...string) openAITunnelToolIdentity {
|
||||
var identity openAITunnelToolIdentity
|
||||
for _, key := range keys {
|
||||
candidate := c.responseTools[key]
|
||||
if identity.id == "" {
|
||||
identity.id = candidate.id
|
||||
}
|
||||
if identity.name == "" {
|
||||
identity.name = candidate.name
|
||||
}
|
||||
}
|
||||
return identity
|
||||
}
|
||||
|
||||
func (c *openAITunnelEndpointCodec) decodeResponsesTunnelFrame(data string) ([]streamgate.NormalizedEvent, error) {
|
||||
if strings.TrimSpace(data) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var payload struct {
|
||||
Type string `json:"type"`
|
||||
Delta string `json:"delta"`
|
||||
ItemID string `json:"item_id"`
|
||||
CallID string `json:"call_id"`
|
||||
Name string `json:"name"`
|
||||
OutputIdx int `json:"output_index"`
|
||||
OutputText string `json:"output_text"`
|
||||
Item struct {
|
||||
ID string `json:"id"`
|
||||
CallID string `json:"call_id"`
|
||||
Name string `json:"name"`
|
||||
} `json:"item"`
|
||||
Output []struct {
|
||||
Type string `json:"type"`
|
||||
ID string `json:"id"`
|
||||
CallID string `json:"call_id"`
|
||||
Name string `json:"name"`
|
||||
Arguments string `json:"arguments"`
|
||||
Content []struct {
|
||||
Type string `json:"type"`
|
||||
Text string `json:"text"`
|
||||
} `json:"content"`
|
||||
} `json:"output"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(data), &payload); err != nil {
|
||||
return nil, nil
|
||||
}
|
||||
outputKey := fmt.Sprintf("output-%d", payload.OutputIdx)
|
||||
c.rememberResponseTool(openAITunnelToolIdentity{id: payload.CallID, name: payload.Name}, payload.CallID, payload.ItemID, outputKey)
|
||||
c.rememberResponseTool(openAITunnelToolIdentity{id: payload.Item.CallID, name: payload.Item.Name}, payload.Item.CallID, payload.Item.ID, outputKey)
|
||||
if payload.Type == "" && (payload.OutputText != "" || len(payload.Output) > 0) {
|
||||
var events []streamgate.NormalizedEvent
|
||||
text := payload.OutputText
|
||||
if text == "" {
|
||||
for _, item := range payload.Output {
|
||||
for _, content := range item.Content {
|
||||
if content.Type == "output_text" {
|
||||
text += content.Text
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if text != "" {
|
||||
event, eventErr := streamgate.NewTextDeltaEvent(streamGateChannelDefault, text, time.Now())
|
||||
if eventErr != nil {
|
||||
return nil, eventErr
|
||||
}
|
||||
events = append(events, event)
|
||||
}
|
||||
for i, item := range payload.Output {
|
||||
key := fmt.Sprintf("output-%d", i)
|
||||
c.rememberResponseTool(openAITunnelToolIdentity{id: item.CallID, name: item.Name}, item.CallID, item.ID, key)
|
||||
if item.Type != "function_call" || item.Arguments == "" {
|
||||
continue
|
||||
}
|
||||
identity := c.responseTool(item.CallID, item.ID, key)
|
||||
id := identity.id
|
||||
if id == "" {
|
||||
id = fmt.Sprintf("call-%d", i)
|
||||
}
|
||||
name := identity.name
|
||||
if name == "" {
|
||||
name = "function"
|
||||
}
|
||||
event, eventErr := streamgate.NewToolCallFragmentEvent(streamGateChannelDefault, id, name, item.Arguments, time.Now())
|
||||
if eventErr != nil {
|
||||
return nil, eventErr
|
||||
}
|
||||
events = append(events, event)
|
||||
}
|
||||
return events, nil
|
||||
}
|
||||
switch payload.Type {
|
||||
case "response.output_text.delta":
|
||||
if payload.Delta == "" {
|
||||
return nil, nil
|
||||
}
|
||||
ev, err := streamgate.NewTextDeltaEvent(streamGateChannelDefault, payload.Delta, time.Now())
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
case "response.reasoning_text.delta", "response.reasoning_summary_text.delta":
|
||||
if payload.Delta == "" {
|
||||
return nil, nil
|
||||
}
|
||||
ev, err := streamgate.NewReasoningDeltaEvent(streamGateChannelDefault, payload.Delta, time.Now())
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
case "response.function_call_arguments.delta":
|
||||
if payload.Delta == "" {
|
||||
return nil, nil
|
||||
}
|
||||
identity := c.responseTool(payload.CallID, payload.ItemID, outputKey)
|
||||
id := identity.id
|
||||
if id == "" {
|
||||
id = fmt.Sprintf("call-%d", payload.OutputIdx)
|
||||
}
|
||||
name := identity.name
|
||||
if name == "" {
|
||||
name = "function"
|
||||
}
|
||||
ev, err := streamgate.NewToolCallFragmentEvent(streamGateChannelDefault, id, name, payload.Delta, time.Now())
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
case "response.completed", "response.incomplete":
|
||||
return nil, nil
|
||||
case "response.failed", "error":
|
||||
ev, err := newOpenAIProviderErrorEvent(streamGateErrorTunnelFailed)
|
||||
return []streamgate.NormalizedEvent{ev}, err
|
||||
default:
|
||||
return nil, nil
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -336,7 +336,7 @@ func (m *modelQueueManager) pumpAllLocked() {
|
|||
// resolver to rebuild the slice from current store/catalog/registry state;
|
||||
// otherwise it returns the enqueue-time snapshot.
|
||||
//
|
||||
// The return value is tri-state:
|
||||
// The return value has four states:
|
||||
// - resolveOk: usable candidates returned; the pump continues to
|
||||
// findAvailableNodeLocked (which may skip the item if all candidates
|
||||
// are at capacity — temporary block, not terminal).
|
||||
|
|
@ -345,6 +345,9 @@ func (m *modelQueueManager) pumpAllLocked() {
|
|||
// - resolveResolverError: the resolver returned an error distinct from
|
||||
// errProviderUnavailable. The item remains queued so a later live-state
|
||||
// re-evaluation can recover from a catalog/config resolver fault.
|
||||
// - resolveTerminalError: request-specific admission policy rejected every
|
||||
// otherwise eligible candidate. The item is removed and the typed error is
|
||||
// delivered without a reservation or provider dispatch.
|
||||
//
|
||||
// Orphaned candidates (disconnected nodes whose resources were cleared by
|
||||
// releaseNode) are filtered out. If all candidates are orphaned, the result
|
||||
|
|
@ -356,6 +359,9 @@ func (m *modelQueueManager) resolveQueuedCandidatesLocked(item *queueItem) ([]ca
|
|||
}
|
||||
candidates, err := item.resolveCandidates()
|
||||
if err != nil {
|
||||
if errors.Is(err, ErrProviderPoolCandidateRejected) {
|
||||
return nil, resolveTerminalError, err
|
||||
}
|
||||
if errors.Is(err, errProviderUnavailable) {
|
||||
return nil, resolveNoCandidates, nil
|
||||
}
|
||||
|
|
@ -397,10 +403,11 @@ func (m *modelQueueManager) resolveQueuedCandidatesLocked(item *queueItem) ([]ca
|
|||
// dispatch, the global enqueue sequence decides, which is what keeps FIFO across
|
||||
// groups and prevents a busy group from starving a quiet one.
|
||||
//
|
||||
// Resolver outcomes are tri-state: resolveOk continues to selection
|
||||
// Resolver outcomes are four-state: resolveOk continues to selection
|
||||
// (findAvailableNodeLocked); resolveNoCandidates is terminal and delivers typed
|
||||
// unavailable; resolveResolverError leaves the item queued for a later live-state
|
||||
// re-evaluation.
|
||||
// re-evaluation; resolveTerminalError removes the item and preserves its typed
|
||||
// request-admission failure.
|
||||
func (m *modelQueueManager) pumpOnceLocked() bool {
|
||||
pending := m.pendingItemsLocked()
|
||||
|
||||
|
|
@ -418,7 +425,7 @@ func (m *modelQueueManager) pumpOnceLocked() bool {
|
|||
}
|
||||
|
||||
for _, ref := range pending {
|
||||
candidates, outcome, _ := m.resolveQueuedCandidatesLocked(ref.item)
|
||||
candidates, outcome, resolveErr := m.resolveQueuedCandidatesLocked(ref.item)
|
||||
switch outcome {
|
||||
case resolveOk:
|
||||
candidate := m.findAvailableNodeLocked(ref.group, candidates, ref.item.long)
|
||||
|
|
@ -459,9 +466,15 @@ func (m *modelQueueManager) pumpOnceLocked() bool {
|
|||
// Resolver returned a non-terminal error. Keep the item queued so a
|
||||
// corrected catalog/config snapshot can dispatch it on a later pump.
|
||||
continue
|
||||
case resolveTerminalError:
|
||||
m.removeQueuedItemLocked(ref.group, ref.item)
|
||||
select {
|
||||
case ref.item.waitCh <- admitResult{err: resolveErr}:
|
||||
default:
|
||||
}
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -99,7 +99,7 @@ func (m *modelQueueManager) resolveAndPumpAllLocked(excludeNodeID string, exclud
|
|||
for _, group := range m.groups {
|
||||
for _, item := range group.queue {
|
||||
if item.resolveCandidates != nil {
|
||||
candidates, outcome, _ := m.resolveQueuedCandidatesLocked(item)
|
||||
candidates, outcome, resolveErr := m.resolveQueuedCandidatesLocked(item)
|
||||
if outcome == resolveNoCandidates {
|
||||
// Terminal no-candidate: remove immediately, before the pump,
|
||||
// so the dispatch pass only sees items that can actually run.
|
||||
|
|
@ -115,6 +115,14 @@ func (m *modelQueueManager) resolveAndPumpAllLocked(excludeNodeID string, exclud
|
|||
// is recoverable and must not block the dispatch pass.
|
||||
continue
|
||||
}
|
||||
if outcome == resolveTerminalError {
|
||||
m.removeQueuedItemLocked(group, item)
|
||||
select {
|
||||
case item.waitCh <- admitResult{err: resolveErr}:
|
||||
default:
|
||||
}
|
||||
continue
|
||||
}
|
||||
if candidates != nil {
|
||||
item.candidates = candidates
|
||||
}
|
||||
|
|
|
|||
|
|
@ -46,6 +46,9 @@ const (
|
|||
// errProviderUnavailable. The item remains queued for a later live-state
|
||||
// re-evaluation and no terminal result is delivered.
|
||||
resolveResolverError
|
||||
// resolveTerminalError preserves a typed, request-specific admission error
|
||||
// and removes the waiter without reserving or dispatching a provider slot.
|
||||
resolveTerminalError
|
||||
)
|
||||
|
||||
// candidateNode pairs a registry entry with the per-request capacity derived
|
||||
|
|
@ -71,8 +74,12 @@ type candidateNode struct {
|
|||
priority int
|
||||
providerID string // non-empty for provider-pool candidates
|
||||
providerType string // non-empty for provider-pool candidates; the raw provider type (e.g. "vllm", "ollama")
|
||||
adapter string // non-empty for provider-pool candidates; dispatch adapter key
|
||||
servedTarget string // concrete served model name; used for target rewrite
|
||||
// lifecycleCapabilities are the provider-advertised capabilities used by
|
||||
// request-local admission policy. They are copied from the Edge-owned
|
||||
// provider catalog for every initial and re-resolved candidate.
|
||||
lifecycleCapabilities []string
|
||||
adapter string // non-empty for provider-pool candidates; dispatch adapter key
|
||||
servedTarget string // concrete served model name; used for target rewrite
|
||||
// longContextCapacity is the provider's concurrent long-context slot limit.
|
||||
// Zero means the provider declares no dedicated long-slot limit and long
|
||||
// requests are not gated on it.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package service
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
|
|
@ -36,15 +37,37 @@ type prepareTunnelFunc func(req SubmitProviderTunnelRequest) (SubmitProviderTunn
|
|||
// present before dispatch, while leaving the tunnel branch untouched.
|
||||
type prepareRunFunc func(req SubmitRunRequest) (SubmitRunRequest, error)
|
||||
|
||||
// ProviderPoolCandidate is the stable, caller-neutral view passed to a
|
||||
// request-local provider-pool admission predicate. It intentionally contains
|
||||
// only actual target and configured provider capability facts; HTTP caller or
|
||||
// agent identity is never part of pool selection.
|
||||
type ProviderPoolCandidate struct {
|
||||
ActualModel string
|
||||
ProviderID string
|
||||
ExecutionPath string
|
||||
LifecycleCapabilities []string
|
||||
}
|
||||
|
||||
// ProviderPoolCandidatePredicate decides whether a resolved provider candidate
|
||||
// can serve one request. The service invokes it for both the first admission
|
||||
// and every queue/recovery re-resolution.
|
||||
type ProviderPoolCandidatePredicate func(ProviderPoolCandidate) bool
|
||||
|
||||
// ErrProviderPoolCandidateRejected reports that otherwise available provider
|
||||
// candidates were all rejected by a request-local admission predicate before a
|
||||
// slot was reserved or a provider dispatch was sent.
|
||||
var ErrProviderPoolCandidateRejected = errors.New("provider pool candidates rejected by request admission policy")
|
||||
|
||||
// ProviderPoolDispatchRequest bundles the Run and Tunnel surface values for
|
||||
// a single one-shot provider-pool dispatch. SubmitProviderPool uses exactly
|
||||
// one queue admission to select a candidate, then dispatches only the
|
||||
// execution path indicated by the candidate's executionPath.
|
||||
type ProviderPoolDispatchRequest struct {
|
||||
Run SubmitRunRequest
|
||||
Tunnel SubmitProviderTunnelRequest
|
||||
PrepareTunnel prepareTunnelFunc
|
||||
PrepareRun prepareRunFunc
|
||||
Run SubmitRunRequest
|
||||
Tunnel SubmitProviderTunnelRequest
|
||||
PrepareTunnel prepareTunnelFunc
|
||||
PrepareRun prepareRunFunc
|
||||
AcceptCandidate ProviderPoolCandidatePredicate
|
||||
}
|
||||
|
||||
// ProviderPoolDispatchResult describes which execution path was selected and
|
||||
|
|
@ -71,6 +94,10 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
candidates, rejected := filterProviderPoolCandidates(candidates, req.AcceptCandidate)
|
||||
if rejected {
|
||||
return nil, ErrProviderPoolCandidateRejected
|
||||
}
|
||||
|
||||
// Provider-pool dispatch uses the canonical policy from the runtime snapshot.
|
||||
var policy groupPolicy
|
||||
|
|
@ -81,7 +108,21 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
}
|
||||
|
||||
long := req.Run.ContextClass == contextClassLong
|
||||
selected, queueReason, err := s.queue.admitWithReason(ctx, req.Run.ModelGroupKey, req.Run.Adapter, req.Run.Target, candidates, policy, s.resolveQueueCandidatesClosure(req.Run), long, req.Run.ProviderPool)
|
||||
resolveCandidates := s.resolveQueueCandidatesClosure(req.Run)
|
||||
if req.AcceptCandidate != nil {
|
||||
resolveCandidates = func() ([]candidateNode, error) {
|
||||
resolved, _, err := s.resolveQueueCandidates(req.Run)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
accepted, rejected := filterProviderPoolCandidates(resolved, req.AcceptCandidate)
|
||||
if rejected {
|
||||
return nil, ErrProviderPoolCandidateRejected
|
||||
}
|
||||
return accepted, nil
|
||||
}
|
||||
}
|
||||
selected, queueReason, err := s.queue.admitWithReason(ctx, req.Run.ModelGroupKey, req.Run.Adapter, req.Run.Target, candidates, policy, resolveCandidates, long, req.Run.ProviderPool)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -123,6 +164,26 @@ func (s *Service) SubmitProviderPool(ctx context.Context, req ProviderPoolDispat
|
|||
}
|
||||
}
|
||||
|
||||
func filterProviderPoolCandidates(candidates []candidateNode, accept ProviderPoolCandidatePredicate) ([]candidateNode, bool) {
|
||||
if accept == nil || len(candidates) == 0 {
|
||||
return candidates, false
|
||||
}
|
||||
accepted := make([]candidateNode, 0, len(candidates))
|
||||
for _, candidate := range candidates {
|
||||
capabilities := append([]string(nil), candidate.lifecycleCapabilities...)
|
||||
if !accept(ProviderPoolCandidate{
|
||||
ActualModel: candidate.servedTarget,
|
||||
ProviderID: candidate.providerID,
|
||||
ExecutionPath: string(candidate.executionPath),
|
||||
LifecycleCapabilities: capabilities,
|
||||
}) {
|
||||
continue
|
||||
}
|
||||
accepted = append(accepted, candidate)
|
||||
}
|
||||
return accepted, len(accepted) == 0
|
||||
}
|
||||
|
||||
// dispatchProviderPoolTunnel relays the selected candidate's raw provider
|
||||
// request after provider-pool admission. The tunnel inherits the Run's
|
||||
// identity, metadata, and long-context classification so passthrough dispatch
|
||||
|
|
|
|||
|
|
@ -559,3 +559,59 @@ func TestProviderPoolMaxQueueIgnoresLegacyPending(t *testing.T) {
|
|||
t.Fatalf("expected total legacy group inflight=0 after holder release, got %d", totalLgInflight)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderPoolQueuedPredicateRejectionIsTerminal(t *testing.T) {
|
||||
store := edgenode.NewNodeStore()
|
||||
store.Add(&edgenode.NodeRecord{
|
||||
ID: "node-policy",
|
||||
Runtime: config.RuntimeConf{Concurrency: 1},
|
||||
Providers: []config.NodeProviderConf{{ID: "prov-policy", Capacity: 1}},
|
||||
})
|
||||
entry := &edgenode.NodeEntry{NodeID: "node-policy"}
|
||||
candidates := []candidateNode{{entry: entry, capacity: 1, providerID: "prov-policy"}}
|
||||
manager := newModelQueueManager(store)
|
||||
manager.setProviderPoolPolicyLocked(store, NewGroupPolicy(4, time.Second))
|
||||
|
||||
first, err := manager.admit(t.Context(), "group-policy", "", "", candidates, groupPolicy{}, nil, false, true)
|
||||
if err != nil {
|
||||
t.Fatalf("initial admit: %v", err)
|
||||
}
|
||||
|
||||
reject := false
|
||||
resolver := func() ([]candidateNode, error) {
|
||||
if reject {
|
||||
return nil, ErrProviderPoolCandidateRejected
|
||||
}
|
||||
return candidates, nil
|
||||
}
|
||||
resultCh := make(chan providerPoolAdmitResult, 1)
|
||||
ctx, cancel := context.WithTimeout(t.Context(), time.Second)
|
||||
defer cancel()
|
||||
go func() {
|
||||
candidate, admitErr := manager.admit(ctx, "group-policy", "", "", candidates, groupPolicy{}, resolver, false, true)
|
||||
resultCh <- providerPoolAdmitResult{candidate: candidate, err: admitErr}
|
||||
}()
|
||||
requireProviderPoolPending(t, manager, 1)
|
||||
|
||||
reject = true
|
||||
manager.mu.Lock()
|
||||
manager.pumpAllLocked()
|
||||
pending := manager.pendingProviderPoolCountLocked()
|
||||
leases := len(manager.leases)
|
||||
manager.mu.Unlock()
|
||||
|
||||
result := <-resultCh
|
||||
if !errors.Is(result.err, ErrProviderPoolCandidateRejected) {
|
||||
t.Fatalf("queued admission err=%v, want ErrProviderPoolCandidateRejected", result.err)
|
||||
}
|
||||
if result.candidate != nil {
|
||||
t.Fatalf("queued policy rejection reserved candidate %+v", result.candidate)
|
||||
}
|
||||
if pending != 0 {
|
||||
t.Fatalf("pending=%d, want zero after terminal rejection", pending)
|
||||
}
|
||||
if leases != 1 {
|
||||
t.Fatalf("leases=%d, want only original reservation", leases)
|
||||
}
|
||||
manager.releaseLease(first.leaseID, "test-cleanup")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -299,6 +299,7 @@ func applyProviderDispatchFields(c *candidateNode, prov config.NodeProviderConf)
|
|||
c.longContextCapacity = prov.LongContextCapacity
|
||||
c.priority = prov.Priority
|
||||
c.providerType = prov.Type
|
||||
c.lifecycleCapabilities = append(c.lifecycleCapabilities[:0], prov.LifecycleCapabilities...)
|
||||
c.adapter = providerAdapterKey(prov)
|
||||
c.executionPath = classifyProviderExecutionPath(prov.Type)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -110,17 +110,46 @@ openai:
|
|||
timeout_sec: 120
|
||||
strict_output: true
|
||||
strict_stream_buffer: false
|
||||
# stream_evidence_gate configures request-local Recovery Coordinator and ingress snapshot limits.
|
||||
# enabled: route normalized live-SSE chat completion and provider tunnel passthrough through the
|
||||
# streamgate request runtime instead of the legacy eager-write path (default: false).
|
||||
# stream_evidence_gate configures request-local Recovery Coordinator, ingress snapshot limits,
|
||||
# and optional caller-neutral semantic output filters.
|
||||
# enabled: route supported Chat Completions, normalized Responses, and provider tunnel passthrough
|
||||
# through the streamgate request runtime instead of the legacy eager-write path (default: false).
|
||||
# max_request_fault_recovery: request fault recovery cap (0..3, default: 3). Explicit 0 disables recovery.
|
||||
# max_strategy_fault_recovery: strategy fault recovery cap (0..max_request_fault_recovery, default: inherits request limit).
|
||||
# max_ingress_snapshot_bytes: ingress snapshot size limit in bytes (1..16777216 [16 MiB], default: 16777216).
|
||||
stream_evidence_gate:
|
||||
enabled: false
|
||||
environment: dev # dev | dev-corp; request-start selector snapshot
|
||||
max_request_fault_recovery: 3
|
||||
max_strategy_fault_recovery: 3
|
||||
max_ingress_snapshot_bytes: 16777216
|
||||
# filters are disabled by omission. Each policy has one unique filter kind:
|
||||
# repeat_guard (request-local history plus Unicode rolling/current-stream
|
||||
# inspection), schema_gate (only when metadata.scheme is present), or
|
||||
# provider_error (error-event lifecycle observation only; matching and
|
||||
# retry semantics belong to a follow-up task). Repeat evidence and
|
||||
# observations retain fingerprints/counts/offsets, never prompt, output,
|
||||
# reasoning, tool arguments, or results. A provider must advertise
|
||||
# the configured capability in lifecycle_capabilities only when a blocking
|
||||
# policy is enabled. Selectors may refine enablement/enforcement by
|
||||
# environment, model_group, model, or provider; caller/agent identity is not
|
||||
# a selector.
|
||||
# filters:
|
||||
# - filter: repeat_guard
|
||||
# enforcement: blocking # blocking | observe_only
|
||||
# capability: output.repeat_guard
|
||||
# hold_evidence_runes: 500
|
||||
# timeout_ms: 5000
|
||||
# - filter: schema_gate
|
||||
# enforcement: blocking
|
||||
# capability: output.schema_gate
|
||||
# - filter: provider_error
|
||||
# enforcement: blocking
|
||||
# capability: output.provider_error
|
||||
# selectors:
|
||||
# - type: provider # environment | model_group | model | provider
|
||||
# key: "provider-id"
|
||||
# enabled: true
|
||||
|
||||
# === Provider-pool (models[] / nodes[].providers[]) — recommended ===
|
||||
# Top-level models[] defines canonical routing keys and their provider-pool mapping.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package config
|
|||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Default provider-pool queue policy values. These are the only canonical
|
||||
|
|
@ -134,14 +135,209 @@ type EdgeOpenAIConf struct {
|
|||
|
||||
// StreamEvidenceGateConf configures request-local Recovery Coordinator and ingress snapshot limits.
|
||||
type StreamEvidenceGateConf struct {
|
||||
// Enabled routes the normalized live-SSE chat completion and provider
|
||||
// tunnel passthrough paths through the packages/go/streamgate request
|
||||
// runtime instead of the legacy eager-write path. Default false preserves
|
||||
// existing passthrough behavior exactly.
|
||||
Enabled bool `mapstructure:"enabled" yaml:"enabled,omitempty"`
|
||||
MaxRequestFaultRecovery *int `mapstructure:"max_request_fault_recovery" yaml:"max_request_fault_recovery,omitempty"`
|
||||
MaxStrategyFaultRecovery *int `mapstructure:"max_strategy_fault_recovery" yaml:"max_strategy_fault_recovery,omitempty"`
|
||||
MaxIngressSnapshotBytes int `mapstructure:"max_ingress_snapshot_bytes" yaml:"max_ingress_snapshot_bytes,omitempty"`
|
||||
// Enabled routes supported Chat Completions, normalized Responses, and
|
||||
// provider-tunnel passthrough paths through the packages/go/streamgate
|
||||
// request runtime instead of the legacy eager-write path. Default false
|
||||
// preserves existing behavior exactly.
|
||||
Enabled bool `mapstructure:"enabled" yaml:"enabled,omitempty"`
|
||||
// Environment is the request-stable deployment selector input. Empty
|
||||
// defaults to dev; only the deployment labels supported by the active
|
||||
// rollout policy are accepted.
|
||||
Environment string `mapstructure:"environment" yaml:"environment,omitempty"`
|
||||
MaxRequestFaultRecovery *int `mapstructure:"max_request_fault_recovery" yaml:"max_request_fault_recovery,omitempty"`
|
||||
MaxStrategyFaultRecovery *int `mapstructure:"max_strategy_fault_recovery" yaml:"max_strategy_fault_recovery,omitempty"`
|
||||
MaxIngressSnapshotBytes int `mapstructure:"max_ingress_snapshot_bytes" yaml:"max_ingress_snapshot_bytes,omitempty"`
|
||||
// Filters declares the semantic output-validation filters that the request
|
||||
// runtime registers on top of the always-applicable core mechanics. An empty
|
||||
// slice keeps the legacy production behavior (core mechanics + request-local
|
||||
// tool validation) exactly. Each entry is keyed by its filter kind, which is
|
||||
// unique within the slice.
|
||||
Filters []StreamGateFilterPolicyConf `mapstructure:"filters" yaml:"filters,omitempty"`
|
||||
}
|
||||
|
||||
// Stream-gate output-filter kinds. These are the caller-neutral semantic
|
||||
// filters that consume the Stream Evidence Gate Core mechanics.
|
||||
const (
|
||||
// StreamGateFilterRepeatGuard is the rolling-window repeat guard.
|
||||
StreamGateFilterRepeatGuard = "repeat_guard"
|
||||
// StreamGateFilterSchemaGate is the metadata.scheme terminal gate.
|
||||
StreamGateFilterSchemaGate = "schema_gate"
|
||||
// StreamGateFilterProviderError is the provider-error lifecycle participant.
|
||||
StreamGateFilterProviderError = "provider_error"
|
||||
|
||||
// Supported request-stable deployment selector values.
|
||||
StreamGateEnvironmentDev = "dev"
|
||||
StreamGateEnvironmentDevCorp = "dev-corp"
|
||||
)
|
||||
|
||||
// Stream-gate output-filter enforcement modes mirrored at the config boundary.
|
||||
const (
|
||||
StreamGateFilterEnforcementBlocking = "blocking"
|
||||
StreamGateFilterEnforcementObserveOnly = "observe_only"
|
||||
)
|
||||
|
||||
// Stream-gate output-filter selector types mirrored at the config boundary.
|
||||
const (
|
||||
StreamGateFilterSelectorEnvironment = "environment"
|
||||
StreamGateFilterSelectorModelGroup = "model_group"
|
||||
StreamGateFilterSelectorModel = "model"
|
||||
StreamGateFilterSelectorProvider = "provider"
|
||||
)
|
||||
|
||||
// Stream-gate output-filter default and absolute bounds.
|
||||
const (
|
||||
DefaultStreamGateFilterHoldEvidenceRunes = 500
|
||||
MaxStreamGateFilterHoldEvidenceRunes = 65536
|
||||
DefaultStreamGateFilterTimeoutMS = 5000
|
||||
MaxStreamGateFilterTimeoutMS = 60000
|
||||
)
|
||||
|
||||
// knownStreamGateFilterKinds is the closed set of configurable filter kinds.
|
||||
var knownStreamGateFilterKinds = map[string]struct{}{
|
||||
StreamGateFilterRepeatGuard: {},
|
||||
StreamGateFilterSchemaGate: {},
|
||||
StreamGateFilterProviderError: {},
|
||||
}
|
||||
|
||||
// knownStreamGateFilterEnforcements is the closed set of enforcement modes.
|
||||
var knownStreamGateFilterEnforcements = map[string]struct{}{
|
||||
StreamGateFilterEnforcementBlocking: {},
|
||||
StreamGateFilterEnforcementObserveOnly: {},
|
||||
}
|
||||
|
||||
// knownStreamGateFilterSelectorTypes is the closed set of selector types.
|
||||
var knownStreamGateFilterSelectorTypes = map[string]struct{}{
|
||||
StreamGateFilterSelectorEnvironment: {},
|
||||
StreamGateFilterSelectorModelGroup: {},
|
||||
StreamGateFilterSelectorModel: {},
|
||||
StreamGateFilterSelectorProvider: {},
|
||||
}
|
||||
|
||||
// StreamGateFilterSelectorConf overrides a filter's enablement and enforcement
|
||||
// at a single selector scope (environment/model_group/model/provider). It never
|
||||
// changes priority, timeout, or hold bounds: those stay request-stable so a
|
||||
// filter's recovery intent priority always matches its resolved registration
|
||||
// priority.
|
||||
type StreamGateFilterSelectorConf struct {
|
||||
Type string `mapstructure:"type" yaml:"type"`
|
||||
Key string `mapstructure:"key" yaml:"key"`
|
||||
// Enabled overrides the base enabled value at this selector. Omitted (nil)
|
||||
// inherits the base filter enabled value.
|
||||
Enabled *bool `mapstructure:"enabled" yaml:"enabled,omitempty"`
|
||||
// Enforcement overrides the base enforcement at this selector. Empty inherits
|
||||
// the base filter enforcement.
|
||||
Enforcement string `mapstructure:"enforcement" yaml:"enforcement,omitempty"`
|
||||
}
|
||||
|
||||
// StreamGateFilterPolicyConf declares one semantic output-validation filter and
|
||||
// its request-stable policy. Enablement and enforcement can be refined per
|
||||
// environment/model/provider selector; priority, timeout, capability, and hold
|
||||
// bounds are request-stable base values.
|
||||
type StreamGateFilterPolicyConf struct {
|
||||
// Filter is the filter kind. It is unique within the Filters slice.
|
||||
Filter string `mapstructure:"filter" yaml:"filter"`
|
||||
// Enabled is the base enablement. Omitted (nil) defaults to true.
|
||||
Enabled *bool `mapstructure:"enabled" yaml:"enabled,omitempty"`
|
||||
// Enforcement is the base enforcement. Empty defaults to blocking.
|
||||
Enforcement string `mapstructure:"enforcement" yaml:"enforcement,omitempty"`
|
||||
// Capability is the provider capability id a candidate must advertise for
|
||||
// this filter to be admissible. Empty defaults to output.<filter>.
|
||||
Capability string `mapstructure:"capability" yaml:"capability,omitempty"`
|
||||
// HoldEvidenceRunes is the rolling-window evidence bound for the repeat
|
||||
// guard. Zero defaults to 500. Ignored by non-rolling filters.
|
||||
HoldEvidenceRunes int `mapstructure:"hold_evidence_runes" yaml:"hold_evidence_runes,omitempty"`
|
||||
// TimeoutMS is the per-evaluation timeout. Zero defaults to 5000.
|
||||
TimeoutMS int `mapstructure:"timeout_ms" yaml:"timeout_ms,omitempty"`
|
||||
// Priority is the evaluation/arbitration priority. Non-negative.
|
||||
Priority int `mapstructure:"priority" yaml:"priority,omitempty"`
|
||||
Selectors []StreamGateFilterSelectorConf `mapstructure:"selectors" yaml:"selectors,omitempty"`
|
||||
}
|
||||
|
||||
// EffectiveEnabled reports the base enablement, defaulting to true when omitted.
|
||||
func (f StreamGateFilterPolicyConf) EffectiveEnabled() bool {
|
||||
if f.Enabled == nil {
|
||||
return true
|
||||
}
|
||||
return *f.Enabled
|
||||
}
|
||||
|
||||
// EffectiveEnforcement reports the base enforcement, defaulting to blocking.
|
||||
func (f StreamGateFilterPolicyConf) EffectiveEnforcement() string {
|
||||
if strings.TrimSpace(f.Enforcement) == "" {
|
||||
return StreamGateFilterEnforcementBlocking
|
||||
}
|
||||
return f.Enforcement
|
||||
}
|
||||
|
||||
// EffectiveCapability reports the required provider capability, defaulting to
|
||||
// output.<filter> when omitted.
|
||||
func (f StreamGateFilterPolicyConf) EffectiveCapability() string {
|
||||
if strings.TrimSpace(f.Capability) == "" {
|
||||
return "output." + f.Filter
|
||||
}
|
||||
return f.Capability
|
||||
}
|
||||
|
||||
// EffectiveHoldEvidenceRunes reports the rolling-window bound, defaulting to 500.
|
||||
func (f StreamGateFilterPolicyConf) EffectiveHoldEvidenceRunes() int {
|
||||
if f.HoldEvidenceRunes == 0 {
|
||||
return DefaultStreamGateFilterHoldEvidenceRunes
|
||||
}
|
||||
return f.HoldEvidenceRunes
|
||||
}
|
||||
|
||||
// EffectiveTimeoutMS reports the per-evaluation timeout, defaulting to 5000.
|
||||
func (f StreamGateFilterPolicyConf) EffectiveTimeoutMS() int {
|
||||
if f.TimeoutMS == 0 {
|
||||
return DefaultStreamGateFilterTimeoutMS
|
||||
}
|
||||
return f.TimeoutMS
|
||||
}
|
||||
|
||||
// Validate checks a single filter policy for known kind/enforcement/selector
|
||||
// values and in-range numeric bounds.
|
||||
func (f StreamGateFilterPolicyConf) Validate() error {
|
||||
if _, ok := knownStreamGateFilterKinds[f.Filter]; !ok {
|
||||
return fmt.Errorf("stream_evidence_gate filter kind %q is not one of repeat_guard/schema_gate/provider_error", f.Filter)
|
||||
}
|
||||
if _, ok := knownStreamGateFilterEnforcements[f.EffectiveEnforcement()]; !ok {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q enforcement %q must be blocking or observe_only", f.Filter, f.Enforcement)
|
||||
}
|
||||
if strings.TrimSpace(f.EffectiveCapability()) == "" {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q capability must not be empty", f.Filter)
|
||||
}
|
||||
if runes := f.EffectiveHoldEvidenceRunes(); runes < 1 || runes > MaxStreamGateFilterHoldEvidenceRunes {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q hold_evidence_runes must be between 1 and %d, got %d", f.Filter, MaxStreamGateFilterHoldEvidenceRunes, runes)
|
||||
}
|
||||
if ms := f.EffectiveTimeoutMS(); ms < 1 || ms > MaxStreamGateFilterTimeoutMS {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q timeout_ms must be between 1 and %d, got %d", f.Filter, MaxStreamGateFilterTimeoutMS, ms)
|
||||
}
|
||||
if f.Priority < 0 {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q priority must be non-negative, got %d", f.Filter, f.Priority)
|
||||
}
|
||||
for i, sel := range f.Selectors {
|
||||
if _, ok := knownStreamGateFilterSelectorTypes[sel.Type]; !ok {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q selector %d type %q must be environment/model_group/model/provider", f.Filter, i, sel.Type)
|
||||
}
|
||||
if strings.TrimSpace(sel.Key) == "" {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q selector %d key must not be empty", f.Filter, i)
|
||||
}
|
||||
if strings.TrimSpace(sel.Enforcement) != "" {
|
||||
if _, ok := knownStreamGateFilterEnforcements[sel.Enforcement]; !ok {
|
||||
return fmt.Errorf("stream_evidence_gate filter %q selector %d enforcement %q must be blocking or observe_only", f.Filter, i, sel.Enforcement)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EffectiveEnvironment returns the deployment selector input, defaulting to
|
||||
// dev when omitted.
|
||||
func (s StreamEvidenceGateConf) EffectiveEnvironment() string {
|
||||
if strings.TrimSpace(s.Environment) == "" {
|
||||
return StreamGateEnvironmentDev
|
||||
}
|
||||
return s.Environment
|
||||
}
|
||||
|
||||
// EffectiveMaxRequestFaultRecovery returns the effective request-total fault recovery limit.
|
||||
|
|
@ -174,6 +370,10 @@ func (s StreamEvidenceGateConf) EffectiveMaxIngressSnapshotBytes() int {
|
|||
// Validate checks internal consistency and boundaries for StreamEvidenceGateConf.
|
||||
func (s StreamEvidenceGateConf) Validate() error {
|
||||
effTotal := s.EffectiveMaxRequestFaultRecovery()
|
||||
environment := s.EffectiveEnvironment()
|
||||
if environment != StreamGateEnvironmentDev && environment != StreamGateEnvironmentDevCorp {
|
||||
return fmt.Errorf("stream_evidence_gate environment %q must be dev or dev-corp", s.Environment)
|
||||
}
|
||||
if effTotal < 0 || effTotal > 3 {
|
||||
return fmt.Errorf("max_request_fault_recovery must be between 0 and 3, got %d", effTotal)
|
||||
}
|
||||
|
|
@ -192,6 +392,17 @@ func (s StreamEvidenceGateConf) Validate() error {
|
|||
return fmt.Errorf("max_ingress_snapshot_bytes must be between 1 and %d bytes (16 MiB), got %d", maxAllowedIngress, effIngress)
|
||||
}
|
||||
|
||||
seenFilterKinds := make(map[string]struct{}, len(s.Filters))
|
||||
for i, filter := range s.Filters {
|
||||
if err := filter.Validate(); err != nil {
|
||||
return fmt.Errorf("stream_evidence_gate filters[%d]: %w", i, err)
|
||||
}
|
||||
if _, dup := seenFilterKinds[filter.Filter]; dup {
|
||||
return fmt.Errorf("stream_evidence_gate filters[%d]: duplicate filter kind %q", i, filter.Filter)
|
||||
}
|
||||
seenFilterKinds[filter.Filter] = struct{}{}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -276,6 +276,131 @@ openai:
|
|||
}
|
||||
}
|
||||
|
||||
// TestStreamGateFilterPolicy_Defaults verifies the per-filter effective
|
||||
// defaults (enabled=true, enforcement=blocking, capability=output.<kind>,
|
||||
// hold=500, timeout=5000) that the runtime relies on when fields are omitted.
|
||||
func TestStreamGateFilterPolicy_Defaults(t *testing.T) {
|
||||
f := config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard}
|
||||
if !f.EffectiveEnabled() {
|
||||
t.Errorf("EffectiveEnabled() = false, want true by default")
|
||||
}
|
||||
if got := f.EffectiveEnforcement(); got != config.StreamGateFilterEnforcementBlocking {
|
||||
t.Errorf("EffectiveEnforcement() = %q, want blocking", got)
|
||||
}
|
||||
if got := f.EffectiveCapability(); got != "output.repeat_guard" {
|
||||
t.Errorf("EffectiveCapability() = %q, want output.repeat_guard", got)
|
||||
}
|
||||
if got := f.EffectiveHoldEvidenceRunes(); got != config.DefaultStreamGateFilterHoldEvidenceRunes {
|
||||
t.Errorf("EffectiveHoldEvidenceRunes() = %d, want %d", got, config.DefaultStreamGateFilterHoldEvidenceRunes)
|
||||
}
|
||||
if got := f.EffectiveTimeoutMS(); got != config.DefaultStreamGateFilterTimeoutMS {
|
||||
t.Errorf("EffectiveTimeoutMS() = %d, want %d", got, config.DefaultStreamGateFilterTimeoutMS)
|
||||
}
|
||||
if err := f.Validate(); err != nil {
|
||||
t.Errorf("Validate() unexpectedly failed on defaulted filter: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestStreamGateFilterPolicy_ValidateTable covers the config validation contract
|
||||
// for the semantic output filters: known kinds, enforcement values, selector
|
||||
// types/keys, and numeric bounds (S01/S02 config).
|
||||
func TestStreamGateFilterPolicy_ValidateTable(t *testing.T) {
|
||||
boolPtr := func(b bool) *bool { return &b }
|
||||
tests := []struct {
|
||||
name string
|
||||
filter config.StreamGateFilterPolicyConf
|
||||
expectErr bool
|
||||
}{
|
||||
{name: "valid repeat guard", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard}},
|
||||
{name: "valid schema gate observe_only", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterSchemaGate, Enforcement: config.StreamGateFilterEnforcementObserveOnly}},
|
||||
{name: "valid provider error with selectors", filter: config.StreamGateFilterPolicyConf{
|
||||
Filter: config.StreamGateFilterProviderError,
|
||||
Priority: 20,
|
||||
Selectors: []config.StreamGateFilterSelectorConf{
|
||||
{Type: config.StreamGateFilterSelectorProvider, Key: "prov-a", Enabled: boolPtr(false)},
|
||||
{Type: config.StreamGateFilterSelectorModel, Key: "ornith:35b", Enforcement: config.StreamGateFilterEnforcementObserveOnly},
|
||||
},
|
||||
}},
|
||||
{name: "unknown kind", filter: config.StreamGateFilterPolicyConf{Filter: "malformed_guard"}, expectErr: true},
|
||||
{name: "unknown enforcement", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, Enforcement: "warn"}, expectErr: true},
|
||||
{name: "hold runes over max", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, HoldEvidenceRunes: config.MaxStreamGateFilterHoldEvidenceRunes + 1}, expectErr: true},
|
||||
{name: "negative hold runes", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, HoldEvidenceRunes: -1}, expectErr: true},
|
||||
{name: "timeout over max", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, TimeoutMS: config.MaxStreamGateFilterTimeoutMS + 1}, expectErr: true},
|
||||
{name: "negative priority", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, Priority: -1}, expectErr: true},
|
||||
{name: "unknown selector type", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, Selectors: []config.StreamGateFilterSelectorConf{{Type: "region", Key: "kr"}}}, expectErr: true},
|
||||
{name: "empty selector key", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, Selectors: []config.StreamGateFilterSelectorConf{{Type: config.StreamGateFilterSelectorModel, Key: " "}}}, expectErr: true},
|
||||
{name: "unknown selector enforcement", filter: config.StreamGateFilterPolicyConf{Filter: config.StreamGateFilterRepeatGuard, Selectors: []config.StreamGateFilterSelectorConf{{Type: config.StreamGateFilterSelectorModel, Key: "m", Enforcement: "warn"}}}, expectErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := tt.filter.Validate()
|
||||
if tt.expectErr && err == nil {
|
||||
t.Fatalf("Validate() succeeded, want error")
|
||||
}
|
||||
if !tt.expectErr && err != nil {
|
||||
t.Fatalf("Validate() failed: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestStreamGateFilterPolicy_LoadAndDuplicateRejection loads a full filter policy
|
||||
// from YAML and confirms a duplicate filter kind is rejected at load time.
|
||||
func TestStreamGateFilterPolicy_LoadAndDuplicateRejection(t *testing.T) {
|
||||
valid := `
|
||||
openai:
|
||||
enabled: true
|
||||
stream_evidence_gate:
|
||||
enabled: true
|
||||
filters:
|
||||
- filter: repeat_guard
|
||||
enforcement: observe_only
|
||||
capability: output.repeat_guard
|
||||
- filter: provider_error
|
||||
priority: 20
|
||||
selectors:
|
||||
- type: provider
|
||||
key: prov-a
|
||||
enabled: false
|
||||
`
|
||||
tmpDir := t.TempDir()
|
||||
tmpFile := filepath.Join(tmpDir, "edge.yaml")
|
||||
if err := os.WriteFile(tmpFile, []byte(valid), 0644); err != nil {
|
||||
t.Fatalf("write yaml: %v", err)
|
||||
}
|
||||
cfg, err := config.LoadEdge(tmpFile)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadEdge valid filters failed: %v", err)
|
||||
}
|
||||
filters := cfg.OpenAI.StreamEvidenceGate.Filters
|
||||
if len(filters) != 2 {
|
||||
t.Fatalf("loaded %d filters, want 2", len(filters))
|
||||
}
|
||||
if filters[0].Filter != config.StreamGateFilterRepeatGuard || filters[0].EffectiveEnforcement() != config.StreamGateFilterEnforcementObserveOnly {
|
||||
t.Errorf("filter[0] = %+v", filters[0])
|
||||
}
|
||||
if len(filters[1].Selectors) != 1 || filters[1].Selectors[0].Type != config.StreamGateFilterSelectorProvider {
|
||||
t.Errorf("filter[1] selectors = %+v", filters[1].Selectors)
|
||||
}
|
||||
|
||||
dup := `
|
||||
openai:
|
||||
enabled: true
|
||||
stream_evidence_gate:
|
||||
enabled: true
|
||||
filters:
|
||||
- filter: repeat_guard
|
||||
- filter: repeat_guard
|
||||
`
|
||||
dupFile := filepath.Join(tmpDir, "dup.yaml")
|
||||
if err := os.WriteFile(dupFile, []byte(dup), 0644); err != nil {
|
||||
t.Fatalf("write dup yaml: %v", err)
|
||||
}
|
||||
if _, err := config.LoadEdge(dupFile); err == nil {
|
||||
t.Fatalf("LoadEdge with duplicate filter kind succeeded, want error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamEvidenceGate_IngressSnapshotBytes_TableFixture(t *testing.T) {
|
||||
maxAllowed := 16 * 1024 * 1024 // 16777216
|
||||
|
||||
|
|
@ -347,3 +472,19 @@ openai:
|
|||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStreamGateFilterPolicy_Environment(t *testing.T) {
|
||||
var defaults config.StreamEvidenceGateConf
|
||||
if got := defaults.EffectiveEnvironment(); got != config.StreamGateEnvironmentDev {
|
||||
t.Fatalf("default environment=%q, want dev", got)
|
||||
}
|
||||
for _, environment := range []string{config.StreamGateEnvironmentDev, config.StreamGateEnvironmentDevCorp} {
|
||||
cfg := config.StreamEvidenceGateConf{Environment: environment}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Errorf("Validate(%q): %v", environment, err)
|
||||
}
|
||||
}
|
||||
if err := (config.StreamEvidenceGateConf{Environment: "production"}).Validate(); err == nil {
|
||||
t.Fatal("Validate(production) succeeded, want closed environment error")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1239,7 +1239,34 @@ func (r *RequestRuntime) Run(ctx context.Context) error {
|
|||
case ArbitrationActionTerminal:
|
||||
useUnbuffered := passThrough || !r.tail.HasUnconfirmedEvents(epoch.ID())
|
||||
var termCauses FailureCauseChain
|
||||
if arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
if arbResult.FilterID() != "" {
|
||||
desc, errD := NewExternalDescriptor("error", "fatal_violation", "fatal_filter_violation", "")
|
||||
if errD != nil {
|
||||
return errD
|
||||
}
|
||||
cause, errC := NewFailureCause("arbiter", "fatal_violation", "", arbResult.FilterID(), arbResult.RuleID())
|
||||
if errC != nil {
|
||||
return errC
|
||||
}
|
||||
causes, errCh := NewFailureCauseChain([]FailureCause{cause})
|
||||
if errCh != nil {
|
||||
return errCh
|
||||
}
|
||||
termCauses = causes
|
||||
termRes, errTerm := NewErrorTerminalResult(ev.Channel(), desc, causes, time.Now())
|
||||
if errTerm != nil {
|
||||
return errTerm
|
||||
}
|
||||
if useUnbuffered {
|
||||
if errRel := r.releaser.CommitTerminalWithoutEpoch(ctx, binding.AttemptID(), termRes); errRel != nil {
|
||||
return errRel
|
||||
}
|
||||
} else {
|
||||
if _, errRel := r.releaser.FailPending(ctx, binding.AttemptID(), termRes); errRel != nil {
|
||||
return errRel
|
||||
}
|
||||
}
|
||||
} else if arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
termRes, errRes := ev.AsTerminal()
|
||||
if errRes != nil {
|
||||
return errRes
|
||||
|
|
@ -1273,41 +1300,19 @@ func (r *RequestRuntime) Run(ctx context.Context) error {
|
|||
}
|
||||
}
|
||||
} else {
|
||||
desc, errD := NewExternalDescriptor("error", "fatal_violation", "fatal_filter_violation", "")
|
||||
if errD != nil {
|
||||
return errD
|
||||
}
|
||||
cause, errC := NewFailureCause("arbiter", "fatal_violation", "", arbResult.FilterID(), arbResult.RuleID())
|
||||
if errC != nil {
|
||||
return errC
|
||||
}
|
||||
causes, errCh := NewFailureCauseChain([]FailureCause{cause})
|
||||
if errCh != nil {
|
||||
return errCh
|
||||
}
|
||||
termCauses = causes
|
||||
termRes, errTerm := NewErrorTerminalResult(ev.Channel(), desc, causes, time.Now())
|
||||
if errTerm != nil {
|
||||
return errTerm
|
||||
}
|
||||
if useUnbuffered {
|
||||
if errRel := r.releaser.CommitTerminalWithoutEpoch(ctx, binding.AttemptID(), termRes); errRel != nil {
|
||||
return errRel
|
||||
}
|
||||
} else {
|
||||
if _, errRel := r.releaser.FailPending(ctx, binding.AttemptID(), termRes); errRel != nil {
|
||||
return errRel
|
||||
}
|
||||
}
|
||||
return errors.New("streamgate: terminal arbitration has no terminal disposition or blocking filter")
|
||||
}
|
||||
if arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
if arbResult.FilterID() == "" &&
|
||||
arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
r.emitObservation(ObservationKindReleaseCommitted, epoch.ID(), func(in *FilterObservationInput) {
|
||||
in.CommitState = CommitStateStreamOpen
|
||||
})
|
||||
}
|
||||
r.emitObservation(ObservationKindTerminalCommitted, epoch.ID(), func(in *FilterObservationInput) {
|
||||
in.CommitState = CommitStateTerminalCommitted
|
||||
if arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
if arbResult.FilterID() != "" {
|
||||
in.Causes = termCauses
|
||||
} else if arbResult.BaseDisposition() == BaseDispositionTerminalSuccessCandidate {
|
||||
in.TerminalReason = TerminalReasonCompleted
|
||||
} else {
|
||||
in.Causes = termCauses
|
||||
|
|
|
|||
|
|
@ -1504,6 +1504,74 @@ func TestRequestRuntimeTerminalFailureFidelity(t *testing.T) {
|
|||
t.Fatalf("fatal causes = %+v", causes)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("fatal_filter_overrides_success_terminal", func(t *testing.T) {
|
||||
sink := newFixtureSink()
|
||||
observationSink := &testObservationRecordingSink{}
|
||||
hold, err := NewFilterHoldRequirementTerminalGate(
|
||||
"default",
|
||||
[]EventKind{EventKindTerminal},
|
||||
EventKindTerminal,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFilterHoldRequirementTerminalGate: %v", err)
|
||||
}
|
||||
filter := &customMockFilter{
|
||||
id: "terminal-fatal-filter",
|
||||
holdReq: &hold,
|
||||
evaluateFn: func(ctx context.Context, fc FilterContext, batch EvidenceBatch) (FilterDecision, error) {
|
||||
evidence := mustSanitizedEvidence(EventKindTerminal, "default", "terminal-fatal-rule", "terminal_fatal_violation", FixedFingerprint{3}, 1, 0, FilterOutcomeKindEvaluated, testNow)
|
||||
return NewFilterDecision(FilterDecisionKindFatal, "consumer1", "terminal-fatal-filter", "terminal-fatal-rule", evidence, nil)
|
||||
},
|
||||
}
|
||||
reg := mustFilterRegistration(filter, "cap1", true, FilterEnforcementBlocking, 5*time.Second, 10)
|
||||
snapshot := createTestRuntimeSnapshot(t, []FilterRegistration{reg}, &fixtureDispatcher{}, &fixtureRebuilder{}, sink).
|
||||
WithObservationSink(observationSink)
|
||||
initial := mustAttemptBinding(t, "att-1", newSliceEventSource([]NormalizedEvent{
|
||||
mustTerminal("default", testNow),
|
||||
}), &fixtureController{})
|
||||
runtime, err := NewRequestRuntime(snapshot, "group-a", initial)
|
||||
if err != nil {
|
||||
t.Fatalf("NewRequestRuntime: %v", err)
|
||||
}
|
||||
if err := runtime.Run(context.Background()); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if len(sink.terminals) != 1 || sink.terminals[0].Success() {
|
||||
t.Fatalf("terminal-filter fatal result = %+v, want one error terminal", sink.terminals)
|
||||
}
|
||||
causes := sink.terminals[0].FailureCauses().All()
|
||||
if len(causes) != 1 ||
|
||||
causes[0].Filter() != "terminal-fatal-filter" ||
|
||||
causes[0].RuleID() != "terminal-fatal-rule" {
|
||||
t.Fatalf("terminal-filter fatal causes = %+v", causes)
|
||||
}
|
||||
var terminalObservations []FilterObservation
|
||||
for _, observation := range observationSink.observations() {
|
||||
if observation.Kind() == ObservationKindTerminalCommitted {
|
||||
terminalObservations = append(terminalObservations, observation)
|
||||
}
|
||||
}
|
||||
if len(terminalObservations) != 1 {
|
||||
t.Fatalf("terminal observations = %+v, want exactly one", terminalObservations)
|
||||
}
|
||||
terminalObservation := terminalObservations[0]
|
||||
if terminalObservation.TerminalReason() != "" ||
|
||||
terminalObservation.CommitState() != CommitStateTerminalCommitted {
|
||||
t.Fatalf(
|
||||
"terminal observation reason/state = %q/%q, want empty/%q",
|
||||
terminalObservation.TerminalReason(),
|
||||
terminalObservation.CommitState(),
|
||||
CommitStateTerminalCommitted,
|
||||
)
|
||||
}
|
||||
observationCauses := terminalObservation.Causes().All()
|
||||
if len(observationCauses) != 1 ||
|
||||
observationCauses[0].Filter() != "terminal-fatal-filter" ||
|
||||
observationCauses[0].RuleID() != "terminal-fatal-rule" {
|
||||
t.Fatalf("terminal observation causes = %+v", observationCauses)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestRequestRuntimeAtomicInstallFailure(t *testing.T) {
|
||||
|
|
|
|||
Loading…
Reference in a new issue