test(dev): tool-call smoke 검증 기준을 갱신한다
This commit is contained in:
parent
c747c395cf
commit
99fcd19b06
3 changed files with 51 additions and 9 deletions
|
|
@ -65,7 +65,7 @@ dev-runtime provider pool과 3-node 연결 상태를 점검할 때는 `agent-tes
|
|||
- capacity baseline: `2`
|
||||
- priority baseline: `2`
|
||||
- workdir: `/Users/toki/agent-work/iop-mlx-vllm`
|
||||
- runtime baseline: `vllm-mlx`, `--max-num-seqs 2`, `--max-kv-size 262144`, `--max-request-tokens 262144`, `--use-paged-cache --paged-cache-block-size 64 --max-cache-blocks 4096`, `--tool-call-parser qwen`, `--reasoning-parser qwen3`, `default_chat_template_kwargs.enable_thinking=true`
|
||||
- runtime baseline: `vllm-mlx`, `--max-num-seqs 2`, `--max-kv-size 262144`, `--max-request-tokens 262144`, `--use-paged-cache --paged-cache-block-size 64 --max-cache-blocks 4096`, `--enable-auto-tool-choice --tool-call-parser qwen --reasoning-parser qwen3`, `--default-chat-template-kwargs {"enable_thinking": true}`
|
||||
- KV policy: per-call window bound `262144`, one full context-size setting
|
||||
- long-context admission baseline: `total_context_tokens=262144`, `long_context_capacity=1` (KV budget `262144` fits one full 262144-window request; long slot count is not the normal capacity `2`)
|
||||
- GX10 vLLM node: `gx10-vllm-node` / `gx10-vllm`
|
||||
|
|
@ -74,8 +74,7 @@ dev-runtime provider pool과 3-node 연결 상태를 점검할 때는 `agent-tes
|
|||
- served model: `nvidia/Qwen3.6-35B-A3B-NVFP4`
|
||||
- capacity baseline: `4`
|
||||
- priority baseline: `0`
|
||||
- runtime baseline: `vllm`, `--max-model-len 262144`, `--max-num-seqs 4`, `--gpu-memory-utilization 0.30`, `--reasoning-parser qwen3`, `default_chat_template_kwargs.enable_thinking=true`
|
||||
- agent/tool-call note: tracked GX10 runtime evidence has not yet fixed a Qwen tool-call parser/template option; before using this provider for Pi/Cline-style agent tools, validate `tools[]` non-stream/stream responses and then record the chosen `--tool-call-parser`/`--chat-template` values in `agent-test/dev/inventory.yaml`.
|
||||
- runtime baseline: `vllm`, `--max-model-len 262144`, `--max-num-seqs 4`, `--gpu-memory-utilization 0.30`, `--reasoning-parser qwen3`, `--default-chat-template-kwargs {"enable_thinking":true}`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`
|
||||
- long-context admission baseline: `total_context_tokens=262144`, `long_context_capacity=1` (conservative; `max-model-len 262144` until runtime evidence proves a larger concurrent long-context budget)
|
||||
- workspace: `/home/toki/iop-gx10-vllm`
|
||||
- OneXPlayer Lemonade node: `onexplayer-lemonade-node` / `onexplayer-lemonade`
|
||||
|
|
@ -111,6 +110,7 @@ mac-mlx-vllm provider는 mac-codex-node 소속 resource로, Edge host와 같은
|
|||
- registry, service, transport, console, HTTP/A2A 입력 표면을 바꾼 경우 edge-node 메시지 2회 왕복과 command 응답을 확인한다.
|
||||
- OpenAI-compatible 경계를 바꾼 경우 dev `18083` 기준 `iop-edge smoke openai` 또는 동등한 `/healthz`, `/v1/models`, `/v1/responses` 확인으로 edge service와 node adapter 경로 수렴을 확인한다.
|
||||
- OpenAI-compatible tool-call 경계(`apps/edge/internal/openai/**`의 text tool-call 합성/validation)를 바꾼 경우 dev `18083` `/v1/chat/completions`에 Pi/Cline형 `tools[]` 요청을 non-stream/stream 각각 최소 1회 보내 raw text tool-call boundary smoke를 수행한다. token은 원격 환경 변수에서 주입하고 원문을 명령/로그/보고에 남기지 않는다.
|
||||
- dev provider-pool tool-call drift를 확인할 때는 `gx10-vllm`, `mac-mlx-vllm`, `onexplayer-lemonade` direct endpoint 각각에 동일한 `tools[]` + `tool_choice:"auto"` Chat Completions 요청을 보내 200과 `tool_calls` 구조를 확인한 뒤, Edge `18083`에도 10개 동시 요청을 보내 10/10 성공, 10/10 `tool_calls`, provider peak `gx10-vllm=4`, `onexplayer-lemonade=3`, `mac-mlx-vllm=2` 사용을 확인한다.
|
||||
- bootstrap/artifact 경계를 바꾼 경우 dev artifact/base URL 후보 `18082`가 local/test `18080` field baseline을 덮어쓰지 않는지 확인한다.
|
||||
|
||||
## 보조 검증
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
test_env: dev
|
||||
profile: dev-runtime-provider-pool
|
||||
last_updated_at: "2026-07-04"
|
||||
last_updated_at: "2026-07-08"
|
||||
|
||||
source:
|
||||
remote_runner:
|
||||
|
|
@ -33,7 +33,7 @@ build:
|
|||
model:
|
||||
alias: qwen3.6:35b
|
||||
provider_capacity_total: 9
|
||||
provider_capacity_status: verified_with_control_plane_provider_snapshots_2026_07_04
|
||||
provider_capacity_status: verified_with_control_plane_provider_snapshots_2026_07_08
|
||||
context_window: 262144
|
||||
default_max_tokens: 32768
|
||||
min_max_tokens: 32768
|
||||
|
|
@ -74,7 +74,7 @@ model:
|
|||
prompt_policy: long_reasoning_allowed
|
||||
exact_output_match: false
|
||||
latest_provider_snapshot:
|
||||
observed_at: "2026-07-04"
|
||||
observed_at: "2026-07-08"
|
||||
total_capacity: 9
|
||||
all_nodes_connected: true
|
||||
providers:
|
||||
|
|
@ -99,6 +99,45 @@ model:
|
|||
queued: 0
|
||||
health: healthy
|
||||
served_model: Qwen3.6-35B-A3B-MTP-GGUF
|
||||
openai_tool_call_smoke:
|
||||
observed_at: "2026-07-08"
|
||||
request_shape: tools_plus_tool_choice_auto
|
||||
direct_provider_expected:
|
||||
http_status: 200
|
||||
minimum_tool_calls: 1
|
||||
direct_provider_result:
|
||||
gx10-vllm:
|
||||
http_status: 200
|
||||
tool_calls: 1
|
||||
mac-mlx-vllm:
|
||||
http_status: 200
|
||||
tool_calls: 1
|
||||
onexplayer-lemonade:
|
||||
http_status: 200
|
||||
tool_calls: 1
|
||||
edge_concurrency_result:
|
||||
endpoint: /v1/chat/completions
|
||||
concurrent_requests: 10
|
||||
success_count: 10
|
||||
failure_count: 0
|
||||
tool_call_response_count: 10
|
||||
finish_reason_tool_calls: 10
|
||||
peak_total_in_flight: 9
|
||||
peak_total_queued: 3
|
||||
provider_result_counts:
|
||||
gx10-vllm: 5
|
||||
onexplayer-lemonade: 3
|
||||
mac-mlx-vllm: 2
|
||||
provider_peaks:
|
||||
gx10-vllm:
|
||||
max_in_flight: 4
|
||||
max_queued: 1
|
||||
onexplayer-lemonade:
|
||||
max_in_flight: 3
|
||||
max_queued: 1
|
||||
mac-mlx-vllm:
|
||||
max_in_flight: 2
|
||||
max_queued: 1
|
||||
final_content_smoke:
|
||||
endpoint: /v1/chat/completions
|
||||
concurrent_requests: 10
|
||||
|
|
@ -188,6 +227,7 @@ nodes:
|
|||
paged_cache_block_size: 64
|
||||
max_cache_blocks: 4096
|
||||
total_kv_tokens: 262144
|
||||
enable_auto_tool_choice: true
|
||||
tool_call_parser: qwen
|
||||
reasoning_parser: qwen3
|
||||
default_chat_template_kwargs:
|
||||
|
|
@ -217,6 +257,8 @@ nodes:
|
|||
gpu_memory_utilization: 0.30
|
||||
default_max_tokens: 32768
|
||||
default_thinking_token_budget: 1024
|
||||
enable_auto_tool_choice: true
|
||||
tool_call_parser: qwen3_xml
|
||||
reasoning_parser: qwen3
|
||||
default_chat_template_kwargs:
|
||||
enable_thinking: true
|
||||
|
|
@ -250,4 +292,5 @@ nodes:
|
|||
mtp_runtime_policy: disabled
|
||||
default_max_tokens: 32768
|
||||
default_thinking_token_budget: 1024
|
||||
tool_choice_auto_verified: true
|
||||
windows_process_start: Win32_Process.Create
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ dev-runtime의 실제 3-node 연결을 점검할 때는 원격 runner `ssh toki@
|
|||
- capacity baseline: `2`
|
||||
- priority baseline: `2`
|
||||
- workdir: `/Users/toki/agent-work/iop-mlx-vllm`
|
||||
- runtime baseline: `vllm-mlx`, `--max-num-seqs 2`, `--max-kv-size 262144`, `--max-request-tokens 262144`, `--use-paged-cache --paged-cache-block-size 64 --max-cache-blocks 4096`, `--tool-call-parser qwen`, `--reasoning-parser qwen3`, `default_chat_template_kwargs.enable_thinking=true`
|
||||
- runtime baseline: `vllm-mlx`, `--max-num-seqs 2`, `--max-kv-size 262144`, `--max-request-tokens 262144`, `--use-paged-cache --paged-cache-block-size 64 --max-cache-blocks 4096`, `--enable-auto-tool-choice --tool-call-parser qwen --reasoning-parser qwen3`, `--default-chat-template-kwargs {"enable_thinking": true}`
|
||||
- KV policy: per-call window bound `262144`, one full context-size setting
|
||||
- long-context admission baseline: `total_context_tokens=262144`, `long_context_capacity=1` (KV budget `262144` fits one full 262144-window request; long slot count is not the normal capacity `2`)
|
||||
- GX10 vLLM node: `gx10-vllm-node` / `gx10-vllm`
|
||||
|
|
@ -63,8 +63,7 @@ dev-runtime의 실제 3-node 연결을 점검할 때는 원격 runner `ssh toki@
|
|||
- served model: `nvidia/Qwen3.6-35B-A3B-NVFP4`
|
||||
- capacity baseline: `4`
|
||||
- priority baseline: `0`
|
||||
- runtime baseline: `vllm`, `--max-model-len 262144`, `--max-num-seqs 4`, `--gpu-memory-utilization 0.30`, `--reasoning-parser qwen3`, `default_chat_template_kwargs.enable_thinking=true`
|
||||
- agent/tool-call note: tracked GX10 runtime evidence has not yet fixed a Qwen tool-call parser/template option; before using this provider for Pi/Cline-style agent tools, validate `tools[]` non-stream/stream responses and then record the chosen `--tool-call-parser`/`--chat-template` values in `agent-test/dev/inventory.yaml`.
|
||||
- runtime baseline: `vllm`, `--max-model-len 262144`, `--max-num-seqs 4`, `--gpu-memory-utilization 0.30`, `--reasoning-parser qwen3`, `--default-chat-template-kwargs {"enable_thinking":true}`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`
|
||||
- long-context admission baseline: `total_context_tokens=262144`, `long_context_capacity=1` (conservative; `max-model-len 262144` until runtime evidence proves a larger concurrent long-context budget)
|
||||
- workspace: `/home/toki/iop-gx10-vllm`
|
||||
- OneXPlayer Lemonade node: `onexplayer-lemonade-node` / `onexplayer-lemonade`
|
||||
|
|
|
|||
Loading…
Reference in a new issue