toki/iop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
iop/agent-test/dev-corp/mac-studio-ornith-think-on-baseline-20260713.md

12 KiB
Raw Blame History

Mac Studio Ornith Think-On Baseline

측정 목적: Mac Studio Ornith runtime의 think off 실험 전 기준 성능을 보존한다.

Runtime Identity

  • 측정 시각: 2026-07-13 17:47:59 KST
  • Host: dc-devui-MacStudio.local
  • Endpoint: http://127.0.0.1:8007/v1
  • Model alias: ornith:35b
  • Current runtime path: ornith35b_alias_proxy.py on 0.0.0.0:8007 -> mlx_lm.server on 127.0.0.1:8008
  • Upstream model path: /Users/dc_dev/.cache/huggingface/manual/mlx-community/Ornith-1.0-35B-8bit
  • Runtime note: the active vllm_mlx.cli serve process on this host is currently Gemma4 on 8004; this Ornith baseline uses the current Mac Studio Ornith alias path above.
  • Thinking mode: current default. Upstream mlx_lm.server was started with --chat-template-args '{"enable_thinking":true}'; benchmark requests did not pass think=false.
  • API shape: OpenAI-compatible streaming /v1/chat/completions
  • Token counting: Qwen2Tokenizer over concatenated streaming reasoning, reasoning_content, and content deltas.

Request Parameters

  • max_tokens: 256
  • temperature: 0.2
  • top_p: 0.95
  • Warmup: one short warmup request, excluded from the table.
  • Prompt: Benchmark generation task. Do not use tools. Produce a compact Korean report with exactly 24 numbered lines about reliable agent runtime operations. Keep each line short, but continue until the list is complete.

Metric Definitions

  • avg_ttft_s: average seconds from request start to first non-empty streaming delta. This includes reasoning deltas and is not the user-visible content latency.
  • avg_per_call_tok_s: average per-call decode throughput, computed as output tokens divided by time from first non-empty delta to completion.
  • total_tok_s: aggregate group throughput, computed as total output tokens divided by wall-clock time from earliest request start to latest completion.
  • group_wall_s: wall-clock duration for the concurrency group.

Content-First Baseline

This corrected measurement records when the first non-empty content delta appears. It is the user-visible answer latency requested for the think-off comparison.

  • 측정 시각: 2026-07-13 17:55:18 KST
  • Endpoint: http://127.0.0.1:8007/v1
  • Model alias: ornith:35b
  • Thinking mode: current default. Upstream mlx_lm.server was started with --chat-template-args '{"enable_thinking":true}'; benchmark requests did not pass think=false.
  • max_tokens: 1024
  • temperature: 0.2
  • top_p: 0.95
  • Prompt: Benchmark generation task. Produce exactly 8 short numbered Korean lines about reliable agent runtime operations.
Concurrent calls Success Avg first reasoning/any delta (s) Avg first content delta (s) Avg reasoning tokens/call Avg content tokens/call Avg all tok/s per call Avg content tok/s per call Total all tok/s Total content tok/s Group wall (s) Finish reason
1 1/1 0.227 17.830 919.0 104.0 52.29 53.00 51.69 5.25 19.792 length
2 2/2 0.339 22.193 978.0 45.0 44.74 44.44 88.17 3.88 23.206 length
3 3/3 0.462 26.260 978.0 45.0 37.93 38.26 111.85 4.92 27.439 length
4 4/4 0.598 30.638 978.0 45.0 32.55 32.34 127.74 5.62 32.035 length
5 5/5 0.728 38.788 978.0 45.0 25.69 25.64 126.13 5.55 40.553 length

Content-First Per-Call Detail

Concurrent calls Call First content delta (s) Reasoning tokens Content tokens All tok/s Content tok/s Duration (s)
1 1 17.830 919 104 52.29 53.00 19.792
2 1 22.195 978 45 44.74 44.49 23.206
2 2 22.191 978 45 44.73 44.38 23.205
3 1 26.258 978 45 37.93 38.33 27.432
3 2 26.263 978 45 37.93 38.29 27.439
3 3 26.258 978 45 37.92 38.15 27.437
4 1 30.637 978 45 32.55 32.47 32.023
4 2 30.637 978 45 32.54 32.22 32.034
4 3 30.642 978 45 32.55 32.31 32.035
4 4 30.637 978 45 32.55 32.34 32.028
5 1 38.787 978 45 25.69 25.50 40.551
5 2 38.787 978 45 25.70 25.70 40.538
5 3 38.787 978 45 25.69 25.61 40.544
5 4 38.793 978 45 25.69 25.58 40.552
5 5 38.788 978 45 25.70 25.81 40.531

Content-First Observations

  • The user-visible first content delta was much later than the first reasoning delta.
  • At concurrency 1, first reasoning/any delta arrived at 0.227s, but first content arrived at 17.830s.
  • At concurrency 5, first reasoning/any delta arrived at 0.728s, but first content arrived at 38.788s.
  • The run consistently spent most of the 1024 token budget on reasoning before content. Most calls produced about 978 reasoning tokens and 45 content tokens before finish_reason=length.
  • For think-off comparison, Avg first content delta (s) is the latency column to compare first.

Think-Off Result

Compatibility Note

The alias proxy currently maps request-level think:false to upstream chat_template_args.enable_thinking=false. The installed mlx_lm.server request parser reads chat_template_kwargs, not request-level chat_template_args (mlx_lm/server.py request parse path stores self.chat_template_kwargs = self.body.get("chat_template_kwargs")). Because of that mismatch, plain think:false did not produce a real think-off result in this runtime path.

Observed think:false compatibility probe:

Mode First content delta (s) Reasoning chars Content chars Finish reason Interpretation
think:false via alias proxy 17.975 3300 203 length Not effective; still generated reasoning first

The effective think-off measurement below used request chat_template_kwargs: {"enable_thinking": false}.

Effective Think-Off Content-First Table

  • 측정 시각: 2026-07-13 18:05:09 KST
  • Endpoint: http://127.0.0.1:8007/v1
  • Model alias: ornith:35b
  • Thinking mode: effective request-level think-off via chat_template_kwargs.enable_thinking=false
  • max_tokens: 1024
  • temperature: 0.2
  • top_p: 0.95
  • Prompt: same as the Content-First Baseline.
Concurrent calls Success Avg first content delta (s) Avg reasoning tokens/call Avg content tokens/call Avg content tok/s per call Total content tok/s Group wall (s) Finish reason
1 1/1 0.208 0.0 104.0 52.77 47.74 2.179 stop
2 2/2 0.333 0.0 113.0 44.81 79.15 2.855 stop
3 3/3 0.457 0.0 113.0 37.79 98.27 3.450 stop
4 4/4 0.580 0.0 113.0 32.47 111.20 4.065 stop
5 5/5 0.703 0.0 113.0 25.82 111.03 5.089 stop

Think-On vs Effective Think-Off

Concurrent calls Think-on first content (s) Effective think-off first content (s) Delta (s) Speedup
1 17.830 0.208 17.622 85.9x
2 22.193 0.333 21.860 66.7x
3 26.260 0.457 25.803 57.5x
4 30.638 0.580 30.058 52.9x
5 38.788 0.703 38.085 55.2x

Effective Think-Off Observations

  • Effective think-off eliminated reasoning output in this benchmark (0 reasoning tokens for all calls).
  • First user-visible content became the first non-empty stream delta.
  • Total content throughput peaked around concurrency 4-5 at about 111 tok/s.
  • The largest improvement is first content latency, not single-stream decode tok/s.
  • To make think:false work through the alias endpoint, update the proxy to map think to chat_template_kwargs.enable_thinking, not only chat_template_args.enable_thinking.

First-Content Optimization Candidates

Priority Candidate Expected effect Risk / note
1 Fix alias proxy think mapping to upstream chat_template_kwargs.enable_thinking Makes simple think:false clients get the measured 0.2-0.7s first-content path Very small code change, but verify /v1/chat/completions streaming and existing chat_template_kwargs passthrough
2 Start a separate always-non-thinking Ornith endpoint with mlx_lm.server --chat-template-args '{"enable_thinking":false}' Makes non-thinking the default without per-request kwargs Keep on a separate port first; quality/tool-call behavior may differ
3 Use vllm_mlx Ornith path if stable, with --default-chat-template-kwargs '{"enable_thinking": false}' or request enable_thinking=false Better OpenAI-compatible semantics, continuous batching, prefix cache, warm prompts, and request-level thinking controls This changes runtime stack from current mlx_lm.server; test as a parallel endpoint before replacing 8007
4 If reasoning must stay on, enforce a small thinking budget where supported Bounds reasoning delay before content instead of fully disabling reasoning Current mlx_lm.server path has no observed thinking budget control; vllm_mlx/vLLM-style serving has this concept
5 Warm repeated prompt prefixes / prompt cache Reduces prefill/TTFT for repeated system prompts and agent templates Helps prefill, not the long reasoning-before-content delay. Think-off gives the bigger win here
6 Tune decode-concurrency, prompt-concurrency, and prefill-step-size on mlx_lm.server May improve batch scheduling and prefill behavior Current content-first bottleneck was reasoning generation. Tune only after think-off route is correct

References checked:

  • Local mlx_lm.server --help: supports --chat-template-args, --decode-concurrency, --prompt-concurrency, --prefill-step-size, --prompt-cache-size, --prompt-cache-bytes, and draft-model options.
  • Local installed mlx_lm/server.py: request body uses chat_template_kwargs; startup CLI uses chat_template_args.
  • Qwen docs: previous Qwen3-style hybrid thinking can be controlled by enable_thinking=False; newer Qwen3 Instruct/Thinking variants may be mode-specific.
  • vLLM docs: request-level chat_template_kwargs can override server defaults, and thinking budgets can bound reasoning tokens.
  • vllm-mlx docs: continuous batching, prefix cache, SSD-tiered cache, and warm prompts are intended Apple Silicon serving optimizations.

Reasoning-Inclusive First-Delta Table

Concurrent calls Success Avg output tokens/call Total output tokens Avg TTFT (s) Avg duration (s) Avg decode duration (s) Avg tok/s per call Total tok/s Group wall (s) Finish reason
1 1/1 256.0 256 0.221 5.085 4.864 52.63 50.34 5.085 length
2 2/2 256.0 512 0.340 6.072 5.732 44.66 84.31 6.073 length
3 3/3 256.0 768 0.462 7.213 6.751 37.92 106.42 7.217 length
4 4/4 256.0 1024 0.570 8.445 7.875 32.51 121.17 8.451 length
5 5/5 256.0 1280 0.698 10.639 9.940 25.75 120.20 10.649 length

Per-Call Detail

Concurrent calls Call Output tokens TTFT (s) Duration (s) Decode duration (s) Tok/s
1 1 256 0.221 5.085 4.864 52.63
2 1 256 0.339 6.072 5.734 44.65
2 2 256 0.342 6.072 5.730 44.68
3 1 256 0.465 7.216 6.751 37.92
3 2 256 0.461 7.216 6.755 37.90
3 3 256 0.460 7.208 6.748 37.94
4 1 256 0.569 8.443 7.874 32.51
4 2 256 0.569 8.438 7.869 32.53
4 3 256 0.569 8.450 7.881 32.48
4 4 256 0.574 8.451 7.877 32.50
5 1 256 0.697 10.647 9.950 25.73
5 2 256 0.697 10.632 9.935 25.77
5 3 256 0.696 10.639 9.943 25.75
5 4 256 0.704 10.648 9.944 25.74
5 5 256 0.698 10.626 9.929 25.78

Observations

  • Every measured request completed successfully and stopped at finish_reason=length.
  • Aggregate throughput increased from concurrency 1 to 4, then flattened at concurrency 5.
  • In this baseline, total throughput was 121.17 tok/s at concurrency 4 and 120.20 tok/s at concurrency 5.
  • Average TTFT rose with concurrency, from 0.221s at concurrency 1 to 0.698s at concurrency 5.