Studio · Bench
Which work belongs on your own machine — and on whose measurement?
Bench is the part of Saphan Studio that answers it — by measuring the box you actually own instead of reading a model card, then pinning the answer to a versioned profile that tells you when it has gone stale.
This is not “run models locally instead of the cloud”
Frontier models stay in the picture. They earn their price on the work where being wrong is expensive — design, hard debugging, review. What they are not earning it on is the mechanical middle: pattern migrations, summaries, scaffolding, the repetitive volume that fills the gaps between the hard parts and arrives on a per-token invoice.
The question is not whether to move that work. It is which work moves — and that turns out to be a property of your specific hardware that nobody can tell you in advance.
The right local setup is not knowable ahead of time. It has to be measured on the machine you actually own, and the gap between guessed and measured is large enough to change what you buy.
Measured, not guessed
Four models, one 128 GB nameplate unified-memory node, llama.cpp over Vulkan/RADV. GLM-4.5-Air is the obvious flagship — the one you reach for when the box is big enough to hold it. On this hardware it generates 2.1× slower than gpt-oss-120b — 2.12×, measured twice and agreeing to within a tenth of a standard deviation. Prompt processing is also more than twice as slow, though that ratio drifts between runs (2.11 and 2.25), so we quote it as a range rather than a number. It carries roughly 9 GiB more weight for the privilege. Neither model card says so, because neither model card is about your machine.
One node, four models, three measures
Single 128 GB nameplate unified-memory node (Ryzen AI Max+ 395 class), llama.cpp + Vulkan/RADV, Q4_K_M except gpt-oss-120b, which is native MXFP4. llama-bench -p 512 -n 128 -ngl 999 -fa 1 -ctk q8_0 -ctv q8_0, and f16 KV for kimi-linear, which cannot quantise it. All three panels share one order, sorted by generation speed.
| Model | pp512 t/s | tg128 t/s | Weights GiB | Cold load s | Warm s |
|---|
These figures describe one measured machine under one set of flags. They are evidence that measurement matters — not a prediction for your hardware. Memory fit does transfer between machines, because bytes are bytes. Throughput does not. Your calibration run produces your numbers, and until it runs we would rather show you nothing than show you someone else's.
What a model card cannot tell you
Five things that only appeared when the thing was actually run. None of them is visible on a spec sheet; each changes a decision.
Cold load is ~23× warm load
GLM-4.5-Air: 42.19 s from disk against 1.85 s warm — and that is the narrowest of the four gaps measured. That ratio, not any throughput number, is what decides whether you batch work or interleave it. It is a queueing policy hidden inside a loading time.
A success that is a failure
Reasoning models return HTTP 200 with empty content when the token budget is too small: the reasoning tokens ate it. Nothing errors. Your pipeline records a completed call and moves on.
One model cannot quantise its KV cache at all
Kimi-Linear's head dimension is 72; the q8_0 block size is 32. That is a hard constraint, not a tuning parameter, and there is no way to learn it except by running the model.
Capability has a measurable ceiling
A 30B-class coder is reliably good at single-file work and falls over above it. The ceiling is a property you can measure, not a function of the parameter count on the box.
The machine lies about its own memory
GPU-addressable memory defaulted to 60.61 of 121.21 GiB — exactly half, because the kernel’s ttm_global_init() does num_pages /= 2 and has done since 6.1. Not a fault and not a bad build: a generic default nobody chose. One boot parameter later it was 112 GiB, and the 120B model that “did not fit” fit.
The counterintuitive part, measured on the same box a week apart: the widely-repeated fix — raise the BIOS graphics carve-out — is the worse option on Linux. The GTT ceiling is derived from system RAM after the carve-out is subtracted, so a large carve-out mechanically lowers the ceiling it is meant to raise.
28 Jul, BIOS carve-out 64 GiB amdgpu: 65536M of VRAM memory ready amdgpu: 31451M of GTT memory ready. 05 Aug, BIOS carve-out 2 GiB + ttm.pages_limit amdgpu: 2048M of VRAM memory ready amdgpu: 114688M of GTT memory ready.
Turning the BIOS setting up to 64 GiB left 30.71 GiB of GTT. Turning it down to the minimum and setting one kernel parameter gave 112 GiB — 3.6× more, on the same hardware. The setting most guides tell you to raise is the one to leave alone.
None of these is visible on a model card. Every one of them was found by running the thing. That is what Bench is.
How it runs — value before installation
You find out whether this is worth doing before anyone asks you to install anything. The diagnosis is free and stays free.
Verdict free · installs nothing
Reads your hardware — installed memory, GPU-addressable memory, driver and kernel versions, disk headroom — and tells you which model classes your box can hold resident, which it can swap, and which will not fit. Every probed value carries the file or command it came from, so you can tell measurement from inference.
Profile
One command, reproducible. Pins the llama.cpp build hash, the serving proxy version, kernel, firmware and Mesa versions, and every model file by repository, filename and sha256. A profile carries a version, a tested-on matrix and an expiry date — see below for why that is not optional.
Calibration you ratify
Runs a curated task set through the real serving path — the transport you will actually use, not a synthetic loop around it. Where a task compiles or tests, CI decides; a model judges only where correctness cannot be checked mechanically. The output is a proposed lane assignment, and it goes through the same gate as everything else. Calibration never applies itself.
Report
Ongoing: local against frontier split, spend delta, and which lane would earn the next hardware upgrade. Computed from the record, which knows the endpoint that actually served each call — because a serving proxy can silently substitute a fallback, and a savings figure that trusts intent instead of the record is worth nothing.
A profile without a version and an expiry is a lie in waiting
During one build week upstream moved three times: a configuration key was removed mid-task, a “known-good kernel” rule from the previous month stopped being true, and a serving-layer feature moved. A setup that was correct on Monday was quietly wrong by Friday, and nothing in it knew.
So every profile states what it was tested on and the date after which it stops claiming to be true. When yours goes stale, or when you deviate from it on purpose, you re-run the test and get re-mapped. That is the supported path, not a failure mode.
$ saphan machine show strix-halo-evo-x2 [current] profile strix-halo-evo-x2 v1.0.0 · expires in 90 days kernel 7.0.0-28-generic · mesa 26.0.3 · vulkan 1.4.341 llama.cpp e031d95 · BIOS EVO-X2 1.12 · GTT 112.00 GiB models: 4 pinned by sha256 · 1 hard constraint recorded $ saphan machine show strix-halo-evo-x2 # ninety-one days later [expired] do not treat the figures below as current → run: saphan machine qualify strix-halo-evo-x2 --reverify
The constraint that cannot be written as prose: n_embd_head_v % kv_block_size == 0 — 72 against a q8_0 block size of 32, so kimi-linear cannot quantise its KV cache at all. Quantising only K does not rescue it. It ships as data with the reproduction attached, so the tooling refuses the impossible configuration instead of letting you discover it at load time, which is how it was found the first time.
Where Bench sits
Bench is a module of Saphan Studio, not a separate product. The lanes it proposes are the lanes the routing plane already runs. The ratification it asks for is the gate you already use. The spend it reports comes off the record you already keep.
The one piece that stands alone is the free verdict — because it has to work on a machine where nothing is installed yet, including Saphan.
Routing decides where a class of work should go. Bench is how you find out what your own hardware can take, so that decision is made on measurement instead of assumption.
When a second person needs the same guarantee
Everything above works the same for a team. What changes is that the split stops being one person's habit and becomes policy: lane assignment is a file someone approved rather than a default nobody chose, and spend becomes attributable rather than personal, because the record knows which endpoint actually served each call.
That is a door, not a different product. Start solo; it is the same profile when the second machine arrives.
Status, plainly
What is true today: the reference deployment runs, and every measurement on this page came off it. Verdict exists as a working prototype and reports against the reference hardware profile and a synthetic fixture set.
What is not true yet: Verdict has not been validated on a second physical machine. Profile, Calibration and Report are specified and not yet built. The supported matrix is narrow on purpose — one hardware class, one serving stack, the models listed above — and it widens by verification, one class at a time.
We would rather say “not measured” than publish a number we cannot stand behind. That is also why this section exists.
Design partner inquiries
Talk to the founder — hello@saphan.ai