18 measured runs across 10 open models

Stop guessing your inference bill

See what each open model really delivers on real GPUs before you commit to hardware: tokens per second, latency, cost per million tokens and energy, measured the same way on vLLM and SGLang.

  • Speed, latency and peak throughput are free for every model
  • No account needed to browse the leaderboard
  • Pro adds cost per token, energy and full concurrency curves

Fastest measured result

Mistral-Small-4-119B-2603

vLLM 0.32.0 · NVIDIA B300 x1

280.7tok/s
per user
46ms
first token
9,558tok/s
peak, 256 users
Cost per 1M tokens
Pro
Energy per 1M tokens
Pro
Unlock cost and energy for every model
Greenest
gemma-4-E4B-it
vLLM · NVIDIA B300 x1
Included in Pro

Lowest GPU energy per million output tokens

Most profitable
gemma-4-E4B-it
vLLM · NVIDIA B300 x1
Included in Pro

Most output tokens per dollar of GPU time

Fastest response
Mistral-Small-4-119B-2603
vLLM · NVIDIA B300 x1
280.7 tok/s per user

Highest single-user generation speed

Best throughput per GPU
gemma-4-E4B-it
vLLM · NVIDIA B300 x1
18,467.7 tok/s per GPU

Most output tokens per second per GPU

Why teams use it

The three numbers that decide an inference budget

Measured, not estimated

Every figure comes from a run on our own hardware: three workloads, with the chat workload swept from 1 to 256 concurrent users. A metric we could not measure is left empty.

Cost per token you can plan on

Cost per million tokens from peak throughput and current GPU rental rates, so you can compare a model on two engines or two GPU counts directly.

Energy at the GPU

GPU board power sampled at 10 Hz and checked against the energy counter, reported as energy per million tokens and tokens per watt.

Leaderboard

Fastest single-user speed

Full leaderboard
#ModelHardwareTok/s per userTTFTPeak tok/s
1Mistral-Small-4-119B-2603
mistralai · vLLM 0.32.0
Eco-friendlyFastest
NVIDIA B300 x1
tp1 × pp1
280.746 ms9,558
@ 256 users
2Mistral-Small-4-119B-2603
mistralai · SGLang 0.5.22
Eco-friendlyProfitable
NVIDIA B300 x1
tp1 × pp1
269.681 ms9,602
@ 256 users
3gemma-4-E4B-it
google · SGLang 0.5.22
Eco-friendlyProfitable
NVIDIA B300 x1
tp1 × pp1
252.545 ms18,061
@ 256 users
4gemma-4-E4B-it
google · vLLM 0.32.0
Eco-friendlyProfitableTop throughput
NVIDIA B300 x1
tp1 × pp1
227.519 ms18,468
@ 256 users
5DeepSeek-V4.1-Flash
deepseek-ai · vLLM 0.32.0
NVIDIA B300 x4
tp4 × pp1
192.733 ms12,730
@ 256 users
6GLM-5.3-Flash
zai-org · SGLang 0.5.22
NVIDIA B300 x2
tp2 × pp1
172.076 ms3,962
@ 256 users

Pipeline

Running now

Loading pipeline…
Watch the pipeline

ModelPerf Pro

The economics of every model, for less than an hour of GPU time

Free shows how fast a model is. Pro shows what it costs: per token, per watt, and at every load level.

$39 a month billed annually (save 20%), or $49 month to month.

  • Cost per million tokens for every model, engine and GPU
  • Energy per million tokens, tokens per watt and GPU power draw
  • Full concurrency curves for all three workload scenarios
  • Unlimited ROI and break-even calculations with PDF export
  • CSV export of the full leaderboard, paid metrics included
  • New models and re-runs included as they are published

Analysis

What the measurements say

All analysis

Questions

Frequently asked

What does ModelPerf.ai measure?
Inference performance of open models on real GPUs: tokens per second per user, time to first token, peak throughput across a concurrency sweep, cost per million tokens, energy per million tokens and GPU power draw. It does not score accuracy.
Which engines and hardware are covered?
The benchmark runner supports NVIDIA (CUDA) and AMD (ROCm) GPU hosts with the vLLM and SGLang inference engines, and can also measure hosted endpoints over HTTPS. The results published today were measured on NVIDIA B300.
What is free and what is paid?
Speed, latency and peak throughput are free for every model. Cost per million tokens, energy, power and full concurrency curves are included in ModelPerf Pro, or can be unlocked for a single model.
How is cost per million tokens calculated?
Hourly GPU cost divided by the output tokens produced per hour at measured peak throughput, scaled to one million tokens.