18 measured runs across 10 open models
Stop guessing your inference bill
See what each open model really delivers on real GPUs before you commit to hardware: tokens per second, latency, cost per million tokens and energy, measured the same way on vLLM and SGLang.
- Speed, latency and peak throughput are free for every model
- No account needed to browse the leaderboard
- Pro adds cost per token, energy and full concurrency curves
Fastest measured result
Mistral-Small-4-119B-2603vLLM 0.32.0 · NVIDIA B300 x1
- 280.7tok/s
- per user
- 46ms
- first token
- 9,558tok/s
- peak, 256 users
- Cost per 1M tokens
- Pro
- Energy per 1M tokens
- Pro
Lowest GPU energy per million output tokens
Most output tokens per dollar of GPU time
Highest single-user generation speed
Most output tokens per second per GPU
Why teams use it
The three numbers that decide an inference budget
Measured, not estimated
Every figure comes from a run on our own hardware: three workloads, with the chat workload swept from 1 to 256 concurrent users. A metric we could not measure is left empty.
Cost per token you can plan on
Cost per million tokens from peak throughput and current GPU rental rates, so you can compare a model on two engines or two GPU counts directly.
Energy at the GPU
GPU board power sampled at 10 Hz and checked against the energy counter, reported as energy per million tokens and tokens per watt.
Leaderboard
Fastest single-user speed
| # | Model | Hardware | Tok/s per user | TTFT | Peak tok/s |
|---|---|---|---|---|---|
| 1 | Mistral-Small-4-119B-2603 mistralai · vLLM 0.32.0 Eco-friendlyFastest | NVIDIA B300 x1 tp1 × pp1 | 280.7 | 46 ms | 9,558 @ 256 users |
| 2 | Mistral-Small-4-119B-2603 mistralai · SGLang 0.5.22 Eco-friendlyProfitable | NVIDIA B300 x1 tp1 × pp1 | 269.6 | 81 ms | 9,602 @ 256 users |
| 3 | gemma-4-E4B-it google · SGLang 0.5.22 Eco-friendlyProfitable | NVIDIA B300 x1 tp1 × pp1 | 252.5 | 45 ms | 18,061 @ 256 users |
| 4 | gemma-4-E4B-it google · vLLM 0.32.0 Eco-friendlyProfitableTop throughput | NVIDIA B300 x1 tp1 × pp1 | 227.5 | 19 ms | 18,468 @ 256 users |
| 5 | DeepSeek-V4.1-Flash deepseek-ai · vLLM 0.32.0 | NVIDIA B300 x4 tp4 × pp1 | 192.7 | 33 ms | 12,730 @ 256 users |
| 6 | GLM-5.3-Flash zai-org · SGLang 0.5.22 | NVIDIA B300 x2 tp2 × pp1 | 172.0 | 76 ms | 3,962 @ 256 users |
ModelPerf Pro
The economics of every model, for less than an hour of GPU time
Free shows how fast a model is. Pro shows what it costs: per token, per watt, and at every load level.
$39 a month billed annually (save 20%), or $49 month to month.
- Cost per million tokens for every model, engine and GPU
- Energy per million tokens, tokens per watt and GPU power draw
- Full concurrency curves for all three workload scenarios
- Unlimited ROI and break-even calculations with PDF export
- CSV export of the full leaderboard, paid metrics included
- New models and re-runs included as they are published
Analysis
What the measurements say
LLM cold start times on NVIDIA B300: 51 seconds to 28 minutes
Measured server start-up times for 18 model and engine combinations on NVIDIA B300, and what they mean for autoscaling and scale-to-zero.
Cost per million tokens: the formula and what moves it
How to calculate self-hosted inference cost per million tokens from GPU price and measured throughput, with a worked example and the common mistakes.
How to read an inference benchmark: speed, latency and throughput
Tokens per second per user, time to first token and peak throughput answer three different questions. Here is what each one means and when it matters.
ROI calculator
Pick a configuration we measured, add your prices and costs, and get the break-even date and return as a PDF.
Run a calculationCustom benchmarks
Your model, your hardware or your hosted endpoint, run through the same suite with a full report.
Request a benchmarkPublished method
Workloads, concurrency levels, power sampling and how failed runs are handled are all written down.
Read the methodologyQuestions
Frequently asked
- What does ModelPerf.ai measure?
- Inference performance of open models on real GPUs: tokens per second per user, time to first token, peak throughput across a concurrency sweep, cost per million tokens, energy per million tokens and GPU power draw. It does not score accuracy.
- Which engines and hardware are covered?
- The benchmark runner supports NVIDIA (CUDA) and AMD (ROCm) GPU hosts with the vLLM and SGLang inference engines, and can also measure hosted endpoints over HTTPS. The results published today were measured on NVIDIA B300.
- What is free and what is paid?
- Speed, latency and peak throughput are free for every model. Cost per million tokens, energy, power and full concurrency curves are included in ModelPerf Pro, or can be unlocked for a single model.
- How is cost per million tokens calculated?
- Hourly GPU cost divided by the output tokens produced per hour at measured peak throughput, scaled to one million tokens.