GPT-OSS-20B on a Single H100

01 / FINDINGS

What changed under load

The GPU reached high aggregate throughput well before the serving system reached a good operating point for users.

01

Throughput plateaued

Short-context throughput flattened between concurrency 400 and 1,600, peaking at 800 before declining.

02

KV pressure appeared as queueing

At saturation, fewer requests stayed resident while waiting queues and time to first token rose sharply.

03

Half KV moved the boundary

With 28.26 GiB, KV reached 100% at concurrency 800 instead of 1,600 under the auto-sized policy.

04

Raw throughput hid the damage

Aggregate output stayed near 8K tokens/s while average TTFT at concurrency 800 rose from 10.0s to 18.3s.

KV cache is primarily a capacity resource. Reducing it did not make token generation faster; it reduced the number of requests vLLM could keep resident.

02 / SHORT CONTEXT

Finding the throughput knee

With 256 input and 128 output tokens, the baseline peaked at concurrency 800. Beyond that point, aggregate throughput and caller fair share both fell.

Output throughput vs. concurrency

OUTPUT TOKENS / SECOND

Baseline: gpu-memory-utilization=0.92, approximately 56.5 GiB KV, 4,096-token maximum model length.

Concurrency	Output tok/s	Request/s	Avg TTFT	Avg ITL	Fair share tok/s
100	3,305	26.45	2.53s	9.9ms	33.05
200	7,968	63.80	1.28s	14.7ms	39.84
400	8,869	71.00	1.71s	31.1ms	22.17
800	9,354	74.88	2.22s	67.0ms	11.69
1,600	8,994	71.99	3.10s	145.2ms	5.62
3,200	8,206	65.68	10.51s	263.6ms	2.56
6,400	5,067	40.56	12.61s	331.7ms	0.79

Short-context KV policy peaks

These runs all remained in a compute-dominated regime. Their peak differences are not evidence that a smaller cache accelerates the model.

Auto KV · utilization 0.92

9,354

Auto KV · utilization 0.98

9,792

Exact KV · 28.26 GiB

10,221

03 / STRESS TEST

Long context exposed capacity

The sustained 2,048-input / 512-output workload made KV capacity visible. The auto-sized cache reached its boundary at concurrency 1,600.

Aggregate throughput stayed close

OUTPUT TOKENS / SECOND

The half-KV run is not faster in a causal sense. Both configurations kept the GPU near compute saturation.

Time to first token

AVERAGE · SECONDS

Maximum KV use

PERCENT

Concurrency	Auto KV tok/s	Half KV tok/s	Auto TTFT	Half TTFT	Auto / half KV use
50	4,089	4,505	1.12s	0.74s	5.1% / 10.6%
100	5,992	6,108	0.98s	0.94s	10.0% / 21.8%
200	7,620	7,773	2.59s	1.70s	19.8% / 42.9%
400	7,878	8,197	4.82s	4.77s	39.6% / 85.5%
800	8,153	8,362	9.96s	18.35s	80.2% / 100%
1,600	7,228	7,663	34.21s	52.93s	100% / 100%

At concurrency 800, cutting KV approximately in half increased average TTFT by 84% while aggregate output throughput changed by only 2.6%.

04 / CAPACITY

The caller sees a different curve

Dividing server-wide output throughput by configured concurrency provides a simple fair-share estimate. It falls rapidly after the useful operating range.

Estimated output rate per configured caller

OUTPUT TOKENS / SECOND / CALLER

Estimate = aggregate output token throughput ÷ configured concurrency. Waiting callers receive no tokens, so this is capacity intuition rather than an individual streaming trace.

For the auto-KV stress run, concurrency 200 delivered about 38.1 fair-share tokens/s. Raising concurrency to 400 added only 3.4% aggregate output throughput, but reduced fair share to 19.7 tokens/s and increased average TTFT by 86%.

05 / DECISIONS

Serving recommendations

Optimize for latency-constrained goodput and queue depth, not the single highest tokens-per-second number.

Default range

Start long-context capacity planning around concurrency 200–400 on this hardware and workload, then tune against an explicit TTFT objective.

KV sizing

Use automatic profiling or a conservative explicit --kv-cache-memory-bytes value. A forced 62.05 GiB allocation failed at startup with CUDA OOM.

Admission

Apply request admission control before KV reaches 100%. Queue depth and waiting requests are leading capacity signals.

Benchmarking

Use sustained, duration-bounded workloads with realistic input/output distributions and evaluate TTFT, ITL, and SLO goodput alongside throughput.

Caller metric

Do not interpret aggregate output tokens/s as the stream rate seen by every caller. Track per-request latency distributions in production.

06 / METHOD

How the experiment ran

AIPerf concurrency was matched to vLLM's --max-num-seqs setting for each sequential server run.

Hardware: 1× NVIDIA H100 80GB HBM3
Model: openai/gpt-oss-20b · MXFP4
Tensor parallelism: 1
KV dtype: auto
Short workload: ISL 256 · OSL 128 · max len 4,096
Stress workload: ISL 2,048 · OSL 512 · max len 16,384
Stress profile: 180s profile · 30s warmup
Forced half KV: 30,343,943,946 bytes · 617,042 tokens

vLLM uses continuous batching. Here, “concurrency” is the number of in-flight AIPerf requests, while max-num-seqs is a scheduler cap—not a promise that every request is resident on the GPU at once. Under half-KV pressure, roughly 450–500 requests were running while the remainder waited.

The complete benchmark matrix, machine-readable results, and runner scripts are available in the experiment directory.