We serve a Qwen3.8-27B on our own hardware in India, on a pair of NVIDIA RTX A5000s (24 GB VRAM each) running vLLM with tensor parallelism. Three questions come up about it every time, in the same order: how fast is it, how much context does it have, and how many agents can run against it at once.

Every one of those has an answer that sounds right and is wrong. So we spent a day measuring the live deployment properly, then changed two server settings and measured again. This post is the whole study: the numbers, the two mistakes the benchmark itself made, a free 2x that costs nothing but prompt ordering, and one tuning change that we reverted because it did nothing at all.

Everything here is measured on the hardware, not modelled.

The short answers

One request, nothing else running	60.3 tokens/sec, first token in 75 ms
The whole card, batched	641 tokens/sec peak, at 64 concurrent
Prompt processing	about 1,000 tokens/sec
Context window	131,072 tokens, prompt and completion share it
Concurrent requests accepted	256; beyond that they queue, they do not fail
Concurrent requests that are useful	about 32

The gap between those last two rows is most of this post.

“How fast is it” is two questions

There is no single tokens-per-second number for a GPU, and publishing one without a label is how a spec sheet turns into a support ticket. Single-stream is what one user feels with nobody else on the card. Aggregate is what the card produces in total while batching. Here they differ by more than 10x, and both are true.

Measured with 256-token replies and end-of-sequence ignored, so every request emits exactly the same count, and with a fresh prompt for every request:

Concurrent requests	Total throughput	Per request	Speedup
1	59 tok/s	59.4	1.0x
2	105 tok/s	52.3	1.8x
4	188 tok/s	47.1	3.2x
8	312 tok/s	39.0	5.3x
16	466 tok/s	29.2	7.9x
32	589 tok/s	18.4	9.9x
64	641 tok/s	10.0	10.8x

Read the last two rows together. Doubling from 32 to 64 concurrent requests buys 9% more total throughput and costs 46% of every individual request’s speed. That is where the card stops working harder and starts only queueing.

It is also why the usual capacity arithmetic — peak throughput divided by your speed floor — overstates what you have. Per-request speed falls as concurrency rises, so peak aggregate always happens at a concurrency where the floor has already been breached. The honest count is the highest concurrency at which each request still meets the floor, and it is a much smaller number.

The context window is not the cost

131,072 tokens, shared between prompt and completion. Ask for more and the server refuses with a 400 rather than quietly truncating, which is the behaviour you want and is worth checking on any endpoint you depend on.

What a long context actually costs is not generation speed:

Prompt size	Time to first token	Generation after that
6,616 tokens	6.6 s	58.8 tok/s
26,452 tokens	27.0 s	57.1 tok/s
82,660 tokens	—	53.5 tok/s

Generation slows about 11% across the entire window. What grows, linearly, is the wait before the first token, at roughly 1,000 tokens/sec of prompt processing. Budget one second of waiting per thousand tokens of prompt. A 100K-token context is about 100 seconds of silence before anything appears. If your agent pastes its whole history into every call, that is where the time goes — not in the model being slow.

“How many agents” has three different answers

There are three separate limits, and people quote whichever one they happened to hit.

The scheduler cap. vLLM will hold 256 sequences at once here. We fired 320 simultaneous requests: 159 ran concurrently, 161 waited in the queue, all 320 completed and none errored. Overload degrades into queueing rather than failure — which is what you want, though it does mean a load test can report total success while hiding a queue.

The KV cache. This is the one that usually binds. Every concurrent sequence holds its own context in that cache, so the window and the concurrency are the same budget spent two ways. These are the figures after the tuning described further down; before it, every row was about 15% smaller:

Context per agent	Agents that fit
4,000 tokens	141
8,000	70
16,000	35
32,000	17
131,072 (the full window)	4

That is the trade to plan around: the context window and the number of concurrent agents are the same pool of memory, spent two ways. Four agents that each fill the full window will saturate this box; seventy agents holding 8K each will not. Size a fleet on the context you will actually use, not on the window in the specification, or you will overestimate what one machine holds by an order of magnitude.

Throughput per agent is the limit that decides what is usable. If each agent needs 40 tok/s you can run 8; at 30 tok/s, 16; at 20 tok/s, 32. The card will accept 256. It will not do useful work for 256.

A free 2x that is only prompt ordering

Reading the server’s own configuration turned up something we would have guessed wrong: the KV block size is 800 tokens, not the usual 16. This model is a hybrid state-space and attention architecture, and vLLM sizes its blocks accordingly.

Block size governs prefix caching. A cached block needs every preceding token to match from position zero, so a shared prompt prefix only pays if it is byte-identical and at least one block long. We tested it with four concurrent agents, identical content in every arm, changing only the order:

Prompt layout	Wall time	Cache hit rate
Shared 2.4K-token prefix first, per-agent task last	6.78 s	62.9%
Same text, per-agent task moved to the front	13.86 s	0%
Four entirely unrelated prompts	13.45 s	0%

Same tokens, same work, twice the wall time — decided entirely by where the variable part sits. Putting the unique part first is indistinguishable from sharing nothing at all.

So: put the system prompt and any shared evidence first, byte-identical across every agent in a fan-out, and put the per-agent instruction last. Never interpolate a timestamp, a request id, a nonce or an agent index ahead of the shared block — one varying character at position zero throws the whole thing away. This is the first change we would make in anyone’s agent code, and it costs nothing.

Two settings changed. One worked.

GPU memory utilisation, 0.90 to 0.95. vLLM’s own startup log points out that 0.90 is effectively 0.812 once CUDA-graph memory profiling is accounted for, and the cards had around 3 GB free each. Recovering it did exactly what it promised:

	Before	After
KV cache	493,447 tokens	567,464 (+15.0%)
Max concurrency at full context	3.76x	4.33x
Agents at 8K context	61	70

Maximum batched tokens, 2,048 to 8,192. This did nothing, and we reverted it. Prompt processing is chunked at 2,048 tokens by default on cards below 70 GB, and it was running at only about 45% of the two cards’ theoretical compute, so a bigger chunk looked like free speed. Measured afterwards: 1,019 tokens/sec against 1,002 before at 6.6K, and 989 against 979 at 26.5K. Aggregate throughput did not move either — 309, 464 and 588 tok/s at 8, 16 and 32 concurrent, against 312, 466 and 589. Every row inside noise.

Prompt processing on this hardware is compute-bound, not chunk-bound. The missing 55% is quantisation overhead and cross-GPU communication, and no scheduler flag reaches it. We put it back, and putting it back actually gained cache: the two settings compete, because vLLM profiles a forward pass at the chunk size and gives the KV cache whatever is left over. 562,838 tokens with the big chunk; 567,464 with the default.

The lesson generalises: raise memory utilisation for concurrency, and leave the batch size alone unless you have measured your own prompt processing against your own peak FLOPs. We are publishing the change that failed because the reasoning behind it was perfectly plausible and completely wrong, which is the only kind of mistake worth writing down.

Two ways the benchmark fooled us

Both are general, and both are the sort of thing that ships as a confident number.

An idle check is a statement about the past. The harness did the standard thing: read the server’s count of running requests, wait for zero, then measure. It reported 36 tok/s for a single request on a model that does 60, because other traffic arrived one second into the measurement window.

The fix is not a longer wait, it is an exact bracket. With end-of-sequence ignored, our own output is precisely concurrency times max_tokens, so we read the server’s cumulative generated-token counter before and after every step and discarded any step where the delta exceeded what we had asked for. Over the day it caught intrusions from 2 tokens to 417. If you benchmark anything with other traffic on it, bracket every step and say how many you threw away.

A benchmark will happily measure its own cache. Our long-context probe reported 25,228 tokens/sec of prompt processing, twenty-five times the real figure. The retry loop rebuilt the identical prompt after a discarded attempt, so the second attempt was served from the prefix cache. We had written that harness specifically to avoid this, with a distinct prompt per request, and still reintroduced it in the retry path.

A third, smaller one: a metric that reads exactly zero deserves a check that the metric exists. We spent a run collecting a counter name that vLLM does not publish, and a silent 0/0 reads exactly like a real result.

One result we cannot explain

At exactly four concurrent requests with multi-thousand-token prompts, throughput is bimodal. Identical workload, four clean measurements: 86 tok/s, 183, 86, 183. Two tight modes, 2.1x apart, roughly half the runs each.

We dismissed the first slow reading as contamination; it survived the bracket four times, so it is real. Preemption is ruled out — the server recorded exactly one in three days. Our remaining suspects are the 800-token block granularity interacting with the 2,048-token processing chunk, and the hybrid model’s state allocation. We have not proven either.

We are publishing it unexplained rather than leaving it out, because four-wide is a very common fan-out width and anyone running one deserves to know it sometimes takes twice as long for the same work.

What we would do next

Free, today: put shared prefixes first; make fewer, larger calls, since any API hop adds a fixed per-call cost that a short reply cannot amortise; and stop running fan-outs wider than the throughput floor you actually need.

Worth a restart: recover the memory that CUDA-graph profiling quietly reserves, as above. Leave the batch size at its default.

The larger levers are not tuning at all. A sparse mixture-of-experts model in the same size class reads roughly a tenth of the bytes per token that a dense 27B does, which is about 6x the generation throughput by our own measurements — more than every tuning change put together, if the quality holds for your workload. And newer silicon is 2 to 3x on memory bandwidth alone, which is what decode speed is made of.

Tensor parallelism is worth checking too. Under it each card reads only its own shard of the weights, so the decode roofline sees the combined bandwidth of the group, not one card’s. We have had an estimate off by 2x from getting that one wrong, and it inverted the commercial conclusion rather than just the number.

What we learned, as a checklist you can run

If you are serving a 27B-class model on your own cards, this is the whole study compressed into things to do, in the order we would do them. Every one of them came out of a measurement above.

Before you tune anything: measure twice, and label both numbers

1. Measure single-stream and aggregate separately. They differ by more than 10x here. One unlabelled figure will be read as whichever one makes it wrong.
2. Fix the output length. Set a token count and ignore end-of-sequence, so every request emits exactly the same number of tokens and runs are comparable.
3. Use a different prompt for every request, including retries. Prefix caching will otherwise serve your second attempt from memory and report a number twenty-five times too high. This caught us inside a harness written to prevent it.
4. If anything else uses the machine, bracket every step. Read the server’s cumulative generated-token counter before and after; with a fixed output length you know exactly how many tokens are yours, so any excess is someone else’s traffic and the step must be thrown away. Waiting for an idle card is not enough — traffic that arrives one second in will halve your number.
5. Check that a metric reading zero actually exists. A misspelt counter returns a silent 0 that looks exactly like a clean result.

Read your own server’s startup log before touching a flag

Nearly every number that mattered was already printed and we had never read it:

• KV cache size, in tokens. Divide it by the context each agent will really hold and you have your true concurrency, which is almost never the scheduler’s limit.
• The KV block size. Ours is 800 tokens, not the 16 we assumed, because the model is a hybrid architecture. That single number decides whether prompt-prefix sharing does anything at all.
• The memory-utilisation note. vLLM states plainly that your setting is effectively lower once CUDA-graph profiling is counted, and tells you what to raise it to. That was our only change that worked.
• Whether the scheduler limit was ever yours to begin with. Ours was a default chosen by card size, not by us.

The changes worth making, in order of payoff

1. Put shared prompt text first, byte-identical. 2x, free, no restart. This is the biggest single win available and it is a code change in your client, not a server flag. Shared system prompt and context at the top of every prompt in a fan-out, per-item task at the bottom, and nothing variable ahead of it.
2. Raise GPU memory utilisation to what the log recommends. +15% KV cache for us, which is +15% concurrent agents at every context size. Watch that you keep a safety margin on the card.
3. Batch into fewer, larger calls. Per-call overhead does not shrink with a shorter reply, so ten small calls pay it ten times.
4. Pick your concurrency from the speed floor you need, not from what the server will accept. It will accept far more than it can serve usefully.

Things not to bother with, which we tried or checked

• Do not raise the prompt-processing batch size unless you have measured your prompt throughput against your cards’ peak FLOPs. Ours was compute-bound already; 4x the chunk size changed nothing and cost KV cache, because the two settings compete for the same memory.
• Do not raise the sequence limit. The KV cache binds long before it does.
• Do not shrink the context window hoping to free cache. The pool does not depend on it; only the per-request ceiling does.
• Do not push concurrency past the knee. Ours is around 32: past it you buy 9% more aggregate for 46% of every request’s speed.
• Be careful with a model-based speculative decoder on a small card. A drafter competes for the same memory as the KV cache; one we tried cut usable cache by nearly 5x and collapsed batched throughput. A draft method that needs no extra weights is the safer experiment.

And the arithmetic that is easy to get backwards

• Capacity is not peak throughput divided by your floor. Per-request speed falls as concurrency rises, so peak aggregate always occurs where the floor is already breached. Count the highest concurrency at which each request still meets the floor.
• Under tensor parallelism, scale by the combined bandwidth of the group, not one card’s. Each card reads only its own shard of the weights. Getting this wrong once put an estimate out by 2x and inverted the conclusion.
• Decode speed follows bytes read per token. That is why a sparse mixture-of-experts model in the same parameter class can be several times faster than a dense one: it only reads its active experts.

The one thing to take away

Publish two numbers, always, and label them. Single-stream is what a user feels; aggregate is what the hardware produces. A single unlabelled tokens-per-second figure will be read as whichever one makes it wrong.

And measure the change you just made. Half of ours did nothing.