All research systems
miniserveActive

What do continuous batching and a paged KV cache each change when a serving engine is built from first principles, and what does each cost?

Question → hypothesis/design → architecture → evaluation → results → limitations → next questions

Motivation

Serving behavior constrains every architecture built on top of it. miniserve is a small LLM serving engine written from scratch in PyTorch to separate two effects that are usually discussed together: how requests are scheduled, and how key-value memory is reserved. It runs Qwen3-0.6B on Apple Silicon or CPU and states what it does not show.

Hypothesis

Continuous batching raises throughput and cuts the wait for new requests at some cost to each user's token rate, and exact-length block reservation helps only when key-value memory is scarce.

Architecture

  • Qwen3-0.6B forward pass written directly in PyTorch from a pinned model snapshot; no transformers model code at runtime
  • Block allocator with ownership invariants, per-sequence block tables, and a paged key-value cache
  • Iteration-level scheduler: strict first-come admission, chunked prefill, bounded queue, deadlines that include queue wait, cancellation, one terminal event per request
  • Benchmark arms that change one thing at a time: sequential, static batching and continuous batching, each with maximum-length or exact-length reservation

Implementation

  • Python 3.13, PyTorch on MPS or CPU, strict type checking, 96 tests
  • Greedy output matches Hugging Face token for token in float32 on CPU, with scrambled block tables, reused blocks and batched padded decode
  • Injected defects in the cache and scheduler make the tests fail, so the tests can catch real mistakes

Evaluation methodology

Measured on one Apple M1 Max. A canary measurement before and after each cell guards against machine slowdown. No comparison with vLLM or any other server was run.

Workloads
  • 48 requests per cell, Poisson arrivals at 1, 2, 4 and 8 times the measured sequential capacity, chat-template prompts of varying length, answers that end on the stop token
Baselines
  • Sequential, one request at a time
  • Static batching held until the longest request ends
Metrics
  • Time to first token, inter-token latency, output tokens per second, queue wait
  • Blocks reserved against blocks holding tokens, across pool sizes

Results

Preliminary results
  • At twice the sequential load the median wait for the first token was 28 s one request at a time, 1.7 s with static batching and 60 ms with continuous batching. At eight times the load, continuous batching produced 267 to 269 output tokens per second against 46 sequential and 186 static
  • Batching cost each user speed: the median gap between tokens rose from 21 ms to 43 ms at the highest load
  • Exact-length reservation changed throughput only when the pool was small (1.24 to 2.2 times at 192 to 32 blocks) and made no difference with a large pool. It still left 38% of reserved blocks unused, because answers end early
  • Paging overhead at one user at a time was 2.3% to 2.8%, from two 8-request runs
  • Bfloat16 on MPS matched the float32 reference exactly for 7 of 16 sequences over 48 tokens, so outputs differ slightly between configurations

One complete repetition. A first attempt was discarded after a four-fold slowdown that coincided with the laptop sleeping and waking, and a second run was stopped early. Treat differences under about 10% as noise.

Limitations

  • Not fast: attention gathers blocks then calls the standard PyTorch routine, with no fused kernel
  • One model, greedy decoding, one machine, one repetition
  • Reservation is worst-case up front, so it captures only part of what paging can give; growth on demand and preemption are deferred
  • No claim about vLLM, other hardware, or large-batch paging cost beyond a microbenchmark

Current status

Public and Apache-2.0. Engine, tests and benchmark harness are complete; an independent review recomputed every headline number from the raw results and all matched.

Roadmap

  • Growth on demand with preemption
  • An HTTP endpoint

Next questions

  • Where reserve-on-demand and preemption beat worst-case reservation, and by how much
  • How much of continuous batching's gain on small models is amortized per-step overhead rather than memory bandwidth