PagedAttention scheduler playground

Several requests share one fixed block of GPU memory for their KV cache. Serve them first-come-first-served and watch what the allocator does. Switch the mode between paged blocks (allocate on demand, swap the latest request to CPU when full) and contiguous reservation (one slab of the maximum length per request). Everything runs in this page; no network calls.

OPT-13B shape40 layershidden 5120fp16800 KiB of KV per tokenoffline, one file
Memory presets:

Scheduling timeline (one row per request, one column per cycle)

waitingprefilldecoding (request colour)swapped to CPUaborted

GPU work per cycle

Estimated as 2 x 13e9 FLOPs per token processed (decode: one token per running request; prefill: the whole prompt). It ignores the cost of attention itself, so treat it as a rough scale, not a measurement. Teal = decode cycles (scaled to the largest decode cycle); amber = cycles that include a prefill, drawn clipped.

Event log

KV bytes per token = 2 (K and V) × 40 layers × 5120 hidden × 2 bytes (fp16) = 819,200 bytes, about 800 KiB. "GiB" means 230 bytes. One cycle is one decode step of the whole batch, not a unit of time: a cycle with more requests does more work. Prefill of a newly admitted request takes the cycle in which it is admitted. Requests are served first-come-first-served and a request that does not fit blocks the ones behind it. Paged mode: blocks are allocated on demand; when none is free, the most recently arrived running request is swapped to CPU memory as a whole (all-or-nothing) and returns, oldest first, when free blocks cover its blocks plus one spare block per running request (that spare-block rule is this simulator's own addition, to avoid swapping a request in and straight out again). As in the paper, no new request is admitted until every preempted request has finished. Contiguous mode: each request gets one slab of the maximum length, so the number of slabs that fit is the memory divided by the slab size, and nothing is swapped. This is a teaching model, not vLLM: no prefix caching, no recomputation, no copy-on-write, no real timings.