https://devopstales.github.io/tags/continuous-batching/ Recent content in Continuous-Batching on DevOpsTales Hugo -- gohugo.io en-US Fri, 02 Oct 2026 00:00:00 +0000 https://devopstales.github.io/ai/vllm-scheduling-throughput-latency/ Fri, 02 Oct 2026 00:00:00 +0000 https://devopstales.github.io/ai/vllm-scheduling-throughput-latency/ At 9 in the morning Alice asks the office’s local model to summarize yesterday’s meeting, and the answer streams back faster than she can read it. By 10 the whole team has found the thing. Same GPU, same model — and now everyone’s replies stutter and stall. Something ran out. The hard part is telling what. The short version: the GPU is being shared, and vLLM’s scheduler is good at the sharing part. What usually decides how many people fit is memory space, and long conversations eat most of it. That also tells you which targets to pick for a personal assistant, an office tool, or a customer-facing app.