Orca: Continuous (Iteration-Level) Batching

Editorial record

Plain-language summary

Traditional batching runs a whole batch of requests together until all finish, so short requests wait for long ones and the GPU sits underused. Orca schedules at the granularity of a single decoding iteration: after each token step it can inject newly arrived requests and remove completed ones, keeping the batch continuously full. Combined with selective batching that handles operations with different sequence lengths correctly, this delivered large throughput and latency gains and became the standard serving pattern later adopted by systems like vLLM.