Tag
A detailed breakdown of why uploading a large PDF to ChatGPT doesn't significantly slow response times, covering techniques like streaming, KV caching, and parallel prefill phase.
A blog post explaining continuous batching, a technique for improving LLM serving throughput by dynamically adding new requests to a batch as old ones finish, keeping the GPU busy and reducing idle time.