Tag
vLLM announces integration with TileRT's decode engine via its V1 connector interface, allowing users to pair vLLM prefill with a specialized latency-optimized decode engine for workloads like agents and real-time assistants, without any changes to vLLM.
Kog has open-sourced the Laneformer 2B model, a 2.3B parameter instruction-tuned coding model designed for high-speed decoding, achieving over 3,000 tokens per second by prioritizing latency from the architecture stage.