inference-infrastructure

作者通过与几位构建 AI agent 的开发者交流，总结出批量 agent 任务的共同模式（延迟容忍、无交互、大批量），并指出现有实时与批量推理 API 均无法很好满足这类工作负载，尤其是 KV cache 命中率问题导致批量 API 实际上难以节省成本。

The author will be speaking at the AI Engineer Paris conference about Modal's playbook for serving low-latency inference.

TamarindBio has been selected to build, host, and operate the inference infrastructure layer for TuneLab 2.0, Eli Lilly's collaborative AI/ML drug discovery platform.