Tag
FluidPD introduces SLO-aware in-place elasticity for prefill-decode disaggregated LLM serving, using FluidToken to offload prefill work during transient imbalance and FluidRole to reassign workers between phases during sustained shifts, improving SLO attainment over static SGLang by up to 94.6 percentage points on Azure production traces.
This article analyzes Kubernetes Horizontal Pod Autoscaler (HPA) through control theory, explaining its behavior as a proportional controller and discussing trade-offs like deadband and slow response in autoscaling.
A promotional post for Modal, an AI infrastructure platform featuring SDK, autoscaling, and production-ready tools, along with an announcement for the Runtime conference for AI engineers.
The blog post analyzes a recent GitHub outage caused by a misconfigured autoscaling policy, discussing the challenges of autoscaling in cloud infrastructure and referencing the component substitution fallacy to highlight systemic issues.
Cline compares the cost of using Kimi vs Fable for token inference, finding Kimi 3-12x cheaper, and predicts that self-hosting open-weight models will become standard for businesses as token consumption scales, especially with models like Kimi K3.
Explored NVIDIA Dynamo, a tool for deploying LLMs across multiple GPU cluster nodes with features like model caching, autoscaling, multinode deployments, and Kubernetes integration.
STARIXNet is a lightweight neural network that improves cloud resource allocation by capturing multivariate spatio-temporal relationships among system metrics, prioritizing service stability over forecast accuracy. Deployed at Walmart, it achieved 10-50% cost savings while maintaining service reliability.
A benchmark study finding that a calibrated rule-based autoscaler beats six mainstream deep RL algorithms on cost across all tested workloads, with RL only showing benefits on bursty patterns at higher cost. The paper introduces RLScale-Bench to improve evaluation protocol and reproducibility.
Discusses practical challenges in scaling infrastructure for AI agent pipelines on a budget, highlighting the inadequacy of CPU/memory-based autoscaling for GPU inference workloads.