optimization

This paper derives batch scaling laws for sketched linear regression under power-law spectra, analyzing one-pass and multi-pass mini-batch SGD. It provides explicit risk decompositions showing how batch size affects bias, variance, and fluctuation terms, and establishes that without-replacement sampling yields lower noise than with-replacement.

Proposes a 'lift' method for training input-convex neural networks (ICNNs) that uses an unconstrained hypernetwork to emit non-negative inter-layer weights, softening the loss landscape and escaping gradient attenuation, achieving lower test loss than projected gradient descent and softplus reparametrization.

This paper analyzes tradeoffs between latency, reliability, and cost in LLM-enabled agentic workflows, introducing performance models and deriving optimal resource allocation policies like water-filling token allocation.

A critical observation about recent industry AI papers lacking novelty, citing examples like SkillOpt that treat natural-language skills as trainable external parameters.

This paper systematically studies scale vectors in LLM normalization layers, showing they optimize training through a self-amplifying preconditioning effect, and proposes three lightweight improvements that enhance performance and scaling behavior with negligible overhead.

Developed a custom C++ inference engine for MiniCPM-V 4.6 on Orange Pi AIPro (Ascend 310B NPU), achieving 2x speedup over stock framework by writing optimized AscendC kernels for matmul and causal-conv1d, reaching 5.90 tokens/s.

This paper introduces SF-NorMuon, a schedule-free spectral optimizer that matches or exceeds tuned AdamW on language models up to 772M parameters, with theoretical guarantees for stationarity and long-horizon stability.

This paper presents the aircraft disassembly scheduling problem, a large-scale combinatorial optimization task involving thousands of tasks, precedence relations, balance constraints, and limited space. It proposes a Constraint Programming model and a MIP model tested on real operational instances with up to 1450 tasks.

DVAO adaptively weights objectives based on reward variance to improve multi-reward RL training stability and multi-objective performance.

Converted the Qwen 3.6 35b a3b model to ROCmfp4 format, leveraging MTP benefits for improved performance on AMD hardware.

A tweet recommending --ddtree-budget 36 for Nvidia RTX 4090, claiming 2.5x speedup during decoding for Qwen3.6_27B.

This paper studies reward hacking in reinforcement learning for language models through the geometry of updates, identifying optimization drift as a key factor. It proposes trusted-direction projection to constrain gradients within a clean reference subspace, delaying shortcut exploitation and preserving task performance.

Turbo3 achieves 5% faster decode tokens per second compared to fp8 at 32K context, a performance improvement in quantization or model optimization.

KDA is an agent-driven kernel design framework that helped HAN Lab achieve top rankings in the MLSys FlashInfer Kernel Contest by minimizing human involvement. The agent leverages Humanize, KernelWiki, and profiler skills to produce state-of-the-art kernels.

A tip on KV-cache compression for transformer models: start with uncompressed keys and compressed values, then adjust based on model family sensitivity; try asymmetric before symmetric compression.

A developer shares that after spending 3 months over-optimizing a Claude code setup with 23 plugins and multiple frameworks, performance was worse than day one; deleting almost everything dramatically improved results, emphasizing that a minimal setup often works best.

Explains the mathematical concepts of gradient, Jacobian, and Hessian as fundamental tools in AI model training, describing how they measure change and their roles in optimization.

A comprehensive blog post explaining how to optimize deep learning performance by understanding three key components: compute, memory bandwidth, and overhead, using first principles to identify the performance regime and focus on effective optimizations.

A Go engineer recounts an incident where an in-memory datastore became overloaded due to slow sorting, and they implemented context cancellation inside sort functions by using panics and recover for non-local flow control, similar to how encoding/json handles errors.

Charlie Marsh (creator of Ruff) rhetorically asks if it's allowed to make things hundreds of times faster, likely referencing a major optimization in a software development tool.