gradient-based-optimization

This paper introduces Prototype-Based Sparse Steering, a method that applies sparse autoencoders to attention query activations in LLMs, then uses gradient-based optimization during inference to steer generation toward target behaviors. The approach is validated in both a logical planning task and a stylistic educational domain, demonstrating interpretable and disentangled control.

This paper proposes Latent Heuristic Search (LHS), a framework that shifts heuristic discovery to a learned continuous latent manifold, using gradient-based optimization and normalizing flows to generate novel heuristics conditioned on large language models, achieving competitive results on TSP, CVRP, KSP, and Online Bin Packing.