Tag
The article discusses various techniques to make LLM-powered agents more deterministic, such as golden sets, guardrails, consensus mechanisms, regression tests, coded logic, and hyperparameter tuning, and asks for additional successful methods.
This paper proposes replacing the stateless autoresearch pattern with a stateful ReAct agent using LangGraph, reducing per-iteration token costs from O(n) to O(1) and achieving 52-90% fewer tokens on hyperparameter tuning and code optimization benchmarks.
A method using Bayesian Probabilistic Context-Free Grammar to generate synthetic regression datasets that structurally resemble physics equations, validated against the Feynman corpus and shown to be effective for hyperparameter tuning.
A user found that reducing the learning rate from 2e-4 to 1e-4 significantly improved QLoRA fine-tuning of Llama 3.1 8B on a small dataset (8k samples), preventing overfitting and leading to better evaluation results.
This paper introduces AutoLLMResearch, an agentic framework that automates the configuration of expensive LLM experiments by learning from low-fidelity environments and extrapolating to high-cost settings. It aims to reduce computational waste and reliance on expert intuition in scalable LLM research.