Tag
Shantanu Goel shared a configuration recipe for optimizing Qwen 3.8 Flash Next on a single DGX Spark, tested for practical tasks and plans to benchmark it further.
This paper establishes tight generalization bounds for multi-dimensional hyperparameter tuning in data-driven algorithm design, using real algebraic geometry and a multi-regime lower-bound framework to resolve theoretical gaps.
A new Meta paper reveals that small models can accurately predict scaling laws but require more extensive hyperparameter tuning. The study finds scaling laws emerge around 4M parameters and become clearer with proper tuning.
The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.
This paper identifies two structural mechanisms causing multi-task Gaussian processes to misestimate cross-task correlation in Bayesian optimization transfer learning, even for affinely related tasks. The authors propose three conservative remedies to mitigate these issues.
The article discusses various techniques to make LLM-powered agents more deterministic, such as golden sets, guardrails, consensus mechanisms, regression tests, coded logic, and hyperparameter tuning, and asks for additional successful methods.
This paper proposes replacing the stateless autoresearch pattern with a stateful ReAct agent using LangGraph, reducing per-iteration token costs from O(n) to O(1) and achieving 52-90% fewer tokens on hyperparameter tuning and code optimization benchmarks.
A method using Bayesian Probabilistic Context-Free Grammar to generate synthetic regression datasets that structurally resemble physics equations, validated against the Feynman corpus and shown to be effective for hyperparameter tuning.
A user found that reducing the learning rate from 2e-4 to 1e-4 significantly improved QLoRA fine-tuning of Llama 3.1 8B on a small dataset (8k samples), preventing overfitting and leading to better evaluation results.
This paper introduces AutoLLMResearch, an agentic framework that automates the configuration of expensive LLM experiments by learning from low-fidelity environments and extrapolating to high-cost settings. It aims to reduce computational waste and reliance on expert intuition in scalable LLM research.