Self-Evolving Search Index

Hugging Face Daily Papers Papers
information-retrieval search-index self-evolution llm-agents autonomous-systems academic-research ai-framework

Summary

The paper introduces SELF-INDEX, a framework that enables search indexes to self-evolve autonomously, improving retrieval performance and benefiting downstream applications such as search agents and agent memory systems.

Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.
Original Article
View Cached Full Text

Cached at: 09/18/26, 07:02 PM

Paper page - Self-Evolving Search Index

Source: https://huggingface.co/papers/2609.19656

Abstract

InformationretrievalisincreasinglyimportantasLLMagentstacklecomplextasksinvolvingdiverseinformationneeds.Becauseretrievalreliesonanindexthatrepresentseachdocumentthroughindexkeys,retrievalqualitydependsheavilyonhoweffectivelythesekeysexposetheknowledgecontainedineachdocument.However,effectiveindexrepresentationsvaryacrossretrievalenvironments,makingitdifficultforanyfixedoptimizationstrategytoperformconsistently.Yetevolvinganindextoitsretrievalenvironmentremainslargelyhuman-driven,requiringhumanstodiagnoseretrievalfailures,refinetheoptimizationstrategy,andreprocesstheindexaccordingly.WeproposeSELF-INDEX,aframeworkthatenablesanindextoself-evolvewithouthumanintervention.ItsOptimizerautonomouslydiagnosesretrievalshortfalls,selectivelyrevisestheresponsibleindexkeys,andvalidateseachrevisionbeforeupdatingtheindex.Beyondreactingtoobservedretrievaldemands,SELF-INDEXproactivelyexploresadditionaldemandsthroughaQuerySimulator,allowingtheindextoevolvebeyondthequeriesalreadyavailableforoptimization.Acrossdiversecorporaandretrievers,SELF-INDEXconsistentlyimprovesretrievalperformancewhileoutperformingexistingindexoptimizationmethods.Wefurthershowthatthesebenefitsextendtodownstreamapplications,improvingtheeffectivenessandefficiencyofsearchagentsandhelpingagentmemorysystemsretrieveusefulpastinteractions.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.19656

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.19656 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.19656 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.19656 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Self-Evolving Coding Agents

Hugging Face Daily Papers

This paper surveys self-evolving coding agents, which improve their future behavior by updating frameworks, memory, skills, tools, or models from prior coding interactions, and presents a taxonomy of what evolves, when, and what software-specific evidence drives it.

ERSkill: Evolving for Skill-Guided Adaptive Memory Retrieval

arXiv cs.CL

Introduces ERSkill, a retrieval-centric framework for self-evolving, skill-guided adaptive memory access in LLM agents. It co-evolves retrieval skills and a routing policy, substantially outperforming strong baselines across agent memory benchmarks.

Self Improvement via Fast Tree-search

arXiv cs.AI

This paper introduces RecursiveSelfImprovement via Fast Tree-search (SIFT), a sample-efficient framework that uses a lightweight tree-search guided by LLM-as-a-judge evaluations to improve coding agents' performance under budget constraints, outperforming existing methods with lower resource costs.