Does a Language Server Save Tokens for Coding Agents? A Measurement Methodology and Preliminary Study

# Does a Language Server Save Tokens for Coding Agents? A Measurement Methodology and Preliminary Study
Source: [https://arxiv.org/html/2608.13568](https://arxiv.org/html/2608.13568)
\(June 2026\)

###### Abstract

Coding agents spend most of their context budget on*retrieval*: deciding which tokens of a repository must enter the model’s context so it can build a faithful internal model of the task\. Two regimes dominate\.Lexical retrieval\(grep,ripgrep,find\) is universal, instant, and zero\-setup, but noisy: it cannot distinguish a definition from a call from a mention in a comment\.Semantic retrievalvia the Language Server Protocol \(LSP\) —textDocument/references,definition,hover,documentSymbol— is precise and typed, but requires a running, indexed server and pays a per\-symbol round\-trip cost\. The widely repeated claim is that semantic retrieval is “more token\-efficient\.” We surveyed the tools and literature that make this claim and found a striking gap:it is asserted almost everywhere and measured almost nowhere\.This paper \(1\) formalizes the question with a single primary metric,*tokens\-to\-success*; \(2\) specifies a five\-arm ablation isolating semantic retrieval from confounds; \(3\) maps three pre\-stated failure modes onto measurable variables; and \(4\) reports apreliminary measurement studyacross four task classes \(Pythonrequestsplus two TypeScript repositories; Claude Opus 4\.8, Sonnet 4\.6, Haiku 4\.5\) that turns the prior into first numbers\. The answer is conditional and, in the common case,negative: on symbol\-named*localization*the LSP*costs*tokens \(\+6%\+6\\%Opus,\+118%\+118\\%Sonnet\) and the agent ignores it entirely when free; on*reference\-completeness*the LSP buysprecision\(1\.001\.00vs0\.760\.76, zero false call sites\) but not token savings \(a∼19%\{\\sim\}19\\%premium\) and cannot raise the recall ceiling set by agent thoroughness; and it nets a tokensaving only for the weakest model\(Haiku,−26%\-26\\%\), as a crutch against lexical noise\. On*editing*scored by real test execution the verdict is sharpest:grepsolves multi\-file renames perfectly while a location\-only LSP fails three\-quarters of them by missing a call site, and even a complete, index\-warmed, text\-enriched LSP \(returning each reference’s line inline, as production LSP\-MCP servers do\) recovers most of that gap — cutting follow\-up file\-reads∼5×\{\\sim\}5\\times— but cannot close it, because a rename must touch comments and strings that semantic references structurally exclude\. The most robust finding is that the agent’s tool choice istask\-dependent: it defaults togrepon localization \(semantic\-tool use0–6%6\\%\) but reaches for the LSP about half the time on reference tasks \(4545–57%57\\%\),*unprompted*\. The grep preference is not a fixed bias but a learned, task\-shaped policy\. This argues not for “LSP\-always” but for an*adaptive router*keyed on task class, model capability, and lexical noise — and, since that routing competence is demonstrably already present in latent form, for reinforcing it into the policy rather than bolting on the tool\.

## 1The fundamental question

An agent navigating a codebase is, in essence, performingretrieval under a token budget\. The model has a finite context window; the task is solvable only if the*right*tokens enter that window and the*wrong*tokens stay out\. Every tool call is a retrieval decision with a price, denominated in tokens\.

From this lens, the two regimes are two points on a precision/recall/cost surface\.Lexical retrievalhas high recall and low precision: it finds every textual match, including matches in comments, strings, and unrelated identifiers, and the agent often pays again to read surrounding lines\.Semantic retrievalhas high precision: “find references” returns exactly the resolved references\. The cost moves elsewhere — a server must run and index, and each query is a JSON\-RPC round\-trip whose latency grows with repository size\. So the question has a precise form:

> At equal task\-success rate, how many fewer tokens does semantic retrieval place into the agent’s context than lexical retrieval, and under what conditions does that delta become negative?

The phrase*at equal task\-success rate*is load\-bearing: a method that “saves tokens” by failing earlier has saved nothing\. Token savings are meaningful onlyconditioned on iso\-accuracy\.

## 2Background and related work

#### Provenance note\.

Live web*search*tooling was degraded during this study \(returning model\-generated synthesis rather than ranked results\)\. Two retrieval paths returned verifiable primary sources, and every claim below is grounded in one:\(F\)fetch\-verified this session \(the OSS tools below, and arXiv papers retrieved via the arXiv API with full abstracts\);\(K\)established background from training knowledge, flagged where used and never relied on for a novel quantitative claim\. This separation is deliberate: the paper’s central claim is about the*absence*of measurement, so it must not itself rest on unverifiable measurement\.

#### Tools that already expose semantic navigation \(F\)\.

Serena\[[7](https://arxiv.org/html/2608.13568#bib.bib7)\]wraps LSP withfind\_symbol,find\_referencing\_symbols, and symbolic editing across 40\+ languages; it states symbolic editing is “much more token\-efficient” and lets an agent explore “without reading entire files” — an explicit claim*with no benchmark numbers*\.mcp\-language\-server\[[8](https://arxiv.org/html/2608.13568#bib.bib8)\]wraps a real language server and calls its edit tool “more reliable and context\-economical” — scoped to editing,*no general token comparison*\.Aider repo\-map\[[9](https://arxiv.org/html/2608.13568#bib.bib9)\]is a static cousin of LSP: tree\-sitter extracts signatures and a PageRank\-style ranking keeps the most\-referenced symbols within a token budget; its stated motivation is exactly our thesis \(“sending whole files … waste\[s\] the precious context window”\) but it reports*no before/after numbers*\.

#### Recent literature \(F, arXiv, full abstracts captured\)\.

TypeScript Repository Indexing for Code Agent Retrieval\[[3](https://arxiv.org/html/2608.13568#bib.bib3)\]— most directly relevant, in our originally\-chosen language\. It observes that “LSP\-based resolution requires a JSON\-RPC call for each symbol lookup, \[so\] these per\-symbol calls become a bottleneck on large TypeScript repositories,” motivating a TS\-Compiler\-API parser\. It measures*index\-construction*efficiency, not an agent’s tokens\-to\-success\.RL from Compiler and Language Server Feedback\[[4](https://arxiv.org/html/2608.13568#bib.bib4)\]argues compilers/type\-checkers/language\-servers “already compute the missing supervision signal … but expose it through interfaces designed for human\-driven IDEs rather than learning loops,” and turns LSP signal into a shaped process reward — the strongest articulation of the interface\-friction hypothesis and of training the signal into the policy\.CORE\-Bench\[[5](https://arxiv.org/html/2608.13568#bib.bib5)\]is a benchmark for requirement\-driven repository search \(180K\+ queries on SWE\-bench\-series\), reporting “a sharp drop from traditional code search to code retrieval in agentic coding settings\.”Code as Agent Harness\[[6](https://arxiv.org/html/2608.13568#bib.bib6)\]frames code/tooling as the agent’s substrate and lists “evaluation beyond final task success” as an open challenge\.

#### The gap\.

Across tools and papers the pattern is consistent:the token\-efficiency of semantic retrieval is asserted, not measured\.The closest quantitative work measures index\-build efficiency, not agent tokens\-to\-success at iso\-accuracy\. That gap is this study’s justification\. Established background \(K\): SWE\-bench\[[1](https://arxiv.org/html/2608.13568#bib.bib1)\]and the SWE\-agent agent\-computer\-interface line\[[2](https://arxiv.org/html/2608.13568#bib.bib2)\]provide the verified\-task harness our protocol reuses\.

## 3Method

#### Primary metric: tokens\-to\-success\.

Let a*task*be a repository state plus a verifiable goal\. For agent configurationccon tasktt, runRRrollouts\. Define

success​\(c,t\)=1R​∑r=1R𝟙​\[rollout​r​verified\],T2S​\(c,t\)=∑r:verifiedtokens​\(c,t,r\)\#​\{r:verified\},\\mathrm\{success\}\(c,t\)=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\mathbb\{1\}\[\\text\{rollout \}r\\text\{ verified\}\],\\qquad\\mathrm\{T2S\}\(c,t\)=\\frac\{\\sum\_\{r:\\,\\text\{verified\}\}\\mathrm\{tokens\}\(c,t,r\)\}\{\\\#\\\{r:\\text\{verified\}\\\}\},wheretokens​\(c,t,r\)\\mathrm\{tokens\}\(c,t,r\)is thetotal context tokensconsumed by rolloutrr\(prompt \+ tool\-result \+ generated, summed over turns — what the user pays for\)\. The headline comparison is always the pair\(success rate,T2S\)\(\\text\{success rate\},\\text\{T2S\}\); we never report a token number without its success rate\. Secondary, per\-rollout: tool\-call counts \(by tool\), read volume, turns, wall\-clock, and thegrep false\-positive rate\(fraction of lexical matches that are not semantically relevant — the theoretical headroom for precision\)\.

#### The ablation: five arms\.

All arms share thesame model, tasks, harness, and prompt scaffold; the only variable is the retrieval tool surface\.

The A–B–C triangle answers “does semantic help when available, and does the agent use it when free?” D vs\. C isolates*habit*from*capability*; E vs\. B isolates how much benefit needs a live server\. \(Arm E is untested here\.\)

#### Tasks and controls\.

We use tasks where reference\-finding is on the critical path: change\-signature/fix\-callers, who\-calls\-X, cross\-file refactor, dead\-code, and issue\-to\-edit localization, drawn from SWE\-bench\-series so success is objectively checkable\. We hold model and decoding fixed, match tool\-description length across arms, measure LSP arms both warm and cold, and log every tool call with its token cost so deltas can be attributed to \(a\) fewer reads, \(b\) less noise per read, or \(c\) fewer turns\.

## 4The three failure modes, made measurable

@pchsu pre\-stated three reasons an LSP might fail in practice\. Each maps to a measurement and a remedy\.H\-install\(hard to install/initialize\)→\\rightarrowmeasure per\-language setup success and cold\-start cost; remedy arm:*index\-on\-build*\(for TS, via the Compiler API rather than a live session, following arXiv:2604\.18413\)\.H\-interface\(API too limited\)→\\rightarrowlog every*LSP\-insufficiency event*where the agent falls back to grep, and categorize; the taxonomy of fallbacks is itself an output pointing at candidate API extensions\.H\-churn\(edits invalidate the index\)→\\rightarrowmeasure re\-index latency and “tokens wasted re\-reading after edit”; remedy arm: partial\-index \+ grep hybrid\. A clean negative on any remedy is a real finding — it tells a team where*not*to invest\.

## 5Why agents prefergrep— mechanism hypotheses

Given a free choice, agents reach forgrepand rarely issue LSP queries\. Ranked by prior: \(1\)Training prior— pretraining/RLHF corpora are saturated with humans runninggrepin terminals and nearly devoid of programmatic LSP JSON\-RPC; the policy has a high prior on the high\-frequency tool\. \(2\)Tool affordance—grepis universal and terse; LSP tools carry higher perceived activation energy\. \(3\)Universality—grepworks in any repo with zero setup\. \(4\)Latency—grepreturns instantly; a cold index stalls\. \(5\)Interface gaps— no single LSP call answers “where is this*conceptually*used” across dynamic dispatch and strings\. The decisive experiment is C vs\. D: if forcing semantic\-first improves T2S over free choice, the agent was*under\-using*a capability it had \(a habit problem\); if not, semantic retrieval genuinely was not better for those tasks \(a coverage problem\)\.

## 6Results

We implemented the harness and ran the study onPython /requests\(and two TypeScript repositories for the cross\-language and edit experiments\) with two retrieval servers \(python\-lsp\-server, andpyrightfor the reference and edit experiments\), driving Claude models through a tool\-use agent loop that logs token usage per turn\. The study is preliminary — small task suites, few repositories — and validates the methodology and surfaces first signals rather than settling the question at scale \(§[7](https://arxiv.org/html/2608.13568#S7)\)\. All raw results are public \(see Code and Data Availability\)\.

### 6\.1Localization — grep wins on tokens

Six SWE\-bench\-Literequestsissues; task = name the file\(s\) to change, verified against the gold patch; four arms×\\times3 rollouts, Opus 4\.8\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x1.png)Figure 1:Localization \(Opus 4\.8\): the LSP costs more, not less\.Tokens\-to\-success by arm\. The LSP\-only arm \(B\) is6%6\\%more expensive than grep\-only \(A\) at identical100%100\\%success\. The free\-choice arm \(C\) collapses onto grep \(it uses the LSP0%0\\%of the time\), and forcing semantic\-first \(D\)*reduces*success to89%89\\%\. On localization where the issue text names the symbol, semantic retrieval is a token tax\.On localization where the issue text typically*names*the symbol,semantic retrieval costs more tokens, not fewer\. Decisively, in arm C the agent used the LSPzero times— C collapses onto A\. Forcing semantic\-first \(D\)*reduced*success and the agent still resisted \(4 semantic vs\. 51 grep calls\)\. This confirmsPrediction 2,*refutes*the localization half ofPrediction 1\(for symbol\-named localization, semantic retrieval is a tax\), and confirmsPrediction 3emphatically: the habit gap is so strong that forcing backfires\.

### 6\.2Model sweep — LSP helps weak models, taxes strong ones

Same localization tasks, three models, 72 episodes each\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x2.png)Figure 2:Capability dependence and the universal grep habit\.*Left:*the LSP token premium over grep, by model\. Only the weakest model \(Haiku\)*saves*tokens with the LSP \(−26%\-26\\%\); both capable models pay a premium \(Opus\+6%\+6\\%, Sonnet\+118%\+118\\%\)\.*Right:*how often each model uses a semantic tool in the free\-choice arm C\. Every model defaults to grep — even Haiku, which would save26%26\\%, uses the LSP only6%6\\%of the time\. Giving the tool is not enough\.Thecapability dependenceis the headline: the weakest model \(Haiku — which flails with grep noise,∼12\{\\sim\}12k tokens and 13\+ turns per task\) is the*only*one thatsavestokens with the LSP; both capable models pay a premium\. Semantic retrieval is a crutch for weak models and a tax for strong ones\. And the grep preference isuniversal: free\-choice semantic use is0%/4%/6%0\\%/4\\%/6\\%\.

### 6\.3Reference\-completeness — LSP buys precision, not tokens, and cannot fix recall

Task = list*every*call site of a target function, scored by F1 againstpyright’s reference set\. \(We switched the oracle frompylsptopyrightafter finding jedi’s reference resolution too incomplete to serve as ground truth — itself a finding about LSP quality variance\.\) Five cross\-filerequestsfunctions×\\times4 arms×\\times3 rollouts, Opus 4\.8\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x3.png)Figure 3:Reference\-completeness: the LSP buys precision, not recall\.Precision, recall, and F1 by arm\. The LSP arm \(B\) achieves*perfect precision*\(1\.00 — zero false call sites\) versus grep’s 0\.76, and wins on F1\. Butrecall is identical \(∼\\sim0\.66\) across all arms: no tool helps the agent find*more*true call sites\. The missing third is an agent\-thoroughness ceiling, not a retrieval problem\.Here — onfind\_references’ home turf —the LSP wins on accuracy\(F10\.7780\.778vs\.0\.7060\.706\), the*opposite*of localization: the task class decides whether semantic retrieval helps\. The decomposition is the real result\. The entire gain isprecision: B reports zero false call sites \(1\.00\) vs\. grep’s 0\.76 \(∼\\sim24% of grep hits are false positives — comments, strings, unrelated same\-named symbols\)\.Recall is identical \(∼\\sim0\.66\) across all four arms: neither tool helps the agent find*more*true sites — the missing third is an*agent\-thoroughness*problem, not a retrieval one, which the LSP cannot fix\. The precision gain costs∼\\sim19% more tokens, appears on every cross\-file target \(\+0\.06\+0\.06to\+0\.13\+0\.13F1\), and vanishes on the one single\-file target \(both arms F1=1\.00=1\.00\)\. Free\-choice arm C reverts to grep’s profile — the agent forgoes the precision gain even where it exists\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x4.png)Figure 4:Why the LSP costs\+19%\+19\\%tokens on reference tasks\.Per\-tool\-call token cost is nearly identical between arms \(163163vs\.160160\); the difference is*call/turn count*\.*Left:*the LSP arm issues more tool calls per episode\.*Right:*it issues far moreread\_filecalls\.find\_referencesreturns*locations only*\(path:line\), so the agent must open those files to verify each call site;grepreturns the matching line’s*content inline*, often needing no follow\-up read\. Each extra turn re\-bills the accumulated conversation context — so the cost compounds\. An LSP tool that returned the referenced line inline would likely erase the penalty while keeping precision=1\.00=1\.00\(an actionable H\-interface fix\)\.
### 6\.4Reference\-completeness across models, and task\-dependent tool choice

We ran the reference experiment on all three models \(60 episodes each\)\. Two results stand out\.

First, the accuracy benefit ismodel\-independent: the LSP lifts F1 for all three models, with precision→1\.00\\to 1\.00for the two capable models and0\.930\.93for Haiku\. Second, the*token*picture is milder than localization: the premium shrinks sharply relative to the localization tax \(Sonnet\+118%\+118\\%on localization→\+12%\\to\+12\\%here; Haiku−26%→−7%\-26\\%\\to\-7\\%\) but stays slightly positive for the two strong models\. Unlike localization, here the premium*buys real F1*\. A clean token saving appears only for the weakest model \(Haiku,−7%\-7\\%\)\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x5.png)Figure 5:Tool choice is task\-dependent — the agent has latent routing competence\.Semantic\-tool use in the free\-choice arm C, by model, for the two task classes\. On*localization*every model defaults togrep\(0–6%\); on*reference\-completeness*the same models reach for the LSP roughly half the time,*unprompted*\(45–57%\)\. The grep preference is not absolute — it is task\-shaped\. The policy already partially recognizes when “find all callers” calls forfind\_references\.This second result is the most important refinement in the study\. Our localization data suggested a*universal*grep preference \(arm C used the LSP0%0\\%/4%4\\%/6%6\\%of the time\)\. The reference\-completeness data overturns the universality: the*same*agents, given the*same*free choice, use semantic tools45%45\\%/50%50\\%/57%57\\%of the time when the task is reference\-shaped \(Figure 5\)\. So the grep default istask\-dependent, not a fixed bias— the policy already has substantial latent routing competence and exercises it when the task obviously suits semantic navigation\. This strengthens, rather than weakens, the case for an adaptive router*and*for training the routing into the policy: the signal is demonstrably already there to be reinforced\.

### 6\.5What actually determines the LSP’s value: lexical noise, not language

The reference experiment so far compared one Python repo \(requests\) and one TypeScript repo \(remeda\), and found the LSP helped on the former \(Δ\\DeltaF1\+0\.072\+0\.072\) but not the latter \(\+0\.000\+0\.000\)\. That invites the wrong conclusion — “TypeScript doesn’t benefit from semantic retrieval\.” It is confounded:remedais not just TypeScript, it is a*clean*codebase \(distinctive function names, imported\-and\-called consistently\), sogrepalready achieved precision1\.001\.00and the LSP had nothing to add\. To separate*language*from*codebase naming noise*, we added a third repo:hono, a TypeScript web framework with deliberately*noisy*target names \(html,stream,parseAccept\) that collide with comments, strings, tests, and similarly\-named symbols\.

The resultgroups by noise, not language\. Both noisy repos — Python*and*TypeScript — show a clear LSP advantage \(\+0\.072\+0\.072,\+0\.245\+0\.245\) with low grep precision; the clean repo shows none\. The two TypeScript repos sit at*opposite extremes*\(\+0\.000\+0\.000vs\.\+0\.245\+0\.245\), separated purely by naming noise\. On the noisiest repo \(hono\) the LSP not only gives the largest F1 gain but also*saves*tokens \(−12%\-12\\%\): when grep is flooded with false positives, the agent reads more to filter them, so semantic retrieval is strictly better\.

The cleanest evidence is*within*a single repo, which eliminates any cross\-repo or cross\-language confound entirely\. Rankinghono’s six targets by grep precision:

As grep’s precision falls, the LSP’s benefit rises, nearly monotonically — same repo, same language, same model\.decodeBase64\(grep precision1\.001\.00\) gets zero benefit;parseAccept\(grep precision0\.070\.07, essentially drowned in false positives\) gets\+0\.550\+0\.550\. Pooling all targets across all three repos \(Figure 6\) gives a single descending relationship \(slope≈−0\.49\{\\approx\}\-0\.49\): every point, regardless of language or repo, falls on the same line\.

![Refer to caption](https://arxiv.org/html/2608.13568v1/x6.png)Figure 6:The LSP helps exactly whengrepis noisy — one relationship across repos and languages\.Each point is one target function: x\-axis isgrep’s precision on that target \(1\.0 = no lexical false positives\), y\-axis is the LSP’s F1 benefit overgrep\. Points from all three repos \(Python and TypeScript, clean and noisy\) lie on a single descending trend: the LSP’s value is determined by the target’s*lexical collision rate*, not by the programming language or its type system\. The clean TypeScript repo \(remeda, green\) clusters at precision1\.01\.0/ benefit0; the noisy TypeScript repo \(hono, red\) spreads across the high\-benefit region exactly wheregrep’s precision is low\.This is the study’s sharpest causal claim, and it resolves the apparent “TypeScript doesn’t benefit” result:whether semantic retrieval helps a reference\-finding task is governed by the symbol’s naming collision rate, not by the language\.The practical rule for an adaptive router follows directly: route to the LSP when the identifier is lexically ambiguous \(a cheapgrepcan estimate its own noise\), and stay lexical when it is distinctive\.

### 6\.6Synthesis

Across two task classes and three models, a single conditional structure holds:

> “Does an LSP save tokens?” — in general, no\.On symbol\-named*localization*it*costs*tokens \(a tax that grows with model strength\)\. On*reference\-completeness*it does not save tokens either \(a smaller\+12\+12–19%19\\%premium for capable models\) but the premium now buys*precision*\(1\.001\.00vs\.0\.760\.76\) and cannot fix the recall ceiling\. It saves tokens only for the*weakest*model, as a crutch against lexical noise\. And the agent’s tool choice istask\-dependent: it defaults togrepon localization \(0–6%6\\%semantic use\) but reaches for the LSP about half the time on reference tasks \(4545–57%57\\%\) — the grep preference is not a fixed bias but a learned, task\-shaped policy\.

This argues against “LSP\-always” and for\(a\)an adaptive router keyed on task class, model capability, and lexical\-noise level, and\(b\)training the*when\-to\-go\-semantic*competence into the policy — because handing the model the tool is empirically insufficient\.

### 6\.7Edit tasks with real test execution

Localization and reference\-completeness are*retrieval*tasks scored against a static oracle\. What matters most for an agent is whether the LSP helps it*edit code correctly*, where correctness is defined by*execution*\. Because the SWE\-bench Docker images do not run under emulation on the available arm64 host, we built a local, SWE\-bench\-style harness on modernrequeststhat runs natively \(Python 3\.11\): the agent edits the working tree via anedit\_filetool, then the task’s test patch is applied and the target tests run\. Resolution is realpass@1\.

Single\-file edits \(6 real bugfix commits\)\.Grep\-only leads \(pass@10\.830\.83\); adding \(C,0\.580\.58\) or forcing \(D,0\.750\.75\) the LSP does not help, and the LSP again carries a token premium\. Failures are*wrong\-rewrite*and*no\-edit*, never missed\-site — but a single\-file task*cannot*exhibit a cross\-file recall miss, so it cannot test whether the LSP’s recall ceiling propagates into edit failures\. That needs multi\-file edits\.

Multi\-file renames \(6 helpers used across 2–6 files\)\.The agent must update every call site; a single miss leaves the old name dangling \(caught byimport requestsor a residual\-name grep\)\. We also measure*site\-recall*= fraction of referencing files fully converted\. The prediction is confirmed:

Two confounds had to be removed first, and isolating them is what made the mechanism precise\.*\(i\) Backend completeness:*pylsp\(jedi\) returns only∼13\{\\sim\}\\tfrac\{1\}\{3\}of cross\-file references on dynamic Python \(5 of 15 forto\_native\_stringvs\.pyright’s 14\); an LSP that under\-reports references fails a rename regardless of formatting, so we usepyright\.*\(ii\) Index warmup:*pyrightresolves references lazily per file, so a coldfind\_referencesissued*at a definition*— the natural query for a rename — returns only the definition \(11result\) until the referencing files are opened; a fixed wait does not help \(11after4040s\), but opening all source files does \(1→141\\\!\\to\\\!14\)\.An LSP\-backed agent that does not warm the index silently gets incomplete references on exactly the queries a refactor depends on\.This warmup is not free — opening every file and letting the server settle costs∼15\{\\sim\}15s per episode even onrequests\(1919files\), growing with repo size — and it is*not*pyright\-specific: a coldtextDocument/referencesat the definition ofremeda’spurry\(used across108108TypeScript files\) likewise returns11until indexing completes \(∼10\{\\sim\}10s\)\. Production LSP\-MCP servers treat this as table stakes: Serena ships a build\-timeserena project indexthat persists symbols to an on\-disk cache,*and*a runtime guard\_wait\_for\_cross\_file\_references\_if\_needed\(\)whose comment notes that some servers “require waiting \[…\] before they can return accurate cross\-file results \[…\] after at least one file was opened\.” A warmed, ideally pre\-cached, index is thus a prerequisite for any LSP\-backed agent, not optional tuning\.

With a complete, warmed backend we isolate the practical question: does returning each reference’s line*inline*\(as production LSP\-MCP servers such as Serena do\) beat the bare\-location default? Arm F attaches±2\\pm 2source lines to eachpyrightreference\.It helps substantially and via the predicted mechanism: vs\. B,pass@10\.67→0\.830\.67\\\!\\to\\\!0\.83, site\-recall0\.930→0\.9580\.930\\\!\\to\\\!0\.958, tokens−19%\-19\\%, and file\-reads/episode15\.2→3\.215\.2\\\!\\to\\\!3\.2\(below grep’s4\.34\.3\) — the location\-only arm wastes its turns reading each file back to see the call site\.Yet even F still loses to grep, and the residual is fundamental: B and F both fail thedefault\_hooksrename by missing the same site, a*comment*\(\# …default\_hooks\(\) return type\)\.pyrightreturns the six real references but never the comment — by design, a comment is not a semantic reference\. A rename is a*textual*operation; semantic references are a strict subset of textual occurrences, sofind\_referencescannot match grep’s completeness on edits spanning non\-code text\. For editing, grep is the better default; if an LSP is used, a warmed index and inline\-context results are table stakes that recover most — but not all — of grep’s edge\.

## 7Limitations and threats to validity

The results arepreliminary:\(i\) few repositories, small N—requestsplusremeda/hono\(TypeScript\), small libraries the models likely saw in pretraining; 6 localization tasks, 5–6 reference targets, and 6 edit / 6 rename tasks, 2–3 rollouts each\. Directions are robust; effect sizes \(e\.g\. Sonnet’s\+118%\+118\\%\) are preliminary and will move with more data\.\(ii\) The edit results are local, not SWE\-bench\-scored: SWE\-bench Docker images do not run under emulation on the available arm64 host, so edit tasks use a locally built set \(realrequestsbugfix commits and constructed multi\-file renames\) run natively\. This is sound for the*relative*arm comparison but thepass@1values are not comparable to a standard SWE\-bench leaderboard\.\(iii\) Oracle qualitydrove a mid\-study change frompylsptopyright\(jedi was too incomplete to be ground truth\) — itself a finding, and a caution that any single LSP\-as\-truth inherits that server’s blind spots\.\(iii\) Task coverage— we did not run the*edit*tasks end\-to\-end with test execution, arm E \(static repo\-map\), or large repositories where per\-symbol LSP cost and the static\-index trade\-off would bite\.\(iv\) Language/server— Python withpylsp/pyright; the originally\-motivatingTypeScriptcase \(statically typed, most reliable references\) is untested and is the most important next stratum\.\(v\) Harness specificityand\(vi\) token accounting\(we count total context tokens; caching can change billed cost\)\.\(vii\) Provenanceas in §2\.

## 8Conclusion

“Does an LSP save tokens” is not a tooling opinion — it is a measurable statement about retrieval precision under a token budget, and it has gone unmeasured in public while being near\-universally asserted\. We reduced it to a single metric \(*tokens\-to\-success at iso\-accuracy*\), a five\-arm ablation, and a mapping of three failure modes onto movable variables — and ran a preliminary study that turns the prior into first numbers\. The answer isconditional, and in the common case negative: on symbol\-named localization the LSP*costs*tokens \(a tax growing with model strength\); on reference\-completeness it buys*precision*, not token savings, at a∼\\sim19% premium and cannot lift the recall ceiling; it nets a saving only for the weakest model\. Underneath sits the most robust finding:every model defaults togrepwhen the LSP is merely available\.That is why the durable solution is not “add an LSP” but to make the routing — and eventually the semantic competence itself —native to the policy rather than a brittle external layer, the direction RL\-from\-language\-server\-feedback already points\. The contribution is to replace an assertion with a measurement, and a blanket recommendation with a conditional one\. The honest next step is scale: more and larger repositories, real SWE\-bench\-scored edits on Docker\-capable hardware, and a second model family\.

## Code and data availability

All code \(harness, task builders, verifiers, analysis\) and raw per\-episode results are public at[https://github\.com/Poytr1/lsp\-vs\-grep\-token\-study](https://github.com/Poytr1/lsp-vs-grep-token-study)\. Key result files:harness/runs/v1\_all\_arms\.jsonl\(localization\),ref\_opus\.jsonl/ref\_ts\_\*\.jsonl\(reference\-completeness\),edit\_opus\_verified\.jsonl\(single\-file edits\), andrename\_opus\_verified\.jsonl/rename\_warm\_BF\_verified\.jsonl\(multi\-file renames; location\-only vs\. text\-inline\)\. Each arm is a tool surface behind one identical agent loop; figures regenerate viaharness/make\_figs\.py\.

## How to cite

```
@misc{xu2026lsptokens,
  title  = {Does a Language Server Save Tokens for Coding Agents?
            A Measurement Methodology and Preliminary Study},
  author = {Xu, Pengcheng},
  year   = {2026},
  note   = {Preprint},
  howpublished = {\url{https://github.com/Poytr1/lsp-vs-grep-token-study}}
}
```

## Appendix ASource provenance

Sources were handled in two tiers\.*Fetch\-verified \(F\)*— retrieved and read this study: Serena\[[7](https://arxiv.org/html/2608.13568#bib.bib7)\], mcp\-language\-server\[[8](https://arxiv.org/html/2608.13568#bib.bib8)\], Aider repo\-map\[[9](https://arxiv.org/html/2608.13568#bib.bib9)\], and arXiv:2604\.18413\[[3](https://arxiv.org/html/2608.13568#bib.bib3)\], 2510\.22907\[[4](https://arxiv.org/html/2608.13568#bib.bib4)\], 2606\.11864\[[5](https://arxiv.org/html/2608.13568#bib.bib5)\], 2605\.18747\[[6](https://arxiv.org/html/2608.13568#bib.bib6)\]\(full abstracts captured\)\.*Established background \(K\)*— relied on from training knowledge and never used for a novel quantitative claim: SWE\-bench\[[1](https://arxiv.org/html/2608.13568#bib.bib1)\], the SWE\-agent agent\-computer\-interface\[[2](https://arxiv.org/html/2608.13568#bib.bib2)\], AutoCodeRover\[[10](https://arxiv.org/html/2608.13568#bib.bib10)\], and Moatless\[[11](https://arxiv.org/html/2608.13568#bib.bib11)\]\.

## References

- \[1\]C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan\.SWE\-bench: Can Language Models Resolve Real\-World GitHub Issues?*arXiv:2310\.06770*, 2023\.
- \[2\]J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press\.SWE\-agent: Agent\-Computer Interfaces Enable Automated Software Engineering\.*arXiv:2405\.15793*, 2024\.
- \[3\]TypeScript Repository Indexing for Code Agent Retrieval\.*arXiv:2604\.18413*, 2026\.
- \[4\]Reinforcement Learning from Compiler and Language Server Feedback\.*arXiv:2510\.22907*, 2025\.
- \[5\]CORE\-Bench: Code Retrieval in the Era of Agentic Coding\.*arXiv:2606\.11864*, 2026\.
- \[6\]Code as Agent Harness\.*arXiv:2605\.18747*, 2026\.
- \[7\]Oraios AI\. Serena: A coding agent toolkit with semantic retrieval and editing\.[https://github\.com/oraios/serena](https://github.com/oraios/serena), 2024–2026\.
- \[8\]I\. Phillips\. mcp\-language\-server: An MCP server exposing language\-server tools\.[https://github\.com/isaacphi/mcp\-language\-server](https://github.com/isaacphi/mcp-language-server), 2024–2026\.
- \[9\]Aider\. Repository map\.[https://aider\.chat/docs/repomap\.html](https://aider.chat/docs/repomap.html), 2023–2026\.
- \[10\]Y\. Zhang, H\. Ruan, Z\. Fan, and A\. Roychoudhury\.AutoCodeRover: Autonomous Program Improvement\.*arXiv:2404\.05427*, 2024\.
- \[11\]A\. Antonio \(Moatless Tools\)\.Moatless Tools: LLM tooling for code editing\.[https://github\.com/aorwall/moatless\-tools](https://github.com/aorwall/moatless-tools), 2024\.