Accelerating Large Language Model Inference with Self-Supervised Early Exits

arXiv cs.CL Papers
large-language-models inference-acceleration early-exits self-supervised-learning speculative-decoding pythia entropy

Summary

This paper introduces a self-supervised early exit method for LLMs, allowing computation to stop early at intermediate layers when confidence is high, thereby reducing inference cost. It also presents Dynamic Self-Speculative Decoding (DSSD) which achieves higher token acceptance than existing baselines.

arXiv:2407.21082v3 Announce Type: replace Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers. Each head is trained in a self-supervised manner to mimic the main model's predictions, allowing computation to stop early when a calibrated confidence threshold is reached. We evaluate several confidence metrics and show that entropy provides the most reliable separation between correct and incorrect predictions. Experiments on the Pythia model suite (70M to 2.8B parameters) demonstrate that our method significantly reduces inference cost while maintaining accuracy across multiple benchmarks. We further adapt this approach to speculative decoding, introducing Dynamic Self-Speculative Decoding (DSSD), which achieves 1.66x higher token acceptance than manually-tuned LayerSkip baselines with minimal hyperparameter tuning.
Original Article
View Cached Full Text

Cached at: 07/13/26, 08:00 AM 

# Accelerating Large Language Model Inference with Self-Supervised Early Exits
Source: [https://arxiv.org/abs/2407.21082](https://arxiv.org/abs/2407.21082)
## Computer Science \> Computation and Language

**arXiv:2407\.21082**\(cs\)

\[Submitted on 30 Jul 2024 \([v1](https://arxiv.org/abs/2407.21082v1)\), last revised 10 Jul 2026 \(this version, v3\)\]

[View PDF](https://arxiv.org/pdf/2407.21082)

> Abstract:This paper presents a modular approach to accelerate inference in large language models \(LLMs\) by adding early exit heads at intermediate transformer layers\. Each head is trained in a self\-supervised manner to mimic the main model's predictions, allowing computation to stop early when a calibrated confidence threshold is reached\. We evaluate several confidence metrics and show that entropy provides the most reliable separation between correct and incorrect predictions\. Experiments on the Pythia model suite \(70M to 2\.8B parameters\) demonstrate that our method significantly reduces inference cost while maintaining accuracy across multiple benchmarks\. We further adapt this approach to speculative decoding, introducing Dynamic Self\-Speculative Decoding \(DSSD\), which achieves 1\.66x higher token acceptance than manually\-tuned LayerSkip baselines with minimal hyperparameter tuning\.

## Submission history

From: florian valade \[[view email](https://arxiv.org/show-email/5fa99cf3/2407.21082)\] \[via CCSD proxy\] **[\[v1\]](https://arxiv.org/abs/2407.21082v1)**Tue, 30 Jul 2024 07:58:28 UTC \(7,190 KB\) **[\[v2\]](https://arxiv.org/abs/2407.21082v2)**Thu, 12 Feb 2026 09:41:15 UTC \(8,307 KB\) **\[v3\]**Fri, 10 Jul 2026 12:19:50 UTC \(1,154 KB\)

Bibliographic Tools

## Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Code, Data, Media

## Code, Data and Media Associated with this Article

Demos

## Demos

Related Papers

## Recommenders and Search Tools

About arXivLabs

## arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website\.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy\. arXiv is committed to these values and only works with partners that adhere to them\.

Have an idea for a project that will add value for arXiv's community?[**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html)\.

Similar Articles