BCMT: Blockwise Causal Memory Transformer

arXiv cs.CL Papers

Summary

BCMT introduces a blockwise causal memory transformer architecture that decouples local and global dependencies in long-context language modeling, improving efficiency while maintaining performance comparable to dense transformers.

arXiv:2608.13578v1 Announce Type: new Abstract: Transformer architectures rely on dense self-attention to model long-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length. We introduce BCMT (Blockwise Causal Memory Transformer), an architecture for long-context language modeling that decouples local token interactions from global context propagation. Dense causal self-attention is applied independently within local blocks, while each block produces an adaptive summary aggregated through an exponential causal memory. This memory is subsequently injected back into the token representations, enabling efficient propagation of long-range contextual information without relying on explicit global attention. Unlike standard Transformers and recurrent memory architectures, BCMT maintains neither dense interactions between distant tokens nor learned memory states. Its memory mechanism is fully parallelizable and remains compatible with standard implementations of dense self-attention. Experiments on language modeling with context lengths of up to 1024 tokens show that BCMT achieves validation performance comparable to that of Dense Transformers while significantly improving training throughput and reducing memory consumption. An ablation study further confirms that these improvements arise from the proposed memory mechanism. These results demonstrate that an exponential causal memory constructed from block summaries provides an effective alternative to dense global attention mechanisms for long-context language modeling.
Original Article
View Cached Full Text

Cached at: 08/17/26, 09:39 AM

# BCMT: Blockwise Causal Memory Transformer
Source: [https://arxiv.org/html/2608.13578](https://arxiv.org/html/2608.13578)
Rachid Arezki Independent Researcher France, Nîmes rachid\.arezki\.pro@gmail\.com

###### Abstract

Transformer architectures rely on dense self\-attention to model long\-range dependencies, but this mechanism exhibits quadratic complexity with respect to sequence length\.

We introduce BCMT \(Blockwise Causal Memory Transformer\), an architecture for long\-context language modeling that decouples local token interactions from global context propagation\. Dense causal self\-attention is applied independently within local blocks, while each block produces an adaptive summary aggregated through an exponential causal memory\. This memory is subsequently injected back into the token representations, enabling efficient propagation of long\-range contextual information without relying on explicit global attention\.

Unlike standard Transformers and recurrent memory architectures, BCMT maintains neither dense interactions between distant tokens nor learned memory states\. Its memory mechanism is fully parallelizable and remains compatible with standard implementations of dense self\-attention\.

Experiments on language modeling with context lengths of up to 1024 tokens show that BCMT achieves validation performance comparable to that of Dense Transformers while significantly improving training throughput and reducing memory consumption\. An ablation study further confirms that these improvements arise from the proposed memory mechanism\.

These results demonstrate that an exponential causal memory constructed from block summaries provides an effective alternative to dense global attention mechanisms for long\-context language modeling\.

## 1Introduction

Transformer architectures have become the dominant paradigm for autoregressive language modeling owing to the expressive power of self\-attention\[[22](https://arxiv.org/html/2608.13578#bib.bib1)\]\. By allowing each token to interact with all preceding tokens, they effectively model complex dependencies over long sequences\.

This flexibility, however, comes at a high computational cost\. Causal self\-attention exhibits quadratic complexity with respect to sequence length, which limits its applicability as the context becomes longer\.

Numerous approaches have therefore been proposed as alternatives to dense global attention, including sparse attention mechanisms\[[25](https://arxiv.org/html/2608.13578#bib.bib3),[1](https://arxiv.org/html/2608.13578#bib.bib2)\], recurrent memory architectures\[[5](https://arxiv.org/html/2608.13578#bib.bib6)\], and state\-space models\[[8](https://arxiv.org/html/2608.13578#bib.bib12),[21](https://arxiv.org/html/2608.13578#bib.bib10),[13](https://arxiv.org/html/2608.13578#bib.bib11)\]\. Although these approaches improve scalability, they generally rely either on modifying the attention structure or on introducing more complex recurrent mechanisms\.

In this work, we propose BCMT \(Blockwise Causal Memory Transformer\)111Official implementation:[https://github\.com/rachidlabs/BCMT](https://github.com/rachidlabs/BCMT), an architecture based on a different principle\. The central idea is to decouple the modeling of local dependencies from global context propagation\. Token interactions are captured by dense causal self\-attention applied independently within local blocks, while long\-range dependencies are propagated through an exponential causal memory constructed from adaptive block summaries\.

The resulting memory representations are then injected back into the token representations through a gating mechanism, enabling the propagation of global contextual information without relying on explicit attention between tokens belonging to different blocks\. Unlike recurrent memory architectures, the memory mechanism of BCMT relies neither on learned hidden states nor on persistent memory tokens\. Instead, it is obtained through a fully parallelizable exponential causal aggregation that remains compatible with standard GPU implementations of dense self\-attention\.

We evaluate BCMT on language modeling tasks with context lengths of up to 1024 tokens\. The results show that the proposed architecture achieves validation performance comparable to that of a Dense Transformer while significantly improving training throughput and reducing memory consumption\. An ablation study further confirms that these improvements arise from the proposed memory mechanism rather than from the sequence partitioning alone\.

Our main contributions are as follows:

- •We propose BCMT, an architecture that explicitly decouples local token interactions from global context propagation through an inter\-block exponential causal memory\.
- •We introduce an exponential causal memory mechanism constructed from adaptive block summaries, enabling efficient modeling of long\-range dependencies without explicit global attention\.
- •We experimentally demonstrate that BCMT preserves modeling performance comparable to that of a Dense Transformer while significantly improving computational efficiency, training throughput, and memory consumption\.

## 2Related Work

### 2\.1Scaling Self\-Attention to Long Sequences

The success of Transformer architectures largely stems from the expressive power of dense self\-attention\[[22](https://arxiv.org/html/2608.13578#bib.bib1)\]\. However, the quadratic complexity of this mechanism with respect to sequence length quickly limits its applicability to long\-context settings\.

A first family of approaches reduces this cost through sparse attention patterns\. Longformer restricts attention to sliding local windows complemented by a limited number of global tokens\[[1](https://arxiv.org/html/2608.13578#bib.bib2)\], while BigBird combines local, random, and global connections to approximate a dense attention graph\[[25](https://arxiv.org/html/2608.13578#bib.bib3)\]\. These methods reduce the computational cost while preserving explicit interactions between distant tokens\.

A second family relies on approximations of the attention operator itself\. Linformer projects keys and values into a lower\-dimensional space\[[23](https://arxiv.org/html/2608.13578#bib.bib4)\], whereas Performer replaces softmax attention with an approximation based on random feature mappings\[[3](https://arxiv.org/html/2608.13578#bib.bib5)\]\. These approaches improve scalability at the cost of modifying the original formulation of self\-attention\.

Finally, more recent architectures designed for very long contexts exploit structured attention mechanisms\. LongNet introduces dilated attention, enabling a substantial increase in the effective context while limiting the number of computed interactions\[[6](https://arxiv.org/html/2608.13578#bib.bib8)\]\. Despite their differences, all of these approaches retain an attention mechanism that explicitly connects distant tokens\.

BCMT follows a different strategy\. Attention is strictly confined to independent local blocks, while long\-range dependencies are propagated through an exponential causal memory constructed from adaptive block summaries\. Consequently, unlike the aforementioned approaches, BCMT neither extends the receptive field nor modifies the structure of the attention graph\. Instead, long\-range interactions emerge from the progressive propagation of a compact contextual representation across blocks, rather than from explicit attention between distant tokens\.

### 2\.2Local and Blockwise Sequence Processing

Another line of research consists in modifying the processing unit of the sequence by replacing global attention with the independent processing of local subsequences\. This strategy relies on the assumption that the most informative interactions are predominantly local, thereby reducing the computational cost of self\-attention while preserving compatibility with standard dense attention implementations\.

BlockBERT is a representative example of this approach\. The input sequence is partitioned into contiguous blocks, each processed independently using dense intra\-block self\-attention\[[15](https://arxiv.org/html/2608.13578#bib.bib9)\]\. This organization reduces the computational cost while naturally leveraging existing GPU implementations\.

The main limitation of these architectures is the absence of an explicit communication mechanism between blocks\. Long\-range dependencies can only be captured by increasing the block size or by introducing additional mechanisms, which progressively reduces the computational benefits of local processing\.

BCMT builds upon the principle of local blockwise processing by introducing an explicit context propagation mechanism\. Each block is summarized into an adaptive vector, and these summaries are subsequently aggregated through an exponential causal memory to construct a compact representation of the sequence history\. Long\-range dependencies are therefore propagated across blocks without reintroducing explicit attention between tokens belonging to different blocks\.

### 2\.3Sequence Compression and Pooling\-Based Architectures

Another family of approaches improves the scalability of Transformer architectures by explicitly reducing the effective sequence length through compression or pooling mechanisms\.

Set Transformer introduces learned aggregation operators that efficiently summarize a set of representations into a lower\-dimensional latent space\[[10](https://arxiv.org/html/2608.13578#bib.bib22)\]\. Funnel Transformer, in contrast, applies hierarchical pooling to progressively reduce the temporal resolution throughout the network, thereby decreasing the computational cost of deeper layers\[[4](https://arxiv.org/html/2608.13578#bib.bib23)\]\.

More recently, several approaches have investigated dynamically reducing the number of tokens\. Token Pooling selects or aggregates intermediate representations to reduce the computational cost of subsequent layers\[[11](https://arxiv.org/html/2608.13578#bib.bib24)\], whereas PoolingFormer partially replaces attention operations with parameterized pooling mechanisms capable of capturing contextual information at low computational cost\[[24](https://arxiv.org/html/2608.13578#bib.bib25)\]\.

These approaches share a common principle: they use compressed representations to reduce the complexity of subsequent processing\. BCMT also relies on block summaries obtained through adaptive pooling, but with a different objective\. The compressed representations neither replace the original tokens nor reduce the sequence length processed by subsequent layers\. Instead, they constitute an auxiliary memory channel dedicated to propagating context across blocks\.

Consequently, compression is not used as a mechanism for reducing computational cost, but rather as a memory support\. The adaptive summaries are aggregated through an exponential causal memory to propagate a compact representation of the sequence history, while the local token representations are preserved and continue to be processed by dense self\-attention within each block\.

### 2\.4Hierarchical and Multi\-Scale Sequence Modeling

Hierarchical representations have long been investigated as a means of modeling dependencies across multiple temporal scales\. Early work on recurrent neural networks already explored mechanisms for temporal abstraction and memory compression\[[7](https://arxiv.org/html/2608.13578#bib.bib17),[18](https://arxiv.org/html/2608.13578#bib.bib18)\]\. These ideas subsequently evolved into hierarchical recurrent architectures and multi\-scale latent sequence models\.

More recently, these principles have been adapted to Transformer architectures\. Hierarchical Autoregressive Transformers exploit representations operating at different linguistic resolutions\[[12](https://arxiv.org/html/2608.13578#bib.bib13)\], while Hierarchical Resolution Transformers introduce multi\-scale processing inspired by wavelet decompositions through multiple levels of abstraction\[[17](https://arxiv.org/html/2608.13578#bib.bib14)\]\. Other works propose hierarchical memory organizations to structure long\-term reasoning or the pretraining of large language models\[[20](https://arxiv.org/html/2608.13578#bib.bib15),[14](https://arxiv.org/html/2608.13578#bib.bib16)\]\.

BCMT shares with these approaches the idea that compact representations can facilitate the modeling of long\-range dependencies\. However, its objective is fundamentally different\. Unlike hierarchical architectures, BCMT neither constructs multiple levels of representation nor builds a hierarchy of progressively coarser resolutions\. Instead, it relies on a single level of adaptive block summaries, aggregated through an exponential causal memory to progressively propagate context along the sequence\.

Thus, BCMT does not seek to learn a hierarchy of representations, but rather to decouple local token processing from global context propagation\. The exponential causal memory acts as a compact context propagation channel between blocks while preserving dense local self\-attention within each block\.

### 2\.5Memory\-Augmented and Recurrent Transformer Architectures

Another major direction for modeling long contexts consists of augmenting Transformer architectures with explicit memory or recurrent mechanisms\.

Transformer\-XL introduces segment\-level recurrence by reusing the hidden states of previous segments, thereby extending the effective context window\[[5](https://arxiv.org/html/2608.13578#bib.bib6)\]\. Compressive Transformer further develops this idea by maintaining compressed historical representations across multiple temporal scales\[[16](https://arxiv.org/html/2608.13578#bib.bib7)\]\.

More recently, several works have proposed integrating persistent memory directly into Transformer architectures\. Recurrent Memory Transformer \(RMT\) introduces dedicated memory tokens that are propagated across successive segments\[[2](https://arxiv.org/html/2608.13578#bib.bib19)\]\. Hierarchical Memory Transformer \(HMT\) organizes these memory representations into multiple hierarchical levels to simultaneously capture short\-, medium\-, and long\-range dependencies\[[9](https://arxiv.org/html/2608.13578#bib.bib20)\]\. Along the same line,Diagonal Batching Unlocks Parallelism in Recurrent Memory Transformersintroduces a computation reordering strategy that enables the parallel execution of RMT\-based architectures while exactly preserving their recurrent dynamics\. This approach significantly improves the hardware efficiency of recurrent memory architectures without modifying their underlying formulation\[[19](https://arxiv.org/html/2608.13578#bib.bib21)\]\.

In parallel, other works have proposed replacing self\-attention with learned recurrent dynamics\. RetNet\[[21](https://arxiv.org/html/2608.13578#bib.bib10)\], RWKV\[[13](https://arxiv.org/html/2608.13578#bib.bib11)\], and Mamba\[[8](https://arxiv.org/html/2608.13578#bib.bib12)\]model long\-range dependencies through recurrently updated internal states rather than explicit interactions between tokens\.

BCMT shares with memory\-augmented architectures the idea of propagating a compact representation of the context across segments instead of maintaining dense attention over the entire sequence\. However, its memory mechanism differs fundamentally from previous approaches\. Unlike Transformer\-XL, RMT, or HMT, BCMT relies neither on recurrent hidden states nor on persistent memory tokens, and it does not learn any recurrent dynamics\.

The memory in BCMT is constructed directly from adaptive block summaries produced after local self\-attention\. These summaries are then aggregated through a normalized exponential causal memory, yielding a compact and stable representation of the sequence history\. This construction is entirely deterministic, fully parallelizable, and independent of any learned recurrent memory state\.

Thus, BCMT differs from existing memory\-based architectures by replacing recurrent mechanisms with context propagation based on an exponential causal aggregation of block summaries, while preserving dense local self\-attention within each block\.

### 2\.6Positioning of BCMT

The approaches reviewed above illustrate several strategies for extending the capabilities of Transformer architectures to long\-context modeling, including reducing attention connectivity, approximating the attention operator, local blockwise processing, hierarchical representation compression, and the introduction of explicit memory or recurrent mechanisms\.

BCMT differs from these approaches by combining two simple principles: dense causal self\-attention strictly confined to independent local blocks and an exponential causal memory constructed from adaptive summaries of these blocks\. Consequently, long\-range dependencies are neither modeled through global attention nor stored in learned recurrent states or persistent memory tokens\.

Rather than extending the receptive field of self\-attention or maintaining a memory state that evolves throughout the sequence, BCMT progressively propagates a compact contextual representation across blocks\. This explicit separation between local token interactions and global context propagation constitutes the central principle of the proposed architecture\.

Through this design, BCMT positions itself as an alternative to both global\-attention Transformers and recurrent memory architectures by introducing a context propagation mechanism that is fully parallelizable and compatible with standard implementations of dense self\-attention\.

## 3Method

BCMT is based on two complementary mechanisms\. Dense causal self\-attention is applied independently within local blocks to model interactions between nearby tokens\. In parallel, an exponential causal memory constructed from adaptive block summaries propagates global context across blocks\. This separation preserves strictly local attention while efficiently propagating long\-range dependencies\.

### 3\.1Problem Formulation

We consider an autoregressive language modeling task over a sequence of lengthTT, represented as

X=\(x1,x2,…,xT\),xt∈ℝD,X=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{T\}\),\\qquad x\_\{t\}\\in\\mathbb\{R\}^\{D\},
whereDDdenotes the dimensionality of the token representations\.

The objective is to model the conditional distributionp​\(xt∣x<t\)p\(x\_\{t\}\\mid x\_\{<t\}\)while preserving autoregressive causality\. To reduce the computational cost of self\-attention for long sequences, BCMT replaces global attention across all tokens with a combination of local attention and inter\-block memory\.

Figure 1:Overview of the BCMT architecture\.
### 3\.2Block Decomposition

The input sequence is partitioned intoNbN\_\{b\}contiguous, non\-overlapping blocks of maximum sizeLL:

X=\{X\(1\),X\(2\),…,X\(Nb\)\},X=\\left\\\{X^\{\(1\)\},X^\{\(2\)\},\\dots,X^\{\(N\_\{b\}\)\}\\right\\\},
with

Nb=⌈TL⌉\.N\_\{b\}=\\left\\lceil\\frac\{T\}\{L\}\\right\\rceil\.
Each blockX\(i\)∈ℝLi×DX^\{\(i\)\}\\in\\mathbb\{R\}^\{L\_\{i\}\\times D\}containsLi≤LL\_\{i\}\\leq Ltokens\. All blocks have sizeLL, except possibly the last one whenTTis not a multiple ofLL\.

This decomposition constitutes the fundamental principle of BCMT\. Token interactions are restricted to each individual block through dense causal self\-attention, while dependencies across blocks are handled by the memory mechanism described in the following sections\.

### 3\.3Local Multi\-Head Causal Attention \(HH\)

Each block is processed independently using a standard multi\-head causal self\-attention layer\.

For a block

X\(b\)∈ℝL×D,X^\{\(b\)\}\\in\\mathbb\{R\}^\{L\\times D\},
and for each attention head

i∈\{1,…,h\},i\\in\\\{1,\\ldots,h\\\},
whereD=h​dhD=h\\,d\_\{h\}, the query, key, and value projections are defined as

Qi\(b\)=X\(b\)​WQ,i,Ki\(b\)=X\(b\)​WK,i,Vi\(b\)=X\(b\)​WV,i\.Q\_\{i\}^\{\(b\)\}=X^\{\(b\)\}W\_\{Q,i\},\\qquad K\_\{i\}^\{\(b\)\}=X^\{\(b\)\}W\_\{K,i\},\\qquad V\_\{i\}^\{\(b\)\}=X^\{\(b\)\}W\_\{V,i\}\.
The output of headiiis then given by

Hi\(b\)=softmax​\(Qi\(b\)​\(Ki\(b\)\)Tdh\+ℳ\)​Vi\(b\),H\_\{i\}^\{\(b\)\}=\\mathrm\{softmax\}\\left\(\\frac\{Q\_\{i\}^\{\(b\)\}\(K\_\{i\}^\{\(b\)\}\)^\{T\}\}\{\\sqrt\{d\_\{h\}\}\}\+\\mathcal\{M\}\\right\)V\_\{i\}^\{\(b\)\},
whereℳ\\mathcal\{M\}denotes the standard causal mask\.

The outputs of all heads are concatenated and projected back into the representation space:

H^\(b\)=Concat​\(H1\(b\),…,Hh\(b\)\),\\hat\{H\}^\{\(b\)\}=\\mathrm\{Concat\}\\left\(H\_\{1\}^\{\(b\)\},\\dots,H\_\{h\}^\{\(b\)\}\\right\),
H\(b\)=H^\(b\)​WO,WO∈ℝD×D,H^\{\(b\)\}=\\hat\{H\}^\{\(b\)\}W\_\{O\},\\qquad W\_\{O\}\\in\\mathbb\{R\}^\{D\\times D\},
with

H\(b\)∈ℝL×D\.H^\{\(b\)\}\\in\\mathbb\{R\}^\{L\\times D\}\.
The local representations of all blocks are then concatenated along the sequence dimension:

H=\[H\(1\)H\(2\)⋮H\(Nb\)\]∈ℝT×D\.H=\\begin\{bmatrix\}H^\{\(1\)\}\\\\ H^\{\(2\)\}\\\\ \\vdots\\\\ H^\{\(N\_\{b\}\)\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{T\\times D\}\.
The tensorHHconstitutes the local representation produced by BCMT\. It serves as the starting point for constructing the adaptive block summaries, from which the inter\-block memory described in the following section is computed\.

### 3\.4Adaptive Block Summaries \(SS\)

Starting from the local representationsH\(i\)H^\{\(i\)\}, each block is summarized by a single contextual vector\. Rather than using a uniform average of the token representations, BCMT learns an adaptive weighting that aggregates the content of each block\.

For a block

H\(i\)∈ℝL×D,H^\{\(i\)\}\\in\\mathbb\{R\}^\{L\\times D\},
the token scores are obtained through a linear projection:

s\(i\)=H\(i\)​Wp,s^\{\(i\)\}=H^\{\(i\)\}W\_\{p\},
where

Wp∈ℝD,W\_\{p\}\\in\\mathbb\{R\}^\{D\},
and

s\(i\)∈ℝL\.s^\{\(i\)\}\\in\\mathbb\{R\}^\{L\}\.
The scores are then normalized using the softmax function:

α\(i\)=softmax​\(s\(i\)\),\\alpha^\{\(i\)\}=\\mathrm\{softmax\}\\left\(s^\{\(i\)\}\\right\),
with

α\(i\)∈ℝL\.\\alpha^\{\(i\)\}\\in\\mathbb\{R\}^\{L\}\.
The adaptive block summary is then computed as

S\(i\)=\(H\(i\)\)⊤​α\(i\),S^\{\(i\)\}=\\left\(H^\{\(i\)\}\\right\)^\{\\top\}\\alpha^\{\(i\)\},
where

S\(i\)∈ℝD\.S^\{\(i\)\}\\in\\mathbb\{R\}^\{D\}\.
The vectorsS\(i\)S^\{\(i\)\}provide a compact representation of each block\. They are subsequently used as the input to the exponential causal memory mechanism described in the following section\.

### 3\.5Exponential Causal Memory

The block summaries\{S\(i\)\}i=1Nb\\\{S^\{\(i\)\}\\\}\_\{i=1\}^\{N\_\{b\}\}are used to construct an inter\-block memory that propagates context throughout the sequence\.

For each blockii, we construct a memory vector obtained through a normalized causal exponential aggregation of the summaries observed up to the current block:

M\(i\)=∑j=1iαi−j​S\(j\)∑j=1iαi−j,M^\{\(i\)\}=\\frac\{\\sum\_\{j=1\}^\{i\}\\alpha^\{\\,i\-j\}S^\{\(j\)\}\}\{\\sum\_\{j=1\}^\{i\}\\alpha^\{\\,i\-j\}\},
whereα∈\[0,1\)\\alpha\\in\[0,1\)denotes the exponential decay factor\.

Unlike a uniform average, this aggregation naturally assigns greater weight to more recent summaries while preserving a progressively decreasing contribution from the entire history\.

The decay factor is fixed toα=0\.8\\alpha=0\.8\. Preliminary experiments showed that values between 0\.7 and 0\.9 yield very similar performance\. We therefore useα=0\.8\\alpha=0\.8throughout all experiments\.

The denominator normalizes the weighting coefficients so that they form a convex combination of the block summaries, regardless of the number of blocks\.

Each memory vector satisfies

M\(i\)∈ℝD\.M^\{\(i\)\}\\in\\mathbb\{R\}^\{D\}\.
The memory representations are then projected into the local representation space:

Mproj\(i\)=M\(i\)​Wm,M\_\{\\mathrm\{proj\}\}^\{\(i\)\}=M^\{\(i\)\}W\_\{m\},
where

Wm∈ℝD×D\.W\_\{m\}\\in\\mathbb\{R\}^\{D\\times D\}\.
To preserve strict autoregressive causality, the projected memory is shifted by one block:

Mcausal\(i\)=Mproj\(i−1\),M\_\{\\mathrm\{causal\}\}^\{\(i\)\}=M\_\{\\mathrm\{proj\}\}^\{\(i\-1\)\},
with

Mcausal\(1\)=𝟎𝐃\.M\_\{\\mathrm\{causal\}\}^\{\(1\)\}=\\mathbf\{0\_\{D\}\}\.
Consequently, blockiireceives information exclusively from preceding blocks and can never access its own summary\.

The resulting memory tensor is therefore

Mcausal∈ℝNb×D\.M\_\{\\mathrm\{causal\}\}\\in\\mathbb\{R\}^\{N\_\{b\}\\times D\}\.

### 3\.6Autoregressive Causality

BCMT satisfies strict autoregressive causality\. No token representation depends on information originating from future positions\. This property results from the combination of local self\-attention and the inter\-block memory mechanism\.

##### Local Self\-Attention Causality

Within each block, representations are computed using standard causal self\-attention\. The lower triangular mask ensures that each token can attend only to itself and to preceding tokens within the same block\.

Consequently, no information from future positions can be accessed through the local attention pathway\.

##### Inter\-Block Memory Causality

The adaptive summaryS\(i\)S^\{\(i\)\}is constructed solely from the local representations of blockii\. The exponential causal memory is then computed from the summaries of all blocks observed up to the current block\.

Before being injected into the local representations, this memory is shifted by one block:

Mcausal\(i\)=Mproj\(i−1\),M\_\{\\mathrm\{causal\}\}^\{\(i\)\}=M\_\{\\mathrm\{proj\}\}^\{\(i\-1\)\},
with

Mcausal\(1\)=𝟎𝐃\.M\_\{\\mathrm\{causal\}\}^\{\(1\)\}=\\mathbf\{0\_\{D\}\}\.
Consequently, blockiireceives contextual information exclusively from blocks

\{1,…,i−1\},\\\{1,\\ldots,i\-1\\\},
and can never access either its own summary or the summaries of future blocks\.

##### Global Causality

The autoregressive causality of BCMT therefore relies on three complementary properties:

- •strictly causal self\-attention within each block;
- •a memory constructed solely from the summaries of the observed blocks;
- •a causal shift of the memory before its injection into the local representations\.

Consequently, there exists no computational path through which information from a future token can influence the representation of the current token\. BCMT therefore satisfies the causality constraints required for autoregressive language modeling\.

### 3\.7Memory Broadcast to Tokens

The memory representation of each block is broadcast to all tokens belonging to that block\.

For each blockii, the memory vector

Mcausal\(i\)∈ℝDM\_\{\\mathrm\{causal\}\}^\{\(i\)\}\\in\\mathbb\{R\}^\{D\}
is replicated across theLLpositions of the block to form a matrix

M~\(i\)∈ℝL×D,\\widetilde\{M\}^\{\(i\)\}\\in\\mathbb\{R\}^\{L\\times D\},
defined by

M~t,:\(i\)=Mcausal\(i\),t=1,…,L\.\\widetilde\{M\}^\{\(i\)\}\_\{t,:\}=M\_\{\\mathrm\{causal\}\}^\{\(i\)\},\\qquad t=1,\\ldots,L\.
The matricesM~\(i\)\\widetilde\{M\}^\{\(i\)\}are then concatenated along the sequence dimension, producing the tensor

Mtokens∈ℝT×D\.M\_\{\\mathrm\{tokens\}\}\\in\\mathbb\{R\}^\{T\\times D\}\.
As a result, every token receives the same contextual representation as the other tokens within its block\. This representation is constructed exclusively from preceding blocks, in accordance with the autoregressive causality established in the previous section\.

The tensorMtokensM\_\{\\mathrm\{tokens\}\}is then combined with the local representations through the gated fusion mechanism described in the following section\.

### 3\.8Gated Memory Injection

The exponential causal memory provides contextual information that complements the local representations produced by self\-attention\. Its contribution is dynamically modulated through a learned gating mechanism\.

A gate vector is computed from the local representations:

G=σ​\(H​Wg\),G=\\sigma\(HW\_\{g\}\),
where

Wg∈ℝD×D,W\_\{g\}\\in\\mathbb\{R\}^\{D\\times D\},
and

G∈ℝT×D\.G\\in\\mathbb\{R\}^\{T\\times D\}\.
The final representation is then computed as

Y=H\+G⊙Mtokens,Y=H\+G\\odot M\_\{\\mathrm\{tokens\}\},
where⊙\\odotdenotes the element\-wise product\.

The coefficients ofGGare computed independently for each token and each representation dimension\. They control the amount of contextual information injected into the local representations: when the coefficients are close to zero, the local representation is largely preserved; when they are close to one, the contribution of the exponential causal memory becomes more significant\.

This mechanism enables the model to adaptively regulate the use of memory according to the context while preserving local information whenever it is sufficient\.

### 3\.9Conceptual Interpretation

BCMT explicitly separates the modeling of local dependencies from that of long\-range dependencies\.

Interactions between neighboring tokens are modeled directly by local causal self\-attention operating independently within each block\. More distant dependencies, in contrast, are no longer modeled through explicit interactions between all tokens in the sequence, but through the propagation of a compact memory constructed from adaptive block summaries\.

This separation introduces an explicit inductive bias into the architecture\. Self\-attention is specialized for modeling fine\-grained, high\-resolution interactions, whereas the exponential causal memory captures a more global and progressively aggregated representation of the sequence history\.

From a signal processing perspective, BCMT can be interpreted as a decomposition into two complementary scales\. The local pathway preserves the high\-frequency components associated with interactions between neighboring tokens, whereas the exponential causal memory conveys lower\-frequency information describing the global evolution of the context\.

This organization differs from that of global\-attention Transformers, in which all dependencies are modeled by a single attention mechanism\. In contrast, BCMT distributes these two functions across two complementary mechanisms, each operating at a different temporal resolution\.

### 3\.10Computational Complexity

LetTTdenote the sequence length,LLthe block size, andDDthe representation dimension\. The sequence is partitioned into

Nb=⌈TL⌉N\_\{b\}=\\left\\lceil\\frac\{T\}\{L\}\\right\\rceil
blocks\.

The computational cost of BCMT consists of four successive operations: local intra\-block attention, adaptive block summarization, exponential causal memory construction, and memory injection into the token representations\.

##### Local Attention

Each block is processed independently using dense causal self\-attention with complexity

𝒪​\(L2\)\.\\mathcal\{O\}\(L^\{2\}\)\.
Since the sequence is partitioned intoNbN\_\{b\}blocks, the overall attention complexity becomes

Nb​𝒪​\(L2\)=𝒪​\(T​L\)\.N\_\{b\}\\,\\mathcal\{O\}\(L^\{2\}\)=\\mathcal\{O\}\(TL\)\.
For a fixed block size, the attention cost therefore grows linearly with the sequence length\.

The memory required to store the attention matrices follows the same analysis and is also

𝒪​\(T​L\),\\mathcal\{O\}\(TL\),
compared with

𝒪​\(T2\)\\mathcal\{O\}\(T^\{2\}\)
for a Dense Transformer with global attention\.

##### Adaptive Block Summaries

Computing the pooling scores and the weighted summary for each block requires a single pass over the local representations\. Its complexity is

𝒪​\(Nb​L​D\)=𝒪​\(T​D\),\\mathcal\{O\}\(N\_\{b\}LD\)=\\mathcal\{O\}\(TD\),
which remains lower than the attention cost as soon asL≫D−1L\\gg D^\{\-1\}, which is the case in practice\.

##### Exponential Causal Memory

The memory is constructed by aggregating theNbN\_\{b\}block summaries through a normalized causal exponential average, followed by a linear projection and a causal shift\.

The overall complexity of these operations is

𝒪​\(Nb​D\),\\mathcal\{O\}\(N\_\{b\}D\),
that is, linear in the number of blocks\.

##### Memory Injection

Broadcasting the memory to the tokens of each block and the subsequent gated fusion rely only on element\-wise operations over representations of dimensionDD\. Their computational cost is

𝒪​\(T​D\)\.\\mathcal\{O\}\(TD\)\.

##### Overall Complexity

The dominant computational cost of BCMT is the local attention, leading to an overall complexity of

𝒪​\(T​L\)\.\\boxed\{\\mathcal\{O\}\(TL\)\}\.
When the block size is fixed \(L≪TL\\ll T\), this complexity becomes linear with respect to the sequence length, whereas that of a Transformer with global attention remains quadratic:

Dense Transformer:𝒪​\(T2\),BCMT:𝒪​\(T​L\)\.\\text\{Dense Transformer\}\\;:\\;\\mathcal\{O\}\(T^\{2\}\),\\qquad\\text\{BCMT\}\\;:\\;\\mathcal\{O\}\(TL\)\.
BCMT therefore reduces both computational and memory costs by restricting self\-attention to local blocks, while long\-range dependencies are propagated through an exponential causal memory whose computational cost is linear\.

## 4Experiments

### 4\.1Experimental Setup

#### 4\.1\.1Evaluated Configurations

All comparisons are conducted between a standard dense causal Transformer and several BCMT variants\.

To isolate the impact of the proposed memory mechanism, all models share exactly the same architecture \(number of layers, hidden dimension, and number of attention heads\)\. The only difference between the Dense Transformer and BCMT lies in the strategy used to model dependencies\.

We evaluate four BCMT variants corresponding to different block sizes:

#### 4\.1\.2Model Architecture

All models use the following configuration:

- •hidden dimension:dmodel=128d\_\{\\text\{model\}\}=128
- •number of layers:66
- •number of attention heads:88
- •batch size:1616

All models are implemented in PyTorch and trained using the same hyperparameters to ensure a fair comparison\.

#### 4\.1\.3Datasets

All experiments are conducted on the WikiText\-103 corpus, a widely used benchmark for evaluating autoregressive language models\.

Two training subsets are constructed from the training split in order to evaluate BCMT under two complementary experimental settings\.

##### WikiText200k

WikiText200k is obtained by randomly sampling 200,000 sequences from the training split222dataset\["train"\]\.shuffle\(seed=999\)\.select\(range\(200000\)\)\.

After preprocessing and tokenization, this corpus contains approximately 12\.60 million training tokens\. It is used in Phases I, II, and IV\.

##### WikiText600k

WikiText600k is constructed using the same protocol by selecting 600,000 sequences333dataset\["train"\]\.shuffle\(seed=123\)\.select\(range\(600000\)\)\.

After preprocessing and tokenization, this corpus contains approximately 38\.04 million training tokens\. It is used in Phase III to evaluate the behavior of BCMT under a more demanding training regime\.

For all experiments, evaluation is performed on the entire WikiText\-103 validation split, comprising 239,536 tokens\.

#### 4\.1\.4Training Protocol

All models are trained in an autoregressive language modeling setting using a causal mask and the AdamW optimizer\.

The learning rate is dynamically adjusted using aReduceLROnPlateauscheduler configured with:

- •reduction factor:0\.50\.5;
- •patience:22epochs;
- •improvement threshold:10−310^\{\-3\};
- •minimum learning rate:10−510^\{\-5\}\.

Training is terminated using anearly stoppingcriterion when the validation loss no longer improves, with:

- •patience:66epochs;
- •minimum required improvement \(min\_delta\):10−310^\{\-3\}\.

Training sequences are generated with 50% overlap\. Each experiment is repeated using two independent random seeds \(seed=42andseed=123\)\. The reported results correspond to the average of the metrics obtained from these two runs, and the standard deviations are computed from the same initializations\.

#### 4\.1\.5Experimental Protocol

The experimental evaluation is organized into four complementary phases\.

##### Phase I: Sensitivity to Context Length

Comparison between the Dense Transformer and BCMT\-128 for sequence lengths

seq\_len∈\{128,256,512,1024\},\\texttt\{seq\\\_len\}\\in\\\{128,256,512,1024\\\},
to evaluate the impact of context length on model performance and computational efficiency\.

##### Phase II: Comparison of BCMT Variants

Comparison between the Dense Transformer and the BCMT\-64, BCMT\-128, BCMT\-256, and BCMT\-512 variants to investigate the impact of block size on the trade\-off between modeling quality and computational efficiency\.

##### Phase III: Large\-Scale Validation

Comparison between the Dense Transformer and BCMT\-256 on the WikiText600k corpus to verify that the modeling performance, computational efficiency gains, and robustness of the proposed memory mechanism are maintained as the amount of training data increases substantially\.

##### Phase IV: Ablation Study

Comparison between BCMT and a variant without inter\-block memory \(HOnly\) to quantify the contribution of the proposed memory mechanism\.

### 4\.2Phase I: Sensitivity to Context Length

#### 4\.2\.1Objective

The first experimental phase evaluates the behavior of BCMT as the context length increases\.

Its objective is to assess BCMT’s ability to preserve its modeling performance as the context length increases, while reducing the computational cost associated with global attention\.

#### 4\.2\.2Experimental Setup

We compare the reference Dense Transformer with BCMT\-128 for different context lengths:

seq\_len∈\{128,256,512,1024\}\.\\texttt\{seq\\\_len\}\\in\\\{128,256,512,1024\\\}\.
All experiments are conducted on the WikiText200k corpus described previously\.

We report the following metrics:

- •validation loss;
- •validation perplexity;
- •training throughput, measured in tokens per second;
- •training time per epoch;
- •peak GPU memory usage\.

#### 4\.2\.3Results

Table 1:Comparison of the scaling behavior with respect to context length between the Dense Transformer and BCMT\-128\. Validation losses are reported as the mean±\\pmstandard deviation over two random seeds\.Table[1](https://arxiv.org/html/2608.13578#S4.T1)reports the results of the context\-length sensitivity study comparing the reference Dense Transformer with BCMT\-128 for context lengths ranging from 128 to 1024 tokens\. Validation losses are reported as the mean±\\pmstandard deviation over two independent random initializations\.

The results show that BCMT maintains modeling performance close to that of the Dense Transformer across all evaluated context lengths while becoming progressively more computationally efficient as the context length increases\.

For a context length of 128 tokens, the entire sequence fits within a single block, meaning that the inter\-block memory mechanism is not activated\. Under this setting, BCMT achieves a validation loss of4\.5359±0\.00384\.5359\\pm 0\.0038, slightly lower than that of the Dense Transformer \(4\.5396±0\.00184\.5396\\pm 0\.0018\)\. This result indicates that partitioning the sequence into blocks does not introduce any intrinsic degradation when the entire sequence is contained within a single block\.

As the context length increases, the performance gap between the two architectures remains moderate\. At a context length of 1024 tokens, BCMT achieves a validation loss of4\.6130±0\.00694\.6130\\pm 0\.0069, compared with4\.5752±0\.00634\.5752\\pm 0\.0063for the Dense Transformer\. Despite the absence of global attention across blocks, the exponential causal memory preserves a large portion of the information required for accurate modeling\.

At the same time, the computational efficiency gains become increasingly pronounced as the context length grows\. For a context length of 512 tokens, BCMT increases the training throughput from 154\.1K to 170\.0K tokens/s \(\+10\.3%\) while reducing the peak GPU memory usage from 6\.16 to 5\.34 GB \(\-13\.3%\)\. At 1024 tokens, these improvements become particularly significant: the training throughput reaches 204\.2K tokens/s compared with 119\.9K tokens/s for the Dense Transformer \(\+70\.3%\), while the peak GPU memory usage decreases from 14\.37 to 10\.48 GB \(\-27\.1%\)\.

These observations are consistent with the complexity analysis presented in Section[3\.10](https://arxiv.org/html/2608.13578#S3.SS10)\. When the block size is fixed, the computational complexity of local attention grows linearly with the context length, whereas dense attention remains quadratic\. Consequently, the computational advantages of BCMT become increasingly apparent as the context length increases\.

Finally, the small standard deviations observed across all experiments indicate stable optimization\. They show that the exponential causal memory mechanism does not introduce measurable optimization instabilities, regardless of the context length considered\.

### 4\.3Phase II: Comparison of BCMT Variants

#### 4\.3\.1Objective

This second phase constitutes the primary evaluation of the paper\.

Its objective is to investigate the impact of block size on the trade\-off between modeling performance and computational efficiency in order to identify the most balanced configuration\.

#### 4\.3\.2Experimental Setup

All experiments are conducted on the WikiText200k corpus with a fixed context length of

seq\_len=1024\.\\texttt\{seq\\\_len\}=1024\.
We compare the reference Dense Transformer with four BCMT variants:

- •BCMT\-64;
- •BCMT\-128;
- •BCMT\-256;
- •BCMT\-512\.

We report the following metrics:

- •validation loss;
- •validation perplexity;
- •training throughput \(tokens/s\);
- •training time per epoch;
- •peak GPU memory usage\.

#### 4\.3\.3Results

Table 2:Comparison of the architectures evaluated in Phase II on WikiText200k with a context length of 1024 tokens\. Validation losses are reported as the mean±\\pmstandard deviation over two independent random seeds\.Table[2](https://arxiv.org/html/2608.13578#S4.T2)compares the Dense Transformer with the different BCMT variants on WikiText200k using a fixed context length of 1024 tokens\. Validation losses are reported as the mean±\\pmstandard deviation over two independent random initializations\.

The results reveal a clear trade\-off between modeling performance and computational efficiency, directly controlled by the block size\. As the block size increases, predictive performance progressively approaches that of the Dense Transformer, while the gains in training throughput and memory efficiency gradually decrease\.

Among the evaluated configurations, BCMT\-256 provides the best trade\-off\. Its validation loss \(4\.5931±0\.00454\.5931\\pm 0\.0045\) remains very close to that of the Dense Transformer \(4\.5752±0\.00454\.5752\\pm 0\.0045\), with a gap of less than 0\.02\. At the same time, the training throughput increases from 119\.9K to 188\.7K tokens/s \(\+57\.4%\), while the peak GPU memory usage decreases from 14\.37 to 11\.05 GB \(\-23\.1%\)\.

At the opposite extreme, BCMT\-64 maximizes computational efficiency\. This configuration achieves the highest training throughput \(211\.6K tokens/s, corresponding to a \+76\.5% improvement\) and the lowest peak GPU memory usage \(10\.20 GB, a reduction of 29\.0%\), at the cost of a more noticeable increase in validation loss \(4\.6344±0\.00374\.6344\\pm 0\.0037\)\. These results suggest that excessively small block sizes limit the amount of contextual information available locally\.

The BCMT\-128 and BCMT\-512 variants illustrate intermediate points along this trade\-off\. BCMT\-128 retains substantial computational gains \(\+70\.4% in training throughput\) with a slightly larger degradation in modeling performance\. Conversely, BCMT\-512 achieves predictive performance very close to that of BCMT\-256 while providing more modest computational gains \(\+29\.8% in training throughput\)\. Increasing the block size therefore progressively improves modeling performance while reducing the computational benefits associated with local attention\.

Overall, all BCMT variants provide substantial computational advantages over the Dense Transformer while maintaining comparable validation performance\. These results show that the exponential causal memory effectively replaces global interactions across blocks with a compact contextual representation, without introducing a significant degradation in modeling performance\.

Finally, the smooth evolution of performance as a function of block size confirms that this parameter provides an explicit means of controlling the trade\-off between modeling performance and computational efficiency\. Among the evaluated configurations, BCMT\-256 offers the best balance between these two objectives\.

### 4\.4Phase III: Large\-Scale Training Validation

#### 4\.4\.1Objective

The third phase aims to verify that the benefits observed in the previous experiments are maintained as the amount of training data increases substantially\.

#### 4\.4\.2Experimental Setup

The Dense Transformer and BCMT\-256 are trained on the WikiText600k corpus with a fixed context length of

seq\_len=1024\.\\texttt\{seq\\\_len\}=1024\.
We report the following metrics:

- •validation loss;
- •validation perplexity;
- •training throughput \(tokens/s\);
- •training time per epoch;
- •peak GPU memory usage\.

#### 4\.4\.3Results

Table 3:Results of Phase III on WikiText600k\. Validation losses are reported as the mean±\\pmstandard deviation over two independent random seeds\.The results reported in Table[3](https://arxiv.org/html/2608.13578#S4.T3)show that the trends observed in the first two phases are preserved when the amount of training data is tripled\. Despite being trained on a corpus containing approximately 38 million tokens, BCMT retains a substantial computational advantage while maintaining modeling performance close to that of the Dense Transformer\.

After training on WikiText600k, BCMT\-256 achieves a validation loss of4\.1470±0\.00034\.1470\\pm 0\.0003, compared with4\.1115±0\.00334\.1115\\pm 0\.0033for the Dense Transformer\. Although the latter retains a slight advantage in modeling performance, the gap remains small relative to the computational gains achieved by BCMT\.

The training throughput increases from 120\.2K to 189\.7K tokens/s, corresponding to a gain of 57\.8%, while the training time per epoch decreases from 633 to 401 seconds \(\-36\.7%\)\. At the same time, peak GPU memory usage is reduced from 14\.37 to 11\.05 GB \(\-23\.1%\)\.

The observed standard deviations remain very small for both architectures, and particularly for BCMT\-256 \(±0\.0003\\pm 0\.0003\), indicating excellent training stability despite the substantial increase in training data\.

This third study confirms that the benefits of BCMT are not limited to a moderate\-scale training regime\. The improvements in training throughput and memory efficiency are preserved as the amount of training data increases substantially, demonstrating that the exponential causal memory mechanism remains both effective and stable under a more demanding training regime\.

### 4\.5Phase IV: Ablation Study

#### 4\.5\.1Objective

The final phase aims to quantify the contribution of the memory mechanism proposed in BCMT\.

More specifically, we compare the complete architecture with a simplified variant, denotedHOnly, in which both the adaptive block summaries and the exponential causal memory are entirely removed\.

The objective is to determine whether the improvements observed in the previous phases genuinely arise from the proposed inter\-block memory mechanism or merely from partitioning the sequence into local blocks\.

#### 4\.5\.2Experimental Setup

All experiments are conducted on the WikiText200k corpus with a fixed context length of

seq\_len=1024\.\\texttt\{seq\\\_len\}=1024\.
For each block size

L∈\{64,128,256,512\},L\\in\\\{64,128,256,512\\\},
we compare:

- •BCMT, which uses adaptive block summaries and exponential causal memory;
- •HOnly, in which each block is processed independently without adaptive summaries or inter\-block memory\.

In the HOnly variant, the local representations are simply concatenated after local attention:

H=Concat⁡\(H\(1\),…,H\(Nb\)\),H=\\operatorname\{Concat\}\\left\(H^\{\(1\)\},\\ldots,H^\{\(N\_\{b\}\)\}\\right\),
without any information propagation across blocks\.

#### 4\.5\.3Results

Table 4:Comparison between BCMT and its HOnly variant\. The gain corresponds to the difference in validation loss between HOnly and BCMT\. A positive value indicates an advantage in favor of BCMT\.Table[4](https://arxiv.org/html/2608.13578#S4.T4)compares BCMT with its HOnly variant, in which the inter\-block memory mechanism is removed\. The gain corresponds to the difference in validation loss between HOnly and BCMT; a positive value indicates an advantage in favor of BCMT\.

Removing the inter\-block memory consistently increases the validation loss, regardless of the block size\. This observation demonstrates that the improvements achieved by BCMT do not arise solely from partitioning the sequence into local blocks, but from the contextual propagation mechanism provided by the exponential causal memory\.

However, the impact of this memory strongly depends on the block size\. For blocks of 64 tokens, removing the memory increases the validation loss by 0\.0682\. The improvement remains substantial for a block size of 128 tokens \(\+0\.0386\), but decreases markedly as the block size increases \(\+0\.0121 for 256\-token blocks and only \+0\.0055 for 512\-token blocks\)\.

This behavior is consistent with the underlying design of BCMT\. When the blocks are small, local attention covers only a limited portion of the context, making the inter\-block memory essential for propagating information from preceding blocks\. As the block size increases, an increasing fraction of the dependencies is captured directly by local attention, naturally reducing the contribution of the memory mechanism\.

These results therefore highlight the complementarity of the two components of the architecture\. Local attention models short\-range dependencies, while the exponential causal memory effectively compensates for the loss of context induced by block\-wise processing when this limitation is most severe\.

Finally, this ablation study validates the central hypothesis underlying BCMT\. The observed performance gains do not result solely from local block processing, but from the combination of local attention and inter\-block memory\. The proposed memory mechanism is therefore the key component that preserves long\-range dependencies while maintaining the computational efficiency benefits of localized attention\.

## 5Conclusion

We have presented BCMT \(Blockwise Causal Memory Transformer\), a long\-context language modeling architecture based on an explicit separation between local interactions and global context propagation\. While interactions between tokens are modeled through dense causal self\-attention applied independently within local blocks, long\-range dependencies are propagated by means of an exponential causal memory constructed from adaptive block summaries\.

This design enables the propagation of global contextual information without relying on explicit attention between all tokens in the sequence, while preserving a simple, fully parallelizable architecture that remains compatible with standard dense self\-attention implementations\.

Experiments on WikiText show that BCMT achieves validation performance comparable to that of a Dense Transformer while significantly improving training throughput and reducing GPU memory consumption\. The ablation study confirms that these improvements arise from the proposed memory mechanism rather than from local block processing alone, highlighting the essential role of context propagation across blocks\.

These results demonstrate that a compact memory constructed from adaptive block summaries constitutes a viable alternative to dense global attention for long\-context language modeling\. More broadly, they suggest that explicitly separating the modeling of local interactions from the propagation of global context is a promising architectural principle for developing more efficient sequence models\.

Several directions for future research can be envisioned\. In particular, it would be interesting to investigate more expressive summarization mechanisms, hierarchical or multi\-scale memory structures, as well as the application of BCMT to larger models and tasks requiring stronger reasoning capabilities or long\-range information retrieval\.

We hope that this work will contribute to the development of a new family of architectures in which local modeling and global context propagation are handled by distinct yet complementary mechanisms, providing a simple, efficient, and fully parallelizable alternative to architectures based on dense global attention\.

## 6Limitations

BCMT propagates long\-range dependencies through an exponential causal memory constructed from adaptive block summaries rather than through explicit interactions between all tokens in the sequence\. This design choice significantly reduces the computational cost, but relies on a deliberately compact representation of the historical context\.

By construction, this memory is not intended to preserve a detailed representation of every observed token\. The exponential causal aggregation naturally emphasizes the most recent information while maintaining a progressively decreasing contribution from earlier blocks\. Although this strategy provides an effective global contextual signal, it does not allow for the exact retrieval of information from the distant past\. Tasks requiring precise localization or faithful reconstruction of distant events may therefore continue to benefit from explicit global attention mechanisms or specialized memory systems\.

Furthermore, the experimental evaluation presented in this work is intentionally focused\. The experiments are primarily conducted on WikiText language modeling tasks and compare BCMT with a standard Dense Transformer in order to isolate the contribution of the proposed memory mechanism\. While this methodology enables a precise assessment of BCMT itself, it does not encompass the full range of recent long\-context architectures\. A systematic comparison with models such as Transformer\-XL, Recurrent Memory Transformer \(RMT\), RetNet, RWKV, and Mamba would constitute an important next step for positioning BCMT more precisely within the current landscape of memory\-augmented architectures\.

Finally, the current version of BCMT relies on a single memory level and a fixed exponential decay factor\. Hierarchical, multi\-scale, or adaptive memory mechanisms could provide a more effective representation of dependencies spanning different temporal ranges while preserving linear computational complexity\. Exploring such extensions constitutes a natural direction for future work\.

## References

- \[1\]I\. Beltagy, M\. E\. Peters, and A\. Cohan\(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p2.1)\.
- \[2\]A\. Bulatov, Y\. Kuratov, and M\. Burtsev\(2022\)Recurrent memory transformer\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 11079–11091\.Cited by:[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p3.1)\.
- \[3\]K\. Choromanskiet al\.\(2021\)Rethinking attention with performers\.InICLR,Cited by:[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p3.1)\.
- \[4\]Z\. Dai, G\. Lai, Y\. Yang, J\. Carbonell, Q\. Le, and R\. Salakhutdinov\(2020\)Funnel\-transformer: filtering out sequential redundancy for efficient language processing\.InNeurIPS,Cited by:[§2\.3](https://arxiv.org/html/2608.13578#S2.SS3.p2.1)\.
- \[5\]Z\. Daiet al\.\(2019\)Transformer\-xl: attentive language models beyond a fixed\-length context\.InACL,Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p2.1)\.
- \[6\]J\. Dinget al\.\(2023\)LongNet: scaling transformers to 1,000,000,000 tokens\.arXiv preprint arXiv:2307\.02486\.Cited by:[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p4.1)\.
- \[7\]J\. L\. Elman\(1990\)Finding structure in time\.Cognitive Science14\(2\),pp\. 179–211\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p1.1)\.
- \[8\]A\. Gu and T\. Dao\(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p4.1)\.
- \[9\]Z\. He, Y\. Li, R\. Zhao,et al\.\(2024\)Hierarchical memory transformer for long context language processing\.arXiv preprint arXiv:2405\.06067\.Cited by:[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p3.1)\.
- \[10\]J\. Lee, Y\. Lee, J\. Kim, A\. Kosiorek, S\. Choi, and Y\. W\. Teh\(2019\)Set transformer: a framework for attention\-based permutation\-invariant neural networks\.InICML,Cited by:[§2\.3](https://arxiv.org/html/2608.13578#S2.SS3.p2.1)\.
- \[11\]D\. Marinet al\.\(2022\)Token pooling in vision transformers\.InICLR,Cited by:[§2\.3](https://arxiv.org/html/2608.13578#S2.SS3.p3.1)\.
- \[12\]P\. Neitemeieret al\.\(2025\)Hierarchical autoregressive transformers: combining byte\- and word\-level processing for robust, adaptable language models\.arXiv preprint arXiv:2501\.10322\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p2.1)\.
- \[13\]B\. Penget al\.\(2023\)RWKV: reinventing rnns for the transformer era\.arXiv preprint arXiv:2305\.13048\.Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p4.1)\.
- \[14\]H\. Pouransariet al\.\(2025\)Pretraining with hierarchical memories: separating long\-tail and common knowledge\.arXiv preprint arXiv:2510\.02375\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p2.1)\.
- \[15\]J\. Qiu, H\. Ma, O\. Levy, W\. Yih, S\. Wang, and J\. Tang\(2019\)Blockwise self\-attention for long document understanding\.arXiv preprint arXiv:1911\.02972\.Cited by:[§2\.2](https://arxiv.org/html/2608.13578#S2.SS2.p2.1)\.
- \[16\]J\. W\. Raeet al\.\(2020\)Compressive transformers for long\-range sequence modelling\.InICLR,Cited by:[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p2.1)\.
- \[17\]A\. Saret al\.\(2025\)Hierarchical resolution transformers: a wavelet\-inspired architecture for multi\-scale language understanding\.arXiv preprint arXiv:2509\.20581\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p2.1)\.
- \[18\]J\. Schmidhuber\(1992\)Learning complex, extended sequences using the principle of history compression\.Neural Computation4\(2\),pp\. 234–242\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p1.1)\.
- \[19\]D\. Sivtsov, I\. Rodkin, G\. Kuzmin, Y\. Kuratov, and I\. Oseledets\(2025\)Diagonal batching unlocks parallelism in recurrent memory transformers for long contexts\.arXiv preprint arXiv:2506\.05229\.Cited by:[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p3.1)\.
- \[20\]H\. Sun and S\. Zeng\(2025\)Hierarchical memory for high\-efficiency long\-term reasoning in llm agents\.arXiv preprint arXiv:2507\.22925\.Cited by:[§2\.4](https://arxiv.org/html/2608.13578#S2.SS4.p2.1)\.
- \[21\]Y\. Sunet al\.\(2023\)Retentive network: a successor to transformer for large language models\.arXiv preprint arXiv:2307\.08621\.Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.5](https://arxiv.org/html/2608.13578#S2.SS5.p4.1)\.
- \[22\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p1.1)\.
- \[23\]S\. Wang, B\. Z\. Li, M\. Khabsa, H\. Fang, and H\. Ma\(2020\)Linformer: self\-attention with linear complexity\.arXiv preprint arXiv:2006\.04768\.Cited by:[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p3.1)\.
- \[24\]W\. Yu, C\. Si, P\. Zhou, M\. Luo, Y\. Zhou, X\. Wang, J\. Feng, and S\. Yan\(2022\)MetaFormer is actually what you need for vision\.arXiv preprint arXiv:2210\.13452\.Cited by:[§2\.3](https://arxiv.org/html/2608.13578#S2.SS3.p3.1)\.
- \[25\]M\. Zaheer, G\. Guruganesh, A\. Dubey, J\. Ainslie, C\. Alberti, S\. Ontañón, P\. Pham, A\. Ravula, Q\. Wang, L\. Yang, and A\. Ahmed\(2020\)Big bird: transformers for longer sequences\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2608.13578#S1.p3.1),[§2\.1](https://arxiv.org/html/2608.13578#S2.SS1.p2.1)\.

Similar Articles

Multi-Branch Transformer

Hugging Face Daily Papers

The Multi-Branch Transformer (MBT) splits Transformer blocks into parallel parameterized branches with shared initial/final layers, and its Balanced variant with a shared Joint-KV cache outperforms sequential Transformers at matched compute while reducing KV-cache storage by 34% and boosting decoding throughput by up to 51.4% on an H200.