Convolution for Large Language Models

# Convolution for Large Language Models Technical Report
Source: [https://arxiv.org/html/2607.18413](https://arxiv.org/html/2607.18413)
Yuchuan Tian1, Yingte Shu1, Wei He2, Shuo Zhang2, Tianchen Zhao3, Chao Xu1, Xinghao Chen2, Yunhe Wang2, Hanting Chen2,†, Yu Wang3 1Peking University2Huawei Technologies3Tsinghua University

\(July 2026\)

###### Abstract

Large language models \(LLMs\) largely rely on Transformers, where self\-attention provides global token interaction but does not explicitly encode the locality of natural language\. We study whether lightweight depthwise convolutions can supply this local inductive bias without materially increasing model size\. Our macro\-level ablation compares convolution at 17 locations in a Qwen3 Transformer block and finds the best results when convolution is applied to the projected queries, keys, and values before attention\. A subsequent micro\-level study favors a residual depthwise convolution with kernel sizek=3k=3, without additional normalization or activation\. Across Qwen3 models and several pre\-training data budgets, this design improves the average accuracy on seven downstream benchmarks while adding less than0\.01%0\.01\\%parameters\. A representation\-level case study further suggests that the convolution makes repeated token IDs more sensitive to their immediate context\. These results support depthwise convolution as a lightweight complement to self\-attention for modeling short\-range token interactions\.

11footnotetext:Corresponding author\.## 1Introduction

Large language models \(LLMs\), including DeepSeek\(Liuet al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib33)\)and Llama\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib32)\), are predominantly built by scaling Transformer architectures\. Their basic structure remains a stack of self\-attention layers for token interaction and feed\-forward networks \(FFNs\) for token\-wise feature transformation\. Variants such as Multi\-Head Latent Attention \(MLA\) and Grouped\-Query Attention \(GQA\) improve the efficiency of attention, but local composition is still learned implicitly rather than encoded directly in the architecture\.

![Refer to caption](https://arxiv.org/html/2607.18413v1/x1.png)Figure 1:Summary of the downstream evaluation\. Left: average accuracy at the 30B\-token training budget for the evaluated Qwen3 model sizes\. Right: average accuracy of Qwen3\-4B at three training\-data budgets\. The post\-QKV depthwise convolution adds less than0\.01%0\.01\\%parameters\.Natural language contains substantial short\-range structure: adjacent tokens form compounds, phrases, and other units that contribute to sentence\-level meaning\. For example, in “neural network optimization,” the tokens “neural” and “network” form a familiar local unit\. Transformers can learn such patterns through self\-attention, but the standard architecture provides no explicit preference for them\. This motivates adding a lightweight operator whose receptive field is local by construction\.

We therefore revisit one\-dimensional convolution as a way to add locality to modern LLMs\. Unlike hybrid architectures that use convolution as a separate block, such as Conformer\(Gulatiet al\.,[2020](https://arxiv.org/html/2607.18413#bib.bib34)\), we use it as a small refinement module within the existing Transformer data path\. Our working hypothesis is that aggregating nearby features before attention can provide locally contextualized representations for subsequent long\-range interaction\.

We organize the study from macro\-level placement to micro\-level module design\. At the macro level, we insert depthwise Conv1D at 17 locations in a Qwen3 Transformer block\. The strongest configuration applies convolution after the query, key, and value projections and before attention\. At the micro level, we compare residual connections, normalization, kernel size, initialization, activation functions, and reparameterization\. Depthwise convolution operates independently on each channel, keeping the added parameter count linear in the hidden dimension\.

Across the evaluated model sizes and training\-data budgets, the resulting model improves average downstream accuracy over its matched Transformer baseline\. Figure[1](https://arxiv.org/html/2607.18413#S1.F1)summarizes these results\. We also examine a WSC example to assess how the convolution changes the representations of repeated token IDs in different local contexts\.

The main contributions are as follows:

1. 1\.We compare 17 convolution locations in a Qwen3 Transformer block and find that post\-QKV convolution performs best in our controlled placement study\.
2. 2\.We identify a compact module design: a residual depthwise Conv1D with kernel size 3 and no added normalization or activation\.
3. 3\.We evaluate the design across Qwen3 model sizes and training\-data budgets, observing higher average accuracy with less than0\.01%0\.01\\%additional parameters\.

## 2Related Work

Convolution has often been combined with Transformer architectures to supplement the global receptive field of self\-attention with local structure\. We review representative approaches in vision and language modeling\.

### 2\.1Convolution in Transformers

In computer vision, convolution is incorporated at several points in a Transformer\. LeViT\(Grahamet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib1)\)and CoAtNet\(Daiet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib2)\)use convolutional embeddings, downsampling, or convolutional blocks alongside attention\. CvT\(Wuet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib3)\)applies convolutional projections before producing attention inputs, whereas ConViT\(d’Ascoliet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib4)\)introduces a gated positional bias toward local context\. TiC\(Zhanget al\.,[2023](https://arxiv.org/html/2607.18413#bib.bib25)\)develops a multi\-head self\-attention convolution as an alternative to global attention\. Convolution also appears within the FFN in PVTv2\(Wanget al\.,[2022](https://arxiv.org/html/2607.18413#bib.bib5)\), SRFormer\(Zhouet al\.,[2023](https://arxiv.org/html/2607.18413#bib.bib8)\), and CeiT\(Yuanet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib24)\)\. Conformer\(Gulatiet al\.,[2020](https://arxiv.org/html/2607.18413#bib.bib34)\)places a convolution module between the FFN and attention sublayers, and depthwise convolution has also been used to bypass selected Transformer blocks\(Zhanget al\.,[2025](https://arxiv.org/html/2607.18413#bib.bib26)\)\. In language modeling, ConvBERT\(Jianget al\.,[2020](https://arxiv.org/html/2607.18413#bib.bib7)\)replaces a subset of attention heads with span\-based dynamic convolution\.

### 2\.2Locality in Language Models

Several language\-model architectures capture local or sequential structure through convolution\-like operations\. RWKV\(Penget al\.,[2023](https://arxiv.org/html/2607.18413#bib.bib9)\)combines linear attention with time decay, and the Gated Delta Network\(Yanget al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib10)\)uses recurrent state updates\. Multi\-Token Attention\(Golovnevaet al\.,[2025](https://arxiv.org/html/2607.18413#bib.bib11)\)allows queries to attend to contiguous token blocks\. Zoology\(Aroraet al\.,[2023](https://arxiv.org/html/2607.18413#bib.bib12)\)and work on simple linear attention\(Aroraet al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib13)\)study the role of local modeling in recall and pattern\-matching tasks\. Other approaches represent recurring phrases with scalable lookup tables\(Chenget al\.,[2026](https://arxiv.org/html/2607.18413#bib.bib14)\)or use parameterized long convolutions to capture dependencies at multiple scales\(Poliet al\.,[2023](https://arxiv.org/html/2607.18413#bib.bib15)\)\. These models differ in mechanism and scope, but they all provide alternatives to unrestricted token\-to\-token attention for modeling sequential structure\.

Our focus is narrower: we keep the Transformer backbone fixed and compare where and how a lightweight depthwise convolution can be inserted\. This controlled design study separates the effects of placement, residual structure, normalization, kernel size, initialization, activation, and reparameterization while preserving the backbone’s global attention mechanism\.

## 3Designing Convolutional Augmentation

### 3\.1A Lightweight Local Operator

Our design objective is to add a local inductive bias with little parameter overhead and minimal disruption to the Transformer block\. We therefore usedepthwise Conv1Das the basic operator and do not include dense cross\-channel mixing within the convolutional layer\.

For a standard 1D convolution with kernel sizekk, the parameter count scales as𝒪​\(k​dmodel2\)\\mathcal\{O\}\(kd\_\{\\mathrm\{model\}\}^\{2\}\)\. Qwen3\-1\.7B hasL=28L=28layers and hidden sizedmodel=2048d\_\{\\mathrm\{model\}\}=2048\. Adding a standard convolution withk=2k=2to every layer would therefore introduce approximately2×20482×28≈234\.882\\times 2048^\{2\}\\times 28\\approx 234\.88million parameters, or about13\.5%13\.5\\%of the baseline parameter count\.

Depthwise Conv1D instead operates within each channel, reducing the parameter complexity to𝒪​\(k​dmodel\)\\mathcal\{O\}\(kd\_\{\\mathrm\{model\}\}\)\. Under the same calculation, it adds approximately2×2048×28≈0\.112\\times 2048\\times 28\\approx 0\.11million parameters, corresponding to about0\.006%of the baseline\. Channel mixing remains the responsibility of the existing projection and FFN layers\.

### 3\.2Comparing Convolution Locations

Table 1:Candidate Conv1D locations in a Qwen3 Transformer block\. We report the convolution input width, average loss, WikiText\-103 perplexity, and total parameter count in millions\.![Refer to caption](https://arxiv.org/html/2607.18413v1/x2.png)Figure 2:A Qwen3 Transformer block and the 17 candidate locations for a Conv1D module\.Figure[2](https://arxiv.org/html/2607.18413#S3.F2)summarizes a Qwen3 Transformer block\(Yanget al\.,[2025](https://arxiv.org/html/2607.18413#bib.bib29)\)\. The input𝐮\\mathbf\{u\}is first normalized with RMSNorm to produce𝐡\\mathbf\{h\}\. Linear projections then produce the queries, keys, and values; QK normalization and rotary positional embeddings \(RoPE\) are applied before grouped\-query attention\. The attention output is projected and added to𝐮\\mathbf\{u\}, producing the intermediate state𝐱\\mathbf\{x\}\. A second RMSNorm and a SwiGLU FFN follow\. In the FFN, separate gate and up projections are combined by applyingSwish\\mathrm\{Swish\}to the gate branch and multiplying it element\-wise by the up branch\. A down projection maps the result back to the model dimension before the second residual addition\.

Table[1](https://arxiv.org/html/2607.18413#S3.T1)defines the 17 candidate locations\. We use Qwen3\-1\.7B as the backbone, add one Conv1D module at each location, and train each model from scratch on FineWeb\-100B\(Penedoet al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib30)\)\. We report the mean training loss over the final 10,000 iterations, perplexity on WikiText\-103\(Merityet al\.,[2018](https://arxiv.org/html/2607.18413#bib.bib31)\), and total parameter count\. For discussion, we group the locations by whether they operate directly on the inputs or outputs of attention\.

The first group contains locations before attention, after attention, or in the FFN pathway \(P1–P4 and P10–P17\)\. At these locations, convolution acts on representations either before they enter attention or after global mixing\. Most configurations improve at least one of loss or perplexity, but the gains vary substantially; P3 performs worse than the baseline on both metrics\.

The second group, P5–P9, acts directly on attention components\. P6–P8 convolve only the query, key, or value representation, whereas P5 jointly processes the concatenated QKV projections\. P9 applies convolution after attention scores have been used for aggregation\. Among these choices, P5 obtains the lowest average loss \(2\.4818\) and perplexity \(12\.85\)\. One possible interpretation is that P5 supplies local context to all three attention inputs before global aggregation; the current ablation establishes the empirical ordering but does not isolate this mechanism\.

We therefore use P5, the post\-QKV location, in the remaining design experiments\.

### 3\.3Comparing Convolution Module Designs

![Refer to caption](https://arxiv.org/html/2607.18413v1/x3.png)Figure 3:The four Conv1D module designs\. From left to right: plain convolution, convolution with a shortcut, convolution with pre\-normalization, and convolution with sandwich normalization\.#### Module design\.

We first compare four internal configurations of the depthwise convolution module \(Figure[3](https://arxiv.org/html/2607.18413#S3.F3)\): \(1\)plain convolution,Y=Conv⁡\(X\)Y=\\operatorname\{Conv\}\(X\); \(2\)convolution with a shortcut,Y=X\+Conv⁡\(X\)Y=X\+\\operatorname\{Conv\}\(X\); \(3\)convolution with pre\-normalization,Y=X\+Conv⁡\(Norm⁡\(X\)\)Y=X\+\\operatorname\{Conv\}\(\\operatorname\{Norm\}\(X\)\); and \(4\)convolution with sandwich normalization,Y=X\+Norm⁡\(Conv⁡\(Norm⁡\(X\)\)\)Y=X\+\\operatorname\{Norm\}\(\\operatorname\{Conv\}\(\\operatorname\{Norm\}\(X\)\)\)\.

Table 2:Effect of the Conv1D module design on Qwen3\-1\.7B\.All four convolutional variants improve average loss and perplexity over the “No Conv1D” baseline in Table[2](https://arxiv.org/html/2607.18413#S3.T2)\.

TheConv \+ Shortcutconfiguration gives the lowest reported perplexity, 12\.79\. The shortcut preserves the projected QKV features and lets the convolution learn a residual local correction\.

Adding normalization does not improve this result: pre\-normalization yields 12\.94 perplexity, and sandwich normalization yields 13\.33\. Because the block already normalizes its input, an additional normalization at the post\-QKV stage may alter the feature scale used by attention\. We therefore retain the simplerConv \+ Shortcutdesign for the remaining experiments\.

### 3\.4Configuration Ablations

Table 3:Effect of kernel size on Qwen3\-1\.7B\.#### Kernel size\.

We comparek∈\{2,3,4\}k\\in\\\{2,3,4\\\}for the depthwise convolution at the post\-QKV location\. As shown in Table[3](https://arxiv.org/html/2607.18413#S3.T3), all three settings improve over the baseline, andk=3k=3gives the lowest reported perplexity, 12\.79\. Increasing the kernel size tok=4k=4does not yield a further improvement\. This result is consistent with a compact kernel being sufficient for short\-range composition, although the ablation does not identify whyk=3k=3performs best\. We usek=3k=3in the remaining experiments\.

#### Initialization\.

Table 4:Effect of Conv1D initialization on Qwen3\-1\.7B\. “Random” denotes sampling the weight or bias from𝒩​\(0,0\.02\)\\mathcal\{N\}\(0,0\.02\)\.Table[4](https://arxiv.org/html/2607.18413#S3.T4)compares four initialization settings\. Random initialization of both weights and biases gives the lowest reported loss and perplexity\. The bias\-free, zero\-weight setting performs substantially worse, whereas the zero\-weight setting with a zero or random bias does not show the same degradation\. We use random initialization for both weights and biases in the remaining experiments\.

### 3\.5Additional Architectural Variants

After selecting the location, module design, kernel size, and initialization, we test three additional choices: a second convolution, nonlinear activation, and multi\-branch reparameterization\. None improves both loss and perplexity over the selected configuration\.

Table 5:Results for Qwen3\-1\.7B with Conv1D at P5, P17, or both locations\.#### Multiple convolution modules\.

We add a second Conv1D module at P17 to the selected P5 configuration\. As shown in Table[5](https://arxiv.org/html/2607.18413#S3.T5), this variant slightly increases both loss and perplexity\. The result may reflect redundant local mixing after attention and the FFN, but the ablation establishes only that adding P17 is not beneficial in this setting\. We therefore retain a single convolution at P5\.

Table 6:Effect of activation functions in the Qwen3\-1\.7B Conv1D module\.
#### Activation function\.

We test SiLU, LeakyReLU, and sigmoid activations after Conv1D\. Each activation increases both loss and perplexity relative to the linear configuration in Table[6](https://arxiv.org/html/2607.18413#S3.T6)\. We therefore use a linear convolution without an activation function\.

Table 7:Effect of convolutional reparameterization in Qwen3\-1\.7B\.
#### Reparameterization\.

We also train multi\-branch Conv1D modules with different kernel sizes and merge the branches into an equivalent convolution for inference\. Both reparameterized variants perform worse than the single\-branch configuration in Table[7](https://arxiv.org/html/2607.18413#S3.T7)\. One possible explanation is that the branches mix local patterns over different spans, although this ablation does not isolate the cause\. We therefore use a single compact Conv1D module\.

## 4Representation Analysis

![Refer to caption](https://arxiv.org/html/2607.18413v1/x4.png)Figure 4:Cosine\-similarity heatmaps of QKV representations from the Qwen3 baseline and Qwen3\+Conv1D in one WSC example\.Red boxes mark repeated token IDs at different positions\. In this example, the Conv1D model assigns lower similarity to these occurrences and produces the correct prediction\.We use one WSC example to examine whether Conv1D changes the representations of repeated token IDs that occur in different local contexts\. RoPE encodes token position, while the neighboring tokens may help determine the interpretation of each occurrence\.

For the same prompt, we extract the QKV projection outputs from the Qwen3 baseline and its Conv1D\-augmented counterpart and compute pairwise cosine similarities across positions\. For repeated token IDs, lower similarity is consistent with greater contextual differentiation, although cosine similarity alone does not establish a semantic interpretation\. The example is drawn from WSC\(Levesqueet al\.,[2012](https://arxiv.org/html/2607.18413#bib.bib23)\)\.

As shown in Figure[4](https://arxiv.org/html/2607.18413#S4.F4), the highlighted repeated\-token occurrences have lower cosine similarity in the Conv1D model than in the baseline\. This case shows that the convolution changes QKV representations according to nearby context and is consistent with the proposed role of the module\. Establishing a general mechanism would require an aggregate analysis over many examples and suitable controls\.

## 5Experiments

Table 8:Main results\.Accuracy on seven benchmarks for Qwen3 backbones trained with different token budgets, with and without Conv1D\.### 5\.1Experimental Setup

#### Causal implementation and decoding\.

To preserve autoregressive causality, we use a causal depthwise convolution\. For kernel sizekk, left padding byk−1k\-1positions ensures that the output at timettdepends only on the current token and the precedingk−1k\-1tokens\.

During incremental decoding, we maintain a convolution cache containing the features of the most recentk−1k\-1tokens\. For each new token, the convolution uses this cache and the current input; the cache is then updated as a sliding window\. The convolutional work per decoded token therefore does not grow with the decoded context length\.

#### Models and data\.

We pretrain Qwen3\-1\.7B and Qwen3\-4B from scratch\. Both backbones use grouped\-query attention, SwiGLU activations, and rotary positional embeddings\. The experiments use FineWeb\-100B\(Penedoet al\.,[2024](https://arxiv.org/html/2607.18413#bib.bib30)\)and its 30B\- and 60B\-token subsets\.

#### Hyperparameters\.

We set the training hyperparameters based on Step Law\(Liet al\.,[2025](https://arxiv.org/html/2607.18413#bib.bib27)\), with adjustments for optimization stability\. The peak learning rates are1\.0×10−31\.0\\times 10^\{\-3\}for Qwen3\-1\.7B and6\.0×10−46\.0\\times 10^\{\-4\}for Qwen3\-4B\. The learning rate is annealed after reaching its peak\. All runs use a context length of 4096 tokens and a global batch size of 256 sequences, corresponding to 1,048,576 tokens per optimization step\.

#### Evaluation\.

We evaluate the models on seven benchmarks: ARC\-Challenge and ARC\-Easy\(Clarket al\.,[2018](https://arxiv.org/html/2607.18413#bib.bib16)\), BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2607.18413#bib.bib17)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2607.18413#bib.bib19)\), PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2607.18413#bib.bib20)\), WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2607.18413#bib.bib21)\), and WSC\(Levesqueet al\.,[2012](https://arxiv.org/html/2607.18413#bib.bib23)\)\.

Table[8](https://arxiv.org/html/2607.18413#S5.T8)summarizes the results across backbone sizes and training\-token budgets\. Conv1D improves the average score in each reported setting: by 1\.99 points for Qwen3\-1\.7B trained on 30B tokens, and by 1\.57, 1\.91, and 3\.76 points for Qwen3\-4B trained on 30B, 60B, and 100B tokens, respectively\. The largest individual gains occur on WSC\.

Although the average improves in every setting, several individual benchmark scores decrease\. The larger average gain at 100B tokens suggests that the benefit may increase with the data budget for Qwen3\-4B, but additional training budgets and repeated runs would be needed to establish a general scaling trend\.

## 6Conclusion

We study depthwise Conv1D as a lightweight local inductive bias for Transformer\-based language models\. Among the 17 insertion points tested, applying convolution to the QKV projections before attention gives the lowest loss and perplexity\. The subsequent ablations favor a residual convolution with kernel sizek=3k=3, random initialization, and no additional normalization or activation\. Across the evaluated Qwen3 model sizes and training\-token budgets, this configuration improves average benchmark accuracy with a small increase in parameter count\. These results indicate that local convolution can complement self\-attention by incorporating neighboring\-token information before global interaction\. The evidence is limited to the evaluated Qwen3 configurations, and broader comparisons are needed to determine how well the result transfers to other model families and training settings\.

## References

- Zoology: measuring and improving recall in efficient language models\.arXiv preprint arXiv:2312\.04927\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- S\. Arora, S\. Eyuboglu, M\. Zhang, A\. Timalsina, S\. Alberti, D\. Zinsley, J\. Zou, A\. Rudra, and C\. Ré \(2024\)Simple linear attention language models balance the recall–throughput tradeoff\.arXiv preprint arXiv:2402\.18668\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- X\. Cheng, W\. Zeng, D\. Dai, Q\. Chen, B\. Wang, Z\. Xie, K\. Huang, X\. Yu, Z\. Hao, Y\. Li,et al\.\(2026\)Conditional memory via scalable lookup: a new axis of sparsity for large language models\.arXiv preprint arXiv:2601\.07372\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- S\. d’Ascoli, H\. Touvron, M\. L\. Leavitt, A\. S\. Morcos, G\. Biroli, and L\. Sagun \(2021\)ConViT: improving vision transformers with soft convolutional inductive biases\.InInternational Conference on Machine Learning,pp\. 2286–2296\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- Z\. Dai, H\. Liu, Q\. V\. Le, and M\. Tan \(2021\)CoAtNet: marrying convolution and attention for all data sizes\.Advances in Neural Information Processing Systems34,pp\. 3965–3977\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- O\. Golovneva, T\. Wang, J\. Weston, and S\. Sukhbaatar \(2025\)Multi\-token attention\.arXiv preprint arXiv:2504\.00927\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- B\. Graham, A\. El\-Nouby, H\. Touvron, P\. Stock, A\. Joulin, H\. Jégou, and M\. Douze \(2021\)LeViT: a vision transformer in ConvNet’s clothing for faster inference\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 12259–12269\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2607.18413#S1.p1.1)\.
- A\. Gulati, J\. Qin, C\. Chiu, N\. Parmar, Y\. Zhang, J\. Yu, W\. Han, S\. Wang, Z\. Zhang, Y\. Wu,et al\.\(2020\)Conformer: convolution\-augmented transformer for speech recognition\.arXiv preprint arXiv:2005\.08100\.Cited by:[§1](https://arxiv.org/html/2607.18413#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- Z\. Jiang, W\. Yu, D\. Zhou, Y\. Chen, J\. Feng, and S\. Yan \(2020\)ConvBERT: improving BERT with span\-based dynamic convolution\.Advances in Neural Information Processing Systems33,pp\. 12837–12848\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- H\. J\. Levesque, E\. Davis, and L\. Morgenstern \(2012\)The winograd schema challenge\.\.KR2012\(13th\),pp\. 3\.Cited by:[§4](https://arxiv.org/html/2607.18413#S4.p2.1),[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- H\. Li, W\. Zheng, Q\. Wang, H\. Zhang, Z\. Wang, S\. Xuyang, Y\. Fan, Z\. Ding, H\. Wang, N\. Ding, S\. Zhou, X\. Zhang, and D\. Jiang \(2025\)Predictable scale: part i – optimal hyperparameter scaling law in large language model pretraining\.External Links:2503\.04715,[Link](https://arxiv.org/abs/2503.04715)Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px3.p1.2)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)DeepSeek\-V3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2607.18413#S1.p1.1)\.
- S\. Merity, N\. S\. Keskar, J\. Bradbury, and R\. Socher \(2018\)Scalable language modeling: WikiText\-103 on a single GPU in 12 hours\.Proceedings of the SYSML18\.Cited by:[§3\.2](https://arxiv.org/html/2607.18413#S3.SS2.p2.1)\.
- G\. Penedo, H\. Kydlíček, A\. Lozhkov, M\. Mitchell, C\. A\. Raffel, L\. Von Werra, T\. Wolf,et al\.\(2024\)The FineWeb datasets: decanting the web for the finest text data at scale\.Advances in Neural Information Processing Systems37,pp\. 30811–30849\.Cited by:[§3\.2](https://arxiv.org/html/2607.18413#S3.SS2.p2.1),[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px2.p1.1)\.
- B\. Peng, E\. Alcaide, Q\. Anthony, A\. Albalak, S\. Arcadinho, S\. Biderman, H\. Cao, X\. Cheng, M\. Chung, M\. Grella,et al\.\(2023\)RWKV: reinventing RNNs for the transformer era\.arXiv preprint arXiv:2305\.13048\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- M\. Poli, S\. Massaroli, E\. Nguyen, D\. Y\. Fu, T\. Dao, S\. Baccus, Y\. Bengio, S\. Ermon, and C\. Ré \(2023\)Hyena hierarchy: towards larger convolutional language models\.InInternational Conference on Machine Learning,pp\. 28043–28078\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- W\. Wang, E\. Xie, X\. Li, D\. Fan, K\. Song, D\. Liang, T\. Lu, P\. Luo, and L\. Shao \(2022\)PVT v2: improved baselines with pyramid vision transformer\.Computational Visual Media8\(3\),pp\. 415–424\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- H\. Wu, B\. Xiao, N\. Codella, M\. Liu, X\. Dai, L\. Yuan, and L\. Zhang \(2021\)CvT: introducing convolutions to vision transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 22–31\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§3\.2](https://arxiv.org/html/2607.18413#S3.SS2.p1.5)\.
- S\. Yang, J\. Kautz, and A\. Hatamizadeh \(2024\)Gated delta networks: improving Mamba2 with delta rule\.arXiv preprint arXiv:2412\.06464\.Cited by:[§2\.2](https://arxiv.org/html/2607.18413#S2.SS2.p1.1)\.
- K\. Yuan, S\. Guo, Z\. Liu, A\. Zhou, F\. Yu, and W\. Wu \(2021\)Incorporating convolution designs into visual transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 579–588\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§5\.1](https://arxiv.org/html/2607.18413#S5.SS1.SSS0.Px4.p1.1)\.
- S\. Zhang, Q\. Wang, J\. Bian, and H\. Xiong \(2023\)TiC: exploring vision transformer in convolution\.arXiv preprint arXiv:2310\.04134\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- T\. Zhang, W\. Xu, B\. Luo, and G\. Wang \(2025\)Depth\-wise convolutions in vision transformers for efficient training on small datasets\.Neurocomputing617,pp\. 128998\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.
- Y\. Zhou, Z\. Li, C\. Guo, S\. Bai, M\. Cheng, and Q\. Hou \(2023\)SRFormer: permuted self\-attention for single image super\-resolution\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 12780–12791\.Cited by:[§2\.1](https://arxiv.org/html/2607.18413#S2.SS1.p1.1)\.